About syslog-ng

Syslog-ng is a logging software developed by Hungarian company BalaBit. Very powerful and flexible tool Syslog-ng (new generation) lets you collect log by local or remote machines and is an alternative to syslogd (another logging software), because it has more functions and extends the normal use of syslog protocol.

Major additions are:

• use of TCP for sending network messages
• TLS support

other features are:

• macros
• filters
• encryption with TLS (starting from syslog-ng-3.0.1)
• ability to interface with MySQL databases
• regular expressions

Last version is 3.1.2, you can get it here:http://www.balabit.com/downloads/files/syslog-ng/sources/3.1.2/source/syslog-ng_3.1.2.tar.gz.

Facilities and Severities

Syslog-ng behavior (as well as others log systems, such syslogd) is dictated by facilities, that represents the source of log messages, and from severities that represents the severity of a message.

In syslog-ng we've these facilities:

Code Name 	Facility
0	kern		kernel messages
1	user		user-level messages
2	mail		mail system
3	daemon	system daemons
4	auth		security/authorization messages
5	syslog	messages generated internally by syslogd
6	lpr		line printer subsystem
7	news		network news subsystem
8	uucp		UUCP subsystem
9	cron		clock daemon
10	auth		security/authorization messages
11	ftp		FTP daemon
12			NTP subsystem
13			log audit
14			log alert
15	cron		clock daemon
16-23	local0-local7	locally used facilities (local0-local7)

And these severities:

Code	 Severity
0	 Emergency: system is unusable
1	 Alert: action must be taken immediately
2	 Critical: critical conditions
3	 Error: error conditions
4	 Warning: warning conditions
5	 Notice: normal but significant condition
6	 Informational: informational messages
7	 Debug: debug-level messages

For every generated log message, will be assigned one facility and a level of severity.

These along filters let us to run a very large number of operations on log messages, like divide messages by severity or facility and so on.

In syslog-ng you can use both numeric codes and name from tables above for represent facilities or severities.

Installation

Syslog-ng is available for all modern distributions and for its installation please refer to the automated package tool of your distribution, just search for syslog-ng.

It's suggested to keep only one logging software in the system, so uninstall or disable the other software if this is not syslog-ng.

Some packages tool, like apt-get propose to uninstall the old log software allowing the installation of a new one.

Server configuration

Configuration file is /etc/syslog-ng.conf, and we can divide it by section:

• options => global syslog-ng options
• sources => source where messages are taken
• desinations => destination where write messages
• filters => filter to execute on log messages
• logs => link source, filters and destination and execute logging operations

Options

In this section are present global options of syslog-ng and they're useful for define software behavior.

We've a lot of options here and the most common are:

create_dirs(val)	 => Creates destination directories for log files; val => yes | no
dir_group(val)	 	 => Set group on created directories; val => group id
dir_owner(val)	 	 => Set owner on created directories; val => user id
dir_perm(val) 	 	 => Set permissions on created dircetories; val => permissions (e.g. 0755)
group(val) 	 	 => Set group on created log files; val => group id
owner(val) 	 	 => Set owner on created log files; val => owner id
perm(val) 	 	 => Set permissions on created file: val => permissions (e.g 0755)
check_hostname(val) 	 => Checks if hostname contains valid character; val => yes | no

Other options are used to manage DNS, log file and timezone.

You can write these options in both source and destination directives.

We set only this option in our server:

options{
	create_dirs(yes);
}

With create_dirs syslog will create the directory when they're necessaries. It can be applied for example when we have a log folder per host to collect log files.

Sources

Sources are source of log messages and we can create how many sources we want in a single configuration file with different options.

The syntax for create a source is the following:

source source_name 
{ 
	source_driver(params), 
	source_driver(params), 
	...... 
}

The possible source_driver from where collect a log message are:

internal()				 => Internal messages of syslog-ng
file()					=> Collect log from text file
unix-stream()				=> Listen on a SOCK_STREAM (most used on linux) such as /dev/log
unix-dgram()				=> Like unix-stream(), but opend socket is of type SOCK_DGRAM(BSD) like /var/log/run
tcp(), tcp6()				=> Listen on a TCP port, default 514, for incoming log messages
udp(), udp6()				=> Like tcp() but uses UDP, default port 514
syslog()				=> Enable receiving log messages with new syslog protocol syslog (http://tools.ietf.org/html/rfc5424)
program()				=> Read stdout of a program
pipe()					=> Listen on a named pipe and collect incoming messages
sun-stream(), sun-streams()		=> Used on solaris machine for collect log

Our source are the following:

## Source

source src_local {
	 internal();
	 unix-stream("/dev/log");
	 file("/proc/kmsg" program_override("kernel: "));
}

source src_remote{
	 tcp( max-connections(20); );
	 udp();
}

We declare a source directive called src_local for collect log messages of the local machine. This source can read messages internally to syslog-ng and from the system stream /dev/log. The "file" source collect kernel messages from /proc/kmsg, and appends "kernel: " at the beginning of the messages.

Second source src_remote is used to collect log from remote hosts through TCP, or through UDP if there are some client that use syslogd.

Destinations

Destinations let us to write log messages from a source in a specified location.

Correct syntax for write a destination rule is the following:

destination destination_name {
	destination_driver(params),
	destination_driver(params), 
	...
}

Like sources we can include how many destinations we need followed by the parameter we want.

The possible destinations (destination_driver) are similar to the sources:

file()					=> Wrote log messages in a text file
unix-stream()				=> Send messages to SOCK_STREAM socket, most used in linux environment
unix-dgram()				=> Send messages to SOCK_DGRAM socket, used in BSD
tcp(), tcp6()				=> Send messages to a TCP port
udp(), udp6()				=> Like tcp(), but with UDP protocol
syslog()				=> Send messages using new syslog protocol to an host
program()				=> Executes specified program and send to its stind log messages 
pipe(), fifo()				=> Wrote messages on a specified pipe

Our destinations:

## Destinations

destination dst_messages {
    file("/var/log/log-ng/messages");
};
destination dst_syslog {
    file("/var/log/log-ng/syslog");
};

destination dst_remote_sysdaemon {
    file("/var/log/log-ng/$HOST/$FACILITY.log");
};

destination dst_remote_internal {
    file("/var/log/log-ng/$HOST/$FACILITY.log");
};

destination dst_remote_auth {
    file("/var/log/log-ng/$HOST/$FACILITY.log");
};

destination dst_remote_cron {
    file("/var/log/log-ng/$HOST/$FACILITY.log");
};

destination dst_remote_authpriv {
    file("/var/log/log-ng/$HOST/$FACILITY.log");
};

destination dst_remote_kernel {
    file("/var/log/log-ng/$HOST/$FACILITY.log");
};

destination dst_remote_user {
    file("/var/log/log-ng/$HOST/$FACILITY.log");
};

In our case the first two destinations are going to be used for local machine log, while the others are used for remote machines log. We've also created a destination for each facilities we want to log for make order.

In this configuration the interesting thing are the macro $HOST and $FACILITY, that add more flexibility, letting us create a folder for each host in the network we want log, and also a log file per facility.

Filters

With filters we've a better possibility to manage incoming messages, for example we can direct what we want where we want. This let us to work with log messages in a better and simple way.

The right way for create a filter is the following:

filter filter_name {
	expression_1
	expression_2
	...
};

We can substitute "expression" with functions or use Boolean operators and, or, not.

Here we've an example of a filter:

filter test { 
	match("linux" value("MESSAGE")) 
};

This filter select all messages that contains the word "linux".

Useful filter function:

facility(facility)	 	 => filters through facility, you can use numeric value and name of a facility
host(reg_exp)	 	 	 => with a regular expression you can filter hosts
level(level)	 	 	 => filters with severities, with ".." you can specify a range of severities 
netmask(ip/mask)	 	 => filter using netmask
match(reg_exp)	 	 	 => Use a regular expression to filter

These are the filters we will create:

## Log

## Local

filter info_to_notice{
    	level(info .. notice);
};

filter warn_to_emergency{
    	level(warn .. emerg);
};

## Remote

filter flt_sysdaemon{
    	facility(daemon);
};

filter flt_internal{
    	facility(syslog);
};

filter flt_auth{
    	facility(auth);
};

filter flt_cron{
    	facility(cron);
};

filter flt_authpriv{
    	facility(authpriv);
};

filter flt_kern{
    	facility(kern);
};

filter flt_user{
    	 facility(user);
};

First two filters are for local messages: we will filter all messages with a priority from info to notice, and from warning to emergency. With this configuration we will have two log file, one with most important advices and one with less important.

In the others filter we use facilities as a filter, in order to obtain a unique log file for a given facility.

Log

Last section of configuration file, contains log directives, that let us to link the sources, destinations and filters seen above together.

A log directive can be::

log {
	source(source_name);
	filter(filter_name);
	destination(destination_name);
}

We must at least write the source and destination of a log, and we can write how many sources, destinations and filters we want.

Filters must be placed before destination otherwise nothing will be filtered.

We have defined these log for our server:

## Log

## Local

log {
	source(src_local);
	filter(info_to_notice);    
	destination(dst_messages);   
};

log {
	source(src_local);
	filter(warn_to_emergency);
	destination(dst_syslog);
};

## Remote

log {
    source(src_remote);
 	filter(flt_auth);
    destination(dst_remote_auth);
};

log {
    source(src_remote);
    filter(flt_internal);
    destination(dst_remote_internal);
};

log {
    source(src_remote);
    filter(flt_cron);
    destination(dst_remote_cron);
};

log {
    	source(src_remote);
    	filter(flt_sysdaemon);
    	destination(dst_remote_sysdaemon);
};

log {
    	source(src_remote);
    	filter(flt_authpriv);
    	destination(dst_remote_authpriv);
};

log {
    	source(src_remote);
    	filter(flt_kern);
    	destination(dst_remote_kernel);
};

log {
    	source(src_remote);
    	filter(flt_user);
    	destination(dst_remote_user);
};

Like the others, first two log are for messages of our local machine, divided in two files with different level of warning.

Others log directives are for remote logging and execute filter according to the facility.

The source is always the network, while change filter and destination.

Client configuration

In the previous part we've configured the server side of our syslog-ng environment, now it's time to configure client that will use our server as end point for log messages, so we must install syslog-ng on the client and configure syslog-ng.conf:

@version: 3.0

## Source

source src_local {
    internal();
    unix-stream("/dev/log");
    file("/proc/kmsg" program_override("kernel: "));
};

## Destination 

destination tcp_dest {
    tcp("192.168.1.27");
};

destination dst_messages {
    file("/var/log/log-ng/messages");
};

destination dst_syslog {
    file("/var/log/log-ng/syslog");
};

## Filter

filter info_to_notice{
    level(info .. notice);
};

filter warn_to_emergency{
    level(warn .. emerg);
};

## Log

log {
    source(src_local);
    filter(info_to_notice);
    destination(dst_messages);
};

log {
    source(src_local);
    filter(warn_to_emergency);
    destination(dst_syslog);
};

log{
    source(src_local);
    destination(tcp_dest);
};

The configuration in this file create two log file for local generated messages on the client, one file with important log and one for less important. Then it sends all the log messages to the remote server.

We can copy this configuration on each client and then restart the syslog-ng service.

If you use syslogd on the client, we can send messages to our syslog-ng server adding this line to syslog.conf:

*.*	@syslog-ng_ip_or_server_name

That will send all messages of every facility and severity to syslog-ng server.

Macro

There is the possibility to use marco in the configuration file of syslog-ng to create dynamic log files.

Here are some useful macros:

$FACILITY	 => facility name
$FACILITY_NUM	 => numeric code of the facility
$PRIORITY	 => severity name, for example "emerg"
$DAY	 	 => day
$MONTH	 	 => month
$HOUR 	 	 => hour
$MIN 	 	 => minute
$YEAR 	 	 => year
$WEEKDAY 	 => Name of the day, in three character format, like "Fri"
$HOST 	 	 => hostname

Server configuration file

@version: 3.0

## Options

options {
    create_dirs(yes);
};

## Sources

source src_local {
    internal();
    unix-stream("/dev/log");
    file("/proc/kmsg" program_override("kernel: "));
};

source src_remote {
	 tcp(max-connections(20));
	 udp();
};

## Destinations

destination dst_messages {
    file("/var/log/log-ng/messages");
};

destination dst_syslog {
    file("/var/log/log-ng/syslog");
};

destination dst_remote_sysdaemon {
    file("/var/log/log-ng/$HOST/$FACILITY.log");
};

destination dst_remote_internal {
    file("/var/log/log-ng/$HOST/$FACILITY.log");
};

destination dst_remote_auth {
    file("/var/log/log-ng/$HOST/$FACILITY.log");
};

destination dst_remote_cron {
    file("/var/log/log-ng/$HOST/$FACILITY.log");
};

destination dst_remote_authpriv {
    file("/var/log/log-ng/$HOST/$FACILITY.log");
};

destination dst_remote_kernel {
    file("/var/log/log-ng/$HOST/$FACILITY.log");
};

destination dst_remote_user {
    file("/var/log/log-ng/$HOST/$FACILITY.log");
};

## Filters

filter flt_sysdaemon{
    facility(daemon);
};

filter flt_internal{
    facility(syslog);
};

filter flt_auth{
    facility(auth);
};

filter flt_cron{
    facility(cron);
};

filter flt_authpriv{
    facility(authpriv);
};

filter flt_kern{
    facility(kern);
};

filter flt_user{
    facility(user);
};

filter info_to_notice{
    level(info .. notice);
};

filter warn_to_emergency{
    level(warn .. emerg);
};

## Log

## Local

log {
    source(src_local);
    filter(info_to_notice);
    destination(dst_messages);
};

log {
    source(src_local);
    filter(warn_to_emergency);
    destination(dst_syslog);
};

## Remote

log {
    source(src_remote);
    filter(flt_auth);
    destination(dst_remote_auth);
};

log {
    source(src_remote);
    filter(flt_internal);
    destination(dst_remote_internal);
};

log {
    source(src_remote);
    filter(flt_cron);
    destination(dst_remote_cron);
};

log {
    source(src_remote);
    filter(flt_sysdaemon);
    destination(dst_remote_sysdaemon);
};

log {
    source(src_remote);
    filter(flt_authpriv);
    destination(dst_remote_authpriv);
};

log {
    source(src_remote);
    filter(flt_kern);
    destination(dst_remote_kernel);
};

log {
    source(src_remote);
    filter(flt_user);
    destination(dst_remote_user);
};

TLS

Starting from version 3.0.1, it's fully implemented support for sending encrypted messages with TLS.

This add an additional security layer and complicates messages interpretation by criminals.

For do this, we've to generate and sign a certificate on the server that will be copied on others client, this to verify that we've the authorization to send messages.

Server configuration

We generate a private key on the server:

openssl genrsa -out srv_private.key 1024

Now we create a certificate and next sign it with our private key:

openssl req -new -x509 -key srv_private.key -out crt_server.pem -days 365

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CH
State or Province Name (full name) [Some-State]:Ticino
Locality Name (eg, city) []:Lugano
Organization Name (eg, company) [Internet Widgits Pty Ltd]:OsGate
Organizational Unit Name (eg, section) []:IT
Common Name (eg, YOUR name) []:192.168.1.27
Email Address []:snake@test.local

It's important to insert in "Common Name" the IP or the name of our syslog-ng server.

After key and certificate operations, we modify the configuration file of syslog-ng in the source directive like this:

source src_remote {
	 tcp(max-connections(20)
	 tls( key_file("/opt/syslog-ng/etc/key.d/srv_private.key") cert_file("/opt/syslog-ng/etc/cert.d/crt_server.pem")
 peer_verify("optional-trusted")) 
	 udp();
};

We've added two folders, /opt/syslog-ng/key.d/ and /opt/syslog-ng/etc/cert.d, one for private key and one for certificate; after this create and copy the equivalent file to their respective folders.

Peer_verifiy rules is used to verifiy other peer, and with "optional-trusted" options we avoid that trusted client send to us another certificate in addition to the one that they have already.

Client configuration

We create the folder that will contain the certificate on the client, then we will copy the certificate from server to the client:

mkdir /opt/syslog-ng/ca.d
scp crt_server.pem root@client1:/opt/syslog-ng/ca.d

After this, modify syslog-ng.conf:

destination tcp_dest {
    tcp("192.168.1.27" port(514) tls(ca_dir("/opt/syslog-ng/etc/ca.d")));
};

As final step, in certificate directory type::

openssl x509 -noout -hash -in crt_server.pem

And create a symbolic link with the hash number visualized:

ln -s crt_server.pem 6d2962a8.0

Restart syslog-ng server and start enjoying TLS.

Utils

logger => invoced with -p "facility.severity" "message" option, send to daemon that manage log a message. Useful in testing phase.