OpenBSD and syslog-ng
Published: 09/09/2009
Logs are everywhere. They hide within the Windows Event Viewer, syslog silently churns away debugs under /var, applications are always banging against the disk with file appends, legal forensics heavily depends on them, and when we as admins need to investigate a problem on the network, something needs to tell us which packets passed through an interface and which ones got dropped by which firewall rule.
But there's a reason why you don't leave bare logs sitting in their original location. If your firewall gets hacked, you can't trust the integrity of the logs on that device anymore because the attacker could have potentially modified them. The only solution is to stream logging information as events happen from the end devices to a remote centralized log server that's well-protected within the core of your company network that only a delegated, trusted few can access. Security personnel with automatic weapons wearing shirts that scream "Layer 1 Defense" should stand guard.
The built-in syslog facility within the Unix realm works, but isn't elegant. Everything is piped into the same set of log files. That's fine if you have only one appliance sending log data, but what happens if you have dozens, hundreds, or thousands of firewalls, routers, switches, VPN gateways, and who knows what else? It's better to have a log server that recognizes each device and gives them their own dedicated log file that can be tailed on-demand. UDP 514, here we come!
Set up an OpenBSD server with a lot of disk space. i386 hardware is used in this article's example along with OpenBSD 4.5 and syslog-ng 2.1.3. Make sure that your corporate backup strategy is in place. Verify that all your devices are synced to a common time source. Then install syslog-ng, the next-generation replacement for syslog. This all should take you less than five minutes. Tell management it took a few hours (unless they're watching over your shoulder). And if you're a consultant being paid by the hour, type slow and don't copy and paste from here.
Install, configure syslog-ng, reboot, start swallowing log data
Install the syslog-ng package and update the system startup file so the original syslog is disabled (by commenting out the specified lines) and enabling syslog-ng by adding two lines below it:
$ sudo pkg_add -v ftp://ftp.openbsd.org/pub/OpenBSD/4.5/packages/i386/syslog-ng-2.1.3.tgz
$ sudo vi /etc/rc
# syslogd_flags="${syslogd_flags} -a /var/named/dev/log"
# syslogd_flags="${syslogd_flags} -a /var/empty/dev/log"
# syslogd ${syslogd_flags}
syslog_ng_flags="-p /var/run/syslog-ng.pid"
/usr/local/sbin/syslog-ng ${syslog_ng_flags}
$ sudo vi /etc/rc.conf
# syslogd_flags=
syslog_ng_flags=
$ sudo cp /etc/syslog-ng/syslog-ng.conf /etc/syslog-ng/syslog-ng.conf.original
|
Edit the syslog-ng config file to save all logging data under /var/log/mycompany (change the values in red to your preference):
$ sudo vi /etc/syslog-ng/syslog-ng.conf
options { sync (0);
time_reopen (10);
log_fifo_size (1000);
long_hostnames (off);
use_dns (yes);
use_fqdn (no);
create_dirs (no);
keep_hostname (yes);
};
source local { internal(); pipe("/dev/klog" log_prefix("kernel: ")); unix-dgram("/dev/log"); };
source net { udp(); };
destination d_cons { file("/dev/console"); };
destination d_mesg { file("/var/log/messages"); };
destination d_auth { file("/var/log/secure"); };
destination d_mail { file("/var/log/maillog"); };
destination d_spol { file("/var/log/spooler"); };
destination d_boot { file("/var/log/boot.log"); };
destination d_cron { file("/var/log/cron"); };
destination d_mlal { usertty("*"); };
destination d_network_hosts { file("/var/log/mycompany/$HOST.log"); };
filter f_filter1 { facility(kern); };
filter f_filter2 { level(info) and
not (facility(mail)
or facility(authpriv) or facility(cron)); };
filter f_filter3 { facility(authpriv); };
filter f_filter4 { facility(mail); };
filter f_filter5 { level(emerg); };
filter f_filter6 { facility(uucp) or
(facility(news) and level(crit)); };
filter f_filter7 { facility(local7); };
filter f_filter8 { facility(cron); };
filter f_net_hosts { level(info, notice, warn, crit, err, debug); };
log { source(local); filter(f_filter2); destination(d_mesg); };
log { source(local); filter(f_filter3); destination(d_auth); };
log { source(local); filter(f_filter4); destination(d_mail); };
log { source(local); filter(f_filter5); destination(d_mlal); };
log { source(local); filter(f_filter6); destination(d_spol); };
log { source(local); filter(f_filter7); destination(d_boot); };
log { source(local); filter(f_filter8); destination(d_cron); };
log { source(net); filter(f_net_hosts); destination(d_network_hosts); };
|
Finish up and reboot.
$ sudo mkdir -p /var/log/mycompany $ sudo kill -9 `cat /var/run/syslog.pid` $ sudo reboot |
Note - while these steps have worked for me in earlier versions of OpenBSD, whenever version 4.5 boots a message appears while the daemons are started such as "can't load library libiconv.so.5.0." Although I'm unsure why this is so, simply adding the following line in /etc/rc.local solves the problem:
/usr/local/sbin/syslog-ng -f /etc/syslog-ng/syslog-ng.conf -p /var/run/syslog-ng.pid |
Now point your various network devices and *nix hosts to send logs to this server. Based on the configuration above, log filenames will include the basic hostname of the sending host, such as myfirewall01.log. If the syslog-ng server cannot resolve the name via reverse-DNS lookup, then the the IP address will be used in its place such as 1.2.3.4.log.
Once logging collection is started, it's recommended to create log rotation jobs. Otherwise, you can now tail individual logs in real-time as events happen on the nodes, assuming the sending nodes send all event data immediately as they occur. I have seen some appliances bundle everything into a queue for a few seconds before sending syslog data over.
More stuff you cannot forget to do
We're talking about OpenBSD here - you did remember to set up pf, right? You'll need to allow UDP 514 (or any other syslog port that your devices use) as well as some method of SSHing into the server from the appropriate management network for the occasional log tailing when yet another IT fire needs to be put out.
Go back to the main articles list.