logcheck – a log file scanner
Every decent system generates logs. They are useful both from a forensic
and from a debug point point of view. Some systems generate huge volumes
of logs. Scanning those logs manually is both tedious and error-prone.
This calls for an automated solution. Enter logcheck.
Logcheck will scan your log files and report any entries which do not match a
list previously flagged as OK to ignore. The pattern matching is flexible and
easily extended.
Background
logcheck has been around at least 10 years. I starting
using logcheck in 1999, just about 10 years ago. Since then, logcheck
underwent quite a transformation. It once had just a handful of matching files.
Now it has over 180 files.
logcheck works by ignoring known benign patterns and reports any log file
entries that do not match those patterns. You can add to these patterns
easily.
Logcheck can scan a number of files. The list is kept in
/usr/local/etc/logcheck/logcheck.logfiles. I choose to scan these files:
# these files will be checked by logcheck # This has been tuned towards a default syslog install /var/log/messages /var/log/auth.log /var/log/maillog
NOTE: the comments are not mine.
For logcheck to scan all the files on a default FreeBSD system, you will need
to make some changes to file permissions, /etc/newsyslog.conf, and /etc/group.
See the next section for details.
Permissions
logcheck runs as the logcheck user:
# grep logcheck /etc/passwd logcheck:*:915:915:Logcheck system account:/var/db/logcheck:/usr/local/bin/bash
This user is created by the install process. I’m assuming you have the ports
tree intact.
cd /usr/ports/security/logcheck make install clean
If the cd fails, you need to do this first because you probably don’t have a
ports tree checked out:
portsnap fetch && portsnap extract
If you do not alter the permission and update some configuration files, you’ll
soon get one of these emails:
To: root@ngaio.example.org Subject: Logcheck: ngaio.example.org 2009-11-20 12:02 exiting due to errors Message-Id: <20091120120201.7ACFF17104@ngaio.example.org> Date: Fri, 20 Nov 2009 12:02:01 +0000 (GMT) From: logcheck@ngaio.example.org (Logcheck system account) Warning: If you are seeing this message, your log files may not have been checked! Details: Could not run logtail or save output Check temporary directory: /tmp/logcheck.ZOjfJO Also verify that the logcheck user can read all files referenced in /etc/logcheck/logcheck.logfiles! declare -x HOME="/var/db/logcheck" declare -x LOGNAME="logcheck" declare -x MAILTO="root" declare -x OLDPWD declare -x PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin" declare -x PWD="/var/db/logcheck" declare -x SHELL="/bin/sh" declare -x SHLVL="1" declare -x USER="logcheck"
The email contains the wrong location for the file. It is assuming an
installation location which has been changed at configuration/install time
and /usr/local/sbin/logcheck has not been refreshed accordingly.
I have submitted a patch for that.
Check the permissions for the files listed in
/usr/local/etc/logcheck/logcheck.logfiles:
# ls -l /var/log/messages /var/log/auth.log /var/log/maillog -rw------- 1 root wheel 6564 Nov 28 21:13 /var/log/auth.log -rw-r----- 1 root wheel 60 Nov 28 00:00 /var/log/maillog -rw-r--r-- 1 root wheel 83127 Nov 28 22:00 /var/log/messages
As you can see, the logcheck user will be unable to read auth.log and maillog.
We can change that.
# chgrp logcheck /var/log/auth.log /var/log/maillog # chmod g+r /var/log/auth.log # ls -l /var/log/messages /var/log/auth.log /var/log/maillog -rw-r----- 1 root logcheck 6564 Nov 28 21:13 /var/log/auth.log -rw-r----- 1 root logcheck 60 Nov 28 00:00 /var/log/maillog -rw-r--r-- 1 root wheel 83277 Nov 28 22:05 /var/log/messages
logcheck will now be able to read the files, but as you know, these files are
rotated by newsyslog.conf. So let’s see the entries for them:
# egrep "/var/log/auth.log|/var/log/maillog" /etc/newsyslog.conf /var/log/auth.log 600 7 100 * JC /var/log/maillog 640 7 * @T00 JC
The above is before my changes, the following is after:
# egrep "/var/log/auth.log|/var/log/maillog" /etc/newsyslog.conf /var/log/auth.log root:logcheck 640 7 100 * JC /var/log/maillog root:logcheck 640 7 * @T00 JC
Note that you have to add the root:logcheck to both *and* change the mode
for auth.log to 640.
Recent versions of logcheck default the outgoing email to the logcheck user.
To get these emails sent to myself, I added this entry to /etc/mail/aliases:
logcheck: dan
Customizations
logcheck will initally produce notices about things you do not care to see
again. They are normal for your system and they do not need to be brought
to your attention again. You can train logcheck to ignore these items.
You will see both System Events and Security Events emails.
For example:
Security Events =-=-=-=-=-=-=-= Nov 28 16:37:55 dbclone postgres[93778]: [2-1] ERROR: table "mac" does not exist System Events =-=-=-=-=-=-= Nov 28 16:28:22 dbclone bacula-dir: Shutting down Bacula service: localhost-dir ...
These items are normal for this system. It is used for
Bacula regression testing. For the Security Events,
I created /usr/local/etc/logcheck/violations.ignore.d/local-postgres with the following
contents:
# grep mac /usr/local/etc/logcheck/violations.ignore.d/local-postgres ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ postgres\[[0-9]+\]: \[[0-9]+\-[0-9]+\] ERROR: table "mac" does not exist
That preamble seems like a lot. But I grabbed it from logcheck-postgres. The logcheck
project recommends that you put your own customizations into files prefixed with local-
so they are easily identified. logcheck itself does not care.
For the System Event, I added this entry to /usr/local/etc/logcheck/ignore.d.server/local-postgres
bacula-fd: Shutting down Bacula service: localhost-fd
Notice that my System Event exceptions are specified in the ignore.d.server directory.
This is because I selected the following option in /usr/local/etc/logcheck/logcheck.conf:
REPORTLEVEL="server"
If you are using “workstation”, you would add your file to the ignore.d.workstation
directory.
EOF
There you go. That should get you started with logcheck. I’ve been using it for
10 years. It’s a great idea. I hope and trust it will save you a great deal
of ready. Best wishes.
If you see the following error, install textproc/docbook-to-man
The clue to this error was a ‘recent’ commit by glarkin: http://www.freshports.org/commit.php?category=security&port=logcheck&files=yes&message_id=200905262025.n4QKPxAO031130@repoman.freebsd.org
Looking at the diffs for that commit gave me the idea to install docbook-to-man manually.
# make
===> Extracting for logcheck-1.2.54_3
=> MD5 Checksum OK for logcheck_1.2.54.tar.gz.
=> SHA256 Checksum OK for logcheck_1.2.54.tar.gz.
===> logcheck-1.2.54_3 depends on file: /usr/local/bin/perl5.8.9 – found
===> Patching for logcheck-1.2.54_3
===> logcheck-1.2.54_3 depends on file: /usr/local/bin/perl5.8.9 – found
===> Applying FreeBSD patches for logcheck-1.2.54_3
===> logcheck-1.2.54_3 depends on executable: docbook2man – found
===> logcheck-1.2.54_3 depends on file: /usr/local/bin/perl5.8.9 – found
===> Configuring for logcheck-1.2.54_3
===> Building for logcheck-1.2.54_3
/usr/bin/sed -i.bak -e ‘s!/var/log/syslog!/var/log/messages!’ /usr/ports/securi
ty/logcheck/work/logcheck-1.2.54/etc/logcheck.logfiles
/usr/bin/sed -i.bak -e ‘s!/etc/logcheck!/usr/local/etc/logcheck!’ -e ‘s!/usr/sh
are/doc/logcheck-database/README.logcheck-database.gz!/usr/local/share/doc/logch
eck/README.logcheck-database!’ /usr/ports/security/logcheck/work/logcheck-1.2.5
4/docs/logcheck.sgml
cd /usr/ports/security/logcheck/work/logcheck-1.2.54/docs && docbook2man -s /us
r/local/share/docbook2X/xslt/man/docbook.xsl –sgml logcheck.sgml 2> /dev/null
&& /bin/mv Logcheck.8 logcheck.8
*** Error code 255
Stop in /usr/ports/security/logcheck.
—
The Man Behind The Curtain
Actually, the bits between ‘Building for logcheck’ and ‘*** Error’ are only displayed if you remove the @ which appear in the do-build section of the Makefile. I made these changes when trying to debug the problem.
—
The Man Behind The Curtain