logcheck – a log file scanner

logcheck – a log file scanner

Every decent system generates logs. They are useful both from a forensic
and from a debug point point of view. Some systems generate huge volumes
of logs. Scanning those logs manually is both tedious and error-prone.
This calls for an automated solution. Enter logcheck.
Logcheck will scan your log files and report any entries which do not match a
list previously flagged as OK to ignore. The pattern matching is flexible and
easily extended.

Background

logcheck has been around at least 10 years. I starting
using logcheck in 1999, just about 10 years ago. Since then, logcheck
underwent quite a transformation. It once had just a handful of matching files.
Now it has over 180 files.

logcheck works by ignoring known benign patterns and reports any log file
entries that do not match those patterns. You can add to these patterns
easily.

Logcheck can scan a number of files. The list is kept in
/usr/local/etc/logcheck/logcheck.logfiles. I choose to scan these files:

# these files will be checked by logcheck
# This has been tuned towards a default syslog install
/var/log/messages
/var/log/auth.log
/var/log/maillog

NOTE: the comments are not mine.

For logcheck to scan all the files on a default FreeBSD system, you will need
to make some changes to file permissions, /etc/newsyslog.conf, and /etc/group.
See the next section for details.

Permissions

logcheck runs as the logcheck user:

# grep logcheck /etc/passwd
logcheck:*:915:915:Logcheck system account:/var/db/logcheck:/usr/local/bin/bash

This user is created by the install process. I’m assuming you have the ports
tree intact.

cd /usr/ports/security/logcheck
make install clean

If the cd fails, you need to do this first because you probably don’t have a
ports tree checked out:

portsnap fetch && portsnap extract

If you do not alter the permission and update some configuration files, you’ll
soon get one of these emails:

To: root@ngaio.example.org
Subject: Logcheck: ngaio.example.org 2009-11-20 12:02 exiting due to errors
Message-Id: <20091120120201.7ACFF17104@ngaio.example.org>
Date: Fri, 20 Nov 2009 12:02:01 +0000 (GMT)
From: logcheck@ngaio.example.org (Logcheck system account)

Warning: If you are seeing this message, your log files may not have been
checked!

Details:
Could not run logtail or save output

Check temporary directory: /tmp/logcheck.ZOjfJO

Also verify that the logcheck user can read all files referenced in
/etc/logcheck/logcheck.logfiles!

declare -x HOME="/var/db/logcheck"
declare -x LOGNAME="logcheck"
declare -x MAILTO="root"
declare -x OLDPWD
declare -x PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin"
declare -x PWD="/var/db/logcheck"
declare -x SHELL="/bin/sh"
declare -x SHLVL="1"
declare -x USER="logcheck"

The email contains the wrong location for the file. It is assuming an
installation location which has been changed at configuration/install time
and /usr/local/sbin/logcheck has not been refreshed accordingly.
I have submitted a patch for that.
Check the permissions for the files listed in
/usr/local/etc/logcheck/logcheck.logfiles:

# ls -l /var/log/messages /var/log/auth.log /var/log/maillog
-rw-------  1 root  wheel   6564 Nov 28 21:13 /var/log/auth.log
-rw-r-----  1 root  wheel     60 Nov 28 00:00 /var/log/maillog
-rw-r--r--  1 root  wheel  83127 Nov 28 22:00 /var/log/messages

As you can see, the logcheck user will be unable to read auth.log and maillog.
We can change that.

# chgrp logcheck /var/log/auth.log /var/log/maillog
# chmod g+r /var/log/auth.log
# ls -l /var/log/messages /var/log/auth.log /var/log/maillog
-rw-r-----  1 root  logcheck   6564 Nov 28 21:13 /var/log/auth.log
-rw-r-----  1 root  logcheck     60 Nov 28 00:00 /var/log/maillog
-rw-r--r--  1 root  wheel     83277 Nov 28 22:05 /var/log/messages

logcheck will now be able to read the files, but as you know, these files are
rotated by newsyslog.conf. So let’s see the entries for them:

# egrep "/var/log/auth.log|/var/log/maillog" /etc/newsyslog.conf
/var/log/auth.log                       600  7     100  *     JC
/var/log/maillog                        640  7     *    @T00  JC

The above is before my changes, the following is after:

# egrep "/var/log/auth.log|/var/log/maillog" /etc/newsyslog.conf
/var/log/auth.log       root:logcheck   640  7     100  *     JC
/var/log/maillog        root:logcheck   640  7     *    @T00  JC

Note that you have to add the root:logcheck to both *and* change the mode
for auth.log to 640.

email

Recent versions of logcheck default the outgoing email to the logcheck user.
To get these emails sent to myself, I added this entry to /etc/mail/aliases:

logcheck:       dan

Customizations

logcheck will initally produce notices about things you do not care to see
again. They are normal for your system and they do not need to be brought
to your attention again. You can train logcheck to ignore these items.
You will see both System Events and Security Events emails.
For example:

Security Events
=-=-=-=-=-=-=-=
Nov 28 16:37:55 dbclone postgres[93778]: [2-1] ERROR:  table "mac" does not exist

System Events
=-=-=-=-=-=-=
Nov 28 16:28:22 dbclone bacula-dir: Shutting down Bacula service: localhost-dir ...

These items are normal for this system. It is used for
Bacula regression testing. For the Security Events,
I created /usr/local/etc/logcheck/violations.ignore.d/local-postgres with the following
contents:

# grep mac /usr/local/etc/logcheck/violations.ignore.d/local-postgres
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ postgres\[[0-9]+\]: \[[0-9]+\-[0-9]+\] ERROR:  table "mac" does not exist

That preamble seems like a lot. But I grabbed it from logcheck-postgres. The logcheck
project recommends that you put your own customizations into files prefixed with local-
so they are easily identified. logcheck itself does not care.

For the System Event, I added this entry to /usr/local/etc/logcheck/ignore.d.server/local-postgres

bacula-fd: Shutting down Bacula service: localhost-fd

Notice that my System Event exceptions are specified in the ignore.d.server directory.
This is because I selected the following option in /usr/local/etc/logcheck/logcheck.conf:

REPORTLEVEL="server"

If you are using “workstation”, you would add your file to the ignore.d.workstation
directory.

EOF

There you go. That should get you started with logcheck. I’ve been using it for
10 years. It’s a great idea. I hope and trust it will save you a great deal
of ready. Best wishes.