Using newsyslog to rotate backups

Using newsyslog to rotate backups

newsyslog(8) allows you to maintain system log files to manageable sizes.
Log files record various system activities and can be useful when diagnosing
problems. The system adds to the end of the file and they can be read in
chronological order from top to bottom. If this growth is not monitored and
limited, storage space will be eventually exhausted. This is where
newsyslog.conf(8) helps.

newsyslog(8) uses the concept of rotation. Old data is removed and new data
added to a fresh empty file. You decide how much data you want to keep,
how often, etc, and newsyslog does the rest.

newsyslog(8) can be choose to archive based upon three reasons:

1. size
2. elapsed time
3. time of day

The motivation

The problem I’m solving is not log files, but it is disk space related to
files. On a daily basis, the databases are dumped to disk and then copied
from the database server to my server at home. At present, I include the
date in the file name after rsyncing. I also tar up the file. The existing
solution looks a bit like this:

YYYYMMDD=`eval date "+%Y.%m.%d"`
FILES=`echo *.sql`
for i in $FILES
do
    tar -cvzf archive/`basename ${i} .sql`.${YYYYMMDD}.sql.tgz ${i}
done

The issue with this solution is the removal of older backups. I’m already
backing up the .sql files via Bacula.
I keep additional copies just because I can. However, they need to be
trimmed.

The newsyslog pattern

Here is what worked for me:

/home/dan/backups/old-backups/*.sql      dan:dan        640     60      *       $D23    GZB

The key point is the G. From man newsyslog.conf

G       indicates that the specified logfile_name is a shell pat-
        tern, and that newsyslog(8) should archive all filenames
        matching that pattern using the other options on this
        line.  See glob(3) for details on syntax and matching
        rules.

This is exactly what I need. The Z flag requests compression. The B flag indicates the log file
has a special format; do not append an ASCII message to it.

I did have to make some fine tuning to this, but over time, it came out to just what I needed:

$ ls -lt | head -20
total 63929678
-rw-r--r--  1 dan  dan        5088 Oct 17 04:29 globals.sql
-rw-r--r--  1 dan  dan  3835195066 Oct 17 04:29 pentabarf_pgcon.sql
-rw-r--r--  1 dan  dan   731189913 Oct 17 04:26 pentabarf_bsdcan.sql
-rw-r--r--  1 dan  dan     1900659 Oct 17 04:26 pentabarf.sql
-rw-r--r--  1 dan  dan   623710100 Oct 17 04:26 openx.sql
-rw-r--r--  1 dan  dan      192664 Oct 17 04:24 fsphorum.sql
-rw-r--r--  1 dan  dan  4280026122 Oct 17 04:24 freshports.org.sql
-rw-r--r--  1 dan  dan     2170423 Oct 17 04:17 fpphorum.sql
-rw-r--r--  1 dan  dan      482262 Oct 17 04:17 bsdcert.sql
-rw-r-----  1 dan  dan        1319 Oct 16 04:29 globals.sql.0.gz
-rw-r-----  1 dan  dan  1496507511 Oct 16 04:29 pentabarf_pgcon.sql.0.gz
-rw-r-----  1 dan  dan   276787790 Oct 16 04:26 pentabarf_bsdcan.sql.0.gz
-rw-r-----  1 dan  dan      680104 Oct 16 04:26 pentabarf.sql.0.gz
-rw-r-----  1 dan  dan    60034865 Oct 16 04:26 openx.sql.0.gz
-rw-r-----  1 dan  dan   819085732 Oct 16 04:24 freshports.org.sql.0.gz
-rw-r-----  1 dan  dan       69310 Oct 16 04:24 fsphorum.sql.0.gz
-rw-r-----  1 dan  dan      113826 Oct 16 04:17 bsdcert.sql.0.gz
-rw-r-----  1 dan  dan      471781 Oct 16 04:17 fpphorum.sql.0.gz
-rw-r-----  1 dan  dan        1319 Oct 15 04:29 globals.sql.1.gz

This solution has been running since September 24. I like it.

What would you have done?