Jan 232007

Bacula – Transport Layer Security (TLS)

Bacula is a very robust and feature-rich network backup solution. Whether you are backing up two computers or two hundred, Bacula can do it. Backup solutions are many and varied. Make sure you choose the right backup software. I’ve been using Bacula since late 2003. The use of Bacula has grown tremendously during that time. The number of active developers has grown steadily and the community has flourished. Many new features and improvements have been added. If you are considering a backup solution, I suggest Bacula. I wrote the PostgreSQL plug-in for Bacula. This plug-in allows Bacula to use PostgreSQL as a Catalog of all files that have been backed up. MySQL and SQLite can also be used as a Catalog. I have been using Bacula to backup across public untrusted networks (e.g. the Internet) for a few years. I secure the data using stunnel. One of these systems sits at CityLink in Wellington, New Zealand. I decided today that I should start using TLS, not stunnel for this node. This article shows how I moved from stunnel to TLS. This will not be a full Bacula TLS tutorial. I suggest reading Bacula/TLS by R.I.Pienaar. It is what I referred to when doing this. When I was first setting up TLS, I did take notes. Now I cannot find them. They may turn up one day….

Get backups working

I strongly urge you to get backups working without TLS first. Do not backup any sensitive data; instead, backup some nonsense files. Remember: what you are backing up will be sent in clear text. NOTE: handshaking between Bacula daemons does not involve sending passwords in clear text; but any backup data is clear. I first ran a test backup of the remote client, without using TLS. This succeeded. Then I started modifying the client resource in the bacula-dir.conf file:
  • Client
    • Address: I changed this from localhost (which was tunnelled to the remote client) to the hostname of the remote client.
    • FDPort: Changed from the stunnel port (some high random number) to the standard Bacula FD port: 9102
  • Job
    • Storage – I changed this from the Storage Daemon used by remote clients to the SD used by TLS clients. In previous setups, remote clients always came in through stunnel.
Note that I have not yet done anything with regards to TLS yet. At this point we are just trying to ensure that communications work.
*status client=lists-fd
Connecting to Client lists-fd at lists.example.org:9102

lists-fd Version: 1.38.11 (28 June 2006)  i386-portbld-freebsd4.8 freebsd 4.8-RC
Daemon started 05-Sep-06 01:31, 0 Jobs run since started.
No Terminated Jobs.
Running Jobs:
Director connected at: 05-Sep-06 01:32
No Jobs running.
There. Proof that Director and Client will speak to each other. Now we move on to creating the keys for the client.

TLS and hostnames

In the next section, you will be creating a certificate request. During that process, you will be asked for a common name. This is usually a fully qualified domain name (FQDN), such as lists.unixathome.org. In most circumstances, the FQDN you want to supply is the hostname of the machine on which the certficate will reside. However, this is not a requirement. I will elaborate. I was backing up a customer’s machine. To do this, I pointed one of my DNS records at their machine. I used that hostname, which wasn’t the same as the hostname of the computer. For example, if their computer was ABC.example.net, I created XYZ.example.com and pointed it at their computer. NOTE: I used .com, not .net in this example. That is not a typo. It is vital to TLS that the value you put in the Address field of the Client resource in the bacula-dir.conf file matches the FQDN that you use for the certificate request. Neither of these values needs to be the hostname of the computer you are backing up. The Address just needs to point at that computer.

Creating the client key

In this section, I will create the client key for bacula-fd. Please read the official Bacula TLS documentation. I am using Bacula/TLS as previously mentioned. First, I create the key for the client. The output from this process will be stored in lists.example.org.key. By convention, I use the hostname in the filename, just to keep things clear.
$ openssl genrsa -des3 -out lists.example.org.key 1024
Generating RSA private key, 1024 bit long modulus
e is 65537 (0x10001)
Enter pass phrase for lists.example.org.key:
Verifying - Enter pass phrase for lists.example.org.key:
You are asked to enter a passphrase. You will use that passphrase in subsequent steps. With that key, you create a certificate request. This goes to your certificate issuer. Much of this process is beyond the scope of this article.

NOTE: It is important to use the hostname of the client using this certificate. In this case, lists.example.org is the hostname of the client that runs bacula-fd. See below as to where you must enter. The passphrase you are asked for in this step is the one you used in the previous step.

$ openssl req -new -key lists.example.org.key -out lists.example.org.csr
Enter pass phrase for lists.example.org.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
Country Name (2 letter code) [AU]:CA
State or Province Name (full name) [Some-State]:Ontario
Locality Name (eg, city) []:Ottawa
Organization Name (eg, company) [Internet Widgits Pty Ltd]:The FreeBSD Diary
Organizational Unit Name (eg, section) []:Backup Division
Common Name (eg, YOUR name) []:lists.example.org
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
The above process created lists.example.org.csr, and this file is what your certificate authority will use to create your certificate. In my case, I use CACert. I put the certificate from CACert into lists.example.org.cert. I now create a non-passworded version of that certificate. You must supply the passphrase from the first step. This is necessary because Bacula does not support supplying the passwords for these certificates.
$ openssl rsa -in lists.example.org.key -out lists.example.org.nopassword.key
Enter pass phrase for lists.example.org.key:
writing RSA key
It is this file that you must copy to your Bacula client.

bacula-fd.conf changes

For bacula-fd.conf, you add TLS specification to the director resource. A bacula-fd.conf can have more than one director resource. Thus, each director gets a TLS specification. Simlarly, if you are specifying TLS for bacula-sd, it goes in the director resource of bacula-sd.conf. Here are the entries I added to /usr/local/etc/bacula-fd.conf. This will indicate that all connections between this FD and this this Director must use TLS.
TLS Enable  = yes
TLS Require = yes

TLS CA Certificate File = /home/bacula/cacert.pem

TLS Certificate         = /home/bacula/lists.example.org.cert
TLS Key                 = /home/bacula/lists.example.org.nopassword.key
I put the certificates in /home/bacula for no particular reason. The directory is, and contents is chown bacula:bacula and the files in the directory are chmod 640. The files in there are:
[root@lists:/home/bacula] # ls -l
total 8
-rw-r-----  1 bacula  bacula  4720 Sep  5 02:19 cacert.pem
-rw-r-----  1 bacula  bacula   887 Sep  5 02:23 lists.example.org.nopassword.key
-rw-r-----  1 bacula  bacula  1524 Sep  5 02:19 lists.example.org.cert
[root@lists:/home/bacula] #
Then I restarted bacula-fd:
# /usr/local/etc/rc.d/bacula-fd.sh restart
Stopping bacula_fd.
Waiting for PIDS: 77379.
Starting bacula_fd.
05-Sep 02:26 lists-fd: ERROR in tls.c:83 Error loading private key:
         ERR=error:02001002:system library:fopen:No such file or directory
05-Sep 02:26 lists-fd: ERROR in tls.c:83 Error loading private key:
          ERR=error:20074002:BIO routines:FILE_CTRL:system lib
05-Sep 02:26 lists-fd: ERROR in tls.c:83 Error loading private key:
           ERR=error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
05-Sep 02:26 lists-fd: Fatal Error at filed.c:327 because:
Failed to initialize TLS context for File daemon "lists-fd" in /usr/local/etc/bacula-fd.conf.
05-Sep 02:26 lists-fd: ERROR in filed.c:188 Please correct configuration
                 file: /usr/local/etc/bacula-fd.conf
[root@lists:/home/bacula] #
Ouch. Simple to fix though. I had /home/bacula/lists.example.org.key (no such file) instead of /home/bacula/lists.example.org.nopassword.key. After making the change to bacula-fd.conf and restarting bacula-fd, all was well. It was off to bconsole to run a status command. That failed with these messages:
04-Sep 10:26 bacula-dir: Start Backup JobId 10621, Job=lists.2006-09-04_10.26.51
04-Sep 10:27 bacula-dir: lists.2006-09-04_10.26.51 Fatal error: Socket error on 
              Storage command: ERR=Resource temporarily unavailable
04-Sep 10:27 polo-sd: lists.2006-09-04_10.26.51 Fatal error: TLS negotiation failed.
04-Sep 10:27 polo-sd: lists.2006-09-04_10.26.51 Fatal error: Incorrect authorization
               key from File daemon at client rejected.
Please see http://www.bacula.org/rel-manual/faq.html#AuthorizationErrors for help.
Now I turn my attention to the Director.

bacula-dir.conf changes

I had not yet set up bacula-dir to speak know to use TLS with this client. bacula-dir will need its own certificate, created in much the same fashion as bacula-fd’s certificate. After doing this, I added the following to the Client resource in /usr/local/etc/bacula-dir.conf on the Bacula server. Not shown here are the certificates steps I performed for the director.
TLS Require = yes
TLS Enable  = yes
TLS CA Certificate File = /home/bacula/certificates/cacert.pem

TLS Certificate = /home/bacula/certificates/bacula.example.org.cert
TLS Key         = /home/bacula/certificates/bacula.example.org.nopassword.key
In bconsole, issue the reload command, and now bacula-dir should be ready to use TLS with that client. Try a status client command to test.