Creating multiple jails

Creating multiple jails

Jails are a useful tool for achieving a certain level of virtualization.
I have used jails in the past for creating sandboxes for testing of
the Bacula project. Today, I will
be setting up 7 jails, each for regression tests on different configurations.
In the process, I’ll show you some shortcuts that Ryan Lortie (desrt) told
me about. It saves time. I’ll also share some scripts that will help with
the setup of various jail items.

This article should be read in conjunction with
my other jail articles. It won’t stand
alone unless you are already familiar with jails and how to configure them.

Building and configuring the jails

According to man jail, these are the basic steps for building a jail:

D=/here/is/the/jail
cd /usr/src
mkdir -p $D
make world DESTDIR=$D
make distribution DESTDIR=$D
mount -t devfs devfs $D/dev

desrt pointed out, and I think I’ve heard this before, after the first
build, you don’t have to do a full make world. For the subsequent jails,
you can get away with this:

D=/here/is/the/OTHER/jail
cd /usr/src
mkdir -p $D
make installworld DESTDIR=$D
make distribution DESTDIR=$D
mount -t devfs devfs $D/dev

The key differences are:

1. Change the value of D to point to the new jail.
2. Do a make installworld, not a make world.

“make world” consists of two main steps: make buildworld && make installworld.
There is no advantage in doing the buildworld part again. That has already
be done.

During the building of my first jail, I tried this:

export D=mysql41.example.org

This fail, because the install could not find the directory mysql41.example.org.
I should have included the full path to the directory: /usr/jail/mysql41.example.org.
To complete the installation of the jail, I issued this series of commands:

export D=/usr/jail/mysql41.example.org
make installworld DESTDIR=$D
make distribution DESTDIR=$D
mount -t devfs devfs $D/dev

After the above completed, I had the following mount points:

# mount
/dev/ad0s1a on / (ufs, local)
/devfs on /dev (devfs, local)
/dev/ad0s1e on /tmp (ufs, local, soft-updates)
/dev/ad0s1f on /usr (ufs, local, soft-updates)
/dev/ad0s1d on /var (ufs, local, soft-updates)
/devfs on /usr/jail/mysql41.example.org/dev (devfs, local)

There are a few things that I like to configure within the jail before
starting it. The files in the jail environment are accessible from
the host environment. So it is a simple matter of using your
favourite editor to make the changes. Here are the files I altered
in /usr/jail/mysql41.example.org:

• etc/rc.conf – added: sshd_enable=”YES”
• etc/resolv.conf – added entries to get DNS working
• etc/syslog.conf – changed the “*.err;kern.warning;auth.notice;mail.crit”
  line to echo to /var/log/messages instead of the console

It would be easy to script that. For example, here is the script
I used to do the installworld for each jail:

#!/bin/sh
JAILDIR="/usr/jail"
JAILS=`ls ${JAILDIR}`
for jail in ${JAILS}
do
   echo $jail
   D=${JAILDIR}/${jail}
   make installworld DESTDIR=$D
   make distribution DESTDIR=$D
   mount -t devfs devfs $D/dev
done

This script assumes that each directory in /usr/jail needs to be setup.
If this was not the case, you could hardcode the value of JAILS like this:

JAILS="mysql51.example.org pg73.example.org pg74.example.org pg80.example.org"

Similar scripts can be created for populating the ports tree. You
may think that a single shared ports tree might be sufficient for multiple
jails. I’ve decided to go with a distinct tree for each jail. I’ll be
building different ports concurrently in each jail. I don’t want any
interference caused by another jail.

Setting up the host environment

Setting up multiple jails involves adding several entries to /etc/rc.conf,
some of which specify defaults for all jails. The default settings in my
server are:

# Defaults for all jails:
jail_interface="fxp0"    # Interface to create the IP alias on
jail_devfs_enable="YES"  # mount devfs in the jail
jail_procfs_enable="YES" # mount procfs in jail

I also specify a list of the jails on this machine:

# list of jails on this machine
jail_list="mysql41 mysql50 mysql51 pg73 pg74 pg80 pg81 pg82"

For each of the above mentioned jails, entries similar to this appear:

# values for each jail listed above
jail_mysql41_rootdir="/usr/home/jails/mysql41.example.org"  # Jail's root directory
jail_mysql41_hostname="mysql41.example.org"  # Jail's hostname
jail_mysql41_ip="192.168.0.100"              # Jail's IP number

It can be tedious to setup thse values. So a script can be useful
and it could be based upon the script that appears above.

PostgreSQL in a jail

Running PostgreSQL in a jail is interesting. There are shared memory issues which can
be resolved by setting security.jail.sysvipc_allowed via sysctl. I tried that.
I failed. The problem I encountered during initdb is:

# su -l pgsql -c initdb
The files belonging to this database system will be owned by user "pgsql".
This user must also own the server process.

The database cluster will be initialized with locale C.

creating directory /usr/local/pgsql/data ... ok
creating directory /usr/local/pgsql/data/global ... ok
creating directory /usr/local/pgsql/data/pg_xlog ... ok
creating directory /usr/local/pgsql/data/pg_xlog/archive_status ... ok
creating directory /usr/local/pgsql/data/pg_clog ... ok
creating directory /usr/local/pgsql/data/pg_subtrans ... ok
creating directory /usr/local/pgsql/data/pg_twophase ... ok
creating directory /usr/local/pgsql/data/pg_multixact/members ... ok
creating directory /usr/local/pgsql/data/pg_multixact/offsets ... ok
creating directory /usr/local/pgsql/data/base ... ok
creating directory /usr/local/pgsql/data/base/1 ... ok
creating directory /usr/local/pgsql/data/pg_tblspc ... ok
selecting default max_connections ... 10
selecting default shared_buffers ... 50
creating configuration files ... ok
creating template1 database in /usr/local/pgsql/data/base/1 ... FATAL:  could 
not create shared memory segment: Function not implemented
DETAIL:  Failed system call was shmget(key=1, size=1327104, 03600).
child process exited with exit code 1
initdb: removing data directory "/usr/local/pgsql/data"

I initially thought I had to set this flag on each of the jails. I was wrong.
After a bit of trial and error, I discovered the problem. I had added this value
to /etc/sysctl.conf:

# For PostgreSQL jails
security.jail.sysvipc_allowed=1

After a reboot, the above entry ensures this setting after a reboot:

# sysctl security.jail.sysvipc_allowed
security.jail.sysvipc_allowed: 1

When starting a jail, I noticed this:

# /etc/rc.d/jail start pg82
Configuring jails: sysvipc_allow=NO.
Starting jails: pg82.unixathome.org.

Why was it saying sysvipc_allow=NO? Additionally, in the jail and in the host
system, I was now seeing this:

# sysctl security.jail.sysvipc_allowed
security.jail.sysvipc_allowed: 0

Not only was the jail not getting the right value, the value in the host system
was being reset.

Looking at /etc/rc.d/jail for “Configuring jails”, I found more clues
leading me to /etc/defaults/rc.conf, where I found this setting:

jail_sysvipc_allow="NO" # Allow SystemV IPC use from within a jail

Ahh, OK, so now I needed to set this in /etc/rc.conf

jail_sysvipc_allow="YES" # For PostgreSQL

After restarting the pg82 jail, I found the security.jail.sysvipc_allowed was correctly
set, in both the jail and the host. I tried the initdb again:

# su -l pgsql -c initdb
The files belonging to this database system will be owned by user "pgsql".
This user must also own the server process.

The database cluster will be initialized with locale C.

creating directory /usr/local/pgsql/data ... ok
creating directory /usr/local/pgsql/data/global ... ok
creating directory /usr/local/pgsql/data/pg_xlog ... ok
creating directory /usr/local/pgsql/data/pg_xlog/archive_status ... ok
creating directory /usr/local/pgsql/data/pg_clog ... ok
creating directory /usr/local/pgsql/data/pg_subtrans ... ok
creating directory /usr/local/pgsql/data/base ... ok
creating directory /usr/local/pgsql/data/base/1 ... ok
creating directory /usr/local/pgsql/data/pg_tblspc ... ok
selecting default max_connections ... 10
selecting default shared_buffers ... 50
creating configuration files ... ok
creating template1 database in /usr/local/pgsql/data/base/1 ... FATAL:  could 
not create semaphores: No space left on device
DETAIL:  Failed system call was semget(1, 17, 03600).
HINT:  This error does *not* mean that you have run out of disk space.
       It occurs when either the system limit for the maximum number of 
semaphore sets (SEMMNI), or the system wide maximum number of semaphores
(SEMMNS), would be exceeded.  You need to raise the respective kernel parameter.
Alternatively, reduce PostgreSQL's consumption of semaphores by reducing its 
max_connections parameter (currently 10).
The PostgreSQL documentation contains more information about configuring your system for PostgreSQL.
child process exited with exit code 1
initdb: removing data directory "/usr/local/pgsql/data"

I am familiar with this message. It is related to kernel settings. I added these values
to /boot/loader.conf on the host system and rebooted:

kern.ipc.semmni=256
kern.ipc.semmns=512
kern.ipc.semmnu=256

The reboot is necessary because those settings are read-only and cannot be modified after
booting the kernel. With those changes, the initdb ran to completion:

# su -l pgsql -c initdb
The files belonging to this database system will be owned by user "pgsql".
This user must also own the server process.

The database cluster will be initialized with locale C.

creating directory /usr/local/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers/max_fsm_pages ... 24MB/153600
creating configuration files ... ok
creating template1 database in /usr/local/pgsql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating conversions ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the -A option the
next time you run initdb.

Success. You can now start the database server using:

    postgres -D /usr/local/pgsql/data
or
    pg_ctl -D /usr/local/pgsql/data -l logfile start

I was then able to start PostgrSQL, create users, etc. I repeated
the process for two other jails, then ran into this problem for the
third jail:

# su -l pgsql -c initdb
The files belonging to this database system will be owned by user "pgsql".
This user must also own the server process.

The database cluster will be initialized with locale C.

creating directory /usr/local/pgsql/data ... ok
creating directory /usr/local/pgsql/data/global ... ok
creating directory /usr/local/pgsql/data/pg_xlog ... ok
creating directory /usr/local/pgsql/data/pg_xlog/archive_status ... ok
creating directory /usr/local/pgsql/data/pg_clog ... ok
creating directory /usr/local/pgsql/data/pg_subtrans ... ok
creating directory /usr/local/pgsql/data/base ... ok
creating directory /usr/local/pgsql/data/base/1 ... ok
creating directory /usr/local/pgsql/data/pg_tblspc ... ok
selecting default max_connections ... 10
selecting default shared_buffers ... 50
creating configuration files ... ok
creating template1 database in /usr/local/pgsql/data/base/1 ... FATAL:  could
not create shared memory segment: Cannot allocate memory
DETAIL:  Failed system call was shmget(key=1, size=1122304, 03600).
HINT:  This error usually means that PostgreSQL's request for a shared memory
segment exceeded available memory or swap space. To reduce the request size 
(currently 1122304 bytes), reduce PostgreSQL's shared_buffers parameter 
(currently 50) and/or its max_connections parameter (currently 10).
The PostgreSQL documentation contains more information about shared memory
configuration.
child process exited with exit code 1
initdb: removing data directory "/usr/local/pgsql/data"

I decided to double the settings in /etc/sysctl.conf on the host system:

kern.ipc.semmni=512
kern.ipc.semmns=1024
kern.ipc.semmnu=512

But that did not solve the problem. Stopping PostgreSQL on the other two jails did.
This allowed the initdb to complete. But I was still unable to get all the
PostgreSQL servers running in each jail.

After a few more trial and error attempts, I discovered that the settings I had in
/boot/loader.conf contained syntax errors. Instead of SETTING=VALUE, I had
SETTING: VALUE. Fixing that, and rebooting, allowed me to get all five instances
of PostgreSQL running concurrently. But there was a bit more to it than just that.

I tried various things. I tried running each postmaster on a different port
(didn’t help). I also tried running each postmaster as a different GID (that did
help). That is the key. I will explain.

Shared memory and UID

PostgreSQL makes use of shared memory. When running multiple instances of PostgreSQL
the shared memory for one instance can be stomped on by another instance. That’s not
nice. The key to avoiding this is using a different UID for each instance. You can see
that here:

$ grep -h pgsql /usr/jail/*.unixathome.org/etc/passwd
pgsql:*:1073:70:PostgreSQL Daemon:/usr/local/pgsql:/bin/sh
pgsql:*:1074:70:PostgreSQL Daemon:/usr/local/pgsql:/bin/sh
pgsql:*:1080:70:PostgreSQL Daemon:/usr/local/pgsql:/bin/sh
pgsql:*:1081:70:PostgreSQL Daemon:/usr/local/pgsql:/bin/sh
pgsql:*:1082:70:PostgreSQL Daemon:/usr/local/pgsql:/bin/sh

I used a UID that would relate to the version of PostgreSQL that was running.
For example, UID=1073 is PostgreSQL version 7.3. There is no need to follow
this convention. I did it merely because I could.

When I changed the UID in each jail, I first made sure PostgreSQL was not running
in that jail. Then I ran vipw and altered the UID and exited. Then I did a chown
to ensure the file attributes were owned by the new UID:

chown -R pgsql /usr/local/pgsql

Then I restarted PostgreSQL. No more memory stomping. The symptoms
of this memory stomping (a term I made up and probably does not reflect
reality) are:

$ psql -l
psql: FATAL: semctl(262144, 4, SETVAL, 0) failed: Invalid argument

The above type of problem would arise only when I ran multiple PostgreSQL servers
with the same UID.

The sysctl settings

These are the various sysctl settings I’m using on this server:

[dan@polo:~] $ cat /boot/loader.conf
kern.ipc.semaem=32767
kern.ipc.semvmx=65534
kern.ipc.semusz=184
kern.ipc.semume=80
kern.ipc.semopm=200
kern.ipc.semmsl=120
kern.ipc.semmnu=4096
kern.ipc.semmns=8192
kern.ipc.semmni=32767
kern.ipc.semmap=60

[dan@polo:~] $ cat /etc/sysctl.conf
# For PostgreSQL jails
security.jail.sysvipc_allowed=1

# for more shared memory for jails/PostgreSQL
kern.ipc.shmall=65536
kern.ipc.shmmax=134217728
kern.ipc.semmap=4096
[dan@polo:~] $

I am confident that I am over allocating resources here. I kept increasing the
values and trying them, without success. Then I discovered my syntax error.
I didn’t want to reverse-engineer my changes so I have left them as they appear
above. You may wish to use lesser values.

You may be interested in the host system dmesg output. As you can see, this
isn’t a very high spec system. It has only 512M of RAM.

/var/run/dmesg.boot

top – do not be afraid

When running top, I noticed something I had not see before. Number in the username
field.

last pid: 82356;  load averages:  2.47,  1.30,  1.01   up 1+21:46:07  10:25:39
285 processes: 6 running, 277 sleeping, 1 stopped, 1 lock
CPU states: 29.2% user,  5.8% nice, 33.1% system,  1.3% interrupt, 30.7% idle
Mem: 173M Active, 100M Inact, 118M Wired, 14M Cache, 60M Buf, 81M Free
Swap: 989M Total, 282M Used, 708M Free, 28% Inuse

  PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
  81261   1080      1 105    0 10432K  4728K *Giant 1   0:28 16.85% postgres
  82147   1074      1   4    0 10392K  3136K sbwait 0   0:07 16.58% postgres
  81830   1073      1   4    0 11016K  4548K sbwait 0   0:16 15.87% postgres
  81322   1081      1 103    0 11664K  4244K CPU0   1   0:23 13.57% postgres
  81826 dan         6 107   10 11832K  3744K RUN    0   0:04  3.13% bacula-dir
  82143 dan         6 108   10 11856K  3852K RUN    0   0:02  3.03% bacula-dir
  81252 dan        12 106   10 15700K  4296K ucond  0   0:07  1.51% bacula-dir
  81257 dan        12 106   10 15700K  4304K ucond  1   0:07  1.12% bacula-dir

With a bit of thought, you can figure this out. top was running in the host
environment. The UID for pgsql has been changed in the jail. The host environment
has no knowledge of UIDs from the jails. When top cannot convert a UID to a username,
it displays the UID. This is fine. An added benefit is seeing exactly which jail
is still running. In this case, it’s PostgreSQL 8.0, 7.4, 7.3, and 8.1.

Jails – just right for this

Jails are just right what for what I’m trying here. I certainly do not want
8 servers sitting around the house, just for regression testing of Bacula.
I’ll be writing up some scripts to automate much of this testing and to upload
the results to a website.

Jails: sometimes the right tool for the job.