Jul 062008

ezjail – A jail administration framework

I want to set up some jails. They will each be very similar. They will each be used to test a slightly different configuration of Bacula. My tool of choice is ezjail, available in the ports tree. With ezjail, I can:
  • create a jail flavour, upon which the creation of other jails can be based
  • centrally update the jail’s ports tree
The above does not fully describe the neat things you can do with ezjail. Read below to discover more fun and interesting things. DISCLAIMER: I installed ezjail several months ago. I am only now getting around to documenting and writing about it. I may have omitted some steps. If so, I apologize; please let me know. While creating my jails I used the following references:

Installation of ezjail

First step installation:
cd /usr/ports/sysutils/ezjail
make install clean
Remember to add this to /etc/rc.conf:
Remember to create an ezjail configuration file:
cd /usr/local/etc
cp ezjail.conf.sample ezjail.conf

Create your base jail

This command creates the base jail. The base jail is the one upon which all other jails will be based. Note that this does not create a jail, it creates a base jail, one of the foundation components of your ezjail configuration.
ezjail-admin update -ip
This assumes you have already done a build world and that it exists within the standard and default location: /usr/src. After running this command you should see this:
# ls /usr/jails
basejail        flavours        newjail
It also installs a copy of the ports tree into the base jail, using portsnap. I encountered this error while trying to create a new basejail. Upgrading to the latest STABLE fixed this problem.
install -o root -g wheel -m 444 dir-tmpl /usr/jails/fulljail/usr/share/info/dir
install:No such file or directory

Creating the IP addresses for the jails

The host system needs to have some IP addresses that will be used exclusively by the jails. That is, one IP address per jail. For ease of use, and personal sanity, I’ve added the following entries to my private DNS server:
; jails for testing Bacula

mysql41 IN A
mysql50 IN A
mysql51 IN A
pg73    IN A
pg74    IN A
pg80    IN A
pg81    IN A
pg82    IN A
pg83    IN A
sqlite3 IN A
Corresponding to the above, here are the ifconfig aliases I added to /etc/rc.conf on the host system:

Creating the first jail

The following command creates the first jail. You can create additional jails by changing the hostname and the IP address. Each jail needs a unique IP address and hostname. Ideally, the IP address will correspond to the IP address you have selected.
# ezjail-admin create -f bacula mysql41.example.org
40797 blocks
Note: Shell scripts installed, flavourizing on jails first startup.
Warning: Some services already seem to be listening on all IP, (including
  This may cause some confusion, here they are:
root     bacula-fd  1023  3  tcp4   *:9102                *:*
root     sshd       1010  4  tcp4   *:22                  *:*
root     master     995   11 tcp4   *:25                  *:*
root     syslogd    816   7  udp4   *:514                 *:*
[root@polo ~]# jls
Of note, you see the services running on the host which are listening on all IP addresses. This will be a problem for the jails. I will need to change the host services to listen only on the main IP address for this host. Sometimes you might want a host service to listen on all IP addresses, but that’s not what I want here. The -f argument indicates that the jail should be created using the bacula flavour. This flavour was set up back in March, and I do not have my notes from that work. I will probably be creating more flavours and if I do, I will document them. But I do have a file list. Here is is the list of files from my bacula ezjail flavour:
[dan@polo /usr/jails/flavours/bacula]$ find .
[dan@polo /usr/jails/flavours/bacula]$ 
See Modifying other daemons in the Jails under FreeBSD 6 article for more information on how I’ve done this in the past.

Starting the jail

Now that I have created my jail, it is time to start it:
#  /usr/local/etc/rc.d/ezjail.sh start
 ezjailConfiguring jails:.
Starting jails: mysql41.example.org.
I verified the jail was running:
# jls
   JID  IP Address    Hostname                Path
     1  mysql41.example.org     /usr/jails/mysql41.example.org
I was able to ssh to the newly started jail:
$ uptime
 1:19PM  up  1:33, 1 user, load averages: 0.00, 0.00, 0.00
$ hostname

Amending your flavour and refreshing the jail

NOTE: this attempt failed. You may not want to try this. Read the whole section first, then see what actually succeeded. After logging into the new jail, I noticed my shell was missing my favourite bash prompt. I decided to copy the required files into the flavour and refresh the jail.
cp ~/.bash_profile ~/.bashrc /usr/jails/flavours/bacula/usr/home/dan/
Now I needed to stop the jail and update its image. Then restart the jail.
# /usr/local/etc/rc.d/ezjail.sh stop
 ezjailStopping jails: mysql41.example.org.
# ezjail-admin delete mysql41.example.org
# ezjail-admin create -x -f bacula mysql41.example.org
Warning: Some services already seem to be listening on all IP, (including
  This may cause some confusion, here they are:
root     bacula-fd  1483  3  tcp4   *:9102                *:*
root     master     994   11 tcp4   *:25                  *:*
root     syslogd    815   7  udp4   *:514                 *:*
As you can see, I still have a few daemons listening on all addresses. I will handle that eventually. However, my goal of recreating the jail with my bash configuration files failed. What I had to do was delete the jail with the -w option, to wipe the jail from the HDD. Then create it again:
ezjail-admin delete -w mysql41.example.org
ezjail-admin create -f bacula mysql41.example.org
/usr/local/etc/rc.d/ezjail.sh start

Creating and starting a second jail

Now I will create another jail for testing MySQL 5.0:
[root@polo /usr/jails/flavours/bacula]# ezjail-admin create -f bacula mysql50.example.org
67260 blocks
Note: Shell scripts installed, flavourizing on jails first startup.
Warning: Some services already seem to be listening on IP
  This may cause some confusion, here they are:
root     ntpd       65252 8  udp4       *:*
Warning: Some services already seem to be listening on all IP, (including
  This may cause some confusion, here they are:
root     ntpd       65252 4  udp4   *:123                 *:*
[root@polo /usr/jails/flavours/bacula]#  /usr/local/etc/rc.d/ezjail.sh      
Usage: /usr/local/etc/rc.d/ezjail.sh [fast|force|one](start|stop|restart|rcvar|startcrypto|stopcrypto)
[root@polo /usr/jails/flavours/bacula]#  /usr/local/etc/rc.d/ezjail.sh start
ezjailConfiguring jails:.
Starting jails: mysql50.example.org [mysql41.example.org already running 
                            (/var/run/jail_mysql41_unixathome_org.id exists)].
[root@polo /usr/jails/flavours/bacula]# 
And here is that jail:
$ hostname
$ uname -a
FreeBSD mysql50.example.org 7.0-STABLE FreeBSD 7.0-STABLE #4: Fri Jul  
4 14:01:32 EDT 2008     dan@polo.example.org:/usr/obj/usr/src/sys/PHENOM  
$ bash
[dan@mysql50:/usr/home/dan] $ cd
[dan@mysql50:~] $ uptime
 6:03PM  up  6:17, 1 user, load averages: 0.07, 0.02, 0.02

Better jail start/stop

You can start/stop individual jails.
#  /usr/local/etc/rc.d/ezjail.sh stop mysql50.example.org
Stopping jails: mysql50.example.org.
#  /usr/local/etc/rc.d/ezjail.sh start mysql50.example.org
Configuring jails:.
Starting jails: mysql50.example.org.

Upgrading a jail (added on 23 Aug 2008)

I’ve decided it is time to upgrade these jails. The host system is now running:
FreeBSD polo.example.org 7.0-STABLE FreeBSD 7.0-STABLE #5: Fri Aug 22 13:45:47 EDT 2008     dan@polo.example.org:/usr/obj/usr/src/sys/PHENOM  amd64
Whereas, the jails are:
FreeBSD mysql50.example.org 7.0-STABLE FreeBSD 7.0-STABLE #5: Fri Aug 22 13:45:47 EDT 2008    dan@polo.example.org:/usr/obj/usr/src/sys/PHENOM  amd64
Yes, they look identical. But to upgrade the ezjail after doing a build world in there, I did this:

What’s next?

That’s my first cut of using ezjail. I’ll be amending and adding to my flavours as I add more jails. I also plan to use cfengine to configure the jails, install/upgrade ports, and to undertake general maintenance. Watch for cfengine in an upcoming article.

  6 Responses to “ezjail – A jail administration framework”

  1. Good article. Ezjail is incredibly useful. I combine it with NATD to create a bunch of jails all accessible through one public IP address. I encode each jail’s unique ports in the last octet of the private IP address space, using NATD directives like this:

    log yes

    # JAIL 10
    redirect_port tcp 1022
    redirect_port tcp 1080

    # JAIL 11
    redirect_port tcp 1122
    redirect_port tcp 1180
    redirect_port tcp 11443

    Each jail’s services are on unique ports, using URLs like https://www.example.com:11443/, for example, and ssh -p 1122. The only problem I have had is that some of our large corporate clients block all incoming and outgoing ports above 999. Baffling, but true!

    Ezjail and natd help us leverage a single inexpensive server to host 35 development environments. Very cool!

    The numbering scheme limits me to 56 jails, because port numbers top out at 65535, making 65443 the highest available https port. We’ve never come close to topping that out, so this works very well for us.


  2. Hi,

    ezjail isn’t the fun way to play with jails. ZFS is.

    First of all, the /etc/rc.d/jail script has been quite updated, and will take care of the day-to-day stuff, such as starting and stopping jails.

    Simply define:

    jail_psql_devfs_ruleset="devfsrules_jail" # devfs ruleset to apply to jail

    And you can do:

    /etc/rc.d/jail start psql

    That’s it. No need for a port to retrofit this.

    Also, for the fun part. If you throw in ZFS in the loop, you get better control over disk usage (pr. jail or even pr. dir quotas, compression, redundancy-levels etc).

    You can also snapshot jails.

    That’s right. If you’re going to upgrade a jailed "machine", just snapshot it, test it, and roll back if there’s a problem. If you have read-write userdata, simply put it in a separate zfs, and you can do it with parts of the "virtual machine".

    But wait! There’s more!

    If you can create a base config, with most of the things you’ll typically need for a jail, you can then snapshot and clone it.

    If you have a base jail, and you’d like to install a 4MB package in addition to your standard stuff, the entire jail will only take up that 4MB of space.

    You save RAM too. I think. Not entirely sure how ZFS fits into the FreeBSD page system, but since you’re running your libc and whatnot from the same disk pages for all your jails, I’d think it’s quite likely you’ll save up some RAM too.

    Sounds like a good plan?

    • tld wrote:

      > Sounds like a good plan?

      All except: ZFS isn’t quite ready for production, from everything I’ve heard.

      The Man Behind The Curtain

    • Details, details. 😉

      ZFS is "different" for a lot of FreeBSD people. We’re used to anything in FreeBSD being stable on anything it can run on. ZFS seems to have issues on low memory i386 machines, but work quite well on amd64 machines with 4GB+.

      In other words, I guess it all depends on your needs. ZFS also offers a lot of convenient features which are tempting to have in production, such as protection against silent datacorruption (seen that a couple of times too many), being able to use all the space effectively without downtime for reinstall, etc etc.

      Might not want to dive in and convert anythign and everything to run ZFS, but it’s definitively ready for testing out.

  3. Note: the "create -f bacula" part of the ezjail command assumes you have already created a flavor named ‘bacula’.

    Sorry. I omitted that stage during my note taking.

    The Man Behind The Curtain