ezjail – A jail administration framework

ezjail – A jail administration framework

I want to set up some jails. They will each be very similar.
They will each be used to test a slightly different configuration
of Bacula. My tool of choice
is ezjail,
available in the ports tree.

With ezjail, I can:

  • create a jail flavour, upon which the creation of other jails can be based
  • centrally update the jail’s ports tree

The above does not fully describe the neat things you can do with ezjail.
Read below to discover more fun and interesting things.

DISCLAIMER: I installed ezjail several months ago. I am only now getting
around to documenting and writing about it. I may have omitted some
steps. If so, I apologize; please let me know.

While creating my jails I used the following references:

Installation of ezjail

First step installation:

cd /usr/ports/sysutils/ezjail
make install clean

Remember to add this to /etc/rc.conf:

ezjail_enable="YES"

Remember to create an ezjail configuration file:

cd /usr/local/etc
cp ezjail.conf.sample ezjail.conf

Create your base jail

This command creates the base jail. The base jail is the one upon which
all other jails will be based. Note that this does not create a jail,
it creates a base jail, one of the foundation components of your
ezjail configuration.

ezjail-admin update -ip

This assumes you have already done a build world and that it exists within
the standard and default location: /usr/src.

After running this command you should see this:

# ls /usr/jails
basejail        flavours        newjail

It also installs a copy of the ports tree into the base jail, using portsnap.

I encountered this error while trying to create a new basejail.
Upgrading to the latest STABLE fixed this problem.

install -o root -g wheel -m 444 dir-tmpl /usr/jails/fulljail/usr/share/info/dir
install:No such file or directory

Creating the IP addresses for the jails

The host system needs to have some IP addresses that will be used
exclusively by the jails. That is, one IP address per jail. For
ease of use, and personal sanity, I’ve added the following entries
to my private DNS server:

; jails for testing Bacula

mysql41 IN A 10.10.10.100
mysql50 IN A 10.10.10.101
mysql51 IN A 10.10.10.102
pg73    IN A 10.10.10.103
pg74    IN A 10.10.10.104
pg80    IN A 10.10.10.105
pg81    IN A 10.10.10.106
pg82    IN A 10.10.10.107
pg83    IN A 10.10.10.109
sqlite3 IN A 10.10.10.108

Corresponding to the above, here are the ifconfig aliases I added to
/etc/rc.conf on the host system:

ifconfig_fxp0_alias0="inet 10.10.10.100/32"
ifconfig_fxp0_alias1="inet 10.10.10.101/32"
ifconfig_fxp0_alias2="inet 10.10.10.102/32"
ifconfig_fxp0_alias3="inet 10.10.10.103/32"
ifconfig_fxp0_alias4="inet 10.10.10.104/32"
ifconfig_fxp0_alias5="inet 10.10.10.105/32"
ifconfig_fxp0_alias6="inet 10.10.10.106/32"
ifconfig_fxp0_alias7="inet 10.10.10.107/32"
ifconfig_fxp0_alias8="inet 10.10.10.108/32"
ifconfig_fxp0_alias9="inet 10.10.10.109/32"

Creating the first jail

The following command creates the first jail. You can create
additional jails by changing the hostname and the IP address.
Each jail needs a unique IP address and hostname. Ideally,
the IP address will correspond to the IP address you have selected.

# ezjail-admin create -f bacula mysql41.example.org 10.10.10.100
/usr/jails/mysql41.example.org/COPYRIGHT
/usr/jails/mysql41.example.org/basejail
/usr/jails/mysql41.example.org/bin
/usr/jails/mysql41.example.org/boot
/usr/jails/mysql41.example.org/dev
...
...
...
/usr/jails/mysql41.example.org/var/cfengine/inputs
/usr/jails/mysql41.example.org/var/cfengine/inputs/update.conf
/usr/jails/mysql41.example.org/var/cfengine/master
40797 blocks
Note: Shell scripts installed, flavourizing on jails first startup.
Warning: Some services already seem to be listening on all IP, (including 10.55.0.100)
  This may cause some confusion, here they are:
root     bacula-fd  1023  3  tcp4   *:9102                *:*
root     sshd       1010  4  tcp4   *:22                  *:*
root     master     995   11 tcp4   *:25                  *:*
root     syslogd    816   7  udp4   *:514                 *:*
[root@polo ~]# jls

Of note, you see the services running on the host which are listening
on all IP addresses. This will be a problem for the jails. I will need
to change the host services to listen only on the main IP address for this
host. Sometimes you might want a host service to listen on all IP addresses,
but that’s not what I want here.

The -f argument indicates that the jail should be created using
the bacula flavour. This flavour was set up back in March, and
I do not have my notes from that work. I will probably be creating
more flavours and if I do, I will document them.
But I do have a file list. Here is is the list
of files from my bacula ezjail flavour:

[dan@polo /usr/jails/flavours/bacula]$ find .
.
./etc
./etc/mail
./etc/mail/aliases
./etc/mail/mailer.conf
./etc/ssh
./etc/ssh/ssh_host_rsa_key.pub
./etc/ssh/ssh_host_rsa_key
./etc/ssh/ssh_host_key.pub
./etc/ssh/ssh_host_key
./etc/ssh/ssh_host_dsa_key.pub
./etc/ssh/ssh_host_dsa_key
./etc/rc.conf
./etc/crontab
./etc/syslog.conf
./etc/make.conf
./etc/resolv.conf
./etc/periodic.conf
./usr
./usr/local
./usr/local/etc
./usr/local/etc/sudoers
./usr/home
./usr/home/dan
./usr/home/dan/.ssh
./usr/home/dan/.ssh/id_dsa
./usr/home/dan/.ssh/id_dsa.pub
./usr/home/dan/.ssh/id_dsa_no_passphrase
./usr/home/dan/.ssh/id_dsa_no_passphrase.pub
./usr/home/dan/.ssh/authorized_keys
./usr/home/dan/.ssh/known_hosts
./pkg
./pkg/joe.tbz
./pkg/libiconv.tbz
./pkg/gettext.tbz
./pkg/cvsup-without-gui.tbz
./pkg/cfengine-2.2.3_1.tbz
./pkg/db46-4.6.21.0.tbz
./pkg/aspell.tbz
./pkg/bash.tbz
./var
./var/cfengine
./var/cfengine/inputs
./var/cfengine/inputs/update.conf
./var/cfengine/master
./ezjail.flavour
[dan@polo /usr/jails/flavours/bacula]$ 

See Modifying other daemons in the
Jails under FreeBSD 6 article for more
information on how I’ve done this in the past.

Starting the jail

Now that I have created my jail, it is time to start it:

#  /usr/local/etc/rc.d/ezjail.sh start
 ezjailConfiguring jails:.
Starting jails: mysql41.example.org.

I verified the jail was running:

# jls
   JID  IP Address    Hostname                Path
     1  10.10.10.100  mysql41.example.org     /usr/jails/mysql41.example.org

I was able to ssh to the newly started jail:

$ uptime
 1:19PM  up  1:33, 1 user, load averages: 0.00, 0.00, 0.00
$ hostname
mysql41.example.org
$ 

Amending your flavour and refreshing the jail

NOTE: this attempt failed. You may not want to try this. Read
the whole section first, then see what actually succeeded.

After logging into the new jail, I noticed my shell was missing my
favourite bash prompt. I decided to copy the required files into the
flavour and refresh the jail.

cp ~/.bash_profile ~/.bashrc /usr/jails/flavours/bacula/usr/home/dan/

Now I needed to stop the jail and update its image. Then restart the jail.

# /usr/local/etc/rc.d/ezjail.sh stop
 ezjailStopping jails: mysql41.example.org.
# ezjail-admin delete mysql41.example.org
# ezjail-admin create -x -f bacula mysql41.example.org 10.10.10.100
Warning: Some services already seem to be listening on all IP, (including 10.10.10.100)
  This may cause some confusion, here they are:
root     bacula-fd  1483  3  tcp4   *:9102                *:*
root     master     994   11 tcp4   *:25                  *:*
root     syslogd    815   7  udp4   *:514                 *:*

As you can see, I still have a few daemons listening on all addresses. I will
handle that eventually. However, my goal of recreating the jail with my
bash configuration files failed.

What I had to do was delete the jail with the -w option, to wipe
the jail from the HDD. Then create it again:

ezjail-admin delete -w mysql41.example.org
ezjail-admin create -f bacula mysql41.example.org 10.10.10.100
/usr/local/etc/rc.d/ezjail.sh start

Creating and starting a second jail

Now I will create another jail for testing MySQL 5.0:

[root@polo /usr/jails/flavours/bacula]# ezjail-admin create -f bacula mysql50.example.org 10.55.0.101
/usr/jails/mysql50.example.org/COPYRIGHT
/usr/jails/mysql50.example.org/basejail
/usr/jails/mysql50.example.org/bin
/usr/jails/mysql50.example.org/boot
/usr/jails/mysql50.example.org/dev
...
...
/usr/jails/mysql50.example.org/var/cfengine
/usr/jails/mysql50.example.org/var/cfengine/inputs
/usr/jails/mysql50.example.org/var/cfengine/inputs/update.conf
/usr/jails/mysql50.example.org/var/cfengine/master
/usr/jails/mysql50.example.org/var/ports
/usr/jails/mysql50.example.org/var/ports/packages
67260 blocks
Note: Shell scripts installed, flavourizing on jails first startup.
Warning: Some services already seem to be listening on IP 10.55.0.101
  This may cause some confusion, here they are:
root     ntpd       65252 8  udp4   10.55.0.101:123       *:*
Warning: Some services already seem to be listening on all IP, (including 10.55.0.101)
  This may cause some confusion, here they are:
root     ntpd       65252 4  udp4   *:123                 *:*
[root@polo /usr/jails/flavours/bacula]#  /usr/local/etc/rc.d/ezjail.sh      
Usage: /usr/local/etc/rc.d/ezjail.sh [fast|force|one](start|stop|restart|rcvar|startcrypto|stopcrypto)
[root@polo /usr/jails/flavours/bacula]#  /usr/local/etc/rc.d/ezjail.sh start
ezjailConfiguring jails:.
Starting jails: mysql50.example.org [mysql41.example.org already running 
                            (/var/run/jail_mysql41_unixathome_org.id exists)].
[root@polo /usr/jails/flavours/bacula]# 

And here is that jail:

$ hostname
mysql50.example.org
$ uname -a
FreeBSD mysql50.example.org 7.0-STABLE FreeBSD 7.0-STABLE #4: Fri Jul  
4 14:01:32 EDT 2008     dan@polo.example.org:/usr/obj/usr/src/sys/PHENOM  
amd64
$ bash
[dan@mysql50:/usr/home/dan] $ cd
[dan@mysql50:~] $ uptime
 6:03PM  up  6:17, 1 user, load averages: 0.07, 0.02, 0.02

Better jail start/stop

You can start/stop individual jails.

#  /usr/local/etc/rc.d/ezjail.sh stop mysql50.example.org
Stopping jails: mysql50.example.org.
#  /usr/local/etc/rc.d/ezjail.sh start mysql50.example.org
Configuring jails:.
Starting jails: mysql50.example.org.
#

Upgrading a jail (added on 23 Aug 2008)

I’ve decided it is time to upgrade these jails. The host system is now
running:


FreeBSD polo.example.org 7.0-STABLE FreeBSD 7.0-STABLE #5: Fri Aug 22 13:45:47 EDT 2008     dan@polo.example.org:/usr/obj/usr/src/sys/PHENOM  amd64

Whereas, the jails are:


FreeBSD mysql50.example.org 7.0-STABLE FreeBSD 7.0-STABLE #5: Fri Aug 22 13:45:47 EDT 2008    dan@polo.example.org:/usr/obj/usr/src/sys/PHENOM  amd64

Yes, they look identical. But to upgrade the ezjail after doing a build
world in there, I did this:

What’s next?

That’s my first cut of using ezjail. I’ll be amending and adding to my
flavours as I add more jails. I also plan to use cfengine to configure
the jails, install/upgrade ports, and to undertake general maintenance.
Watch for cfengine in an upcoming article.

6 thoughts on “ezjail – A jail administration framework”

  1. Salvor Hardin

    Good article. Ezjail is incredibly useful. I combine it with NATD to create a bunch of jails all accessible through one public IP address. I encode each jail’s unique ports in the last octet of the private IP address space, using NATD directives like this:

    log yes

    # JAIL 10
    redirect_port tcp 192.168.9.10:22 1022
    redirect_port tcp 192.168.9.10:80 1080

    # JAIL 11
    redirect_port tcp 192.168.9.11:22 1122
    redirect_port tcp 192.168.9.11:80 1180
    redirect_port tcp 192.168.9.11:443 11443

    Each jail’s services are on unique ports, using URLs like https://www.example.com:11443/, for example, and ssh -p 1122. The only problem I have had is that some of our large corporate clients block all incoming and outgoing ports above 999. Baffling, but true!

    Ezjail and natd help us leverage a single inexpensive server to host 35 development environments. Very cool!

    The numbering scheme limits me to 56 jails, because port numbers top out at 65535, making 65443 the highest available https port. We’ve never come close to topping that out, so this works very well for us.

    [%sig%]

  2. Hi,

    ezjail isn’t the fun way to play with jails. ZFS is.

    First of all, the /etc/rc.d/jail script has been quite updated, and will take care of the day-to-day stuff, such as starting and stopping jails.

    Simply define:

    jail_enable="YES"
    jail_list="psql"
    jail_psql_rootdir="/usr/local/jails/psql.freebsddiary.org"
    jail_psql_hostname="psql.freebsddiary.org"
    jail_psql_ip="10.10.10.10"
    jail_psql_devfs_enable="YES"
    jail_psql_devfs_ruleset="devfsrules_jail" # devfs ruleset to apply to jail

    And you can do:

    /etc/rc.d/jail start psql

    That’s it. No need for a port to retrofit this.

    Also, for the fun part. If you throw in ZFS in the loop, you get better control over disk usage (pr. jail or even pr. dir quotas, compression, redundancy-levels etc).

    You can also snapshot jails.

    That’s right. If you’re going to upgrade a jailed "machine", just snapshot it, test it, and roll back if there’s a problem. If you have read-write userdata, simply put it in a separate zfs, and you can do it with parts of the "virtual machine".

    But wait! There’s more!

    If you can create a base config, with most of the things you’ll typically need for a jail, you can then snapshot and clone it.

    If you have a base jail, and you’d like to install a 4MB package in addition to your standard stuff, the entire jail will only take up that 4MB of space.

    You save RAM too. I think. Not entirely sure how ZFS fits into the FreeBSD page system, but since you’re running your libc and whatnot from the same disk pages for all your jails, I’d think it’s quite likely you’ll save up some RAM too.

    Sounds like a good plan?

    1. tld wrote:

      > Sounds like a good plan?

      All except: ZFS isn’t quite ready for production, from everything I’ve heard.


      The Man Behind The Curtain

    2. Details, details. 😉

      ZFS is "different" for a lot of FreeBSD people. We’re used to anything in FreeBSD being stable on anything it can run on. ZFS seems to have issues on low memory i386 machines, but work quite well on amd64 machines with 4GB+.

      In other words, I guess it all depends on your needs. ZFS also offers a lot of convenient features which are tempting to have in production, such as protection against silent datacorruption (seen that a couple of times too many), being able to use all the space effectively without downtime for reinstall, etc etc.

      Might not want to dive in and convert anythign and everything to run ZFS, but it’s definitively ready for testing out.

  3. Note: the "create -f bacula" part of the ezjail command assumes you have already created a flavor named ‘bacula’.

    Sorry. I omitted that stage during my note taking.


    The Man Behind The Curtain

Leave a Comment

Scroll to Top