ezjail – A jail administration frameworkI want to set up some jails. They will each be very similar. They will each be used to test a slightly different configuration of Bacula. My tool of choice is ezjail, available in the ports tree. With ezjail, I can:
- create a jail flavour, upon which the creation of other jails can be based
- centrally update the jail’s ports tree
Installation of ezjailFirst step installation:
cd /usr/ports/sysutils/ezjail make install cleanRemember to add this to /etc/rc.conf:
ezjail_enable="YES"Remember to create an ezjail configuration file:
cd /usr/local/etc cp ezjail.conf.sample ezjail.conf
Create your base jailThis command creates the base jail. The base jail is the one upon which all other jails will be based. Note that this does not create a jail, it creates a base jail, one of the foundation components of your ezjail configuration.
ezjail-admin update -ipThis assumes you have already done a build world and that it exists within the standard and default location: /usr/src. After running this command you should see this:
# ls /usr/jails basejail flavours newjailIt also installs a copy of the ports tree into the base jail, using portsnap. I encountered this error while trying to create a new basejail. Upgrading to the latest STABLE fixed this problem.
install -o root -g wheel -m 444 dir-tmpl /usr/jails/fulljail/usr/share/info/dir install:No such file or directory
Creating the IP addresses for the jailsThe host system needs to have some IP addresses that will be used exclusively by the jails. That is, one IP address per jail. For ease of use, and personal sanity, I’ve added the following entries to my private DNS server:
; jails for testing Bacula mysql41 IN A 10.10.10.100 mysql50 IN A 10.10.10.101 mysql51 IN A 10.10.10.102 pg73 IN A 10.10.10.103 pg74 IN A 10.10.10.104 pg80 IN A 10.10.10.105 pg81 IN A 10.10.10.106 pg82 IN A 10.10.10.107 pg83 IN A 10.10.10.109 sqlite3 IN A 10.10.10.108Corresponding to the above, here are the ifconfig aliases I added to /etc/rc.conf on the host system:
ifconfig_fxp0_alias0="inet 10.10.10.100/32" ifconfig_fxp0_alias1="inet 10.10.10.101/32" ifconfig_fxp0_alias2="inet 10.10.10.102/32" ifconfig_fxp0_alias3="inet 10.10.10.103/32" ifconfig_fxp0_alias4="inet 10.10.10.104/32" ifconfig_fxp0_alias5="inet 10.10.10.105/32" ifconfig_fxp0_alias6="inet 10.10.10.106/32" ifconfig_fxp0_alias7="inet 10.10.10.107/32" ifconfig_fxp0_alias8="inet 10.10.10.108/32" ifconfig_fxp0_alias9="inet 10.10.10.109/32"
Creating the first jailThe following command creates the first jail. You can create additional jails by changing the hostname and the IP address. Each jail needs a unique IP address and hostname. Ideally, the IP address will correspond to the IP address you have selected.
# ezjail-admin create -f bacula mysql41.example.org 10.10.10.100 /usr/jails/mysql41.example.org/COPYRIGHT /usr/jails/mysql41.example.org/basejail /usr/jails/mysql41.example.org/bin /usr/jails/mysql41.example.org/boot /usr/jails/mysql41.example.org/dev ... ... ... /usr/jails/mysql41.example.org/var/cfengine/inputs /usr/jails/mysql41.example.org/var/cfengine/inputs/update.conf /usr/jails/mysql41.example.org/var/cfengine/master 40797 blocks Note: Shell scripts installed, flavourizing on jails first startup. Warning: Some services already seem to be listening on all IP, (including 10.55.0.100) This may cause some confusion, here they are: root bacula-fd 1023 3 tcp4 *:9102 *:* root sshd 1010 4 tcp4 *:22 *:* root master 995 11 tcp4 *:25 *:* root syslogd 816 7 udp4 *:514 *:* [root@polo ~]# jlsOf note, you see the services running on the host which are listening on all IP addresses. This will be a problem for the jails. I will need to change the host services to listen only on the main IP address for this host. Sometimes you might want a host service to listen on all IP addresses, but that’s not what I want here. The -f argument indicates that the jail should be created using the bacula flavour. This flavour was set up back in March, and I do not have my notes from that work. I will probably be creating more flavours and if I do, I will document them. But I do have a file list. Here is is the list of files from my bacula ezjail flavour:
[dan@polo /usr/jails/flavours/bacula]$ find . . ./etc ./etc/mail ./etc/mail/aliases ./etc/mail/mailer.conf ./etc/ssh ./etc/ssh/ssh_host_rsa_key.pub ./etc/ssh/ssh_host_rsa_key ./etc/ssh/ssh_host_key.pub ./etc/ssh/ssh_host_key ./etc/ssh/ssh_host_dsa_key.pub ./etc/ssh/ssh_host_dsa_key ./etc/rc.conf ./etc/crontab ./etc/syslog.conf ./etc/make.conf ./etc/resolv.conf ./etc/periodic.conf ./usr ./usr/local ./usr/local/etc ./usr/local/etc/sudoers ./usr/home ./usr/home/dan ./usr/home/dan/.ssh ./usr/home/dan/.ssh/id_dsa ./usr/home/dan/.ssh/id_dsa.pub ./usr/home/dan/.ssh/id_dsa_no_passphrase ./usr/home/dan/.ssh/id_dsa_no_passphrase.pub ./usr/home/dan/.ssh/authorized_keys ./usr/home/dan/.ssh/known_hosts ./pkg ./pkg/joe.tbz ./pkg/libiconv.tbz ./pkg/gettext.tbz ./pkg/cvsup-without-gui.tbz ./pkg/cfengine-2.2.3_1.tbz ./pkg/db46-184.108.40.206.tbz ./pkg/aspell.tbz ./pkg/bash.tbz ./var ./var/cfengine ./var/cfengine/inputs ./var/cfengine/inputs/update.conf ./var/cfengine/master ./ezjail.flavour [dan@polo /usr/jails/flavours/bacula]$See Modifying other daemons in the Jails under FreeBSD 6 article for more information on how I’ve done this in the past.
Starting the jailNow that I have created my jail, it is time to start it:
# /usr/local/etc/rc.d/ezjail.sh start ezjailConfiguring jails:. Starting jails: mysql41.example.org.I verified the jail was running:
# jls JID IP Address Hostname Path 1 10.10.10.100 mysql41.example.org /usr/jails/mysql41.example.orgI was able to ssh to the newly started jail:
$ uptime 1:19PM up 1:33, 1 user, load averages: 0.00, 0.00, 0.00 $ hostname mysql41.example.org $
Amending your flavour and refreshing the jailNOTE: this attempt failed. You may not want to try this. Read the whole section first, then see what actually succeeded. After logging into the new jail, I noticed my shell was missing my favourite bash prompt. I decided to copy the required files into the flavour and refresh the jail.
cp ~/.bash_profile ~/.bashrc /usr/jails/flavours/bacula/usr/home/dan/Now I needed to stop the jail and update its image. Then restart the jail.
# /usr/local/etc/rc.d/ezjail.sh stop ezjailStopping jails: mysql41.example.org. # ezjail-admin delete mysql41.example.org # ezjail-admin create -x -f bacula mysql41.example.org 10.10.10.100 Warning: Some services already seem to be listening on all IP, (including 10.10.10.100) This may cause some confusion, here they are: root bacula-fd 1483 3 tcp4 *:9102 *:* root master 994 11 tcp4 *:25 *:* root syslogd 815 7 udp4 *:514 *:*As you can see, I still have a few daemons listening on all addresses. I will handle that eventually. However, my goal of recreating the jail with my bash configuration files failed. What I had to do was delete the jail with the -w option, to wipe the jail from the HDD. Then create it again:
ezjail-admin delete -w mysql41.example.org ezjail-admin create -f bacula mysql41.example.org 10.10.10.100 /usr/local/etc/rc.d/ezjail.sh start
Creating and starting a second jailNow I will create another jail for testing MySQL 5.0:
[root@polo /usr/jails/flavours/bacula]# ezjail-admin create -f bacula mysql50.example.org 10.55.0.101 /usr/jails/mysql50.example.org/COPYRIGHT /usr/jails/mysql50.example.org/basejail /usr/jails/mysql50.example.org/bin /usr/jails/mysql50.example.org/boot /usr/jails/mysql50.example.org/dev ... ... /usr/jails/mysql50.example.org/var/cfengine /usr/jails/mysql50.example.org/var/cfengine/inputs /usr/jails/mysql50.example.org/var/cfengine/inputs/update.conf /usr/jails/mysql50.example.org/var/cfengine/master /usr/jails/mysql50.example.org/var/ports /usr/jails/mysql50.example.org/var/ports/packages 67260 blocks Note: Shell scripts installed, flavourizing on jails first startup. Warning: Some services already seem to be listening on IP 10.55.0.101 This may cause some confusion, here they are: root ntpd 65252 8 udp4 10.55.0.101:123 *:* Warning: Some services already seem to be listening on all IP, (including 10.55.0.101) This may cause some confusion, here they are: root ntpd 65252 4 udp4 *:123 *:* [root@polo /usr/jails/flavours/bacula]# /usr/local/etc/rc.d/ezjail.sh Usage: /usr/local/etc/rc.d/ezjail.sh [fast|force|one](start|stop|restart|rcvar|startcrypto|stopcrypto) [root@polo /usr/jails/flavours/bacula]# /usr/local/etc/rc.d/ezjail.sh start ezjailConfiguring jails:. Starting jails: mysql50.example.org [mysql41.example.org already running (/var/run/jail_mysql41_unixathome_org.id exists)]. [root@polo /usr/jails/flavours/bacula]#And here is that jail:
$ hostname mysql50.example.org $ uname -a FreeBSD mysql50.example.org 7.0-STABLE FreeBSD 7.0-STABLE #4: Fri Jul 4 14:01:32 EDT 2008 firstname.lastname@example.org:/usr/obj/usr/src/sys/PHENOM amd64 $ bash [dan@mysql50:/usr/home/dan] $ cd [dan@mysql50:~] $ uptime 6:03PM up 6:17, 1 user, load averages: 0.07, 0.02, 0.02
Better jail start/stopYou can start/stop individual jails.
# /usr/local/etc/rc.d/ezjail.sh stop mysql50.example.org Stopping jails: mysql50.example.org. # /usr/local/etc/rc.d/ezjail.sh start mysql50.example.org Configuring jails:. Starting jails: mysql50.example.org. #
Upgrading a jail (added on 23 Aug 2008)I’ve decided it is time to upgrade these jails. The host system is now running:
Whereas, the jails are:
FreeBSD polo.example.org 7.0-STABLE FreeBSD 7.0-STABLE #5: Fri Aug 22 13:45:47 EDT 2008 email@example.com:/usr/obj/usr/src/sys/PHENOM amd64
Yes, they look identical. But to upgrade the ezjail after doing a build world in there, I did this:
FreeBSD mysql50.example.org 7.0-STABLE FreeBSD 7.0-STABLE #5: Fri Aug 22 13:45:47 EDT 2008 firstname.lastname@example.org:/usr/obj/usr/src/sys/PHENOM amd64
Good article. Ezjail is incredibly useful. I combine it with NATD to create a bunch of jails all accessible through one public IP address. I encode each jail’s unique ports in the last octet of the private IP address space, using NATD directives like this:
# JAIL 10
redirect_port tcp 192.168.9.10:22 1022
redirect_port tcp 192.168.9.10:80 1080
# JAIL 11
redirect_port tcp 192.168.9.11:22 1122
redirect_port tcp 192.168.9.11:80 1180
redirect_port tcp 192.168.9.11:443 11443
Each jail’s services are on unique ports, using URLs like https://www.example.com:11443/, for example, and ssh -p 1122. The only problem I have had is that some of our large corporate clients block all incoming and outgoing ports above 999. Baffling, but true!
Ezjail and natd help us leverage a single inexpensive server to host 35 development environments. Very cool!
The numbering scheme limits me to 56 jails, because port numbers top out at 65535, making 65443 the highest available https port. We’ve never come close to topping that out, so this works very well for us.
Rather than using natd, you could probably do just fine with pf. That’d get you out of all the packets having to go to userland and back as well.
ezjail isn’t the fun way to play with jails. ZFS is.
First of all, the /etc/rc.d/jail script has been quite updated, and will take care of the day-to-day stuff, such as starting and stopping jails.
jail_psql_devfs_ruleset="devfsrules_jail" # devfs ruleset to apply to jail
And you can do:
/etc/rc.d/jail start psql
That’s it. No need for a port to retrofit this.
Also, for the fun part. If you throw in ZFS in the loop, you get better control over disk usage (pr. jail or even pr. dir quotas, compression, redundancy-levels etc).
You can also snapshot jails.
That’s right. If you’re going to upgrade a jailed "machine", just snapshot it, test it, and roll back if there’s a problem. If you have read-write userdata, simply put it in a separate zfs, and you can do it with parts of the "virtual machine".
But wait! There’s more!
If you can create a base config, with most of the things you’ll typically need for a jail, you can then snapshot and clone it.
If you have a base jail, and you’d like to install a 4MB package in addition to your standard stuff, the entire jail will only take up that 4MB of space.
You save RAM too. I think. Not entirely sure how ZFS fits into the FreeBSD page system, but since you’re running your libc and whatnot from the same disk pages for all your jails, I’d think it’s quite likely you’ll save up some RAM too.
Sounds like a good plan?
> Sounds like a good plan?
All except: ZFS isn’t quite ready for production, from everything I’ve heard.
The Man Behind The Curtain
Details, details. 😉
ZFS is "different" for a lot of FreeBSD people. We’re used to anything in FreeBSD being stable on anything it can run on. ZFS seems to have issues on low memory i386 machines, but work quite well on amd64 machines with 4GB+.
In other words, I guess it all depends on your needs. ZFS also offers a lot of convenient features which are tempting to have in production, such as protection against silent datacorruption (seen that a couple of times too many), being able to use all the space effectively without downtime for reinstall, etc etc.
Might not want to dive in and convert anythign and everything to run ZFS, but it’s definitively ready for testing out.
Note: the "create -f bacula" part of the ezjail command assumes you have already created a flavor named ‘bacula’.
Sorry. I omitted that stage during my note taking.
The Man Behind The Curtain