ezjail – A jail administration framework
I want to set up some jails. They will each be very similar.
They will each be used to test a slightly different configuration
of Bacula. My tool of choice
is ezjail,
available in the ports tree.
With ezjail, I can:
- create a jail flavour, upon which the creation of other jails can be based
- centrally update the jail’s ports tree
The above does not fully describe the neat things you can do with ezjail.
Read below to discover more fun and interesting things.
DISCLAIMER: I installed ezjail several months ago. I am only now getting
around to documenting and writing about it. I may have omitted some
steps. If so, I apologize; please let me know.
While creating my jails I used the following references:
Installation of ezjail
First step installation:
cd /usr/ports/sysutils/ezjail make install clean
Remember to add this to /etc/rc.conf:
ezjail_enable="YES"
Remember to create an ezjail configuration file:
cd /usr/local/etc cp ezjail.conf.sample ezjail.conf
Create your base jail
This command creates the base jail. The base jail is the one upon which
all other jails will be based. Note that this does not create a jail,
it creates a base jail, one of the foundation components of your
ezjail configuration.
ezjail-admin update -ip
This assumes you have already done a build world and that it exists within
the standard and default location: /usr/src.
After running this command you should see this:
# ls /usr/jails basejail flavours newjail
It also installs a copy of the ports tree into the base jail, using portsnap.
I encountered this error while trying to create a new basejail.
Upgrading to the latest STABLE fixed this problem.
install -o root -g wheel -m 444 dir-tmpl /usr/jails/fulljail/usr/share/info/dir install:No such file or directory
Creating the IP addresses for the jails
The host system needs to have some IP addresses that will be used
exclusively by the jails. That is, one IP address per jail. For
ease of use, and personal sanity, I’ve added the following entries
to my private DNS server:
; jails for testing Bacula mysql41 IN A 10.10.10.100 mysql50 IN A 10.10.10.101 mysql51 IN A 10.10.10.102 pg73 IN A 10.10.10.103 pg74 IN A 10.10.10.104 pg80 IN A 10.10.10.105 pg81 IN A 10.10.10.106 pg82 IN A 10.10.10.107 pg83 IN A 10.10.10.109 sqlite3 IN A 10.10.10.108
Corresponding to the above, here are the ifconfig aliases I added to
/etc/rc.conf on the host system:
ifconfig_fxp0_alias0="inet 10.10.10.100/32" ifconfig_fxp0_alias1="inet 10.10.10.101/32" ifconfig_fxp0_alias2="inet 10.10.10.102/32" ifconfig_fxp0_alias3="inet 10.10.10.103/32" ifconfig_fxp0_alias4="inet 10.10.10.104/32" ifconfig_fxp0_alias5="inet 10.10.10.105/32" ifconfig_fxp0_alias6="inet 10.10.10.106/32" ifconfig_fxp0_alias7="inet 10.10.10.107/32" ifconfig_fxp0_alias8="inet 10.10.10.108/32" ifconfig_fxp0_alias9="inet 10.10.10.109/32"
Creating the first jail
The following command creates the first jail. You can create
additional jails by changing the hostname and the IP address.
Each jail needs a unique IP address and hostname. Ideally,
the IP address will correspond to the IP address you have selected.
# ezjail-admin create -f bacula mysql41.example.org 10.10.10.100 /usr/jails/mysql41.example.org/COPYRIGHT /usr/jails/mysql41.example.org/basejail /usr/jails/mysql41.example.org/bin /usr/jails/mysql41.example.org/boot /usr/jails/mysql41.example.org/dev ... ... ... /usr/jails/mysql41.example.org/var/cfengine/inputs /usr/jails/mysql41.example.org/var/cfengine/inputs/update.conf /usr/jails/mysql41.example.org/var/cfengine/master 40797 blocks Note: Shell scripts installed, flavourizing on jails first startup. Warning: Some services already seem to be listening on all IP, (including 10.55.0.100) This may cause some confusion, here they are: root bacula-fd 1023 3 tcp4 *:9102 *:* root sshd 1010 4 tcp4 *:22 *:* root master 995 11 tcp4 *:25 *:* root syslogd 816 7 udp4 *:514 *:* [root@polo ~]# jls
Of note, you see the services running on the host which are listening
on all IP addresses. This will be a problem for the jails. I will need
to change the host services to listen only on the main IP address for this
host. Sometimes you might want a host service to listen on all IP addresses,
but that’s not what I want here.
The -f argument indicates that the jail should be created using
the bacula flavour. This flavour was set up back in March, and
I do not have my notes from that work. I will probably be creating
more flavours and if I do, I will document them.
But I do have a file list. Here is is the list
of files from my bacula ezjail flavour:
[dan@polo /usr/jails/flavours/bacula]$ find . . ./etc ./etc/mail ./etc/mail/aliases ./etc/mail/mailer.conf ./etc/ssh ./etc/ssh/ssh_host_rsa_key.pub ./etc/ssh/ssh_host_rsa_key ./etc/ssh/ssh_host_key.pub ./etc/ssh/ssh_host_key ./etc/ssh/ssh_host_dsa_key.pub ./etc/ssh/ssh_host_dsa_key ./etc/rc.conf ./etc/crontab ./etc/syslog.conf ./etc/make.conf ./etc/resolv.conf ./etc/periodic.conf ./usr ./usr/local ./usr/local/etc ./usr/local/etc/sudoers ./usr/home ./usr/home/dan ./usr/home/dan/.ssh ./usr/home/dan/.ssh/id_dsa ./usr/home/dan/.ssh/id_dsa.pub ./usr/home/dan/.ssh/id_dsa_no_passphrase ./usr/home/dan/.ssh/id_dsa_no_passphrase.pub ./usr/home/dan/.ssh/authorized_keys ./usr/home/dan/.ssh/known_hosts ./pkg ./pkg/joe.tbz ./pkg/libiconv.tbz ./pkg/gettext.tbz ./pkg/cvsup-without-gui.tbz ./pkg/cfengine-2.2.3_1.tbz ./pkg/db46-4.6.21.0.tbz ./pkg/aspell.tbz ./pkg/bash.tbz ./var ./var/cfengine ./var/cfengine/inputs ./var/cfengine/inputs/update.conf ./var/cfengine/master ./ezjail.flavour [dan@polo /usr/jails/flavours/bacula]$
See Modifying other daemons in the
Jails under FreeBSD 6 article for more
information on how I’ve done this in the past.
Starting the jail
Now that I have created my jail, it is time to start it:
# /usr/local/etc/rc.d/ezjail.sh start ezjailConfiguring jails:. Starting jails: mysql41.example.org.
I verified the jail was running:
# jls
JID IP Address Hostname Path
1 10.10.10.100 mysql41.example.org /usr/jails/mysql41.example.org
I was able to ssh to the newly started jail:
$ uptime 1:19PM up 1:33, 1 user, load averages: 0.00, 0.00, 0.00 $ hostname mysql41.example.org $
Amending your flavour and refreshing the jail
NOTE: this attempt failed. You may not want to try this. Read
the whole section first, then see what actually succeeded.
After logging into the new jail, I noticed my shell was missing my
favourite bash prompt. I decided to copy the required files into the
flavour and refresh the jail.
cp ~/.bash_profile ~/.bashrc /usr/jails/flavours/bacula/usr/home/dan/
Now I needed to stop the jail and update its image. Then restart the jail.
# /usr/local/etc/rc.d/ezjail.sh stop ezjailStopping jails: mysql41.example.org. # ezjail-admin delete mysql41.example.org # ezjail-admin create -x -f bacula mysql41.example.org 10.10.10.100 Warning: Some services already seem to be listening on all IP, (including 10.10.10.100) This may cause some confusion, here they are: root bacula-fd 1483 3 tcp4 *:9102 *:* root master 994 11 tcp4 *:25 *:* root syslogd 815 7 udp4 *:514 *:*
As you can see, I still have a few daemons listening on all addresses. I will
handle that eventually. However, my goal of recreating the jail with my
bash configuration files failed.
What I had to do was delete the jail with the -w option, to wipe
the jail from the HDD. Then create it again:
ezjail-admin delete -w mysql41.example.org ezjail-admin create -f bacula mysql41.example.org 10.10.10.100 /usr/local/etc/rc.d/ezjail.sh start
Creating and starting a second jail
Now I will create another jail for testing MySQL 5.0:
[root@polo /usr/jails/flavours/bacula]# ezjail-admin create -f bacula mysql50.example.org 10.55.0.101
/usr/jails/mysql50.example.org/COPYRIGHT
/usr/jails/mysql50.example.org/basejail
/usr/jails/mysql50.example.org/bin
/usr/jails/mysql50.example.org/boot
/usr/jails/mysql50.example.org/dev
...
...
/usr/jails/mysql50.example.org/var/cfengine
/usr/jails/mysql50.example.org/var/cfengine/inputs
/usr/jails/mysql50.example.org/var/cfengine/inputs/update.conf
/usr/jails/mysql50.example.org/var/cfengine/master
/usr/jails/mysql50.example.org/var/ports
/usr/jails/mysql50.example.org/var/ports/packages
67260 blocks
Note: Shell scripts installed, flavourizing on jails first startup.
Warning: Some services already seem to be listening on IP 10.55.0.101
This may cause some confusion, here they are:
root ntpd 65252 8 udp4 10.55.0.101:123 *:*
Warning: Some services already seem to be listening on all IP, (including 10.55.0.101)
This may cause some confusion, here they are:
root ntpd 65252 4 udp4 *:123 *:*
[root@polo /usr/jails/flavours/bacula]# /usr/local/etc/rc.d/ezjail.sh
Usage: /usr/local/etc/rc.d/ezjail.sh [fast|force|one](start|stop|restart|rcvar|startcrypto|stopcrypto)
[root@polo /usr/jails/flavours/bacula]# /usr/local/etc/rc.d/ezjail.sh start
ezjailConfiguring jails:.
Starting jails: mysql50.example.org [mysql41.example.org already running
(/var/run/jail_mysql41_unixathome_org.id exists)].
[root@polo /usr/jails/flavours/bacula]#
And here is that jail:
$ hostname mysql50.example.org $ uname -a FreeBSD mysql50.example.org 7.0-STABLE FreeBSD 7.0-STABLE #4: Fri Jul 4 14:01:32 EDT 2008 dan@polo.example.org:/usr/obj/usr/src/sys/PHENOM amd64 $ bash [dan@mysql50:/usr/home/dan] $ cd [dan@mysql50:~] $ uptime 6:03PM up 6:17, 1 user, load averages: 0.07, 0.02, 0.02
Better jail start/stop
You can start/stop individual jails.
# /usr/local/etc/rc.d/ezjail.sh stop mysql50.example.org Stopping jails: mysql50.example.org. # /usr/local/etc/rc.d/ezjail.sh start mysql50.example.org Configuring jails:. Starting jails: mysql50.example.org. #
Upgrading a jail (added on 23 Aug 2008)
I’ve decided it is time to upgrade these jails. The host system is now
running:
FreeBSD polo.example.org 7.0-STABLE FreeBSD 7.0-STABLE #5: Fri Aug 22 13:45:47 EDT 2008 dan@polo.example.org:/usr/obj/usr/src/sys/PHENOM amd64
Whereas, the jails are:
FreeBSD mysql50.example.org 7.0-STABLE FreeBSD 7.0-STABLE #5: Fri Aug 22 13:45:47 EDT 2008 dan@polo.example.org:/usr/obj/usr/src/sys/PHENOM amd64
Yes, they look identical. But to upgrade the ezjail after doing a build
world in there, I did this:
What’s next?
That’s my first cut of using ezjail. I’ll be amending and adding to my
flavours as I add more jails. I also plan to use cfengine to configure
the jails, install/upgrade ports, and to undertake general maintenance.
Watch for cfengine in an upcoming article.
Good article. Ezjail is incredibly useful. I combine it with NATD to create a bunch of jails all accessible through one public IP address. I encode each jail’s unique ports in the last octet of the private IP address space, using NATD directives like this:
log yes
# JAIL 10
redirect_port tcp 192.168.9.10:22 1022
redirect_port tcp 192.168.9.10:80 1080
# JAIL 11
redirect_port tcp 192.168.9.11:22 1122
redirect_port tcp 192.168.9.11:80 1180
redirect_port tcp 192.168.9.11:443 11443
Each jail’s services are on unique ports, using URLs like https://www.example.com:11443/, for example, and ssh -p 1122. The only problem I have had is that some of our large corporate clients block all incoming and outgoing ports above 999. Baffling, but true!
Ezjail and natd help us leverage a single inexpensive server to host 35 development environments. Very cool!
The numbering scheme limits me to 56 jails, because port numbers top out at 65535, making 65443 the highest available https port. We’ve never come close to topping that out, so this works very well for us.
[%sig%]
Rather than using natd, you could probably do just fine with pf. That’d get you out of all the packets having to go to userland and back as well.
Hi,
ezjail isn’t the fun way to play with jails. ZFS is.
First of all, the /etc/rc.d/jail script has been quite updated, and will take care of the day-to-day stuff, such as starting and stopping jails.
Simply define:
jail_enable="YES"
jail_list="psql"
jail_psql_rootdir="/usr/local/jails/psql.freebsddiary.org"
jail_psql_hostname="psql.freebsddiary.org"
jail_psql_ip="10.10.10.10"
jail_psql_devfs_enable="YES"
jail_psql_devfs_ruleset="devfsrules_jail" # devfs ruleset to apply to jail
And you can do:
/etc/rc.d/jail start psql
That’s it. No need for a port to retrofit this.
Also, for the fun part. If you throw in ZFS in the loop, you get better control over disk usage (pr. jail or even pr. dir quotas, compression, redundancy-levels etc).
You can also snapshot jails.
That’s right. If you’re going to upgrade a jailed "machine", just snapshot it, test it, and roll back if there’s a problem. If you have read-write userdata, simply put it in a separate zfs, and you can do it with parts of the "virtual machine".
But wait! There’s more!
If you can create a base config, with most of the things you’ll typically need for a jail, you can then snapshot and clone it.
If you have a base jail, and you’d like to install a 4MB package in addition to your standard stuff, the entire jail will only take up that 4MB of space.
You save RAM too. I think. Not entirely sure how ZFS fits into the FreeBSD page system, but since you’re running your libc and whatnot from the same disk pages for all your jails, I’d think it’s quite likely you’ll save up some RAM too.
Sounds like a good plan?
tld wrote:
> Sounds like a good plan?
All except: ZFS isn’t quite ready for production, from everything I’ve heard.
—
The Man Behind The Curtain
Details, details. 😉
ZFS is "different" for a lot of FreeBSD people. We’re used to anything in FreeBSD being stable on anything it can run on. ZFS seems to have issues on low memory i386 machines, but work quite well on amd64 machines with 4GB+.
In other words, I guess it all depends on your needs. ZFS also offers a lot of convenient features which are tempting to have in production, such as protection against silent datacorruption (seen that a couple of times too many), being able to use all the space effectively without downtime for reinstall, etc etc.
Might not want to dive in and convert anythign and everything to run ZFS, but it’s definitively ready for testing out.
Note: the "create -f bacula" part of the ezjail command assumes you have already created a flavor named ‘bacula’.
Sorry. I omitted that stage during my note taking.
—
The Man Behind The Curtain