Using a jail as a virtual machine
This article shows you how I created a jail for the
OSW website. It runs in a jail on the
same system as this website. I originally did this install back in
November 2003 and the notes from that session form the basis of this article.
I have need to recreate the jail now as we recently had an HDD failure.
NOTE: This article applies to FreeBSD 4. I have since written about
jail on FreeBSD 5, and it appears to apply to
FreeBSD 6.x just as well. I recommend read that article over this one.
A jail is useful for many purposes. In my case, I wanted to give the OSW
project a place to run their websites, mailing lists, etc, but at the same
time keep them isolated from the rest of the machine. In short, it gives them
a virtual machine, and it gives me peace of mind knowing that I have less to
worry about with respect to the rest of the machine.
The main document for creating a jail is
man jail.
I followed the instructions listed under Setting up a Jail Directory
Tree. I followed those instructions to create the jail.
Of note, I did not do this:
- I left sendmail (actually, postfix) running. I just changed it so that
it did not listen on all IP addresses. This allowed the jail to run its own
mail server. - I did not specify the
portmap_enable="NO"
directive as that is default setting (see
/etc/defaults/rc.conf).
Modifying other daemons
Most daemons will listen to whatever IP addresses are available to them. After starting your jail,
if you try to ssh to it, you will not get into it. You’ll be in the host environment instead.
To get into the jail environment via ssh, you need to:
- Tell the host environment sshd not to listen to the jail’s IP address
- run sshd in the jail
Host environment sshd
To alter the host environment sshd so it listens only to host environment IP addresses,
modify /etc/ssh/sshd_config and set the IP address for the Listen directive:
ListenAddress 192.168.0.100
Then restart the main sshd process:
kill -HUP `cat /var/run/sshd.pid`
Use telnet to verify that the host environment is not listening on the jail address:
$ telnet 192.168.0.155 22
Trying 192.168.0.155...
telnet: connect to address 192.168.0.155: Connection refused
telnet: Unable to connect to remote host
If you don’t get a connection, the host environment is not listening. This assumes that you have not yet
started sshd in the jail environment.
Jail environment sshd
To start sshd in the jail environment, add the following line to /etc/rc.conf:
sshd_enable="YES"
Starting the jail for the first time
From man jail, to start a jail, issue this command:
[root@mtwenty:/home/dan] # jail /usr/jails/192.168.0.155 osw.example.org 192.168.0.155 /bin/sh
$
That prompt (#) indicates you are now in the jail environment. Now you can run the start up processes:
$ sh /etc/rc
Loading configuration files.
mdmfs: mdconfig (attach) exited with error code 1
cd: can't cd to ppp
rm: utmp: Permission denied
cp: utmp: Permission denied
/etc/rc.d/cleanvar: cannot create /var/run/clean_var: Permission denied
undef.unixathome.org
Setting hostname: undef.unixathome.org.
Generating nsswitch.conf.
eval: cannot create /etc/nsswitch.conf: Permission denied
Generating host.conf.
eval: cannot open /etc/nsswitch.conf: No such file or directory
ln: /dev/log: Operation not permitted
eval: cannot create /var/run/syslogd.sockets: Permission denied
Starting syslogd.
syslogd: child pid 97899 exited with return code 1
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
ldconfig: mkstemp(/var/run/ld-elf.so.hints.EO1FRT): Permission denied
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout
ldconfig: /var/run/ld.so.hints.QnuzP1cZaF: Permission denied
Starting local daemons:.
Updating motd ... /etc/motd is not writable, update failed.
/etc/rc: WARNING: Setting entropy source to blocking mode.
====================================================
Type a full screenful of random junk to unblock
it and remember to finish with. This will
timeout in 300 seconds, but waiting for
the timeout without typing junk may make the
entropy source deliver predictable output.Just hit
for fast+insecure startup.
====================================================
kern.random.sys.seeded: 1
lkajdflkjadsflkjsdfalk; voiusfady 098125 09okjcv lkhq234ou 8g09fuzohj adjf
You don't exist, go away!
You don't exist, go away!
You don't exist, go away!
sendmail_submit: /etc/mail/aliases.db not present, generating
Permission denied (real uid not trusted)
asendmail_clientmqueue: /etc/mail/aliases.db not present, generating
Permission denied (real uid not trusted)
dStarting cron.
cron: can't open or create /var/run/cron.pid: Permission denied
Local package initialization:.
/etc/rc.d/msgs: cannot create /var/msgs/bounds: Permission denied
l
Sun Sep 11 21:05:35 UTC 2005
$
For the most part, this looks exactly like a normal startup. A few things to note
- /etc/fstab – if you see this, you didn’t do as man jail said. You need to create an empty /etc/fstab file.
- adjkerntz – Not sure about this. I know you should comment out the /etc/crontab entry for adjkerntz within your jail environment.
- sendmail – My host environment does not use sendmail (i.e. I have “NO_SENDMAIL= true” in /etc/make.conf). I think this will go away once I install Postfix within the jail.
- net.inet.tcp.always_keepalive – I have no idea why this occurs
Starting and stopping the jail automagically
I found two interesting tools for starting and stopping the jails:
sysutils/jailer
and sysutils/jailutils.
sysutils/jailer is installed in the jail environment. sysutils/jailutils
should be installed in the host environment.
Using those two tools, I created this start/stop script:
#!/bin/shcase "$1" in
start)
jail /usr/jail/192.168.0.155 osw.example.org 192.168.0.155 /usr/local/sbin/jailer > /dev/null && echo -n ' jail 66.154.97.254'
;;
stop)
/usr/local/sbin/jails | /usr/bin/xargs /usr/local/sbin/killjail > /dev/null && echo -n ' jail'
;;
*)
echo "Usage: `basename $0` {start|stop}" >&2
y;;
esacexit 0
This is a very limited script. It doesn’t check that a jail is already running
before starting it. That would be a nice addition. If you want to add it, I
look forward to your patch.
In addition, you might want to add this to the host environment’s
/etc/sysctl.conf
jail.set_hostname_allowed=0
Under 5.*, this variable has a slightly different name.
There is a missing parenthesis in:
I did not specify the portmap_enable="NO" directive as that is default setting (see /etc/defaults/rc.conf.
Regarding the sendmail error in the jail startup screendump, shouldn’t setting sendmail_enable="NONE" in the jail’s /etc/rc.conf stop Sendmail from loading at all like a non-jailed system?
As far as detecting if the jail is already running, the man page provides a hint into finding the jail’s PID.
Quotheth the man page:
The /proc/pid/status file contains, as its last field, the hostname of
the jail in which the process runs, or “-” to indicate that the process
is not running within a jail. The ps(1) command also shows a `J’ flag
for processes in a jail.
Mind you, I have never setup a jail before 🙂
questionlp wrote:
> Regarding the sendmail error in the jail startup screendump,
> shouldn’t setting sendmail_enable="NONE" in the jail’s
> /etc/rc.conf stop Sendmail from loading at all like a
> non-jailed system?
I don’t know… what did you get when you tried it?
Note: I was intending to run postfix in that jail.
—
The Man Behind The Curtain
We’ve been running jail successfully since FreeBSD-4.8 or even earlier.
The jail was "getting up" when ppp session started up and was killed when ppp went down (the IP was dynamically changing – so there was no other way) – all magic done by 2 simple scripts (called from /etc/ppp/ppp.linkup and ppp.linkdown) – one to determine an IP of the tun0 interface, check if the jail was running and if not – create jail startup script and launch it, and second to kill all jail’s processes using `ps axw|grep J|awk ‘{print $1}’` as seems there was either no jailer in ports at that time or I knew nothing about it.
Running several jails at once requires a little more sophisticated approach – you need to check if the process belongs to some definite jail by checking entries in /proc for it.
Sendmail and bind run perfectly inside jail (other things like pop3/httpd do well as well 😉 ).
There is no need to explicitely bind sshd to certain IP’s on the host system – the jailed sshd takes over the TCP port – same thing about any application using TCP (so you may get connected to a host system if the jail is not running or has nothing listening on that port – causing some confusion 😉 ). Not sure what was happening with UDP – seems such careless approach didn’t work.
about "adjkerntz – Not sure about this." and other similiar things – all of that is in /etc/rc or called from it. I guess there’s a need to finally create a separate set of startup scripts for the jail, the scripts may not try doing weird things like playing with kernel variables, video modes and other things that belong to the host system.
Also – each jail has its own loopback (127.0.0.1) that prooved to have nothing to do with the host system’s one, so don’t expect you’ll get connected to a host system from the jail using 127.0.0.1 – add some alias IP to a host system’s lo0, like 192.168.0.1 or anything else.
Summarizing all the experience of running (you surely googled for the name?) all the *beep* in a not so friendly Internet environment in a jail I must admit that it is a pretty robust solution, allowing pretty fast migration from one host system to another (just pack and unpack directory with the jailed environment), though not very straightforward when you need it to do strange things.
> Also – each jail has its own loopback (127.0.0.1) that prooved to have
> nothing to do with the host system’s one, so don’t expect you’ll get
> connected to a host system from the jail using 127.0.0.1 – add some
> alias IP to a host system’s lo0, like 192.168.0.1 or anything else.
I’m not sure If I understand you right, but each jail having it’s OWN 127.0.0.1/loopback is a good thing (TM).
Hypothetically, why would I add 192.168.0.1 to my HOST’s lo0? and not my GUEST’s?
Okay… one more typo found in the start/stop script:
echo "Usage: asename $0 {start|stop}" >&2
Not sure what that character between Usage and asename is but that is how it shows up in View Source under Firefox.
Shame all I get when trying to use jailer is
# jail /usr/jails/192.168.1.1 jail-test 192.168.1.1 /usr/local/sbin/jailer
jailer: couldn’t determine if running in a jail.
where as
# jail /usr/jails/192.168.1.1 jail-test 192.168.1.1 /bin/sh /etc/rc
works fine..
Grez..
Grez wrote:
> Shame all I get when trying to use jailer is
>
> # jail /usr/jails/192.168.1.1 jail-test 192.168.1.1
> /usr/local/sbin/jailer
> jailer: couldn’t determine if running in a jail.
>
> where as
>
> # jail /usr/jails/192.168.1.1 jail-test 192.168.1.1 /bin/sh
> /etc/rc
>
> works fine..
I would check everything and try again. It definitely works.
—
The Man Behind The Curtain
Fixed. Thanks.
—
The Man Behind The Curtain
questionlp wrote:
> There is a missing parenthesis in:
>
> I did not specify the portmap_enable="NO" directive as that is
> default setting (see /etc/defaults/rc.conf.
Fixed. Thank you.
—
The Man Behind The Curtain