This article was written by Greg Panula.

is just my notes from setting up ipsec between two test boxes; phoenix and cardinal.

See also this ONLamp article.

The test uses both AH and ESP; packet and payload encryption.  It is a test of
transport mode, i.e. direct communications between two hosts.

In order to use IPSEC, you must first add it to your kernel.  You’ll need the
following options:

options  IPSEC           #IP security
options  IPSEC_ESP       #IP security (crypto; define w/ IPSEC)
options  IPSEC_DEBUG     #debug for IP security

For instructions on how to create a new kernel, refer to the Configuring the FreeBSD Kernel
section in the FreeBSD handbook.  Pay
special attention to the section on Building and Installing
a Custom Kernel

Transport mode

Useful references:

page for setkey

The initial test is between phoenix ( and cardinal (  This
test will use both AH and ESP; packet and payload encryption.  The test is
using only transport mode (i.e. it is not creating a tunnel).  Phoenix and Cardinal
reside on different network segments, there is a router between  them but no
firewall, nat or packet filtering. 

Phoenix setup

This is the AH protocol setup:

setkey -c
add ah 15700 -A hmac-md5 "1234567890123456";
add ah 24500 -A hmac-md5 "1234567890123456";

This sets up the AH protocol using hmac-md5 as the encryption algorithm. 
hmac-md5 requires a 128bit key, which means it must be 16 characters (bytes)

Breakdown of the first line:

  • source 
  • destination
  • AH protocol
  • 15700 SPI (unique ID#)
  • -A hmac-md5 authenication algorithm to use
  • "1234567890123456"  secretkey 

SPI tells the kernel which encryption rule and algorithm to use on  traffic tagged
with that SPI

This is the ESP protocol setup:

setkey -c
add esp 15701 -E blowfish-cbc "12345";
add esp 24501 -E blowfish-cbc "12345";

This sets up the ESP protocol using the blowfish-cbc encryption algortihm.  
blowfish-cbc allows for key lengths of 40-448 bits (5 to 56 bytes).

Cardinal setup

This is the AH protocol setup:

setkey -c
add ah 15700 -A hmac-md5 "1234567890123456";
add ah 24500 -A hmac-md5 "1234567890123456";

This is the ESP protocol setup:

setkey -c
add esp 15701 -E blowfish-cbc "12345";
add esp 24501 -E blowfish-cbc "12345";

Using ESP

Now we will configure both machines to use AH and ESP.  We will use the level of
default.  Which means the kernel consults the sysctl variable esp_trans_deflev
to determine whether to encrypt or not.  This allows us to remotely setup one
end (phoenix in this case) without losing connectivity.  The default value of esp_trans_deflev
is 1.  After looking at /sys/netinet6/ipsec.h and then doing a quick tcpdump
of the traffic between phoenix and cardinal, I have selected a value of 1, means
use encryption if available.

This is the relevent section from ipsec.h:

/* Security protocol level */  
#define IPSEC_LEVEL_DEFAULT    0  /* reference to system default */
#define IPSEC_LEVEL_USE        1  /* use SA if present. */
#define IPSEC_LEVEL_REQUIRE    2  /* require SA. */
#define IPSEC_LEVEL_UNIQUE     3  /* unique SA. */


setkey -c  
spdadd any -P out ipsec


setkey -c
spdadd any -P out ipsec

Note: the above spdadd line is one line and it encrypts all traffic (the `any’ before
the -P).  Check the setkey man page for info on how to encrypt certain
packets, e.g. rsh.

I’m only encrypting outbound traffic. And the format for the spdadd line is: spdadd
<src> <dst> <upperspec> <policy>

<upperspec> is upper layer protocol; tcp, udp or any are the options

<policy> is one of the following:
-P direction discard
-P direction none
-P direction ipsec protocol/mode/src-dst/level

What’s it look like?

A tcpdump of the traffic between phoenix and cardinal shows this:

cardinal.foo.bar > phoenix.foo.bar: AH(spi=24500,seq=0x2ac): ESP(spi=24501,seq=0x59e)
14:15:30.924053 phoenix.foo.bar > cardinal.foo.bar: AH(spi=15700,seq=0x38):
ESP(spi=15701,seq=0x198) (DF) 
14:15:31.002687 cardinal.foo.bar > phoenix.foo.bar: AH(spi=24500,seq=0x2ad):
ESP(spi=24501,seq=0x59f) (DF) 
14:15:31.007360 phoenix.foo.bar > cardinal.foo.bar: AH(spi=15700,seq=0x39):
ESP(spi=15701,seq=0x199) (DF) 
14:15:31.098242 cardinal.foo.bar > phoenix.foo.bar: AH(spi=24500,seq=0x2ae):
ESP(spi=24501,seq=0x5a0) (DF) 
14:15:32.826557 cardinal.foo.bar > phoenix.foo.bar: AH(spi=24500,seq=0x2af):
ESP(spi=24501,seq=0x5a1) (DF) 
14:15:32.832951 phoenix.foo.bar > cardinal.foo.bar: AH(spi=15700,seq=0x3a):
ESP(spi=15701,seq=0x19a) (DF) 

Setting the keys at boot time

One thing learned afterwards was the kernel loses the keys after a reboot.
  There is a mechanism built into /etc/rc.conf which set the keys for
you at boot time.  Note: do not modify /etc/defaults/rc.conf, make the
changes to /etc/rc.conf instead.

$ grep -i IPSEC /etc/defaults/rc.conf
ipsec_enable="NO"      # Set to YES to run setkey on
ipsec_file="/etc/ipsec.conf"   # Name of config file for setkey

If you put your setkey commands into /etc/ipsec.conf, and set ipsec_enable
to true, your keys will be set for you at system startup.  My thanks to Michael C.
Cambria for pointing this out to me.

17 thoughts on “IPsec”

  1. (posted by Dan Langille)

    First off, thanks! I now have IPSEC up and running between my two freeBSD
    hosts. However, there were a few things unclear that I wanted to touch
    upon. First off, the section which talks about sysctl under "Using ESP"
    through me way off. I still don’t see how this applies to getting IPSEC
    running. 2nd, the thing that killed 4 hours of my time :), you may want
    to make it VERY clear that the identification after "ah/es" in the AH/ESP
    setup needs to be the same on each host. I for some reason overlooked
    that which caused it to fail. However, after looking over it the 400th
    time, I was able to find what the issue was. I would have been nice to
    have that stated (for those of this who attempt to get IPSEC working after
    a VERY long day) hehe, anyways, thanks so much.. I’ve used your site many
    times and am very thankful for the information you provide.

  2. Hi! I have tried running the command

    # setkey -c
    add ah 1000 -m transport -A hmac-md5 "1234567890123456" ;

    and I always get the error

    line 1: Not supported at [1234567890123456]
    parse failed, line 1.

    Could I get some help??



    1. Maybe try removing the "-m transport" switch? -A hmac-md5 is just for authentication of the packets, so transport or tunnel mode isn’t really into play yet.

    2. have u included ipsec options in kernal file??

      previously, i am also got same problem
      when i enable ipsec options in kernal file
      options IPSEC
      options IPSEC_ESP
      options IPSEC_DEBUG

      and recompile ur kernal with IPSEC options …


  3. I have learned a little bit more and put another article/paper as part of some training. I still haven’t played with ipsec between different OSes, guess that makes me a bit of a freebsd bigot. 🙂

    Anyways, the follow-up article covers setting up a vpn between two freebsd boxes using automatic key exchange. Uses pre-shared secret for inital handshake, haven’t played much with X.509 certs, yet. It also covers doing nat across the vpn tunnel. Network Diagrams and Config examples are included.

    The article is available at:
    <A HREF="http://www.sans.org/infosecFAQ/firewall/IPSec_VPN.htm">http://www.sans.org/infosecFAQ/firewall/IPSec_VPN.htm

    Any questions or problems, let me know. I might be able to help.


  4. 1) In our FreeBSD machines, kernel is compiled with
    IPSEC options (IPSEC, IPSEC_ESP, IPSEC_DEBUG) . While trying ping6
    using the configured
    policy (using: ping6 -P <policyname> <destaddr>), the message
    "unable to set IPSEC policy" is coming. Any soulution for this?


    consider the following setup (All are FreeBSD machines):


    Policies (AH, Tunnelmode, hmac-md)are configured successfully in
    R1 and R3.
    When Ping6 is initiated from H1 to H2, packets coming
    from R1 do not contain AH header. But there is a reply
    from H2 without AH header. How to make R1 to send
    IP packets with Security headers.

    Is there any options to be included in kernal??

    1. Hi All,

      I also compiled and configured my kernel as mentioned in this document.
      I am trying to ping the destination machine, machine A is able to send neighbor solicitation message to m/c B, but machine B is sending some ESP packet to m/c A. I guess m/c B should send neighbor advertisement message to m/c A.

      Any help in this will be highly appreciated.


  5. Lars: Would you like to post your notes below as comments to the article?
    At http://www.freebsddiary.org/ipsec.php, click on "Post" just right the
    right of the article title and just below the date.


    On 9 Apr 2002 at 10:14, Lars Eggert wrote:

    > Hi,
    > please note that your IPsec how-to at
    > http://www.freebsddiary.org/ipsec.php or
    > http://rr.sans.org/firewall/IPSec_VPN.php instructs people to set up
    > IPIP tunnels in parallel to IPsec tunnel mode SAs.
    > This is NOT required. In fact, with this approach you are setting up two
    > tunnels between a node pair (one secure, one insecure). It "works"
    > because the kernel will hijack packets forwarded over the insecure IP
    > tunnel and push them over the secure IPsec SA. This depends on a
    > specific interaction of side effects in the kernel and has all kinds of
    > interesting failure modes.
    > It also confuses people into thinking that IPIP tunnels (gif interfaces)
    > and IPsec tunnel mode are related, or even dependent on one another,
    > when in reality they are completely separate concepts.
    > Please see the KAME newsletters (http://www.kame.net/newsletter/) for
    > correct configuration of IPSec tunnel mode.
    > Lars

  6. I have ipsec and nat working seperately, but I can’t get them to work at the same. I have:

    network 1 — Gateway –Internet –Gateway —– Network2

    I want all traffic from network 1 to nat to a single address when going to network2. any pointers or howto’s available?

    1. As far as I have read this doesn’t seem to be possible.
      I could think of a little trick:

      Network1 — NAT-PC — Gateway — Internet — Gateway — Network2

      Unfortunately, you will need another PC for this.


Leave a Comment

Scroll to Top