IPSec transport mode with X.509 certificates


This article describes how to create a secure network-level transport between two hosts. All traffic sent between both hosts will be encrypted automatically as in enters the TCP/IP stack at the network (IP) level by using IPSec Encapsulating Security Payload (ESP) protocol.

For more details about IPSec, read IPSec pilot between glass and teapot.

Security Policy Database (SPD) configuration

The Security Policy Database (SPD) defines which IP traffic flows are to be affected by IPSec. Any traffic protected by IPSec can be protected by two different security mechanisms:

  • Confidentiality.

    Is achieved by using IPSec Encapsulating Security Payload (ESP) protocol.

    The IPSec ESP protocol cyphers the contents of the payload to be transported over IP. ESP, optionally, offers authentication and integrity, but it is considered weak since it does only affect the payload but not the header of the encapsulating IP datagram.

  • Authentication and Integrity.

    Is achieved by using IPSec Authentication Header (AH) protocol.

    The IPSec AH protocol protects that payload and any unmutable field of the encapsulating IP header.

By configuring the SPD, it is possible to use:

  • IPSec Transport Mode to protect all the traffic sent between two hosts.
  • IPSec Tunnel Mode to protect all the traffic sent between to networks.
For the machine ipsec-a:

Create the file /etc/racoon/setkey.sh with the following lines:

#!/sbin/setkey -f spdflush ; spdadd any -P out ipsec esp/transport//require ; spdadd any -P in ipsec esp/transport//require ;

This file should be marked executable and will be executed before launching racoon in order to populate the SPD with the proper entries.

For the machine ipsec-b:

Create the file /etc/racoon/setkey.sh with the following lines:

#!/sbin/setkey -f spdflush ; spdadd any -P out ipsec esp/transport//require ; spdadd any -P in ipsec esp/transport//require ;

This file should be marked executable and will be executed before launching racoon in order to populate the SPD with the proper entries.

Racoon configuration

racoon is a user-space daemon in charge of negotiating and establishing the Security Associations (SA) between two peer.

When the kernel sees an IP datagram, affected by a SPD rule, for which there is no SA yet established, the kernel will invoke racoon in order to negotiate and set it up with the corresponding peer defined in the SPD.

The peers can authenticate using some of the following:

  • Pre-Shared Keys (PSK)

    Both peers mutually agree on a shared secret, which is manually configured by the administrator and stored in the file /etc/racoon/psk.txt.

  • RSA Signature

    Each peer has an associated private key and public key X.509 certificate. Authentication takes place by exchanging certificates between peers and validating them, while RSA is used for authentication.


    Kerberos is used for authentication of both peers.

In out scenario, RSA Signature using X.509 public key certificates will be used for authentication between the peers so, in first place, we need to generate private keys and their corresponding certificates for each peer. The steps used to generate the certificates are described in Setting up Certificate Authority (CA) using OpenSSL.

The configuration for both peers is identical, so we will use the same racoon configuration file. However, the private key and public key certificate for each peer is different, so we should take this into consideration.

The peer private key must be installed into /etc/racoon/certs/key.pem, the peer signed public key certificate into /etc/racooon/certs/cert.pem and the CA public key certificate into /etc/racoon/certs/cacert.pem.

This is the /etc/racoon/racoon.conf configuration file:

path include "/etc/racoon"; path pre_shared_key "/etc/racoon/psk.txt"; path certificate "/etc/racoon/certs"; remote anonymous { # Some IPSec implementations have been found to # be vulnerable when used in aggressive exchange # mode exchange_mode main ; # Allow for the extension described in RFC 2407 # called Domain of Interpretation which allows # negotiation of the traditional 32-bit sequence # numbers or extended 64-bit sequence numbers doi ipsec_doi ; # Local identifier is taken from the Subject field # of the X.509 certificate (Distinguised Name) my_identifier asn1dn ; # Remote identifier is taken from the Subject field # of the X.509 certificate presented by the remote # peer (Distinguised Name) peers_identifier asn1dn ; # Checks that the oeer identity that appears in the # ID payload matches the identity specified in the # peers_identifier option verify_identifier on ; # Specifies the path to the certificate and private # key files, encoded in PEM, relative to the # "path certificate" option specified above certificate_type x509 "cert.pem" "key.pem" ; # Specifies the path to the CA certificate file, # encoded in PEM, relative to the "path certificate" # option specified above ca_type x509 "cacert.pem" ; # Configures the size of the nonce in bytes, which # must be no less than 8 and no more than 256 nonce_size 16 ; # Lifetime the Phase 1 SA proposal lifetime time 24 hour ; proposal { # Encryption algorithm for phase 1 encryption_algorithm 3des ; # Hash algorithm for phase 1 hash_algorithm sha1 ; # RSA Signature authentication authentication_method rsasig ; # Diffie-Hellman group for phase 1 dh_group 2 ; } } sainfo anonymous { # Diffie-Hellman group for phase 2 pfs_group 2; # Lifetime for the SA lifetime time 12 hour ; # Encryption algorithms to be used in the SA encryption_algorithm 3des, blowfish, des, rijndael ; # Authentication algorithms to be used in the SA authentication_algorithm hmac_sha1, hmac_md5 ; # Use deflate compression (IPComp) compression_algorithm deflate ; }

Starting peers

For every peer, we need to launch racoon. For testing purpouses, we will launch racoon in foreground, so all messages are dumped to the screen:

/usr/sbin/racoon -F

racoon will dump the following messages to the console:

INFO: @(#)ipsec-tools 0.5 (http://ipsec-tools.sourceforge.net)
INFO: @(#)This product linked OpenSSL 0.9.7f 22 Mar 2005 
INFO:[500] used as isakmp port (fd=7)
INFO:[500] used for NAT-T
INFO:[500] used as isakmp port (fd=8)
INFO:[500] used for NAT-T
INFO: ::1[500] used as isakmp port (fd=9)
INFO: fe80::20c:29ff:fea1:d55c%eth0[500] used as isakmp port (fd=10)

Next, we need to initialize the SPD:


Testing connectivity

To trigger the SA establishment we can ping the other host. The kernel will apply the SPD policy and will ask racoon to negotiate and set up the proper SA between both peers. racoon should dump something like this to the console:

INFO: IPsec-SA request for queued due to no phase1 found.
INFO: initiate new phase 1 negotiation:[500][500]
INFO: begin Identity Protection mode.
INFO: received Vendor ID: DPD
WARNING: unable to get certificate CRL(3) at depth:0 SubjectName:
         /C=ES/ST=Madrid/O=Software AG/OU=IT/CN=ipsec-a
WARNING: unable to get certificate CRL(3) at depth:1 SubjectName:
         /C=ES/ST=Madrid/L=Madrid/O=Software AG/OU=IT/CN=ca-server
INFO: ISAKMP-SA established[500]-[500] 
INFO: initiate new phase 2 negotiation:[0][0]
INFO: IPsec-SA established: ESP/Transport> 
INFO: IPsec-SA established: ESP/Transport> 

IPSec pilot between glass and teapot

I have established a secured ESP-only IPSec link between glass and teapot using Fedora Linux Core 2 native network interface configuration. I have opted for manually keyed Security Associations since there are a few glitches when using racoon‘s ISAKMP implementation and KAME 2.6 kernels IPSec implementation that I’m describing right now:

  1. Userspace and on-demand SA establishment interaction problems.

    When using ISAKMP on KAME-based IPSec implementation kernels (2.6 kernels or newer), the IPSec Security Association (SA) is established on demand (for tunnel and transport-mode links): the kernel delegates the SA establishment process to a userspace daemon that implenents the ISAKMP protocol (for KAME-based IPSec stacks, this daemon is called racoon).

    When any of the IPSec peers sends a packet, if no IPSec SA has been yet established, the ISAKMP negotiation starts. The problem is that the kernel does not queue the packet that triggered the ISAKMP SA negotiation until the SA has been established and then transmits it using IPSec. Instead, the kernel simply returns -EAGAIN to the userspace process that triggered the negotiation and thus will receive an error message. Once the ISAKMP SA has been established, any packet sent by any userspace process will succeed.

    This can be easily checked by pinging the other IPSec peer when no SA has been yet established. ping will return a “resource temporarily unavailable” error until the ISAKMP SA negotiation has finished. Once the SA has been established, IP traffic will flow normally between both peers.

  2. ISAKMP interaction problems with IPv6.

    IPv6 behaves in a very different manner with respect to link-layer address resolution. While IPv4 uses subnet broadcast traffic to guess the link-layer (MAC address for Ethernet and TokenRing networks, DLCI for FrameRelay, etc) address of a peer node (using ARP), IPv6 uses well-defined methods based on multicast (see Neighbor solicitation and Neighbor advertising ICMPv6 messages) and ICMPv6 instead of relying on broadcasts.

    This seems to cause problems when deploying racoon-based automatic SA establishment. By snooping at the traffic generated between two IPv6 peers trying to set a Security Association, and seeing no progress at all, I have reached the following conclusions:

    1. When node A tries to send a packet to node B when no SA between A and B has been established, the kernel triggers the ISAKMP protocol to set up an IPSec SA between both nodes. The kernel informs the userspace racoon daemon and thus the ISAKMP daemon triggers Phase 1 of the ISAKMP protocol.

      For A to start the Phase 1 ISAKMP SA negotiation, A needs to know the link-layer address of host B. Thus, A sends an ICMPv6 Neighbor solicitation message. Since this is a multicast message, it is not affected by any IPSec policy (AFAIK, only unicast traffic can be the target of IPSec policies).

    2. Host B receives the Neighbor solicitation message and tries to respond to A using a unicast IPv6 packet. Since the Neighbor discovery packet is targeted at a unicast IPv6 address, it’s affected by the IPSec policy and since no SA between A and B is present, host B triggers Phase 1 of the ISAKMP protocol.

      So, node B sends the ISAKMP Phase 1 initiator to node A, but since node A is already in its own ISAKMP Phase 1 and waiting for a response from node B, both nodes enter a deadlock and the ISAKMP SA negotiation never ends. After a while, racoon‘s SA timers expire and no SA establishment succeeds. Since no SA between host A and B can be set up, no IPSec traffic can flow between both nodes. This is a serious problem if a policy enforces mandatory IPSec traffic flow between both nodes..

      NOTE: This can be easily fixed if IPSec is not used for the ICMPv6 protocol. For example, when using setkey to manually alter the SPD database, you can issue:

        # spdadd   ipv6-icmp -P out none;
        # spdadd   ipv6-icmp -P in  none;

For all these reasons, I have decided to implement manual keyed IPSec SA between glass and teapot to avoid the above problems.

A few concepts:

  • SA (Security Association).

    Agreement between two nodes on a series of parameters and standards used for every IPSec packet exchange. The SA includes the cypher algorithm (DES, 3DES, AES, NULL if no encryption is used), the hash algorithm (HMAC-MD5-96 or HMAC-SHA1-96) if AH protocol authentication is being used, secret keys, SPIs, etc.

    NOTE: AH protocol is not required to provide minimal authentication support. ESP itself is able to provide authentication on the payload (that is, the daya from upper layer protocols, like TCP or UDP, but not the packet IP header).

  • SPD (Security Policy Database).

    Defines the IPSec policies that are going to be applied. A policy mainly dictates whether ESP, AH or both are to be used between two nodes and if its use is mandatory or optional. Optionally, the policy can protect any traffic, or just specific protocols like TCP or UDP.

  • ESP (Encapsulated Security Payload).

    Basically, the IPSec protocol (50) used to provide for confidentiality (optional if the NULL algorithm is used) and limited authentication (no authentication of the IP header is provided).

  • AH (Authentication Header).

    IPSec protocol (51) that provides strong authentication of both the IP header (except for mutable fields like the TTL) and it’s payload.

In Fedora Linux Core 2, there exists special interfaces (not phyiscal ones) to manage SA associations and modify the SPD properly. Those interfaces, usually with a name in the form ipsec, where n is an interface ordinal, are configured with a text file located at /etc/sysconfig/network-scripts. The file defines the source address (the IPv4 or IPv6 address from which IPSec packets will be sent) and the IPSec peer address (destination), the interface kind (always IPSEC) and how the IPSec keys are to be managed: manually or automatically (using pre-shared keys, X.509 certificates or Kerberos V via GSSAPI). In this pilot scenario, I’m going to use a manually keyed link.

Manually keyed connections require generating random keys of a predefined length (128 or 192 bits, for example) which must be manually set up at both IPSec peers. Although this is the simplest method of setting up an IPSec SA between to nodes, it lacks scalability, but this is not a concern for small networks like mine.

The random keys can be generated with the following command:

  # dd if=/dev/random bs=1 count=24 | xxd -ps
  24+0 records in
  24+0 records out

Where * 8 is equal to the number of bits of the key. If 24 is used, a 192 bits key is used.

A single key can be used for all incoming and outgoing traffic, or two independent keys can be used: one for incoming traffic and another one for outgoing traffic. Also, different keys for the ESP and AH protocol can be used.

The ifcfg-ipsec0 interface configuration file for teapot is as follows:

<pre # This is ifcfg-ipsec0 for teapot

SRC specifies the interface IPv4 address of the source (teapot’s IPv4 address). DST specifies the peer’s IPv4 address (glass’ IPv4 address). TYPE must always be set to IPSEC to state this is not a true physical interface by itself. KEY_ESP_OUT and KEY_ESP_IN define the keys used in the ESP packets. KEY_ESP_OUT is used to encrypt all outgoing (from teapot to glass) traffic, while KEY_ESP_IN is used to decrypt all incoming traffic. KEY_AH_OUT and KEY_AH_IN would set the keys used for the AH protocol (AH is optional, and can be left out if ESP authentication is enough).

SPI_ESP_OUT and SPI_ESP_IN specify the Security Parameter Index (SPI) for outgoing ESP traffic and incoming ESP traffic, respectively. Additionally, if AH is to be used, SPI_AH_OUT and SPI_AH_IN would set up SPI’s for outgoing and incoming AH protocol traffic. The SPI is a randomly chosen unique number used to identify each direction of either ESP or AH protocol.

ONBOOT is set to YES so that the IPSec virtual interface is set up at boot, effectively creating the proper SA entries and modiying the SPD policy database to “require” the use of either ESP, AH or both.

SECURITY NOTE: The ifcfg-ipsec0 must be properly secured from non root access since it contains hardcoded IPSec keys. Tus, the file should be owned by root and its permissions bits set to 0400. This, however, won’t prevent a hacker from gaining access to the file by using any kind of media forensics tools or low-level disk access software.

  # This is ifcfg-ipsec0 for glass

This file is generated from the contents of the teapot file by reversing the direction of all the fields: IN becomes OUT, OUT becomes IN, SRC becomes DST and DST becomes SRC. This seems pretty logical.