How PKI-based tokens from Keystone are authenticated

This article tries to explain how tokens generated by Keystone (using the PKI token format, not UUID) can be authenticated by clients (e.g. cinder, neutron, nova, etc.)

The relevant fragment from /etc/keystone/keystone.conf that specifies the PKI material used to sign Keystone tokens (the signing key, the signing certificate and its corresponding CA certificate, together with key size and key expiration period) usually looks like this (default values are used next):

[signing]
token_format = PKI
certfile = /etc/keystone/ssl/certs/signing_cert.pem
keyfile = /etc/keystone/ssl/private/signing_key.pem
ca_certs = /etc/keystone/ssl/certs/ca.pem
cert_subject = /C=US/ST=Unset/L=Unset/O=Unset/CN=www.example.com
key_size = 2048
valid_days = 3650

The Keystone client middleware — implemented in the keystone client.middleware.auth_token Python module — verifies the signature of a given Keystone token (data is in IAW CMS syntax). The actual method from this module is cms_verify. This method relies on its counterpart cms_verify defined in keystoneclient.common.cms and requires the actual data, the signing certificate and corresponding CA certificate.

The token’s data, signing certificate and its corresponding CA certificate are stored on local disk, inside a directory specified by the signing_dir option in the keystone_authtoken section. By default, this option is set to None. When None or absent, a temporary directory is created, as one can see in the verify_signing_dir method:

def verify_signing_dir(self):
    if os.path.exists(self.signing_dirname):
        if not os.access(self.signing_dirname, os.W_OK):
            raise ConfigurationError(
                'unable to access signing_dir %s' % self.signing_dirname)
        uid = os.getuid()
        if os.stat(self.signing_dirname).st_uid != uid:
            self.LOG.warning(
                'signing_dir is not owned by %s', uid)
        current_mode = stat.S_IMODE(os.stat(self.signing_dirname).st_mode)
        if current_mode != stat.S_IRWXU:
            self.LOG.warning(
                'signing_dir mode is %s instead of %s',
                oct(current_mode), oct(stat.S_IRWXU))
    else:
        os.makedirs(self.signing_dirname, stat.S_IRWXU)

When debug is True for any particular OpenStack service, one can see the value of the signing_dir option during startup in the logs:

2015-04-15 19:03:25.069 9449 DEBUG glance.common.config [-] keystone_authtoken.signing_dir = None log_opt_values /usr/lib/python2.6/site-packages/oslo/config/cfg.py:1953

The signing certificate and its corresponding CA certificate are retrieved from Keystone via an HTTP request, and stored on local disk. The methods that implement this in keystone client.middleware.auth_token look like this:

def _fetch_cert_file(self, cert_file_name, cert_type):
    path = '/v2.0/certificates/' + cert_type
    response = self._http_request('GET', path)
    if response.status_code != 200:
        raise exceptions.CertificateConfigError(response.text)
    self._atomic_write_to_signing_dir(cert_file_name, response.text)

def fetch_signing_cert(self):
    self._fetch_cert_file(self.signing_cert_file_name, 'signing')

def fetch_ca_cert(self):
    self._fetch_cert_file(self.signing_ca_file_name, 'ca')

Which translates to HTTP requests to Keystone like this:

2015-04-15 19:03:34.704 9462 DEBUG urllib3.connectionpool [-] "GET /v2.0/certificates/signing HTTP/1.1" 200 4251 _make_request /usr/lib/python2.6/site-packages/urllib3/connectionpool.py:295
2015-04-15 19:03:34.727 9462 DEBUG urllib3.connectionpool [-] "GET /v2.0/certificates/ca HTTP/1.1" 200 1277 _make_request /usr/lib/python2.6/site-packages/urllib3/connectionpool.py:295

As said before, in order to verify the Keystone token, the cms_verify method uses the signing certificate and corresponding CA certificates (as stored on local disk) plus the token data, and passes them to an external openssl process for verification:

def cms_verify(self, data):
    """Verifies the signature of the provided data's IAW CMS syntax.

    If either of the certificate files are missing, fetch them and
    retry.
    """
    while True:
        try:
            output = cms.cms_verify(data, self.signing_cert_file_name,
                                    self.signing_ca_file_name)
        except exceptions.CertificateConfigError as err:
            if self.cert_file_missing(err.output,
                                      self.signing_cert_file_name):
                self.fetch_signing_cert()
                continue
            if self.cert_file_missing(err.output,
                                      self.signing_ca_file_name):
                self.fetch_ca_cert()
                continue
            self.LOG.error('CMS Verify output: %s', err.output)
            raise
...

This translates to having the Keystone middleware spawning a process to run an openssl command to validate the input (the Keystone token). Something like:

openssl cms -verify -certfile /tmp/keystone-signing-OFShms/signing_cert.pem -CAfile /tmp/keystone-signing-OFShms/cacert.pem -inform PEM -nosmimecap -nodetach -nocerts -noattr << EOF
-----BEGIN CMS-----
MIIBxgYJKoZIhvcNAQcCoIIBtzCCAbMCAQExCTAHBgUrDgMCGjAeBgkqhkiG9w0B
BwGgEQQPeyJyZXZva2VkIjogW119MYIBgTCCAX0CAQEwXDBXMQswCQYDVQQGEwJV
UzEOMAwGA1UECAwFVW5zZXQxDjAMBgNVBAcMBVVuc2V0MQ4wDAYDVQQKDAVVbnNl
dDEYMBYGA1UEAwwPd3d3LmV4YW1wbGUuY29tAgEBMAcGBSsOAwIaMA0GCSqGSIb3
DQEBAQUABIIBABzCPXw9Kv49gArUWpAOWPsK8WRRnt6WS9gMaACvkllQs8vHEN11
nLBFGmO/dSTQdyXR/gQU4TuohsJfnYdh9rr/lrC3sVp1pCO0TH/GKmf4Lp1axrQO
c/gZym7qCpFKDNv8mAAHIbGFWvBa8H8J+sos/jC/RQYDbX++7TgPTCZdCbLlzglh
jKZko07P86o3k14Hq6o7VGpMGu9EjOziM6uOg391yylCVbqRazwoSszKm29s/LHH
dyvEc+RM9iRaNNTiP5Sa/bU3Oo25Ke6cleTcTqIdBaw+H5C1XakCkhpw3f8z0GkY
h0CAN2plwwqkT8xPYavBLjccOz6Hl3MrjSU=
-----END CMS-----
EOF

One has to pay attention to the purposes of the signing certificate. If its purposes are wrong, tokens generated by Keystone won’t be validated by Keystone clients (middleware). This is reflected in the logs with an error message that typically looks like this:

2015-04-15 18:52:13.027 29533 WARNING keystoneclient.middleware.auth_token [-] Verify error: Command 'openssl' returned non-zero exit status 4
2015-04-15 18:52:13.027 29533 DEBUG keystoneclient.middleware.auth_token [-] Token validation failure. _validate_user_token /usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py:836
2015-04-15 18:52:13.027 29533 TRACE keystoneclient.middleware.auth_token Traceback (most recent call last):
2015-04-15 18:52:13.027 29533 TRACE keystoneclient.middleware.auth_token File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 823, in _validate_user_token
2015-04-15 18:52:13.027 29533 TRACE keystoneclient.middleware.auth_token verified = self.verify_signed_token(user_token)
2015-04-15 18:52:13.027 29533 TRACE keystoneclient.middleware.auth_token File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 1258, in verify_signed_token
2015-04-15 18:52:13.027 29533 TRACE keystoneclient.middleware.auth_token if self.is_signed_token_revoked(signed_text):
2015-04-15 18:52:13.027 29533 TRACE keystoneclient.middleware.auth_token File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 1216, in is_signed_token_revoked
2015-04-15 18:52:13.027 29533 TRACE keystoneclient.middleware.auth_token revocation_list = self.token_revocation_list
2015-04-15 18:52:13.027 29533 TRACE keystoneclient.middleware.auth_token File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 1312, in token_revocation_list
2015-04-15 18:52:13.027 29533 TRACE keystoneclient.middleware.auth_token self.token_revocation_list = self.fetch_revocation_list()
2015-04-15 18:52:13.027 29533 TRACE keystoneclient.middleware.auth_token File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 1358, in fetch_revocation_list
2015-04-15 18:52:13.027 29533 TRACE keystoneclient.middleware.auth_token return self.cms_verify(data['signed'])
2015-04-15 18:52:13.027 29533 TRACE keystoneclient.middleware.auth_token File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 1239, in cms_verify
2015-04-15 18:52:13.027 29533 TRACE keystoneclient.middleware.auth_token self.signing_ca_file_name)
2015-04-15 18:52:13.027 29533 TRACE keystoneclient.middleware.auth_token File "/usr/lib/python2.6/site-packages/keystoneclient/common/cms.py", line 148, in cms_verify
2015-04-15 18:52:13.027 29533 TRACE keystoneclient.middleware.auth_token raise e
2015-04-15 18:52:13.027 29533 TRACE keystoneclient.middleware.auth_token CalledProcessError: Command 'openssl' returned non-zero exit status 4
2015-04-15 18:52:13.027 29533 TRACE keystoneclient.middleware.auth_token
2015-04-15 18:52:13.028 29533 DEBUG keystoneclient.middleware.auth_token [-] Marking token as unauthorized in cache _cache_store_invalid /usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py:1154
2015-04-15 18:52:13.028 29533 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2015-04-15 18:52:13.029 29533 INFO keystoneclient.middleware.auth_token [-] Invalid user token - deferring reject downstream
Advertisements

IPSec transport mode with X.509 certificates

Scenario

This article describes how to create a secure network-level transport between two hosts. All traffic sent between both hosts will be encrypted automatically as in enters the TCP/IP stack at the network (IP) level by using IPSec Encapsulating Security Payload (ESP) protocol.

For more details about IPSec, read IPSec pilot between glass and teapot.

Security Policy Database (SPD) configuration

The Security Policy Database (SPD) defines which IP traffic flows are to be affected by IPSec. Any traffic protected by IPSec can be protected by two different security mechanisms:

  • Confidentiality.

    Is achieved by using IPSec Encapsulating Security Payload (ESP) protocol.

    The IPSec ESP protocol cyphers the contents of the payload to be transported over IP. ESP, optionally, offers authentication and integrity, but it is considered weak since it does only affect the payload but not the header of the encapsulating IP datagram.

  • Authentication and Integrity.

    Is achieved by using IPSec Authentication Header (AH) protocol.

    The IPSec AH protocol protects that payload and any unmutable field of the encapsulating IP header.

By configuring the SPD, it is possible to use:

  • IPSec Transport Mode to protect all the traffic sent between two hosts.
  • IPSec Tunnel Mode to protect all the traffic sent between to networks.
For the machine ipsec-a:

Create the file /etc/racoon/setkey.sh with the following lines:

#!/sbin/setkey -f spdflush ; spdadd 192.168.0.40 192.168.0.41 any -P out ipsec esp/transport//require ; spdadd 192.168.0.41 192.168.0.40 any -P in ipsec esp/transport//require ;

This file should be marked executable and will be executed before launching racoon in order to populate the SPD with the proper entries.

For the machine ipsec-b:

Create the file /etc/racoon/setkey.sh with the following lines:

#!/sbin/setkey -f spdflush ; spdadd 192.168.0.41 192.168.0.40 any -P out ipsec esp/transport//require ; spdadd 192.168.0.40 192.168.0.41 any -P in ipsec esp/transport//require ;

This file should be marked executable and will be executed before launching racoon in order to populate the SPD with the proper entries.

Racoon configuration

racoon is a user-space daemon in charge of negotiating and establishing the Security Associations (SA) between two peer.

When the kernel sees an IP datagram, affected by a SPD rule, for which there is no SA yet established, the kernel will invoke racoon in order to negotiate and set it up with the corresponding peer defined in the SPD.

The peers can authenticate using some of the following:

  • Pre-Shared Keys (PSK)

    Both peers mutually agree on a shared secret, which is manually configured by the administrator and stored in the file /etc/racoon/psk.txt.

  • RSA Signature

    Each peer has an associated private key and public key X.509 certificate. Authentication takes place by exchanging certificates between peers and validating them, while RSA is used for authentication.

  • GSSAPI

    Kerberos is used for authentication of both peers.

In out scenario, RSA Signature using X.509 public key certificates will be used for authentication between the peers so, in first place, we need to generate private keys and their corresponding certificates for each peer. The steps used to generate the certificates are described in Setting up Certificate Authority (CA) using OpenSSL.

The configuration for both peers is identical, so we will use the same racoon configuration file. However, the private key and public key certificate for each peer is different, so we should take this into consideration.

The peer private key must be installed into /etc/racoon/certs/key.pem, the peer signed public key certificate into /etc/racooon/certs/cert.pem and the CA public key certificate into /etc/racoon/certs/cacert.pem.

This is the /etc/racoon/racoon.conf configuration file:

path include "/etc/racoon"; path pre_shared_key "/etc/racoon/psk.txt"; path certificate "/etc/racoon/certs"; remote anonymous { # Some IPSec implementations have been found to # be vulnerable when used in aggressive exchange # mode exchange_mode main ; # Allow for the extension described in RFC 2407 # called Domain of Interpretation which allows # negotiation of the traditional 32-bit sequence # numbers or extended 64-bit sequence numbers doi ipsec_doi ; # Local identifier is taken from the Subject field # of the X.509 certificate (Distinguised Name) my_identifier asn1dn ; # Remote identifier is taken from the Subject field # of the X.509 certificate presented by the remote # peer (Distinguised Name) peers_identifier asn1dn ; # Checks that the oeer identity that appears in the # ID payload matches the identity specified in the # peers_identifier option verify_identifier on ; # Specifies the path to the certificate and private # key files, encoded in PEM, relative to the # "path certificate" option specified above certificate_type x509 "cert.pem" "key.pem" ; # Specifies the path to the CA certificate file, # encoded in PEM, relative to the "path certificate" # option specified above ca_type x509 "cacert.pem" ; # Configures the size of the nonce in bytes, which # must be no less than 8 and no more than 256 nonce_size 16 ; # Lifetime the Phase 1 SA proposal lifetime time 24 hour ; proposal { # Encryption algorithm for phase 1 encryption_algorithm 3des ; # Hash algorithm for phase 1 hash_algorithm sha1 ; # RSA Signature authentication authentication_method rsasig ; # Diffie-Hellman group for phase 1 dh_group 2 ; } } sainfo anonymous { # Diffie-Hellman group for phase 2 pfs_group 2; # Lifetime for the SA lifetime time 12 hour ; # Encryption algorithms to be used in the SA encryption_algorithm 3des, blowfish, des, rijndael ; # Authentication algorithms to be used in the SA authentication_algorithm hmac_sha1, hmac_md5 ; # Use deflate compression (IPComp) compression_algorithm deflate ; }

Starting peers

For every peer, we need to launch racoon. For testing purpouses, we will launch racoon in foreground, so all messages are dumped to the screen:

/usr/sbin/racoon -F

racoon will dump the following messages to the console:

INFO: @(#)ipsec-tools 0.5 (http://ipsec-tools.sourceforge.net)
INFO: @(#)This product linked OpenSSL 0.9.7f 22 Mar 2005 
      (http://www.openssl.org/)
INFO: 127.0.0.1[500] used as isakmp port (fd=7)
INFO: 127.0.0.1[500] used for NAT-T
INFO: 192.168.0.41[500] used as isakmp port (fd=8)
INFO: 192.168.0.41[500] used for NAT-T
INFO: ::1[500] used as isakmp port (fd=9)
INFO: fe80::20c:29ff:fea1:d55c%eth0[500] used as isakmp port (fd=10)

Next, we need to initialize the SPD:

/etc/racoon/setkey.sh

Testing connectivity

To trigger the SA establishment we can ping the other host. The kernel will apply the SPD policy and will ask racoon to negotiate and set up the proper SA between both peers. racoon should dump something like this to the console:

INFO: IPsec-SA request for 192.168.0.40 queued due to no phase1 found.
INFO: initiate new phase 1 negotiation: 
      192.168.0.41[500]192.168.0.40[500]
INFO: begin Identity Protection mode.
INFO: received Vendor ID: DPD
WARNING: unable to get certificate CRL(3) at depth:0 SubjectName:
         /C=ES/ST=Madrid/O=Software AG/OU=IT/CN=ipsec-a
WARNING: unable to get certificate CRL(3) at depth:1 SubjectName:
         /C=ES/ST=Madrid/L=Madrid/O=Software AG/OU=IT/CN=ca-server
INFO: ISAKMP-SA established 192.168.0.41[500]-192.168.0.40[500] 
      spi:2698c81446191f6c:9b9127e3b6956065
INFO: initiate new phase 2 negotiation: 192.168.0.41[0]192.168.0.40[0]
INFO: IPsec-SA established: ESP/Transport 192.168.0.40->192.168.0.41 
      spi=78608282(0x4af779a)
INFO: IPsec-SA established: ESP/Transport 192.168.0.41->192.168.0.40 
      spi=118550227(0x710eed3)

Setting up Certificate Authority (CA) using OpenSSL

OpenSSL command-line tools offer a lot of functionality. OpenSSL can generate private keys and their corresponding public key certificate requests, can sign those certificate requests, publish certificate revocation lists (CRLs), convert between several encoding formats like DER, PEM, PKCS#12, etc.

In this article I will describe how to use OpenSSL to set up a Certificate Authority (CA), how to generate private keys, generate certificate requests and sign them, using OpenSSL and the command-line on a Fedora Core 4 Linux system.

Configuring OpenSSL

Edit /etc/pki/tls/openssl.cnf and make sure the [ CA_default ] section looks like this:

[ CA_default ]

dir             = /etc/pki/CA           # Where everything is kept
certs           = $dir/certs            # Where the issued certs are kept
crl_dir         = $dir/crl              # Where the issued crl are kept
database        = $dir/index.txt        # database index file.
unique_subject  = no                    # Set to 'no' to allow creation of
                                        # several ctificates with same subject.
new_certs_dir   = $dir/newcerts         # default place for new certs.

certificate     = $dir/cacert.pem       # The CA certificate
serial          = $dir/serial           # The current serial number
#crlnumber      = $dir/crlnumber        # the current crl number must be
                                        # commented out to leave a V1 CRL
crl             = $dir/crl.pem          # The current CRL
private_key     = $dir/private/cakey.pem# The private key
RANDFILE        = $dir/private/.rand    # private random number file

x509_extensions = usr_cert              # The extentions to add to the cert

# Comment out the following two lines for the "traditional"
# (and highly broken) format.
name_opt        = ca_default            # Subject Name options
cert_opt        = ca_default            # Certificate field options

# Extension copying option: use with caution.
# copy_extensions = copy

# Extensions to add to a CRL. Note: Netscape communicator chokes on V2 CRLs
# so this is commented out by default to leave a V1 CRL.
# crlnumber must also be commented out to leave a V1 CRL.
# crl_extensions        = crl_ext

default_days    = 365                   # how long to certify for
default_crl_days= 30                    # how long before next CRL
default_md      = sha1                  # which md to use.
preserve        = no                    # keep passed DN ordering

# A few difference way of specifying how similar the request should look
# For type CA, the listed attributes must be the same, and the optional
# and supplied fields are just that.
policy          = policy_match

Creating the Certificate Authority (CA)

This step will create the CA private key and a self-signed certificate (the CA certificate).

The CA certificate will be stored in /etc/pki/CA/cacert.pem, while the CA private key will be stored in /etc/pki/CA/private/cakey.pem:

openssl req -new -x509 -days 365 -newkey rsa:1024 
  -keyout /etc/pki/CA/private/cakey.pem 
  -out /etc/pki/CA/cacert.pem
chmod 600 /etc/pki/CA/private/cakey.pem

A copy of every signed certificate will be stored into /etc/pki/CA/newcerts, with a name matching the certificate serial number plus the .pem extension:

mkdir /etc/pki/CA/newcerts

The /etc/pki/CA/index.txt file holds a log of every signed certificate:

touch /etc/pki/CA/index.txt

The /etc/pki/CA/serial file holds the next available X.509 serial number:

echo 01 > /etc/pki/CA/serial

Generating a certificate request and its corresponding private key

The following command will generate a random, 1024-bit private RSA key and its corresponding public key will be wrapped into a PEM-encoded certificate. This certificate is still unsigned and will be submitted later to the CA for signing:

openssl req -new -days 365 -newkey rsa:1024 
  -keyout /etc/pki/CA/sslkey.pem
  -out /etc/pki/CA/sslcert.pem

The private key will get written to /etc/pki/CA/sslkey.pem while the public key, encoded inside an unsigned certificate, will get written to /etc/pki/CA/sslcert.pem.

The -nodes option can be used to avoid using a pass-phrase to protect the private key. This is optional, but some applications are unable to read the private key if it was protected by a pass-phrase, while others like FreeRADIUS can do so with no problems at all.

Signing a certificate

To sign a certificate stored in /etc/pki/CA/sslcert.pem, use the following command:

openssl ca -in /etc/pki/CA/sslcert.pem -out /etc/pki/CA/cert.pem

The resulting signed certificate will get outputted to /etc/pki/CA/cert.pem. Once the certificate has been signed, the unsigned certificate can be safely deleted.