How to run Docker inside a Nova/LXD container

I’ve been experimenting with deploying OpenStack using Nova/LXD (instead of Nova/KVM) for quite some time, using conjure-up as the deployment tool. It is simple, easy to set up and use and produces a usable OpenStack cluster.

However, I’ve been unable to run Docker inside a Nova instance (implemented as an LXD container) using an out-of-the-box installation deployed by conjure-up. The underlying reason is that the LXD container where nova-compute is hosted lacks some privileges. Also, inside this nova-compute container Nova/LXD spawns nested LXD containers, one for each Nova instance, which again lack some additional privileges required by Docker.

Short story, you can apply the docker LXD profile to both the nova-compute container and those nested LXD containers inside it where you want to run Docker, and Docker will run fine:

⟫ juju status nova-compute
⟫ juju status nova-compute
Model                         Controller                Cloud/Region         Version    SLA
conjure-openstack-novalx-1d1  conjure-up-localhost-718  localhost/localhost  2.2-beta4  unsupported

App                  Version  Status  Scale  Charm                Store       Rev  OS      Notes
lxd                  2.0.9    active      1  lxd                  jujucharms   10  ubuntu
neutron-openvswitch  10.0.0   active      1  neutron-openvswitch  jujucharms  240  ubuntu
nova-compute         15.0.2   active      1  nova-compute         jujucharms  266  ubuntu

Unit                      Workload  Agent  Machine  Public address  Ports  Message
nova-compute/0*           active    idle   4        10.0.8.61              Unit is ready
  lxd/0*                  active    idle            10.0.8.61              Unit is ready
  neutron-openvswitch/0*  active    idle            10.0.8.61              Unit is ready

Machine  State    DNS        Inst id        Series  AZ  Message
4        started  10.0.8.61  juju-59ffc3-4  xenial      Running
...

From the previous output, notice how the nova-compute/0 unit is running in machine #4, and that the underlying LXD container is named juju-59ffc3-4. Now, let’s see the LXD profiles used by this container:

⟫ lxc info juju-59ffc3-4 | grep Profiles
Profiles: default, juju-conjure-openstack-novalx-1d1

The docker LXD profile is missing from this container, and this will cause that any nested container trying to use Docker will fail. Entering the nova-compute/0 container, we see initially no nested containers. That is, since there are no Nova instances, there are no LXD containers. Remember that when using Nova/LXD, there is a 1:1 mapping between a Nova instance and an LXD container:

⟫ lxc exec juju-59ffc3-4 /bin/bash
root@juju-59ffc3-4:~# lxc list
+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+

Let’s spawn a Nova instance for testing:

⟫ juju ssh nova-cloud-controller/0
ubuntu@juju-59ffc3-13:~$ source novarc
ubuntu@juju-59ffc3-13:~$ openstack server create --flavor m1.small --image xenial-lxd --nic net-id=ubuntu-net test1

Now, if we take a look inside the nova-compute/0 container, we will see a nested container:

⟫ juju ssh nova-compute/0
ubuntu@juju-59ffc3-4:~$ sudo -i
root@juju-59ffc3-4:~# lxc list
+-------------------+---------+-------------------+------+------------+-----------+
|       NAME        |  STATE  |       IPV4        | IPV6 |    TYPE    | SNAPSHOTS |
+-------------------+---------+-------------------+------+------------+-----------+
| instance-00000001 | RUNNING | 10.101.0.9 (eth0) |      | PERSISTENT | 0         |
+-------------------+---------+-------------------+------+------------+-----------+
root@juju-59ffc3-4:~# lxc info instance-00000001 | grep Profiles
Profiles: instance-00000001

Here one can see that the nested container is using a profile named after the Nova instance. Let’s enter this nested container, install Docker and try to spawn a Docker container:

root@juju-59ffc3-4:~# lxc exec instance-00000001 /bin/bash
root@test1:~# apt-get update
...
root@test1:~# apt-get -y install docker.io
...
root@test1:~# docker run -it ubuntu /bin/bash
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
b6f892c0043b: Pull complete
55010f332b04: Pull complete
2955fb827c94: Pull complete
3deef3fcbd30: Pull complete
cf9722e506aa: Pull complete
Digest: sha256:382452f82a8bbd34443b2c727650af46aced0f94a44463c62a9848133ecb1aa8
Status: Downloaded newer image for ubuntu:latest
docker: Error response from daemon: containerd: container not started.

Here we can see that Docker was unable to spawn the Docker container.

First thing we are going to try is to add the docker LXD profile to the nested container, the one hosting our Nova instance:

⟫ juju ssh nova-compute/0
ubuntu@juju-59ffc3-4:~$ sudo -i
root@juju-59ffc3-4:~# lxc list
+-------------------+---------+-------------------+------+------------+-----------+
|       NAME        |  STATE  |       IPV4        | IPV6 |    TYPE    | SNAPSHOTS |
+-------------------+---------+-------------------+------+------------+-----------+
| instance-00000001 | RUNNING | 10.101.0.5 (eth0) |      | PERSISTENT | 0         |
+-------------------+---------+-------------------+------+------------+-----------+
root@juju-59ffc3-4:~# lxc info instance-00000001 | grep Profiles
Profiles: instance-00000001
root@juju-59ffc3-4:~# lxc profile apply instance-00000001 instance-00000001,docker
Profile instance-00000001,docker applied to instance-00000001

Now, let’s try again to run a Docker container:

root@juju-59ffc3-4:~# lxc exec instance-00000001 /bin/bash
root@test1:~# docker run -it ubuntu /bin/bash
root@7fc441a9b0a5:/# uname -r
4.10.0-21-generic
root@7fc441a9b0a5:/#

But this, besides being a manual process, it is not elegant. There’s another solution which requires no operation intervention. It consists of a Python code patch to the Nova/LXD driver that allows selectively adding additional LXD profiles to Nova containers:

$ juju ssh nova-compute/0
ubuntu@juju-59ffc3-4:~$ sudo -i
root@juju-59ffc3-4:~# patch -d/ -p0 << EOF
--- /usr/lib/python2.7/dist-packages/nova_lxd/nova/virt/lxd/config.py.orig      2017-06-07 19:41:47.685278274 +0000
+++ /usr/lib/python2.7/dist-packages/nova_lxd/nova/virt/lxd/config.py   2017-06-07 19:42:58.891624467 +0000
@@ -56,11 +56,17 @@
         instance_name = instance.name
         try:
 
+            # Profiles to be applied to the container
+            profiles = [str(instance.name)]
+            lxd_profiles = instance.flavor.extra_specs.get('lxd:profiles')
+            if lxd_profiles:
+                profiles += lxd_profiles.split(',')
+
             # Fetch the container configuration from the current nova
             # instance object
             container_config = {
                 'name': instance_name,
-                'profiles': [str(instance.name)],
+                'profiles': profiles,
                 'source': self.get_container_source(instance),
                 'devices': {}
             }
EOF
root@juju-59ffc3-4:~# service nova-compute restart

Now, let’s create a new flavor named docker with the extra spec to include the docker LXD profile to all instances that rely on this flavor:

⟫ juju ssh nova-cloud-controller/0
ubuntu@juju-59ffc3-13:~$ source novarc
ubuntu@juju-59ffc3-13:~$ openstack flavor create --disk 20 --vcpus 2 --ram 1024 docker
ubuntu@juju-59ffc3-13:~$ openstack flavor set --property lxd:profiles=docker docker
ubuntu@juju-59ffc3-13:~$ openstack server create --flavor docker --image xenial-lxd --nic net-id=ubuntu-net test2

Then, inside the nova-compute container:

⟫ juju ssh nova-compute/0
ubuntu@juju-59ffc3-4:~$ sudo -i
root@juju-59ffc3-4:~# lxc list
+-------------------+---------+--------------------------------+------+------------+-----------+
|       NAME        |  STATE  |              IPV4              | IPV6 |    TYPE    | SNAPSHOTS |
+-------------------+---------+--------------------------------+------+------------+-----------+
| instance-00000001 | RUNNING | 172.17.0.1 (docker0)           |      | PERSISTENT | 0         |
|                   |         | 10.101.0.9 (eth0)              |      |            |           |
+-------------------+---------+--------------------------------+------+------------+-----------+
| instance-00000003 | RUNNING | 10.101.0.8 (eth0)              |      | PERSISTENT | 0         |
+-------------------+---------+--------------------------------+------+------------+-----------+
root@juju-59ffc3-4:~# lxc info instance-00000003 | grep Profiles
Profiles: instance-00000003, docker
root@juju-59ffc3-4:~# lxc exec instance-00000003 /bin/bash
root@test2:~# apt-get update
...
root@test2:~# apt-get -y install docker.io
...
root@test2:~# docker run -it ubuntu /bin/bash
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
b6f892c0043b: Pull complete
55010f332b04: Pull complete
2955fb827c94: Pull complete
3deef3fcbd30: Pull complete
cf9722e506aa: Pull complete
Digest: sha256:382452f82a8bbd34443b2c727650af46aced0f94a44463c62a9848133ecb1aa8
Status: Downloaded newer image for ubuntu:latest
root@fd74cfa04876:/# uname -r
4.10.0-21-generic
root@fd74cfa04876:/#

So, that’s it. With this small patch, which enables support for the lxd:profiles extra spec, it is easier to allow Docker to run inside Nova instances hosted in LXD containers.

Advertisements

One thought on “How to run Docker inside a Nova/LXD container

  1. Thanks a lot. Really nice tutorial !!! By the way do you think we can use Docker image directly import to Nova-lxd ? I was trying various way on that. But it seems not work. Do you have any idea.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s