Debugging MAAS 2.x Ephemeral images

MAAS 2.x relies on Ephemeral images during commissioning of nodes. Basically, an Ephemeral image consists of a kernel, a RAM disk and a squashfs file-system that is booted over the network (PXE) and relies on cloud-init to perform discovery of a node’s hardware (e.g. number of CPUs, RAM, disk, etc.)

There are times that, for some reason, the commissioning process fails and you need to perform some troubleshooting. Typically, the node boots over PXE but cloud-init fails and you are left on the login screen with an non-configured host (e.g. hostname is ‘ubuntu”). But Ephemeral images don’t allow anyone to log in interactively. The solution consists of injecting some backdoor into the Ephemeral image. Such backdoor could be enabling some password for the root user, for example. Next, I will explain how to do this.

Ephemeral images are downloaded from the Internet by the MAAS region controller and synchronized to MAAS rack controllers. These files are kept on disk under:

Inside this directory, there is a subdirectory named after the Ubuntu release code name (e.g. Xenial):

Under this, another subdirectory named after the CPU architecture (e.g. AMD64):

And under this, another subdirectory named with some timestamp:

If you browse this location, you will find something like this:

[DIR]  ga-16.04/                12-Oct-2017 01:57.      -    
[DIR]  hwe-16.04-edge/          12-Oct-2017 01:57       -    
[DIR]  hwe-16.04/               12-Oct-2017 01:57.      -    
[   ]  squashfs                 12-Oct-2017 01:57       156M     
[TXT]  squashfs.manifest        12-Oct-2017 01:57       13K

The squashfs filesystem is shared among different types of kernels/ramdisk combinations (GA which stands for General Availability, HWE or HWE Edge). As mentioned before, these files are downloaded and kept updated in MAAS rack controllers under:


On-disk layout is different from the Web layout, as each kernel/ramdisk combination has its own subdirectory together with the squashfs filesystem. But let’s no diverge. To introduce a backdoor, such as a password for the root user, let’s do the following:

# cd /var/lib/maas/boot-resources/snapshot-20171020-091808/ubuntu/amd64/hwe-16.04-edge/xenial/daily
# unsquashfs squashfs
# openssl passwd -1 ubuntu
# cp -r squashfs-root/etc/passwd squashfs-root/etc/passwd~
# sed 's,^root:x:0:0:root:/root:/bin/bash$,root:$1$lqVUYmVl$6QatT6qYPVpFo8nbEO4Ve1:0:0:root:/root:/bin/bash,g' > squashfs-root/etc/passwd < squashfs-root/etc/passwd~
# cp -r squashfs squashfs.dist
# mksquashfs squashfs-root squashfs -xattrs -comp xz -noappend
# chown maas:maas squashfs

Now that the squashfs image has been unpacked, patched and re-packed, one can try commissioning the node again. If it fails, one can log in interactively as user root and password ubuntu.

Juju and apt-cacher

I’ve been playing quite a lot lately with Juju and other related software projects, like conjure-up and LXD. They make so easy to spin up and down complex software stacks like OpenStack that you don’t even realize until your hosting provider start alerting you of high traffic consumption. And guess where most of this traffic usage comes from? From installing packages.

So I decided to save on bandwidth by using apt-cacher. It is straightforward and easy to set up and getting it running. In the end, if you follow the steps described in the previous link or this, you will end up with a Perl program listening on your machine in port 3142 that you can use as an Apt cache.

For Juju, one can use a YAML configuration file like this:

apt-http-proxy: http://localhost:3142
apt-https-proxy: http://localhost:3142

Then bootstrap Juju using the following command:

$ juju bootstrap --config config.yaml localhost lxd

For conjure-up is also very easy:

$ conjure-up \
    --apt-proxy http://localhost:3142 \
    --apt-https-proxy http://localhost:3142 \

OpenStack Newton and LXD


This post is about deploying a minimal OpenStack newton cluster atop LXD on a single machine. Most of what is mentioned here is based on OpenStack on LXD.


The rationale behind using LXD is simplicity and feasibility: it doesn’t require more than one x86_64 server with 8 CPU cores, 64GB of RAM and a SSD drive large enough to perform an all-in-one deployment of OpenStack Newton.

According to Canonical, “LXD is a pure-container hypervisor that runs unmodified Linux guest operating systems with VM-style operations at incredible speed and density.”. Instead of using pure virtual machines to run the different OpenStack components, LXD is used which allows for higher “machine” (container) density. In practice, an LXD container behaves pretty much like a virtual or baremetal machine.

For all purposes, I will be using Ubuntu 16.04.02 for this experiment on a 128GB machine with 12 CPU cores and 4x240GB SSD drives configured using software RAID0. For increased performance and efficiency ZFS is also used (dedicated partition separate from the base OS) as a backing store for LXD.


$ sudo add-apt-repository ppa:juju/devel
$ sudo add-apt-repository ppa:ubuntu-lxc/lxd-stable
$ sudo apt update
$ sudo apt install \
    juju lxd zfsutils-linux squid-deb-proxy \
    python-novaclient python-keystoneclient \
    python-glanceclient python-neutronclient \
    python-openstackclient curl
$ git clone

It is important to run all the following commands inside the openstack-on-lxd directory where the Git repository has been cloned locally.

LXD set up
$ sudo lxd init

The relevant part here is the network configuration. IPv6 is not properly supported by Juju so make sure to not enable. For IPv4 use the subnet and assign the IPv4 address for LXD itself. The DHCP range could be something like to

NOTE: Having LXD listen on the network is also an option for remotely managing LXD, but beware of security issues when exposing it over a public network. Using ZFS (or btrfs) should also increase performance and efficiency (e.g. copy-on-write shall save disk space by prevent duplicate bits from all the containers running the same base image).

Using an MTU of 9000 for container interfaces will likely increase performance:

$ lxc profile device set default eth0 mtu 9000

Next step is to spawn an LXC container for testing purposes:

$ lxc launch ubuntu-daily:xenial openstack
$ lxc exec openstack bash
# exit

An specific LXC profile named juju-default will be used when deploying OpenStack. In particular this profile allows for nesting LXD (required by nova-compute), allows running privileged containers, and preloads certain kernel modules required inside OpenStack containers.

$ lxc profile create juju-default 2>/dev/null || \
  echo "juju-default profile already exists"
$ cat lxd-profile.yaml | lxc profile edit juju-default
Bootstrap Juju controller
$ juju bootstrap --config config.yaml localhost lxd
Deploy OpenStack
$ juju deploy bundle-newton-novalxd.yaml
$ watch juju status

After Juju has finished deploying OpenStack, make sure there is a file named novarc in the current directory. This file is required to be sourced in order to use the OpenStack CLI:

$ source novarc
$ openstack catalog list
$ nova service-list
$ neutron agent-list
$ cinder service-list

Create Nova flavors:

$ openstack flavor create --public \
    --ram   512 --disk  1 --ephemeral  0 --vcpus 1 m1.tiny
$ openstack flavor create --public \
    --ram  1024 --disk 20 --ephemeral 40 --vcpus 1 m1.small
$ openstack flavor create --public \
    --ram  2048 --disk 40 --ephemeral 40 --vcpus 2 m1.medium
$ openstack flavor create --public \
    --ram  8192 --disk 40 --ephemeral 40 --vcpus 4 m1.large
$ openstack flavor create --public \
    --ram 16384 --disk 80 --ephemeral 40 --vcpus 8 m1.xlarge

Add the typical SSH key:

$ openstack keypair create --public-key ~/.ssh/ mykey

Create a Neutron external network and a virtual network for testing:

$ ./neutron-ext-net \
    -g -c \
    -f ext_net
$ ./neutron-tenant-net \
    -t admin -r provider-router \
    -N internal

CAVEAT: Nova/LXD does not support use of QCOW2 images in Glance. Instead one has to use RAW images. For example:

$ curl | \
  glance image-create --name xenial --disk-format raw --container-format bare


$ openstack server create \
    --image xenial --flavor m1.tiny --key-name mykey --wait \
    --nic net-id=$(neutron net-list | grep internal | awk '{ print $2 }') \

NOTE: For reasons I yet do not understand, one can’t use a flavor other than m1.tiny. Reason is that this flavor is the only one that does not request any ephemeral disk. As soon as ephemeral disk is requested, the LXD subsystem inside the nova-compute container will complain with the following error:

$ juju ssh nova-compute/0
$ sudo tail -f /var/log/nova/nova-compute.log
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nova/compute/", line 2078, in _build_resources
    yield resources
  File "/usr/lib/python2.7/dist-packages/nova/compute/", line 1920, in _build_and_run_instance
  File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/", line 317, in spawn
    self._add_ephemeral(block_device_info, lxd_config, instance)
  File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/", line 1069, in _add_ephemeral
    raise exception.NovaException(reason)
NovaException: Unsupport LXD storage detected. Supported storage drivers are zfs and btrfs.

If Cinder is available, create a test Cinder volume:

$ cinder create --name testvolume 10

Default user in WSL

The Windows Subsystem for Linux (WSL) defaults to running as the “root” user. In order to change that behavior, just create a Linux user.Let’s imagine this user is named “jdoe”. To have WSL start the session as “jdoe” instead of “root”, just run the following command from a “cmd.exe” window:

C:\Users\JohnDoe> lxrun /setdefaultuser jdoe

Take into account that running any running WSL will be killed inmmediately.

Create a local mirror of Ubuntu packages using apt-mirror

Sometimes, having a local mirror of Ubuntu packages can be useful. Not only this can save tons of network bandwidth when installing an Ubuntu system multiple times. An example of this are testing, development and QA environments that rely on virtual machines. When installing a new Ubuntu system, just point the installer to the local Ubuntu mirror and you’ll save time and reduce your WAN/Internet traffic considerably.

In order to create and keep a local mirror of Ubuntu, you can use apt-mirror which is available in the universe repository. And, for the record, this post is heavily based on another one — Ubuntu – Set Up A Local Mirror.

Ubuntu, as many other Linux distributions, retrieve packages for installation over HTTP. Therefore, the first thing to do is to install Apache, if not already installed. And, at the same time, let’s install apt-mirror too:

$ sudo apt-get install apache2 apt-mirror

Next step consists of configuring apt-mirror. The configuration is very similar to /etc/apt/sources.list. apt-mirror reads its configuration from /etc/apt/mirror.list. By default, it mirrors packages for the architecture on which it’s running, but you’ll likely want it to mirror packages for x86_64 and i386 systems. Also, beware of the size of the local mirror: mirroring all the repositories can consume quite a lot of disk space in the local system (30GB or even more). It’s a good idea to mirror those repositories that you need. Here’s an example of my /etc/apt/mirror.list:

############# config ##################
# set base_path    /var/spool/apt-mirror
# set mirror_path  $base_path/mirror
# set skel_path    $base_path/skel
# set var_path     $base_path/var
# set cleanscript $var_path/
# set defaultarch  
# set postmirror_script $var_path/
# set run_postmirror 0
set nthreads     20
set _tilde 0
############# end config ##############

deb-amd64 trusty main restricted
deb-amd64 trusty-security main restricted
deb-amd64 trusty-updates main restricted
deb-i386 trusty main restricted
deb-i386 trusty-security main restricted
deb-i386 trusty-updates main restricted


This configuration requests 20 download threads, and mirrors the main and restricted repositories for x86_64 and i386 systems exclusively.

To initiate the mirror process, just run:

$ sudo apt-mirror

This will spawn workers threads that will mirror the configured repositories into /var/spool/apt-mirror.

In order to serve this mirror via Apache, just create a symlink into the root Apache directory:

$ sudo ln -s /var/spool/apt-mirror/mirror/ /var/www/html/ubuntu

It might also be a good idea to remove or rename /var/www/html/index.html so that one can browse the repository using a Web browser.

And finally, you can configure cron to run apt-mirror periodically. For example, by adding the following line to your crontab:

@daily /usr/bin/apt-mirror

Self-signed certificates with OpenSSL

I’ve found that the easiest way to generate self-signed certificates in Debian derivatives, like Ubuntu, is by installing and using make-ssl-cert:

$ sudo apt-get install ssl-cert
$ make-ssl-cert /usr/share/ssl-cert/ssleay.cnf /path/to/cert-file.crt

This will invoke OpenSSL to generate a pair of RSA public and private keys. OpenSSL will ask for some information, like the Common Name for the certificate. When used to protect Web sites, the Common Name has to match the associated FQDN (fully-qualified domain name). For example,

More information can be found by reading the README.Debian.gz file from Apache2 documentation set:

$ zless /usr/share/doc/apache2/README.Debian.gz

Or online, by reading Apache and SSL, The Easy Way.

BTRFS and Ubuntu Lucid 10.04

Although Ubuntu Lucid 10.04 has native support for BTRFS, it does not like very much auto-mounting a BTRFS volume during start up. The problem seems to be that btrfsctl -a is not invoked during the boot process.

It takes some hacks to udev and initramfs to get this working. I found the solution in HowTO: Btrfs Root Installation:

This initramfs script will make sure the btrfsctl binary gets copied to the RAM disk:

$ cat /usr/share/initramfs-tools/hooks/btrfs
#!/bin/sh -e
# initramfs hook for btrfs

if [ "$1" = "prereqs" ]; then
    exit 0

. /usr/share/initramfs-tools/hook-functions

if [ -x "`which btrfsctl`" ]; then
    copy_exec "`which btrfsctl`" /sbin

I believe the following is not strictly necessary unless you plan on having a BTRFS-based root filesystem:

$ cat /usr/share/initramfs-tools/modules.d/btrfs
# initramfs modules for btrfs

This will load the BTRFS module while the system boots up, and calls btrfsctl -a to prepare the BTRFS volumes:

$ cat /usr/share/initramfs-tools/scripts/local-premount/btrfs
#!/bin/sh -e
# initramfs script for btrfs

if [ "$1" = "prereqs" ]; then
    exit 0

modprobe btrfs

if [ -x /sbin/btrfsctl ]; then
    /sbin/btrfsctl -a 2>/dev/null

Mark the scripts executable:

chmod +x /usr/share/initramfs-tools/scripts/local-premount/btrfs
chmod +x /usr/share/initramfs-tools/hooks/btrfs

Rebuild the initial RAM disks and GRUB environment:

update-initramfs -u -k all

That should do it.

vsftpd: anonymous FTP uploads

In order to have an anonymous-only FTP server that allows anonymous uploads, based on vsftpd running on Ubuntu 9.10, I applied the following configuration changes:

--- /etc/vsftpd.conf.orig	2010-01-13 02:00:46.287216196 +0100
+++ /etc/vsftpd.conf	2010-01-13 01:59:34.787215294 +0100
@@ -26,16 +26,18 @@
 # Uncomment this to enable any form of FTP write command.
 # Default umask for local users is 077. You may wish to change this to 022,
 # if your users expect that (022 is used by most other ftpd's)
 # Uncomment this to allow the anonymous FTP user to upload files. This only
 # has an effect if the above global write enable is activated. Also, you will
 # obviously need to create a directory writable by the FTP user.
 # Uncomment this if you want the anonymous FTP user to be able to create
 # new directories.


While trying to use Chromium on a Ubuntu 64-bit machine, I discovered I wasn’t able to browse to any web page. I always got the following error message:

This webpage is not available.

The webpage at might be temporarily down or it may have moved permanently to a new web address.

Here are some suggestions:
Reload this web page later.
More information on this error
Below is the original error message

Error 105 (net::ERR_NAME_NOT_RESOLVED): The server could not be found.

DNS name resolution was working properly, so it was something else. I searched for this error and most of the search results were about Chrome on Windows having problems with proxy or firewall configuration. But, who cares about Windows? So, after spending a little bit more, I found the following issue in the official Google Code web site.

In the end, it was just a matter of:

$ sudo apt-get install lib32nss-mdns

Why Chromium has a an explicit dependency on mDNS is something that still puzzles me out.

libvirt and bridged networking

libvirt and virt-manager are a blessing. They bring powerful, free, open source management to Xen- and KVM-based virtualization environments.

I’ve been using both for quite a while. Also, I’ve always prefered bridged networking support for my virtual machines over NAT. While NAT is non-disruptive and allows for isolation, I typically like to easily access services provided by my virtual machines, like SSH or NFSv4. Turns out that setting bridged networking support in libvirt is very easy, as long as bridged interface is detected by libvirt.

The simplest solution consists of creating a bridge interface that enslaves all the physical networks interfaces used to connect to the LAN or the Internet. For example, in Ubuntu, in order to enslave eth0 to a br0 bridge interface, while using DHCP for IPv4 address configuration, /etc/network/interfaces needs to look like this:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet manual

# The bridge
auto br0
iface br0 inet dhcp
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0
        bridge_maxwait 0

Next time, when creating a new virtual machine, it will be possible to use bridged networking in addition to NAT-based networking. There is one caveat, at least in Ubuntu: libvirt and virt-manager by default connect to qemu:///user instead of qemu:///system. This is neither good nor bad by itself. qemu:///user allows a non-privileged user to create and use virtual machines and the process of creating and destroying the virtual network interfaces used by the virtual machines is done within the context of the user running virt-manager. Due to lack of root privileges, virtual machines are limited to QEMU’s usermode networking support. In order to use advanced networking feautures like bridged networking, make sure you connect to qemu:///system instead. That is typically achieved by running virt-manager as root (which is not necessarily nice). I tried playing with udev and device ownership and permission masks but it all boils down to the inability of a non-privileged user to use brcrl to enslave network interfaces to a bridge.