Debugging MAAS 2.x Ephemeral images

MAAS 2.x relies on Ephemeral images during commissioning of nodes. Basically, an Ephemeral image consists of a kernel, a RAM disk and a squashfs file-system that is booted over the network (PXE) and relies on cloud-init to perform discovery of a node’s hardware (e.g. number of CPUs, RAM, disk, etc.)

There are times that, for some reason, the commissioning process fails and you need to perform some troubleshooting. Typically, the node boots over PXE but cloud-init fails and you are left on the login screen with an non-configured host (e.g. hostname is ‘ubuntu”). But Ephemeral images don’t allow anyone to log in interactively. The solution consists of injecting some backdoor into the Ephemeral image. Such backdoor could be enabling some password for the root user, for example. Next, I will explain how to do this.

Ephemeral images are downloaded from the Internet by the MAAS region controller and synchronized to MAAS rack controllers. These files are kept on disk under:

https://images.maas.io/ephemeral-v3/daily/

Inside this directory, there is a subdirectory named after the Ubuntu release code name (e.g. Xenial):

https://images.maas.io/ephemeral-v3/daily/xenial/amd64/20171011/

Under this, another subdirectory named after the CPU architecture (e.g. AMD64):

https://images.maas.io/ephemeral-v3/daily/xenial/amd64/

And under this, another subdirectory named with some timestamp:

https://images.maas.io/ephemeral-v3/daily/xenial/amd64/20171011/

If you browse this location, you will find something like this:

[DIR]  ga-16.04/                12-Oct-2017 01:57.      -    
[DIR]  hwe-16.04-edge/          12-Oct-2017 01:57       -    
[DIR]  hwe-16.04/               12-Oct-2017 01:57.      -    
[   ]  squashfs                 12-Oct-2017 01:57       156M     
[TXT]  squashfs.manifest        12-Oct-2017 01:57       13K

The squashfs filesystem is shared among different types of kernels/ramdisk combinations (GA which stands for General Availability, HWE or HWE Edge). As mentioned before, these files are downloaded and kept updated in MAAS rack controllers under:

/var/lib/maas/boot-resources/snapshot-20171020-091808/ubuntu/amd64/hwe-16.04-edge/xenial/daily

On-disk layout is different from the Web layout, as each kernel/ramdisk combination has its own subdirectory together with the squashfs filesystem. But let’s no diverge. To introduce a backdoor, such as a password for the root user, let’s do the following:

# cd /var/lib/maas/boot-resources/snapshot-20171020-091808/ubuntu/amd64/hwe-16.04-edge/xenial/daily
# unsquashfs squashfs
# openssl passwd -1 ubuntu
$1$lqVUYmVl$6QatT6qYPVpFo8nbEO4Ve1
# cp -r squashfs-root/etc/passwd squashfs-root/etc/passwd~
# sed 's,^root:x:0:0:root:/root:/bin/bash$,root:$1$lqVUYmVl$6QatT6qYPVpFo8nbEO4Ve1:0:0:root:/root:/bin/bash,g' > squashfs-root/etc/passwd < squashfs-root/etc/passwd~
# cp -r squashfs squashfs.dist
# mksquashfs squashfs-root squashfs -xattrs -comp xz -noappend
# chown maas:maas squashfs

Now that the squashfs image has been unpacked, patched and re-packed, one can try commissioning the node again. If it fails, one can log in interactively as user root and password ubuntu.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s