Xen is one of the coolest pieces of software I have ever used. It allows me to partition my box into manageable pieces, for increased security and increased resource utilization. I have been playing extensively with Xen for more than a year and have also written some posts about it.

NetBSD is a lean, mean, fast free, open source operating system and is nicely supported under Xen, has nice features like the PF packet filter and the pkgsrc ports-like collection and runs in nearly every single hardware architecture on earth. Because of this, I decided to run NetBSD 3.1 on Xen. NetBSD can run either as the privileged domain (called dom0) or as an unprivileged guest (called domU) domain. Since I was already running Linux under Xen as a domU, I am mostly interested in running NetBSD 3.1 as a domU guest on Xen. dom0 can be either Red Hat Enterprise Linux 5.0 or Fedora Core 6, but feel free to use any other Linux distribution as most of them are Xen-ready.

As far as I know, there are some restrictions between the Xen hypervisor + dom0 kernel and domU kernel:

• You cannot mix PAE-enabled and non-PAE kernels.

  For example, you cannot run a PAE-enabled dom0 kernel and/or PAE-enabled hypervisor and a non-PAE dom0/domU kernel.

  This is currently a problem since Fedora Core 6 and Red Hat Enterprise Linux 5.0 both ship with a PAE-enabled Xen hypervisor and Xen-enabled kernels, but NetBSD does not currently ship a PAE-compatible, Xen-enabled kernel.

• You cannot mix 64-bit and 32-bit kernels.

  You cannot run a 64-bit Xen hypervisor and 64-bit dom0 kernel and a 32-bit domU kernel.

Since both Fedora Core 6 and Red Hat Enterprise Linux 5.0 ship by default with a PAE-enabled (36-bit addressable memory space) Xen hypervisor and dom0 Xen-enabled Linux kernel, the first thing that I had to do in order to run NetBSD 3.1 as domU under Xen was to recompile the Linux kernel and the Xen hypervisor with PAE support completely disabled. This is described next.

Build Xen hypervisor and dom0 kernel without PAE

You can skip to the next section if you already have a non-PAE, working Xen installation.

The first thing I had to do is to downl the SRPM (source RPM) for the latest Linux kernel, for example kernel-2.6.19-1.2895.fc6.src.rpm, then install it by running:

# rpm -i kernel-2.6.19-1.2895.fc6.src.rpm

In file /usr/src/redhat/SPECS/kernel-2.6.spec replace the following:

%ifarch i686
%define buildpae 1
# we build always xen HV with pae
%define xen_flags verbose=y crash_debug=y pae=y
%endif

with:

%ifarch i686
%define buildpae 0
# we build always xen HV with pae
%define xen_flags verbose=y crash_debug=y
%endif

This will cause the Xen hypervisor to be built without PAE support. Additionally, no PAE-enabled extra kernels will be built. The Xen kernel, however, uses its specific configuration file that has to be changed in order to disable PAE support. To disable PAE support for the Xen kernel, I reconfigured the kernel with no PAE support by running:

# rpmbuild -bp /usr/src/redhat/SPECS/kernel-2.6.spec
# cd /usr/src/redhat/BUILD/kernel-2.6.19/linux-2.6.19.i386
# cp configs/kernel-2.6.19-i686-xen.config .config
# make menuconfig

Make sure PAE is disabled by navigating to Processor type and features, then High Memory Support is set to either off or 4GB (but not 64GB).

Next, I copied the updated configuration file back to /usr/src/redhat/SOURCES, where it belongs. Also, we need to insert # i386 at the beginning of the file so that the RPM build process can derive the exact processor architecture from the config file when building the RPMs:

# cat <(echo "# i386") .config > ../../../SOURCES/kernel-2.6.19-i686-xen.config

The processor architecture is supplied to make during the build process in the form of ARCH=i386.

Now, let’s build the RPMs:

# rpmbuild -ba --target i686 ../../../SPECS/kernel-2.6.spec

We need to specify i686 as the target architecture since Fedora and Red Hat don’t use i386 anymore for kernels themselves — i386 is now only used for some common RPMs like kernel-headers.

Once the RPMs have been built, check the files under /usr/src/redhat/RPMS/i686. At least there should be a file called kernel-xen-2.6.19-1.2895.i686.rpm. This RPM contains several files, but the ones that we are interested in are:

• /boot/config-2.6.19-1.2895xen

  Contains the kernel configuration. Make sure either CONFIG_X86_PAE is set to n or is undefined.

• /boot/vmlinuz-2.6.19-1.2895xen

  The Linux Xen-enabled kernel.

• /boot/xen.gz-2.6.19-1.2895

  The Xen hypervisor. In the most recent versions of Fedora Core and Red Hat Enterprise Linux, the Xen hypervisor and the Xen-enabled kernel are packaged in the same RPM. This is the right thing to do since both are tightly coupled.

Install the new Xen kernel and hypervisor:

# rpm -ivh --force /usr/src/redhat/RPMS/i686/kernel-xen-2.6.19-1.2895.i686.rpm

Reboot:

# reboot

I assume the system will boot correctly and into the new Xen hypervisor and Xen-enabled Linux kernel. You can check that by running:

# uname -a
Linux xen 2.6.19-1.2895xen #1 SMP Sat Feb 3 16:56:34 CET 2007 i686 i686 i386 GNU/Linux

The next step is installing NetBSD 3.1 as a domU. This is covered next.

Installing NetBSD 3.1

The first step is preparing the Xen’s domU configuration file and its corresponding storage backend. Xen can use file-backed storage for a domU or block-backed storage (i.e. a disk partition or logical volume). Typically, block-backed storage is faster than file-backed storage, so I set up a 10GiB logical volume for NetBSD:

# lvcreate -n netbsd xen -L 10G

I also used NetBSD’s Internet-based installation since it’s the easiest way to get a working NetBSD installation and the NetBSD community have built Xen-enabled NetBSD kernels:

• netbsd-INSTALL_XEN3_DOMU

  A Xen-based, domU kernel used to install NetBSD.

• netbsd-XEN3_DOMU

  A Xen-based, domU kernel used to run the installed system.

Both files can be downloaded from /pub/NetBSD/NetBSD-3.1/i386/binary/kernel. Download and uncompress both of them:

# wget ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-3.1/i386/binary/kernel/netbsd-*XEN3_DOMU.gz
# zcat netbsd-INSTALL_XEN3_DOMU.gz > /boot/netbsd-INSTALL_XEN3_DOMU
# zcat netbsd-XEN3_DOMU.gz > /boot/netbsd-XEN3_DOMU

If you are running SELinux, you will need to relabel these files properly or xm will be unable to load them into memory:

# chcon root system_u:object_r:boot_t /boot/netbsd*

Next, create the Xen configuration file for NetBSD. In my case, it looked like this:

# cat /etc/xen/auto/netbsd
kernel = "/boot/netbsd-INSTALL_XEN3_DOMU"
memory = 256
name = "netbsd"
vif = [ 'mac=00:16:3e:00:00:11, bridge=xenbr0' ]
disk = [ 'phy:/dev/xen/netbsd,hda,w' ]
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

Now, we will install NetBSD by starting the domain:

# xm create -c /etc/xen/auto/netbsd

This will start the new domain and will attach to its console. You can follow the Example Installation NetBSD document to assist you in installing NetBSD and also Xensource NetBSDdomU Wiki page.

Once the installer has finished, do not reboot. At the end of the installation process, you’ll be brought back to the main install screen. Select e: Utility menu, then a: Run /bin/sh, then type the following at the shell:

mount /dev/xbd0a /mnt
cp -pR /dev/rxbd* /mnt/dev
cp -pR /dev/xbd* /mnt/dev
halt -p

This will copy the required special device files and shut down the guest. Now, you will have to modify the domain config file in order to use the standard NetBSD domU kernel, /boot/netbsd-XEN3_DOMU. Edit /etc/xen/auto/netbsd and replace:

kernel = "/boot/netbsd-INSTALL_XEN3_DOMU"

with:

kernel = "/boot/netbsd-XEN3_DOMU"

And boot the domain again:

# xm create -c /etc/xen/auto/netbsd

During boot, you will see some errors like:


wsconscfg: /dev/ttyEcfg: Device not configured


This is due to the NetBSD guest only having access to one physical console. To kill those errors, edit /etc/ttys from within the NetBSD guest and turn off all terminals except "console", like:

console "/usr/libexec/getty Pc"         vt100   on  secure
ttyE0   "/usr/libexec/getty Pc"         vt220   off secure
ttyE1   "/usr/libexec/getty Pc"         vt220   off secure
ttyE2   "/usr/libexec/getty Pc"         vt220   off secure
ttyE3   "/usr/libexec/getty Pc"         vt220   off secure
...

Also, comment out all screens in /etc/wscons.conf:

#screen 0       -       vt100
#screen 1       -       vt100
#screen 2       -       vt100
#screen 3       -       vt100
#screen 4       -       -
#screen 4       80x25bf vt100

That’s all. Now we have a fully functional NetBSD 3.1 domU guest running on Xen 🙂

References

The information and instructions on this post are based on:

1. NetBSDdomU — How to install NetBSD as a domU on a Linux host.
2. Example Installation — NetBSD example installation.