Xen is one of the coolest pieces of software I have ever used. It allows me to partition my box into manageable pieces, for increased security and increased resource utilization. I have been playing extensively with Xen for more than a year and have also written some posts about it.
NetBSD is a lean, mean, fast free, open source operating system and is nicely supported under Xen, has nice features like the PF packet filter and the pkgsrc ports-like collection and runs in nearly every single hardware architecture on earth. Because of this, I decided to run NetBSD 3.1 on Xen. NetBSD can run either as the privileged domain (called dom0) or as an unprivileged guest (called domU) domain. Since I was already running Linux under Xen as a domU, I am mostly interested in running NetBSD 3.1 as a domU guest on Xen. dom0 can be either Red Hat Enterprise Linux 5.0 or Fedora Core 6, but feel free to use any other Linux distribution as most of them are Xen-ready.
As far as I know, there are some restrictions between the Xen hypervisor + dom0 kernel and domU kernel:
- You cannot mix PAE-enabled and non-PAE kernels.
For example, you cannot run a PAE-enabled dom0 kernel and/or PAE-enabled hypervisor and a non-PAE dom0/domU kernel.
This is currently a problem since Fedora Core 6 and Red Hat Enterprise Linux 5.0 both ship with a PAE-enabled Xen hypervisor and Xen-enabled kernels, but NetBSD does not currently ship a PAE-compatible, Xen-enabled kernel.
- You cannot mix 64-bit and 32-bit kernels.
You cannot run a 64-bit Xen hypervisor and 64-bit dom0 kernel and a 32-bit domU kernel.
Since both Fedora Core 6 and Red Hat Enterprise Linux 5.0 ship by default with a PAE-enabled (36-bit addressable memory space) Xen hypervisor and dom0 Xen-enabled Linux kernel, the first thing that I had to do in order to run NetBSD 3.1 as domU under Xen was to recompile the Linux kernel and the Xen hypervisor with PAE support completely disabled. This is described next.
Build Xen hypervisor and dom0 kernel without PAE
You can skip to the next section if you already have a non-PAE, working Xen installation.
The first thing I had to do is to downl the SRPM (source RPM) for the latest Linux kernel, for example kernel-2.6.19-1.2895.fc6.src.rpm, then install it by running:
# rpm -i kernel-2.6.19-1.2895.fc6.src.rpm
/usr/src/redhat/SPECS/kernel-2.6.spec replace the following:
%ifarch i686 %define buildpae 1 # we build always xen HV with pae %define xen_flags verbose=y crash_debug=y pae=y %endif
%ifarch i686 %define buildpae 0 # we build always xen HV with pae %define xen_flags verbose=y crash_debug=y %endif
This will cause the Xen hypervisor to be built without PAE support. Additionally, no PAE-enabled extra kernels will be built. The Xen kernel, however, uses its specific configuration file that has to be changed in order to disable PAE support. To disable PAE support for the Xen kernel, I reconfigured the kernel with no PAE support by running:
# rpmbuild -bp /usr/src/redhat/SPECS/kernel-2.6.spec # cd /usr/src/redhat/BUILD/kernel-2.6.19/linux-2.6.19.i386 # cp configs/kernel-2.6.19-i686-xen.config .config # make menuconfig
Make sure PAE is disabled by navigating to
Processor type and features, then
High Memory Support is set to either
4GB (but not
Next, I copied the updated configuration file back to
/usr/src/redhat/SOURCES, where it belongs. Also, we need to insert
# i386 at the beginning of the file so that the RPM build process can derive the exact processor architecture from the config file when building the RPMs:
# cat <(echo "# i386") .config > ../../../SOURCES/kernel-2.6.19-i686-xen.config
The processor architecture is supplied to
make during the build process in the form of
Now, let’s build the RPMs:
# rpmbuild -ba --target i686 ../../../SPECS/kernel-2.6.spec
We need to specify
i686 as the target architecture since Fedora and Red Hat don’t use
i386 anymore for kernels themselves —
i386 is now only used for some common RPMs like
Once the RPMs have been built, check the files under
/usr/src/redhat/RPMS/i686. At least there should be a file called
kernel-xen-2.6.19-1.2895.i686.rpm. This RPM contains several files, but the ones that we are interested in are:
Contains the kernel configuration. Make sure either
CONFIG_X86_PAEis set to
nor is undefined.
The Linux Xen-enabled kernel.
The Xen hypervisor. In the most recent versions of Fedora Core and Red Hat Enterprise Linux, the Xen hypervisor and the Xen-enabled kernel are packaged in the same RPM. This is the right thing to do since both are tightly coupled.
Install the new Xen kernel and hypervisor:
# rpm -ivh --force /usr/src/redhat/RPMS/i686/kernel-xen-2.6.19-1.2895.i686.rpm
I assume the system will boot correctly and into the new Xen hypervisor and Xen-enabled Linux kernel. You can check that by running:
# uname -a Linux xen 2.6.19-1.2895xen #1 SMP Sat Feb 3 16:56:34 CET 2007 i686 i686 i386 GNU/Linux
The next step is installing NetBSD 3.1 as a domU. This is covered next.
Installing NetBSD 3.1
The first step is preparing the Xen’s domU configuration file and its corresponding storage backend. Xen can use file-backed storage for a domU or block-backed storage (i.e. a disk partition or logical volume). Typically, block-backed storage is faster than file-backed storage, so I set up a 10GiB logical volume for NetBSD:
# lvcreate -n netbsd xen -L 10G
I also used NetBSD’s Internet-based installation since it’s the easiest way to get a working NetBSD installation and the NetBSD community have built Xen-enabled NetBSD kernels:
A Xen-based, domU kernel used to install NetBSD.
A Xen-based, domU kernel used to run the installed system.
Both files can be downloaded from /pub/NetBSD/NetBSD-3.1/i386/binary/kernel. Download and uncompress both of them:
# wget ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-3.1/i386/binary/kernel/netbsd-*XEN3_DOMU.gz # zcat netbsd-INSTALL_XEN3_DOMU.gz > /boot/netbsd-INSTALL_XEN3_DOMU # zcat netbsd-XEN3_DOMU.gz > /boot/netbsd-XEN3_DOMU
If you are running SELinux, you will need to relabel these files properly or
xm will be unable to load them into memory:
# chcon root system_u:object_r:boot_t /boot/netbsd*
Next, create the Xen configuration file for NetBSD. In my case, it looked like this:
# cat /etc/xen/auto/netbsd kernel = "/boot/netbsd-INSTALL_XEN3_DOMU" memory = 256 name = "netbsd" vif = [ 'mac=00:16:3e:00:00:11, bridge=xenbr0' ] disk = [ 'phy:/dev/xen/netbsd,hda,w' ] on_poweroff = 'destroy' on_reboot = 'restart' on_crash = 'restart'
Now, we will install NetBSD by starting the domain:
# xm create -c /etc/xen/auto/netbsd
Once the installer has finished, do not reboot. At the end of the installation process, you’ll be brought back to the main install screen. Select
e: Utility menu, then
a: Run /bin/sh, then type the following at the shell:
mount /dev/xbd0a /mnt cp -pR /dev/rxbd* /mnt/dev cp -pR /dev/xbd* /mnt/dev halt -p
This will copy the required special device files and shut down the guest. Now, you will have to modify the domain config file in order to use the standard NetBSD domU kernel,
/etc/xen/auto/netbsd and replace:
kernel = "/boot/netbsd-INSTALL_XEN3_DOMU"
kernel = "/boot/netbsd-XEN3_DOMU"
And boot the domain again:
# xm create -c /etc/xen/auto/netbsd
During boot, you will see some errors like:
wsconscfg: /dev/ttyEcfg: Device not configured
This is due to the NetBSD guest only having access to one physical console. To kill those errors, edit
/etc/ttys from within the NetBSD guest and turn off all terminals except
console "/usr/libexec/getty Pc" vt100 on secure ttyE0 "/usr/libexec/getty Pc" vt220 off secure ttyE1 "/usr/libexec/getty Pc" vt220 off secure ttyE2 "/usr/libexec/getty Pc" vt220 off secure ttyE3 "/usr/libexec/getty Pc" vt220 off secure ...
Also, comment out all screens in
#screen 0 - vt100 #screen 1 - vt100 #screen 2 - vt100 #screen 3 - vt100 #screen 4 - - #screen 4 80x25bf vt100
That’s all. Now we have a fully functional NetBSD 3.1 domU guest running on Xen 🙂
The information and instructions on this post are based on: