libvirt and bridged networking

07/06/2009 / Felipe Alfaro Solana / 1 Comment

libvirt and virt-manager are a blessing. They bring powerful, free, open source management to Xen- and KVM-based virtualization environments.

I’ve been using both for quite a while. Also, I’ve always prefered bridged networking support for my virtual machines over NAT. While NAT is non-disruptive and allows for isolation, I typically like to easily access services provided by my virtual machines, like SSH or NFSv4. Turns out that setting bridged networking support in libvirt is very easy, as long as bridged interface is detected by libvirt.

The simplest solution consists of creating a bridge interface that enslaves all the physical networks interfaces used to connect to the LAN or the Internet. For example, in Ubuntu, in order to enslave eth0 to a br0 bridge interface, while using DHCP for IPv4 address configuration, /etc/network/interfaces needs to look like this:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet manual

# The bridge
auto br0
iface br0 inet dhcp
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0
        bridge_maxwait 0

Next time, when creating a new virtual machine, it will be possible to use bridged networking in addition to NAT-based networking. There is one caveat, at least in Ubuntu: libvirt and virt-manager by default connect to qemu:///user instead of qemu:///system. This is neither good nor bad by itself. qemu:///user allows a non-privileged user to create and use virtual machines and the process of creating and destroying the virtual network interfaces used by the virtual machines is done within the context of the user running virt-manager. Due to lack of root privileges, virtual machines are limited to QEMU’s usermode networking support. In order to use advanced networking feautures like bridged networking, make sure you connect to qemu:///system instead. That is typically achieved by running virt-manager as root (which is not necessarily nice). I tried playing with udev and device ownership and permission masks but it all boils down to the inability of a non-privileged user to use brcrl to enslave network interfaces to a bridge.

Free VMware ESXi

29/07/2008 / Felipe Alfaro Solana / 2 Comments

It seems that the oust of Diane Greene is having big consequences. One of them is that VMware ESXi is now a free product. I see this like a direct attack to Microsoft’s attempt to get into the virtualization market. What does this mean? That there is little reason to choose Microsoft’s hypervisor when you can choose a more mature one that is now free.

Of course, hardware requirements for ESXi still make it hard to afford for small companies, but these are very good news in any case.

The hypervisors war is getting very hot!

Installing Solaris 10 under VMware Fusion on an Intel Core 2 Duo Mac

12/10/2007 / Felipe Alfaro Solana / 8 Comments

I have been fighting for quite some time to get Solaris 10 installed and working properly under VMwae Fusion 1.1 on an Intel Core 2 Duo Mac. I think this is an interaction problem between VMware Fusion and Solaris 10. Even if you tell VMware Fusion that you want to install Solaris in 32-bit mode, VMware doesn’t disable the 64-bit, long-word instruction support (available from the host). Thus, Solaris 10 is able to detect that 64-bit, long-word instructions are available and boots a 64-bit kernel. VMware then complains about the fact that the virtual machine was configured in 32-bit mode but the guest is trying to execute 64-bit instructions.

The work-around is pretty easy: it consists of disabling 64-bit, long-word instructions in the VMware configuration file for the guest. This is described in greater detail in article Installing Solaris 10 as a 32-Bit Guest Operating System on a 64-Bit Host Machine but consists mainly in editing the .vmx virtual machine’s configuration file and adding the following line:

monitor_control.disable_longmode = 1

Rebooting the installation, or booting an already installed virtual machine system should make Solaris boot into 32-bit mode.

QEMU, KQEMU and udev

08/02/2007 / Felipe Alfaro Solana / 2 Comments

Now that KQEMU has switched to the GPL v2 license, I’m starting to get interested on it.

One problem with KQEMU is that modprobing the kernel module, kqemu.ko, doesn’t automatically create /dev/kqemu unless the proper udev rules are defined.

A cannonical udev rule file to get /dev/kqemu created automatically when kqemu.ko is loaded is:

# cat /etc/udev/rules.d/60-kqemu.rules
KERNEL=="kqemu", NAME="%k", MODE="0666"'

Creating this file will tell udev to automatically create the corresponding special device file with permissions 0666 and owner root.root, but this can be easily changed to specify a different user group so that only a limited number of users, members of that group, can access KQEMU.

NetBSD 3.1 as a domU/guest on Xen

04/02/2007 / Felipe Alfaro Solana / 4 Comments

Xen is one of the coolest pieces of software I have ever used. It allows me to partition my box into manageable pieces, for increased security and increased resource utilization. I have been playing extensively with Xen for more than a year and have also written some posts about it.

NetBSD is a lean, mean, fast free, open source operating system and is nicely supported under Xen, has nice features like the PF packet filter and the pkgsrc ports-like collection and runs in nearly every single hardware architecture on earth. Because of this, I decided to run NetBSD 3.1 on Xen. NetBSD can run either as the privileged domain (called dom0) or as an unprivileged guest (called domU) domain. Since I was already running Linux under Xen as a domU, I am mostly interested in running NetBSD 3.1 as a domU guest on Xen. dom0 can be either Red Hat Enterprise Linux 5.0 or Fedora Core 6, but feel free to use any other Linux distribution as most of them are Xen-ready.

As far as I know, there are some restrictions between the Xen hypervisor + dom0 kernel and domU kernel:

You cannot mix PAE-enabled and non-PAE kernels.
For example, you cannot run a PAE-enabled dom0 kernel and/or PAE-enabled hypervisor and a non-PAE dom0/domU kernel.

This is currently a problem since Fedora Core 6 and Red Hat Enterprise Linux 5.0 both ship with a PAE-enabled Xen hypervisor and Xen-enabled kernels, but NetBSD does not currently ship a PAE-compatible, Xen-enabled kernel.
You cannot mix 64-bit and 32-bit kernels.
You cannot run a 64-bit Xen hypervisor and 64-bit dom0 kernel and a 32-bit domU kernel.

Since both Fedora Core 6 and Red Hat Enterprise Linux 5.0 ship by default with a PAE-enabled (36-bit addressable memory space) Xen hypervisor and dom0 Xen-enabled Linux kernel, the first thing that I had to do in order to run NetBSD 3.1 as domU under Xen was to recompile the Linux kernel and the Xen hypervisor with PAE support completely disabled. This is described next.

Build Xen hypervisor and dom0 kernel without PAE

You can skip to the next section if you already have a non-PAE, working Xen installation.

The first thing I had to do is to downl the SRPM (source RPM) for the latest Linux kernel, for example kernel-2.6.19-1.2895.fc6.src.rpm, then install it by running:

# rpm -i kernel-2.6.19-1.2895.fc6.src.rpm

In file /usr/src/redhat/SPECS/kernel-2.6.spec replace the following:

%ifarch i686
%define buildpae 1
# we build always xen HV with pae
%define xen_flags verbose=y crash_debug=y pae=y
%endif

with:

%ifarch i686
%define buildpae 0
# we build always xen HV with pae
%define xen_flags verbose=y crash_debug=y
%endif

This will cause the Xen hypervisor to be built without PAE support. Additionally, no PAE-enabled extra kernels will be built. The Xen kernel, however, uses its specific configuration file that has to be changed in order to disable PAE support. To disable PAE support for the Xen kernel, I reconfigured the kernel with no PAE support by running:

# rpmbuild -bp /usr/src/redhat/SPECS/kernel-2.6.spec
# cd /usr/src/redhat/BUILD/kernel-2.6.19/linux-2.6.19.i386
# cp configs/kernel-2.6.19-i686-xen.config .config
# make menuconfig

Make sure PAE is disabled by navigating to Processor type and features, then High Memory Support is set to either off or 4GB (but not 64GB).

Next, I copied the updated configuration file back to /usr/src/redhat/SOURCES, where it belongs. Also, we need to insert # i386 at the beginning of the file so that the RPM build process can derive the exact processor architecture from the config file when building the RPMs:

# cat <(echo "# i386") .config > ../../../SOURCES/kernel-2.6.19-i686-xen.config

The processor architecture is supplied to make during the build process in the form of ARCH=i386.

Now, let’s build the RPMs:

# rpmbuild -ba --target i686 ../../../SPECS/kernel-2.6.spec

We need to specify i686 as the target architecture since Fedora and Red Hat don’t use i386 anymore for kernels themselves — i386 is now only used for some common RPMs like kernel-headers.

Once the RPMs have been built, check the files under /usr/src/redhat/RPMS/i686. At least there should be a file called kernel-xen-2.6.19-1.2895.i686.rpm. This RPM contains several files, but the ones that we are interested in are:

/boot/config-2.6.19-1.2895xen
Contains the kernel configuration. Make sure either CONFIG_X86_PAE is set to n or is undefined.
/boot/vmlinuz-2.6.19-1.2895xen
The Linux Xen-enabled kernel.
/boot/xen.gz-2.6.19-1.2895
The Xen hypervisor. In the most recent versions of Fedora Core and Red Hat Enterprise Linux, the Xen hypervisor and the Xen-enabled kernel are packaged in the same RPM. This is the right thing to do since both are tightly coupled.

Install the new Xen kernel and hypervisor:

# rpm -ivh --force /usr/src/redhat/RPMS/i686/kernel-xen-2.6.19-1.2895.i686.rpm

Reboot:

# reboot

I assume the system will boot correctly and into the new Xen hypervisor and Xen-enabled Linux kernel. You can check that by running:

# uname -a
Linux xen 2.6.19-1.2895xen #1 SMP Sat Feb 3 16:56:34 CET 2007 i686 i686 i386 GNU/Linux

The next step is installing NetBSD 3.1 as a domU. This is covered next.

Installing NetBSD 3.1

The first step is preparing the Xen’s domU configuration file and its corresponding storage backend. Xen can use file-backed storage for a domU or block-backed storage (i.e. a disk partition or logical volume). Typically, block-backed storage is faster than file-backed storage, so I set up a 10GiB logical volume for NetBSD:

# lvcreate -n netbsd xen -L 10G

I also used NetBSD’s Internet-based installation since it’s the easiest way to get a working NetBSD installation and the NetBSD community have built Xen-enabled NetBSD kernels:

netbsd-INSTALL_XEN3_DOMU
A Xen-based, domU kernel used to install NetBSD.
netbsd-XEN3_DOMU
A Xen-based, domU kernel used to run the installed system.

Both files can be downloaded from /pub/NetBSD/NetBSD-3.1/i386/binary/kernel. Download and uncompress both of them:

# wget ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-3.1/i386/binary/kernel/netbsd-*XEN3_DOMU.gz
# zcat netbsd-INSTALL_XEN3_DOMU.gz > /boot/netbsd-INSTALL_XEN3_DOMU
# zcat netbsd-XEN3_DOMU.gz > /boot/netbsd-XEN3_DOMU

If you are running SELinux, you will need to relabel these files properly or xm will be unable to load them into memory:

# chcon root system_u:object_r:boot_t /boot/netbsd*

Next, create the Xen configuration file for NetBSD. In my case, it looked like this:

# cat /etc/xen/auto/netbsd
kernel = "/boot/netbsd-INSTALL_XEN3_DOMU"
memory = 256
name = "netbsd"
vif = [ 'mac=00:16:3e:00:00:11, bridge=xenbr0' ]
disk = [ 'phy:/dev/xen/netbsd,hda,w' ]
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

Now, we will install NetBSD by starting the domain:

# xm create -c /etc/xen/auto/netbsd

This will start the new domain and will attach to its console. You can follow the Example Installation NetBSD document to assist you in installing NetBSD and also Xensource NetBSDdomU Wiki page.

Once the installer has finished, do not reboot. At the end of the installation process, you’ll be brought back to the main install screen. Select e: Utility menu, then a: Run /bin/sh, then type the following at the shell:

mount /dev/xbd0a /mnt
cp -pR /dev/rxbd* /mnt/dev
cp -pR /dev/xbd* /mnt/dev
halt -p

This will copy the required special device files and shut down the guest. Now, you will have to modify the domain config file in order to use the standard NetBSD domU kernel, /boot/netbsd-XEN3_DOMU. Edit /etc/xen/auto/netbsd and replace:

kernel = "/boot/netbsd-INSTALL_XEN3_DOMU"

with:

kernel = "/boot/netbsd-XEN3_DOMU"

And boot the domain again:

# xm create -c /etc/xen/auto/netbsd

During boot, you will see some errors like:

wsconscfg: /dev/ttyEcfg: Device not configured

This is due to the NetBSD guest only having access to one physical console. To kill those errors, edit /etc/ttys from within the NetBSD guest and turn off all terminals except "console", like:

console "/usr/libexec/getty Pc"         vt100   on  secure
ttyE0   "/usr/libexec/getty Pc"         vt220   off secure
ttyE1   "/usr/libexec/getty Pc"         vt220   off secure
ttyE2   "/usr/libexec/getty Pc"         vt220   off secure
ttyE3   "/usr/libexec/getty Pc"         vt220   off secure
...

Also, comment out all screens in /etc/wscons.conf:

#screen 0       -       vt100
#screen 1       -       vt100
#screen 2       -       vt100
#screen 3       -       vt100
#screen 4       -       -
#screen 4       80x25bf vt100

That’s all. Now we have a fully functional NetBSD 3.1 domU guest running on Xen 🙂

References

The information and instructions on this post are based on:

NetBSDdomU — How to install NetBSD as a domU on a Linux host.
Example Installation — NetBSD example installation.

Xen network configuration and multiple VLANs (II)

21/08/2006 / Felipe Alfaro Solana / 21 Comments

Since I first write my first attempt at trying to get VLAN support working under Xen, I’ve received some reports for people stating that it doesn’t work as expected. And they are right.

At the end of the first article, I pointed out I was having problems with UDP traffic. In turn, it was worse than I ever expected, since it was affecting DNS name resolution, DHCP services and other services running as inside a domainU. This is the reason why I rethought the implementation and now have it working on a production machine acting, among as other things, as a DHCP server and DNS server.

In this second try I decided not to mess around with Xen’s default network configuration, so please undo all the changes you did so you end end up with a pristine Xen configuration. In this new scenario all the native traffic (tagged an untagged Ethernet frames) is being captured by Xen’s switch, xenbr0, and sent to the right network interface. If the traffic being received is a 802.11q tagged frame, the target will receive it tagged and thus will have to implement measures to untag and process it accordingly.

Introduction

So, let’s say we have the following logical network topology and virtual machines:

                   |
                  LAN
                   |
-------------------+----------------------------------
|                  |                                 |
|                peth0 ---- xen-br0                  |
|                              |                     |
|            -----------------------------           |
|            |                           |           |
|          vif0.0                     vif1.0         |
|            |                           |           |
|            |            +--------------+------------
|            |            |              |
|            |            |  ------------+------------
|            |            |  |           |           |
|           eth0          |  |          eth0         |
|            |            |  |           |           |
|     -------+-------     |  |     ------+-------    |
|     |             |     |  |     |            |    |
| eth0.1000      eth0.10  |  | eth0.2000     eth0.10 |
|     |             |     |  |     |            |    |
| VLAN 1000      VLAN 10  |  | VLAN 2000     VLAN 10 |
|     |             |     |  |     |            |    |
|    www           ssh    |  |    ftp          ssh   |
|                         |  |                       |
|        Domain0          |  |        DomainU        |
---------------------------  -------------------------

The Xen’s switch configuration can be seen with the following command:

root@xen:~# brctl show
bridge name  bridge id           STP enabled   interfaces
xenbr0       8000.feffffffffff   no            peth0
                                               vif0.0
                                               vif1.0

For each domain — this includes domain0 or any domainU — there is a vif|X|.|Y| interface attached to Xen’s bridge xen-br0, where |X| is the domain ID (0 for domain0 and a monotonically increasing number for every domainU). Then, we have every network interface card inside the domain, in the form of eth|Y|. Thus, if a domainU with ID #3 defines two network interfaces, eth0 and eth1, there will two corresponding virtual network interfaces in domain0, named vif3.0 and vif3.1.

Instead of trying to export VLAN interfaces to one or more domainUs, we export the whole, native (tagged or not) network interface to the domainU and, inside this domainU, we can configure VLAN subinterfaces if needed.

Sample scenario

Let’s say we want to offer the following services per VLAN:

WWW server on VLAN 1000
FTP server on VLAN 2000
SSH access to administer the WWW sever, reachable only through the VLAN 10
SSH server to administer the FTP server, reachable only through the VLAN 10

But we also want to partition the physical machine in two, so domain0 serves WWW traffic while domainU servers FTP traffic:

	WWW	FTP	SSH
domain0	VLAN 1000	–	VLAN 10
domainU	–	VLAN 2000	VLAN 10

Thus, we need the following VLAN subinterfaces:

eth0.10 and eth0.1000 on domain0
eth0.10 and eth0.2000 on domainU

Configuring VLAN subinterfaces in domainU is straight forward. However, it’s a little bit more difficult for domain0.

Configuring VLAN subinterfaces for domain0

First of all, make sure you are using bridging for your Xen configuration. Make sure the following line is uncommented in /etc/xen/xend-config.sxp:

(network-script network-bridge)

And comment any other network-script configuration lines, like:

(network-script network-nat)

(network-script network-route)

It seems we can’t bring up VLAN subinterfaces before Xen’s network script is fired up since Xen’s network scripts perform some black magic on the network interfaces, mainly renaming eth0 to peth0 and bringing up a dummy interface named eth0. Any subinterface related to the original eth0 seems to stop working after the renaming takes place.

Thus, I coded up an init script used to bring up the VLAN subinterfaces that gets invoked just after Xen’s network script has finished. Note that it’s targeted for RedHat-based distributions:

#!/bin/sh
#
# Init file for Network-VLAN
# STARTS AFTER XEN (which is S50 and K01)
#
# chkconfig: 2345 51 89
# description: VLAN networking

. /etc/init.d/functions

case "$1" in
start)
 echo -n $"Configuring VLAN interfaces:"

 if [ ! -f /var/lock/subsys/network-vlan ]; then
  (
  modprobe 8021q || exit 1
  vconfig add eth0 10 || exit 2
  ifconfig eth0.10 up 10.0.0.1 netmask 255.0.0.0 || exit 3
  vconfig add eth0 1000 || exit 2
  ifconfig eth0.1000 up 11.0.0.1 netmask 255.0.0.0 || exit 3
  ) > /dev/null 2>&1

  RETVAL=$?
  [ "$RETVAL" = 0 ] && ( success ;\
    touch /var/lock/subsys/network-vlan ) || failure
 fi
 echo

 ;;

stop)
 echo -n $"Unconfiguring VLAN interfaces:"

 if [ -f /var/lock/subsys/network-vlan ]; then
  (
  ifconfig eth0.10 down && vconfig rem eth0.10 ;
  ifconfig eth0.1000 down && vconfig rem eth0.1000
  ) > /dev/null 2>&1

  RETVAL=$?
  [ "$RETVAL" = 0 ] && ( rm -f /var/lock/subsys/network-vlan ;\
    success ) || failure
 fi
 echo
esac

Save this script as /etc/init.d/network-vlan, then run:

chmod +x /etc/init.d/network-vlan
chkconfig --add /etc/init.d/network-vlan

The script runs just after Xen’s init script has renamed the real Ethernet interface and has brought up a dummy interface called eth0. Then, the network-vlan script brings up two VLAN subinterfaces, one for VLAN 10 and another one for VLAN 1000, and then assigns each one its own IP address.

Additionally, these are the contents of /etc/sysconfig/network-scripts/ifcfg-eth0:

DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
TYPE=Ethernet

Note that eth0 in this context refers to the real Ethernet interface, since Xen’s init script has not been ran yet. I didn’t configure any IP address for this interface since I only want to process tagged traffic. Beware that on many switches — i.e., Cisco 2960 and 3560 —, VLAN1 is, by default, the native VLAN and traffic on the native VLAN doesn’t get tagged.

Configuring VLAN subinterfaces for domainU

These are the contents of /etc/sysconfig/network-scripts/ifcfg-eth0:

DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
TYPE=Ethernet

I didn’t configure any IP address for this interface since I only want to process tagged traffic. Read the note above on untagged frames and native VLANs.

These are the contents of /etc/sysconfig/network-scripts/ifcfg-eth0.10:

DEVICE=eth0.10
BOOTPROTO=static
IPADDR=10.0.0.2
NETMASK=255.0.0.0
ONBOOT=yes
TYPE=Ethernet
VLAN=yes

These are the contents of /etc/sysconfig/network-scripts/ifcfg-eth0.2000:

DEVICE=eth0.2000
BOOTPROTO=static
IPADDR=12.0.0.1
NETMASK=255.0.0.0
ONBOOT=yes
TYPE=Ethernet
VLAN=yes

Bonding

For those who desire to use bonding, it seems some tweaking of the networking scripts is required. I recommend them to look at this post on Bonding not working with network-bridge.

Conclusion

I’m sure there are better ways to configure VLAN subinterfaces in domain0, but it was in a hurry and couldn’t find of a better way to get it done.

If anyone out there has a different way of achieving this, please let me know 🙂

Xen network configuration and multiple VLANs

21/07/2006 / Felipe Alfaro Solana / 23 Comments

Xen networking is powerful enough to allow for extreme customization. Although the default networking configuration is usually more than enough for simple scenarios, it can fall short when trying to support multiple guests standing on different VLANs.

In this short article, I describe the steps needed to configure Xen to attach itself to multiple VLANs using a one-bridge-per-VLAN network interface mapping, then attaching each Xen domainU on as many VLANs as needed.

In the sample scenario, we will use a Cisco Catalyst 3560G-24TS switch carrying traffic from five different VLANs:

VLAN2 is the administrative VLAN used to administer all the networking gear and boxes.
VLAN10 carries Internet traffic coming from the first ISP.
VLAN20 carries Internet traffic coming from the second ISP.
VLAN100 carries the access network traffic.
VLAN200 carries the core network traffic.

The final Xen configuration will provide five bridging network interfaces, one per VLAN. Each Xen domainU can freely attach to any of these bridging network interfaces in order to gain access to the traffic being carried by each VLAN.

The bridging interface, |brname| is named after the following convention: xenbr|vlan|:

xenbr2 is the bridging interface standing on VLAN2.
xenbr10 is the bridging interface standing on VLAN10.
xenbr20 is the bridging interface standing on VLAN20.
xenbr100 is the bridging interface standing on VLAN100.
xenbr200 is the bridging interface standing on VLAN200.

Also, Xen creates an manages several virtual network interfaces, named in the form of vif|X|.|Y|, where |X| equals the Xen domain numeric ID and |Y| is a sequential interface index. Thus, starting up a Xen domainU given the following virtual network interface definition:

vif = [ 'mac=00:16:3e:00:00:44, bridge=xenbr10',
        'mac=00:16:e3:00:00:45, bridge=xenbr20' ]

Will cause the Xen domain to get assigned, let’s say, a domain ID of 2, and two virtual network interfaces named vif2.0 — attached to xenbr10 — and vif2.1 — attached to xenbr20.

Setting up the bridging interfaces:

This can be done manually, by invoking brctl addbr |brname| in order to create a new bridging interface.

For example, the following commands will create five bridging interfaces, one for each supported VLAN:

brctl addbr xenbr2
brctl addbr xenbr10
brctl addbr xenbr20
brctl addbr xenbr100
brctl addbr xenbr200

or else can be automated to get done during system startup, by creating a file named /etc/sysconfig/network-scripts/ifcfg-|brname|, where |brname| is the name assigned to the bridging interface, like /etc/sysconfig/network-scripts/ifcfg-xenbr2 (the configuration file for the bridging interface standing on VLAN2):

DEVICE=xenbr2
BOOTPROTO=static
IPADDR=192.168.0.10
NETMASK=255.255.0.0
ONBOOT=yes
TYPE=Bridge

Setting up the VLAN interfaces and add them up to the existing bridging interfaces:

Any traffic sent to this interface will get tagged for VLAN |vlan|.
Any traffic received from interface |ifname| carrying an 802.1q VLAN tag matching |vlan| will be untagged and received by this interface.

vconfig add eth0 2
vconfig add eth0 10
vconfig add eth0 20
vconfig add eth0 100
vconfig add eth0 200

This will add five new VLAN interfaces, one for every supported VLAN.


brctl addif xenbr2 eth0.2
brctl addif xenbr10 eth0.10
brctl addif xenbr20 eth0.20
brctl addif xenbr100 eth0.100
brctl addif xenbr200 eth0.200

The process of adding up a new VLAN interface and then adding it up to an existing bridging interface can be configured using a single configuration file named ifcfg-|ifname|.|vlan|, like /etc/sysconfig/network-scripts/ifcfg-eth0.2:


DEVICE=eth0.2
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
VLAN=yes
BRIDGE=xenbr2

Keeping Xen from reconfiguring the network:

Since we have already configured the network manually, we don’t want Xen to mess up with the configuration. In order to keep Xen from reconfiguring the network, simply make sure none of the following lines appear uncommented in the file /etc/xen/xend-config.sxp:

(network-script network-bridge)
(network-script network-route)
(network-script network-nat)

Additional notes:

I have been experiencing a very strange behavior on Xen domainU guests while using this network configuration: it seems that UDP traffic gets stuck at the network stack and does not flow through unless I load the ip_conntrack.ko kernel module.

Failing to load the ip_conntrack.ko kernel module, even with an unconfigured, empty firewall, allows ICMP and TCP traffic to flow from and to the guest network stack, but UDP traffic, like DNS queries, gets stuck and doesn’t even touch the physical network interface.

This is really strange, isn’t it?

VMWare performance tricks

22/11/2005 / Felipe Alfaro Solana / 6 Comments

From How to improve disk I/O performances with VMware Workstation 5:

Memory trimming

Workstation checks which part of the guest OS virtual memory is not used and allocates it back to the host OS. This permits to have more concurrent virtual machines running but everytime the guest OS asks back for its memory it suffers a performance degradation.

So, if you have enough free RAM for all planned concurrent VMs, be sure to disable memory trimming for guest OSes adding the following line to the virtual machine configuration (.vmx) file:
MemTrimRate=0
Page sharing

VMware uses a page sharing technique to allow guest memory pages with identical contents to be stored as a single copy-on-write page. Page sharing decreases host memory usage, but consumes system resources, potentially including I/O bandwidth.

You may want to avoid this overhead for guests for which host memory is plentiful and I/O latency is important. To disable page sharing, add the following line to the virtual machine configuration (.vmx) file:
sched.mem.pshare.enable=FALSE option

XEN

18/08/2005 / Felipe Alfaro Solana / 2 Comments

Xen implements paravirtualization to allow for software partitioning.

Xen is an hypervisor that allows running several partitions, called “domains”. Each domain is totally independent, virtualized and solated by the Xen hypervisor, which controls access to the hardware and shares the physical resources among all running domains.

There are two domain classes:

Privileged domain: has some limited access to the hardware, and is used to provide for virtual driver support for all the non-privileged domains. There can only be one privileged domain running on a machine at any time and its called “domain0”.
For example, the privileged domain has access to the video frame buffer and can run an accelerated X server, and has limited access to real devices, like eth0, hda, and so on.
Unprivileged domain: has no direct hardware access, and all the I/O operations are virtualized by using virtual devices, like the virtual block device or virtual network device, which are in turn hooked to the privileged domain. There can be as many unprivileged domains as system resources allow for.
Unprivileged domains don’t have a video frame buffer and, thus, all GUI support must get implented using software like VNCserver or FreeNX, then remotely controlling the session from another domain or machine.

Xen selects a random hardware MAC address if not specifically told to use a particular one. This causes problems with networking since the MAC address for the domain virtual interface (called vif) changes between domain reboots and switches and computers keep the corresponding entry on their ARP caches for a few minutes.

Let be A a virtual unprivileged domain running under Xen’s control. After A has started, any other domain or real machine, will be unable to talk to A until the moment A forces an ARP cache update by sending out a packet (for example, an ICMP packet).

I see two solutions to this problem:

When domain A starts, it initiates an ARP cache refresh of other domains or machines.
This can be achieved by sending a gratuitious ARP cache on interface wakeup, which not every Linux distribution does, or send some ICMP echo request packets to force sending ARP requests. This can be easily implemented in init scripts.
Forcing domain A to use a fixed MAC address.
This can be achieved by using the following line in A domain configuration file:

vif = [ 'mac = XX:XX:XX:XX:XX:XX' ]

The latter solution is the easiest and, usually, the preferred one.

NOTE: This also covered in the Xen FAQ.

felipe-alfaro.com

Not just another WordPress.com weblog

Virtualization