Xen network configuration and multiple VLANs (II)

/ Felipe Alfaro Solana

Since I first write my first attempt at trying to get VLAN support working under Xen, I’ve received some reports for people stating that it doesn’t work as expected. And they are right.

At the end of the first article, I pointed out I was having problems with UDP traffic. In turn, it was worse than I ever expected, since it was affecting DNS name resolution, DHCP services and other services running as inside a domainU. This is the reason why I rethought the implementation and now have it working on a production machine acting, among as other things, as a DHCP server and DNS server.

In this second try I decided not to mess around with Xen’s default network configuration, so please undo all the changes you did so you end end up with a pristine Xen configuration. In this new scenario all the native traffic (tagged an untagged Ethernet frames) is being captured by Xen’s switch, xenbr0, and sent to the right network interface. If the traffic being received is a 802.11q tagged frame, the target will receive it tagged and thus will have to implement measures to untag and process it accordingly.

Introduction

So, let’s say we have the following logical network topology and virtual machines:

                   |
                  LAN
                   |
-------------------+----------------------------------
|                  |                                 |
|                peth0 ---- xen-br0                  |
|                              |                     |
|            -----------------------------           |
|            |                           |           |
|          vif0.0                     vif1.0         |
|            |                           |           |
|            |            +--------------+------------
|            |            |              |
|            |            |  ------------+------------
|            |            |  |           |           |
|           eth0          |  |          eth0         |
|            |            |  |           |           |
|     -------+-------     |  |     ------+-------    |
|     |             |     |  |     |            |    |
| eth0.1000      eth0.10  |  | eth0.2000     eth0.10 |
|     |             |     |  |     |            |    |
| VLAN 1000      VLAN 10  |  | VLAN 2000     VLAN 10 |
|     |             |     |  |     |            |    |
|    www           ssh    |  |    ftp          ssh   |
|                         |  |                       |
|        Domain0          |  |        DomainU        |
---------------------------  -------------------------

The Xen’s switch configuration can be seen with the following command:

root@xen:~# brctl show
bridge name  bridge id           STP enabled   interfaces
xenbr0       8000.feffffffffff   no            peth0
                                               vif0.0
                                               vif1.0

For each domain — this includes domain0 or any domainU — there is a vif|X|.|Y| interface attached to Xen’s bridge xen-br0, where |X| is the domain ID (0 for domain0 and a monotonically increasing number for every domainU). Then, we have every network interface card inside the domain, in the form of eth|Y|. Thus, if a domainU with ID #3 defines two network interfaces, eth0 and eth1, there will two corresponding virtual network interfaces in domain0, named vif3.0 and vif3.1.

Instead of trying to export VLAN interfaces to one or more domainUs, we export the whole, native (tagged or not) network interface to the domainU and, inside this domainU, we can configure VLAN subinterfaces if needed.

Sample scenario

Let’s say we want to offer the following services per VLAN:

  • WWW server on VLAN 1000
  • FTP server on VLAN 2000
  • SSH access to administer the WWW sever, reachable only through the VLAN 10
  • SSH server to administer the FTP server, reachable only through the VLAN 10

But we also want to partition the physical machine in two, so domain0 serves WWW traffic while domainU servers FTP traffic:

WWW FTP SSH
domain0 VLAN 1000 – VLAN 10
domainU – VLAN 2000 VLAN 10

Thus, we need the following VLAN subinterfaces:

  • eth0.10 and eth0.1000 on domain0
  • eth0.10 and eth0.2000 on domainU

Configuring VLAN subinterfaces in domainU is straight forward. However, it’s a little bit more difficult for domain0.

Configuring VLAN subinterfaces for domain0

First of all, make sure you are using bridging for your Xen configuration. Make sure the following line is uncommented in /etc/xen/xend-config.sxp:

(network-script network-bridge)

And comment any other network-script configuration lines, like:

(network-script network-nat)

or

(network-script network-route)

It seems we can’t bring up VLAN subinterfaces before Xen’s network script is fired up since Xen’s network scripts perform some black magic on the network interfaces, mainly renaming eth0 to peth0 and bringing up a dummy interface named eth0. Any subinterface related to the original eth0 seems to stop working after the renaming takes place.

Thus, I coded up an init script used to bring up the VLAN subinterfaces that gets invoked just after Xen’s network script has finished. Note that it’s targeted for RedHat-based distributions:

#!/bin/sh
#
# Init file for Network-VLAN
# STARTS AFTER XEN (which is S50 and K01)
#
# chkconfig: 2345 51 89
# description: VLAN networking

. /etc/init.d/functions

case "$1" in
start)
 echo -n $"Configuring VLAN interfaces:"

 if [ ! -f /var/lock/subsys/network-vlan ]; then
  (
  modprobe 8021q || exit 1
  vconfig add eth0 10 || exit 2
  ifconfig eth0.10 up 10.0.0.1 netmask 255.0.0.0 || exit 3
  vconfig add eth0 1000 || exit 2
  ifconfig eth0.1000 up 11.0.0.1 netmask 255.0.0.0 || exit 3
  ) > /dev/null 2>&1

  RETVAL=$?
  [ "$RETVAL" = 0 ] && ( success ;\
    touch /var/lock/subsys/network-vlan ) || failure
 fi
 echo

 ;;

stop)
 echo -n $"Unconfiguring VLAN interfaces:"

 if [ -f /var/lock/subsys/network-vlan ]; then
  (
  ifconfig eth0.10 down && vconfig rem eth0.10 ;
  ifconfig eth0.1000 down && vconfig rem eth0.1000
  ) > /dev/null 2>&1

  RETVAL=$?
  [ "$RETVAL" = 0 ] && ( rm -f /var/lock/subsys/network-vlan ;\
    success ) || failure
 fi
 echo
esac

Save this script as /etc/init.d/network-vlan, then run:

chmod +x /etc/init.d/network-vlan
chkconfig --add /etc/init.d/network-vlan

The script runs just after Xen’s init script has renamed the real Ethernet interface and has brought up a dummy interface called eth0. Then, the network-vlan script brings up two VLAN subinterfaces, one for VLAN 10 and another one for VLAN 1000, and then assigns each one its own IP address.

Additionally, these are the contents of /etc/sysconfig/network-scripts/ifcfg-eth0:

DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
TYPE=Ethernet

Note that eth0 in this context refers to the real Ethernet interface, since Xen’s init script has not been ran yet. I didn’t configure any IP address for this interface since I only want to process tagged traffic. Beware that on many switches — i.e., Cisco 2960 and 3560 —, VLAN1 is, by default, the native VLAN and traffic on the native VLAN doesn’t get tagged.

Configuring VLAN subinterfaces for domainU

These are the contents of /etc/sysconfig/network-scripts/ifcfg-eth0:

DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
TYPE=Ethernet

I didn’t configure any IP address for this interface since I only want to process tagged traffic. Read the note above on untagged frames and native VLANs.

These are the contents of /etc/sysconfig/network-scripts/ifcfg-eth0.10:

DEVICE=eth0.10
BOOTPROTO=static
IPADDR=10.0.0.2
NETMASK=255.0.0.0
ONBOOT=yes
TYPE=Ethernet
VLAN=yes

These are the contents of /etc/sysconfig/network-scripts/ifcfg-eth0.2000:

DEVICE=eth0.2000
BOOTPROTO=static
IPADDR=12.0.0.1
NETMASK=255.0.0.0
ONBOOT=yes
TYPE=Ethernet
VLAN=yes

Bonding

For those who desire to use bonding, it seems some tweaking of the networking scripts is required. I recommend them to look at this post on Bonding not working with network-bridge.

Conclusion

I’m sure there are better ways to configure VLAN subinterfaces in domain0, but it was in a hurry and couldn’t find of a better way to get it done.

If anyone out there has a different way of achieving this, please let me know 🙂

21 thoughts on “Xen network configuration and multiple VLANs (II)”

  1. I’ve tested you configuration as this and works well.

    I’m trying to setup bond0 instead of eth0 but It’s little bite more complicated. For example you can’t enslave eth0+eth1 before run Xend because to rename an interface, this interface must be down. So you should enslave directly in network-bridge script …

    I will post moredetails ASAP !

  2. It seems that only one change is required in network-bridge script.

    You just need to enslave manualy
    a 2nd time. But be carefull, you must bring up “bond0” and enslave “eth0+eth1” correctly in your main network config.

    So after that, In you main xend-config.sxp you should have a line like this :

    (network-script my-network-bridge netdev=bond0)

    In the file “my-network-bridge” that is just a copy of the original “network-bridge” script you must add just after :

    ip set link ${pdev} up

    this line :

    ifenslave ${pdev} eth0 eth1

    Seems that works for me with VLAN inside DOMU …

    If anyone could test it …

  3. After many test …
    bonding not works well !

    It’s very very slow or loose packet…

    So I’ve deciced to use only eth0

    With you config I must force vlan interface mtu to 1496 in each domU. My e1000 are compatible with packets > 1500 but the virtual Xen network driver seems that not …

  6. it work only with the mtu of the DomU vlan at 1492 … but does not work fine. I can mount nfs partition but I can’t browse it.

  7. Hi,
    With you configuration, is it possible to:
    sniff arp broadcast for eth0-DOM0
    from eth0-DOMU?

    I want to disenabled that, and I can’t solve it.
    I’ve tryed ebtable, but nothing!
    So I would like to try with VLAN

    Thanks.

    Laurent.

  8. Hi Felipe,

    I’m trying to implement a configuration like yours on a CentOS5.1 machine. But, I can’t access the outside network (via a Cisco switch). I can ping Dom0 from DomU and isolate DomUs on different VLANs. I can access the network from Dom0 if I didn’t start the bridge, but after that no. I can sniff the network traffic (all VLANs) when sniff the bridge, and can “see” the packets from/to the DomUs from/to the network, but they are lost at some point on the xen bridge. Can You helpme ?

    Thanks,

    Antonio

  9. Pingback: Running Systems » Blog Archive » Xen guests cannot serve NFS requirests

  13. An fascinating discussion is worth comment. I believe that you ought to write regarding this topic, it may not be a taboo topic but generally persons are too few to chat on such topics. To another location. Cheers

  16. I simply needed to thank you very much again. I do not know the things I would’ve used in the absence of the actual thoughts provided by you regarding my problem. Certainly was the difficult case for me, but seeing a skilled avenue you resolved the issue took me to leap over fulfillment. Now i’m thankful for this guidance and thus hope you realize what an amazing job you happen to be putting in training people today by way of your site. I am sure you’ve never come across any of us.

  19. hello web master, your weblog pagea€?s themes is striking mixed with supporting undertaking it. Any composes seem to be astounding. Ensure to help keep pace the very good give fantastic outcomes. Meets.

  20. I truly wanted to construct a simple word in order to thank you for some of the precious information you are sharing at this website. My extended internet look up has at the end been compensated with good facts and techniques to go over with my visitors. I would express that we website visitors are very much blessed to dwell in a remarkable site with many awesome professionals with great things. I feel very privileged to have encountered your website and look forward to so many more entertaining minutes reading here. Thank you once again for all the details.

Leave a Reply

Fill in your details below or click an icon to log in:

Gravatar
WordPress.com Logo

You are commenting using your WordPress.com account. ( Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. ( Log Out /  Change )

Cancel

Connecting to %s