Xen network configuration and multiple VLANs

/ Felipe Alfaro Solana

Xen networking is powerful enough to allow for extreme customization. Although the default networking configuration is usually more than enough for simple scenarios, it can fall short when trying to support multiple guests standing on different VLANs.

In this short article, I describe the steps needed to configure Xen to attach itself to multiple VLANs using a one-bridge-per-VLAN network interface mapping, then attaching each Xen domainU on as many VLANs as needed.

In the sample scenario, we will use a Cisco Catalyst 3560G-24TS switch carrying traffic from five different VLANs:

  • VLAN2 is the administrative VLAN used to administer all the networking gear and boxes.
  • VLAN10 carries Internet traffic coming from the first ISP.
  • VLAN20 carries Internet traffic coming from the second ISP.
  • VLAN100 carries the access network traffic.
  • VLAN200 carries the core network traffic.

The final Xen configuration will provide five bridging network interfaces, one per VLAN. Each Xen domainU can freely attach to any of these bridging network interfaces in order to gain access to the traffic being carried by each VLAN.

The bridging interface, |brname| is named after the following convention: xenbr|vlan|:

  • xenbr2 is the bridging interface standing on VLAN2.
  • xenbr10 is the bridging interface standing on VLAN10.
  • xenbr20 is the bridging interface standing on VLAN20.
  • xenbr100 is the bridging interface standing on VLAN100.
  • xenbr200 is the bridging interface standing on VLAN200.

Also, Xen creates an manages several virtual network interfaces, named in the form of vif|X|.|Y|, where |X| equals the Xen domain numeric ID and |Y| is a sequential interface index. Thus, starting up a Xen domainU given the following virtual network interface definition:

vif = [ 'mac=00:16:3e:00:00:44, bridge=xenbr10',
        'mac=00:16:e3:00:00:45, bridge=xenbr20' ]

Will cause the Xen domain to get assigned, let’s say, a domain ID of 2, and two virtual network interfaces named vif2.0 — attached to xenbr10 — and vif2.1 — attached to xenbr20.

Setting up the bridging interfaces:

This can be done manually, by invoking brctl addbr |brname| in order to create a new bridging interface.

For example, the following commands will create five bridging interfaces, one for each supported VLAN:

brctl addbr xenbr2
brctl addbr xenbr10
brctl addbr xenbr20
brctl addbr xenbr100
brctl addbr xenbr200

or else can be automated to get done during system startup, by creating a file named /etc/sysconfig/network-scripts/ifcfg-|brname|, where |brname| is the name assigned to the bridging interface, like /etc/sysconfig/network-scripts/ifcfg-xenbr2 (the configuration file for the bridging interface standing on VLAN2):

DEVICE=xenbr2
BOOTPROTO=static
IPADDR=192.168.0.10
NETMASK=255.255.0.0
ONBOOT=yes
TYPE=Bridge

Setting up the VLAN interfaces and add them up to the existing bridging interfaces:

This can be done manually, by invoking vconfig add |ifname| |vlan| to configure VLAN number |vlan| by using 802.1q tagging on interface |ifname|. This will active a virtual interface named |ifname|.|vlan|:

  • Any traffic sent to this interface will get tagged for VLAN |vlan|.
  • Any traffic received from interface |ifname| carrying an 802.1q VLAN tag matching |vlan| will be untagged and received by this interface.
vconfig add eth0 2
vconfig add eth0 10
vconfig add eth0 20
vconfig add eth0 100
vconfig add eth0 200

This will add five new VLAN interfaces, one for every supported VLAN.

Once the VLAN interfaces are ready, we add them to their corresponding bridging interfaces by using brctl addif |brname| |ifname|.|vlan|:


brctl addif xenbr2 eth0.2
brctl addif xenbr10 eth0.10
brctl addif xenbr20 eth0.20
brctl addif xenbr100 eth0.100
brctl addif xenbr200 eth0.200

The process of adding up a new VLAN interface and then adding it up to an existing bridging interface can be configured using a single configuration file named ifcfg-|ifname|.|vlan|, like /etc/sysconfig/network-scripts/ifcfg-eth0.2:


DEVICE=eth0.2
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
VLAN=yes
BRIDGE=xenbr2

Keeping Xen from reconfiguring the network:

Since we have already configured the network manually, we don’t want Xen to mess up with the configuration. In order to keep Xen from reconfiguring the network, simply make sure none of the following lines appear uncommented in the file /etc/xen/xend-config.sxp:

(network-script network-bridge)
(network-script network-route)
(network-script network-nat)

Additional notes:

I have been experiencing a very strange behavior on Xen domainU guests while using this network configuration: it seems that UDP traffic gets stuck at the network stack and does not flow through unless I load the ip_conntrack.ko kernel module.

Failing to load the ip_conntrack.ko kernel module, even with an unconfigured, empty firewall, allows ICMP and TCP traffic to flow from and to the guest network stack, but UDP traffic, like DNS queries, gets stuck and doesn’t even touch the physical network interface.

This is really strange, isn’t it?

23 thoughts on “Xen network configuration and multiple VLANs”

  1. i’ve been trying to set up this exact scenario but have not been successful. everything works fine in the dom0:

    # vconfig add eth0 560
    # ifconfig eth0.560 10.10.10.182/30
    root@xen-server-1 ~]# ping -c 1 10.10.10.181
    64 bytes from 10.10.10.181: icmp_seq=1 ttl=255 time=0.493 ms

    note: 10.10.10.181 is only reachable via vlan 560

    but if i follow your steps:

    # brctl addbr xenbr560
    # vconfig add eth0 560
    # brctl addif xenbr560 eth0.560

    then boot my domU with vif = [‘bridge=xenbr560’] in the config:

    [root@xen-server-1 ~]# brctl show
    bridge name bridge id STP enabled interfaces
    xenbr560 8000.000d566ffea8 no eth0.560
    vif1.0

    eth0 in the domU is set to 10.10.10.182/30, but it cannot ping 10.10.10.181.

    am i mising something?

  2. Don’t know exactly what’s going on, but I’ve used the exact configuration described in the article… However, I had serious problems making it work correctly (read the notes at the end of the article wrt. UDP and some other traffic).

    In the end, I left the networking configuration of dom0 alone and, instead, did deploy VLAN subinterfaces inside the virtual machines. For example, I leave the networking configuration as it comes by default (i.e. bridge mode where all domU’s VNIC’s attach to the virtual switch xen-br0). Then, if a domU guest needs to attach to both VLAN ID 3 and VLAN ID 500, I have to manually configure two VLAN subinterfaces inside that domU using vconfig or the networking scripts.

    This has proven to work much better, although this doesn’t isolate the domU machines from VLAN changes. That is, if I decide to move a domU from a VLAN to another, I have to reconfigure the domU machine.

  3. hmm, i’m pretty sure i tried that as well and still couldn’t get it working. but thank you very much for the tip (as well as for the initial writeup), i’ll give it another try.

  4. Currently, I don’t have access to the Xen machine where all this is running. However, as soon as I get access back, I’ll post another article on how I did manage to get it working.

  5. w00t, i do believe i have it working. you’re right, leaving the dom0 stuff totally alone and setting things inside the domU did the trick. thanks a ton, drop me a line if you’re ever in raleigh, nc 🙂

  7. I’ve tried to setup vlan with xen and not works 😦

    Xen server runs debian sarge 3.1 + 2.6.16.19 + xen 3.0.2

    when I try todo a simple ping of a VM inside vlan I loose packet or sometimes not answer.

    It’s very strange.
    Any ideas ?

    br200 8000.00304877ef62 no vif2.0
    vlan200
    br300 8000.00304877ef62 no vif5.0
    vif6.0
    vif8.0
    vlan300
    br400 8000.00304877ef62 no vif3.0
    vlan400
    br500 8000.00304877ef62 no vif7.0
    vlan500
    br700 8000.00304877ef62 no vif4.0
    vlan700

  8. Yeah! At the end of the post I comment on I had similar problems: some UDP datagrams were getting lost and, at the end, I revert to using the old-style, default networking scripts and stuff from Xen. Instead of doing what I describe in this post, I end up defining the VLANs in the domU guests, which works reliably.

    I hope to write a short addendum to this article very soon on how to configure Xen for VLAN support, in dom0 and domU’s, using a different way which works perfectly.

  9. Very good initiative 🙂

    Please let me a mail when you addendum is ready …

    I need to find a very reliable solution to my config asap.

    Moreover Have you test vlan over bonding ?

  13. Jeje, bueno estoy virtualizando con xen un servidor. cual tiene dns, imap,postfix bla bla, en cuatro maquinas virtuales. EL problema es que yo quiero que esas maquinas virtuales no se vean entre si es decir desde afuera.. por eso pensaba en hacer un bridge para cada maquina virtual o crear un vlan y hacer lo mismo con los bridge.. no se que me recomiendas tu y se me podrias guiar un poco gracias

    ___________ __________
    > |VM1 |—————-bridge1——————-| |
    > |________|
    > | |
    > ____________
    > | |—–BALBLA
    > |VM2 |
    > | VM4 |
    > |__________ |—————–bridge2—————-|Firewall |
    > ____________ |
    > |
    > |VM3 |
    > | |
    > |__________ |—————–bridge3—————— |________|

  14. Hi Felipe,
    my setup is quite the same, except I’m using SLES10. I too I’m experiencing some packet loss: this is particularly visible for UDP traffic. Around 4% of bytes transferred get lost (in UDP). I’ve a thread on xen-users ML, if you have comments.
    Experimenting with VMware ESX V3 give me — for udp traffic — a 0.8% loss for bytes transferred.
    XenSource will release XenEnterprise 3.2 (still in beta) at the end of March. They will provide Vlan management, but still don’t know if this problem will be present.
    As for the other solution you suggest (Vlan’s inside the guest) I think it’s less secure and so less acceptable.
    Regards,
    — Marco

  15. Felipe:

    como puedo configurar el eth1 ??

    no he podido configurarla… sabes como se hace?

  20. This article is really very useful. I admire the moment you invested writing this write-up. I would like to become a nurse, do you have any strategy where I could get details about nurse pays, career descriptions and what is the basis. Any option where I could get this data?

  22. An unputdownable discussion is designer account. I opine that you should indite solon on this issue, it strength not be a prejudice study but mostly fill are not sufficiency to mouth on specified topics. To the next. Cheers like your Anti Wrinkle Face Cream A Surprising New Discovery Aloebeauty Designerwear.

  23. Hello there! Quick question that’s totally off topic.
    Do you know how to make your site mobile friendly? My website looks weird when browsing from my
    iphone 4. I’m trying to find a template or plugin that might be able to correct this problem.
    If you have any suggestions, please share. Thanks!

Leave a Reply

Fill in your details below or click an icon to log in:

Gravatar
WordPress.com Logo

You are commenting using your WordPress.com account. ( Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. ( Log Out /  Change )

Cancel

Connecting to %s