Xen networking is powerful enough to allow for extreme customization. Although the default networking configuration is usually more than enough for simple scenarios, it can fall short when trying to support multiple guests standing on different VLANs.
In this short article, I describe the steps needed to configure Xen to attach itself to multiple VLANs using a one-bridge-per-VLAN network interface mapping, then attaching each Xen domainU on as many VLANs as needed.
In the sample scenario, we will use a Cisco Catalyst 3560G-24TS switch carrying traffic from five different VLANs:
- VLAN2 is the administrative VLAN used to administer all the networking gear and boxes.
- VLAN10 carries Internet traffic coming from the first ISP.
- VLAN20 carries Internet traffic coming from the second ISP.
- VLAN100 carries the access network traffic.
- VLAN200 carries the core network traffic.
The final Xen configuration will provide five bridging network interfaces, one per VLAN. Each Xen domainU can freely attach to any of these bridging network interfaces in order to gain access to the traffic being carried by each VLAN.
The bridging interface,
|brname| is named after the following convention:
xenbr2is the bridging interface standing on VLAN2.
xenbr10is the bridging interface standing on VLAN10.
xenbr20is the bridging interface standing on VLAN20.
xenbr100is the bridging interface standing on VLAN100.
xenbr200is the bridging interface standing on VLAN200.
Also, Xen creates an manages several virtual network interfaces, named in the form of
|X| equals the Xen domain numeric ID and
|Y| is a sequential interface index. Thus, starting up a Xen domainU given the following virtual network interface definition:
vif = [ 'mac=00:16:3e:00:00:44, bridge=xenbr10', 'mac=00:16:e3:00:00:45, bridge=xenbr20' ]
Will cause the Xen domain to get assigned, let’s say, a domain ID of 2, and two virtual network interfaces named
vif2.0 — attached to
xenbr10 — and
vif2.1 — attached to
Setting up the bridging interfaces:
This can be done manually, by invoking
brctl addbr |brname| in order to create a new bridging interface.
For example, the following commands will create five bridging interfaces, one for each supported VLAN:
brctl addbr xenbr2 brctl addbr xenbr10 brctl addbr xenbr20 brctl addbr xenbr100 brctl addbr xenbr200
or else can be automated to get done during system startup, by creating a file named
|brname| is the name assigned to the bridging interface, like
/etc/sysconfig/network-scripts/ifcfg-xenbr2 (the configuration file for the bridging interface standing on VLAN2):
DEVICE=xenbr2 BOOTPROTO=static IPADDR=192.168.0.10 NETMASK=255.255.0.0 ONBOOT=yes TYPE=Bridge
Setting up the VLAN interfaces and add them up to the existing bridging interfaces:
This can be done manually, by invoking
vconfig add |ifname| |vlan| to configure VLAN number
|vlan| by using 802.1q tagging on interface
|ifname|. This will active a virtual interface named
- Any traffic sent to this interface will get tagged for VLAN
- Any traffic received from interface
|ifname|carrying an 802.1q VLAN tag matching
|vlan|will be untagged and received by this interface.
vconfig add eth0 2 vconfig add eth0 10 vconfig add eth0 20 vconfig add eth0 100 vconfig add eth0 200
This will add five new VLAN interfaces, one for every supported VLAN.
Once the VLAN interfaces are ready, we add them to their corresponding bridging interfaces by using
brctl addif |brname| |ifname|.|vlan|:
brctl addif xenbr2 eth0.2 brctl addif xenbr10 eth0.10 brctl addif xenbr20 eth0.20 brctl addif xenbr100 eth0.100 brctl addif xenbr200 eth0.200
The process of adding up a new VLAN interface and then adding it up to an existing bridging interface can be configured using a single configuration file named
DEVICE=eth0.2 BOOTPROTO=none ONBOOT=yes TYPE=Ethernet VLAN=yes BRIDGE=xenbr2
Keeping Xen from reconfiguring the network:
Since we have already configured the network manually, we don’t want Xen to mess up with the configuration. In order to keep Xen from reconfiguring the network, simply make sure none of the following lines appear uncommented in the file
(network-script network-bridge) (network-script network-route) (network-script network-nat)
I have been experiencing a very strange behavior on Xen domainU guests while using this network configuration: it seems that UDP traffic gets stuck at the network stack and does not flow through unless I load the
ip_conntrack.ko kernel module.
Failing to load the
ip_conntrack.ko kernel module, even with an unconfigured, empty firewall, allows ICMP and TCP traffic to flow from and to the guest network stack, but UDP traffic, like DNS queries, gets stuck and doesn’t even touch the physical network interface.
This is really strange, isn’t it?
23 thoughts on “Xen network configuration and multiple VLANs”
i’ve been trying to set up this exact scenario but have not been successful. everything works fine in the dom0:
# vconfig add eth0 560
# ifconfig eth0.560 10.10.10.182/30
root@xen-server-1 ~]# ping -c 1 10.10.10.181
64 bytes from 10.10.10.181: icmp_seq=1 ttl=255 time=0.493 ms
note: 10.10.10.181 is only reachable via vlan 560
but if i follow your steps:
# brctl addbr xenbr560
# vconfig add eth0 560
# brctl addif xenbr560 eth0.560
then boot my domU with vif = [‘bridge=xenbr560’] in the config:
[root@xen-server-1 ~]# brctl show
bridge name bridge id STP enabled interfaces
xenbr560 8000.000d566ffea8 no eth0.560
eth0 in the domU is set to 10.10.10.182/30, but it cannot ping 10.10.10.181.
am i mising something?
Don’t know exactly what’s going on, but I’ve used the exact configuration described in the article… However, I had serious problems making it work correctly (read the notes at the end of the article wrt. UDP and some other traffic).
In the end, I left the networking configuration of dom0 alone and, instead, did deploy VLAN subinterfaces inside the virtual machines. For example, I leave the networking configuration as it comes by default (i.e. bridge mode where all domU’s VNIC’s attach to the virtual switch
xen-br0). Then, if a domU guest needs to attach to both VLAN ID 3 and VLAN ID 500, I have to manually configure two VLAN subinterfaces inside that domU using
vconfigor the networking scripts.
This has proven to work much better, although this doesn’t isolate the domU machines from VLAN changes. That is, if I decide to move a domU from a VLAN to another, I have to reconfigure the domU machine.
hmm, i’m pretty sure i tried that as well and still couldn’t get it working. but thank you very much for the tip (as well as for the initial writeup), i’ll give it another try.
Currently, I don’t have access to the Xen machine where all this is running. However, as soon as I get access back, I’ll post another article on how I did manage to get it working.
w00t, i do believe i have it working. you’re right, leaving the dom0 stuff totally alone and setting things inside the domU did the trick. thanks a ton, drop me a line if you’re ever in raleigh, nc 🙂
Glad to know you got it working 🙂
I’ve tried to setup vlan with xen and not works 😦
Xen server runs debian sarge 3.1 + 22.214.171.124 + xen 3.0.2
when I try todo a simple ping of a VM inside vlan I loose packet or sometimes not answer.
It’s very strange.
Any ideas ?
br200 8000.00304877ef62 no vif2.0
br300 8000.00304877ef62 no vif5.0
br400 8000.00304877ef62 no vif3.0
br500 8000.00304877ef62 no vif7.0
br700 8000.00304877ef62 no vif4.0
Yeah! At the end of the post I comment on I had similar problems: some UDP datagrams were getting lost and, at the end, I revert to using the old-style, default networking scripts and stuff from Xen. Instead of doing what I describe in this post, I end up defining the VLANs in the domU guests, which works reliably.
I hope to write a short addendum to this article very soon on how to configure Xen for VLAN support, in dom0 and domU’s, using a different way which works perfectly.
Very good initiative 🙂
Please let me a mail when you addendum is ready …
I need to find a very reliable solution to my config asap.
Moreover Have you test vlan over bonding ?
It is very expressive and educative in the implementation of VLAN on Xen setup
Hey very good this, i used xen but with debian, i need this.. you speak spanish? you can helpme ? thanks
Sí, hablo español 🙂
¿En qué puedo ayudarte?
Jeje, bueno estoy virtualizando con xen un servidor. cual tiene dns, imap,postfix bla bla, en cuatro maquinas virtuales. EL problema es que yo quiero que esas maquinas virtuales no se vean entre si es decir desde afuera.. por eso pensaba en hacer un bridge para cada maquina virtual o crear un vlan y hacer lo mismo con los bridge.. no se que me recomiendas tu y se me podrias guiar un poco gracias
> |VM1 |—————-bridge1——————-| |
> | |
> | |—–BALBLA
> |VM2 |
> | VM4 |
> |__________ |—————–bridge2—————-|Firewall |
> ____________ |
> |VM3 |
> | |
> |__________ |—————–bridge3—————— |________|
my setup is quite the same, except I’m using SLES10. I too I’m experiencing some packet loss: this is particularly visible for UDP traffic. Around 4% of bytes transferred get lost (in UDP). I’ve a thread on xen-users ML, if you have comments.
Experimenting with VMware ESX V3 give me — for udp traffic — a 0.8% loss for bytes transferred.
XenSource will release XenEnterprise 3.2 (still in beta) at the end of March. They will provide Vlan management, but still don’t know if this problem will be present.
As for the other solution you suggest (Vlan’s inside the guest) I think it’s less secure and so less acceptable.
como puedo configurar el eth1 ??
no he podido configurarla… sabes como se hace?
Thanks for this, now xen networking and vlan config makes sense.
In debian I figured out this simply way to manage VLANs in debian:
Anyway, in xend-config.sxp you need to specifify “network-dummy”, otherwise default “network-bridge” will be used.
Sorry, broken link. This is an enhanced mini-howto on xen and vlan in Debian/Ubuntu.
Thanks for posting are there feeds to your blog? I’d like to save them
This article is really very useful. I admire the moment you invested writing this write-up. I would like to become a nurse, do you have any strategy where I could get details about nurse pays, career descriptions and what is the basis. Any option where I could get this data?
Why did you join the blogging challenge? Because Mrs. Rauser made us.
An unputdownable discussion is designer account. I opine that you should indite solon on this issue, it strength not be a prejudice study but mostly fill are not sufficiency to mouth on specified topics. To the next. Cheers like your Anti Wrinkle Face Cream A Surprising New Discovery Aloebeauty Designerwear.
Hello there! Quick question that’s totally off topic.
Do you know how to make your site mobile friendly? My website looks weird when browsing from my
iphone 4. I’m trying to find a template or plugin that might be able to correct this problem.
If you have any suggestions, please share. Thanks!