[Libvir] Inconsistent networking between Xen & QEMU
In Xen it is possible to configure a VIF to enslave with an arbitrary bridge device using a fragment of XML like: <interface type='bridge'> <source bridge='xenbr0'/> <mac address='00:16:3E:66:94:9C'/> </interface> In this example I've enslaved to xenbr0 which is created by the Xen networking script, but this will work with any bridge device really. In QEMU the only bridging is available via the 'virtual networks' support in libvirt, by referencing a named bridge. For example <interface type='network'> <source network='default'/> <mac address='00:16:3E:66:94:9C'/> </interface> Now 'default' is merely defined to be backed by a bridge device 'virbr0' (although it does not neccessary get backed by a bridge) so the basic impl is basically just enslaving a TAP device. Anyway, we now have two different syntaxes in the XML for defining what is more or less the same thing which is a real PITA for the tools. It is useful to have the distinct representations - one is for 'managed' bridges, and the other is for 'unmanaged' bridges - we can do some much more interesting things with managed bridges. To improve life for the tools though I'd like to do two things as a high priority - Support managed networks in Xen - we can simply lookup the bridge device associated with a network and tell Xen to use vif-bridge with this device. For the reverse Xen will tell us what bridge device a guest is using, and we can reverse lookup the corresponding network - Support non-managed networks in QEMU - we can simply enslave the QEMU tap device to an arbitrary user specific bridge device. Until one of these two options is done, its going to be a PITA to support networking across QEMU & Xen in virt-manager/virt-install. Regards, Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
On Fri, 2007-03-09 at 03:03 +0000, Daniel P. Berrange wrote:
Anyway, we now have two different syntaxes in the XML for defining what is more or less the same thing which is a real PITA for the tools. It is useful to have the distinct representations - one is for 'managed' bridges, and the other is for 'unmanaged' bridges - we can do some much more interesting things with managed bridges.
As I said in another mail, the way I'd imagined virt-install working would be that by default it would connect guest NICs to the default network (i.e. not the bridge associated with the default network), but you could still use --bridge to explicitly connect the guest to a bridge. And we'd have a --network option.
To improve life for the tools though I'd like to do two things as a high priority
- Support managed networks in Xen - we can simply lookup the bridge device associated with a network and tell Xen to use vif-bridge with this device.
We already have this in libvirt. See virDomainParseXMLIfDesc()
For the reverse Xen will tell us what bridge device a guest is using, and we can reverse lookup the corresponding network
I'm not sure why you want this?
- Support non-managed networks in QEMU - we can simply enslave the QEMU tap device to an arbitrary user specific bridge device.
Yes, we should definitely have this. Cheers, Mark.
On Fri, Mar 09, 2007 at 02:27:02PM +0000, Mark McLoughlin wrote:
On Fri, 2007-03-09 at 03:03 +0000, Daniel P. Berrange wrote:
To improve life for the tools though I'd like to do two things as a high priority
- Support managed networks in Xen - we can simply lookup the bridge device associated with a network and tell Xen to use vif-bridge with this device.
We already have this in libvirt. See virDomainParseXMLIfDesc()
Ahh I missed that. I will have to write some tests for that I think, although it'll require extending the test:///default driver to support the networking APIs :-)
For the reverse Xen will tell us what bridge device a guest is using, and we can reverse lookup the corresponding network
I'm not sure why you want this?
Without this you be sending in <interface type='network'> <source network='foo'> </interface> And be getting back <interface type='bridge'> <source bridge='virbr0'> </interface> As far as is possible the XML used to create a VM should be identical to that you get back from libvirt once it is running.
- Support non-managed networks in QEMU - we can simply enslave the QEMU tap device to an arbitrary user specific bridge device.
Yes, we should definitely have this.
Ok, I'll add that - it should be trivial since the tricky enslaving code is already there. Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
On Fri, 2007-03-09 at 14:32 +0000, Daniel P. Berrange wrote:
On Fri, Mar 09, 2007 at 02:27:02PM +0000, Mark McLoughlin wrote:
On Fri, 2007-03-09 at 03:03 +0000, Daniel P. Berrange wrote:
For the reverse Xen will tell us what bridge device a guest is using, and we can reverse lookup the corresponding network
I'm not sure why you want this?
Without this you be sending in
<interface type='network'> <source network='foo'> </interface>
And be getting back
<interface type='bridge'> <source bridge='virbr0'> </interface>
As far as is possible the XML used to create a VM should be identical to that you get back from libvirt once it is running.
Absolutely, very good point. There's a few places where we return the current running state rather than the initial configuration we provided, but this would be a particularly bad case. It's a bit fugly, though - should libvirt do ListDomains(), GetBridgName() and match the bridge name, or should we have virNetworkLookupByBridge() and make qemud do the lookup itself? Cheers, Mark.
On Fri, Mar 09, 2007 at 04:20:58PM +0000, Mark McLoughlin wrote:
On Fri, 2007-03-09 at 14:32 +0000, Daniel P. Berrange wrote:
On Fri, Mar 09, 2007 at 02:27:02PM +0000, Mark McLoughlin wrote:
On Fri, 2007-03-09 at 03:03 +0000, Daniel P. Berrange wrote:
For the reverse Xen will tell us what bridge device a guest is using, and we can reverse lookup the corresponding network
I'm not sure why you want this?
Without this you be sending in
<interface type='network'> <source network='foo'> </interface>
And be getting back
<interface type='bridge'> <source bridge='virbr0'> </interface>
As far as is possible the XML used to create a VM should be identical to that you get back from libvirt once it is running.
Absolutely, very good point.
There's a few places where we return the current running state rather than the initial configuration we provided, but this would be a particularly bad case.
It's a bit fugly, though - should libvirt do ListDomains(), GetBridgName() and match the bridge name, or should we have virNetworkLookupByBridge() and make qemud do the lookup itself?
I was actually thinking virNetworkLookupByBridge could be a very handy API to have Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
On Fri, Mar 09, 2007 at 02:27:02PM +0000, Mark McLoughlin wrote:
On Fri, 2007-03-09 at 03:03 +0000, Daniel P. Berrange wrote:
Anyway, we now have two different syntaxes in the XML for defining what is more or less the same thing which is a real PITA for the tools. It is useful to have the distinct representations - one is for 'managed' bridges, and the other is for 'unmanaged' bridges - we can do some much more interesting things with managed bridges.
As I said in another mail, the way I'd imagined virt-install working would be that by default it would connect guest NICs to the default network (i.e. not the bridge associated with the default network), but you could still use --bridge to explicitly connect the guest to a bridge. And we'd have a --network option.
I think we need to explore this a little further - the APIs for virtual networks are very nice froma tools POV because we can trivially enumerate networks & connect VMs to them. Its not pleasent to have to enumerate the regular non-virtual network devices using a different set of APIs - indeed its impossible if we're talking to a remote libvirt. So I think we need to look at also exposing the 'shared physical device' style networks via the libvirt APIs too. This could be expressed with something like: <network> <name>default</name> <bridge name="virbr0" /> <device name="eth0" /> </network> Which would setup a virbr0 bridge,and enslave eth0 to it. It would assume that the network eth0 is connected to has DHCP already, so there's no need to define an IP for the bridge itself, or run dnsmasq in this case. Is it actually possible to take an already configured eth0 device and enslave it to virbr0 without eth0 loosing its address config/routes ? If we have this kind of support could we auto-generate network configs at runtime based on any bridge/network device the adminsitrator has already manually configured via /etc/sysconfig/networking-scripts ? Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
On Fri, 2007-03-09 at 14:40 +0000, Daniel P. Berrange wrote:
On Fri, Mar 09, 2007 at 02:27:02PM +0000, Mark McLoughlin wrote:
On Fri, 2007-03-09 at 03:03 +0000, Daniel P. Berrange wrote:
Anyway, we now have two different syntaxes in the XML for defining what is more or less the same thing which is a real PITA for the tools. It is useful to have the distinct representations - one is for 'managed' bridges, and the other is for 'unmanaged' bridges - we can do some much more interesting things with managed bridges.
As I said in another mail, the way I'd imagined virt-install working would be that by default it would connect guest NICs to the default network (i.e. not the bridge associated with the default network), but you could still use --bridge to explicitly connect the guest to a bridge. And we'd have a --network option.
I think we need to explore this a little further - the APIs for virtual networks are very nice froma tools POV because we can trivially enumerate networks & connect VMs to them. Its not pleasent to have to enumerate the regular non-virtual network devices using a different set of APIs - indeed its impossible if we're talking to a remote libvirt. So I think we need to look at also exposing the 'shared physical device' style networks via the libvirt APIs too.
Took me a second to understand you, but what I'm saying implies a dialog like: --- Add Virtual Network Interface: Mac Address : [ ] Connect to: * Network : [ Default ] * Physical Interface : [ eth0 ] --- And your point is that in order to populate the latter dropdown list, you need to use a sucky API which doesn't work in the remote case. Rather you suggest that if people want to use bridging, then they should modify the default network XML config by hand and not have the latter option in the UI? I dunno, the concept of "connect the guest to a physical network" should be in the UI IMHO, and if you do then you need a list of physical interfaces ...
This could be expressed with something like:
<network> <name>default</name> <bridge name="virbr0" /> <device name="eth0" /> </network>
Which would setup a virbr0 bridge,and enslave eth0 to it. It would assume that the network eth0 is connected to has DHCP already, so there's no need to define an IP for the bridge itself, or run dnsmasq in this case.
Yeah, it's like taking your virtual netgear router, turning off IP forwarding and DHCP and enabling bridging. That introduces this user visible notion of a bridge vs. a router, which is just horrible. But, I guess you're saying you wouldn't have it in the UI.
Is it actually possible to take an already configured eth0 device and enslave it to virbr0 without eth0 loosing its address config/routes ?
Well, where would the address from eth0 go if you had an ip address configured for the network already? It would have to be an either or thing then ... Also, what if you try and bridge multiple networks out over the physical interface? That's why I'm suggesting that if you want to bridge guests over a physical interface, then you need to change the interfaces configuration so that it's "shared" ... which in effect, makes it a bridge which you can add guest's backend interfaces too. ... I guess it all comes down to two options: a) Expose, in the UI, the concept of "either connect the guest to a virtual network or to a shared physical interface" or b) Make it an admin decision, not available in the UI, whether the default network is a private network or bridged to a physical interface I prefer a), but I can see the argument for b).
If we have this kind of support could we auto-generate network configs at runtime based on any bridge/network device the adminsitrator has already manually configured via /etc/sysconfig/networking-scripts ?
Not really following ... Cheers, Mark.
On Fri, Mar 09, 2007 at 04:44:17PM +0000, Mark McLoughlin wrote:
On Fri, 2007-03-09 at 14:40 +0000, Daniel P. Berrange wrote:
On Fri, Mar 09, 2007 at 02:27:02PM +0000, Mark McLoughlin wrote:
As I said in another mail, the way I'd imagined virt-install working would be that by default it would connect guest NICs to the default network (i.e. not the bridge associated with the default network), but you could still use --bridge to explicitly connect the guest to a bridge. And we'd have a --network option.
I think we need to explore this a little further - the APIs for virtual networks are very nice froma tools POV because we can trivially enumerate networks & connect VMs to them. Its not pleasent to have to enumerate the regular non-virtual network devices using a different set of APIs - indeed its impossible if we're talking to a remote libvirt. So I think we need to look at also exposing the 'shared physical device' style networks via the libvirt APIs too.
Took me a second to understand you, but what I'm saying implies a dialog like:
--- Add Virtual Network Interface:
Mac Address : [ ]
Connect to:
* Network : [ Default ]
* Physical Interface : [ eth0 ] ---
And your point is that in order to populate the latter dropdown list, you need to use a sucky API which doesn't work in the remote case.
Pretty much, yes.
Rather you suggest that if people want to use bridging, then they should modify the default network XML config by hand and not have the latter option in the UI?
How they configure the network XML is a completely separate issue - we could easily have UI in virt-manager for creating/deleteing/editing networks in the same way we have UI for creating/deleting/editing domains. So while today they'd have to edit the XML to bridge to the physical network, that need only be a short term issue.
I dunno, the concept of "connect the guest to a physical network" should be in the UI IMHO, and if you do then you need a list of physical interfaces ...
So yes, we need list of physical interfaces / networks, in addition to the existing list of virtual networks. Perhaps the same libvirt API can be used to provide both listings, or maybe we need separate APIs. Even if we used a single API at the libvirt level for listing virtual & physical networks, we can have the option of displaying a dialog with the struture you outline above - just key off some attribute in the XML to determine whether to put a particular virNetworkPtr in the 'Network' drop down vs the 'physical interface' drop down (or whatever they end up being called).
This could be expressed with something like:
<network> <name>default</name> <bridge name="virbr0" /> <device name="eth0" /> </network>
Which would setup a virbr0 bridge,and enslave eth0 to it. It would assume that the network eth0 is connected to has DHCP already, so there's no need to define an IP for the bridge itself, or run dnsmasq in this case.
Yeah, it's like taking your virtual netgear router, turning off IP forwarding and DHCP and enabling bridging.
That introduces this user visible notion of a bridge vs. a router, which is just horrible. But, I guess you're saying you wouldn't have it in the UI.
Just because the different bridge vs routed confoigs are represented in the libvirt XML one way, doesn't force our hand in our we present it to the user. Any network with a '<device name='eth0'> tag could be displayed in the 'physical interfaces' drop down, while any without that tag would be in the 'Network' drop down. My concerns at this stage are really about the being able to use a consistent API in libvirt for everything we ever want to do with networking, and not ever have to call out to non-libvirt APIs ourselves.
Is it actually possible to take an already configured eth0 device and enslave it to virbr0 without eth0 loosing its address config/routes ?
Well, where would the address from eth0 go if you had an ip address configured for the network already? It would have to be an either or thing then ...
In my example I'm not configuring any address for the network - after initial enslaving, the only device which would have an address would be eth0 which preserved its address config from before it was enslaved. If that's possible - i've not actually tested this theory ? I know Xen has horrible manual address copying, but I figure that is only a result of the need to add in netloop.
Also, what if you try and bridge multiple networks out over the physical interface?
That'd be impossible because eth0 can only be in one bridge at once - it only really makes sense to bridge one network into any physical device in any case. Semantically bridging multiple networks is just the same has having one larger network, so there'd be no point in trying to bridge multiple networks into one physical device.
That's why I'm suggesting that if you want to bridge guests over a physical interface, then you need to change the interfaces configuration so that it's "shared" ... which in effect, makes it a bridge which you can add guest's backend interfaces too.
...
I guess it all comes down to two options:
a) Expose, in the UI, the concept of "either connect the guest to a virtual network or to a shared physical interface" or
b) Make it an admin decision, not available in the UI, whether the default network is a private network or bridged to a physical interface
I prefer a), but I can see the argument for b).
If we have this kind of support could we auto-generate network configs at runtime based on any bridge/network device the adminsitrator has already manually configured via /etc/sysconfig/networking-scripts ?
Not really following ...
ie, if the user has create /etc/sysconfig/networking-scripts/ifcrg-br5 for whatever reason, then we can automatically instantiate a network at the libvirt level to represent this externally created network. Semantically an admin creating arbitrary bridge devices with their regular distro specific network init scripts is no different to libvirt createing bridge devices. In both cases you end up with a bridge device, to which you'd like to connect guests. In both cases you may or may not have a physical interface enslaved in the bridge. So if the admin has created some bridge device I'd like it to appear when doing 'virsh net-list'. Regards, Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
On Fri, 2007-03-09 at 18:52 +0000, Daniel P. Berrange wrote:
On Fri, Mar 09, 2007 at 04:44:17PM +0000, Mark McLoughlin wrote:
Rather you suggest that if people want to use bridging, then they should modify the default network XML config by hand and not have the latter option in the UI?
How they configure the network XML is a completely separate issue - we could easily have UI in virt-manager for creating/deleteing/editing networks in the same way we have UI for creating/deleting/editing domains.
... except you'd again have need an API for iterating physical network devices ...
That introduces this user visible notion of a bridge vs. a router, which is just horrible. But, I guess you're saying you wouldn't have it in the UI.
Just because the different bridge vs routed confoigs are represented in the libvirt XML one way, doesn't force our hand in our we present it to the user. Any network with a '<device name='eth0'> tag could be displayed in the 'physical interfaces' drop down, while any without that tag would be in the 'Network' drop down.
Well, except that you're suggesting "connect to physical interface eth0" should be a property of a virtual network, but I think it makes more sense for it to be the property of a virtual nic. e.g. connect your qemu guests to the default network, connect your Xen guests to the eth0 bridge. Cheers, Mark.
On Fri, Mar 09, 2007 at 07:17:04PM +0000, Mark McLoughlin wrote:
On Fri, 2007-03-09 at 18:52 +0000, Daniel P. Berrange wrote:
On Fri, Mar 09, 2007 at 04:44:17PM +0000, Mark McLoughlin wrote:
Rather you suggest that if people want to use bridging, then they should modify the default network XML config by hand and not have the latter option in the UI?
How they configure the network XML is a completely separate issue - we could easily have UI in virt-manager for creating/deleteing/editing networks in the same way we have UI for creating/deleting/editing domains.
... except you'd again have need an API for iterating physical network devices ...
You say that like its a bad thing ? Virt-manager or other apps will need to be able to enumerate physical devices somehow. So if libvirt is already providing APIs for virtual networks a VM can connect to, why not also provide APIs for physical devices (or figure out a way to let the existing virtual networks API also deal with phys devices).
That introduces this user visible notion of a bridge vs. a router, which is just horrible. But, I guess you're saying you wouldn't have it in the UI.
Just because the different bridge vs routed confoigs are represented in the libvirt XML one way, doesn't force our hand in our we present it to the user. Any network with a '<device name='eth0'> tag could be displayed in the 'physical interfaces' drop down, while any without that tag would be in the 'Network' drop down.
Well, except that you're suggesting "connect to physical interface eth0" should be a property of a virtual network, but I think it makes more sense for it to be the property of a virtual nic.
Well actually I'm suggesting 'connect to the physical network that eth0 is a part of' which doesn't feel much different to 'connect to the virtual network' NB, in all this discussion I'm talking about a scenario where the guest can be connect to eth0 via a bridge. I've not even thought about a case where you'd NAT a VIF directly into eth0 - that's probably irrelevant use case now we have proper virtual networks - i only bring it up because that is what xen's vif-route or vif-nat scripts do.
e.g. connect your qemu guests to the default network, connect your Xen guests to the eth0 bridge.
I'm just wondering whether this is making a distinction, where no real distinction exists? If you run 'ifconfig' or 'brctl show' in either of these two cases its going to look basically identical to the admin. ie, there a bridge device, with one of more NICs in it, some virtual NICs or TAPs, some physical NICs. If you run 'virsh net-list' you're only going to see one of those cases Regards, Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
On Fri, 2007-03-09 at 19:38 +0000, Daniel P. Berrange wrote:
On Fri, Mar 09, 2007 at 07:17:04PM +0000, Mark McLoughlin wrote:
On Fri, 2007-03-09 at 18:52 +0000, Daniel P. Berrange wrote:
On Fri, Mar 09, 2007 at 04:44:17PM +0000, Mark McLoughlin wrote:
Rather you suggest that if people want to use bridging, then they should modify the default network XML config by hand and not have the latter option in the UI?
How they configure the network XML is a completely separate issue - we could easily have UI in virt-manager for creating/deleteing/editing networks in the same way we have UI for creating/deleting/editing domains.
... except you'd again have need an API for iterating physical network devices ...
You say that like its a bad thing ?
Not at all, I'm just saying that making the physical interface bridging configuration a part of the virtual network description does not make the problem of listing physical interfaces go away.
e.g. connect your qemu guests to the default network, connect your Xen guests to the eth0 bridge.
I'm just wondering whether this is making a distinction, where no real distinction exists? If you run 'ifconfig' or 'brctl show' in either of these two cases its going to look basically identical to the admin. ie, there a bridge device, with one of more NICs in it, some virtual NICs or TAPs, some physical NICs. If you run 'virsh net-list' you're only going to see one of those cases
I think there's a big distinction between the concepts which we want users to understand, but not such a big distinction in how they are implemented. Cheers, Mark.
participants (2)
-
Daniel P. Berrange -
Mark McLoughlin