Re: [libvirt] how do I stop libvirt futzing with my network configuration?
----- "Nix" <nix@esperi.org.uk> wrote:
From: "Nix" <nix@esperi.org.uk> To: "Daniel P. Berrange" <berrange@redhat.com> Cc: libvir-list@redhat.com Sent: Saturday, November 28, 2009 4:10:28 PM GMT -05:00 US/Canada Eastern Subject: Re: [libvirt] how do I stop libvirt futzing with my network configuration?
On 26 Nov 2009, Daniel P. Berrange spake thusly:
On Thu, Nov 26, 2009 at 06:25:07PM +0000, Nix wrote:
However, there appears to be no way to say 'this is what the network is already like'. That network is considered 'inactive' and can't be used by any guests, and if I try to make it active, I get this:
virsh # net-start default error: Failed to start network default error: cannot create bridge 'vm-net': File exists
Of course it bloody can't create that bridge: it's already there, has an IP address on the host, and has the host routing packets to it. There appears to be no option to allow libvirt to assign IPs on the host...
... should I fix that, 'net-start' tries to update iptables rules! How should I put this: I do not *not not* want libvirt pissing with the firewall in any way at all. If I want firewall rules, I'll create them. But there's no way to tell it 'hands off! This network is already active, don't try to *make* it active!'
If you don't want libvirt to create the bridge + setup IPtables rules then don't use the net-XXX commands / XML. That functionality is not there for pointing libvirt to existing bridge devices.
If you already have a bridge configured, then just point the guest directly at that bridge by name.
OK, I still can't make this work: it worked briefly but then stopped. As far as I can tell tools like virt-manager are unwilling to *let* you connect to a network considered 'inactive', and networks are only considered active if they have a configuration file under /var/run/libvirt/network. These files are only created if libvirt has created the bridge itself as well. If no networks are considerd active, virt-manager won't let you create a guest at all: it insists on trying to start the sodding network, and when that fails doesn't let you get any further.
I've been running with this configuration for many months on dozens of hosts. - Created a bridge (the old fashion way) in /etc/sysconfig/network-scripts Bridge called br0 with one device eth1. - Created a VM in virt manager (or edit existing) Picked "Shared Physical Device" Device "eth1 (Bridge br0)" in the GUI. Or just add it to the VMs XML <interface type='bridge'> <mac address='52:54:00:4f:0a:76'/> <source bridge='br0'/> </interface> Works like a charm and there's certainly no configuration in libvirt for this interface, ie. nothing in /var/run/libvirt/network, and *nothing* set up in Virt Manager under "Host Details->Virtual networks You do need to make sure that you disable netfilter on the bridge or setup the appropriate iptables rules ( see http://wiki.libvirt.org/page/Networking#Bridged_networking_.28aka_.22shared_...)
So as far as I can tell, if you don't want libvirt creating all your bridges for you, you may as well give up hope of using virt-manager, or start hacking all this stuff out of the source.
I hoped I could use libvirt in conjunction with raw qemu. So much for that, it seems :( it *really* wants to take over the world...
(aside: ideally I should not have to spend half an hour crawling around the source to figure this out. The only other program I've ever seen that was this hard to set up was Oracle! Whole *Linux distros* take less work than this. I have half a dozen patches I'll send your way, but I wasn't going to send any of them until I'd actually managed to get a VM working. I got one up last night, somehow -- I no longer have any idea how, obviously one network had somehow got marked active -- whereupon KVM fell over. *sigh*)
-- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On 28 Nov 2009, Ian Woodstock spake thusly:
I've been running with this configuration for many months on dozens of hosts.
- Created a bridge (the old fashion way) in /etc/sysconfig/network-scripts Bridge called br0 with one device eth1.
- Created a VM in virt manager (or edit existing) Picked "Shared Physical Device" Device "eth1 (Bridge br0)" in the GUI.
That's the mystery. I did it (directly via brctl, as it happens, 'cos I'm making several with particular properties and want to enforce them): spindle:/etc/libvirt/qemu# brctl show bridge name bridge id STP enabled interfaces linux-net 8000.06eb4e4985df no dummy0 [...] 52: linux-net: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether 06:eb:4e:49:85:df brd ff:ff:ff:ff:ff:ff inet 192.168.20.1/32 scope global linux-net (hm, the state UNKNOWN is sort of bizarre. It's up...) (dummy0 is a member of this bridge because I've previously noted that a bridge with no members at all isn't picked up by the GUI). It appears in the GUI, all right: as 'host device linux-net (not bridged)', greyed out and unselectable. Calling a bridge 'not bridged' is more than slightly bizarre. Perhaps the netcf thing means I'm *required* to stick stuff in /etc/sysconfig/network-scripts even though I'm not using RH so it would never otherwise be used? From my reading of the source, that's not so... I suppose I should just have it check for the bridge and rip out all this other stuff, but it seems bizarre that I have to do it.
Or just add it to the VMs XML
That's too late to run the installer if it needs to access the network :/ I suppose I could continue doing the VM installation qemu run by hand...
Works like a charm and there's certainly no configuration in libvirt for this interface, ie. nothing in /var/run/libvirt/network, and *nothing* set up in Virt Manager under "Host Details->Virtual networks
You do need to make sure that you disable netfilter on the bridge or setup the appropriate iptables rules ( see http://wiki.libvirt.org/page/Networking#Bridged_networking_.28aka_.22shared_...)
There's no iptables at all on this particular box (at least not yet, although it may turn up later on when I put Windows guests on here: I'm not having *them* running around free).
On Sat, Nov 28, 2009 at 5:06 PM, Nix <nix@esperi.org.uk> wrote:
On 28 Nov 2009, Ian Woodstock spake thusly:
I've been running with this configuration for many months on dozens of hosts.
- Created a bridge (the old fashion way) in /etc/sysconfig/network-scripts Bridge called br0 with one device eth1.
- Created a VM in virt manager (or edit existing) Picked "Shared Physical Device" Device "eth1 (Bridge br0)" in the GUI.
That's the mystery. I did it (directly via brctl, as it happens, 'cos I'm making several with particular properties and want to enforce them):
spindle:/etc/libvirt/qemu# brctl show bridge name bridge id STP enabled interfaces linux-net 8000.06eb4e4985df no dummy0 [...]
52: linux-net: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether 06:eb:4e:49:85:df brd ff:ff:ff:ff:ff:ff inet 192.168.20.1/32 scope global linux-net
(hm, the state UNKNOWN is sort of bizarre. It's up...)
It's unknown because you're using the dummy device (which will just send all bits to a deep dark hole) and we can't do things like check the status of the interface/link etc. I suspect that's why libvirt won't let you connect to it, since libvirt is looking for a "shared physical device" and there's not a device in the bridge.
(dummy0 is a member of this bridge because I've previously noted that a bridge with no members at all isn't picked up by the GUI).
It appears in the GUI, all right: as 'host device linux-net (not bridged)', greyed out and unselectable. Calling a bridge 'not bridged' is more than slightly bizarre.
Actually I think this is correct. It's not bridged to a physical device, it's plumbed to nothing.
Perhaps the netcf thing means I'm *required* to stick stuff in /etc/sysconfig/network-scripts even though I'm not using RH so it would never otherwise be used? From my reading of the source, that's not so...
No, it's not required, it just must be up some how (brctl works fine)
I suppose I should just have it check for the bridge and rip out all this other stuff, but it seems bizarre that I have to do it.
Or just add it to the VMs XML
That's too late to run the installer if it needs to access the network :/ I suppose I could continue doing the VM installation qemu run by hand...
Works like a charm and there's certainly no configuration in libvirt for this interface, ie. nothing in /var/run/libvirt/network, and *nothing* set up in Virt Manager under "Host Details->Virtual networks
You do need to make sure that you disable netfilter on the bridge or setup the appropriate iptables rules ( see http://wiki.libvirt.org/page/Networking#Bridged_networking_.28aka_.22shared_...)
There's no iptables at all on this particular box (at least not yet, although it may turn up later on when I put Windows guests on here: I'm not having *them* running around free).
So it sounds like the root of your issue now is that you're using dummy network device. Is that being done temporarily now because you don't have a network plumbed in or is there some other use case?
On 28 Nov 2009, Ian Woodstock verbalised:
On Sat, Nov 28, 2009 at 5:06 PM, Nix <nix@esperi.org.uk> wrote:
(hm, the state UNKNOWN is sort of bizarre. It's up...)
It's unknown because you're using the dummy device (which will just send all bits to a deep dark hole) and we can't do things like check the status of the interface/link etc.
I just tried that by removing everything from the bridge. Still state UNKNOWN.
I suspect that's why libvirt won't let you connect to it, since libvirt is looking for a "shared physical device" and there's not a device in the bridge.
Gah. So libvirt won't let me connect a bunch of devices to a bridge without that bridge being bridged to something already? So you can't have a pile of bridges with VMs on them *routed* to the rest of the net?
It appears in the GUI, all right: as 'host device linux-net (not bridged)', greyed out and unselectable. Calling a bridge 'not bridged' is more than slightly bizarre.
Actually I think this is correct. It's not bridged to a physical device, it's plumbed to nothing.
I don't want it bridged to a physical device. Why should libvirt require any such thing? It's not necessary to get packets out of it: all you need for *that* is a routing table entry. (The bridge has an IP address on the host and everything.)
There's no iptables at all on this particular box (at least not yet, although it may turn up later on when I put Windows guests on here: I'm not having *them* running around free).
So it sounds like the root of your issue now is that you're using dummy network device. Is that being done temporarily now because you don't have a network plumbed in or is there some other use case?
It was an emergency hack when I found virt-manager not noticing bridges that had nothing on them (it said 'not bridged'). I stuck the dummy device on it and it started working. However, this appears to have been transient. ... In the code, the only place where it checks if a bridge exists is in src/network/bridge_driver.c:networkFindActiveConfigs(), and it only bothers to check *that* if there's a config file in the NETWORK_STATE_DIR (/var/lib/libvirt/network): ,---- | for (i = 0 ; i < driver->networks.count ; i++) { | virNetworkObjPtr obj = driver->networks.objs[i]; | virNetworkDefPtr tmp; | char *config; | | virNetworkObjLock(obj); | | if ((config = virNetworkConfigFile(NULL, | NETWORK_STATE_DIR, | obj->def->name)) == NULL) { | virNetworkObjUnlock(obj); | continue; | } | | if (access(config, R_OK) < 0) { | VIR_FREE(config); | virNetworkObjUnlock(obj); | continue; | } | | /* Try and load the live config */ | tmp = virNetworkDefParseFile(NULL, config); | VIR_FREE(config); | if (tmp) { | obj->newDef = obj->def; | obj->def = tmp; | } | | /* If bridge exists, then mark it active */ | if (obj->def->bridge && | brHasBridge(driver->brctl, obj->def->bridge) == 0) { | obj->active = 1; `---- So, no, I don't see how this can possibly work without a config file, and you only get a config file by creating the bridge through libvirt. I wonder if any of the Dans can tell us what's going on? (It's not surprising I can't figure it out. My first name is wrong. ;) )
On Sat, Nov 28, 2009 at 10:41:58PM +0000, Nix wrote:
On 28 Nov 2009, Ian Woodstock verbalised:
I suspect that's why libvirt won't let you connect to it, since libvirt is looking for a "shared physical device" and there's not a device in the bridge.
Gah. So libvirt won't let me connect a bunch of devices to a bridge without that bridge being bridged to something already? So you can't have a pile of bridges with VMs on them *routed* to the rest of the net?
Please distinguish between libvirt & virt-manager. libvirt allows this find - it is happy to connect VMs to any bridge device. virt-manager is providing a higher level UI which only presents a smaller subset of the functionality. In this case is only supporting the libvirt managed NAT mode, and bridging of a physical inteface. An alternative to using virt-manager is to install your VMs using the command line 'virt-install' tool which offers a much greater level of functionality - in this case you'd be able to simply add --network bridge:linux-net to the args when installing a VM to make it use your bridge. The virt-install manpage has lots of examples of all the other args.
There's no iptables at all on this particular box (at least not yet, although it may turn up later on when I put Windows guests on here: I'm not having *them* running around free).
So it sounds like the root of your issue now is that you're using dummy network device. Is that being done temporarily now because you don't have a network plumbed in or is there some other use case?
It was an emergency hack when I found virt-manager not noticing bridges that had nothing on them (it said 'not bridged'). I stuck the dummy device on it and it started working. However, this appears to have been transient.
... In the code, the only place where it checks if a bridge exists is in src/network/bridge_driver.c:networkFindActiveConfigs(), and it only bothers to check *that* if there's a config file in the NETWORK_STATE_DIR (/var/lib/libvirt/network):
Again the network driver here is libvirt's NAT based networking, not regular physical device bridging. Yes, its a bad naming convention :-) The physical interface management APIs are provided by the 'interface' driver (ie the virsh iface-XXX commands). Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Sat, Nov 28, 2009 at 10:06:16PM +0000, Nix wrote:
On 28 Nov 2009, Ian Woodstock spake thusly:
I've been running with this configuration for many months on dozens of hosts.
- Created a bridge (the old fashion way) in /etc/sysconfig/network-scripts Bridge called br0 with one device eth1.
- Created a VM in virt manager (or edit existing) Picked "Shared Physical Device" Device "eth1 (Bridge br0)" in the GUI.
That's the mystery. I did it (directly via brctl, as it happens, 'cos I'm making several with particular properties and want to enforce them):
spindle:/etc/libvirt/qemu# brctl show bridge name bridge id STP enabled interfaces linux-net 8000.06eb4e4985df no dummy0 [...]
52: linux-net: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether 06:eb:4e:49:85:df brd ff:ff:ff:ff:ff:ff inet 192.168.20.1/32 scope global linux-net
(hm, the state UNKNOWN is sort of bizarre. It's up...)
(dummy0 is a member of this bridge because I've previously noted that a bridge with no members at all isn't picked up by the GUI).
It appears in the GUI, all right: as 'host device linux-net (not bridged)', greyed out and unselectable. Calling a bridge 'not bridged' is more than slightly bizarre.
What's confusing it here is that you've not connected a real physical network interface to the bridge - it is expecting to find a ethernet device enslaved, and instead finds 'dummy0'. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On 30 Nov 2009, Daniel P. Berrange stated:
What's confusing it here is that you've not connected a real physical network interface to the bridge - it is expecting to find a ethernet device enslaved, and instead finds 'dummy0'.
I'm not bothering with dummy0 anymore: my bridges are empty save for VM network interfaces (connected to the real network via ordinary routing). Let's see if this works with the virt-manager/libvirt fix Cole mentioned...
participants (3)
-
Daniel P. Berrange -
Ian Woodstock -
Nix