[libvirt] how do I stop libvirt futzing with my network configuration?
So I have an existing QEMU/KVM setup with networks connected to a bridge which is routed/forwarded to the host's other interfaces. I'm trying to switch to administering at least *some* of its VMs through libvirt (probably not all as I use QEMU snapshots extensively for some of them and libvirt doesn't support that yet, if ever). So libvirt has to be able to start guests connected to a particular host bridge, and give them appropriate IP addresses. I thought it was easy to teach libvirt about this: spindle:/etc/libvirt/qemu/networks# cat /etc/libvirt/qemu/networks/default.xml <network> <name>default</name> <uuid>ee573497-1d74-473b-a2cb-9916229209b8</uuid> <forward mode='route'/> <bridge name='vm-net' stp='on' delay='0' /> <ip address='192.168.20.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.20.1' end='192.168.20.254' /> </dhcp> </ip> </network> However, there appears to be no way to say 'this is what the network is already like'. That network is considered 'inactive' and can't be used by any guests, and if I try to make it active, I get this: virsh # net-start default error: Failed to start network default error: cannot create bridge 'vm-net': File exists Of course it bloody can't create that bridge: it's already there, has an IP address on the host, and has the host routing packets to it. There appears to be no option to allow libvirt to assign IPs on the host... ... should I fix that, 'net-start' tries to update iptables rules! How should I put this: I do not *not not* want libvirt pissing with the firewall in any way at all. If I want firewall rules, I'll create them. But there's no way to tell it 'hands off! This network is already active, don't try to *make* it active!' ... at least, there's none in the documentation, and there's none in the code that I can see. Or is there?
On Thu, Nov 26, 2009 at 06:25:07PM +0000, Nix wrote:
I thought it was easy to teach libvirt about this:
spindle:/etc/libvirt/qemu/networks# cat /etc/libvirt/qemu/networks/default.xml <network> <name>default</name> <uuid>ee573497-1d74-473b-a2cb-9916229209b8</uuid> <forward mode='route'/> <bridge name='vm-net' stp='on' delay='0' /> <ip address='192.168.20.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.20.1' end='192.168.20.254' /> </dhcp> </ip> </network>
However, there appears to be no way to say 'this is what the network is already like'. That network is considered 'inactive' and can't be used by any guests, and if I try to make it active, I get this:
virsh # net-start default error: Failed to start network default error: cannot create bridge 'vm-net': File exists
Of course it bloody can't create that bridge: it's already there, has an IP address on the host, and has the host routing packets to it. There appears to be no option to allow libvirt to assign IPs on the host...
... should I fix that, 'net-start' tries to update iptables rules! How should I put this: I do not *not not* want libvirt pissing with the firewall in any way at all. If I want firewall rules, I'll create them. But there's no way to tell it 'hands off! This network is already active, don't try to *make* it active!'
If you don't want libvirt to create the bridge + setup IPtables rules then don't use the net-XXX commands / XML. That functionality is not there for pointing libvirt to existing bridge devices. If you already have a bridge configured, then just point the guest directly at that bridge by name. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On 26 Nov 2009, Daniel P. Berrange verbalised:
If you don't want libvirt to create the bridge + setup IPtables rules then don't use the net-XXX commands / XML. That functionality is not there for pointing libvirt to existing bridge devices.
Ah. All comes clear.
If you already have a bridge configured, then just point the guest directly at that bridge by name.
That seems to work: I thought it didn't, but that was my DHCP server deciding to go catatonic on me. The only remaining obvious problem is virt-manager-related, I think: this log spam: Nov 27 00:04:38 spindle err: 00:04:38.977: error : virLibConnError:383 : this function is not supported by the hypervisor: virConnectNumOfInterfaces Nov 27 00:04:38 spindle err: 00:04:38.977: error : virLibConnError:383 : this function is not supported by the hypervisor: virConnectNumOfDefinedInterfaces Emitting this *once a second* seems a bit... extreme. Can't it try once and give up?
On Fri, Nov 27, 2009 at 12:06:13AM +0000, Nix wrote:
On 26 Nov 2009, Daniel P. Berrange verbalised:
If you don't want libvirt to create the bridge + setup IPtables rules then don't use the net-XXX commands / XML. That functionality is not there for pointing libvirt to existing bridge devices.
Ah. All comes clear.
If you already have a bridge configured, then just point the guest directly at that bridge by name.
That seems to work: I thought it didn't, but that was my DHCP server deciding to go catatonic on me.
The only remaining obvious problem is virt-manager-related, I think: this log spam:
Nov 27 00:04:38 spindle err: 00:04:38.977: error : virLibConnError:383 : this function is not supported by the hypervisor: virConnectNumOfInterfaces Nov 27 00:04:38 spindle err: 00:04:38.977: error : virLibConnError:383 : this function is not supported by the hypervisor: virConnectNumOfDefinedInterfaces
Emitting this *once a second* seems a bit... extreme. Can't it try once and give up?
libvirtd always logs all errors, so if you want these to go away, then virt-manager will have to stop issuing these APi calls. Best file a bug against virt-manager about this, or raise it on the virt-manager mailing list (virt-tools-list@redhat.com) Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On 11/26/2009 07:06 PM, Nix wrote:
The only remaining obvious problem is virt-manager-related, I think: this log spam:
Nov 27 00:04:38 spindle err: 00:04:38.977: error : virLibConnError:383 : this function is not supported by the hypervisor: virConnectNumOfInterfaces Nov 27 00:04:38 spindle err: 00:04:38.977: error : virLibConnError:383 : this function is not supported by the hypervisor: virConnectNumOfDefinedInterfaces
Emitting this *once a second* seems a bit... extreme. Can't it try once and give up?
Sounds like you are using a recent upstream virt-manager checkout? This was a bug I just fixed yesterday (the fix is actually in virtinst, so you'll want to pull there): http://hg.fedorahosted.org/hg/python-virtinst/rev/05adb0b921e2 - Cole
On 30 Nov 2009, Cole Robinson spake thusly:
Sounds like you are using a recent upstream virt-manager checkout? This was a bug I just fixed yesterday (the fix is actually in virtinst, so you'll want to pull there):
Ah, I love preemptive bugfixing. Looks fixed, thanks! :)
On 26 Nov 2009, Daniel P. Berrange spake thusly:
On Thu, Nov 26, 2009 at 06:25:07PM +0000, Nix wrote:
However, there appears to be no way to say 'this is what the network is already like'. That network is considered 'inactive' and can't be used by any guests, and if I try to make it active, I get this:
virsh # net-start default error: Failed to start network default error: cannot create bridge 'vm-net': File exists
Of course it bloody can't create that bridge: it's already there, has an IP address on the host, and has the host routing packets to it. There appears to be no option to allow libvirt to assign IPs on the host...
... should I fix that, 'net-start' tries to update iptables rules! How should I put this: I do not *not not* want libvirt pissing with the firewall in any way at all. If I want firewall rules, I'll create them. But there's no way to tell it 'hands off! This network is already active, don't try to *make* it active!'
If you don't want libvirt to create the bridge + setup IPtables rules then don't use the net-XXX commands / XML. That functionality is not there for pointing libvirt to existing bridge devices.
If you already have a bridge configured, then just point the guest directly at that bridge by name.
OK, I still can't make this work: it worked briefly but then stopped. As far as I can tell tools like virt-manager are unwilling to *let* you connect to a network considered 'inactive', and networks are only considered active if they have a configuration file under /var/run/libvirt/network. These files are only created if libvirt has created the bridge itself as well. If no networks are considerd active, virt-manager won't let you create a guest at all: it insists on trying to start the sodding network, and when that fails doesn't let you get any further. So as far as I can tell, if you don't want libvirt creating all your bridges for you, you may as well give up hope of using virt-manager, or start hacking all this stuff out of the source. I hoped I could use libvirt in conjunction with raw qemu. So much for that, it seems :( it *really* wants to take over the world... (aside: ideally I should not have to spend half an hour crawling around the source to figure this out. The only other program I've ever seen that was this hard to set up was Oracle! Whole *Linux distros* take less work than this. I have half a dozen patches I'll send your way, but I wasn't going to send any of them until I'd actually managed to get a VM working. I got one up last night, somehow -- I no longer have any idea how, obviously one network had somehow got marked active -- whereupon KVM fell over. *sigh*)
On Sat, Nov 28, 2009 at 09:10:28PM +0000, Nix wrote:
On 26 Nov 2009, Daniel P. Berrange spake thusly:
On Thu, Nov 26, 2009 at 06:25:07PM +0000, Nix wrote:
However, there appears to be no way to say 'this is what the network is already like'. That network is considered 'inactive' and can't be used by any guests, and if I try to make it active, I get this:
virsh # net-start default error: Failed to start network default error: cannot create bridge 'vm-net': File exists
Of course it bloody can't create that bridge: it's already there, has an IP address on the host, and has the host routing packets to it. There appears to be no option to allow libvirt to assign IPs on the host...
... should I fix that, 'net-start' tries to update iptables rules! How should I put this: I do not *not not* want libvirt pissing with the firewall in any way at all. If I want firewall rules, I'll create them. But there's no way to tell it 'hands off! This network is already active, don't try to *make* it active!'
If you don't want libvirt to create the bridge + setup IPtables rules then don't use the net-XXX commands / XML. That functionality is not there for pointing libvirt to existing bridge devices.
If you already have a bridge configured, then just point the guest directly at that bridge by name.
OK, I still can't make this work: it worked briefly but then stopped. As far as I can tell tools like virt-manager are unwilling to *let* you connect to a network considered 'inactive', and networks are only considered active if they have a configuration file under /var/run/libvirt/network. These files are only created if libvirt has created the bridge itself as well. If no networks are considerd active, virt-manager won't let you create a guest at all: it insists on trying to start the sodding network, and when that fails doesn't let you get any further.
These files are all related to the libvirt net-XXX commands, so not things that are relevant to what you are trying todo.
So as far as I can tell, if you don't want libvirt creating all your bridges for you, you may as well give up hope of using virt-manager, or start hacking all this stuff out of the source.
virt-manager supports two primary networking modes described here: http://wiki.libvirt.org/page/Networking The first is the NAT based networking as managed by the net-XXX commands that you don't want to use. The second is bridging of a physical ethXX device to share it with a guest Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On 30 Nov 2009, Daniel P. Berrange spake thusly:
On Sat, Nov 28, 2009 at 09:10:28PM +0000, Nix wrote:
So as far as I can tell, if you don't want libvirt creating all your bridges for you, you may as well give up hope of using virt-manager, or start hacking all this stuff out of the source.
virt-manager supports two primary networking modes described here:
http://wiki.libvirt.org/page/Networking
The first is the NAT based networking as managed by the net-XXX commands that you don't want to use. The second is bridging of a physical ethXX device to share it with a guest
So you have a choice of letting libvirt mess with your host's networking, or messing with its physical ethernet configuration yourself? You can't just use ordinary routing? Blast. :/
On 11/28/2009 04:10 PM, Nix wrote:
On 26 Nov 2009, Daniel P. Berrange spake thusly:
On Thu, Nov 26, 2009 at 06:25:07PM +0000, Nix wrote:
However, there appears to be no way to say 'this is what the network is already like'. That network is considered 'inactive' and can't be used by any guests, and if I try to make it active, I get this:
virsh # net-start default error: Failed to start network default error: cannot create bridge 'vm-net': File exists
Of course it bloody can't create that bridge: it's already there, has an IP address on the host, and has the host routing packets to it. There appears to be no option to allow libvirt to assign IPs on the host...
... should I fix that, 'net-start' tries to update iptables rules! How should I put this: I do not *not not* want libvirt pissing with the firewall in any way at all. If I want firewall rules, I'll create them. But there's no way to tell it 'hands off! This network is already active, don't try to *make* it active!'
If you don't want libvirt to create the bridge + setup IPtables rules then don't use the net-XXX commands / XML. That functionality is not there for pointing libvirt to existing bridge devices.
If you already have a bridge configured, then just point the guest directly at that bridge by name.
OK, I still can't make this work: it worked briefly but then stopped. As far as I can tell tools like virt-manager are unwilling to *let* you connect to a network considered 'inactive', and networks are only considered active if they have a configuration file under /var/run/libvirt/network. These files are only created if libvirt has created the bridge itself as well. If no networks are considerd active, virt-manager won't let you create a guest at all: it insists on trying to start the sodding network, and when that fails doesn't let you get any further.
virt-manager should be fixed here, it should offer some fallback (at least 'no networking') if it can't start the virtual network. But if you don't want the virtual network at all and it can't be started without erroring, you want to virsh net-undefine it, or delete it in virt-manager via: Edit->Host Details->Networks
So as far as I can tell, if you don't want libvirt creating all your bridges for you, you may as well give up hope of using virt-manager, or start hacking all this stuff out of the source.
If you create a manual bridge, virt-manager should see it. However, the virtinst bug I mentioned in my other email was conspiring to make this not work for you. Pull from virtinst, and your manually created bridge devices should show up in virt-manager. - Cole
--- src/qemu/qemu_conf.c | 8 +++++++- 1 files changed, 7 insertions(+), 1 deletions(-) On 30 Nov 2009, Cole Robinson uttered the following:
If you create a manual bridge, virt-manager should see it. However, the virtinst bug I mentioned in my other email was conspiring to make this not work for you. Pull from virtinst, and your manually created bridge devices should show up in virt-manager.
Pulled. Bridges still show up as 'not bridged', inactive. I'm giving up. I've spent a week of free time fighting libvirt and virt-manager and they simply don't seem to be ready for anything but the simplest of use cases: by comparison I had qemu pretty much working from the source in about twenty minutes, including reading the manuals from zero qemu knowledge. Maybe libvirt works better if you're using one of the subset of popular distros for which netcf knows how to parse and manipulate the network config (something which no serious site is going to allow in any case, because they have their own network control scripts). Maybe libvirt and virt-manager are only meant to be useful for single-user desktops and very small sites, in which case it would be nice to, y'know, *say* that somewhere, perhaps along with a big banner saying "don't bother using the code from upstream if you're not running unmodified Fedora or RHEL", so people don't waste their time trying to make it work otherwise. Maybe I overstate the case, but I don't think by much. I've looked at libvirt support in a pile of not-completely-mainstream distros now and I haven't yet found one in which networking works without manual bringup, because netcf fails to parse the distro config files (yes, I know Augeas support is possible, but not many people have written that and nobody but Debian has contributed it upstream: a shame, Augeas is *lovely*). And that then lands me in the same situation I've just been trying and failing to get out of. Whenever anything goes wrong in the virtualization layer I have to fall back to raw qemu for monitor access in any case, because for reasons incomprehensible to me libvirt actually bans me from accessing the monitor or even adding a -serial mon:telnet stanza (or *anything* user-definable) to the command line: and it insists on using VNC consoles so I can't use the SDL console's monitor either (though this is reasonable: you want to be able to stop virt-manager and leave the emulator running, which VNC makes easier). So the ability to ignore the nuts and bolts of your virtualization system's command line is effectively lost the first time anything goes wrong, and you have to keep everything working without libvirt as well. (Of course the qemu guys assume you have monitor access when trying to debug problems.) While I wish I *could* use libvirt (the ability to start VMs as a regular user with working tun/tap alone would be excellent), it's just not remotely ready for serious use yet, though it's a very promising start and has all the infrastructure that it needs to be something very nice indeed in the future. I think I'll hack up something much simpler for now which just kicks up tun/tap devices as root and passes their fds down to setuid() children that run specified qemus as a regular user. You'd think someone would have written this already... Have a patch to be going on with, anyway: libvirt can't parse the version output of qemu-kvm 0.11.0-stable, and doesn't know the name of its x86_64 upstream binary. (I'm not certain that this is right: I'm not sure it can distinguish between KMM and non-KVM. I'm equally unsure that this matters anymore now that enough KVM support to be getting on with is upstream.) diff --git a/src/qemu/qemu_conf.c b/src/qemu/qemu_conf.c index 8f8d490..10efdae 100644 --- a/src/qemu/qemu_conf.c +++ b/src/qemu/qemu_conf.c @@ -642,7 +642,8 @@ qemudCapsInitGuest(virCapsPtr caps, if (STREQ(info->arch, hostmachine) || (STREQ(hostmachine, "x86_64") && STREQ(info->arch, "i686"))) { const char *const kvmbins[] = { "/usr/bin/qemu-kvm", /* Fedora */ - "/usr/bin/kvm" }; /* Upstream .spec */ + "/usr/bin/kvm", /* Upstream .spec */ + "/usr/bin/qemu-system-x86_64" }; /* Upstream sources */ for (i = 0; i < ARRAY_CARDINALITY(kvmbins); ++i) { if (access(kvmbins[i], X_OK) == 0 && @@ -953,6 +954,7 @@ static unsigned int qemudComputeCmdFlags(const char *help, * in parenthesis as follows: * - qemu-kvm-x.y.z in stable releases * - kvm-XX for kvm versions up to kvm-85 + * - kvm-devel for kvm 0.11.0-stable * - qemu-kvm-devel-XX for kvm version kvm-86 and later * * For qemu-kvm versions before 0.10.z, we need to detect @@ -962,6 +964,7 @@ static unsigned int qemudComputeCmdFlags(const char *help, */ #define QEMU_VERSION_STR "QEMU PC emulator version" #define QEMU_KVM_VER_PREFIX "(qemu-kvm-" +#define KVM_GIT_STABLE_VER_PREFIX "(kvm-devel)" #define KVM_VER_PREFIX "(kvm-" #define SKIP_BLANKS(p) do { while ((*(p) == ' ') || (*(p) == '\t')) (p)++; } while (0) @@ -1005,6 +1008,9 @@ int qemudParseHelpStr(const char *help, if (STRPREFIX(p, QEMU_KVM_VER_PREFIX)) { *is_kvm = 1; p += strlen(QEMU_KVM_VER_PREFIX); + } else if (STRPREFIX(p, KVM_GIT_STABLE_VER_PREFIX)) { + *is_kvm = 1; + p += strlen(KVM_GIT_STABLE_VER_PREFIX); } else if (STRPREFIX(p, KVM_VER_PREFIX)) { int ret; -- 1.6.5.3.100.g75959
On 11/30/2009 05:17 PM, Nix wrote:
--- src/qemu/qemu_conf.c | 8 +++++++- 1 files changed, 7 insertions(+), 1 deletions(-)
On 30 Nov 2009, Cole Robinson uttered the following:
If you create a manual bridge, virt-manager should see it. However, the virtinst bug I mentioned in my other email was conspiring to make this not work for you. Pull from virtinst, and your manually created bridge devices should show up in virt-manager.
Pulled. Bridges still show up as 'not bridged', inactive.
It might just be unfortunate timing. If you look at the recent virt-manager commits (as in, within the past week), the bridging code was largely refactored to deal with HAL deprecation, and optionally use libvirt nodedev and interface (aka netcf) APIs if available. This may very well have caused regressions. Might be worth a shot to try the 0.8.0 release and see if you still hit issues. Also, can you provide the output of: ~/.virt-manager/virt-manager.log brctl show virsh iface-list --all - Cole
participants (3)
-
Cole Robinson -
Daniel P. Berrange -
Nix