Re: [libvirt] KVM doesn't send an arp announce after live migrating a domain
On 08/25/2010 12:21 PM, Nils Cant wrote:
On 08/25/2010 10:38 AM, Gleb Natapov wrote:
qemu sends gratuitous ARP after migration. Check forward delay setting on your bridge interface. It should be set to zero.
Aha! That fixed it. Turns out that debian bridge-utils sets the default to 15 for bridges. Manually setting it to 0 with 'brctl setfd br0 0' or setting the 'bridge_fd' parameter to 0 in /etc/network/interfaces solves the issue.
I think libvirt is doing something about this, copying list for further info. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
On Wed, Aug 25, 2010 at 01:40:19PM +0300, Avi Kivity wrote:
On 08/25/2010 12:21 PM, Nils Cant wrote:
On 08/25/2010 10:38 AM, Gleb Natapov wrote:
qemu sends gratuitous ARP after migration. Check forward delay setting on your bridge interface. It should be set to zero.
Aha! That fixed it. Turns out that debian bridge-utils sets the default to 15 for bridges. Manually setting it to 0 with 'brctl setfd br0 0' or setting the 'bridge_fd' parameter to 0 in /etc/network/interfaces solves the issue.
I think libvirt is doing something about this, copying list for further info.
libvirt doesn't set a policy for this. It provides an API for configuring host networking, but we don't override the kernel's forward delay policy, since we don't presume that all bridges are going to have VMs attached. In any case the API isn't available for Debian yet, since no one has ported netcf to Debian, so I assume the OP set bridging up manually. The '15' second default is actually a kernel level default IIRC. The two main host network configs recommended for use with libvirt+KVM (either NAT or bridging) are documented here: http://wiki.libvirt.org/page/Networking Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On 08/25/2010 01:52 PM, Daniel P. Berrange wrote:
I think libvirt is doing something about this, copying list for further info. libvirt doesn't set a policy for this. It provides an API for configuring host networking, but we don't override the kernel's forward delay policy, since we don't presume that all bridges are going to have VMs attached. In any case the API isn't available for Debian yet, since no one has ported netcf to Debian, so I assume the OP set bridging up manually. The '15' second default is actually a kernel level default IIRC.
The two main host network configs recommended for use with libvirt+KVM (either NAT or bridging) are documented here:
From that page: # virsh net-define /usr/share/libvirt/networks/default.xml From my copy of that file: <network> <name>default</name> <bridge name="virbr0" /> <forward/> <ip address="192.168.122.1" netmask="255.255.255.0"> <dhcp> <range start="192.168.122.2" end="192.168.122.254" /> </dhcp> </ip> </network> So it looks like the default config uses the kernel default? If libvirt uses an existing bridge I agree it shouldn't hack it, but if it creates its own can't it use a sensible default? -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
On Wed, Aug 25, 2010 at 02:05:45PM +0300, Avi Kivity wrote:
On 08/25/2010 01:52 PM, Daniel P. Berrange wrote:
I think libvirt is doing something about this, copying list for further info. libvirt doesn't set a policy for this. It provides an API for configuring host networking, but we don't override the kernel's forward delay policy, since we don't presume that all bridges are going to have VMs attached. In any case the API isn't available for Debian yet, since no one has ported netcf to Debian, so I assume the OP set bridging up manually. The '15' second default is actually a kernel level default IIRC.
The two main host network configs recommended for use with libvirt+KVM (either NAT or bridging) are documented here:
From that page:
# virsh net-define /usr/share/libvirt/networks/default.xml
From my copy of that file:
<network> <name>default</name> <bridge name="virbr0" /> <forward/> <ip address="192.168.122.1" netmask="255.255.255.0"> <dhcp> <range start="192.168.122.2" end="192.168.122.254" /> </dhcp> </ip> </network>
So it looks like the default config uses the kernel default? If libvirt uses an existing bridge I agree it shouldn't hack it, but if it creates its own can't it use a sensible default?
That is the NAT virtual network. That one *does* default to a forward delay of 0, but since it is NAT, it is fairly useless for migration in anycase. If you do 'virsh net-dumpxml default' you should see that delay='0' was added The OP was using bridging rather than NAT though, so this XML example doesn't apply. My comments about libvirt not overriding kenrel policy for forward delay were WRT full bridging mode, not the NAT mode[1] Regards, Daniel [1] Yes, the NAT mode uses a bridge as an implementation detail, but there's no physical NIC in that bridge - it is merely to connect the TAP devices together. Connection to the LAN is forwarded + NAT. -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On 08/25/2010 02:15 PM, Daniel P. Berrange wrote:
So it looks like the default config uses the kernel default? If libvirt uses an existing bridge I agree it shouldn't hack it, but if it creates its own can't it use a sensible default? That is the NAT virtual network. That one *does* default to a forward delay of 0, but since it is NAT, it is fairly useless for migration in anycase. If you do 'virsh net-dumpxml default' you should see that delay='0' was added
The OP was using bridging rather than NAT though, so this XML example doesn't apply. My comments about libvirt not overriding kenrel policy for forward delay were WRT full bridging mode, not the NAT mode[1]
Yes, of course. Can't libvirt also create a non-NAT bridge? Looks like it would prevent a lot of manual work and opportunity for misconfiguration. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
On Wed, Aug 25, 2010 at 02:30:01PM +0300, Avi Kivity wrote:
On 08/25/2010 02:15 PM, Daniel P. Berrange wrote:
So it looks like the default config uses the kernel default? If libvirt uses an existing bridge I agree it shouldn't hack it, but if it creates its own can't it use a sensible default? That is the NAT virtual network. That one *does* default to a forward delay of 0, but since it is NAT, it is fairly useless for migration in anycase. If you do 'virsh net-dumpxml default' you should see that delay='0' was added
The OP was using bridging rather than NAT though, so this XML example doesn't apply. My comments about libvirt not overriding kenrel policy for forward delay were WRT full bridging mode, not the NAT mode[1]
Yes, of course.
Can't libvirt also create a non-NAT bridge? Looks like it would prevent a lot of manual work and opportunity for misconfiguration.
Yes, it can on latest Fedora/RHEL6, using the netcf library. This is the new 'virsh iface-XXX' command set (and equivalent APIs). I've not updated the docs to cover this functionality yet though. It also does bonding, and vlans, etc Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On 08/25/2010 02:36 PM, Daniel P. Berrange wrote:
Can't libvirt also create a non-NAT bridge? Looks like it would prevent a lot of manual work and opportunity for misconfiguration. Yes, it can on latest Fedora/RHEL6, using the netcf library. This is the new 'virsh iface-XXX' command set (and equivalent APIs). I've not updated the docs to cover this functionality yet though. It also does bonding, and vlans, etc
Great. Is virt-manager able to drive this? it would be great if you could drive everything from there. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
On Wed, Aug 25, 2010 at 02:38:25PM +0300, Avi Kivity wrote:
On 08/25/2010 02:36 PM, Daniel P. Berrange wrote:
Can't libvirt also create a non-NAT bridge? Looks like it would prevent a lot of manual work and opportunity for misconfiguration. Yes, it can on latest Fedora/RHEL6, using the netcf library. This is the new 'virsh iface-XXX' command set (and equivalent APIs). I've not updated the docs to cover this functionality yet though. It also does bonding, and vlans, etc
Great.
Is virt-manager able to drive this? it would be great if you could drive everything from there.
Yes, it does now, under the menu Edit -> Host Details -> Network Interfaces NetworkManager has also finally learnt to ignore ifcfg-XXX files which have a BRIDGE= setting in them, so it shouldn't totally trash your guest bridge networking if you leave NM running. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On 08/25/2010 02:42 PM, Daniel P. Berrange wrote:
Is virt-manager able to drive this? it would be great if you could drive everything from there. Yes, it does now, under the menu Edit -> Host Details -> Network Interfaces NetworkManager has also finally learnt to ignore ifcfg-XXX files which have a BRIDGE= setting in them, so it shouldn't totally trash your guest bridge networking if you leave NM running.
Cool. I guess what remains is to get people to unlearn all the previous hacks. (also would be nice to have libvirt talk to NetworkManager instead of /etc/sysconfig) -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
On 08/25/2010 07:59 AM, Avi Kivity wrote:
On 08/25/2010 02:42 PM, Daniel P. Berrange wrote:
Is virt-manager able to drive this? it would be great if you could drive everything from there. Yes, it does now, under the menu Edit -> Host Details -> Network Interfaces NetworkManager has also finally learnt to ignore ifcfg-XXX files which have a BRIDGE= setting in them, so it shouldn't totally trash your guest bridge networking if you leave NM running.
There are a couple of caveats remaining, though: 1) When a bridge is created and a physical interface is moved to it, there is a short delay (< 1 sec, but > 0) when NM hasn't yet processed the inotify message from the kernel, and so doesn't yet know that it's supposed to stop managing that interface. The result is that if you attempt to do "virsh iface-create br0.xml && virsh iface-start br0" (where br0 is becoming the master for an interface, eg "eth0" that previously was being used directly), the resulting "ifup" (part of iface-start) will fail. A delay of even 1 sec. eliminates this problem. This race condition will hopefully soon be remedied in NM. Having this functionality in virt-manager really has made it dead simple to create a bridge, especially if NM is disabled, in which case the host's ethernet switches from direct connnect to bridged with no interruption in open connections on the interface (ie, it's possible to create and bring up the bridge while connected via the interface you are bridging). 2) If the physical interface being bridged is the only interface managed by NM, once it is moved to the bridge (and thus no longer managed by NM), NM will believe that the system is "offline", and any applications paying attention to that status will lose their connectivity. This isn't a problem if there is any other interface that's still managed by NM (the bridges themselves are not yet managed by NM - so far it just pretends they don't exist, as it does with vlans and bonds, as well as interfaces that are attached to a bridge).
Cool. I guess what remains is to get people to unlearn all the previous hacks.
Yes, I need to add a section to the docs Daniel pointed to that suggest using the virsh iface-* commands (or virt-manager's functionality) when available, and to only do it the handmade way if netcf isn't there (meaning that iface-* won't be there).
(also would be nice to have libvirt talk to NetworkManager instead of /etc/sysconfig)
To solve the variations of problem (2) above, NM needs to understand bridges, including configuring them, and there is some amount of consensus (including from the NM team :-) that a good way for NM to configure bridges (as well as vlans and bonds) is to use the netcf library, as libvirt does. The biggest hurdle is finding the people resources to get it done ;-) So, instead of libvirt-->netcf-->NM-->sysconfig, the diagram would be libvirt/NM-->netcf-->sysconfig.
On 08/25/2010 06:52 AM, Daniel P. Berrange wrote:
libvirt doesn't set a policy for this. It provides an API for configuring host networking, but we don't override the kernel's forward delay policy, since we don't presume that all bridges are going to have VMs attached. In any case the API isn't available for Debian yet, since no one has ported netcf to Debian,
"Patches welcome!" (as are questions/suggestions/complaints/sarcastic taunts/etc) :-) https://fedorahosted.org/netcf/ If you look in the mailing list archives, you'll find a couple of messages from people porting netcf to other Linux distributions (although none has gotten to the point of a patch submitted for pushing). It would be really great to see netcf running on other platforms, as all the iface-* functionality in libvirt (and thus virt-manager) would magically start working on that platform).
participants (3)
-
Avi Kivity -
Daniel P. Berrange -
Laine Stump