[libvirt] virtio network problems with kvm guests with 2.6.26 kernel
Hi, After upgrading libvirt on one system, my kvm guests running a 2.6.26 kernel with virtio networking could no longer communicate with the network. It seems that the problem is caused by newer libvirt versions unconditionally enabling GSO support by setting IFF_VNET_HDR [1]. However, support for this feature is apparently broken in 2.6.26. Ubuntu seems to have also discovered this problem [2] and they currently work around it by removing GSO support from kvm [3]. Currently, I'm running with a patched libvirt that never sets IFF_VNET_HDR, and it's working fine once again. While this is not a libvirt problem per se, it certainly violated the principle of least surprise given that libvirt was the only thing that changed! But I don't know if there's much that libvirt can do to avoid this problem, as it's really something that needs to get fixed in kvm, or the guest needs to be upgraded, or switched away from virtio. At the very least, maybe this info helps someone else who runs across the same problem. -jim [1] http://www.redhat.com/archives/libvir-list/2009-January/msg00642.html [2] https://bugs.launchpad.net/kvm/+bug/331128 [3] http://patches.ubuntu.com/k/kvm/extracted/virtio-net_disable_gso.patch
On Tue, Sep 29, 2009 at 02:39:59AM -0400, Jim Paris wrote:
Hi,
After upgrading libvirt on one system, my kvm guests running a 2.6.26 kernel with virtio networking could no longer communicate with the network.
It seems that the problem is caused by newer libvirt versions unconditionally enabling GSO support by setting IFF_VNET_HDR [1].
However, support for this feature is apparently broken in 2.6.26. Ubuntu seems to have also discovered this problem [2] and they currently work around it by removing GSO support from kvm [3].
Currently, I'm running with a patched libvirt that never sets IFF_VNET_HDR, and it's working fine once again.
While this is not a libvirt problem per se, it certainly violated the principle of least surprise given that libvirt was the only thing that changed! But I don't know if there's much that libvirt can do to avoid this problem, as it's really something that needs to get fixed in kvm, or the guest needs to be upgraded, or switched away from virtio.
libvirt probes to see if IFF_VNET_HDR is supported by the kernel and if so, enables it. If its broken in the kernel, the kernel should have been patch, or had the flag disabled as its not really practical for libvirt to special case this thing further Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Daniel P. Berrange wrote:
On Tue, Sep 29, 2009 at 02:39:59AM -0400, Jim Paris wrote:
Hi,
After upgrading libvirt on one system, my kvm guests running a 2.6.26 kernel with virtio networking could no longer communicate with the network.
It seems that the problem is caused by newer libvirt versions unconditionally enabling GSO support by setting IFF_VNET_HDR [1].
However, support for this feature is apparently broken in 2.6.26. Ubuntu seems to have also discovered this problem [2] and they currently work around it by removing GSO support from kvm [3].
Currently, I'm running with a patched libvirt that never sets IFF_VNET_HDR, and it's working fine once again.
While this is not a libvirt problem per se, it certainly violated the principle of least surprise given that libvirt was the only thing that changed! But I don't know if there's much that libvirt can do to avoid this problem, as it's really something that needs to get fixed in kvm, or the guest needs to be upgraded, or switched away from virtio.
libvirt probes to see if IFF_VNET_HDR is supported by the kernel and if so, enables it. If its broken in the kernel, the kernel should have been patch, or had the flag disabled as its not really practical for libvirt to special case this thing further
Libvirt probes support in the host kernel. The problem lies with the guest kernel, which fails to communicate with the host when the host has IFF_VNET_HDR enabled. -jim
Jim Paris wrote:
Daniel P. Berrange wrote:
On Tue, Sep 29, 2009 at 02:39:59AM -0400, Jim Paris wrote:
Hi,
After upgrading libvirt on one system, my kvm guests running a 2.6.26 kernel with virtio networking could no longer communicate with the network.
It seems that the problem is caused by newer libvirt versions unconditionally enabling GSO support by setting IFF_VNET_HDR [1].
However, support for this feature is apparently broken in 2.6.26. Ubuntu seems to have also discovered this problem [2] and they currently work around it by removing GSO support from kvm [3].
Currently, I'm running with a patched libvirt that never sets IFF_VNET_HDR, and it's working fine once again.
While this is not a libvirt problem per se, it certainly violated the principle of least surprise given that libvirt was the only thing that changed! But I don't know if there's much that libvirt can do to avoid this problem, as it's really something that needs to get fixed in kvm, or the guest needs to be upgraded, or switched away from virtio.
libvirt probes to see if IFF_VNET_HDR is supported by the kernel and if so, enables it. If its broken in the kernel, the kernel should have been patch, or had the flag disabled as its not really practical for libvirt to special case this thing further
Libvirt probes support in the host kernel. The problem lies with the guest kernel, which fails to communicate with the host when the host has IFF_VNET_HDR enabled.
Nevermind that, my conclusions were bogus. Things still didn't quite add up, so I tracked down what's really going on here: - Libvirt 0.7.1 (as packaged by Debian) has IFF_VNET_HDR support. - Qemu-kvm 0.11.0 (as built by myself) did NOT include IFF_VNET_HDR support, because it was built against the older headers on my system. Libvirt assumes that if it can support IFF_VNET_HDR, and qemu is new enough, then qemu must support IFF_VNET_HDR too. This assumption was wrong in my case, but it seems there's no easy way for libvirt to figure out the correct thing to do. Anyway, after updating the linux-libc-dev package and rebuilding qemu-kvm, things work now. -jim
On Tue, 2009-09-29 at 15:20 -0400, Jim Paris wrote:
Nevermind that, my conclusions were bogus. Things still didn't quite add up, so I tracked down what's really going on here:
- Libvirt 0.7.1 (as packaged by Debian) has IFF_VNET_HDR support. - Qemu-kvm 0.11.0 (as built by myself) did NOT include IFF_VNET_HDR support, because it was built against the older headers on my system.
Libvirt assumes that if it can support IFF_VNET_HDR, and qemu is new enough, then qemu must support IFF_VNET_HDR too. This assumption was wrong in my case, but it seems there's no easy way for libvirt to figure out the correct thing to do.
Anyway, after updating the linux-libc-dev package and rebuilding qemu-kvm, things work now.
Yeah, that makes some sense - we have no way of knowing whether qemu was compiled with IFF_VNET_HDR support, all we can do is check that that version of qemu would have support if compiled against headers with IFF_VNET_HDR. Thanks, Mark.
participants (3)
-
Daniel P. Berrange -
Jim Paris -
Mark McLoughlin