[libvirt] About the vfio error when using SR-IOV
Hi, All I want to use the SR-IOV of intel 82576 NIC. I enabled IOMMU and VT-d and SR-IOV in BIOS. And enabled VT-d in kernel. The OS information is bellow: [root@host3 nova]# cat /etc/redhat-release CentOS Linux release 7.1.1503 (Core) [root@host3 nova]# uname -an Linux host3.localdomain 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux [root@host3 nova]# rpm -qa|grep qemu libvirt-daemon-driver-qemu-1.2.8-16.el7.x86_64 qemu-kvm-1.5.3-86.el7.x86_64 qemu-kvm-common-1.5.3-86.el7.x86_64 ipxe-roms-qemu-20130517-6.gitc4bce43.el7.noarch qemu-img-1.5.3-86.el7.x86_64 [root@host3 nova]# rpm -qa|grep libvirt libvirt-daemon-1.2.8-16.el7.x86_64 libvirt-daemon-driver-nwfilter-1.2.8-16.el7.x86_64 libvirt-python-1.2.8-7.el7.x86_64 libvirt-daemon-driver-storage-1.2.8-16.el7.x86_64 libvirt-daemon-driver-qemu-1.2.8-16.el7.x86_64 libvirt-daemon-config-nwfilter-1.2.8-16.el7.x86_64 libvirt-daemon-driver-secret-1.2.8-16.el7.x86_64 libvirt-daemon-driver-interface-1.2.8-16.el7.x86_64 libvirt-client-1.2.8-16.el7.x86_64 libvirt-glib-0.1.7-3.el7.x86_64 libvirt-daemon-driver-network-1.2.8-16.el7.x86_64 libvirt-daemon-driver-nodedev-1.2.8-16.el7.x86_64 libvirt-daemon-kvm-1.2.8-16.el7.x86_64 And I can see the vf of the NIC after ‘ echo '7' > /sys/class/net/ens1f1/device/sriov_numvfs ' [root@host3 VTS2.1-demo]# lspci |grep -i ethernet 08:00.0 Ethernet controller: QLogic Corp. 10GbE Converged Network Adapter (TCP/IP Networking) (rev 02) 08:00.1 Ethernet controller: QLogic Corp. 10GbE Converged Network Adapter (TCP/IP Networking) (rev 02) 0f:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 0f:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 10:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) I configured the interface as bellow in XML: <interface type="hostdev" managed="yes"> <mac address="fa:16:3e:f7:57:5f"/> <source> <address type="pci" domain="0x0000" bus="0x10" slot="0x10" function="0x3"/> </source> <vlan> <tag id="1000"/> </vlan> </interface> But the error output when I boot one vm: [root@host3 VTS2.1-demo]# virsh create vtc.demo.xml error: Failed to create domain from vtc.demo.xml error: internal error: early end of file from monitor: possible problem: 2016-02-22T07:38:42.169035Z qemu-kvm: -device vfio-pci,host=10:10.3,id=hostdev0,bus=pci.0,addr=0x3: vfio: error, group 17 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. 2016-02-22T07:38:42.169215Z qemu-kvm: -device vfio-pci,host=10:10.3,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to get group 17 2016-02-22T07:38:42.169233Z qemu-kvm: -device vfio-pci,host=10:10.3,id=hostdev0,bus=pci.0,addr=0x3: Device initialization failed. 2016-02-22T07:38:42.169248Z qemu-kvm: -device vfio-pci,host=10:10.3,id=hostdev0,bus=pci.0,addr=0x3: Device 'vfio-pci' could not be initialized Could you please help to solve it? Looking forward for your reply. Thanks.
On 02/22/2016 04:29 AM, Xiao Ma (xima2) wrote:
Hi, All
I want to use the SR-IOV of intel 82576 NIC.
I enabled IOMMU and VT-d and SR-IOV in BIOS. And enabled VT-d in kernel. The OS information is bellow: [root@host3 nova]# cat /etc/redhat-release CentOS Linux release 7.1.1503 (Core) [root@host3 nova]# uname -an Linux host3.localdomain 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux [root@host3 nova]# rpm -qa|grep qemu libvirt-daemon-driver-qemu-1.2.8-16.el7.x86_64 qemu-kvm-1.5.3-86.el7.x86_64 qemu-kvm-common-1.5.3-86.el7.x86_64 ipxe-roms-qemu-20130517-6.gitc4bce43.el7.noarch qemu-img-1.5.3-86.el7.x86_64 [root@host3 nova]# rpm -qa|grep libvirt libvirt-daemon-1.2.8-16.el7.x86_64 libvirt-daemon-driver-nwfilter-1.2.8-16.el7.x86_64 libvirt-python-1.2.8-7.el7.x86_64 libvirt-daemon-driver-storage-1.2.8-16.el7.x86_64 libvirt-daemon-driver-qemu-1.2.8-16.el7.x86_64 libvirt-daemon-config-nwfilter-1.2.8-16.el7.x86_64 libvirt-daemon-driver-secret-1.2.8-16.el7.x86_64 libvirt-daemon-driver-interface-1.2.8-16.el7.x86_64 libvirt-client-1.2.8-16.el7.x86_64 libvirt-glib-0.1.7-3.el7.x86_64 libvirt-daemon-driver-network-1.2.8-16.el7.x86_64 libvirt-daemon-driver-nodedev-1.2.8-16.el7.x86_64 libvirt-daemon-kvm-1.2.8-16.el7.x86_64
And I can see the vf of the NIC after ‘ echo '7' > /sys/class/net/ens1f1/device/sriov_numvfs '
[root@host3 VTS2.1-demo]# lspci |grep -i ethernet 08:00.0 Ethernet controller: QLogic Corp. 10GbE Converged Network Adapter (TCP/IP Networking) (rev 02) 08:00.1 Ethernet controller: QLogic Corp. 10GbE Converged Network Adapter (TCP/IP Networking) (rev 02) 0f:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 0f:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 10:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
I configured the interface as bellow in XML:
<interface type="hostdev" managed="yes"> <mac address="fa:16:3e:f7:57:5f"/> <source> <address type="pci" domain="0x0000" bus="0x10" slot="0x10" function="0x3"/> </source> <vlan> <tag id="1000"/> </vlan> </interface>
But the error output when I boot one vm:
[root@host3 VTS2.1-demo]# virsh create vtc.demo.xml error: Failed to create domain from vtc.demo.xml error: internal error: early end of file from monitor: possible problem: 2016-02-22T07:38:42.169035Z qemu-kvm: -device vfio-pci,host=10:10.3,id=hostdev0,bus=pci.0,addr=0x3: vfio: error, group 17 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
One possible meaning is that vfio sees multiple devices in the same iommu group as the VF at 10:10.3 (in case there is some other possible cause, I'm Cc'ing Alex Williamson, the vfio author). You can check this by looking at the output of "virsh nodedev-dumpxml pci_0000_10_10_3" and look at the "iommuGroup" section - if there are multiple addresses listed there, then there are multiple devices in the same iommu group. It could be that your particular chipset needs a "quirk" in the kernel to be told that the VFs really can be in separate iommu groups; without that quirk, all 14 VFs show up in the same iommu group, so the only way to assign one to a guest is to assign *all* of them to the same guest (or at least detach all of them from the VF driver and attach to vfio-pci, then only assign one of them to a guest while the others sit unused). Since you're still running a 7.1 kernel, you may want to try updating to the latest available and see if that solves your problem. If you have further questions, please include the output of "virsh nodedev-dumpxxml pci_0000_10_10_3" and full lspci output (among other things, that should tell us which chipset your machine uses).
2016-02-22T07:38:42.169215Z qemu-kvm: -device vfio-pci,host=10:10.3,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to get group 17 2016-02-22T07:38:42.169233Z qemu-kvm: -device vfio-pci,host=10:10.3,id=hostdev0,bus=pci.0,addr=0x3: Device initialization failed. 2016-02-22T07:38:42.169248Z qemu-kvm: -device vfio-pci,host=10:10.3,id=hostdev0,bus=pci.0,addr=0x3: Device 'vfio-pci' could not be initialized
Could you please help to solve it?
Looking forward for your reply.
Thanks.
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On Mon, 22 Feb 2016 10:23:16 -0500 Laine Stump <laine@laine.org> wrote:
On 02/22/2016 04:29 AM, Xiao Ma (xima2) wrote:
Hi, All
I want to use the SR-IOV of intel 82576 NIC.
I enabled IOMMU and VT-d and SR-IOV in BIOS. And enabled VT-d in kernel. The OS information is bellow: [root@host3 nova]# cat /etc/redhat-release CentOS Linux release 7.1.1503 (Core) [root@host3 nova]# uname -an Linux host3.localdomain 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux [root@host3 nova]# rpm -qa|grep qemu libvirt-daemon-driver-qemu-1.2.8-16.el7.x86_64 qemu-kvm-1.5.3-86.el7.x86_64 qemu-kvm-common-1.5.3-86.el7.x86_64 ipxe-roms-qemu-20130517-6.gitc4bce43.el7.noarch qemu-img-1.5.3-86.el7.x86_64 [root@host3 nova]# rpm -qa|grep libvirt libvirt-daemon-1.2.8-16.el7.x86_64 libvirt-daemon-driver-nwfilter-1.2.8-16.el7.x86_64 libvirt-python-1.2.8-7.el7.x86_64 libvirt-daemon-driver-storage-1.2.8-16.el7.x86_64 libvirt-daemon-driver-qemu-1.2.8-16.el7.x86_64 libvirt-daemon-config-nwfilter-1.2.8-16.el7.x86_64 libvirt-daemon-driver-secret-1.2.8-16.el7.x86_64 libvirt-daemon-driver-interface-1.2.8-16.el7.x86_64 libvirt-client-1.2.8-16.el7.x86_64 libvirt-glib-0.1.7-3.el7.x86_64 libvirt-daemon-driver-network-1.2.8-16.el7.x86_64 libvirt-daemon-driver-nodedev-1.2.8-16.el7.x86_64 libvirt-daemon-kvm-1.2.8-16.el7.x86_64
And I can see the vf of the NIC after ‘ echo '7' > /sys/class/net/ens1f1/device/sriov_numvfs '
[root@host3 VTS2.1-demo]# lspci |grep -i ethernet 08:00.0 Ethernet controller: QLogic Corp. 10GbE Converged Network Adapter (TCP/IP Networking) (rev 02) 08:00.1 Ethernet controller: QLogic Corp. 10GbE Converged Network Adapter (TCP/IP Networking) (rev 02) 0f:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 0f:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 10:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
I configured the interface as bellow in XML:
<interface type="hostdev" managed="yes"> <mac address="fa:16:3e:f7:57:5f"/> <source> <address type="pci" domain="0x0000" bus="0x10" slot="0x10" function="0x3"/> </source> <vlan> <tag id="1000"/> </vlan> </interface>
But the error output when I boot one vm:
[root@host3 VTS2.1-demo]# virsh create vtc.demo.xml error: Failed to create domain from vtc.demo.xml error: internal error: early end of file from monitor: possible problem: 2016-02-22T07:38:42.169035Z qemu-kvm: -device vfio-pci,host=10:10.3,id=hostdev0,bus=pci.0,addr=0x3: vfio: error, group 17 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
One possible meaning is that vfio sees multiple devices in the same iommu group as the VF at 10:10.3 (in case there is some other possible cause, I'm Cc'ing Alex Williamson, the vfio author). You can check this by looking at the output of "virsh nodedev-dumpxml pci_0000_10_10_3" and look at the "iommuGroup" section - if there are multiple addresses listed there, then there are multiple devices in the same iommu group.
It could be that your particular chipset needs a "quirk" in the kernel to be told that the VFs really can be in separate iommu groups; without that quirk, all 14 VFs show up in the same iommu group, so the only way to assign one to a guest is to assign *all* of them to the same guest (or at least detach all of them from the VF driver and attach to vfio-pci, then only assign one of them to a guest while the others sit unused).
Since you're still running a 7.1 kernel, you may want to try updating to the latest available and see if that solves your problem.
If you have further questions, please include the output of "virsh nodedev-dumpxxml pci_0000_10_10_3" and full lspci output (among other things, that should tell us which chipset your machine uses).
It's almost certain this is due to the 82576 card being installed into a root port that does not have native ACS support. Updating to a newer kernel may or may not solve that problem, it really depends on what the upstream port is from the device. In addition to the libvirt commands above, /sys/kernel/iommu_groups/ will show you all of the isolated groups on the system. For further details, see: http://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html Thanks, Alex
participants (3)
-
Alex Williamson -
Laine Stump -
Xiao Ma (xima2)