Re: [libvirt] Error starting domain: internal error: missing IFLA_VF_INFO in netlink response
Hi Laine, Sorry to disturb you. It seemed this issue had been fixed in libvirt-1.2.2/libnl-3.2.22/linux-3.12. But we still got the error on PowerPC platform. I'll appreciate if you could give any suggestion. We are not sure if any netlink implementation in kernel space is missed. The scenario is a little complicated. We installed internal PF and VF kernel modules and want to use <interface type="hostdev" managed="yes"> syntax to start a guest domain with MAC address. # insmod fslinic.ko max_vfs=2 Freescale 10 Gigabit PCI Express Network Driver fslinic 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 1, Tx Queue count = 1 fslinic 0000:01:00.0: Freescale (R) 10 Gigabit Network Connection fslinic 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 1, Tx Queue count = 1 fslinic 0000:01:00.1: Freescale (R) 10 Gigabit Network Connection # insmod fslinicvf.ko Freescale 10 Gigabit PCI Express Network Driver # lspci -mk 00:00.0 "PCI bridge" "Freescale Semiconductor Inc" "Device 0440" -r20 "" "" 01:00.0 "Power PC" "Freescale Semiconductor Inc" "Device 0440" -r20 "" "" 01:00.1 "Power PC" "Freescale Semiconductor Inc" "Device 0440" -r20 "" "" 01:00.4 "Power PC" "Freescale Semiconductor Inc" "Device 0000" -r20 "" "" 01:00.5 "Power PC" "Freescale Semiconductor Inc" "Device 1957" -r20 "" "" 01:01.0 "Power PC" "Freescale Semiconductor Inc" "Device 0000" -r20 "" "" 01:01.1 "Power PC" "Freescale Semiconductor Inc" "Device 1957" -r20 "" "" 0001:00:00.0 "PCI bridge" "Freescale Semiconductor Inc" "Device 0440" -r20 "" "" 0002:00:00.0 "PCI bridge" "Freescale Semiconductor Inc" "Device 0440" -r20 "" "" # echo 1957 0000 > /sys/bus/pci/drivers/vfio-pci/new_id # echo 1957 1957 > /sys/bus/pci/drivers/vfio-pci/new_id # cat interface.xml <domain type='kvm'> <name>interface</name> <memory unit='KiB'>524288</memory> <currentMemory unit='KiB'>524288</currentMemory> <vcpu placement='static'>1</vcpu> <os> <type arch='ppc64' machine='ppce500'>hvm</type> <kernel>/dev/shm/uImage</kernel> <initrd>/dev/shm/ramdisk</initrd> <cmdline>root=/dev/ram rw console=ttyS0,115200</cmdline> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-ppc64</emulator> <controller type='usb' index='0'/> <controller type='pci' index='0' model='pci-root'/> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <interface type="hostdev" managed="yes"> <mac address="00:e0:0c:00:20:01"/> <source> <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x04"/> </source> </interface> <memballoon model='virtio'/> </devices> </domain> root@t4240rdb:/var/volatile# virsh start interface error: Failed to start domain interface error: internal error: missing IFLA_VF_INFO in netlink response The debug message is as below: error : virNetClientProgramDispatchError:175 : internal error: missing IFLA_VF_INFO in netlink response Do you have any suggestion to investigate/debug this error? Wish you could reply. Thank you in advance. Best Regards, Olivia
-----Original Message----- From: Yin Olivia-R63875 Sent: Monday, November 24, 2014 7:37 PM To: 'libvirt-users@redhat.com'; libvir-list@redhat.com Subject: Error starting domain: internal error: missing IFLA_VF_INFO in netlink response
Hi,
We try PCI Passthrough of host network devices on PPC platform. http://wiki.libvirt.org/page/Networking#Assignment_with_.3Cinterface_type.3D .27hostdev.27.3E_.28SRIOV_devices_only.29
But we got a similar issue as below that reported on RedHat before: https://bugzilla.redhat.com/show_bug.cgi?id=1040626
With <hostdev>, it could start VM successfully. <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x4'/> </source> </hostdev>
Logged into VM and checked for VF using lspci root@model : qemu ppce500:~# lspci 00:00.0 PCI bridge: Freescale Semiconductor Inc MPC8533E 00:02.0 Power PC: Freescale Semiconductor Inc Device 0000 (rev 20) 00:03.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon
But it will fail if using <interface type="hostdev">: <interface type="hostdev" managed="yes"> <mac address="00:e0:0c:00:20:01"/> <source> <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x4"/> </source> </interface>
# virsh create vf.xml 2014-11-24 11:08:31.390+0000: 3556: info : libvirt version: 1.2.2 2014-11-24 11:08:31.390+0000: 3556: debug : virLogParseOutputs:1378 : outputs=1:file:virsh.log error: Failed to create domain from vf.xml error: internal error: missing IFLA_VF_INFO in netlink response
Exactly We're using max_vfs=2 and libnl-3.2.22 # ls -l /usr/lib64/libnl-3.so.200.17.0 -rwxr-xr-x 1 root root 154440 Sep 17 07:15 /usr/lib64/libnl-3.so.200.17.0
Why this issue happened on PPC? Is there anything architecture-specific support needed to add?
Thanks, Olivia
On 01/08/2015 04:40 AM, Hong-Hua.Yin@freescale.com wrote:
Hi Laine,
Sorry to disturb you. It seemed this issue had been fixed in libvirt-1.2.2/libnl-3.2.22/linux-3.12. But we still got the error on PowerPC platform. I'll appreciate if you could give any suggestion. We are not sure if any netlink implementation in kernel space is missed.
The scenario is a little complicated. We installed internal PF and VF kernel modules and want to use <interface type="hostdev" managed="yes"> syntax to start a guest domain with MAC address.
# insmod fslinic.ko max_vfs=2 Freescale 10 Gigabit PCI Express Network Driver fslinic 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 1, Tx Queue count = 1 fslinic 0000:01:00.0: Freescale (R) 10 Gigabit Network Connection fslinic 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 1, Tx Queue count = 1 fslinic 0000:01:00.1: Freescale (R) 10 Gigabit Network Connection
# insmod fslinicvf.ko Freescale 10 Gigabit PCI Express Network Driver
# lspci -mk 00:00.0 "PCI bridge" "Freescale Semiconductor Inc" "Device 0440" -r20 "" "" 01:00.0 "Power PC" "Freescale Semiconductor Inc" "Device 0440" -r20 "" "" 01:00.1 "Power PC" "Freescale Semiconductor Inc" "Device 0440" -r20 "" "" 01:00.4 "Power PC" "Freescale Semiconductor Inc" "Device 0000" -r20 "" "" 01:00.5 "Power PC" "Freescale Semiconductor Inc" "Device 1957" -r20 "" "" 01:01.0 "Power PC" "Freescale Semiconductor Inc" "Device 0000" -r20 "" "" 01:01.1 "Power PC" "Freescale Semiconductor Inc" "Device 1957" -r20 "" "" 0001:00:00.0 "PCI bridge" "Freescale Semiconductor Inc" "Device 0440" -r20 "" "" 0002:00:00.0 "PCI bridge" "Freescale Semiconductor Inc" "Device 0440" -r20 "" ""
What is the output of "virsh nodedev-dumpxml pci_0000_01_00_4"? It should contain an element like this: <capability type='phys_function'> ... This indicates that it is an sriov VF (its PF will be listed in the above <capability> element). If this element is missing, there is something incomplete about the SRIOV implementation for your card's driver.
# echo 1957 0000 > /sys/bus/pci/drivers/vfio-pci/new_id # echo 1957 1957 > /sys/bus/pci/drivers/vfio-pci/new_id
You shouldn't need to do the above when managed='yes', since libvirt does it for you (doing it manually like this may or may not have adverse consequences, I haven't tried it)
<interface type="hostdev" managed="yes"> <mac address="00:e0:0c:00:20:01"/> <source> <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x04"/> </source> </interface> <memballoon model='virtio'/> </devices> </domain>
root@t4240rdb:/var/volatile# virsh start interface error: Failed to start domain interface error: internal error: missing IFLA_VF_INFO in netlink response
At some point (it's been quite awhile, so I've forgotten exactly when) the kernel stopped automatically including the IFLA_VF_INFO object in the response to a netlink RTM_GETLINK request. In order for this info to be included, you need to add an IFLA_EXT_MASK with the RTEXT_FILTER_VF bit set into the RTM_GETLINK request. This is done in src/util/virnetdev.c:virNetDevLinkDump() - here are two bugzilla records describing the problem and pointing at the patches added to fix it: https://bugzilla.redhat.com/show_bug.cgi?id=889249 https://bugzilla.redhat.com/show_bug.cgi?id=889319 That particular code isn't compiled into libvirt unless RTEXT_FILTER_VF is #defined, which should be done in /usr/include/linux/rtnetlink.h. If the kernel headers used when building libvirt were missing that #define (or maybe instead of it being #defined, it is defined as a value in an enum?), then RTEXT_FILTER_VF wouldn't be set in the request. Of course if RTEXT_FILTER_VF really doesn't exist, then the kernel should be returning IFLA_VF_INFO without needing the extra flag in the request. Hopefully something in here will point you at the source of your problem.
Hi Laine, Thank you for the reply. I tried to dump the information.
-----Original Message----- From: sendmail [mailto:justsendmailnothingelse@gmail.com] On Behalf Of Laine Stump Sent: Thursday, January 08, 2015 11:13 PM To: Yin Olivia-R63875 Cc: libvir-list@redhat.com Subject: Re: Error starting domain: internal error: missing IFLA_VF_INFO in netlink response
On 01/08/2015 04:40 AM, Hong-Hua.Yin@freescale.com wrote:
Hi Laine,
Sorry to disturb you. It seemed this issue had been fixed in libvirt-1.2.2/libnl-3.2.22/linux-3.12. But we still got the error on PowerPC platform. I'll appreciate if you could give any suggestion. We are not sure if any netlink implementation in kernel space is missed.
The scenario is a little complicated. We installed internal PF and VF kernel modules and want to use <interface type="hostdev" managed="yes"> syntax to start a guest domain with MAC address.
# insmod fslinic.ko max_vfs=2 Freescale 10 Gigabit PCI Express Network Driver fslinic 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 1, Tx Queue count = 1 fslinic 0000:01:00.0: Freescale (R) 10 Gigabit Network Connection fslinic 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 1, Tx Queue count = 1 fslinic 0000:01:00.1: Freescale (R) 10 Gigabit Network Connection
# insmod fslinicvf.ko Freescale 10 Gigabit PCI Express Network Driver
# lspci -mk 00:00.0 "PCI bridge" "Freescale Semiconductor Inc" "Device 0440" -r20 "" "" 01:00.0 "Power PC" "Freescale Semiconductor Inc" "Device 0440" -r20 "" "" 01:00.1 "Power PC" "Freescale Semiconductor Inc" "Device 0440" -r20 "" "" 01:00.4 "Power PC" "Freescale Semiconductor Inc" "Device 0000" -r20 "" "" 01:00.5 "Power PC" "Freescale Semiconductor Inc" "Device 1957" -r20 "" "" 01:01.0 "Power PC" "Freescale Semiconductor Inc" "Device 0000" -r20 "" "" 01:01.1 "Power PC" "Freescale Semiconductor Inc" "Device 1957" -r20 "" "" 0001:00:00.0 "PCI bridge" "Freescale Semiconductor Inc" "Device 0440" -r20 "" "" 0002:00:00.0 "PCI bridge" "Freescale Semiconductor Inc" "Device 0440" -r20 "" ""
What is the output of "virsh nodedev-dumpxml pci_0000_01_00_4"? It should contain an element like this:
<capability type='phys_function'> ...
This indicates that it is an sriov VF (its PF will be listed in the above <capability> element). If this element is missing, there is something incomplete about the SRIOV implementation for your card's driver.
It seemed that card driver is OK. ~# virsh nodedev-dumpxml pci_0000_01_00_4 <device> <name>pci_0000_01_00_4</name> <path>/sys/devices/ffe240000.pcie/pci0000:00/0000:00:00.0/0000:01:00.4</path> <parent>pci_0000_00_00_0</parent> <driver> <name>fslinicvf</name> </driver> <capability type='pci'> <domain>0</domain> <bus>1</bus> <slot>0</slot> <function>4</function> <product id='0x0000' /> <vendor id='0x1957' /> <capability type='phys_function'> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </capability> <iommuGroup number='78'> <address domain='0x0000' bus='0x01' slot='0x00' function='0x4'/> </iommuGroup> </capability> </device>
# echo 1957 0000 > /sys/bus/pci/drivers/vfio-pci/new_id # echo 1957 1957 > /sys/bus/pci/drivers/vfio-pci/new_id
You shouldn't need to do the above when managed='yes', since libvirt does it for you (doing it manually like this may or may not have adverse consequences, I haven't tried it)
OK, I got it.
<interface type="hostdev" managed="yes"> <mac address="00:e0:0c:00:20:01"/> <source> <address type="pci" domain="0x0000" bus="0x01" slot="0x00"
function="0x04"/>
</source> </interface> <memballoon model='virtio'/> </devices> </domain>
root@t4240rdb:/var/volatile# virsh start interface error: Failed to start domain interface error: internal error: missing IFLA_VF_INFO in netlink response
At some point (it's been quite awhile, so I've forgotten exactly when) the kernel stopped automatically including the IFLA_VF_INFO object in the response to a netlink RTM_GETLINK request. In order for this info to be included, you need to add an IFLA_EXT_MASK with the RTEXT_FILTER_VF bit set into the RTM_GETLINK request. This is done in src/util/virnetdev.c:virNetDevLinkDump() - here are two bugzilla records describing the problem and pointing at the patches added to fix it:
https://bugzilla.redhat.com/show_bug.cgi?id=889249 https://bugzilla.redhat.com/show_bug.cgi?id=889319
That particular code isn't compiled into libvirt unless RTEXT_FILTER_VF is #defined, which should be done in /usr/include/linux/rtnetlink.h. If the kernel headers used when building libvirt were missing that #define (or maybe instead of it being #defined, it is defined as a value in an enum?), then RTEXT_FILTER_VF wouldn't be set in the request. Of course if RTEXT_FILTER_VF really doesn't exist, then the kernel should be returning IFLA_VF_INFO without needing the extra flag in the request.
Hopefully something in here will point you at the source of your problem.
We use linux-3.12 with RTEXT_FILTER_VF defined. $ grep -r RTEXT_FILTER_VF ./ ./include/uapi/linux/rtnetlink.h:#define RTEXT_FILTER_VF (1 << 0) So I guess your patches had been compiled into libvirt. Best Regards, Olivia
Hi Laine, I'm not sure if /usr/include/linux/rtnetlink.h is necessary in the filesystem on host machine. In our scenario, both host machine and iNIC card are both PowerPC boards. I just checked the linux source code we used $ grep -r RTEXT_FILTER_VF ./ ./include/uapi/linux/rtnetlink.h:#define RTEXT_FILTER_VF (1 << 0) But we didn't install the linux header files on the PowerPC host machine. Best Regards, Olivia
-----Original Message----- From: Yin Olivia-R63875 Sent: Monday, January 12, 2015 5:48 PM To: 'Laine Stump' Cc: libvir-list@redhat.com Subject: RE: Error starting domain: internal error: missing IFLA_VF_INFO in netlink response
Hi Laine,
Thank you for the reply. I tried to dump the information.
-----Original Message----- From: sendmail [mailto:justsendmailnothingelse@gmail.com] On Behalf Of Laine Stump Sent: Thursday, January 08, 2015 11:13 PM To: Yin Olivia-R63875 Cc: libvir-list@redhat.com Subject: Re: Error starting domain: internal error: missing IFLA_VF_INFO in netlink response
On 01/08/2015 04:40 AM, Hong-Hua.Yin@freescale.com wrote:
Hi Laine,
Sorry to disturb you. It seemed this issue had been fixed in libvirt-1.2.2/libnl-3.2.22/linux-3.12. But we still got the error on PowerPC platform. I'll appreciate if you could give any suggestion. We are not sure if any netlink implementation in kernel space is missed.
The scenario is a little complicated. We installed internal PF and VF kernel modules and want to use <interface type="hostdev" managed="yes"> syntax to start a guest domain with MAC address.
# insmod fslinic.ko max_vfs=2 Freescale 10 Gigabit PCI Express Network Driver fslinic 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 1, Tx Queue count = 1 fslinic 0000:01:00.0: Freescale (R) 10 Gigabit Network Connection fslinic 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 1, Tx Queue count = 1 fslinic 0000:01:00.1: Freescale (R) 10 Gigabit Network Connection
# insmod fslinicvf.ko Freescale 10 Gigabit PCI Express Network Driver
# lspci -mk 00:00.0 "PCI bridge" "Freescale Semiconductor Inc" "Device 0440" -r20 "" "" 01:00.0 "Power PC" "Freescale Semiconductor Inc" "Device 0440" -r20 "" "" 01:00.1 "Power PC" "Freescale Semiconductor Inc" "Device 0440" -r20 "" "" 01:00.4 "Power PC" "Freescale Semiconductor Inc" "Device 0000" -r20 "" "" 01:00.5 "Power PC" "Freescale Semiconductor Inc" "Device 1957" -r20 "" "" 01:01.0 "Power PC" "Freescale Semiconductor Inc" "Device 0000" -r20 "" "" 01:01.1 "Power PC" "Freescale Semiconductor Inc" "Device 1957" -r20 "" "" 0001:00:00.0 "PCI bridge" "Freescale Semiconductor Inc" "Device 0440" -r20 "" "" 0002:00:00.0 "PCI bridge" "Freescale Semiconductor Inc" "Device 0440" -r20 "" ""
What is the output of "virsh nodedev-dumpxml pci_0000_01_00_4"? It should contain an element like this:
<capability type='phys_function'> ...
This indicates that it is an sriov VF (its PF will be listed in the above <capability> element). If this element is missing, there is something incomplete about the SRIOV implementation for your card's driver.
It seemed that card driver is OK.
~# virsh nodedev-dumpxml pci_0000_01_00_4 <device> <name>pci_0000_01_00_4</name> <path>/sys/devices/ffe240000.pcie/pci0000:00/0000:00:00.0/0000:01:00.4</path> <parent>pci_0000_00_00_0</parent> <driver> <name>fslinicvf</name> </driver> <capability type='pci'> <domain>0</domain> <bus>1</bus> <slot>0</slot> <function>4</function> <product id='0x0000' /> <vendor id='0x1957' /> <capability type='phys_function'> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </capability> <iommuGroup number='78'> <address domain='0x0000' bus='0x01' slot='0x00' function='0x4'/> </iommuGroup> </capability> </device>
# echo 1957 0000 > /sys/bus/pci/drivers/vfio-pci/new_id # echo 1957 1957 > /sys/bus/pci/drivers/vfio-pci/new_id
You shouldn't need to do the above when managed='yes', since libvirt does it for you (doing it manually like this may or may not have adverse consequences, I haven't tried it)
OK, I got it.
<interface type="hostdev" managed="yes"> <mac address="00:e0:0c:00:20:01"/> <source> <address type="pci" domain="0x0000" bus="0x01" slot="0x00"
function="0x04"/>
</source> </interface> <memballoon model='virtio'/> </devices> </domain>
root@t4240rdb:/var/volatile# virsh start interface error: Failed to start domain interface error: internal error: missing IFLA_VF_INFO in netlink response
At some point (it's been quite awhile, so I've forgotten exactly when) the kernel stopped automatically including the IFLA_VF_INFO object in the response to a netlink RTM_GETLINK request. In order for this info to be included, you need to add an IFLA_EXT_MASK with the RTEXT_FILTER_VF bit set into the RTM_GETLINK request. This is done in src/util/virnetdev.c:virNetDevLinkDump() - here are two bugzilla records describing the problem and pointing at the patches added to fix it:
https://bugzilla.redhat.com/show_bug.cgi?id=889249 https://bugzilla.redhat.com/show_bug.cgi?id=889319
That particular code isn't compiled into libvirt unless RTEXT_FILTER_VF is #defined, which should be done in /usr/include/linux/rtnetlink.h. If the kernel headers used when building libvirt were missing that #define (or maybe instead of it being #defined, it is defined as a value in an enum?), then RTEXT_FILTER_VF wouldn't be set in the request. Of course if RTEXT_FILTER_VF really doesn't exist, then the kernel should be returning
IFLA_VF_INFO without needing the extra flag in the request.
Hopefully something in here will point you at the source of your problem.
We use linux-3.12 with RTEXT_FILTER_VF defined. $ grep -r RTEXT_FILTER_VF ./ ./include/uapi/linux/rtnetlink.h:#define RTEXT_FILTER_VF (1 << 0)
So I guess your patches had been compiled into libvirt.
Best Regards, Olivia
On 01/12/2015 05:09 AM, Hong-Hua.Yin@freescale.com wrote:
Hi Laine,
I'm not sure if /usr/include/linux/rtnetlink.h is necessary in the filesystem on host machine.
In our scenario, both host machine and iNIC card are both PowerPC boards. I just checked the linux source code we used $ grep -r RTEXT_FILTER_VF ./ ./include/uapi/linux/rtnetlink.h:#define RTEXT_FILTER_VF (1 << 0) But we didn't install the linux header files on the PowerPC host machine.
It's only necessary in the header file on the machine where you are building libvirt. The machine that is *running* libvirt just needs the proper support built into the kernel.
participants (2)
-
Hong-Hua.Yin＠freescale.com -
Laine Stump