10:40 a.m.
Hi Laine,
Sorry to disturb you.
It seemed this issue had been fixed in libvirt-1.2.2/libnl-3.2.22/linux-3.12. But we still
got the error on PowerPC platform.
I'll appreciate if you could give any suggestion. We are not sure if any netlink
implementation in kernel space is missed.
The scenario is a little complicated. We installed internal PF and VF kernel modules and
want to use
<interface type="hostdev" managed="yes"> syntax to start a guest
domain with MAC address.
# insmod fslinic.ko max_vfs=2
Freescale 10 Gigabit PCI Express Network Driver
fslinic 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 1, Tx Queue count = 1
fslinic 0000:01:00.0: Freescale (R) 10 Gigabit Network Connection
fslinic 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 1, Tx Queue count = 1
fslinic 0000:01:00.1: Freescale (R) 10 Gigabit Network Connection
# insmod fslinicvf.ko
Freescale 10 Gigabit PCI Express Network Driver
# lspci -mk
00:00.0 "PCI bridge" "Freescale Semiconductor Inc" "Device
0440" -r20 "" ""
01:00.0 "Power PC" "Freescale Semiconductor Inc" "Device
0440" -r20 "" ""
01:00.1 "Power PC" "Freescale Semiconductor Inc" "Device
0440" -r20 "" ""
01:00.4 "Power PC" "Freescale Semiconductor Inc" "Device
0000" -r20 "" ""
01:00.5 "Power PC" "Freescale Semiconductor Inc" "Device
1957" -r20 "" ""
01:01.0 "Power PC" "Freescale Semiconductor Inc" "Device
0000" -r20 "" ""
01:01.1 "Power PC" "Freescale Semiconductor Inc" "Device
1957" -r20 "" ""
0001:00:00.0 "PCI bridge" "Freescale Semiconductor Inc" "Device
0440" -r20 "" ""
0002:00:00.0 "PCI bridge" "Freescale Semiconductor Inc" "Device
0440" -r20 "" ""
# echo 1957 0000 > /sys/bus/pci/drivers/vfio-pci/new_id
# echo 1957 1957 > /sys/bus/pci/drivers/vfio-pci/new_id
# cat interface.xml
<domain type='kvm'>
<name>interface</name>
<memory unit='KiB'>524288</memory>
<currentMemory unit='KiB'>524288</currentMemory>
<vcpu placement='static'>1</vcpu>
<os>
<type arch='ppc64' machine='ppce500'>hvm</type>
<kernel>/dev/shm/uImage</kernel>
<initrd>/dev/shm/ramdisk</initrd>
<cmdline>root=/dev/ram rw console=ttyS0,115200</cmdline>
</os>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/qemu-system-ppc64</emulator>
<controller type='usb' index='0'/>
<controller type='pci' index='0' model='pci-root'/>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<interface type="hostdev" managed="yes">
<mac address="00:e0:0c:00:20:01"/>
<source>
<address type="pci" domain="0x0000" bus="0x01"
slot="0x00" function="0x04"/>
</source>
</interface>
<memballoon model='virtio'/>
</devices>
</domain>
root@t4240rdb:/var/volatile# virsh start interface
error: Failed to start domain interface
error: internal error: missing IFLA_VF_INFO in netlink response
The debug message is as below:
error : virNetClientProgramDispatchError:175 : internal error: missing IFLA_VF_INFO in
netlink response
Do you have any suggestion to investigate/debug this error?
Wish you could reply. Thank you in advance.
Best Regards,
Olivia
-----Original Message-----
From: Yin Olivia-R63875
Sent: Monday, November 24, 2014 7:37 PM
To: 'libvirt-users(a)redhat.com'; libvir-list(a)redhat.com
Subject: Error starting domain: internal error: missing IFLA_VF_INFO in netlink
response
Hi,
We try PCI Passthrough of host network devices on PPC platform.
http://wiki.libvirt.org/page/Networking#Assignment_with_.3Cinterface_type.3D
.27hostdev.27.3E_.28SRIOV_devices_only.29
But we got a similar issue as below that reported on RedHat before:
https://bugzilla.redhat.com/show_bug.cgi?id=1040626
With <hostdev>, it could start VM successfully.
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x4'/>
</source>
</hostdev>
Logged into VM and checked for VF using lspci root@model : qemu ppce500:~#
lspci
00:00.0 PCI bridge: Freescale Semiconductor Inc MPC8533E
00:02.0 Power PC: Freescale Semiconductor Inc Device 0000 (rev 20)
00:03.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon
But it will fail if using <interface type="hostdev">:
<interface type="hostdev" managed="yes">
<mac address="00:e0:0c:00:20:01"/>
<source>
<address type="pci" domain="0x0000" bus="0x01"
slot="0x00"
function="0x4"/>
</source>
</interface>
# virsh create vf.xml
2014-11-24 11:08:31.390+0000: 3556: info : libvirt version: 1.2.2
2014-11-24 11:08:31.390+0000: 3556: debug : virLogParseOutputs:1378 :
outputs=1:file:virsh.log
error: Failed to create domain from vf.xml
error: internal error: missing IFLA_VF_INFO in netlink response
Exactly We're using max_vfs=2 and libnl-3.2.22 # ls -l
/usr/lib64/libnl-3.so.200.17.0
-rwxr-xr-x 1 root root 154440 Sep 17 07:15 /usr/lib64/libnl-3.so.200.17.0
Why this issue happened on PPC? Is there anything architecture-specific support
needed to add?
Thanks,
Olivia
4:13 p.m.
New subject: [libvirt] Error starting domain: internal error: missing IFLA_VF_INFO in netlink response
On 01/08/2015 04:40 AM, Hong-Hua.Yin(a)freescale.com wrote:
Hi Laine,
Sorry to disturb you.
It seemed this issue had been fixed in libvirt-1.2.2/libnl-3.2.22/linux-3.12. But we
still got the error on PowerPC platform.
I'll appreciate if you could give any suggestion. We are not sure if any netlink
implementation in kernel space is missed.
The scenario is a little complicated. We installed internal PF and VF kernel modules and
want to use
<interface type="hostdev" managed="yes"> syntax to start a
guest domain with MAC address.
# insmod fslinic.ko max_vfs=2
Freescale 10 Gigabit PCI Express Network Driver
fslinic 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 1, Tx Queue count = 1
fslinic 0000:01:00.0: Freescale (R) 10 Gigabit Network Connection
fslinic 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 1, Tx Queue count = 1
fslinic 0000:01:00.1: Freescale (R) 10 Gigabit Network Connection
# insmod fslinicvf.ko
Freescale 10 Gigabit PCI Express Network Driver
# lspci -mk
00:00.0 "PCI bridge" "Freescale Semiconductor Inc" "Device
0440" -r20 "" ""
01:00.0 "Power PC" "Freescale Semiconductor Inc" "Device
0440" -r20 "" ""
01:00.1 "Power PC" "Freescale Semiconductor Inc" "Device
0440" -r20 "" ""
01:00.4 "Power PC" "Freescale Semiconductor Inc" "Device
0000" -r20 "" ""
01:00.5 "Power PC" "Freescale Semiconductor Inc" "Device
1957" -r20 "" ""
01:01.0 "Power PC" "Freescale Semiconductor Inc" "Device
0000" -r20 "" ""
01:01.1 "Power PC" "Freescale Semiconductor Inc" "Device
1957" -r20 "" ""
0001:00:00.0 "PCI bridge" "Freescale Semiconductor Inc" "Device
0440" -r20 "" ""
0002:00:00.0 "PCI bridge" "Freescale Semiconductor Inc" "Device
0440" -r20 "" ""
What is the output of "virsh nodedev-dumpxml pci_0000_01_00_4"? It
should contain an element like this:
<capability type='phys_function'>
...
This indicates that it is an sriov VF (its PF will be listed in the
above <capability> element). If this element is missing, there is
something incomplete about the SRIOV implementation for your card's driver.
# echo 1957 0000 > /sys/bus/pci/drivers/vfio-pci/new_id
# echo 1957 1957 > /sys/bus/pci/drivers/vfio-pci/new_id
You shouldn't need to do the above when managed='yes', since libvirt
does it for you (doing it manually like this may or may not have adverse
consequences, I haven't tried it)
<interface type="hostdev" managed="yes">
<mac address="00:e0:0c:00:20:01"/>
<source>
<address type="pci" domain="0x0000" bus="0x01"
slot="0x00" function="0x04"/>
</source>
</interface>
<memballoon model='virtio'/>
</devices>
</domain>
root@t4240rdb:/var/volatile# virsh start interface
error: Failed to start domain interface
error: internal error: missing IFLA_VF_INFO in netlink response
At some point (it's been quite awhile, so I've forgotten exactly when)
the kernel stopped automatically including the IFLA_VF_INFO object in
the response to a netlink RTM_GETLINK request. In order for this info to
be included, you need to add an IFLA_EXT_MASK with the RTEXT_FILTER_VF
bit set into the RTM_GETLINK request. This is done in
src/util/virnetdev.c:virNetDevLinkDump() -
here are two bugzilla records describing the problem and pointing at the
patches added to fix it:
https://bugzilla.redhat.com/show_bug.cgi?id=889249
https://bugzilla.redhat.com/show_bug.cgi?id=889319
That particular code isn't compiled into libvirt unless RTEXT_FILTER_VF
is #defined, which should be done in /usr/include/linux/rtnetlink.h. If
the kernel headers used when building libvirt were missing that #define
(or maybe instead of it being #defined, it is defined as a value in an
enum?), then RTEXT_FILTER_VF wouldn't be set in the request. Of course
if RTEXT_FILTER_VF really doesn't exist, then the kernel should be
returning IFLA_VF_INFO without needing the extra flag in the request.
Hopefully something in here will point you at the source of your problem.
10:48 a.m.
New subject: [libvirt] Error starting domain: internal error: missing IFLA_VF_INFO in netlink response
Hi Laine,
Thank you for the reply. I tried to dump the information.
-----Original Message-----
From: sendmail [mailto:justsendmailnothingelse@gmail.com] On Behalf Of Laine
Stump
Sent: Thursday, January 08, 2015 11:13 PM
To: Yin Olivia-R63875
Cc: libvir-list(a)redhat.com
Subject: Re: Error starting domain: internal error: missing IFLA_VF_INFO in netlink
response
On 01/08/2015 04:40 AM, Hong-Hua.Yin(a)freescale.com wrote:
> Hi Laine,
>
> Sorry to disturb you.
> It seemed this issue had been fixed in libvirt-1.2.2/libnl-3.2.22/linux-3.12. But
we still got the error on PowerPC platform.
> I'll appreciate if you could give any suggestion. We are not sure if any
netlink
implementation in kernel space is missed.
>
> The scenario is a little complicated. We installed internal PF and VF
> kernel modules and want to use <interface type="hostdev"
managed="yes">
syntax to start a guest domain with MAC address.
>
> # insmod fslinic.ko max_vfs=2
> Freescale 10 Gigabit PCI Express Network Driver fslinic 0000:01:00.0:
> Multiqueue Enabled: Rx Queue count = 1, Tx Queue count = 1 fslinic
> 0000:01:00.0: Freescale (R) 10 Gigabit Network Connection fslinic
> 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 1, Tx Queue count =
> 1 fslinic 0000:01:00.1: Freescale (R) 10 Gigabit Network Connection
>
> # insmod fslinicvf.ko
> Freescale 10 Gigabit PCI Express Network Driver
>
> # lspci -mk
> 00:00.0 "PCI bridge" "Freescale Semiconductor Inc" "Device
0440" -r20 "" ""
> 01:00.0 "Power PC" "Freescale Semiconductor Inc" "Device
0440" -r20 "" ""
> 01:00.1 "Power PC" "Freescale Semiconductor Inc" "Device
0440" -r20 "" ""
> 01:00.4 "Power PC" "Freescale Semiconductor Inc" "Device
0000" -r20 "" ""
> 01:00.5 "Power PC" "Freescale Semiconductor Inc" "Device
1957" -r20 "" ""
> 01:01.0 "Power PC" "Freescale Semiconductor Inc" "Device
0000" -r20 "" ""
> 01:01.1 "Power PC" "Freescale Semiconductor Inc" "Device
1957" -r20 "" ""
> 0001:00:00.0 "PCI bridge" "Freescale Semiconductor Inc"
"Device 0440" -r20 "" ""
> 0002:00:00.0 "PCI bridge" "Freescale Semiconductor Inc"
"Device 0440" -r20 "" ""
What is the output of "virsh nodedev-dumpxml pci_0000_01_00_4"? It should
contain an element like this:
<capability type='phys_function'>
...
This indicates that it is an sriov VF (its PF will be listed in the above
<capability>
element). If this element is missing, there is something incomplete about the
SRIOV implementation for your card's driver.
It seemed that card driver is OK.
~# virsh nodedev-dumpxml pci_0000_01_00_4
<device>
<name>pci_0000_01_00_4</name>
<path>/sys/devices/ffe240000.pcie/pci0000:00/0000:00:00.0/0000:01:00.4</path>
<parent>pci_0000_00_00_0</parent>
<driver>
<name>fslinicvf</name>
</driver>
<capability type='pci'>
<domain>0</domain>
<bus>1</bus>
<slot>0</slot>
<function>4</function>
<product id='0x0000' />
<vendor id='0x1957' />
<capability type='phys_function'>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x0'/>
</capability>
<iommuGroup number='78'>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x4'/>
</iommuGroup>
</capability>
</device>
>
> # echo 1957 0000 > /sys/bus/pci/drivers/vfio-pci/new_id
> # echo 1957 1957 > /sys/bus/pci/drivers/vfio-pci/new_id
You shouldn't need to do the above when managed='yes', since libvirt does it
for
you (doing it manually like this may or may not have adverse consequences, I
haven't tried it)
OK, I got it.
>
> <interface type="hostdev" managed="yes">
> <mac address="00:e0:0c:00:20:01"/>
> <source>
> <address type="pci" domain="0x0000"
bus="0x01" slot="0x00"
function="0x04"/>
> </source>
> </interface>
> <memballoon model='virtio'/>
> </devices>
> </domain>
>
> root@t4240rdb:/var/volatile# virsh start interface
> error: Failed to start domain interface
> error: internal error: missing IFLA_VF_INFO in netlink response
At some point (it's been quite awhile, so I've forgotten exactly when) the
kernel
stopped automatically including the IFLA_VF_INFO object in the response to a
netlink RTM_GETLINK request. In order for this info to be included, you need to
add an IFLA_EXT_MASK with the RTEXT_FILTER_VF bit set into the RTM_GETLINK
request. This is done in
src/util/virnetdev.c:virNetDevLinkDump() - here are two bugzilla records
describing the problem and pointing at the patches added to fix it:
https://bugzilla.redhat.com/show_bug.cgi?id=889249
https://bugzilla.redhat.com/show_bug.cgi?id=889319
That particular code isn't compiled into libvirt unless RTEXT_FILTER_VF is #defined,
which should be done in /usr/include/linux/rtnetlink.h. If the kernel headers
used when building libvirt were missing that #define (or maybe instead of it being
#defined, it is defined as a value in an enum?), then RTEXT_FILTER_VF wouldn't
be set in the request. Of course if RTEXT_FILTER_VF really doesn't exist, then the
kernel should be returning IFLA_VF_INFO without needing the extra flag in the
request.
Hopefully something in here will point you at the source of your problem.
We use linux-3.12 with RTEXT_FILTER_VF defined.
$ grep -r RTEXT_FILTER_VF ./
./include/uapi/linux/rtnetlink.h:#define RTEXT_FILTER_VF (1 << 0)
So I guess your patches had been compiled into libvirt.
Best Regards,
Olivia
11:09 a.m.
New subject: [libvirt] Error starting domain: internal error: missing IFLA_VF_INFO in netlink response
Hi Laine,
I'm not sure if /usr/include/linux/rtnetlink.h is necessary in the filesystem on host
machine.
In our scenario, both host machine and iNIC card are both PowerPC boards.
I just checked the linux source code we used
$ grep -r RTEXT_FILTER_VF ./
./include/uapi/linux/rtnetlink.h:#define RTEXT_FILTER_VF (1 << 0)
But we didn't install the linux header files on the PowerPC host machine.
Best Regards,
Olivia
-----Original Message-----
From: Yin Olivia-R63875
Sent: Monday, January 12, 2015 5:48 PM
To: 'Laine Stump'
Cc: libvir-list(a)redhat.com
Subject: RE: Error starting domain: internal error: missing IFLA_VF_INFO in netlink
response
Hi Laine,
Thank you for the reply. I tried to dump the information.
> -----Original Message-----
> From: sendmail [mailto:justsendmailnothingelse@gmail.com] On Behalf Of
> Laine Stump
> Sent: Thursday, January 08, 2015 11:13 PM
> To: Yin Olivia-R63875
> Cc: libvir-list(a)redhat.com
> Subject: Re: Error starting domain: internal error: missing
> IFLA_VF_INFO in netlink response
>
> On 01/08/2015 04:40 AM, Hong-Hua.Yin(a)freescale.com wrote:
> > Hi Laine,
> >
> > Sorry to disturb you.
> > It seemed this issue had been fixed in
> > libvirt-1.2.2/libnl-3.2.22/linux-3.12. But
> we still got the error on PowerPC platform.
> > I'll appreciate if you could give any suggestion. We are not sure if
> > any netlink
> implementation in kernel space is missed.
> >
> > The scenario is a little complicated. We installed internal PF and
> > VF kernel modules and want to use <interface type="hostdev"
> > managed="yes">
> syntax to start a guest domain with MAC address.
> >
> > # insmod fslinic.ko max_vfs=2
> > Freescale 10 Gigabit PCI Express Network Driver fslinic 0000:01:00.0:
> > Multiqueue Enabled: Rx Queue count = 1, Tx Queue count = 1 fslinic
> > 0000:01:00.0: Freescale (R) 10 Gigabit Network Connection fslinic
> > 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 1, Tx Queue count
> > =
> > 1 fslinic 0000:01:00.1: Freescale (R) 10 Gigabit Network Connection
> >
> > # insmod fslinicvf.ko
> > Freescale 10 Gigabit PCI Express Network Driver
> >
> > # lspci -mk
> > 00:00.0 "PCI bridge" "Freescale Semiconductor Inc"
"Device 0440" -r20 "" ""
> > 01:00.0 "Power PC" "Freescale Semiconductor Inc"
"Device 0440" -r20 "" ""
> > 01:00.1 "Power PC" "Freescale Semiconductor Inc"
"Device 0440" -r20 "" ""
> > 01:00.4 "Power PC" "Freescale Semiconductor Inc"
"Device 0000" -r20 "" ""
> > 01:00.5 "Power PC" "Freescale Semiconductor Inc"
"Device 1957" -r20 "" ""
> > 01:01.0 "Power PC" "Freescale Semiconductor Inc"
"Device 0000" -r20 "" ""
> > 01:01.1 "Power PC" "Freescale Semiconductor Inc"
"Device 1957" -r20 "" ""
> > 0001:00:00.0 "PCI bridge" "Freescale Semiconductor Inc"
"Device 0440" -r20 ""
""
> > 0002:00:00.0 "PCI bridge" "Freescale Semiconductor Inc"
"Device 0440" -r20 ""
""
>
> What is the output of "virsh nodedev-dumpxml pci_0000_01_00_4"? It
> should contain an element like this:
>
> <capability type='phys_function'>
> ...
>
> This indicates that it is an sriov VF (its PF will be listed in the
> above <capability> element). If this element is missing, there is
> something incomplete about the SRIOV implementation for your card's driver.
>
>
It seemed that card driver is OK.
~# virsh nodedev-dumpxml pci_0000_01_00_4 <device>
<name>pci_0000_01_00_4</name>
<path>/sys/devices/ffe240000.pcie/pci0000:00/0000:00:00.0/0000:01:00.4</path>
<parent>pci_0000_00_00_0</parent>
<driver>
<name>fslinicvf</name>
</driver>
<capability type='pci'>
<domain>0</domain>
<bus>1</bus>
<slot>0</slot>
<function>4</function>
<product id='0x0000' />
<vendor id='0x1957' />
<capability type='phys_function'>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x0'/>
</capability>
<iommuGroup number='78'>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x4'/>
</iommuGroup>
</capability>
</device>
> >
> > # echo 1957 0000 > /sys/bus/pci/drivers/vfio-pci/new_id
> > # echo 1957 1957 > /sys/bus/pci/drivers/vfio-pci/new_id
>
>
> You shouldn't need to do the above when managed='yes', since libvirt
> does it for you (doing it manually like this may or may not have
> adverse consequences, I haven't tried it)
OK, I got it.
>
>
> >
> > <interface type="hostdev" managed="yes">
> > <mac address="00:e0:0c:00:20:01"/>
> > <source>
> > <address type="pci" domain="0x0000"
bus="0x01" slot="0x00"
> function="0x04"/>
> > </source>
> > </interface>
> > <memballoon model='virtio'/>
> > </devices>
> > </domain>
> >
> > root@t4240rdb:/var/volatile# virsh start interface
> > error: Failed to start domain interface
> > error: internal error: missing IFLA_VF_INFO in netlink response
>
> At some point (it's been quite awhile, so I've forgotten exactly when)
> the kernel stopped automatically including the IFLA_VF_INFO object in
> the response to a netlink RTM_GETLINK request. In order for this info
> to be included, you need to add an IFLA_EXT_MASK with the
> RTEXT_FILTER_VF bit set into the RTM_GETLINK request. This is done in
> src/util/virnetdev.c:virNetDevLinkDump() - here are two bugzilla
> records describing the problem and pointing at the patches added to fix it:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=889249
> https://bugzilla.redhat.com/show_bug.cgi?id=889319
>
> That particular code isn't compiled into libvirt unless
> RTEXT_FILTER_VF is #defined, which should be done in
> /usr/include/linux/rtnetlink.h. If the kernel headers used when
> building libvirt were missing that #define (or maybe instead of it
> being #defined, it is defined as a value in an enum?), then
> RTEXT_FILTER_VF wouldn't be set in the request. Of course if
> RTEXT_FILTER_VF really doesn't exist, then the kernel should be returning
IFLA_VF_INFO without needing the extra flag in the request.
>
> Hopefully something in here will point you at the source of your problem.
We use linux-3.12 with RTEXT_FILTER_VF defined.
$ grep -r RTEXT_FILTER_VF ./
./include/uapi/linux/rtnetlink.h:#define RTEXT_FILTER_VF (1 << 0)
So I guess your patches had been compiled into libvirt.
Best Regards,
Olivia
3:38 p.m.
New subject: [libvirt] Error starting domain: internal error: missing IFLA_VF_INFO in netlink response
On 01/12/2015 05:09 AM, Hong-Hua.Yin(a)freescale.com wrote:
Hi Laine,
I'm not sure if /usr/include/linux/rtnetlink.h is necessary in the filesystem on host
machine.
In our scenario, both host machine and iNIC card are both PowerPC boards.
I just checked the linux source code we used
$ grep -r RTEXT_FILTER_VF ./
./include/uapi/linux/rtnetlink.h:#define RTEXT_FILTER_VF (1 << 0)
But we didn't install the linux header files on the PowerPC host machine.
It's only necessary in the header file on the machine where you are
building libvirt. The machine that is *running* libvirt just needs the
proper support built into the kernel.
3538
days inactive
3542
days old
4 comments
2 participants
participants (2)
-
Hong-Hua.Yin＠freescale.com -
Laine Stump