Re: [libvirt] two hostdev devices problem

Hi, > First - I just got confirmation from the libnl maintainer that the bug I > mentioned causing problems with max_vfs > 50 *is* an issue in libnl > 3.2.16 - it is fixed in libnl 3.2.22. Try to upgrade to that version and > it should solve at least some of your problems. Upgrade libnl to version 3.2.22 helps. Thanks. > 1) when you have max_vfs = 10, you can assign the devices and mac > addresses are set properly, i.e. everything works. > 2) when you have max_vfs = 35, you can assign the devices, but the mac > addresses are not set properly. > 3) when you have max_vfs = 63, you can't assign the devices because you > get this error: > > internal error missing IFLA_VF_INFO in netlink response > > Is this correct? Yes. Before upgrade libnl :-) Upgrede helps only for limit max_vfs. > (BTW, it may get a bit confusing if you leave your networks named as > "vnet0" and "vnet1" - although it's a different namespace, libvirt uses > "vnetN" (where n is 0 - whatever) as the name for the tap device > associated with each guest interface). You're right. I changed this. > the output of "ip link show dev $PF". Hmm, > ip link show dev eth0 Shows only: 2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000 link/ether b8:ca:3a:5b:a6:3a brd ff:ff:ff:ff:ff:ff I see VFs like network interfaces eth2-eth127. When VF is attached to VM it disappear from list on host. -- Regards Dominik 2013/5/24 Laine Stump <laine(a)laine.org&gt;: > On 05/24/2013 09:16 AM, Dominik Mostowiec wrote: > >>>> Do you have this problem with mac addresses when max_vfs is set to an >> even lower number? > > > First - I just got confirmation from the libnl maintainer that the bug I > mentioned causing problems with max_vfs > 50 *is* an issue in libnl > 3.2.16 - it is fixed in libnl 3.2.22. Try to upgrade to that version and > it should solve at least some of your problems. > > >> >> No, >> When max_fs=10, macs in VM are OK. > > > Interesting. > > So now I'm curious about something - can you try adding a vlan tag to > your networks and see if the vlan tag is set properly when max_vfs is a > low number (10 or 7): > > <network> > <name>vnet0</name> > <uuid>ec49896a-a0b5-4944-a81f-9f0cdf578871</uuid> > <vlan> > <tag id='42'/> > </vlan> > <forward mode='hostdev' managed='yes'> > <pf dev='eth0'/> > </forward> > </network> > > > (you will have to do "virsh net-destroy vnet0; virsh net-start vnet0" > after making the modification to the network). If the vlan tag is > successfully set, it will show up next to the mac address for the VF in > the output of "ip link show dev $PF". > > (BTW, it may get a bit confusing if you leave your networks named as > "vnet0" and "vnet1" - although it's a different namespace, libvirt uses > "vnetN" (where n is 0 - whatever) as the name for the tap device > associated with each guest interface). > > To summarize: > > 1) when you have max_vfs = 10, you can assign the devices and mac > addresses are set properly, i.e. everything works. > 2) when you have max_vfs = 35, you can assign the devices, but the mac > addresses are not set properly. > 3) when you have max_vfs = 63, you can't assign the devices because you > get this error: > > internal error missing IFLA_VF_INFO in netlink response > > Is this correct? > >> >> Phisical interfaces: >>> ifconfig eth0 >> eth0 Link encap:Ethernet HWaddr b8:ca:3a:5b:a6:38 >> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 > > Okay, so that's not the problem. > >> >>> ifconfig eth1 >> eth1 Link encap:Ethernet HWaddr b8:ca:3a:5b:a6:38 >> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 >> >>>> Hmm. ixgbe. I just got a report yesterday that setting of the vlan tag >>>> for ixgbe vfs was not working properly (although it works fine for igb >>>> devices) >> Domain config: >> <interface type="network"> >> <alias name="hostdev0"/> >> <source network="vnet0"/> >> <mac address="52:54:0a:b1:48:fc"/> >> </interface> >> <interface type="network"> >> <alias name="hostdev1"/> >> <source network="vnet1"/> >> <mac address="52:54:0a:b1:48:fc"/> >> </interface> >> >> Networks defined: >> <network> >> <name>vnet0</name> >> <uuid>ec49896a-a0b5-4944-a81f-9f0cdf578871</uuid> >> <forward mode='hostdev' managed='yes'> >> <pf dev='eth0'/> >> </forward> >> </network> >> >> <network> >> <name>vnet1</name> >> <uuid>6c8319e8-8b53-4382-9f4e-2400819b00a9</uuid> >> <forward mode='hostdev' managed='yes'> >> <pf dev='eth1'/> >> </forward> >> </network> >> >> Regards >> Dominik >> >> 2013/5/24 Laine Stump <laine(a)laine.org&gt;: >>> On 05/24/2013 04:45 AM, Dominik Mostowiec wrote: >>>> Hi, >>>> I have libnl3, kernel 3.8.8, os ubuntu 12.04, net:intel10G. >>>> Upgrade libnl to 3.2.16 not helps, >>>> >>>> I don't know - this is probably another problem, when I have set >>>> max_vfs=35,35 VF is attaching to VM but mac in VM is not set >>>> propertly. >>> Do you have this problem with mac addresses when max_vfs is set to an >>> even lower number? >>> >>> Ah, another thing - did you "ifconfig up" the PF before you started the >>> guest? >>>> Libvirt probably set this macs because in syslog: >>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.171792] ixgbe >>> >>> Hmm. ixgbe. I just got a report yesterday that setting of the vlan tag >>> for ixgbe vfs was not working properly (although it works fine for igb >>> devices) >>> >>> >>>> 0000:01:00.0: setting MAC 52:54:0a:b1:48:f7 on VF 0 >>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.171799] ixgbe >>>> 0000:01:00.0: Reload the VF driver to make this change effective. >>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.175054] ixgbe >>>> 0000:01:00.1: setting MAC 52:54:0a:b1:48:f7 on VF 0 >>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.175060] ixgbe >>>> 0000:01:00.1: Reload the VF driver to make this change effective. >>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.307172] pci-stub >>>> 0000:01:12.4: enabling device (0000 -> 0002) >>>> May 23 14:49:00 on-10-177-32-62 kernel: [12575.662890] assign device 0:1:12.4 >>>> May 23 14:49:00 on-10-177-32-62 kernel: [12575.665210] pci-stub >>>> 0000:01:12.5: enabling device (0000 -> 0002) >>>> May 23 14:49:00 on-10-177-32-62 kernel: [12575.765609] assign device 0:1:12.5 >>>> But in VM i have another macs probably old from VF. >>>> >>>> --- >>>> Regards >>>> Dominik >>>> >>>> >>>> >>>> 2013/5/24 Laine Stump <laine(a)laine.org&gt;: >>>>> On 05/22/2013 06:32 PM, Dominik Mostowiec wrote: >>>>>> I tested on 1.0.5 patched version and vm with 2 vfs working fine. >>>>>> >>>>>> I have another problem >>>>>> When i set max_vfs=63: >>>>>> internal error missing IFLA_VF_INFO in netlink response >>>>>> >>>>>> Its working when max_vfs=31 >>>>>> >>>>> What is the platform, kernel version, and what version of libnl are you >>>>> using (libnl 1 or libnl3?) >>>>> >>>>> There have been multiple bugs recently filed and fixed in this area on >>>>> RHEL6/CentOS6. In particular, there was a problem with max_vfs > 50 that >>>>> had exactly this symptom. It was solved by a patch to libnl-1.1. If your >>>>> platform uses libnl-1.1, this could be the source of the problem. It is >>>>> fixed in the upstream libnl-1.1.4 maintenance release. >>>>> >>>>> I unfortunately am not sure where the libnl-1.1 git is (I only have the >>>>> git tree for libnl3), so I can't give you the exact commit message, but >>>>> in short the problem was that libnl was using too small of a buffer for >>>>> the netlink socket, so when there were a lot of vfs, the netlink message >>>>> containing vf info was truncated. >>>>> >>>>> >>>>>> -- >>>>>> Dominik >>>>>> >>>>>> 21 maj 2013 15:19, "Dominik Mostowiec" <dominikmostowiec(a)gmail.com >>>>>> <mailto:dominikmostowiec@gmail.com>> napisał(a): >>>>>> >>>>>> Hmm, >>>>>> It seems to be working (after only simple tests). >>>>>> >>>>>> -- >>>>>> Dominik >>>>>> >>>>>> >>>>>> 2013/5/21 Ján Tomko <jtomko(a)redhat.com <mailto:jtomko@redhat.com>> >>>>>> >>>>>> On 05/21/2013 01:37 PM, Laine Stump wrote: >>>>>> > On 05/21/2013 04:03 AM, Ján Tomko wrote: >>>>>> >> On 05/21/2013 09:32 AM, Dominik Mostowiec wrote: >>>>>> >>> hi, >>>>>> >>> I try to add 2 VF by "hostdev". >>>>>> >>> Networks (vnet0, vnet1) with: >>>>>> >>> <forward mode='hostdev' managed='yes'> >>>>>> >>> <pf dev='eth1'/> >>>>>> >>> ..... >>>>>> >>> >>>>>> >>> Domain: >>>>>> >>> <interface type="network"> >>>>>> >>> <source network="vnet0"/> >>>>>> >>> .... >>>>>> >>> <interface type="network"> >>>>>> >>> <source network="vnet1"/> >>>>>> >>> .... >>>>>> >>> >>>>>> >>> virsh create error: >>>>>> >>> "error: internal error process exited while connecting to >>>>>> monitor: kvm: >>>>>> >>> -device >>>>>> pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4: >>>>>> >>> Duplicate ID 'hostdev0' for device" >>>>>> >>> >>>>>> >> Hi, >>>>>> >> >>>>>> >> it seems we have been assigning the same id to all network >>>>>> hostdevs until this >>>>>> >> recent commit (not yet released): >>>>>> >> >>>>>> >> http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6597cc25 >>>>>> > >>>>>> > >>>>>> > If that patch has such an effect, it's purely accidental. >>>>>> Have you >>>>>> > tested this? (I plan to test a before and after as soon as >>>>>> I've had >>>>>> > breakfast) >>>>>> > >>>>>> >>>>>> I haven't tested it and looking at the code again, I might've >>>>>> been wrong :( >>>>>> >>>>>> Sorry about that. >>>>>> >>>>>> Jan >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Pozdrawiam >>>>>> Dominik >>>>>> >>>> >> >> >> -- >> Pozdrawiam >> Dominik >> > -- Pozdrawiam Dominik