Re: [libvirt] <interface type='hostdev'>vf configuration cleanup when VM is delete

-----Original Message----- From: sendmail [mailto:justsendmailnothingelse@gmail.com] On Behalf Of Laine Stump Sent: Wednesday, December 16, 2015 12:31 AM To: Libvirt <libvir-list(a)redhat.com&gt; Cc: vyasevic(a)redhat.com; Moshe Levi <moshele(a)mellanox.com&gt; Subject: Re: [libvirt] <interface type='hostdev'>vf configuration cleanup when VM is delete On 12/15/2015 04:12 PM, Vlad Yasevich wrote: > On 12/15/2015 02:45 PM, Laine Stump wrote: >> On 12/15/2015 01:34 PM, Laine Stump wrote: >>> On 12/13/2015 10:51 AM, Moshe Levi wrote: >>>> Hi, >>>> >>>> I have a setup with libvirt 1.3.0 and OpenStack trunk. >>>> >>>> Before launched the VM ip link command show the following VF >>>> mac/vlan configuration [1] >>>> >>>> When I launch a VM with <interface type='hostdev'> via openstack >>>> api (OpenStack direct >>>> port) >>>> >>>> I can see that the VF get the mac/vlan according to libvrit xml [2] >>>> and ip link command [3], but when I delete the VM the mac/vlan >>>> config are still shown as in [3] and not restored to [1] >>>> >>>> Shouldn’t libvirt restore the mac/vlan to [1]. >>>> >>>> The same problem exists when using <interface type='direct'> >>>> (OpenStack macvtap port) but just for the MAC configuration of the VF. >>>> >>> What libvirt does is to restore the MAC address to whatever it was >>> before we set it up for use with a guest. Although there are some >>> sriov net drivers that (for some unfathomable reason) think it's >>> cool to assign a random MAC address to each VF at boot time, the "normal" thing is for the VFs to have a MAC address of all 0's to start with. >>> So libvirt should be saving 00:00:00:00:00:00 (it will be in the >>> file >>> /var/run/libvirt/hostdevmgr/$ifname_vf$vfnum) then setting the MAC >>> to use; when done, libvirt will read the 00:00:00:00:00:00 and use >>> netlink to set the MAC address, but this is apparently failing. >>> >>> I checked on my Fedora 22 system with the igb driver, and found that >>> if the MAC address was originally set to something other than 0's, >>> it was restored properly by libvirt, but if it was set to all 0's originally, the attempt to set it back to 0 would fail. >>> >>> I then tried doing the same thing with the "ip" utility: >>> >>> # ip link set dev p4p2 vf 0 mac 00:00:00:00:00:00 >>> >>> and I get the following response: >>> >>> RTNETLINK answers: Invalid argument >>> >>> So it appears that either the kernel or the NIC driver is refusing >>> to set the MAC address to all 0's. I'm reasonably certain this is a >>> regression in the kernel, >> Sigh. It appears that this has "always" been the case - I just >> checked on a 2.6.32-573 RHEL kernel, and a 3.10.x RHEL7.2 kernel, and >> 4.1 (Fedora 22) and both of them also refuse to set the MAC address >> to 00:00:00:00:00:00. I'm not sure if this limitation is in the NIC driver or some basic code in the kernel. > It's considered to be an invalid address by is_valid_ether_addr() function. > > There appear to be different behaviour in some adapters. In current > upstream, it looks like a quarter of the VF capable drivers (bnxt, > enic, fm10k, sfc) permit VF mac setting of all zeros. The others > simply use is_valid_ether_addr function without specifically testing > for all-0. I am not sure if this is HW related or simply oversights... Might want to bringing this up on netdev. Thanks, Vlad! Moshe, It sounds like in your case it is caused by code in the mlx driver, so hopefully you can have some influence there. My path is a bit more difficult, as the failure on mine is in the igb driver :-)

Sending a message to netdev is a good idea. It would be wonderful if all the vendors could agree to: 1) initialize all VFs with a MAC address of 0 2) allow setting VF MAC address to 0 at any time. I'll put that on my to-do list :-P

> -vlad > >> >>> although I can't say how long it's been there, as I don't normally >>> pay attention to this (and as I said, many SRIOV NIC drivers don't >>> default their VFs to 0 MAC addresses) >>> >>> What distro and kernel are you using for your tests? >>> >>> >>>> [1] - 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 >>>> qdisc mq master ovs-system state UP mode DEFAULT group default qlen >>>> 1000 >>>> >>>> link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff >>>> >>>> vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state >>>> auto >>>> >>>> vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state >>>> auto >>>> >>>> vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state >>>> auto >>>> >>>> vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state >>>> auto >>>> >>>> [2] - <interface type='hostdev' managed='yes'> >>>> >>>> <mac address=' fa:16:3e:11:af:fe '/> >>>> >>>> <driver name='kvm'/> >>>> >>>> <source> >>>> >>>> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' >>>> function='0x7'/> >>>> >>>> </source> >>>> >>>> <vlan> >>>> >>>> <tag id='190'/> >>>> >>>> </vlan> >>>> >>>> <alias name='hostdev0'/> >>>> >>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' >>>> function='0x0'/> >>>> >>>> </interface> >>>> >>>> [3] 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc >>>> mq master ovs-system state UP mode DEFAULT group default qlen 1000 >>>> >>>> link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff >>>> >>>> vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state >>>> auto >>>> >>>> vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state >>>> auto >>>> >>>> vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state >>>> auto >>>> >>>> vf 3 MAC fa:16:3e:11:af:fe, vlan 190, spoof checking off, >>>> link-state enable >>>> >>>> >>>> >>>> -- >>>> libvir-list mailing list >>>> libvir-list(a)redhat.com >>>> https://www.redhat.com/mailman/listinfo/libvir-list >>> F15 >>> >>> >>> -- >>> libvir-list mailing list >>> libvir-list(a)redhat.com >>> https://www.redhat.com/mailman/listinfo/libvir-list >> >