Re: [libvirt] <interface type='hostdev'>vf configuration cleanup when VM is delete

On 12/15/2015 02:45 PM, Laine Stump wrote: > On 12/15/2015 01:34 PM, Laine Stump wrote: >> On 12/13/2015 10:51 AM, Moshe Levi wrote: >>> Hi, >>> >>> I have a setup with libvirt 1.3.0 and OpenStack trunk. >>> >>> Before launched the VM ip link command show the following VF mac/vlan configuration [1] >>> >>> When I launch a VM with <interface type='hostdev'> via openstack api (OpenStack direct >>> port) >>> >>> I can see that the VF get the mac/vlan according to libvrit xml [2] and ip link >>> command [3], but when I delete the VM the mac/vlan config are still shown as in [3] >>> and not restored to [1] >>> >>> Shouldn’t libvirt restore the mac/vlan to [1]. >>> >>> The same problem exists when using <interface type='direct'> (OpenStack macvtap port) >>> but just for the MAC configuration of the VF. >>> >> What libvirt does is to restore the MAC address to whatever it was before we set it up >> for use with a guest. Although there are some sriov net drivers that (for some >> unfathomable reason) think it's cool to assign a random MAC address to each VF at boot >> time, the "normal" thing is for the VFs to have a MAC address of all 0's to start with. >> So libvirt should be saving 00:00:00:00:00:00 (it will be in the file >> /var/run/libvirt/hostdevmgr/$ifname_vf$vfnum) then setting the MAC to use; when done, >> libvirt will read the 00:00:00:00:00:00 and use netlink to set the MAC address, but this >> is apparently failing. >> >> I checked on my Fedora 22 system with the igb driver, and found that if the MAC address >> was originally set to something other than 0's, it was restored properly by libvirt, but >> if it was set to all 0's originally, the attempt to set it back to 0 would fail. >> >> I then tried doing the same thing with the "ip" utility: >> >> # ip link set dev p4p2 vf 0 mac 00:00:00:00:00:00 >> >> and I get the following response: >> >> RTNETLINK answers: Invalid argument >> >> So it appears that either the kernel or the NIC driver is refusing to set the MAC >> address to all 0's. I'm reasonably certain this is a regression in the kernel, > Sigh. It appears that this has "always" been the case - I just checked on a 2.6.32-573 > RHEL kernel, and a 3.10.x RHEL7.2 kernel, and 4.1 (Fedora 22) and both of them also refuse > to set the MAC address to 00:00:00:00:00:00. I'm not sure if this limitation is in the NIC > driver or some basic code in the kernel. It's considered to be an invalid address by is_valid_ether_addr() function. There appear to be different behaviour in some adapters. In current upstream, it looks like a quarter of the VF capable drivers (bnxt, enic, fm10k, sfc) permit VF mac setting of all zeros. The others simply use is_valid_ether_addr function without specifically testing for all-0. I am not sure if this is HW related or simply oversights... Might want to bringing this up on netdev.

-vlad > >> although I can't say how long it's been there, as I don't normally pay attention to this >> (and as I said, many SRIOV NIC drivers don't default their VFs to 0 MAC addresses) >> >> What distro and kernel are you using for your tests? >> >> >>> [1] - 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master >>> ovs-system state UP mode DEFAULT group default qlen 1000 >>> >>> link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff >>> >>> vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto >>> >>> vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto >>> >>> vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto >>> >>> vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto >>> >>> [2] - <interface type='hostdev' managed='yes'> >>> >>> <mac address=' fa:16:3e:11:af:fe '/> >>> >>> <driver name='kvm'/> >>> >>> <source> >>> >>> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x7'/> >>> >>> </source> >>> >>> <vlan> >>> >>> <tag id='190'/> >>> >>> </vlan> >>> >>> <alias name='hostdev0'/> >>> >>> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> >>> >>> </interface> >>> >>> [3] 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system >>> state UP mode DEFAULT group default qlen 1000 >>> >>> link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff >>> >>> vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto >>> >>> vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto >>> >>> vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto >>> >>> vf 3 MAC fa:16:3e:11:af:fe, vlan 190, spoof checking off, link-state enable >>> >>> >>> >>> -- >>> libvir-list mailing list >>> libvir-list(a)redhat.com >>> https://www.redhat.com/mailman/listinfo/libvir-list >> F15 >> >> >> -- >> libvir-list mailing list >> libvir-list(a)redhat.com >> https://www.redhat.com/mailman/listinfo/libvir-list >