Re: [libvirt] <interface type='direct'>

On 09/03/2016 11:08 AM, Moshe Levi wrote: Look at the explanation in commit cb3fe38c and also https://bugzilla.redhat.com/show_bug.cgi?id=1113474 That commit switched from using a simple ioctl(SIOCSIFHWADDR) to the VF's netdev name, to using a netlink RTM_SETLINK message to the netdev of the *PF* for the given VF. This was done because the latter is the *only* way you can set the MAC address for a VF that you're going to assign to the guest with vfio device assignment, and once you've set the MAC address that way, future attempts to set the MAC address with ioctl(SIOCSIGHWADDR) result in failure and a kernel log like this: kernel: igb 0000:0e:00.1: VF 1 attempted to override administratively set MAC address kernel: Reload the VF driver to resume operations Looking into the kernel, it appears that once the MAC address for a VF has been set via RTM_SETLINK, the igb driver (and I believe also the ixgbe driver, not sure about others) doesn't allow it to be changed via ioctl until the PF driver is reloaded (which can't realistically be done on an active system) But recently there was another report of the MAC address not getting set properly for macvtap passthrough mode when the device is an SRIOV VF (I can't find it in bugzilla, so it must have been an email to one of the lists) and when I tried it myself I found they were correct - in the output of "ip link show" the MAC address showed in the list of VFs under the PF is correctly modified, but it's not set properly in the VF's netdev instance - apparently the MAC addresses in the VF list aren't set in the VF's netdev immediately, they're just saved to be set *the next time the VF is re-bound to the VF netdev driver*. I think in the past the interface may have been in promiscuous mode so it didn't matter, but now it isn't? I'm not sure as I haven't had much time to investigate. Does that make any more sense now? What command do you use to set it? Do you use "ip link set $PF vf $VF# vlan $VLANID" ? I think that's what it's showing here: https://review.openstack.org/#/c/364121/1/nova/network/linux_net.py (I don't know my way around openstack code, but arrived at that page via clicking on links from a google search) libvirt *should* do it. If libvirt isn't getting the job done, and you can set it yourself, then that's a workaround. I don't know that I'd call it "reasaonable" though. If everybody puts in special code to workaround bugs in libvirt (which is apparently what's been done) rather than actually reporting the bug (what you're doing now - Thanks!) then we are tricked into thinking that either the code works, or that nobody is using it so it doesn't matter if it's broken. The *best* way of overcoming this problem is to fix libvirt so it does what it's supposed to do. It's possible we can make it work by adding some operation after we send the RTM_SETLINK (maybe unbind the VF from its netdev driver, then re-bind, but that seems so drastic and time consuming!), or maybe we'll have to revert to using ioctl(SIOCSIFHWADDR), but of course that will fail if the interface has been used for hostdev assignment since the last host reboot. It's interesting that openstack is apparently using the RTM_SETLINK method to set the mac address (afaik, that's what is used by the "ip link set $pf_ifname vf $vf_num mac $mac_addr vlan $vlanid" command that's shown in the bit of code from nova/network/linux_net.py at the link I posted above).