Re: [libvirt] <interface type='direct'>

-----Original Message----- From: sendmail [mailto:justsendmailnothingelse@gmail.com] On Behalf Of Laine Stump Sent: Thursday, September 01, 2016 5:59 PM To: Libvirt <libvir-list(a)redhat.com&gt; Cc: Moshe Levi <moshele(a)mellanox.com>; Edan David <edand(a)mellanox.com&gt; Subject: Re: [libvirt] <interface type='direct'> On 09/01/2016 04:05 AM, Moshe Levi wrote: > Hi, > > In OpenStack we have a port type macvtap. > Mavtap port is just a tap device connected to VF. > > In Libvirt the guest xml look like > <interface type='direct'> > <mac address='fa:16:3e:b1:06:4e'/> > <source dev='p1p6' mode='passthrough'/> > <target dev='macvtap1'/> > <model type='virtio'/> > <driver name='vhost'/> > <alias name='net0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' > function='0x0'/> </interface> > > > In the hypervisor we can see that the mac of the VF which is > fa:16:3e:f3:9b:e8 - is set by OpenStack see [1] > 9: ens3f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000 > link/ether 7c:fe:90:29:24:4e brd ff:ff:ff:ff:ff:ff > vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state disable > vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state disable > vf 2 MAC fa:16:3e:f3:9b:e8, vlan 48, spoof checking on, link-state enable > vf 3 MAC fa:16:3e:f6:02:c8, vlan 48, spoof checking on, > link-state enable > 41: ens3f4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 > link/ether fa:16:3e:f6:02:c8 brd ff:ff:ff:ff:ff:ff > 42: macvtap0@ens3f4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 500 > link/ether fa:16:3e:f6:02:c8, brd ff:ff:ff:ff:ff:ff > > The netdevice of the VF which is ens3f4 has also the same mac. This > mac is set when using Libvirt 1.2.2 (Ubuntu 14.04), But when we tested with new Libvirt versions >= 1.2.17 (Fedora 23/Ubuntu 16.04) the mac netdevice of the VF (ens3f4) is not set. > This change in Libvirt breaks the guest from getting DHCP in OpenStack. > Do you know why the behavior change in newer releases? The MAC address is now set with a netlink command to set the VFINFO of the particular VF# of the PF. This change was made in response to a bug report stating that once the MAC address had been set for a hostdev assignment of a VF (in which case this method is required), it was no longer possible to set the MAC address for macvtap passthrough (the VF driver would complain "MAC has been administratively set", on Intel igbvf at least). Unfortunately I recently found that when you set the MAC address in this manner, it doesn't take effect on the actual device - it's only saved in memory to be applied the *next time* the host driver is rebound to the VF.

Since I don't see a reasonably efficient way to get this to work, I need to make a patch to revert to the old behavior, and we'll then just have to tell people "If you do hostdev device assignment of VFs, then you can't later re- use the same device for macvtap passthrough mode". (actually, I *think* an alternative would be to unbind/rebind the host driver to the VF after setting the VF MAC address, but that seems a bit disruptive/extreme to work around a problem that is probably only seen in QE labs, but not in the real world (realistically, production systems likely use either hostdev or macvtap, and don't switch back and forth between them). A question - I notice you have the vlan set for the VF. Does *that* properly take effect? (it's set in the same manner as the MAC address, via a netlink command to set the VFINFO)

> We have a WIP patch in OpenStack for setting also the mac for the netdevice of the VF [2]. Just wanted to know that this is the correct approach.

> > > [1] - > https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py > #L616-L621 [2] - https://review.openstack.org/#/c/364121/ > > > > Thanks, > Moshe Levi > > > > -- > libvir-list mailing list > libvir-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/libvir-list >