6:25 a.m.
Hi Laine,
Thanks for taking the time to respond to my question. I think I have not described my
problem clearly.
Let me explain my issue below with the information that you had requested.
My assumption according to the information you gave me is that the admin MAC and VF MAC
are the same in my case.
I see a PF (GE0-0) interface but I don’t see a vfnetdev interface as you mentioned in your
email.
* Given this assumption, when the host is booted, the admin MAC and VF MAC are both
00:00:00:00:00:00.
[root@nfvis ~]# ip link show GE0-0
3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216<tel:9216> qdisc
mq master ovs-system state UP mode DEFAULT qlen 1000<tel:1000>
link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
[root@nfvis ~]#
* the VF is assigned to a guest, so the admin MAC is set to the configured value
(52:54:00:29:3c:bf) as shown below
[root@nfvis ~]# ip link show GE0-0
3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216<tel:9216> qdisc
mq master ovs-system state UP mode DEFAULT qlen 1000<tel:1000>
link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 52:54:00:29:3c:bf, spoof checking on, link-state auto, trust on
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on
[root@nfvis ~]#
* Configure bond interface on the guest and added member interface to the bond
On doing the above step, bond interface mac (52:54:00:29:3c:be) gets assigned to VF
[root@nfvis libvirt]# ip link show GE0-0
3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216<tel:9216> qdisc
mq master ovs-system state UP mode DEFAULT qlen 1000<tel:1000>
link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 52:54:00:29:3c:be, spoof checking on, link-state auto, trust on
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on
[root@nfvis libvirt]#
* The next step would be to restart the VM from within the VM console (* we are not doing
a shut down and start from the host *).
At this stage, the admin MAC is not reverted to the domain xml mac(52:54:00:29:3c:bf)
Instead it retains the same bond mac (52:54:00:29:3c:be) which was set before the VM
was restarted from the console.
[root@nfvis libvirt]# ip link show GE0-0
3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216<tel:9216> qdisc
mq master ovs-system state UP mode DEFAULT qlen 1000<tel:1000>
link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 52:54:00:29:3c:be, spoof checking on, link-state auto, trust on
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on
So the actual problem we face here, is after the VM is restarted and after the members of
bond interface are released, VF mac is not getting reverted back to domain xml
mac(52:54:00:29:3c:bf).
Instead the VF mac and admin mac are set to the bond mac “52:54:00:29:3c:be” and leading
to a duplicate mac issue.
My expectation was that after the VM is restarted from the console, libvirt should revert
the VF’s MAC to the original MAC,
which in our case was the domain xml MAC, which is not happening in my case.
Attached is the log file as requested.
Interface related contents from domain xml:
<interface type='hostdev' managed='yes'>
<mac address='52:54:00:29:3c:bf'/>
<driver name='vfio'/>
<source>
<address type='pci' domain='0x0000' bus='0x02'
slot='0x10' function='0x0'/>
</source>
<target dev='vnic1'/>
<model type='virtio'/>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x04' function='0x0'/>
</interface>
Thanks
Thiru.
From: sendmail <justsendmailnothingelse(a)gmail.com> On Behalf Of Laine Stump
Sent: Wednesday, May 2, 2018 10:01 AM
To: Thirunavukarasu Sengalvarayan -X (tsengalv - HCL TECHNOLOGIES LIMITED at Cisco)
<tsengalv(a)cisco.com>
Cc: Chanda Mendon (cmendon) <cmendon(a)cisco.com>
Subject: Re: VF MAC not reverted to all zero MAC on VM restart
On 04/27/2018 11:08 PM, Thirunavukarasu Sengalvarayan -X (tsengalv - HCL TECHNOLOGIES
LIMITED at Cisco) wrote:
Hi Laine Stump,
We are running linux based VM on KVM based Hypervisor and facing issue with respect to VF
MAC.
Our Libvirt version: 3.2.0, host driver IGB version 5.2.16 and VF driver IGBVF version
2.3.7.1
Description of problem:
When passing a VF to a guest, libvirt sets its MAC according to the domain xml.
On shutting down the VM or power off the VM, libvirt attempts to restore its original
MAC(all-zero mac). There is no issue with this scenario.
When we restart the VM, there is no attempt to restore/revert its original MAC(all zero
mac).
This problem leads to duplicate MAC issue on the guest (if configured with portchannel).
Could you please help us in resolving the issue?
Questions like this should be sent to
libvirt-users@redhat.com<mailto:libvirt-users@redhat.com> rather than to an
individual's email address. This makes the likelyhood of getting an answer much
higher, and also creates an archive of the problem and eventual solution, which may help
others in the future. I'm Cc'ing this response to that list so that any further
communication will happen there.
Before reading any of the following, it's useful to know this - for SRIOV network
devices that *properly* support SRIOV (and I consider those using the igb driver to be in
this category) each VF has two MAC addresses that need to be discussed:
1) the VF netdev MAC address (which I will just call the "mac address" - this is
the MAC that is known to the VF net driver, and displayed in the output of "ip link
show dev $vfnetdev".
2) the MAC address that is stored in the PF driver for each VF, displayed in the
"VF" lines immediately following the "ip link" info of the *PF*, and
which will be used to initialize the VF's own MAC address when its driver is
re-initialized (I will call this the "admin mac")
Some VF drivers initialize the mac to 00:00:00:00:00:00 (e.g. the Cisco enic driver) and
some initialize it to a random number (e.g. igb and all the other Intel VF net drivers).
Likewise, some PF drivers initialize the admin macs to 00:00:00:00:00:00 and some to
random numbers (as a matter of fact, this behavior changes between different versions of
the same driver in at least one case!).
Beyond this, some PF and VF drivers allow setting the mac/admin mac to 00:00:00:00:00:00,
and some prohibit one or the other. (in many cases, a driver that itself initializes the
mac/admin mac to 00:00:00:00:00:00 also prohibits setting it back to that same value once
it has been changed!)
When libvirt sets up a VF to be used by a guest (either for vfio device assignment, or for
macvtap passthrough), it saves both the mac and the admin mac with the intent of restoring
them later.
Once these values have been saved, libvirt will set the mac (in the case of macvtap
passthrough) or the admin mac (in the case of vfio device assignment), and send the device
on to qemu to be used by the guest.
After the guest is finished running (or the device is detached), libvirt attempts to
restore the settings it had saved at the beginning. Trouble comes up in 2 ways though - 1)
once the admin mac has been set for a VF (via the PF), it is no longer possible to
directly set the mac (in the VF) (this is done for security reasons, to prevent the guest
from changing its mac address), and 2) as said above, many drivers don't allow setting
00:00:00:00:00:00, but in many cases that was the original setting.
As of libvirt 3.2.0, libvirt uses the following strategies to work around these
problems:
a) if setting a non-0 mac (to the VF) fails, then libvirt will try setting the *admin MAC*
to that value, then detaching/re-attaching the VF net driver. This will cause the PF to
reinitialize the mac in the VF driver. (NB: if we do this, then we don't bother trying
to later re-set the admin MAC to its own original value; it's really not important).
b) if the failure was in setting the MAC to 00:00:00:00:00:00, then libvirt will set the
mac/admin mac to 02:00:00:00:00:00 (which should be a legal value for any driver, but also
should not conflict with and "real" mac on the network).
Okay, enough background. On to your problem.
To make sure I understand your situation correctly:
* When the host is booted, the admin MAC is 00:00:00:00:00:00, and VF mac is random (call
it rr:rr:rr:rr:rr:rr).
* the VF is assigned to a guest (with <interface type='hostdev'> I guess?),
so the admin MAC is set to the configured value (let's call it gg:gg:gg:gg:gg:gg)
* the guest is shutdown, which causes *both* mac and admin mac to be set to
rr:rr:rr:rr:rr:rr
* the next time the guest is start *its mac is not changed*?
There must be something I'm not getting. Can you maybe perform this test and run
"ip link show $PF; ip link show $VF" (substituting the netdev names of the PF
and VF for $PF and $VF) after each step to illustrate the problem? Also, it would be
helpful to add this to /etc/libvirt/libvirtd.conf:
log_filters="1:util.netdev"
log_outputs="1:file:/var/log/libvirt/libvirtd-netdev.log<file://var/log/libvirt/libvirtd-netdev.log>"
then restart libvirtd (systemctl restart libvirtd.service), and post the contents of
libvirtd-netdev.log somewhere and reference it in your next reply. (don't forget to
remove those lines and restart libvirtd again after your testing is done!)
Finally, you should include the contents of the <interface ...> section of your
config in your next response.
(BTW, since you're using libvirt-3.2.0, I assume that you are using either RHEL 7.4 or
CentOS 7.4. If you are using RHEL, you should open a customer case with Red Hat so that
your problem is appropriately prioritized).