On 05/04/2018 06:25 PM, Thirunavukarasu Sengalvarayan -X (tsengalv - HCL
TECHNOLOGIES LIMITED at Cisco) wrote:
Hi Laine,
Thanks for taking the time to respond to my question. I think I have
not described my problem clearly.
Let me explain my issue below with the information that you had requested.
My assumption according to the information you gave me is that the
admin MAC and VF MAC are the same in my case.
I see a PF (GE0-0) interface but I don’t see a vfnetdev interface as
you mentioned in your email.
If the VF has no separate netdev visible on the host, then it is not
being bound to the VF net driver for some reason. This should cause no
ill effect - it just means there is no "original" VF mac to restore.
* Given this assumption, when the host is booted, the admin MAC and VF
MAC are both*00:00:00:00:00:00. *
Well, if there is no VF netdev on the host, there is no "vf mac" (and
the igb driver wouldn't allow it to be 00:00:00:00:00:00). But again,
not important
**
[root@nfvis ~]# ip link show GE0-0
"GEO-0"? That's not a normal name for an igb PF. Are you manually
selecting this name?
3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216
<tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
<tel:1000>
link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
[root@nfvis ~]#
* the VF is assigned to a guest, so the admin MAC is set to the
configured value *(52:54:00:29:3c:bf) *as shown below
[root@nfvis ~]# ip link show GE0-0
3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216
<tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
<tel:1000>
link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
vf 0 MAC *52:54:00:29:3c:bf*, spoof checking on, link-state auto, trust on
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on
[root@nfvis ~]#
* Configure bond interface on the guest and added member interface to
the bond
On doing the above step, bond interface mac*(52:54:00:29:3c:be)*
gets assigned to VF
[root@nfvis libvirt]# ip link show GE0-0
3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216
<tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
<tel:1000>
link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
vf 0 MAC *52:54:00:29:3c:be*, spoof checking on, link-state auto, trust on
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on
[root@nfvis libvirt]#
Okay, this part makes no sense, for 2 reasons:
1) once the admin MAC has been set by the host, a flag is set in the PF
driver marking this VF as having an "administratively set" MAC (that's
why I call it the "admin mac"), and after that point it should not be
possible for a guest to modify the MAC address. If your guest is
successfully setting the MAC of the VF, then either you don't have a
device that uses the igb driver, or there is a bug in the driver.
2) setting the MAC address of the VF shouldn't update the admin mac for
that VF - they are separate entities, and only synched up when the VF
driver is reloaded (and in that case it is the *admin mac* that is used
as the master copy to set both of them)
* The next step would be to restart the VM from within the VM console
(* we are not doing a shut down and start from the host *).
Okay, so that reloads the VF driver in the guest, which would set it to
the admin mac (which you have said showed to be ....:be just before the
reset.
At this stage, the admin MAC is not reverted to the domain xml
mac*(52:54:00:29:3c:bf)*
Instead it retains the same bond mac *(52:54:00:29:3c:be)* which
was set before the VM was restarted from the console. **
[root@nfvis libvirt]# ip link show GE0-0
3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216
<tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
<tel:1000>
link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
vf 0 MAC *52:54:00:29:3c:be*, spoof checking on, link-state auto, trust on
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on
So the actual problem we face here, is after the VM is restarted and
after the members of bond interface are released, VF mac is not
getting reverted back to domain xml mac(*52:54:00:29:3c:bf*).
Instead the VF mac and admin mac are set to the bond mac
“*52:54:00:29:3c:be*” and leading to a duplicate mac issue.
My expectation was that after the VM is restarted from the console,
libvirt should revert the VF’s MAC to the original MAC,
which in our case was the domain xml MAC, which is not happening in my
case.
My expectation was that the guest would not be allowed to modify the MAC
address of the VF at all, and certainly that even if a change to the
guest MAC was allowed, that it wouldn't propagate to the PF's list of
admin mac addresses for the VFs. So we've both been disappointed :-)
Your VF and PF drivers are behaving strangely. But if you completely
shutdown the guest, then restart it, you should get the original
"...:bf" back. Even if that works for you, I would not count on the
ability of the guest to modify the VF's MAC address - that is not the
defined behavior and thus is likely to change in the future.
Attached is the log file as requested.
*_Interface related contents from domain xml: _*
<interface type='hostdev' managed='yes'>
<mac address='52:54:00:29:3c:bf'/>
<driver name='vfio'/>
<source>
<address type='pci' domain='0x0000' bus='0x02'
slot='0x10'
function='0x0'/>
</source>
<target dev='vnic1'/>
<model type='virtio'/>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x04'
function='0x0'/>
</interface>
Thanks
Thiru.
*From:*sendmail <justsendmailnothingelse(a)gmail.com> *On Behalf Of
*Laine Stump
*Sent:* Wednesday, May 2, 2018 10:01 AM
*To:* Thirunavukarasu Sengalvarayan -X (tsengalv - HCL TECHNOLOGIES
LIMITED at Cisco) <tsengalv(a)cisco.com>
*Cc:* Chanda Mendon (cmendon) <cmendon(a)cisco.com>
*Subject:* Re: VF MAC not reverted to all zero MAC on VM restart
On 04/27/2018 11:08 PM, Thirunavukarasu Sengalvarayan -X (tsengalv -
HCL TECHNOLOGIES LIMITED at Cisco) wrote:
Hi Laine Stump,
We are running linux based VM on KVM based Hypervisor and facing
issue with respect to VF MAC.
Our Libvirt version: 3.2.0, host driver IGB version 5.2.16 and VF
driver IGBVF version 2.3.7.1
*_Description of problem:_*
When passing a VF to a guest, libvirt sets its MAC according to
the domain xml.
On shutting down the VM or power off the VM, libvirt attempts to
restore its original MAC(all-zero mac). There is no issue with
this scenario.
*When we restart the VM, there is no attempt to restore/revert its
original MAC(all zero mac). *
This problem leads to duplicate MAC issue on the guest (if
configured with portchannel).
Could you please help us in resolving the issue?
Questions like this should be sent to libvirt-users(a)redhat.com
<mailto:libvirt-users@redhat.com>rather than to an individual's email
address. This makes the likelyhood of getting an answer much higher,
and also creates an archive of the problem and eventual solution,
which may help others in the future. I'm Cc'ing this response to that
list so that any further communication will happen there.
Before reading any of the following, it's useful to know this - for
SRIOV network devices that *properly* support SRIOV (and I consider
those using the igb driver to be in this category) each VF has two MAC
addresses that need to be discussed:
1) the VF netdev MAC address (which I will just call the "mac address"
- this is the MAC that is known to the VF net driver, and displayed in
the output of "ip link show dev $vfnetdev".
2) the MAC address that is stored in the PF driver for each VF,
displayed in the "VF" lines immediately following the "ip link" info
of the *PF*, and which will be used to initialize the VF's own MAC
address when its driver is re-initialized (I will call this the "admin
mac")
Some VF drivers initialize the mac to 00:00:00:00:00:00 (e.g. the
Cisco enic driver) and some initialize it to a random number (e.g. igb
and all the other Intel VF net drivers). Likewise, some PF drivers
initialize the admin macs to 00:00:00:00:00:00 and some to random
numbers (as a matter of fact, this behavior changes between different
versions of the same driver in at least one case!).
Beyond this, some PF and VF drivers allow setting the mac/admin mac to
00:00:00:00:00:00, and some prohibit one or the other. (in many cases,
a driver that itself initializes the mac/admin mac to
00:00:00:00:00:00 also prohibits setting it back to that same value
once it has been changed!)
When libvirt sets up a VF to be used by a guest (either for vfio
device assignment, or for macvtap passthrough), it saves both the mac
and the admin mac with the intent of restoring them later.
Once these values have been saved, libvirt will set the mac (in the
case of macvtap passthrough) or the admin mac (in the case of vfio
device assignment), and send the device on to qemu to be used by the
guest.
After the guest is finished running (or the device is detached),
libvirt attempts to restore the settings it had saved at the
beginning. Trouble comes up in 2 ways though - 1) once the admin mac
has been set for a VF (via the PF), it is no longer possible to
directly set the mac (in the VF) (this is done for security reasons,
to prevent the guest from changing its mac address), and 2) as said
above, many drivers don't allow setting 00:00:00:00:00:00, but in many
cases that was the original setting.
As of libvirt 3.2.0, libvirt uses the following strategies to work
around these problems:
a) if setting a non-0 mac (to the VF) fails, then libvirt will try
setting the *admin MAC* to that value, then detaching/re-attaching the
VF net driver. This will cause the PF to reinitialize the mac in the
VF driver. (NB: if we do this, then we don't bother trying to later
re-set the admin MAC to its own original value; it's really not
important).
b) if the failure was in setting the MAC to 00:00:00:00:00:00, then
libvirt will set the mac/admin mac to 02:00:00:00:00:00 (which should
be a legal value for any driver, but also should not conflict with and
"real" mac on the network).
Okay, enough background. On to your problem.
To make sure I understand your situation correctly:
* When the host is booted, the admin MAC is 00:00:00:00:00:00, and VF
mac is random (call it rr:rr:rr:rr:rr:rr).
* the VF is assigned to a guest (with <interface type='hostdev'> I
guess?), so the admin MAC is set to the configured value (let's call
it gg:gg:gg:gg:gg:gg)
* the guest is shutdown, which causes *both* mac and admin mac to be
set to rr:rr:rr:rr:rr:rr
* the next time the guest is start *its mac is not changed*?
There must be something I'm not getting. Can you maybe perform this
test and run "ip link show $PF; ip link show $VF" (substituting the
netdev names of the PF and VF for $PF and $VF) after each step to
illustrate the problem? Also, it would be helpful to add this to
/etc/libvirt/libvirtd.conf:
log_filters="1:util.netdev"
log_outputs="1:file:/var/log/libvirt/libvirtd-netdev.log
<file://var/log/libvirt/libvirtd-netdev.log>"
then restart libvirtd (systemctl restart libvirtd.service), and post
the contents of libvirtd-netdev.log somewhere and reference it in your
next reply. (don't forget to remove those lines and restart libvirtd
again after your testing is done!)
Finally, you should include the contents of the <interface ...>
section of your config in your next response.
(BTW, since you're using libvirt-3.2.0, I assume that you are using
either RHEL 7.4 or CentOS 7.4. If you are using RHEL, you should open
a customer case with Red Hat so that your problem is appropriately
prioritized).