[libvirt] <interface type='hostdev'>vf configuration cleanup when VM is delete
Hi, I have a setup with libvirt 1.3.0 and OpenStack trunk. Before launched the VM ip link command show the following VF mac/vlan configuration [1] When I launch a VM with <interface type='hostdev'> via openstack api (OpenStack direct port) I can see that the VF get the mac/vlan according to libvrit xml [2] and ip link command [3], but when I delete the VM the mac/vlan config are still shown as in [3] and not restored to [1] Shouldn't libvirt restore the mac/vlan to [1]. The same problem exists when using <interface type='direct'> (OpenStack macvtap port) but just for the MAC configuration of the VF. [1] - 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000 link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto [2] - <interface type='hostdev' managed='yes'> <mac address=' fa:16:3e:11:af:fe '/> <driver name='kvm'/> <source> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x7'/> </source> <vlan> <tag id='190'/> </vlan> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </interface> [3] 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000 link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto vf 3 MAC fa:16:3e:11:af:fe, vlan 190, spoof checking off, link-state enable
On 12/13/2015 10:51 AM, Moshe Levi wrote:
Hi,
I have a setup with libvirt 1.3.0 and OpenStack trunk.
Before launched the VM ip link command show the following VF mac/vlan configuration [1]
When I launch a VM with <interface type='hostdev'> via openstack api (OpenStack direct port)
I can see that the VF get the mac/vlan according to libvrit xml [2] and ip link command [3], but when I delete the VM the mac/vlan config are still shown as in [3] and not restored to [1]
Shouldn’t libvirt restore the mac/vlan to [1].
The same problem exists when using <interface type='direct'> (OpenStack macvtap port) but just for the MAC configuration of the VF.
What libvirt does is to restore the MAC address to whatever it was before we set it up for use with a guest. Although there are some sriov net drivers that (for some unfathomable reason) think it's cool to assign a random MAC address to each VF at boot time, the "normal" thing is for the VFs to have a MAC address of all 0's to start with. So libvirt should be saving 00:00:00:00:00:00 (it will be in the file /var/run/libvirt/hostdevmgr/$ifname_vf$vfnum) then setting the MAC to use; when done, libvirt will read the 00:00:00:00:00:00 and use netlink to set the MAC address, but this is apparently failing. I checked on my Fedora 22 system with the igb driver, and found that if the MAC address was originally set to something other than 0's, it was restored properly by libvirt, but if it was set to all 0's originally, the attempt to set it back to 0 would fail. I then tried doing the same thing with the "ip" utility: # ip link set dev p4p2 vf 0 mac 00:00:00:00:00:00 and I get the following response: RTNETLINK answers: Invalid argument So it appears that either the kernel or the NIC driver is refusing to set the MAC address to all 0's. I'm reasonably certain this is a regression in the kernel, although I can't say how long it's been there, as I don't normally pay attention to this (and as I said, many SRIOV NIC drivers don't default their VFs to 0 MAC addresses) What distro and kernel are you using for your tests?
[1] - 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
[2] - <interface type='hostdev' managed='yes'>
<mac address=' fa:16:3e:11:af:fe '/>
<driver name='kvm'/>
<source>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x7'/>
</source>
<vlan>
<tag id='190'/>
</vlan>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</interface>
[3] 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 3 MAC fa:16:3e:11:af:fe, vlan 190, spoof checking off, link-state enable
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list F15
On 12/15/2015 01:34 PM, Laine Stump wrote:
On 12/13/2015 10:51 AM, Moshe Levi wrote:
Hi,
I have a setup with libvirt 1.3.0 and OpenStack trunk.
Before launched the VM ip link command show the following VF mac/vlan configuration [1]
When I launch a VM with <interface type='hostdev'> via openstack api (OpenStack direct port)
I can see that the VF get the mac/vlan according to libvrit xml [2] and ip link command [3], but when I delete the VM the mac/vlan config are still shown as in [3] and not restored to [1]
Shouldn’t libvirt restore the mac/vlan to [1].
The same problem exists when using <interface type='direct'> (OpenStack macvtap port) but just for the MAC configuration of the VF.
What libvirt does is to restore the MAC address to whatever it was before we set it up for use with a guest. Although there are some sriov net drivers that (for some unfathomable reason) think it's cool to assign a random MAC address to each VF at boot time, the "normal" thing is for the VFs to have a MAC address of all 0's to start with. So libvirt should be saving 00:00:00:00:00:00 (it will be in the file /var/run/libvirt/hostdevmgr/$ifname_vf$vfnum) then setting the MAC to use; when done, libvirt will read the 00:00:00:00:00:00 and use netlink to set the MAC address, but this is apparently failing.
I checked on my Fedora 22 system with the igb driver, and found that if the MAC address was originally set to something other than 0's, it was restored properly by libvirt, but if it was set to all 0's originally, the attempt to set it back to 0 would fail.
I then tried doing the same thing with the "ip" utility:
# ip link set dev p4p2 vf 0 mac 00:00:00:00:00:00
and I get the following response:
RTNETLINK answers: Invalid argument
So it appears that either the kernel or the NIC driver is refusing to set the MAC address to all 0's. I'm reasonably certain this is a regression in the kernel,
Sigh. It appears that this has "always" been the case - I just checked on a 2.6.32-573 RHEL kernel, and a 3.10.x RHEL7.2 kernel, and 4.1 (Fedora 22) and both of them also refuse to set the MAC address to 00:00:00:00:00:00. I'm not sure if this limitation is in the NIC driver or some basic code in the kernel.
although I can't say how long it's been there, as I don't normally pay attention to this (and as I said, many SRIOV NIC drivers don't default their VFs to 0 MAC addresses)
What distro and kernel are you using for your tests?
[1] - 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
[2] - <interface type='hostdev' managed='yes'>
<mac address=' fa:16:3e:11:af:fe '/>
<driver name='kvm'/>
<source>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x7'/>
</source>
<vlan>
<tag id='190'/>
</vlan>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</interface>
[3] 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 3 MAC fa:16:3e:11:af:fe, vlan 190, spoof checking off, link-state enable
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list F15
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On 12/15/2015 02:45 PM, Laine Stump wrote:
On 12/15/2015 01:34 PM, Laine Stump wrote:
On 12/13/2015 10:51 AM, Moshe Levi wrote:
Hi,
I have a setup with libvirt 1.3.0 and OpenStack trunk.
Before launched the VM ip link command show the following VF mac/vlan configuration [1]
When I launch a VM with <interface type='hostdev'> via openstack api (OpenStack direct port)
I can see that the VF get the mac/vlan according to libvrit xml [2] and ip link command [3], but when I delete the VM the mac/vlan config are still shown as in [3] and not restored to [1]
Shouldn’t libvirt restore the mac/vlan to [1].
The same problem exists when using <interface type='direct'> (OpenStack macvtap port) but just for the MAC configuration of the VF.
What libvirt does is to restore the MAC address to whatever it was before we set it up for use with a guest. Although there are some sriov net drivers that (for some unfathomable reason) think it's cool to assign a random MAC address to each VF at boot time, the "normal" thing is for the VFs to have a MAC address of all 0's to start with. So libvirt should be saving 00:00:00:00:00:00 (it will be in the file /var/run/libvirt/hostdevmgr/$ifname_vf$vfnum) then setting the MAC to use; when done, libvirt will read the 00:00:00:00:00:00 and use netlink to set the MAC address, but this is apparently failing.
I checked on my Fedora 22 system with the igb driver, and found that if the MAC address was originally set to something other than 0's, it was restored properly by libvirt, but if it was set to all 0's originally, the attempt to set it back to 0 would fail.
I then tried doing the same thing with the "ip" utility:
# ip link set dev p4p2 vf 0 mac 00:00:00:00:00:00
and I get the following response:
RTNETLINK answers: Invalid argument
So it appears that either the kernel or the NIC driver is refusing to set the MAC address to all 0's. I'm reasonably certain this is a regression in the kernel,
Sigh. It appears that this has "always" been the case - I just checked on a 2.6.32-573 RHEL kernel, and a 3.10.x RHEL7.2 kernel, and 4.1 (Fedora 22) and both of them also refuse to set the MAC address to 00:00:00:00:00:00. I'm not sure if this limitation is in the NIC driver or some basic code in the kernel.
It's considered to be an invalid address by is_valid_ether_addr() function. There appear to be different behaviour in some adapters. In current upstream, it looks like a quarter of the VF capable drivers (bnxt, enic, fm10k, sfc) permit VF mac setting of all zeros. The others simply use is_valid_ether_addr function without specifically testing for all-0. I am not sure if this is HW related or simply oversights... Might want to bringing this up on netdev. -vlad
although I can't say how long it's been there, as I don't normally pay attention to this (and as I said, many SRIOV NIC drivers don't default their VFs to 0 MAC addresses)
What distro and kernel are you using for your tests?
[1] - 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
[2] - <interface type='hostdev' managed='yes'>
<mac address=' fa:16:3e:11:af:fe '/>
<driver name='kvm'/>
<source>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x7'/>
</source>
<vlan>
<tag id='190'/>
</vlan>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</interface>
[3] 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 3 MAC fa:16:3e:11:af:fe, vlan 190, spoof checking off, link-state enable
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list F15
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On 12/15/2015 04:12 PM, Vlad Yasevich wrote:
On 12/15/2015 02:45 PM, Laine Stump wrote:
On 12/15/2015 01:34 PM, Laine Stump wrote:
On 12/13/2015 10:51 AM, Moshe Levi wrote:
Hi,
I have a setup with libvirt 1.3.0 and OpenStack trunk.
Before launched the VM ip link command show the following VF mac/vlan configuration [1]
When I launch a VM with <interface type='hostdev'> via openstack api (OpenStack direct port)
I can see that the VF get the mac/vlan according to libvrit xml [2] and ip link command [3], but when I delete the VM the mac/vlan config are still shown as in [3] and not restored to [1]
Shouldn’t libvirt restore the mac/vlan to [1].
The same problem exists when using <interface type='direct'> (OpenStack macvtap port) but just for the MAC configuration of the VF.
What libvirt does is to restore the MAC address to whatever it was before we set it up for use with a guest. Although there are some sriov net drivers that (for some unfathomable reason) think it's cool to assign a random MAC address to each VF at boot time, the "normal" thing is for the VFs to have a MAC address of all 0's to start with. So libvirt should be saving 00:00:00:00:00:00 (it will be in the file /var/run/libvirt/hostdevmgr/$ifname_vf$vfnum) then setting the MAC to use; when done, libvirt will read the 00:00:00:00:00:00 and use netlink to set the MAC address, but this is apparently failing.
I checked on my Fedora 22 system with the igb driver, and found that if the MAC address was originally set to something other than 0's, it was restored properly by libvirt, but if it was set to all 0's originally, the attempt to set it back to 0 would fail.
I then tried doing the same thing with the "ip" utility:
# ip link set dev p4p2 vf 0 mac 00:00:00:00:00:00
and I get the following response:
RTNETLINK answers: Invalid argument
So it appears that either the kernel or the NIC driver is refusing to set the MAC address to all 0's. I'm reasonably certain this is a regression in the kernel, Sigh. It appears that this has "always" been the case - I just checked on a 2.6.32-573 RHEL kernel, and a 3.10.x RHEL7.2 kernel, and 4.1 (Fedora 22) and both of them also refuse to set the MAC address to 00:00:00:00:00:00. I'm not sure if this limitation is in the NIC driver or some basic code in the kernel. It's considered to be an invalid address by is_valid_ether_addr() function.
There appear to be different behaviour in some adapters. In current upstream, it looks like a quarter of the VF capable drivers (bnxt, enic, fm10k, sfc) permit VF mac setting of all zeros. The others simply use is_valid_ether_addr function without specifically testing for all-0. I am not sure if this is HW related or simply oversights... Might want to bringing this up on netdev.
Thanks, Vlad! Moshe, It sounds like in your case it is caused by code in the mlx driver, so hopefully you can have some influence there. My path is a bit more difficult, as the failure on mine is in the igb driver :-) Sending a message to netdev is a good idea. It would be wonderful if all the vendors could agree to: 1) initialize all VFs with a MAC address of 0 2) allow setting VF MAC address to 0 at any time. I'll put that on my to-do list :-P
-vlad
although I can't say how long it's been there, as I don't normally pay attention to this (and as I said, many SRIOV NIC drivers don't default their VFs to 0 MAC addresses)
What distro and kernel are you using for your tests?
[1] - 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
[2] - <interface type='hostdev' managed='yes'>
<mac address=' fa:16:3e:11:af:fe '/>
<driver name='kvm'/>
<source>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x7'/>
</source>
<vlan>
<tag id='190'/>
</vlan>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</interface>
[3] 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 3 MAC fa:16:3e:11:af:fe, vlan 190, spoof checking off, link-state enable
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list F15
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
-----Original Message----- From: sendmail [mailto:justsendmailnothingelse@gmail.com] On Behalf Of Laine Stump Sent: Wednesday, December 16, 2015 12:31 AM To: Libvirt <libvir-list@redhat.com> Cc: vyasevic@redhat.com; Moshe Levi <moshele@mellanox.com> Subject: Re: [libvirt] <interface type='hostdev'>vf configuration cleanup when VM is delete
On 12/15/2015 04:12 PM, Vlad Yasevich wrote:
On 12/15/2015 02:45 PM, Laine Stump wrote:
On 12/15/2015 01:34 PM, Laine Stump wrote:
On 12/13/2015 10:51 AM, Moshe Levi wrote:
Hi,
I have a setup with libvirt 1.3.0 and OpenStack trunk.
Before launched the VM ip link command show the following VF mac/vlan configuration [1]
When I launch a VM with <interface type='hostdev'> via openstack api (OpenStack direct port)
I can see that the VF get the mac/vlan according to libvrit xml [2] and ip link command [3], but when I delete the VM the mac/vlan config are still shown as in [3] and not restored to [1]
Shouldn’t libvirt restore the mac/vlan to [1].
The same problem exists when using <interface type='direct'> (OpenStack macvtap port) but just for the MAC configuration of the VF.
What libvirt does is to restore the MAC address to whatever it was before we set it up for use with a guest. Although there are some sriov net drivers that (for some unfathomable reason) think it's cool to assign a random MAC address to each VF at boot time, the "normal" thing is for the VFs to have a MAC address of all 0's to start with. So libvirt should be saving 00:00:00:00:00:00 (it will be in the file /var/run/libvirt/hostdevmgr/$ifname_vf$vfnum) then setting the MAC to use; when done, libvirt will read the 00:00:00:00:00:00 and use netlink to set the MAC address, but this is apparently failing.
I checked on my Fedora 22 system with the igb driver, and found that if the MAC address was originally set to something other than 0's, it was restored properly by libvirt, but if it was set to all 0's originally, the attempt to set it back to 0 would fail.
I then tried doing the same thing with the "ip" utility:
# ip link set dev p4p2 vf 0 mac 00:00:00:00:00:00
and I get the following response:
RTNETLINK answers: Invalid argument
So it appears that either the kernel or the NIC driver is refusing to set the MAC address to all 0's. I'm reasonably certain this is a regression in the kernel, Sigh. It appears that this has "always" been the case - I just checked on a 2.6.32-573 RHEL kernel, and a 3.10.x RHEL7.2 kernel, and 4.1 (Fedora 22) and both of them also refuse to set the MAC address to 00:00:00:00:00:00. I'm not sure if this limitation is in the NIC driver or some basic code in the kernel. It's considered to be an invalid address by is_valid_ether_addr() function.
There appear to be different behaviour in some adapters. In current upstream, it looks like a quarter of the VF capable drivers (bnxt, enic, fm10k, sfc) permit VF mac setting of all zeros. The others simply use is_valid_ether_addr function without specifically testing for all-0. I am not sure if this is HW related or simply oversights... Might want to bringing this up on netdev.
Thanks, Vlad!
Moshe,
It sounds like in your case it is caused by code in the mlx driver, so hopefully you can have some influence there. My path is a bit more difficult, as the failure on mine is in the igb driver :-) Sure I will take it with Mellanox Driver team Ok, just saw the latest thread mail you can discard my previous one
Sending a message to netdev is a good idea. It would be wonderful if all the vendors could agree to:
1) initialize all VFs with a MAC address of 0 2) allow setting VF MAC address to 0 at any time.
I'll put that on my to-do list :-P
-vlad
although I can't say how long it's been there, as I don't normally pay attention to this (and as I said, many SRIOV NIC drivers don't default their VFs to 0 MAC addresses)
What distro and kernel are you using for your tests?
[1] - 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
[2] - <interface type='hostdev' managed='yes'>
<mac address=' fa:16:3e:11:af:fe '/>
<driver name='kvm'/>
<source>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x7'/>
</source>
<vlan>
<tag id='190'/>
</vlan>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</interface>
[3] 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc
mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 3 MAC fa:16:3e:11:af:fe, vlan 190, spoof checking off, link-state enable
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list F15
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
To clean up the VF I use ip link set dev p4p2 vf 0 mac 0 and it working 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000 link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto vf 2 MAC 00:00:00:00:00:b1, vlan 190, spoof checking off, link-state enable vf 3 MAC aa:bb:cc:00:00:12, vlan 190, spoof checking off, link-state enable [root@r-ufm160 devstack]# ip link set dev enp3s0f0 vf 3 mac 0 [root@r-ufm160 devstack]# ip link show enp3s0f0 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000 link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto vf 2 MAC 00:00:00:00:00:b1, vlan 190, spoof checking off, link-state enable vf 3 MAC 00:00:00:00:00:b1, vlan 190, spoof checking off, link-state enable It just put the address 00:00:00:00:00:b1 which I don’t know why, but as I remember the same behavior is in intel cards (I think is related to iproute) I used fedora 2.1 with kernel 4.1.13-100. The most annoying part is that in OpenStack if I use an SR-IOV VF (interface hostdev) for VM and delete it I can’t reuse it for macvtap (interface direct) so I have to clean the mac by running ip link set dev p4p2 vf 0 mac 0 I guess I will need to workaround it in OpenStack. From: sendmail [mailto:justsendmailnothingelse@gmail.com] On Behalf Of Laine Stump Sent: Tuesday, December 15, 2015 9:45 PM To: Libvirt <libvir-list@redhat.com> Cc: Moshe Levi <moshele@mellanox.com>; vyasevic@redhat.com Subject: Re: [libvirt] <interface type='hostdev'>vf configuration cleanup when VM is delete On 12/15/2015 01:34 PM, Laine Stump wrote: On 12/13/2015 10:51 AM, Moshe Levi wrote: Hi, I have a setup with libvirt 1.3.0 and OpenStack trunk. Before launched the VM ip link command show the following VF mac/vlan configuration [1] When I launch a VM with <interface type='hostdev'> via openstack api (OpenStack direct port) I can see that the VF get the mac/vlan according to libvrit xml [2] and ip link command [3], but when I delete the VM the mac/vlan config are still shown as in [3] and not restored to [1] Shouldn’t libvirt restore the mac/vlan to [1]. The same problem exists when using <interface type='direct'> (OpenStack macvtap port) but just for the MAC configuration of the VF. What libvirt does is to restore the MAC address to whatever it was before we set it up for use with a guest. Although there are some sriov net drivers that (for some unfathomable reason) think it's cool to assign a random MAC address to each VF at boot time, the "normal" thing is for the VFs to have a MAC address of all 0's to start with. So libvirt should be saving 00:00:00:00:00:00 (it will be in the file /var/run/libvirt/hostdevmgr/$ifname_vf$vfnum) then setting the MAC to use; when done, libvirt will read the 00:00:00:00:00:00 and use netlink to set the MAC address, but this is apparently failing. I checked on my Fedora 22 system with the igb driver, and found that if the MAC address was originally set to something other than 0's, it was restored properly by libvirt, but if it was set to all 0's originally, the attempt to set it back to 0 would fail. I then tried doing the same thing with the "ip" utility: # ip link set dev p4p2 vf 0 mac 00:00:00:00:00:00 and I get the following response: RTNETLINK answers: Invalid argument So it appears that either the kernel or the NIC driver is refusing to set the MAC address to all 0's. I'm reasonably certain this is a regression in the kernel, Sigh. It appears that this has "always" been the case - I just checked on a 2.6.32-573 RHEL kernel, and a 3.10.x RHEL7.2 kernel, and 4.1 (Fedora 22) and both of them also refuse to set the MAC address to 00:00:00:00:00:00. I'm not sure if this limitation is in the NIC driver or some basic code in the kernel. although I can't say how long it's been there, as I don't normally pay attention to this (and as I said, many SRIOV NIC drivers don't default their VFs to 0 MAC addresses) What distro and kernel are you using for your tests? [1] - 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000 link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto [2] - <interface type='hostdev' managed='yes'> <mac address=' fa:16:3e:11:af:fe '/> <driver name='kvm'/> <source> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x7'/> </source> <vlan> <tag id='190'/> </vlan> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </interface> [3] 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000 link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto vf 3 MAC fa:16:3e:11:af:fe, vlan 190, spoof checking off, link-state enable -- libvir-list mailing list libvir-list@redhat.com<mailto:libvir-list@redhat.com> https://www.redhat.com/mailman/listinfo/libvir-list F15 -- libvir-list mailing list libvir-list@redhat.com<mailto:libvir-list@redhat.com> https://www.redhat.com/mailman/listinfo/libvir-list
On 12/16/2015 07:56 AM, Moshe Levi wrote:
To clean up the VF I use
ip link set dev p4p2 vf 0 mac 0 and it working
Now *that* is interesting...
24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:b1, vlan 190, spoof checking off, link-state enable
vf 3 MAC aa:bb:cc:00:00:12, vlan 190, spoof checking off, link-state enable
[root@r-ufm160 devstack]# ip link set dev enp3s0f0 vf 3 mac 0
[root@r-ufm160 devstack]# ip link show enp3s0f0
24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:b1, vlan 190, spoof checking off, link-state enable
vf 3 MAC 00:00:00:00:00:b1, vlan 190, spoof checking off, link-state enable
It just put the address 00:00:00:00:00:b1 which I don’t know why, but as I remember the same behavior is in intel cards (I think is related to iproute)
I just tried this with the igb driver on both 2.6.32 and 4.1 kernels, and a plain "0" is successful for me too. But, as you've experienced, it doesn't actually set the MAC address to 00:00:00:00:00:00, but instead puts random numbers in the final two bytes :-/ So I investigated further, and found that if I use: ip link set dev p4p2 vf 0 mac 00:00:00:00:00 <-- note 5 bytes, not 6 then all bytes except the *final* byte are 0, and the final byte is two seemingly random bytes. But if I re-run the same command many times I find that it just rotates between 10 or so different values; not so random (when I give "0", or "00:00:00:00" to ip link set, the 2nd to last byte is always the *exact same* value. So I looked in the source for the ip utility (in the iproute package) and I found that the function parsing mac addresses from the commandline just creates the buffer on the stack, doesn't initialize it, then parses in as many digits as you specify, leaving the rest with whatever happened to be sitting on the stack at the time :-O. In other words, it's just a happy coincidence of a bug in iproute's mac address parser that "ip link set .... mac 0" happens to be successful (and that bytes 2-4 are 0 and 5-6 are non-0). I really don't know where to start / what to do with this information. There is obviously a bug in iproute that should be fixed, but if it is fixed before all the places in the kernel are adjusted to allow an all-0 MAC, then users will be complaining that their script which was working for years and years (although probably not doing exactly what they believed) is suddenly broken. And who knows what Hell-fury will be unleashed by some unknown bit of code in the kernel if a 0 mac address suddenly shows up for the first time ever. Sigh. (BTW, Cisco's enic driver, on the other hand, doesn't support setting VF MAC addresses via a netlink message to the PF *at all* (so libvirt has to make special accommodations), but it happily accepts requests to directly set the MAC address to 00:00:00:00:00:00 via ioctl(SIOCSIFHWADDR) (and the interface MAC address really does get set to all 0's). There is a script for ovirt that uses a MAC address of all 0's to recognize that an interface is unused, and can thus be included in a pool of interfaces in a libvirt network. That won't work with any other SRIOV drivers though, because even if they initialize their VF macs to 0 (e.g. mlx and *new* (3.10+) igb (but *not* 2.6.32 igb!)), they can't be set back to 0 when they are once again unused. Again sigh.)
I used fedora 2.1 with kernel 4.1.13-100.
The most annoying part is that in OpenStack if I use an SR-IOV VF (interface hostdev) for VM and delete it I can’t reuse it for macvtap (interface direct) so I have to clean the mac
by running ip link set dev p4p2 vf 0 mac 0
I guess I will need to workaround it in OpenStack.
*From:*sendmail [mailto:justsendmailnothingelse@gmail.com] *On Behalf Of *Laine Stump *Sent:* Tuesday, December 15, 2015 9:45 PM *To:* Libvirt <libvir-list@redhat.com> *Cc:* Moshe Levi <moshele@mellanox.com>; vyasevic@redhat.com *Subject:* Re: [libvirt] <interface type='hostdev'>vf configuration cleanup when VM is delete
On 12/15/2015 01:34 PM, Laine Stump wrote:
On 12/13/2015 10:51 AM, Moshe Levi wrote:
Hi,
I have a setup with libvirt 1.3.0 and OpenStack trunk.
Before launched the VM ip link command show the following VF mac/vlan configuration [1]
When I launch a VM with <interface type='hostdev'> via openstack api (OpenStack direct port)
I can see that the VF get the mac/vlan according to libvrit xml [2] and ip link command [3], but when I delete the VM the mac/vlan config are still shown as in [3] and not restored to [1]
Shouldn’t libvirt restore the mac/vlan to [1].
The same problem exists when using <interface type='direct'> (OpenStack macvtap port) but just for the MAC configuration of the VF.
What libvirt does is to restore the MAC address to whatever it was before we set it up for use with a guest. Although there are some sriov net drivers that (for some unfathomable reason) think it's cool to assign a random MAC address to each VF at boot time, the "normal" thing is for the VFs to have a MAC address of all 0's to start with. So libvirt should be saving 00:00:00:00:00:00 (it will be in the file /var/run/libvirt/hostdevmgr/$ifname_vf$vfnum) then setting the MAC to use; when done, libvirt will read the 00:00:00:00:00:00 and use netlink to set the MAC address, but this is apparently failing.
I checked on my Fedora 22 system with the igb driver, and found that if the MAC address was originally set to something other than 0's, it was restored properly by libvirt, but if it was set to all 0's originally, the attempt to set it back to 0 would fail.
I then tried doing the same thing with the "ip" utility:
# ip link set dev p4p2 vf 0 mac 00:00:00:00:00:00
and I get the following response:
RTNETLINK answers: Invalid argument
So it appears that either the kernel or the NIC driver is refusing to set the MAC address to all 0's. I'm reasonably certain this is a regression in the kernel,
Sigh. It appears that this has "always" been the case - I just checked on a 2.6.32-573 RHEL kernel, and a 3.10.x RHEL7.2 kernel, and 4.1 (Fedora 22) and both of them also refuse to set the MAC address to 00:00:00:00:00:00. I'm not sure if this limitation is in the NIC driver or some basic code in the kernel.
although I can't say how long it's been there, as I don't normally pay attention to this (and as I said, many SRIOV NIC drivers don't default their VFs to 0 MAC addresses)
What distro and kernel are you using for your tests?
[1] - 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
[2] - <interface type='hostdev' managed='yes'>
<mac address=' fa:16:3e:11:af:fe '/>
<driver name='kvm'/>
<source>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x7'/>
</source>
<vlan>
<tag id='190'/>
</vlan>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</interface>
[3] 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 3 MAC fa:16:3e:11:af:fe, vlan 190, spoof checking off, link-state enable
--
libvir-list mailing list
libvir-list@redhat.com <mailto:libvir-list@redhat.com>
https://www.redhat.com/mailman/listinfo/libvir-list
F15
--
libvir-list mailing list
libvir-list@redhat.com <mailto:libvir-list@redhat.com>
On 12/16/2015 12:22 PM, Laine Stump wrote:
On 12/16/2015 07:56 AM, Moshe Levi wrote:
To clean up the VF I use
ip link set dev p4p2 vf 0 mac 0 and it working
Now *that* is interesting...
24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:b1, vlan 190, spoof checking off, link-state enable
vf 3 MAC aa:bb:cc:00:00:12, vlan 190, spoof checking off, link-state enable
[root@r-ufm160 devstack]# ip link set dev enp3s0f0 vf 3 mac 0
[root@r-ufm160 devstack]# ip link show enp3s0f0
24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:b1, vlan 190, spoof checking off, link-state enable
vf 3 MAC 00:00:00:00:00:b1, vlan 190, spoof checking off, link-state enable
It just put the address 00:00:00:00:00:b1 which I don’t know why, but as I remember the same behavior is in intel cards (I think is related to iproute)
I just tried this with the igb driver on both 2.6.32 and 4.1 kernels, and a plain "0" is successful for me too. But, as you've experienced, it doesn't actually set the MAC address to 00:00:00:00:00:00, but instead puts random numbers in the final two bytes :-/
So I investigated further, and found that if I use:
ip link set dev p4p2 vf 0 mac 00:00:00:00:00 <-- note 5 bytes, not 6
then all bytes except the *final* byte are 0, and the final byte is two seemingly random bytes. But if I re-run the same command many times I find that it just rotates between 10 or so different values; not so random (when I give "0", or "00:00:00:00" to ip link set, the 2nd to last byte is always the *exact same* value.
So I looked in the source for the ip utility (in the iproute package) and I found that the function parsing mac addresses from the commandline just creates the buffer on the stack, doesn't initialize it, then parses in as many digits as you specify, leaving the rest with whatever happened to be sitting on the stack at the time :-O.
In other words, it's just a happy coincidence of a bug in iproute's mac address parser that "ip link set .... mac 0" happens to be successful (and that bytes 2-4 are 0 and 5-6 are non-0).
I really don't know where to start / what to do with this information. There is obviously a bug in iproute that should be fixed, but if it is fixed before all the places in the kernel are adjusted to allow an all-0 MAC, then users will be complaining that their script which was working for years and years (although probably not doing exactly what they believed) is suddenly broken. And who knows what Hell-fury will be unleashed by some unknown bit of code in the kernel if a 0 mac address suddenly shows up for the first time ever. Sigh.
But they wouldn't be there for the first time as we can plainly see from above output for vf 0 and vf 1. Not sure if users depending on the stack contents of a buffer in iproute2 would ever be really working as they expected. Might just be a necessary patch.
(BTW, Cisco's enic driver, on the other hand, doesn't support setting VF MAC addresses via a netlink message to the PF *at all* (so libvirt has to make special accommodations), but
Looking at upstream, it looks like it offers support for setting VF mac via VFINFO data in the netlink message. May be it got fixed? -vlad
it happily accepts requests to directly set the MAC address to 00:00:00:00:00:00 via ioctl(SIOCSIFHWADDR) (and the interface MAC address really does get set to all 0's). There is a script for ovirt that uses a MAC address of all 0's to recognize that an interface is unused, and can thus be included in a pool of interfaces in a libvirt network. That won't work with any other SRIOV drivers though, because even if they initialize their VF macs to 0 (e.g. mlx and *new* (3.10+) igb (but *not* 2.6.32 igb!)), they can't be set back to 0 when they are once again unused. Again sigh.)
I used fedora 2.1 with kernel 4.1.13-100.
The most annoying part is that in OpenStack if I use an SR-IOV VF (interface hostdev) for VM and delete it I can’t reuse it for macvtap (interface direct) so I have to clean the mac
by running ip link set dev p4p2 vf 0 mac 0
I guess I will need to workaround it in OpenStack.
*From:*sendmail [mailto:justsendmailnothingelse@gmail.com] *On Behalf Of *Laine Stump *Sent:* Tuesday, December 15, 2015 9:45 PM *To:* Libvirt <libvir-list@redhat.com> *Cc:* Moshe Levi <moshele@mellanox.com>; vyasevic@redhat.com *Subject:* Re: [libvirt] <interface type='hostdev'>vf configuration cleanup when VM is delete
On 12/15/2015 01:34 PM, Laine Stump wrote:
On 12/13/2015 10:51 AM, Moshe Levi wrote:
Hi,
I have a setup with libvirt 1.3.0 and OpenStack trunk.
Before launched the VM ip link command show the following VF mac/vlan configuration [1]
When I launch a VM with <interface type='hostdev'> via openstack api (OpenStack direct port)
I can see that the VF get the mac/vlan according to libvrit xml [2] and ip link command [3], but when I delete the VM the mac/vlan config are still shown as in [3] and not restored to [1]
Shouldn’t libvirt restore the mac/vlan to [1].
The same problem exists when using <interface type='direct'> (OpenStack macvtap port) but just for the MAC configuration of the VF.
What libvirt does is to restore the MAC address to whatever it was before we set it up for use with a guest. Although there are some sriov net drivers that (for some unfathomable reason) think it's cool to assign a random MAC address to each VF at boot time, the "normal" thing is for the VFs to have a MAC address of all 0's to start with. So libvirt should be saving 00:00:00:00:00:00 (it will be in the file /var/run/libvirt/hostdevmgr/$ifname_vf$vfnum) then setting the MAC to use; when done, libvirt will read the 00:00:00:00:00:00 and use netlink to set the MAC address, but this is apparently failing.
I checked on my Fedora 22 system with the igb driver, and found that if the MAC address was originally set to something other than 0's, it was restored properly by libvirt, but if it was set to all 0's originally, the attempt to set it back to 0 would fail.
I then tried doing the same thing with the "ip" utility:
# ip link set dev p4p2 vf 0 mac 00:00:00:00:00:00
and I get the following response:
RTNETLINK answers: Invalid argument
So it appears that either the kernel or the NIC driver is refusing to set the MAC address to all 0's. I'm reasonably certain this is a regression in the kernel,
Sigh. It appears that this has "always" been the case - I just checked on a 2.6.32-573 RHEL kernel, and a 3.10.x RHEL7.2 kernel, and 4.1 (Fedora 22) and both of them also refuse to set the MAC address to 00:00:00:00:00:00. I'm not sure if this limitation is in the NIC driver or some basic code in the kernel.
although I can't say how long it's been there, as I don't normally pay attention to this (and as I said, many SRIOV NIC drivers don't default their VFs to 0 MAC addresses)
What distro and kernel are you using for your tests?
[1] - 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
[2] - <interface type='hostdev' managed='yes'>
<mac address=' fa:16:3e:11:af:fe '/>
<driver name='kvm'/>
<source>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x7'/>
</source>
<vlan>
<tag id='190'/>
</vlan>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</interface>
[3] 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
vf 3 MAC fa:16:3e:11:af:fe, vlan 190, spoof checking off, link-state enable
--
libvir-list mailing list
libvir-list@redhat.com <mailto:libvir-list@redhat.com>
https://www.redhat.com/mailman/listinfo/libvir-list
F15
--
libvir-list mailing list
libvir-list@redhat.com <mailto:libvir-list@redhat.com>
On 12/16/2015 12:35 PM, Vlad Yasevich wrote:
(BTW, Cisco's enic driver, on the other hand, doesn't support setting VF MAC addresses via a netlink message to the PF *at all* (so libvirt has to make special accommodations), but Looking at upstream, it looks like it offers support for setting VF mac via VFINFO data in the netlink message. May be it got fixed?
Interesting. If I had one of those systems of my own to test on, I'd give it a try. The only one I have access to is running a 2.6.32 RHEL6 kernel though :-/
On 12/16/2015 01:17 PM, Laine Stump wrote:
On 12/16/2015 12:35 PM, Vlad Yasevich wrote:
(BTW, Cisco's enic driver, on the other hand, doesn't support setting VF MAC addresses via a netlink message to the PF *at all* (so libvirt has to make special accommodations), but Looking at upstream, it looks like it offers support for setting VF mac via VFINFO data in the netlink message. May be it got fixed?
Interesting. If I had one of those systems of my own to test on, I'd give it a try. The only one I have access to is running a 2.6.32 RHEL6 kernel though :-/
It also occurred to me during my copious "time to think" over the holidays that device assignment using <interface type='hostdev'> wouldn't work for enic cards if that wasn't supported, and I recently saw a bug filed where someone was doing exactly that on a 3.10 kernel (RHEL7.x), so yeah it must have been fixed between 2.6 and 3.10). To pick up this discussion again: Moshe - am I correct to assume that you've filed a bug somewhere about the mlx driver not accepting 00:00:00:00:00:00? In the meantime, if setting the MAC address to, e.g., 00:00:00:00:00:01 works for you (as implied when you said you use "ip link set dev p4p2 vf 0 mac 0") then you could temporarily work around the problem by setting all the VFs to something like that at boot time - libvirt would then succeed at setting them *back* to those addresses when it was finished with each device.
From: sendmail [mailto:justsendmailnothingelse@gmail.com] On Behalf Of Laine Stump Sent: Wednesday, January 06, 2016 9:26 PM To: Libvirt <libvir-list@redhat.com> Cc: vyasevic@redhat.com; Moshe Levi <moshele@mellanox.com> Subject: Re: [libvirt] <interface type='hostdev'>vf configuration cleanup when VM is delete On 12/16/2015 01:17 PM, Laine Stump wrote: On 12/16/2015 12:35 PM, Vlad Yasevich wrote: (BTW, Cisco's enic driver, on the other hand, doesn't support setting VF MAC addresses via a netlink message to the PF *at all* (so libvirt has to make special accommodations), but Looking at upstream, it looks like it offers support for setting VF mac via VFINFO data in the netlink message. May be it got fixed? Interesting. If I had one of those systems of my own to test on, I'd give it a try. The only one I have access to is running a 2.6.32 RHEL6 kernel though :-/ It also occurred to me during my copious "time to think" over the holidays that device assignment using <interface type='hostdev'> wouldn't work for enic cards if that wasn't supported, and I recently saw a bug filed where someone was doing exactly that on a 3.10 kernel (RHEL7.x), so yeah it must have been fixed between 2.6 and 3.10). To pick up this discussion again: Moshe - am I correct to assume that you've filed a bug somewhere about the mlx driver not accepting 00:00:00:00:00:00? Yes I filled bugs against ConnectX3 and ConnectX4 drivers. In the meantime, if setting the MAC address to, e.g., 00:00:00:00:00:01 works for you (as implied when you said you use "ip link set dev p4p2 vf 0 mac 0") then you could temporarily work around the problem by setting all the VFs to something like that at boot time - libvirt would then succeed at setting them *back* to those addresses when it was finished with each device. Yes I will use this workaround.
participants (3)
-
Laine Stump -
Moshe Levi -
Vlad Yasevich