8:50 a.m.
On 10/17/2013 10:43 AM, Niilona wrote:
libvirt points to wrong VF -channel, when trying to change MAC
-address.
libvirt v1 1.0.2
linux driver : ixgbe 3.17.3
Symptom description
----------------------------------
1.) How the VF channels are seen in the Host ( correct in increasing
address order )
command : ls -la /sys/bus/pci/devices/0000\:04\:00.0/
..................
lrwxrwxrwx 1 root root 0 Oct 17 12:25 virtfn0 -> ../0000:04:10.0/
lrwxrwxrwx 1 root root 0 Oct 17 12:25 virtfn1 -> ../0000:04:10.2/
lrwxrwxrwx 1 root root 0 Oct 17 12:25 virtfn10 -> ../0000:04:12.4/
lrwxrwxrwx 1 root root 0 Oct 17 12:25 virtfn11 -> ../0000:04:12.6/
lrwxrwxrwx 1 root root 0 Oct 17 12:25 virtfn12 -> ../0000:04:13.0/
lrwxrwxrwx 1 root root 0 Oct 17 12:25 virtfn13 -> ../0000:04:13.2/
lrwxrwxrwx 1 root root 0 Oct 17 12:25 virtfn2 -> ../0000:04:10.4/
lrwxrwxrwx 1 root root 0 Oct 17 12:25 virtfn3 -> ../0000:04:10.6/
lrwxrwxrwx 1 root root 0 Oct 17 12:25 virtfn4 -> ../0000:04:11.0/
lrwxrwxrwx 1 root root 0 Oct 17 12:25 virtfn5 -> ../0000:04:11.2/
lrwxrwxrwx 1 root root 0 Oct 17 12:25 virtfn6 -> ../0000:04:11.4/
lrwxrwxrwx 1 root root 0 Oct 17 12:25 virtfn7 -> ../0000:04:11.6/
lrwxrwxrwx 1 root root 0 Oct 17 12:25 virtfn8 -> ../0000:04:12.0/
lrwxrwxrwx 1 root root 0 Oct 17 12:25 virtfn9 -> ../0000:04:12.2/
-----------------------
2.) How the libvirt sees them
virsh nodedev-dumpxml pci_0000_04_00_0
<device>
<name>pci_0000_04_00_0</name>
<parent>pci_0000_00_03_0</parent>
<driver>
<name>ixgbe</name>
</driver>
<capability type='pci'>
<domain>0</domain>
<bus>4</bus>
<slot>0</slot>
<function>0</function>
<product id='0x10fb'>82599EB 10-Gigabit SFI/SFP+ Network
Connection</product>
<vendor id='0x8086'>Intel Corporation</vendor>
<capability type='virt_functions'>
<address domain='0x0000' bus='0x04' slot='0x12'
function='0x4'/>
<address domain='0x0000' bus='0x04' slot='0x12'
function='0x6'/>
<address domain='0x0000' bus='0x04' slot='0x13'
function='0x0'/>
<address domain='0x0000' bus='0x04' slot='0x13'
function='0x2'/>
<address domain='0x0000' bus='0x04' slot='0x10'
function='0x0'/>
<address domain='0x0000' bus='0x04' slot='0x10'
function='0x2'/>
<address domain='0x0000' bus='0x04' slot='0x10'
function='0x4'/>
<address domain='0x0000' bus='0x04' slot='0x10'
function='0x6'/>
<address domain='0x0000' bus='0x04' slot='0x11'
function='0x0'/>
<address domain='0x0000' bus='0x04' slot='0x11'
function='0x2'/>
<address domain='0x0000' bus='0x04' slot='0x11'
function='0x4'/>
<address domain='0x0000' bus='0x04' slot='0x11'
function='0x6'/>
<address domain='0x0000' bus='0x04' slot='0x12'
function='0x0'/>
<address domain='0x0000' bus='0x04' slot='0x12'
function='0x2'/>
</capability>
</capability>
</device>
3.) Libvirt sets MAC -address in slot/position 5 instead of 1
ip link show eth10
87: eth10: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 2100 qdisc mq
state UP mode DEFAULT qlen 1000
link/ether 00:1b:21:b9:a5:60 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 92:e2:6b:53:a7:c2, spoof checking off <------ this is the
slot, MAC tried to set
vf 1 MAC 0a:04:38:2c:79:35, spoof checking off
vf 2 MAC b2:46:b6:91:44:2e, spoof checking off
vf 3 MAC de:b6:28:da:20:44, spoof checking off
vf 4 MAC 02:00:00:80:00:01, spoof checking off <--------- MAC sets
in wrong slot
vf 5 MAC 02:00:00:80:00:02, spoof checking off
vf 6 MAC 02:00:00:80:00:20, spoof checking off
vf 7 MAC 02:00:00:80:00:30, spoof checking off
vf 8 MAC 52:54:00:de:f8:08, spoof checking off
vf 9 MAC 5e:b0:d0:34:d6:cc, spoof checking off
vf 10 MAC d2:88:5c:2d:db:b5, spoof checking off
vf 11 MAC 6a:06:8a:9c:31:8b, spoof checking off
vf 12 MAC 4a:38:be:8f:b3:bf, spoof checking off
vf 13 MAC b6:c1:d4:12:d3:3d, spoof checking off
5.) Question : Why this happens in ixgbe -driver and when we try to
use more than 10 VF's/physical port. ?
1) You haven't included any of the libvirt config that led to this
situation, thus it isn't possible for us to see the supposed mismatch
between which VF was assigned to the guest and which VF had its MAC
address changed. Please include the <interface> section of the guest
config, as well as any applicable <network> config (if you are
allocating the VF from a network pool).
2) I have noticed in the past that the MAC address shown for a VF by "ip
link show" did not match the MAC address seen by the guest (and used for
the packets that egress from the guest). Have you verified that the
guest truly isn't seeing the device with the proper MAC address? Take a
look at the output of ip link show on the guest.
3) Are you running the stock igxbe driver? There was a recent case of a
user whose VFs were not getting the proper vlan ID set, and after much
investigation, someone noticed that this user had installed some forked
version of the ixgbe driver which was faulty.