[libvirt] sr-iov ethernet VF is not assigned correctly to guest
Hi, I'm hoping someone could possibly show me what I may be doing wrong here, and why I'm not seeing the expected results Thanks in advance! Server Information:HP Product Name: ProLiant DL165 G7 Ethernet Card Information: Intel 82599EB 10-Gigabit SFI/SFP+ uname: Linux 3.0.0-16-server x86_64 GNU/Linux OS Release: Ubuntu 11.10 qemu-kvm version 0.14.1+noroms-0ubuntu6.2 First, here's a list of my VFs: # ip link show: 2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000 link/ether 00:1b:21:7d:70:16 brd ff:ff:ff:ff:ff:ff vf 0 MAC 02:22:25:00:00:00 vf 1 MAC 02:22:25:00:00:01 vf 2 MAC 02:22:25:00:00:02 vf 3 MAC 02:22:25:00:00:03 vf 4 MAC 02:22:25:00:00:04 vf 5 MAC 02:22:25:00:00:05 vf 6 MAC 02:22:25:00:00:06 3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN qlen 1000 link/ether 00:1b:21:7d:70:16 brd ff:ff:ff:ff:ff:ff vf 0 MAC 02:22:25:01:00:00 vf 1 MAC 02:22:25:01:00:01 vf 2 MAC 02:22:25:01:00:02 vf 3 MAC 02:22:25:01:00:03 vf 4 MAC 02:22:25:01:00:04 vf 5 MAC 02:22:25:01:00:05 vf 6 MAC 02:22:25:01:00:06 # lspci |grep 05 05:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 05:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 05:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) # virsh nodedev-list |grep 05 pci_0000_05_00_0 pci_0000_05_00_1 pci_0000_05_10_0 pci_0000_05_10_1 pci_0000_05_10_2 pci_0000_05_10_3 pci_0000_05_10_4 pci_0000_05_10_5 pci_0000_05_10_6 pci_0000_05_10_7 pci_0000_05_11_0 pci_0000_05_11_1 pci_0000_05_11_2 pci_0000_05_11_3 pci_0000_05_11_4 pci_0000_05_11_5 4 Now detach 05.10.0 and 05.11.0 from the host: # virsh nodedev-dettach pci_0000_05_10_0 Device pci_0000_05_10_0 dettached # virsh nodedev-dettach pci_0000_05_11_0 Device pci_0000_05_11_0 dettached 5. Get the host, slot, function information: # virsh nodedev-dumpxml pci_0000_05_10_0|grep -e bus -e slot -e fun |head -3 <bus>5</bus> <slot>16</slot> <function>0</function> # virsh nodedev-dumpxml pci_0000_05_11_0|grep -e bus -e slot -e fun |head -3 <bus>5</bus> <slot>17</slot> <function>0</function> == GUEST-A == XML: ... <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x05' slot='0x10' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x05' slot='0x11' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> ... CLI: /usr/bin/kvm -S \ -M pc-0.14 \ -enable-kvm \ -m 256 \ -smp 1,sockets=1,cores=1,threads=1 \ -name GUEST-A \ -uuid 9bac7b2f-dbb4-0f2a-08e6-47018d6630db \ -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/HOST-A.monitor,server,nowait \ -mon chardev=charmonitor,id=monitor,mode=readline \ -rtc base=utc \ -boot cn \ -drive file=/srv/node/sdc1/HOST-A.raw,if=none,id=drive-virtio-disk0,boot=on,format=raw,cache=none \ -device virtio-blk-pci,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0 \ -chardev pty,id=charserial0 \ -device isa-serial,chardev=charserial0,id=serial0 \ -usb \ -vnc 0.0.0.0:0 \ -vga cirrus \ -device pci-assign,host=05:10.0,id=hostdev0,configfd=20,bus=pci.0,addr=0x4 \ -device pci-assign,host=05:11.0,id=hostdev1,configfd=21,bus=pci.0,addr=0x5 \ -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 dmesg: [ 3.072509] igbvf 0000:00:04.0: Intel(R) 82576 Virtual Function [ 3.072514] igbvf 0000:00:04.0: Address: 02:22:25:00:00:00 [ 3.072516] igbvf 0000:00:04.0: MAC: 1 [ 3.241325] igbvf 0000:00:05.0: Intel(R) 82576 Virtual Function [ 3.241329] igbvf 0000:00:05.0: Address: 02:22:25:00:00:04 [ 3.241332] igbvf 0000:00:05.0: MAC: 1 ip link: 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 02:22:25:00:00:00 brd ff:ff:ff:ff:ff:ff 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 02:22:25:00:00:04 brd ff:ff:ff:ff:ff:ff Expected Results: GUEST-A should have the following assignment: eth0 should = eth0-vf0 (bus:5, slot:10, fun:0), mac: 02:22:25:00:00:00 eth1 should = eth1-vf0 (bus:5, slot:11, fun:0), mac: 02:22:25:01:00:00
On Mon, Mar 26, 2012 at 11:12:56AM -0400, Scott wrote:
Hi, I'm hoping someone could possibly show me what I may be doing wrong here, and why I'm not seeing the expected results
Thanks in advance!
Server Information:HP Product Name: ProLiant DL165 G7 Ethernet Card Information: Intel 82599EB 10-Gigabit SFI/SFP+ uname: Linux 3.0.0-16-server x86_64 GNU/Linux OS Release: Ubuntu 11.10 qemu-kvm version 0.14.1+noroms-0ubuntu6.2
First, here's a list of my VFs: # ip link show: 2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000 link/ether 00:1b:21:7d:70:16 brd ff:ff:ff:ff:ff:ff vf 0 MAC 02:22:25:00:00:00 vf 1 MAC 02:22:25:00:00:01 vf 2 MAC 02:22:25:00:00:02 vf 3 MAC 02:22:25:00:00:03 vf 4 MAC 02:22:25:00:00:04 vf 5 MAC 02:22:25:00:00:05 vf 6 MAC 02:22:25:00:00:06 3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN qlen 1000 link/ether 00:1b:21:7d:70:16 brd ff:ff:ff:ff:ff:ff vf 0 MAC 02:22:25:01:00:00 vf 1 MAC 02:22:25:01:00:01 vf 2 MAC 02:22:25:01:00:02 vf 3 MAC 02:22:25:01:00:03 vf 4 MAC 02:22:25:01:00:04 vf 5 MAC 02:22:25:01:00:05 vf 6 MAC 02:22:25:01:00:06
# lspci |grep 05 05:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 05:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 05:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
# virsh nodedev-list |grep 05 pci_0000_05_00_0 pci_0000_05_00_1
pci_0000_05_10_0 pci_0000_05_10_1 pci_0000_05_10_2 pci_0000_05_10_3 pci_0000_05_10_4 pci_0000_05_10_5 pci_0000_05_10_6 pci_0000_05_10_7
pci_0000_05_11_0 pci_0000_05_11_1 pci_0000_05_11_2 pci_0000_05_11_3 pci_0000_05_11_4 pci_0000_05_11_5
4 Now detach 05.10.0 and 05.11.0 from the host: # virsh nodedev-dettach pci_0000_05_10_0 Device pci_0000_05_10_0 dettached
# virsh nodedev-dettach pci_0000_05_11_0 Device pci_0000_05_11_0 dettached
I don't see where you check that pci_0000_05_10_0 and pci_0000_05_11_0 really do correspond to the virtual functions you want. ie where are you checking that they actually have the expected MAC addrs, before assigning them to the guest
ip link: 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 02:22:25:00:00:00 brd ff:ff:ff:ff:ff:ff 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 02:22:25:00:00:04 brd ff:ff:ff:ff:ff:ff
Expected Results: GUEST-A should have the following assignment: eth0 should = eth0-vf0 (bus:5, slot:10, fun:0), mac: 02:22:25:00:00:00 eth1 should = eth1-vf0 (bus:5, slot:11, fun:0), mac: 02:22:25:01:00:00
Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Mar 26, 2012, at 11:20 AM, Daniel P. Berrange wrote:
On Mon, Mar 26, 2012 at 11:12:56AM -0400, Scott wrote:
Hi, I'm hoping someone could possibly show me what I may be doing wrong here, and why I'm not seeing the expected results
Thanks in advance!
Server Information:HP Product Name: ProLiant DL165 G7 Ethernet Card Information: Intel 82599EB 10-Gigabit SFI/SFP+ uname: Linux 3.0.0-16-server x86_64 GNU/Linux OS Release: Ubuntu 11.10 qemu-kvm version 0.14.1+noroms-0ubuntu6.2
First, here's a list of my VFs: # ip link show: 2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000 link/ether 00:1b:21:7d:70:16 brd ff:ff:ff:ff:ff:ff vf 0 MAC 02:22:25:00:00:00 vf 1 MAC 02:22:25:00:00:01 vf 2 MAC 02:22:25:00:00:02 vf 3 MAC 02:22:25:00:00:03 vf 4 MAC 02:22:25:00:00:04 vf 5 MAC 02:22:25:00:00:05 vf 6 MAC 02:22:25:00:00:06 3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN qlen 1000 link/ether 00:1b:21:7d:70:16 brd ff:ff:ff:ff:ff:ff vf 0 MAC 02:22:25:01:00:00 vf 1 MAC 02:22:25:01:00:01 vf 2 MAC 02:22:25:01:00:02 vf 3 MAC 02:22:25:01:00:03 vf 4 MAC 02:22:25:01:00:04 vf 5 MAC 02:22:25:01:00:05 vf 6 MAC 02:22:25:01:00:06
# lspci |grep 05 05:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 05:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 05:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 05:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
# virsh nodedev-list |grep 05 pci_0000_05_00_0 pci_0000_05_00_1
pci_0000_05_10_0 pci_0000_05_10_1 pci_0000_05_10_2 pci_0000_05_10_3 pci_0000_05_10_4 pci_0000_05_10_5 pci_0000_05_10_6 pci_0000_05_10_7
pci_0000_05_11_0 pci_0000_05_11_1 pci_0000_05_11_2 pci_0000_05_11_3 pci_0000_05_11_4 pci_0000_05_11_5
4 Now detach 05.10.0 and 05.11.0 from the host: # virsh nodedev-dettach pci_0000_05_10_0 Device pci_0000_05_10_0 dettached
# virsh nodedev-dettach pci_0000_05_11_0 Device pci_0000_05_11_0 dettached ip link: 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 02:22:25:00:00:00 brd ff:ff:ff:ff:ff:ff 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 02:22:25:00:00:04 brd ff:ff:ff:ff:ff:ff
Expected Results: GUEST-A should have the following assignment: eth0 should = eth0-vf0 (bus:5, slot:10, fun:0), mac: 02:22:25:00:00:00 eth1 should = eth1-vf0 (bus:5, slot:11, fun:0), mac: 02:22:25:01:00:00
I don't see where you check that pci_0000_05_10_0 and pci_0000_05_11_0 really do correspond to the virtual functions you want. ie where are you checking that they actually have the expected MAC addrs, before assigning them to the guest
Thank you for your response. I had set the MAC addresses via: for i in {0..6}; do ip link set eth0 vf $i mac 02:22:25:00:00:0$i; done for i in {0..6}; do ip link set eth1 vf $i mac 02:22:25:01:00:0$i; done then made sure they corresponded via ip link show (see output above) Is there something else I should be doing/checking? Thanks.
participants (2)
-
Daniel P. Berrange -
Scott