[libvirt] two hostdev devices problem
hi, I try to add 2 VF by "hostdev". Networks (vnet0, vnet1) with: <forward mode='hostdev' managed='yes'> <pf dev='eth1'/> ..... Domain: <interface type="network"> <source network="vnet0"/> .... <interface type="network"> <source network="vnet1"/> .... virsh create error: "error: internal error process exited while connecting to monitor: kvm: -device pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4: Duplicate ID 'hostdev0' for device" I am doing somthing wrong? My configs: domain: <domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm"> <name>one-466</name> <vcpu>1</vcpu> <memory>1781760</memory> <os> <type arch="x86_64">hvm</type> <boot dev="hd"/> </os> <devices> <emulator>/usr/bin/kvm</emulator> <disk type="file" device="disk"> <source file="/nebula/nebula/var//datastores/0/466/disk.0"/> <target dev="hda" bus="virtio"/> <driver name="qemu" type="raw" cache="default"/> </disk> <disk type="file" device="disk"> <source file="/nebula/nebula/var//datastores/0/466/disk.1"/> <target dev="vdb" bus="virtio"/> <driver name="qemu" type="raw" cache="default"/> </disk> <graphics type="vnc" listen="0.0.0.0" port="-1"/> <interface type="network"> <source network="vnet0"/> <mac address="52:54:0a:b1:48:91"/> </interface> <interface type="network"> <source network="vnet1"/> <mac address="52:54:0a:b1:48:92"/> </interface> </devices> <features> <acpi/> </features> <devices><serial type="pty"><source path="/dev/pts/477"/><target port="0"/></serial><console type="pty" tty="/dev/pts/477"><source path="/dev/pts/477"/><target port="0"/></console></devices> </domain> Networks: cat /etc/libvirt/qemu/networks/vnet0.xml <!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh net-edit vnet0 or other application using the libvirt API. --> <network> <name>vnet0</name> <uuid>1b6a3506-d9e0-4e1e-bbff-6f700b8dc80b</uuid> <forward mode='hostdev' managed='yes'> <pf dev='eth0'/> </forward> </network> cat /etc/libvirt/qemu/networks/vnet1.xml <!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh net-edit vnet1 or other application using the libvirt API. --> <network> <name>vnet1</name> <uuid>11feb088-d609-4177-86d7-ac3b9467a3e8</uuid> <forward mode='hostdev' managed='yes'> <pf dev='eth1'/> </forward> </network> I wrote libvirt-users mailing list too. -- Pozdrawiam Dominik
On 05/21/2013 09:32 AM, Dominik Mostowiec wrote:
hi, I try to add 2 VF by "hostdev". Networks (vnet0, vnet1) with: <forward mode='hostdev' managed='yes'> <pf dev='eth1'/> .....
Domain: <interface type="network"> <source network="vnet0"/> .... <interface type="network"> <source network="vnet1"/> ....
virsh create error: "error: internal error process exited while connecting to monitor: kvm: -device pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4: Duplicate ID 'hostdev0' for device"
Hi, it seems we have been assigning the same id to all network hostdevs until this recent commit (not yet released): http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6597cc25 Jan
On 05/21/2013 04:03 AM, Ján Tomko wrote:
On 05/21/2013 09:32 AM, Dominik Mostowiec wrote:
hi, I try to add 2 VF by "hostdev". Networks (vnet0, vnet1) with: <forward mode='hostdev' managed='yes'> <pf dev='eth1'/> .....
Domain: <interface type="network"> <source network="vnet0"/> .... <interface type="network"> <source network="vnet1"/> ....
virsh create error: "error: internal error process exited while connecting to monitor: kvm: -device pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4: Duplicate ID 'hostdev0' for device"
Hi,
it seems we have been assigning the same id to all network hostdevs until this recent commit (not yet released):
http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6597cc25
If that patch has such an effect, it's purely accidental. Have you tested this? (I plan to test a before and after as soon as I've had breakfast)
On 05/21/2013 01:37 PM, Laine Stump wrote:
On 05/21/2013 04:03 AM, Ján Tomko wrote:
On 05/21/2013 09:32 AM, Dominik Mostowiec wrote:
hi, I try to add 2 VF by "hostdev". Networks (vnet0, vnet1) with: <forward mode='hostdev' managed='yes'> <pf dev='eth1'/> .....
Domain: <interface type="network"> <source network="vnet0"/> .... <interface type="network"> <source network="vnet1"/> ....
virsh create error: "error: internal error process exited while connecting to monitor: kvm: -device pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4: Duplicate ID 'hostdev0' for device"
Hi,
it seems we have been assigning the same id to all network hostdevs until this recent commit (not yet released):
http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6597cc25
If that patch has such an effect, it's purely accidental. Have you tested this? (I plan to test a before and after as soon as I've had breakfast)
I haven't tested it and looking at the code again, I might've been wrong :( Sorry about that. Jan
Hmm, It seems to be working (after only simple tests). -- Dominik 2013/5/21 Ján Tomko <jtomko@redhat.com>
On 05/21/2013 04:03 AM, Ján Tomko wrote:
On 05/21/2013 09:32 AM, Dominik Mostowiec wrote:
hi, I try to add 2 VF by "hostdev". Networks (vnet0, vnet1) with: <forward mode='hostdev' managed='yes'> <pf dev='eth1'/> .....
Domain: <interface type="network"> <source network="vnet0"/> .... <interface type="network"> <source network="vnet1"/> ....
virsh create error: "error: internal error process exited while connecting to monitor: kvm: -device
On 05/21/2013 01:37 PM, Laine Stump wrote: pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4:
Duplicate ID 'hostdev0' for device"
Hi,
it seems we have been assigning the same id to all network hostdevs until this recent commit (not yet released):
http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6597cc25
If that patch has such an effect, it's purely accidental. Have you tested this? (I plan to test a before and after as soon as I've had breakfast)
I haven't tested it and looking at the code again, I might've been wrong :(
Sorry about that.
Jan
-- Pozdrawiam Dominik
I tested on 1.0.5 patched version and vm with 2 vfs working fine. I have another problem When i set max_vfs=63: internal error missing IFLA_VF_INFO in netlink response Its working when max_vfs=31 -- Dominik 21 maj 2013 15:19, "Dominik Mostowiec" <dominikmostowiec@gmail.com> napisał(a):
Hmm, It seems to be working (after only simple tests).
-- Dominik
2013/5/21 Ján Tomko <jtomko@redhat.com>
On 05/21/2013 04:03 AM, Ján Tomko wrote:
On 05/21/2013 09:32 AM, Dominik Mostowiec wrote:
hi, I try to add 2 VF by "hostdev". Networks (vnet0, vnet1) with: <forward mode='hostdev' managed='yes'> <pf dev='eth1'/> .....
Domain: <interface type="network"> <source network="vnet0"/> .... <interface type="network"> <source network="vnet1"/> ....
virsh create error: "error: internal error process exited while connecting to monitor: kvm: -device
On 05/21/2013 01:37 PM, Laine Stump wrote: pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4:
Duplicate ID 'hostdev0' for device"
Hi,
it seems we have been assigning the same id to all network hostdevs until this recent commit (not yet released):
http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6597cc25
If that patch has such an effect, it's purely accidental. Have you tested this? (I plan to test a before and after as soon as I've had breakfast)
I haven't tested it and looking at the code again, I might've been wrong :(
Sorry about that.
Jan
-- Pozdrawiam Dominik
On 05/22/2013 06:32 PM, Dominik Mostowiec wrote:
I tested on 1.0.5 patched version and vm with 2 vfs working fine.
I have another problem When i set max_vfs=63: internal error missing IFLA_VF_INFO in netlink response
Its working when max_vfs=31
What is the platform, kernel version, and what version of libnl are you using (libnl 1 or libnl3?) There have been multiple bugs recently filed and fixed in this area on RHEL6/CentOS6. In particular, there was a problem with max_vfs > 50 that had exactly this symptom. It was solved by a patch to libnl-1.1. If your platform uses libnl-1.1, this could be the source of the problem. It is fixed in the upstream libnl-1.1.4 maintenance release. I unfortunately am not sure where the libnl-1.1 git is (I only have the git tree for libnl3), so I can't give you the exact commit message, but in short the problem was that libnl was using too small of a buffer for the netlink socket, so when there were a lot of vfs, the netlink message containing vf info was truncated.
-- Dominik
21 maj 2013 15:19, "Dominik Mostowiec" <dominikmostowiec@gmail.com <mailto:dominikmostowiec@gmail.com>> napisał(a):
Hmm, It seems to be working (after only simple tests).
-- Dominik
2013/5/21 Ján Tomko <jtomko@redhat.com <mailto:jtomko@redhat.com>>
On 05/21/2013 01:37 PM, Laine Stump wrote: > On 05/21/2013 04:03 AM, Ján Tomko wrote: >> On 05/21/2013 09:32 AM, Dominik Mostowiec wrote: >>> hi, >>> I try to add 2 VF by "hostdev". >>> Networks (vnet0, vnet1) with: >>> <forward mode='hostdev' managed='yes'> >>> <pf dev='eth1'/> >>> ..... >>> >>> Domain: >>> <interface type="network"> >>> <source network="vnet0"/> >>> .... >>> <interface type="network"> >>> <source network="vnet1"/> >>> .... >>> >>> virsh create error: >>> "error: internal error process exited while connecting to monitor: kvm: >>> -device pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4: >>> Duplicate ID 'hostdev0' for device" >>> >> Hi, >> >> it seems we have been assigning the same id to all network hostdevs until this >> recent commit (not yet released): >> >> http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6597cc25 > > > If that patch has such an effect, it's purely accidental. Have you > tested this? (I plan to test a before and after as soon as I've had > breakfast) >
I haven't tested it and looking at the code again, I might've been wrong :(
Sorry about that.
Jan
-- Pozdrawiam Dominik
Hi, I have libnl3, kernel 3.8.8, os ubuntu 12.04, net:intel10G. Upgrade libnl to 3.2.16 not helps, I don't know - this is probably another problem, when I have set max_vfs=35,35 VF is attaching to VM but mac in VM is not set propertly. Libvirt probably set this macs because in syslog: May 23 14:48:59 on-10-177-32-62 kernel: [12574.171792] ixgbe 0000:01:00.0: setting MAC 52:54:0a:b1:48:f7 on VF 0 May 23 14:48:59 on-10-177-32-62 kernel: [12574.171799] ixgbe 0000:01:00.0: Reload the VF driver to make this change effective. May 23 14:48:59 on-10-177-32-62 kernel: [12574.175054] ixgbe 0000:01:00.1: setting MAC 52:54:0a:b1:48:f7 on VF 0 May 23 14:48:59 on-10-177-32-62 kernel: [12574.175060] ixgbe 0000:01:00.1: Reload the VF driver to make this change effective. May 23 14:48:59 on-10-177-32-62 kernel: [12574.307172] pci-stub 0000:01:12.4: enabling device (0000 -> 0002) May 23 14:49:00 on-10-177-32-62 kernel: [12575.662890] assign device 0:1:12.4 May 23 14:49:00 on-10-177-32-62 kernel: [12575.665210] pci-stub 0000:01:12.5: enabling device (0000 -> 0002) May 23 14:49:00 on-10-177-32-62 kernel: [12575.765609] assign device 0:1:12.5 But in VM i have another macs probably old from VF. --- Regards Dominik 2013/5/24 Laine Stump <laine@laine.org>:
On 05/22/2013 06:32 PM, Dominik Mostowiec wrote:
I tested on 1.0.5 patched version and vm with 2 vfs working fine.
I have another problem When i set max_vfs=63: internal error missing IFLA_VF_INFO in netlink response
Its working when max_vfs=31
What is the platform, kernel version, and what version of libnl are you using (libnl 1 or libnl3?)
There have been multiple bugs recently filed and fixed in this area on RHEL6/CentOS6. In particular, there was a problem with max_vfs > 50 that had exactly this symptom. It was solved by a patch to libnl-1.1. If your platform uses libnl-1.1, this could be the source of the problem. It is fixed in the upstream libnl-1.1.4 maintenance release.
I unfortunately am not sure where the libnl-1.1 git is (I only have the git tree for libnl3), so I can't give you the exact commit message, but in short the problem was that libnl was using too small of a buffer for the netlink socket, so when there were a lot of vfs, the netlink message containing vf info was truncated.
-- Dominik
21 maj 2013 15:19, "Dominik Mostowiec" <dominikmostowiec@gmail.com <mailto:dominikmostowiec@gmail.com>> napisał(a):
Hmm, It seems to be working (after only simple tests).
-- Dominik
2013/5/21 Ján Tomko <jtomko@redhat.com <mailto:jtomko@redhat.com>>
On 05/21/2013 01:37 PM, Laine Stump wrote: > On 05/21/2013 04:03 AM, Ján Tomko wrote: >> On 05/21/2013 09:32 AM, Dominik Mostowiec wrote: >>> hi, >>> I try to add 2 VF by "hostdev". >>> Networks (vnet0, vnet1) with: >>> <forward mode='hostdev' managed='yes'> >>> <pf dev='eth1'/> >>> ..... >>> >>> Domain: >>> <interface type="network"> >>> <source network="vnet0"/> >>> .... >>> <interface type="network"> >>> <source network="vnet1"/> >>> .... >>> >>> virsh create error: >>> "error: internal error process exited while connecting to monitor: kvm: >>> -device pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4: >>> Duplicate ID 'hostdev0' for device" >>> >> Hi, >> >> it seems we have been assigning the same id to all network hostdevs until this >> recent commit (not yet released): >> >> http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6597cc25 > > > If that patch has such an effect, it's purely accidental. Have you > tested this? (I plan to test a before and after as soon as I've had > breakfast) >
I haven't tested it and looking at the code again, I might've been wrong :(
Sorry about that.
Jan
-- Pozdrawiam Dominik
-- Pozdrawiam Dominik
On 05/24/2013 04:45 AM, Dominik Mostowiec wrote:
Hi, I have libnl3, kernel 3.8.8, os ubuntu 12.04, net:intel10G. Upgrade libnl to 3.2.16 not helps,
I don't know - this is probably another problem, when I have set max_vfs=35,35 VF is attaching to VM but mac in VM is not set propertly.
Do you have this problem with mac addresses when max_vfs is set to an even lower number? Ah, another thing - did you "ifconfig up" the PF before you started the guest?
Libvirt probably set this macs because in syslog: May 23 14:48:59 on-10-177-32-62 kernel: [12574.171792] ixgbe
Hmm. ixgbe. I just got a report yesterday that setting of the vlan tag for ixgbe vfs was not working properly (although it works fine for igb devices)
0000:01:00.0: setting MAC 52:54:0a:b1:48:f7 on VF 0 May 23 14:48:59 on-10-177-32-62 kernel: [12574.171799] ixgbe 0000:01:00.0: Reload the VF driver to make this change effective. May 23 14:48:59 on-10-177-32-62 kernel: [12574.175054] ixgbe 0000:01:00.1: setting MAC 52:54:0a:b1:48:f7 on VF 0 May 23 14:48:59 on-10-177-32-62 kernel: [12574.175060] ixgbe 0000:01:00.1: Reload the VF driver to make this change effective. May 23 14:48:59 on-10-177-32-62 kernel: [12574.307172] pci-stub 0000:01:12.4: enabling device (0000 -> 0002) May 23 14:49:00 on-10-177-32-62 kernel: [12575.662890] assign device 0:1:12.4 May 23 14:49:00 on-10-177-32-62 kernel: [12575.665210] pci-stub 0000:01:12.5: enabling device (0000 -> 0002) May 23 14:49:00 on-10-177-32-62 kernel: [12575.765609] assign device 0:1:12.5 But in VM i have another macs probably old from VF.
--- Regards Dominik
2013/5/24 Laine Stump <laine@laine.org>:
On 05/22/2013 06:32 PM, Dominik Mostowiec wrote:
I tested on 1.0.5 patched version and vm with 2 vfs working fine.
I have another problem When i set max_vfs=63: internal error missing IFLA_VF_INFO in netlink response
Its working when max_vfs=31
What is the platform, kernel version, and what version of libnl are you using (libnl 1 or libnl3?)
There have been multiple bugs recently filed and fixed in this area on RHEL6/CentOS6. In particular, there was a problem with max_vfs > 50 that had exactly this symptom. It was solved by a patch to libnl-1.1. If your platform uses libnl-1.1, this could be the source of the problem. It is fixed in the upstream libnl-1.1.4 maintenance release.
I unfortunately am not sure where the libnl-1.1 git is (I only have the git tree for libnl3), so I can't give you the exact commit message, but in short the problem was that libnl was using too small of a buffer for the netlink socket, so when there were a lot of vfs, the netlink message containing vf info was truncated.
-- Dominik
21 maj 2013 15:19, "Dominik Mostowiec" <dominikmostowiec@gmail.com <mailto:dominikmostowiec@gmail.com>> napisał(a):
Hmm, It seems to be working (after only simple tests).
-- Dominik
2013/5/21 Ján Tomko <jtomko@redhat.com <mailto:jtomko@redhat.com>>
On 05/21/2013 01:37 PM, Laine Stump wrote: > On 05/21/2013 04:03 AM, Ján Tomko wrote: >> On 05/21/2013 09:32 AM, Dominik Mostowiec wrote: >>> hi, >>> I try to add 2 VF by "hostdev". >>> Networks (vnet0, vnet1) with: >>> <forward mode='hostdev' managed='yes'> >>> <pf dev='eth1'/> >>> ..... >>> >>> Domain: >>> <interface type="network"> >>> <source network="vnet0"/> >>> .... >>> <interface type="network"> >>> <source network="vnet1"/> >>> .... >>> >>> virsh create error: >>> "error: internal error process exited while connecting to monitor: kvm: >>> -device pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4: >>> Duplicate ID 'hostdev0' for device" >>> >> Hi, >> >> it seems we have been assigning the same id to all network hostdevs until this >> recent commit (not yet released): >> >> http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6597cc25 > > > If that patch has such an effect, it's purely accidental. Have you > tested this? (I plan to test a before and after as soon as I've had > breakfast) >
I haven't tested it and looking at the code again, I might've been wrong :(
Sorry about that.
Jan
-- Pozdrawiam Dominik
Hi,
Do you have this problem with mac addresses when max_vfs is set to an even lower number?
No, When max_fs=10, macs in VM are OK. Phisical interfaces:
ifconfig eth0 eth0 Link encap:Ethernet HWaddr b8:ca:3a:5b:a6:38 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
ifconfig eth1 eth1 Link encap:Ethernet HWaddr b8:ca:3a:5b:a6:38 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
Hmm. ixgbe. I just got a report yesterday that setting of the vlan tag for ixgbe vfs was not working properly (although it works fine for igb devices)
Domain config: <interface type="network"> <alias name="hostdev0"/> <source network="vnet0"/> <mac address="52:54:0a:b1:48:fc"/> </interface> <interface type="network"> <alias name="hostdev1"/> <source network="vnet1"/> <mac address="52:54:0a:b1:48:fc"/> </interface> Networks defined: <network> <name>vnet0</name> <uuid>ec49896a-a0b5-4944-a81f-9f0cdf578871</uuid> <forward mode='hostdev' managed='yes'> <pf dev='eth0'/> </forward> </network> <network> <name>vnet1</name> <uuid>6c8319e8-8b53-4382-9f4e-2400819b00a9</uuid> <forward mode='hostdev' managed='yes'> <pf dev='eth1'/> </forward> </network> Regards Dominik 2013/5/24 Laine Stump <laine@laine.org>:
On 05/24/2013 04:45 AM, Dominik Mostowiec wrote:
Hi, I have libnl3, kernel 3.8.8, os ubuntu 12.04, net:intel10G. Upgrade libnl to 3.2.16 not helps,
I don't know - this is probably another problem, when I have set max_vfs=35,35 VF is attaching to VM but mac in VM is not set propertly.
Do you have this problem with mac addresses when max_vfs is set to an even lower number?
Ah, another thing - did you "ifconfig up" the PF before you started the guest?
Libvirt probably set this macs because in syslog: May 23 14:48:59 on-10-177-32-62 kernel: [12574.171792] ixgbe
Hmm. ixgbe. I just got a report yesterday that setting of the vlan tag for ixgbe vfs was not working properly (although it works fine for igb devices)
0000:01:00.0: setting MAC 52:54:0a:b1:48:f7 on VF 0 May 23 14:48:59 on-10-177-32-62 kernel: [12574.171799] ixgbe 0000:01:00.0: Reload the VF driver to make this change effective. May 23 14:48:59 on-10-177-32-62 kernel: [12574.175054] ixgbe 0000:01:00.1: setting MAC 52:54:0a:b1:48:f7 on VF 0 May 23 14:48:59 on-10-177-32-62 kernel: [12574.175060] ixgbe 0000:01:00.1: Reload the VF driver to make this change effective. May 23 14:48:59 on-10-177-32-62 kernel: [12574.307172] pci-stub 0000:01:12.4: enabling device (0000 -> 0002) May 23 14:49:00 on-10-177-32-62 kernel: [12575.662890] assign device 0:1:12.4 May 23 14:49:00 on-10-177-32-62 kernel: [12575.665210] pci-stub 0000:01:12.5: enabling device (0000 -> 0002) May 23 14:49:00 on-10-177-32-62 kernel: [12575.765609] assign device 0:1:12.5 But in VM i have another macs probably old from VF.
--- Regards Dominik
2013/5/24 Laine Stump <laine@laine.org>:
On 05/22/2013 06:32 PM, Dominik Mostowiec wrote:
I tested on 1.0.5 patched version and vm with 2 vfs working fine.
I have another problem When i set max_vfs=63: internal error missing IFLA_VF_INFO in netlink response
Its working when max_vfs=31
What is the platform, kernel version, and what version of libnl are you using (libnl 1 or libnl3?)
There have been multiple bugs recently filed and fixed in this area on RHEL6/CentOS6. In particular, there was a problem with max_vfs > 50 that had exactly this symptom. It was solved by a patch to libnl-1.1. If your platform uses libnl-1.1, this could be the source of the problem. It is fixed in the upstream libnl-1.1.4 maintenance release.
I unfortunately am not sure where the libnl-1.1 git is (I only have the git tree for libnl3), so I can't give you the exact commit message, but in short the problem was that libnl was using too small of a buffer for the netlink socket, so when there were a lot of vfs, the netlink message containing vf info was truncated.
-- Dominik
21 maj 2013 15:19, "Dominik Mostowiec" <dominikmostowiec@gmail.com <mailto:dominikmostowiec@gmail.com>> napisał(a):
Hmm, It seems to be working (after only simple tests).
-- Dominik
2013/5/21 Ján Tomko <jtomko@redhat.com <mailto:jtomko@redhat.com>>
On 05/21/2013 01:37 PM, Laine Stump wrote: > On 05/21/2013 04:03 AM, Ján Tomko wrote: >> On 05/21/2013 09:32 AM, Dominik Mostowiec wrote: >>> hi, >>> I try to add 2 VF by "hostdev". >>> Networks (vnet0, vnet1) with: >>> <forward mode='hostdev' managed='yes'> >>> <pf dev='eth1'/> >>> ..... >>> >>> Domain: >>> <interface type="network"> >>> <source network="vnet0"/> >>> .... >>> <interface type="network"> >>> <source network="vnet1"/> >>> .... >>> >>> virsh create error: >>> "error: internal error process exited while connecting to monitor: kvm: >>> -device pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4: >>> Duplicate ID 'hostdev0' for device" >>> >> Hi, >> >> it seems we have been assigning the same id to all network hostdevs until this >> recent commit (not yet released): >> >> http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6597cc25 > > > If that patch has such an effect, it's purely accidental. Have you > tested this? (I plan to test a before and after as soon as I've had > breakfast) >
I haven't tested it and looking at the code again, I might've been wrong :(
Sorry about that.
Jan
-- Pozdrawiam Dominik
-- Pozdrawiam Dominik
On 05/24/2013 09:16 AM, Dominik Mostowiec wrote:
Do you have this problem with mac addresses when max_vfs is set to an even lower number?
First - I just got confirmation from the libnl maintainer that the bug I mentioned causing problems with max_vfs > 50 *is* an issue in libnl 3.2.16 - it is fixed in libnl 3.2.22. Try to upgrade to that version and it should solve at least some of your problems.
No, When max_fs=10, macs in VM are OK.
Interesting. So now I'm curious about something - can you try adding a vlan tag to your networks and see if the vlan tag is set properly when max_vfs is a low number (10 or 7): <network> <name>vnet0</name> <uuid>ec49896a-a0b5-4944-a81f-9f0cdf578871</uuid> <vlan> <tag id='42'/> </vlan> <forward mode='hostdev' managed='yes'> <pf dev='eth0'/> </forward> </network> (you will have to do "virsh net-destroy vnet0; virsh net-start vnet0" after making the modification to the network). If the vlan tag is successfully set, it will show up next to the mac address for the VF in the output of "ip link show dev $PF". (BTW, it may get a bit confusing if you leave your networks named as "vnet0" and "vnet1" - although it's a different namespace, libvirt uses "vnetN" (where n is 0 - whatever) as the name for the tap device associated with each guest interface). To summarize: 1) when you have max_vfs = 10, you can assign the devices and mac addresses are set properly, i.e. everything works. 2) when you have max_vfs = 35, you can assign the devices, but the mac addresses are not set properly. 3) when you have max_vfs = 63, you can't assign the devices because you get this error: internal error missing IFLA_VF_INFO in netlink response Is this correct?
Phisical interfaces:
ifconfig eth0 eth0 Link encap:Ethernet HWaddr b8:ca:3a:5b:a6:38 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
Okay, so that's not the problem.
ifconfig eth1 eth1 Link encap:Ethernet HWaddr b8:ca:3a:5b:a6:38 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
Hmm. ixgbe. I just got a report yesterday that setting of the vlan tag for ixgbe vfs was not working properly (although it works fine for igb devices) Domain config: <interface type="network"> <alias name="hostdev0"/> <source network="vnet0"/> <mac address="52:54:0a:b1:48:fc"/> </interface> <interface type="network"> <alias name="hostdev1"/> <source network="vnet1"/> <mac address="52:54:0a:b1:48:fc"/> </interface>
Networks defined: <network> <name>vnet0</name> <uuid>ec49896a-a0b5-4944-a81f-9f0cdf578871</uuid> <forward mode='hostdev' managed='yes'> <pf dev='eth0'/> </forward> </network>
<network> <name>vnet1</name> <uuid>6c8319e8-8b53-4382-9f4e-2400819b00a9</uuid> <forward mode='hostdev' managed='yes'> <pf dev='eth1'/> </forward> </network>
Regards Dominik
2013/5/24 Laine Stump <laine@laine.org>:
On 05/24/2013 04:45 AM, Dominik Mostowiec wrote:
Hi, I have libnl3, kernel 3.8.8, os ubuntu 12.04, net:intel10G. Upgrade libnl to 3.2.16 not helps,
I don't know - this is probably another problem, when I have set max_vfs=35,35 VF is attaching to VM but mac in VM is not set propertly. Do you have this problem with mac addresses when max_vfs is set to an even lower number?
Ah, another thing - did you "ifconfig up" the PF before you started the guest?
Libvirt probably set this macs because in syslog: May 23 14:48:59 on-10-177-32-62 kernel: [12574.171792] ixgbe
Hmm. ixgbe. I just got a report yesterday that setting of the vlan tag for ixgbe vfs was not working properly (although it works fine for igb devices)
0000:01:00.0: setting MAC 52:54:0a:b1:48:f7 on VF 0 May 23 14:48:59 on-10-177-32-62 kernel: [12574.171799] ixgbe 0000:01:00.0: Reload the VF driver to make this change effective. May 23 14:48:59 on-10-177-32-62 kernel: [12574.175054] ixgbe 0000:01:00.1: setting MAC 52:54:0a:b1:48:f7 on VF 0 May 23 14:48:59 on-10-177-32-62 kernel: [12574.175060] ixgbe 0000:01:00.1: Reload the VF driver to make this change effective. May 23 14:48:59 on-10-177-32-62 kernel: [12574.307172] pci-stub 0000:01:12.4: enabling device (0000 -> 0002) May 23 14:49:00 on-10-177-32-62 kernel: [12575.662890] assign device 0:1:12.4 May 23 14:49:00 on-10-177-32-62 kernel: [12575.665210] pci-stub 0000:01:12.5: enabling device (0000 -> 0002) May 23 14:49:00 on-10-177-32-62 kernel: [12575.765609] assign device 0:1:12.5 But in VM i have another macs probably old from VF.
--- Regards Dominik
2013/5/24 Laine Stump <laine@laine.org>:
On 05/22/2013 06:32 PM, Dominik Mostowiec wrote:
I tested on 1.0.5 patched version and vm with 2 vfs working fine.
I have another problem When i set max_vfs=63: internal error missing IFLA_VF_INFO in netlink response
Its working when max_vfs=31
What is the platform, kernel version, and what version of libnl are you using (libnl 1 or libnl3?)
There have been multiple bugs recently filed and fixed in this area on RHEL6/CentOS6. In particular, there was a problem with max_vfs > 50 that had exactly this symptom. It was solved by a patch to libnl-1.1. If your platform uses libnl-1.1, this could be the source of the problem. It is fixed in the upstream libnl-1.1.4 maintenance release.
I unfortunately am not sure where the libnl-1.1 git is (I only have the git tree for libnl3), so I can't give you the exact commit message, but in short the problem was that libnl was using too small of a buffer for the netlink socket, so when there were a lot of vfs, the netlink message containing vf info was truncated.
-- Dominik
21 maj 2013 15:19, "Dominik Mostowiec" <dominikmostowiec@gmail.com <mailto:dominikmostowiec@gmail.com>> napisał(a):
Hmm, It seems to be working (after only simple tests).
-- Dominik
2013/5/21 Ján Tomko <jtomko@redhat.com <mailto:jtomko@redhat.com>>
On 05/21/2013 01:37 PM, Laine Stump wrote: > On 05/21/2013 04:03 AM, Ján Tomko wrote: >> On 05/21/2013 09:32 AM, Dominik Mostowiec wrote: >>> hi, >>> I try to add 2 VF by "hostdev". >>> Networks (vnet0, vnet1) with: >>> <forward mode='hostdev' managed='yes'> >>> <pf dev='eth1'/> >>> ..... >>> >>> Domain: >>> <interface type="network"> >>> <source network="vnet0"/> >>> .... >>> <interface type="network"> >>> <source network="vnet1"/> >>> .... >>> >>> virsh create error: >>> "error: internal error process exited while connecting to monitor: kvm: >>> -device pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4: >>> Duplicate ID 'hostdev0' for device" >>> >> Hi, >> >> it seems we have been assigning the same id to all network hostdevs until this >> recent commit (not yet released): >> >> http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6597cc25 > > > If that patch has such an effect, it's purely accidental. Have you > tested this? (I plan to test a before and after as soon as I've had > breakfast) >
I haven't tested it and looking at the code again, I might've been wrong :(
Sorry about that.
Jan
-- Pozdrawiam Dominik
-- Pozdrawiam Dominik
Hi,
First - I just got confirmation from the libnl maintainer that the bug I mentioned causing problems with max_vfs > 50 *is* an issue in libnl 3.2.16 - it is fixed in libnl 3.2.22. Try to upgrade to that version and it should solve at least some of your problems.
Upgrade libnl to version 3.2.22 helps. Thanks.
1) when you have max_vfs = 10, you can assign the devices and mac addresses are set properly, i.e. everything works. 2) when you have max_vfs = 35, you can assign the devices, but the mac addresses are not set properly. 3) when you have max_vfs = 63, you can't assign the devices because you get this error:
internal error missing IFLA_VF_INFO in netlink response
Is this correct?
Yes. Before upgrade libnl :-) Upgrede helps only for limit max_vfs.
(BTW, it may get a bit confusing if you leave your networks named as "vnet0" and "vnet1" - although it's a different namespace, libvirt uses "vnetN" (where n is 0 - whatever) as the name for the tap device associated with each guest interface).
You're right. I changed this.
the output of "ip link show dev $PF". Hmm, ip link show dev eth0 Shows only: 2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000 link/ether b8:ca:3a:5b:a6:3a brd ff:ff:ff:ff:ff:ff
I see VFs like network interfaces eth2-eth127. When VF is attached to VM it disappear from list on host. -- Regards Dominik 2013/5/24 Laine Stump <laine@laine.org>:
On 05/24/2013 09:16 AM, Dominik Mostowiec wrote:
Do you have this problem with mac addresses when max_vfs is set to an even lower number?
First - I just got confirmation from the libnl maintainer that the bug I mentioned causing problems with max_vfs > 50 *is* an issue in libnl 3.2.16 - it is fixed in libnl 3.2.22. Try to upgrade to that version and it should solve at least some of your problems.
No, When max_fs=10, macs in VM are OK.
Interesting.
So now I'm curious about something - can you try adding a vlan tag to your networks and see if the vlan tag is set properly when max_vfs is a low number (10 or 7):
<network> <name>vnet0</name> <uuid>ec49896a-a0b5-4944-a81f-9f0cdf578871</uuid> <vlan> <tag id='42'/> </vlan> <forward mode='hostdev' managed='yes'> <pf dev='eth0'/> </forward> </network>
(you will have to do "virsh net-destroy vnet0; virsh net-start vnet0" after making the modification to the network). If the vlan tag is successfully set, it will show up next to the mac address for the VF in the output of "ip link show dev $PF".
(BTW, it may get a bit confusing if you leave your networks named as "vnet0" and "vnet1" - although it's a different namespace, libvirt uses "vnetN" (where n is 0 - whatever) as the name for the tap device associated with each guest interface).
To summarize:
1) when you have max_vfs = 10, you can assign the devices and mac addresses are set properly, i.e. everything works. 2) when you have max_vfs = 35, you can assign the devices, but the mac addresses are not set properly. 3) when you have max_vfs = 63, you can't assign the devices because you get this error:
internal error missing IFLA_VF_INFO in netlink response
Is this correct?
Phisical interfaces:
ifconfig eth0 eth0 Link encap:Ethernet HWaddr b8:ca:3a:5b:a6:38 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
Okay, so that's not the problem.
ifconfig eth1 eth1 Link encap:Ethernet HWaddr b8:ca:3a:5b:a6:38 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
Hmm. ixgbe. I just got a report yesterday that setting of the vlan tag for ixgbe vfs was not working properly (although it works fine for igb devices) Domain config: <interface type="network"> <alias name="hostdev0"/> <source network="vnet0"/> <mac address="52:54:0a:b1:48:fc"/> </interface> <interface type="network"> <alias name="hostdev1"/> <source network="vnet1"/> <mac address="52:54:0a:b1:48:fc"/> </interface>
Networks defined: <network> <name>vnet0</name> <uuid>ec49896a-a0b5-4944-a81f-9f0cdf578871</uuid> <forward mode='hostdev' managed='yes'> <pf dev='eth0'/> </forward> </network>
<network> <name>vnet1</name> <uuid>6c8319e8-8b53-4382-9f4e-2400819b00a9</uuid> <forward mode='hostdev' managed='yes'> <pf dev='eth1'/> </forward> </network>
Regards Dominik
2013/5/24 Laine Stump <laine@laine.org>:
On 05/24/2013 04:45 AM, Dominik Mostowiec wrote:
Hi, I have libnl3, kernel 3.8.8, os ubuntu 12.04, net:intel10G. Upgrade libnl to 3.2.16 not helps,
I don't know - this is probably another problem, when I have set max_vfs=35,35 VF is attaching to VM but mac in VM is not set propertly. Do you have this problem with mac addresses when max_vfs is set to an even lower number?
Ah, another thing - did you "ifconfig up" the PF before you started the guest?
Libvirt probably set this macs because in syslog: May 23 14:48:59 on-10-177-32-62 kernel: [12574.171792] ixgbe
Hmm. ixgbe. I just got a report yesterday that setting of the vlan tag for ixgbe vfs was not working properly (although it works fine for igb devices)
0000:01:00.0: setting MAC 52:54:0a:b1:48:f7 on VF 0 May 23 14:48:59 on-10-177-32-62 kernel: [12574.171799] ixgbe 0000:01:00.0: Reload the VF driver to make this change effective. May 23 14:48:59 on-10-177-32-62 kernel: [12574.175054] ixgbe 0000:01:00.1: setting MAC 52:54:0a:b1:48:f7 on VF 0 May 23 14:48:59 on-10-177-32-62 kernel: [12574.175060] ixgbe 0000:01:00.1: Reload the VF driver to make this change effective. May 23 14:48:59 on-10-177-32-62 kernel: [12574.307172] pci-stub 0000:01:12.4: enabling device (0000 -> 0002) May 23 14:49:00 on-10-177-32-62 kernel: [12575.662890] assign device 0:1:12.4 May 23 14:49:00 on-10-177-32-62 kernel: [12575.665210] pci-stub 0000:01:12.5: enabling device (0000 -> 0002) May 23 14:49:00 on-10-177-32-62 kernel: [12575.765609] assign device 0:1:12.5 But in VM i have another macs probably old from VF.
--- Regards Dominik
2013/5/24 Laine Stump <laine@laine.org>:
On 05/22/2013 06:32 PM, Dominik Mostowiec wrote:
I tested on 1.0.5 patched version and vm with 2 vfs working fine.
I have another problem When i set max_vfs=63: internal error missing IFLA_VF_INFO in netlink response
Its working when max_vfs=31
What is the platform, kernel version, and what version of libnl are you using (libnl 1 or libnl3?)
There have been multiple bugs recently filed and fixed in this area on RHEL6/CentOS6. In particular, there was a problem with max_vfs > 50 that had exactly this symptom. It was solved by a patch to libnl-1.1. If your platform uses libnl-1.1, this could be the source of the problem. It is fixed in the upstream libnl-1.1.4 maintenance release.
I unfortunately am not sure where the libnl-1.1 git is (I only have the git tree for libnl3), so I can't give you the exact commit message, but in short the problem was that libnl was using too small of a buffer for the netlink socket, so when there were a lot of vfs, the netlink message containing vf info was truncated.
-- Dominik
21 maj 2013 15:19, "Dominik Mostowiec" <dominikmostowiec@gmail.com <mailto:dominikmostowiec@gmail.com>> napisał(a):
Hmm, It seems to be working (after only simple tests).
-- Dominik
2013/5/21 Ján Tomko <jtomko@redhat.com <mailto:jtomko@redhat.com>>
On 05/21/2013 01:37 PM, Laine Stump wrote: > On 05/21/2013 04:03 AM, Ján Tomko wrote: >> On 05/21/2013 09:32 AM, Dominik Mostowiec wrote: >>> hi, >>> I try to add 2 VF by "hostdev". >>> Networks (vnet0, vnet1) with: >>> <forward mode='hostdev' managed='yes'> >>> <pf dev='eth1'/> >>> ..... >>> >>> Domain: >>> <interface type="network"> >>> <source network="vnet0"/> >>> .... >>> <interface type="network"> >>> <source network="vnet1"/> >>> .... >>> >>> virsh create error: >>> "error: internal error process exited while connecting to monitor: kvm: >>> -device pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4: >>> Duplicate ID 'hostdev0' for device" >>> >> Hi, >> >> it seems we have been assigning the same id to all network hostdevs until this >> recent commit (not yet released): >> >> http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6597cc25 > > > If that patch has such an effect, it's purely accidental. Have you > tested this? (I plan to test a before and after as soon as I've had > breakfast) >
I haven't tested it and looking at the code again, I might've been wrong :(
Sorry about that.
Jan
-- Pozdrawiam Dominik
-- Pozdrawiam Dominik
-- Pozdrawiam Dominik
Hi, Mabye that's the problem ? : May 27 09:13:44 on-10-177-32-62 kernel: [140204.533733] ixgbe 0000:01:00.0 eth0: VF Reset msg received from vf 13 May 27 09:13:44 on-10-177-32-62 kernel: [140204.533970] ixgbe 0000:01:00.0: VF 13 has no MAC address assigned, you may have to assign one manually May 27 09:13:44 on-10-177-32-62 kernel: [140204.552220] ixgbe 0000:01:00.1 eth1: VF Reset msg received from vf 13 May 27 09:13:44 on-10-177-32-62 kernel: [140204.552228] ixgbe 0000:01:00.1: VF 13 has no MAC address assigned, you may have to assign one manually May 27 09:13:45 on-10-177-32-62 kernel: [140205.674551] ixgbe 0000:01:00.1 eth1: VF Reset msg received from vf 12 May 27 09:13:45 on-10-177-32-62 kernel: [140205.694783] pci-stub 0000:01:13.1: irq 288 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140205.726747] pci-stub 0000:01:13.1: irq 288 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140205.726766] pci-stub 0000:01:13.1: irq 289 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140205.786692] pci-stub 0000:01:13.1: irq 288 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140205.786702] pci-stub 0000:01:13.1: irq 289 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140205.786710] pci-stub 0000:01:13.1: irq 290 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140206.254300] ixgbe 0000:01:00.0 eth0: VF Reset msg received from vf 13 May 27 09:13:45 on-10-177-32-62 kernel: [140206.286346] pci-stub 0000:01:13.2: irq 407 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140206.306280] pci-stub 0000:01:13.2: irq 407 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140206.306293] pci-stub 0000:01:13.2: irq 408 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140206.354379] pci-stub 0000:01:13.2: irq 407 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140206.354393] pci-stub 0000:01:13.2: irq 408 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140206.354403] pci-stub 0000:01:13.2: irq 409 for MSI/MSI-X May 27 09:13:49 on-10-177-32-62 kernel: [140210.174615] ixgbe 0000:01:00.1 eth1: VF Reset msg received from vf 13 May 27 09:13:49 on-10-177-32-62 kernel: [140210.207755] pci-stub 0000:01:13.3: irq 410 for MSI/MSI-X May 27 09:13:49 on-10-177-32-62 kernel: [140210.251553] pci-stub 0000:01:13.3: irq 410 for MSI/MSI-X May 27 09:13:49 on-10-177-32-62 kernel: [140210.251568] pci-stub 0000:01:13.3: irq 411 for MSI/MSI-X May 27 09:13:49 on-10-177-32-62 kernel: [140210.331491] pci-stub 0000:01:13.3: irq 410 for MSI/MSI-X May 27 09:13:49 on-10-177-32-62 kernel: [140210.331506] pci-stub 0000:01:13.3: irq 411 for MSI/MSI-X May 27 09:13:49 on-10-177-32-62 kernel: [140210.331518] pci-stub 0000:01:13.3: irq 412 for MSI/MSI-X Regards Dominik 2013/5/26 Dominik Mostowiec <dominikmostowiec@gmail.com>:
Hi,
First - I just got confirmation from the libnl maintainer that the bug I mentioned causing problems with max_vfs > 50 *is* an issue in libnl 3.2.16 - it is fixed in libnl 3.2.22. Try to upgrade to that version and it should solve at least some of your problems.
Upgrade libnl to version 3.2.22 helps. Thanks.
1) when you have max_vfs = 10, you can assign the devices and mac addresses are set properly, i.e. everything works. 2) when you have max_vfs = 35, you can assign the devices, but the mac addresses are not set properly. 3) when you have max_vfs = 63, you can't assign the devices because you get this error:
internal error missing IFLA_VF_INFO in netlink response
Is this correct?
Yes. Before upgrade libnl :-) Upgrede helps only for limit max_vfs.
(BTW, it may get a bit confusing if you leave your networks named as "vnet0" and "vnet1" - although it's a different namespace, libvirt uses "vnetN" (where n is 0 - whatever) as the name for the tap device associated with each guest interface).
You're right. I changed this.
the output of "ip link show dev $PF". Hmm, ip link show dev eth0 Shows only: 2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000 link/ether b8:ca:3a:5b:a6:3a brd ff:ff:ff:ff:ff:ff
I see VFs like network interfaces eth2-eth127. When VF is attached to VM it disappear from list on host.
-- Regards Dominik
2013/5/24 Laine Stump <laine@laine.org>:
On 05/24/2013 09:16 AM, Dominik Mostowiec wrote:
Do you have this problem with mac addresses when max_vfs is set to an even lower number?
First - I just got confirmation from the libnl maintainer that the bug I mentioned causing problems with max_vfs > 50 *is* an issue in libnl 3.2.16 - it is fixed in libnl 3.2.22. Try to upgrade to that version and it should solve at least some of your problems.
No, When max_fs=10, macs in VM are OK.
Interesting.
So now I'm curious about something - can you try adding a vlan tag to your networks and see if the vlan tag is set properly when max_vfs is a low number (10 or 7):
<network> <name>vnet0</name> <uuid>ec49896a-a0b5-4944-a81f-9f0cdf578871</uuid> <vlan> <tag id='42'/> </vlan> <forward mode='hostdev' managed='yes'> <pf dev='eth0'/> </forward> </network>
(you will have to do "virsh net-destroy vnet0; virsh net-start vnet0" after making the modification to the network). If the vlan tag is successfully set, it will show up next to the mac address for the VF in the output of "ip link show dev $PF".
(BTW, it may get a bit confusing if you leave your networks named as "vnet0" and "vnet1" - although it's a different namespace, libvirt uses "vnetN" (where n is 0 - whatever) as the name for the tap device associated with each guest interface).
To summarize:
1) when you have max_vfs = 10, you can assign the devices and mac addresses are set properly, i.e. everything works. 2) when you have max_vfs = 35, you can assign the devices, but the mac addresses are not set properly. 3) when you have max_vfs = 63, you can't assign the devices because you get this error:
internal error missing IFLA_VF_INFO in netlink response
Is this correct?
Phisical interfaces:
ifconfig eth0 eth0 Link encap:Ethernet HWaddr b8:ca:3a:5b:a6:38 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
Okay, so that's not the problem.
ifconfig eth1 eth1 Link encap:Ethernet HWaddr b8:ca:3a:5b:a6:38 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
Hmm. ixgbe. I just got a report yesterday that setting of the vlan tag for ixgbe vfs was not working properly (although it works fine for igb devices) Domain config: <interface type="network"> <alias name="hostdev0"/> <source network="vnet0"/> <mac address="52:54:0a:b1:48:fc"/> </interface> <interface type="network"> <alias name="hostdev1"/> <source network="vnet1"/> <mac address="52:54:0a:b1:48:fc"/> </interface>
Networks defined: <network> <name>vnet0</name> <uuid>ec49896a-a0b5-4944-a81f-9f0cdf578871</uuid> <forward mode='hostdev' managed='yes'> <pf dev='eth0'/> </forward> </network>
<network> <name>vnet1</name> <uuid>6c8319e8-8b53-4382-9f4e-2400819b00a9</uuid> <forward mode='hostdev' managed='yes'> <pf dev='eth1'/> </forward> </network>
Regards Dominik
2013/5/24 Laine Stump <laine@laine.org>:
On 05/24/2013 04:45 AM, Dominik Mostowiec wrote:
Hi, I have libnl3, kernel 3.8.8, os ubuntu 12.04, net:intel10G. Upgrade libnl to 3.2.16 not helps,
I don't know - this is probably another problem, when I have set max_vfs=35,35 VF is attaching to VM but mac in VM is not set propertly. Do you have this problem with mac addresses when max_vfs is set to an even lower number?
Ah, another thing - did you "ifconfig up" the PF before you started the guest?
Libvirt probably set this macs because in syslog: May 23 14:48:59 on-10-177-32-62 kernel: [12574.171792] ixgbe
Hmm. ixgbe. I just got a report yesterday that setting of the vlan tag for ixgbe vfs was not working properly (although it works fine for igb devices)
0000:01:00.0: setting MAC 52:54:0a:b1:48:f7 on VF 0 May 23 14:48:59 on-10-177-32-62 kernel: [12574.171799] ixgbe 0000:01:00.0: Reload the VF driver to make this change effective. May 23 14:48:59 on-10-177-32-62 kernel: [12574.175054] ixgbe 0000:01:00.1: setting MAC 52:54:0a:b1:48:f7 on VF 0 May 23 14:48:59 on-10-177-32-62 kernel: [12574.175060] ixgbe 0000:01:00.1: Reload the VF driver to make this change effective. May 23 14:48:59 on-10-177-32-62 kernel: [12574.307172] pci-stub 0000:01:12.4: enabling device (0000 -> 0002) May 23 14:49:00 on-10-177-32-62 kernel: [12575.662890] assign device 0:1:12.4 May 23 14:49:00 on-10-177-32-62 kernel: [12575.665210] pci-stub 0000:01:12.5: enabling device (0000 -> 0002) May 23 14:49:00 on-10-177-32-62 kernel: [12575.765609] assign device 0:1:12.5 But in VM i have another macs probably old from VF.
--- Regards Dominik
2013/5/24 Laine Stump <laine@laine.org>:
On 05/22/2013 06:32 PM, Dominik Mostowiec wrote: > I tested on 1.0.5 patched version and vm with 2 vfs working fine. > > I have another problem > When i set max_vfs=63: > internal error missing IFLA_VF_INFO in netlink response > > Its working when max_vfs=31 > What is the platform, kernel version, and what version of libnl are you using (libnl 1 or libnl3?)
There have been multiple bugs recently filed and fixed in this area on RHEL6/CentOS6. In particular, there was a problem with max_vfs > 50 that had exactly this symptom. It was solved by a patch to libnl-1.1. If your platform uses libnl-1.1, this could be the source of the problem. It is fixed in the upstream libnl-1.1.4 maintenance release.
I unfortunately am not sure where the libnl-1.1 git is (I only have the git tree for libnl3), so I can't give you the exact commit message, but in short the problem was that libnl was using too small of a buffer for the netlink socket, so when there were a lot of vfs, the netlink message containing vf info was truncated.
> -- > Dominik > > 21 maj 2013 15:19, "Dominik Mostowiec" <dominikmostowiec@gmail.com > <mailto:dominikmostowiec@gmail.com>> napisał(a): > > Hmm, > It seems to be working (after only simple tests). > > -- > Dominik > > > 2013/5/21 Ján Tomko <jtomko@redhat.com <mailto:jtomko@redhat.com>> > > On 05/21/2013 01:37 PM, Laine Stump wrote: > > On 05/21/2013 04:03 AM, Ján Tomko wrote: > >> On 05/21/2013 09:32 AM, Dominik Mostowiec wrote: > >>> hi, > >>> I try to add 2 VF by "hostdev". > >>> Networks (vnet0, vnet1) with: > >>> <forward mode='hostdev' managed='yes'> > >>> <pf dev='eth1'/> > >>> ..... > >>> > >>> Domain: > >>> <interface type="network"> > >>> <source network="vnet0"/> > >>> .... > >>> <interface type="network"> > >>> <source network="vnet1"/> > >>> .... > >>> > >>> virsh create error: > >>> "error: internal error process exited while connecting to > monitor: kvm: > >>> -device > pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4: > >>> Duplicate ID 'hostdev0' for device" > >>> > >> Hi, > >> > >> it seems we have been assigning the same id to all network > hostdevs until this > >> recent commit (not yet released): > >> > >> http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6597cc25 > > > > > > If that patch has such an effect, it's purely accidental. > Have you > > tested this? (I plan to test a before and after as soon as > I've had > > breakfast) > > > > I haven't tested it and looking at the code again, I might've > been wrong :( > > Sorry about that. > > Jan > > > > > > > -- > Pozdrawiam > Dominik >
-- Pozdrawiam Dominik
-- Pozdrawiam Dominik
-- Pozdrawiam Dominik
On 05/21/2013 07:44 AM, Ján Tomko wrote:
On 05/21/2013 01:37 PM, Laine Stump wrote:
On 05/21/2013 04:03 AM, Ján Tomko wrote:
On 05/21/2013 09:32 AM, Dominik Mostowiec wrote:
hi, I try to add 2 VF by "hostdev". Networks (vnet0, vnet1) with: <forward mode='hostdev' managed='yes'> <pf dev='eth1'/> .....
Domain: <interface type="network"> <source network="vnet0"/> .... <interface type="network"> <source network="vnet1"/> ....
virsh create error: "error: internal error process exited while connecting to monitor: kvm: -device pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4: Duplicate ID 'hostdev0' for device"
Hi,
it seems we have been assigning the same id to all network hostdevs until this recent commit (not yet released):
http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6597cc25
If that patch has such an effect, it's purely accidental. Have you tested this? (I plan to test a before and after as soon as I've had breakfast)
I haven't tested it and looking at the code again, I might've been wrong :(
Sorry about that.
Nothing to be sorry about even if you weren't right. And as it turns out you *were* right! I was curious why, so I tested with a 1.0.4 build running under gdb. The reason for the failure was that the person who wrote the code in qemuBuildCommandLine that called qemuAssignDeviceHostdevAlias() (as well as the person who reviewed that code, which would be me :-/) missed a detail of qemuAssignDeviceHostdevAlias() - it only searches for a free alias index if you send -1 as the index, otherwise it just uses the index you send. This is what was being called: qemuAssignDeviceHostdevAlias(def, hostdev, (def->nhostdevs-1)) So the first time def->nhostdevs-1 == -1, and a search would be made in the (empty) hostdevs array, resulting in an index of 0 being used. The second time it's called, def->nhostdevs-1 == 0, so it would just use 0 without even checking for a conflict. My patch for VFIO (the one Jan pointed out as being a fix) completely removed this code from qemuBuildCommandLine(), relying instead on the alias being assigned with all other aliases in qemuAssignDeviceAliases(). (I didn't do this on purpose to fix this bug though; I did it because the old code had become redundant). Anyway, thanks for being so observant. I otherwise would have wasted investigation time wondering why it wasn't broken for me.
On Tue, May 21, 2013 at 11:29:26AM -0400, Laine Stump wrote:
On 05/21/2013 07:44 AM, Ján Tomko wrote:
On 05/21/2013 01:37 PM, Laine Stump wrote:
On 05/21/2013 04:03 AM, Ján Tomko wrote:
On 05/21/2013 09:32 AM, Dominik Mostowiec wrote:
hi, I try to add 2 VF by "hostdev". Networks (vnet0, vnet1) with: <forward mode='hostdev' managed='yes'> <pf dev='eth1'/> .....
Domain: <interface type="network"> <source network="vnet0"/> .... <interface type="network"> <source network="vnet1"/> ....
virsh create error: "error: internal error process exited while connecting to monitor: kvm: -device pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4: Duplicate ID 'hostdev0' for device"
Hi,
it seems we have been assigning the same id to all network hostdevs until this recent commit (not yet released):
http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6597cc25
If that patch has such an effect, it's purely accidental. Have you tested this? (I plan to test a before and after as soon as I've had breakfast)
I haven't tested it and looking at the code again, I might've been wrong :(
Sorry about that.
Nothing to be sorry about even if you weren't right. And as it turns out you *were* right!
I was curious why, so I tested with a 1.0.4 build running under gdb. The reason for the failure was that the person who wrote the code in qemuBuildCommandLine that called qemuAssignDeviceHostdevAlias() (as well as the person who reviewed that code, which would be me :-/) missed a detail of qemuAssignDeviceHostdevAlias() - it only searches for a free alias index if you send -1 as the index, otherwise it just uses the index you send. This is what was being called:
qemuAssignDeviceHostdevAlias(def, hostdev, (def->nhostdevs-1))
So the first time def->nhostdevs-1 == -1, and a search would be made in the (empty) hostdevs array, resulting in an index of 0 being used. The second time it's called, def->nhostdevs-1 == 0, so it would just use 0 without even checking for a conflict.
My patch for VFIO (the one Jan pointed out as being a fix) completely removed this code from qemuBuildCommandLine(), relying instead on the alias being assigned with all other aliases in qemuAssignDeviceAliases(). (I didn't do this on purpose to fix this bug though; I did it because the old code had become redundant).
Anyway, thanks for being so observant. I otherwise would have wasted investigation time wondering why it wasn't broken for me.
So the fact that we missed this suggests we have a gap in our XML->ARGV test coverage for host device assignment, right ? Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 05/21/2013 11:34 AM, Daniel P. Berrange wrote:
On Tue, May 21, 2013 at 11:29:26AM -0400, Laine Stump wrote:
On 05/21/2013 07:44 AM, Ján Tomko wrote:
On 05/21/2013 01:37 PM, Laine Stump wrote:
On 05/21/2013 04:03 AM, Ján Tomko wrote:
On 05/21/2013 09:32 AM, Dominik Mostowiec wrote:
hi, I try to add 2 VF by "hostdev". Networks (vnet0, vnet1) with: <forward mode='hostdev' managed='yes'> <pf dev='eth1'/> .....
Domain: <interface type="network"> <source network="vnet0"/> .... <interface type="network"> <source network="vnet1"/> ....
virsh create error: "error: internal error process exited while connecting to monitor: kvm: -device pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4: Duplicate ID 'hostdev0' for device"
Hi,
it seems we have been assigning the same id to all network hostdevs until this recent commit (not yet released):
http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6597cc25 If that patch has such an effect, it's purely accidental. Have you tested this? (I plan to test a before and after as soon as I've had breakfast)
I haven't tested it and looking at the code again, I might've been wrong :(
Sorry about that. Nothing to be sorry about even if you weren't right. And as it turns out you *were* right!
I was curious why, so I tested with a 1.0.4 build running under gdb. The reason for the failure was that the person who wrote the code in qemuBuildCommandLine that called qemuAssignDeviceHostdevAlias() (as well as the person who reviewed that code, which would be me :-/) missed a detail of qemuAssignDeviceHostdevAlias() - it only searches for a free alias index if you send -1 as the index, otherwise it just uses the index you send. This is what was being called:
qemuAssignDeviceHostdevAlias(def, hostdev, (def->nhostdevs-1))
So the first time def->nhostdevs-1 == -1, and a search would be made in the (empty) hostdevs array, resulting in an index of 0 being used. The second time it's called, def->nhostdevs-1 == 0, so it would just use 0 without even checking for a conflict.
My patch for VFIO (the one Jan pointed out as being a fix) completely removed this code from qemuBuildCommandLine(), relying instead on the alias being assigned with all other aliases in qemuAssignDeviceAliases(). (I didn't do this on purpose to fix this bug though; I did it because the old code had become redundant).
Anyway, thanks for being so observant. I otherwise would have wasted investigation time wondering why it wasn't broken for me. So the fact that we missed this suggests we have a gap in our XML->ARGV test coverage for host device assignment, right ?
It seems you're on a mission today :-) Yes. The problem with network commandlines has been that the setup of the tap devices has always been depply intertwined with the commandline generation, making it impossible to test from make check. It's been an annoyance to me for a long time, but finally got to my conscious brain last month when I put in the vfio patches. (this was the side topic I opened when Osier sent his patches that add a callback object to the qemuBuildCommandLine() args). This is also a problem with generic legacy hostdev devices (generic vfio host devices don't seem to have a problem), but their only need is a bit of sysfs. I would actually prefer the network device host-side setup to be handled completely outside of qemuBuildCommandLine() in qemuNetworkPrepareDevices() (as is done for usb and pci passthrough devices) rather than in a callback, but that would require sending an array of file descriptors into qemuBuildCommandLine(), and there's no convenient place to do that. (Note that the arg "vmop" is also only needed for network device host-side setup, and so moving that setup outside of qemuBuildCommandline() would eliminate the need for that arg). You've said that you're against adding hidden driver specific state information to the virDomainDef (which makes sense. although I will point out that there is already both driver-specific (everything in <driver>) and state data (the <actual> subelement which isn't driver-specific, but describes what type of network device was actually assigned to the guest) in the virDomainDef), so we're left with either: 1) moving the network device "preparation" into qemuNetworkPrepareDevices, then replacing "vmop" with a "networkData" arg (or maybe "deviceData", so that it can easily have similar stuff for other devices added to it) that contains all the fd's for the tap and vhost-net devices. or 2) moving the network device preparation into a callback that's called from qemuBuildCommandLine(), and keeping vmop simply to pass it into this callback. (unless someone else has a better idea). My opinion is that option (1) is better, because it puts all the netdev setup in a single place (rather than some being in qemuNetworkPrepareDevices() and some in a new callback function), it doesn't add to the qemuBuildCommandLine() arglist, since it removes one arg while adding another, and it makes the netdev setup more symmetrical with error-recovery teardown, which is also called from qemuProcessStart() (as is qemuNetworkPrepareDevices()). As you can tell, I'm still resistant to have a callback (even though the infrastructure is already there), because I think that's still giving qemuBuildCommandLine() an unlisted side effect of messing around with device setup, while it should be simply taking some input and using it to construct a commandline (I don't really have a problem with simply sending it a fake sysfs, as that is just const input data; what I don't like is when it is creating tap devices, etc).
participants (4)
-
Daniel P. Berrange -
Dominik Mostowiec -
Ján Tomko -
Laine Stump