[libvirt] PF interface brought down if guest using a VF in hostdev mode under 802.1Qbh
Hi, In 802.1Qbh case where exists a SR-IOV capable network device, if any of the virtual functions of this device is used in hostdev mode for the guest, port-profile disassociate will also cause physical function interface to go down. This appears to be a bug, but wanted to find if this was done intentionally for some reasons. To be more specific, if a network device supports SRIOV and its VF is being used in pci passthrough mode, when the guest is shutdown or destroyed following happens - qemuProcessStop \_ qemuDomainReAttachHostdevDevices(hostdevs) \_ qemuDomainHostdevNetConfigRestore(hostdev) \_ virNetDevPortProfileDisassociate(linkdev) \_ virNetDevSetOnline(linkdev, false) In above, qemuDomainHostdevNetConfigRestore() finds out the PF for provided hostdev (which is VF) and passes it to virNetDevPortProfileDisassociate() as linkdev. Later, linkdev gets passed to virNetDevSetOnline() where the interface is brought down by clearing IFF_UP flag. However, in macvtap emulation mode, virNetDevMacVLanDeleteWithVPortProfile() passes VF as linkdev (and -1 as vf) to virNetDevPortProfileDisassociate(), hence, not affecting PF at all. Bringing down a PF, when only VF is being brought down is not expected behavior (unless, I'm missing something here). A way to get around it would be to check if there exists vf (>=0) in virNetDevPortProfileDisassociate(), and if so, it should only pass the interface name of this VF rather than PF itself to virNetDevSetOnline(). If it sounds reasonable, I'll send out a fix. Thanks, Nishank
On 08/07/2012 01:53 PM, Nishank Trivedi (nistrive) wrote:
Hi,
In 802.1Qbh case where exists a SR-IOV capable network device, if any of the virtual functions of this device is used in hostdev mode for the guest, port-profile disassociate will also cause physical function interface to go down. This appears to be a bug, but wanted to find if this was done intentionally for some reasons.
To be more specific, if a network device supports SRIOV and its VF is being used in pci passthrough mode, when the guest is shutdown or destroyed following happens -
qemuProcessStop \_ qemuDomainReAttachHostdevDevices(hostdevs) \_ qemuDomainHostdevNetConfigRestore(hostdev) \_ virNetDevPortProfileDisassociate(linkdev) \_ virNetDevSetOnline(linkdev, false)
In above, qemuDomainHostdevNetConfigRestore() finds out the PF for provided hostdev (which is VF) and passes it to virNetDevPortProfileDisassociate() as linkdev. Later, linkdev gets passed to virNetDevSetOnline() where the interface is brought down by clearing IFF_UP flag.
However, in macvtap emulation mode, virNetDevMacVLanDeleteWithVPortProfile() passes VF as linkdev (and -1 as vf) to virNetDevPortProfileDisassociate(), hence, not affecting PF at all.
Bringing down a PF, when only VF is being brought down is not expected behavior (unless, I'm missing something here). A way to get around it would be to check if there exists vf (>=0) in virNetDevPortProfileDisassociate(), and if so, it should only pass the interface name of this VF rather than PF itself to virNetDevSetOnline(). If it sounds reasonable, I'll send out a fix.
Yes, I believe you are correct that virNetDevSetOnline is being called with the wrong device name (I could be wrong though - is there maybe some odd rule that the PF must be UP before the VFs will be up? In case case, we certainly shouldn't be setting the PF to IFF_DOWN). But in the case of PCI passthrough, does it even need to be called at all? Cc'ing Roopa in case she missed this in all the other traffic.
On 8/7/12 8:15 PM, "Laine Stump" <laine@laine.org> wrote:
Yes, I believe you are correct that virNetDevSetOnline is being called with the wrong device name (I could be wrong though - is there maybe some odd rule that the PF must be UP before the VFs will be up?
PF should be UP prior to VF only if there is some proxy'ing or some control data being sent by PF on behalf of VF. Same could be true for bringing VF down.
In case case, we certainly shouldn't be setting the PF to IFF_DOWN). But in the case of PCI passthrough, does it even need to be called at all?
Not for PF for sure. While VF is getting disassociated with port profile, VF interface still needs to be brought down. So virNetDevSetOnline should be passed VF interface, if vf >= 0.
Cc'ing Roopa in case she missed this in all the other traffic.
Invalid email id, cc'ing Christian instead. Thanks, Nishank
-----Original Message----- From: Nishank Trivedi (nistrive) Sent: Wednesday, August 08, 2012 10:22 AM To: Laine Stump; libvir-list@redhat.com Cc: Christian Benvenuti (benve) Subject: Re: [libvirt] PF interface brought down if guest using a VF in hostdev mode under 802.1Qbh
On 8/7/12 8:15 PM, "Laine Stump" <laine@laine.org> wrote:
Yes, I believe you are correct that virNetDevSetOnline is being called with the wrong device name (I could be wrong though - is there maybe some odd rule that the PF must be UP before the VFs will be up?
PF should be UP prior to VF only if there is some proxy'ing or some control data being sent by PF on behalf of VF. Same could be true for bringing VF down.
In case case, we certainly shouldn't be setting the PF to IFF_DOWN). But in the case of PCI passthrough, does it even need to be called at all?
Not for PF for sure. While VF is getting disassociated with port profile, VF interface still needs to be brought down. So virNetDevSetOnline should be passed VF interface, if vf >= 0.
Right, this looks like a bug. Most likely we somehow did not catch it when this patchset https://www.redhat.com/archives/libvir-list/2012-March/msg00015.html got reviewed. Thanks /Chris
Cc'ing Roopa in case she missed this in all the other traffic.
Invalid email id, cc'ing Christian instead.
Thanks, Nishank
On 08/08/2012 01:21 PM, Nishank Trivedi (nistrive) wrote:
On 8/7/12 8:15 PM, "Laine Stump" <laine@laine.org> wrote:
Yes, I believe you are correct that virNetDevSetOnline is being called with the wrong device name (I could be wrong though - is there maybe some odd rule that the PF must be UP before the VFs will be up? PF should be UP prior to VF only if there is some proxy'ing or some control data being sent by PF on behalf of VF. Same could be true for bringing VF down.
In case case, we certainly shouldn't be setting the PF to IFF_DOWN). But in the case of PCI passthrough, does it even need to be called at all? Not for PF for sure. While VF is getting disassociated with port profile, VF interface still needs to be brought down. So virNetDevSetOnline should be passed VF interface, if vf >= 0.
Cc'ing Roopa in case she missed this in all the other traffic. Invalid email id, cc'ing Christian instead.
Yes, I didn't realize until I got the bounce from that mail that Roopa had moved. (also didn't pay attention to the fact that you are at Cisco, so would have contacted her anyway :-) It sounds like you guys have a good understanding of the details, and the proper setup to test it with 802.1Qbh, so please write up a patch and send it in. I'll review it and test it here too (although I can only test netdev PCI passthrough *without* 802.1Qbh). Thanks for finding the root of the problem!
participants (3)
-
Christian Benvenuti (benve) -
Laine Stump -
Nishank Trivedi (nistrive)