Re: [libvirt] RFC: PCI-Passthrough of SRIOV VF's new forward mode

Hello Laine, Many Thanks for reviewing the RFC. Please find my reply inline. On 02/07/2012 02:36 AM, Laine Stump wrote: I was thinking more like <interface type='network'> and in the network xml <forward mode='pci-passthrough'/'hostdev'> since I was thinking adding a new mode to the existing enum virNetworkForwardType. So currently the virNetworkForwardType has vepa, private, bridge, passthrough. I was thinking of adding 1) pci-passthrough or hostdev (VF passthrough to the guest, no virtio interface in the guest, as suggested in your previous proposals) 2) pci-passthrough-hybrid or hostdev-hybrid (VF passthrough to the guest + virtio interface in the guest to support migration with maximum performance results) I was thinking on terms of having the source dev mentioned in the network XML which will suppress any problems we might face while migration. Having a <source dev='eth22> in the domain XML will mean that a similar device needs to be present on the destination host after migration else migration would fail. This sounds like a good idea when using Solarflare network adapter as Solarflare VF's do not have a net device name and operate with PCI addresses. The work that we are trying to achieve follows definitely the same path and it would indeed be a great idea to share notes and part of code between ourselves before we submit patches upstream. Solarflare do not provide a separate VF driver (like the ixgbevf), we provide only a PF driver (sfc) hence the VF doesn't correspond to an ethernet device. Yes, that is correct. In the hybrid model the guest will have a virtio network device along with a passthrough VF as a PCI device. Solarflare provide a guest driver that works along with the virtio driver. The guest driver is called XNAP. The XNAP driver is also called the 'plugin' The Solarflare model for SR-IOV support is a hybrid of the two approaches. It uses a “plugin” approach which maintains the traditional (software) data path through virtio frontend to the KVM host (and then through the Linux bridge to the PF network driver). However, there is also an alternative (accelerated) data path through the VF directly to the network adapter from the guest. Packets can be received on either data path transparently to the guest VM’s network stack and on transmit the plugin (if loaded and enabled) takes the decision on whether to use the accelerated path. A VM can be created/cloned using traditional tools and networking to/from the VM initially uses the standard software network path. If a VF on the network adapter is then passed-through into the guest, the guest sees new hardware has been “hot-plugged” and binds the Solarflare plugin driver to this VF. This plugin driver automatically registers with the virtio driver as an accelerated network plugin. Once the VF driver has registered, subsequent traffic to/from the guest uses the accelerated data path accessing the adapter directly from the guest. If the VF is hot “unplugged” (i.e. removed from the guest), the plugin deregisters with the virtio front end and the networking traffic reverts to the software data path. This approach means there is no dependency on the VF or its driver for the networking data path of the VM. Acceleration can be disabled at any time if needed without losing network connectivity. Migration is fully supported in this model – between hosts with identical network adapters AND also between non-identical hosts. In this XML snippet I use <pf dev='eth2'>. Libvirt will first implicitly autogenerate a list of VF's from the PF. Apart from setting the MAC address of the VF and associating with a virtual port libvirt will have to create a macvtap/bridged interface on eth2 (virtio in the guest) and pci-passthrough a VF (attached to eth2) into the guest Yes the actual part will not show in the static config I agree with using the PCI API I was thinking of adding a field called vf_pci_addr, and saving the PCI addr as a string, to the ActualNetDef structure like below: struct _virDomainActualNetDef { int type; /* enum virDomainNetType */ union { struct { char *brname; } bridge; struct { char *linkdev; char *vf_pci_addr; ( This will store the vf_pci_addr) int mode; /* enum virMacvtapMode from util/macvtap.h */ virVirtualPortProfileParamsPtr virtPortProfile; } direct; } data; virBandwidthPtr bandwidth; }; :-) We actually have a working hybrid model with exceptional performance available with the libvirt patches for RHEL6.1. I am currently working on porting these libvirt patches for RHEL6.2 for some of our customers as well as working on cleaning these patches to submit upstream. We would be happy to send you some of our hardware if you wish to test our hybrid model on RHEL6.1. I definitely think we should work together offline. I would be happy to share notes and code that I am currently working on. Do let me know how I could be of help in our joint effort to get pci-passthrough support for SRIOV VF's. Many Thanks, Regards, Shradha Shah