Re: [libvirt] [RFC PATCH 00/28] Enable multifunction pci hotplug

On Thu, 15 Mar 2018 14:33:55 +0000 Daniel P. Berrangé <berrange(a)redhat.com&gt; wrote: > On Thu, Mar 15, 2018 at 07:54:47PM +0530, Shivaprasad G Bhat wrote: > > > > > > On 03/15/2018 03:31 PM, Daniel P. Berrangé wrote: > > > On Wed, Mar 14, 2018 at 10:44:30PM +0530, Shivaprasad G Bhat wrote: > > > > Hi All, > > > > > > > > I have revisited/rewritten my previously posted patches. Here is > > > > the RFC. Since this patchset is a complete rewrite, I am starting > > > > with v1 here. > > > > > > > > The semantics is as discussed before > > > > https://www.redhat.com/archives/libvir-list/2016-April/msg01057.html > > > > > > > > As I went on to refactor the code to support multifunction virtio devices, > > > > I realised the abort/cleanup path would be a nightmare there, in case of > > > > failures. So, dropped that attempt. The current RFC limits to the real > > > > practical use cases of Multifunction PCI hostdevices. All new test code > > > > to support multifunction PCI hostdevices and test cases are added to > > > > prove the functionality. > > > I guess I'm not really understanding the use case here. With SRIOV > > > devices, you can already choose between assigning either the physical > > > function (which gives the guest access to all virtual functions), or Say what? If a guest is assigned a PF, they get the PF, they don't get to enable SR-IOV to also get the VFs. But SR-IOV and multifunction are far from synonymous nor is SR-IOV ubiquitous to all use cases, so I don't know why we're bringing SR-IOV into this discussion. > > > to assign an arbitrary set of individiual functions to various guests. > > > Why do we need to be able to list many <hostdev> at the same time > > > when hotplugging to assign multiple functions. > > > > > > Basically can you provide a full description of the problem you are > > > trying to solve and why existing functionality isn't sufficient. > > > > Hi Daniel, > > > > This is for cards which may not necessarily be networking cards. Or may be a > > mix of > > networking and storage. > > > > Suppose, user has below card > > 0005:01:00.0 Ethernet controller: Emulex Corporation OneConnect NIC (Lancer) > > (rev 10) > > 0005:01:00.1 Ethernet controller: Emulex Corporation OneConnect NIC (Lancer) > > (rev 10) > > 0005:01:00.2 Ethernet controller: Emulex Corporation OneConnect NIC (Lancer) > > (rev 10) > > 0005:01:00.3 Ethernet controller: Emulex Corporation OneConnect NIC (Lancer) > > (rev 10) > > 0005:01:00.4 Fibre Channel: Emulex Corporation OneConnect FCoE Initiator > > (Lancer) (rev 10) > > 0005:01:00.5 Fibre Channel: Emulex Corporation OneConnect FCoE Initiator > > (Lancer) (rev 10) > > Ok, so this is a device with many functions, but which isn't SRIOV > based, and the goal is to assign the physical device to the guest, > such that guest has all functions available. > > > If user wants to hotplug this card to guest, He has to detach all the > > functions from host driver, > > then hotplug 0005:01:00.0, 0005:01:00.1, so on individually. But, today with > > each hotplug > > of the function, each <hostdev> goes to different guest slot. Whereas, PCI > > requires all of > > them to be on the same slot. This is not supported on libvirt today. > > > > The multifunction cards cant be hotplugged to guest today with the > > individual > > <hostdev>, as the operation is queued by qemu till the function zero of > > guest slot is > > hotplugged. On function zero hotplug, the qemu sends out the event to guest > > for device probing where all the previously hotplugged functions from the > > same slot are discovered. So, grouping the <hostdev>s within the <devices> > > would become necessary to make the whole thing a single operation. > > So IIUC, from the patches, if the user wants to assign the physical > device to the guest, they would need to provide XML that looked like > this to the virDomainAttachDevice() method: > > <devices> > <hostdev mode='subsystem' type='pci' managed='yes'> > <driver name='vfio'/> > <source> > <address domain='0x0000' bus='0x05' slot='0x1' function='0x0'/> > </source> > </hostdev> > <hostdev mode='subsystem' type='pci' managed='yes'> > <driver name='vfio'/> > <source> > <address domain='0x0000' bus='0x05' slot='0x1' function='0x1'/> > </source> > </hostdev> > <hostdev mode='subsystem' type='pci' managed='yes'> > <driver name='vfio'/> > <source> > <address domain='0x0000' bus='0x05' slot='0x1' function='0x2'/> > </source> > </hostdev> > <hostdev mode='subsystem' type='pci' managed='yes'> > <driver name='vfio'/> > <source> > <address domain='0x0000' bus='0x05' slot='0x1' function='0x3'/> > </source> > </hostdev> > <hostdev mode='subsystem' type='pci' managed='yes'> > <driver name='vfio'/> > <source> > <address domain='0x0000' bus='0x05' slot='0x1' function='0x4'/> > </source> > </hostdev> > <hostdev mode='subsystem' type='pci' managed='yes'> > <driver name='vfio'/> > <source> > <address domain='0x0000' bus='0x05' slot='0x1' function='0x5'/> > </source> > </hostdev> > </devices> > > > Where as if the device were SRIOV based, they would only have to > provide > > <device> > <hostdev mode='subsystem' type='pci' managed='yes'> > <driver name='vfio'/> > <source> > <address domain='0x0000' bus='0x05' slot='0x1' function='0x0'/> > </source> > </hostdev> > </device> > > for the guest to get access to all functions. Since when has this been the case? (nit, the example is domain=0x5, bus=0x1,...) > I find this difference in behaviour and approach really unpleasant. > > I think that they user should only need to provide the the address > of the physical device, in both cases. At most perhaps we need a > new attribute multifunction="on" on the source address to tell > libvirt that it should attach all the functions, not just the > first > > <device> > <hostdev mode='subsystem' type='pci' managed='yes'> > <driver name='vfio'/> > <source> > <address domain='0x0000' bus='0x05' slot='0x1' function='0x0' mutlifunction="on"/> > </source> > </hostdev> > </device> Neither really bothers me, but I'm confused by the claimed existing handling of SR-IOV. Either you're assigning a PF and SR-IOV is irrelevant and unavailable to the guest or you're assigning a VF and, well, SR-IOV is still mostly irrelevant to libvirt unless someone decides to assign the PF hosting the VF or libvirt needs to do VF configuration via the PF. Thanks,