Re: [libvirt] [PATCH v4 05/10] Split reprobe action from the virPCIUnbindFromStub into a new function

On Fri, 2015-11-20 at 12:24 -0500, Laine Stump wrote: > On 11/20/2015 11:58 AM, Andrea Bolognani wrote: >> On Fri, 2015-11-20 at 11:33 -0500, Laine Stump wrote: >>> Seems safe, but is this really what we want to do? I haven't >>> read/understood the remaining patches yet, but this makes it sound like >>> what is going to happen is that all of the devices will be unbound from >>> vfio-pci immediately, so they are "in limbo", and will then be reprobed >>> once all devices are unused (and therefore unbound from vfio-pci). >>> >>> I think that may be a bit dangerous. Instead, we should leave the >>> devices bound to vfio-pci until all of them are unused, and at that >>> time, we should unbind them all from vfio-pci, then reprobe them all. >>> (again, I may have misunderstood the direction, if so ignore this). >> I agree, we should not unbind any device from vfio-pci until >> all the devices in the IOMMU group have been detached from >> the guest. > ... and I've just looked back at my original comment about this in the > BZ, and see that at that time I only suggested delaying the reprobe, but > said nothing about delaying the unbind. And I'm not as sure about the > necessity of waiting as I was 1/2 an hour ago. I suppose the issue is > that it brings all those unbound devices one step closer to getting > bound to the host driver. However, that will happen only if those > device's PCI addresses are written to "drivers_reprobe" in sysfs (right? > is there any other way a more "global" reprobe could happen and snatch > up everything that's currently unbound?) Any load of a module will snatch up any unclaimed devices that match it, so if you unbind and leave the devices orpaned, a random module load could cause much badness. Adding a new_id will also cause a device scan, so if that happened to match the device: random badness. > So maybe I'd better ask someone who knows more about this than me - > Alex, is there an issue with unbinding some devices in an iommu group > from vfio-pci at an earlier time, and leaving then unbound to any driver > at all while some other devices in the group are still in use by the > guest? Is there an advantage to keeping them all bound to vfio-pci until > none of them are used, and then unbinding/reprobing them all at the same > time? Or should we unbind each from vfio-pci immediately when they are > detached from the guest, and reprobe them all once they're all unbound? Unbinding them from vfio-pci leaves them susceptible to random bad things happen, as outlined above, and potentially limits vfio's ability to do things like bus resets. For instance imagine a 2-port NIC where each port is a PCI function, the functions are grouped together and the devices don't support any sort of internal reset. If both devices are bound to vfio-pci, then the user owns them both and we can do a bus reset. If one of those devices gets released from the user, as soon as it's unbound from vfio-pci it's no longer in our control and the bus rest option is gone. The best course of action would be to leave any managed devices bound to vfio-pci until all of the devices within the group are no longer in use. Thanks,

Alex