On Wed, 2015-12-09 at 13:51 +0100, Andrea Bolognani wrote:
On Wed, 2015-12-02 at 18:17 +0100, Andrea Bolognani wrote:
This series is my attempt at fixing
[...]
The problem being solved is that, when using VFIO, IOMMU group ownership can't be shared, eg. two devices that are in the same IOMMU group can't be assigned to different guests, or to the host and a guest. If that happens, the host will probably crash.
The series deals with this issue by making sure safety conditions are met before detaching devices from the host or reattaching them to the host. In praticular, when we're asked to reattach a device to the host but doing so would lead to sharing IOMMU group ownership, we delay the operation until we can guarantee this will not cause problems. As a nice side effect of the changes we check for this when starting a guest too, instead of assuming it will work and having QEMU error out immediately afterwards.
Shivaprasad raised a concern on IRC, which I'm sharing here for wider discussion. I'm CC'ing Laine and Alex, hopefully they don't mind - let me know otherwise.
Assume we have a PCI device with two functions. With this series applied, when reattaching both functions to the host this would happen:
f0 remove from guest f1 remove from guest f0 unbind from vfio-pci f0 trigger host driver reprobe f1 unbind from vfio-pci f1 trigger host driver reprobe
Shivaprasad is concerned this is not actually safe, and the proper sequence would rather be:
f0 remove from guest f1 remove from guest f0 unbind from vfio-pci f1 unbind from vfio-pci f0 trigger host driver reprobe f1 trigger host driver reprobe
Doing so would AFAICT mean basically duplicating the delay logic this series adds to virHostdev into virPCI, to ensure that devices are unbound from vfio-pci only once the same operation has been requested for all devices in the IOMMU group, and reprobe is triggered only after all devices have been unbound from vfio-pci.
I was under the impression that what the current series does, eg. sharing devices in the same IOMMU group between the host driver and vfio-pci is safe as long as no guest is using them at the same time, and that devices could be safely "moved" between the host driver (eg. in use) and vfio-pci (eg. idle, waiting to be assigned to a guest) as many times as desired without ill consequences.
Is my understanding wrong? Do I need to rework the series so that unbinds and reprobes are always executed across the IOMMU group?
Any suggestion or pointers to relevant documentation will be very much appreciated.
Hi Andrea, Your understanding is correct, so I think it just comes down to how sure you are that the vfio group is idle/unused. If there's any risk that a device is still in use by QEMU, then we haven't solved the original problem. Unbinding isn't the only way to have good confidence of this though. You could track the QEMU pid, you could use fuser to make sure the group is not in use, you could try to open the group yourself to make sure it's not in use, and of course you can unbind as proposed in the second option. So long as you know the group is idle, somehow, it shouldn't matter what order you unbind and reprobe. Thanks, Alex