Re: [libvirt] [RFC PATCH 0/5] hotplug: fix premature rebinding of VFIO devices to host

On 06/28/2017 08:24 PM, Michael Roth wrote: > Hi everyone. Hoping to get some feedback on this approach, or some > alternatives proposed below, to the following issue: > > Currently libvirt immediately attempts to rebind a managed device back to the > host driver when it receives a DEVICE_DELETED event from QEMU. This is > problematic for 2 reasons: > > 1) If multiple devices from a group are attached to a guest, this can move > the group into a "non-viable" state where some devices are assigned to > the host and some to the guest. Since we don't support hotplug with managed='yes' of individual (or multiple) functions of a multifunction host device, I don't know that it's very useful to support hot *un*plug of it - it would only be useful if the multi-function device were present in the guest when it was started, and then was hot-unplugged later. And this is all a lot of extra complexity, though, so it would be useful to know what are the scenarios where it would actually be used (i.e. is this a legitimate need, or just an interesting exercise?)

> 2) When QEMU emits the DEVICE_DELETED event, there's still a "finalize" phase > where additional cleanup occurs. In most cases libvirt can ignore this > cleanup, but in the case of VFIO devices this is where closing of a VFIO > group FD occurs, and failing to wait before rebinding the device to the > host driver can result in unexpected behavior. In the case of powernv > hosts at least, this can lead to a host driver crashing due to the default > DMA windows not having been fully-restored yet. The window between this is > and the initial DEVICE_DELETED seems to be ~6 seconds in practice. We've > seen host dumps with Mellanox CX4 VFs being rebound to host driver during > this period (on powernv hosts). I agree with Dan that the situation described here should be considered a qemu bug - according to my understanding (from back at the time DEVICE_DELETED was added to qemu (I think at libvirt's request) qemu should never emit the DEVICE_DELETED event until *everything* related to the device is finished - that was the whole point of adding the event in the first palce. Covering up this bug with a bunch of extra libvirt complexity is just creating the potential for even more bugs in the more complex code.