Re: [PATCH RESEND 06/20] virhostdev.c: virHostdevGetPCIHostDevice() now reports missing device

Sorry for the delay in answering. I usually get the replies to my patches in my inbox because I usually get my email CC'ed in the reply. On 1/20/21 10:33 PM, Laine Stump wrote: This was the idea with this series. I'm trying my best to not interfere with working code/logic by adding these mitigations. ---- long story warning ---- We have 2 scenarios I'm trying to deal with in this work: a SR-IOV disappearing from the host, while a running domain was using one of its functions, in a condition where: 1) the VF was coldplugged in the domain (i.e. it was in the domain XML before starting the domain) 2) the VF was hotplugged in the domain In scenario (2), QEMU will throw a DEVICE_DEL event and we'll capture it accordingly via processDeviceDeleteEvent ->qemuDomainRemoveDevice -> qemuDomainRemoveHostDevice. The problem is,by the time we reach qemuDomainRemoveHostDevice, the VF is already gone from the host and we didn't foresee this as a possibility. All of this is pinned on how we interpret virPCIDeviceNew(). My initial idea for this work was to change virPCIDeviceNew() to allow the object to be created, regardless of whether the device actually exists in the host, and then we would have a 'present' flag to determine if the device is present or not. I got terrified by the idea of changing the semantics of virPCIDeviceNew(), which will return -1 if the device isn't present, for all Libvirt devices that uses the object, just to handle a niche case. I decided then to do the 'device is present' checks you'll see around in this series. As far as scenario (1) goes, I don't fully understand why it happens TBH. For QEMU there is no difference - QEMU will throw a DEVICE_DEL event for a VF that disappears from the host, even if the VF was coldplugged. When launching the VM with Libvirt, QEMU internals aren't aware of kernel intent to remove the device. Looking in the vfio driver in the kernel I learned that this intent is expressed by an eventfd call of vfio_pci_request. My somewhat educated guess is that Libvirt is wrapping up the QEMU process in a way that QEMU never sees this eventfd. The result, as you can see in the bug, is that unless the admin kills the terminal where "echo 0 > num_vfs" was issue, 'echo' will lock for as long as the domain using the VF keeps running. And when the domain is shut down the VF is removed from the process, and the host, and then we have a similar problem in our shutdown path, via qemuProcessStop. I'm not sure what would happen. I never though that this could trigger a TOCTOU situation. I **think** that this wouldn't be too nefarious - we would remove the device from the domain definition and the device would be there in the host. Would need to try to simulate this to be sure though. Thanks, DHB