[libvirt] [RFC] On present using dummy hostdev usb device
Hi, all! We use an interesting approach when starting/migrating/etc domain with usb hostdev with startupPolicy=optional. We add qemu usb-host device with missing hostaddr/hostbus parameters (dummy device). I guess there are 2 reasons why we do it. First without dummy device migration will fail as described in [1]. Second is an interesting property of dummy device that qemu starts to monitor for attaching of usb devices and binds the first attached to node to the dummy device. So one can start a domain with missing hostdev and attach it later or migrate a domain then detach hostdev on source and attach it on destination. But as qemu binds the first attached device this is not reliable, to say the least. And after all this does not work if domain uses distinct mount namespace which is default. So I question does it make sense to use dummy device at all? In case of migration/resume from suspend/revert to snapshot we can either fix qemu to ignore incoming missing hostdev data or add dummy device temporarily. The latter solution is worse as it brings dummy device behaviour even for a short period of time. However having a temporary dummy device is neccessary step towards the time when all supported versions of qemu do the mentioned ignoring. As to handling attaching of missing hostdev device to node it should be done in libvirt which can do necessary mount namespace actions. (Actually I developing such patches right now but some peculiarities of dummy device bring me here). Nikolay [1] https://www.redhat.com/archives/libvir-list/2012-October/msg00440.html
On Fri, Aug 30, 2019 at 08:44:03AM +0000, Nikolay Shirokovskiy wrote:
Hi, all!
We use an interesting approach when starting/migrating/etc domain with usb hostdev with startupPolicy=optional. We add qemu usb-host device with missing hostaddr/hostbus parameters (dummy device). I guess there are 2 reasons why we do it. First without dummy device migration will fail as described in [1]. Second is an interesting property of dummy device that qemu starts to monitor for attaching of usb devices and binds the first attached to node to the dummy device. So one can start a domain with missing hostdev and attach it later or migrate a domain then detach hostdev on source and attach it on destination. But as qemu binds the first attached device this is not reliable, to say the least. And after all this does not work if domain uses distinct mount namespace which is default.
Even without mount namespaces, it should fail as QEMU is running non-root and libvirt won't have granted access to any host USB devices in /dev, and also SELinux policy will forbid this.
So I question does it make sense to use dummy device at all? In case of migration/resume from suspend/revert to snapshot we can either fix qemu to ignore incoming missing hostdev data or add dummy device temporarily. The latter solution is worse as it brings dummy device behaviour even for a short period of time. However having a temporary dummy device is neccessary step towards the time when all supported versions of qemu do the mentioned ignoring. As to handling attaching of missing hostdev device to node it should be done in libvirt which can do necessary mount namespace actions. (Actually I developing such patches right now but some peculiarities of dummy device bring me here).
The problems around host USB device passthrough are conceptually similar to the problems of hots PCI device passthrough. In both cases we cannot assume the device present on the source device exists on the target device in the same way. In both cases, even if the device does exist on the target, we cannot serialize the state of the host device across the migration. For PCI devices we simply refuse to initiate the migration if any host PCI devices are attached. The mgmt app has to hot-unplug all devices before migration, and hot-plug new devices after migration if desired. I'm inclined to suggest that same approach of hotunplug + hotplug either side of migration is the only viable option for host USB devices too. As such any mgmt app could do this dance today without any changes in libvirt. If we turned host USB devices into a migration blocker though, that could be considered a significant change of behaviour for mgmt apps, even though this dummy USB device is effectively useless due to our security policies. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Fri, Aug 30, 2019 at 10:09:06AM +0100, Daniel P. Berrangé wrote:
On Fri, Aug 30, 2019 at 08:44:03AM +0000, Nikolay Shirokovskiy wrote:
Hi, all!
We use an interesting approach when starting/migrating/etc domain with usb hostdev with startupPolicy=optional. We add qemu usb-host device with missing hostaddr/hostbus parameters (dummy device). I guess there are 2 reasons why we do it. First without dummy device migration will fail as described in [1]. Second is an interesting property of dummy device that qemu starts to monitor for attaching of usb devices and binds the first attached to node to the dummy device. So one can start a domain with missing hostdev and attach it later or migrate a domain then detach hostdev on source and attach it on destination. But as qemu binds the first attached device this is not reliable, to say the least. And after all this does not work if domain uses distinct mount namespace which is default.
Even without mount namespaces, it should fail as QEMU is running non-root and libvirt won't have granted access to any host USB devices in /dev, and also SELinux policy will forbid this.
Right, but the case with mount namespaces is particularly problematic: if the device open fails due to missing device node, libusb removes the device from its internal device list. This results in the following scenario: - libvirt adds a dummy usb-host device to QEMU in place of a missing device - QEMU (via libusb) installs a watch for udev add events - the physical device is plugged into the host - QEMU detects the addition of the device and, since the dummy device matches everything, tries to open it - by this time libvirt may have not created a device node in QEMU's mount namespace, so the open fails due to missing device node, and libusb removes the device from its internal list - libvirt removes the dummy usb-host device and adds the actual usb-host device - QEMU fails to open it because it's no longer seen by libusb IOW a usb-host device with missing=true can't (reliably, because sometimes libvirt is quick enough to create the device node before QEMU gives up opening it) turn into a working one without QEMU restart.
So I question does it make sense to use dummy device at all? In case of migration/resume from suspend/revert to snapshot we can either fix qemu to ignore incoming missing hostdev data or add dummy device temporarily. The latter solution is worse as it brings dummy device behaviour even for a short period of time. However having a temporary dummy device is neccessary step towards the time when all supported versions of qemu do the mentioned ignoring. As to handling attaching of missing hostdev device to node it should be done in libvirt which can do necessary mount namespace actions. (Actually I developing such patches right now but some peculiarities of dummy device bring me here).
The problems around host USB device passthrough are conceptually similar to the problems of hots PCI device passthrough.
In both cases we cannot assume the device present on the source device exists on the target device in the same way.
In both cases, even if the device does exist on the target, we cannot serialize the state of the host device across the migration.
Right.
For PCI devices we simply refuse to initiate the migration if any host PCI devices are attached. The mgmt app has to hot-unplug all devices before migration, and hot-plug new devices after migration if desired.
I'm inclined to suggest that same approach of hotunplug + hotplug either side of migration is the only viable option for host USB devices too.
As such any mgmt app could do this dance today without any changes in libvirt.
Are you trying to say that the mgmt app should just refrain from creating usb-host devices with missing=true?
If we turned host USB devices into a migration blocker though, that could be considered a significant change of behaviour for mgmt apps, even though this dummy USB device is effectively useless due to our security policies.
I'm afraid the issue is a bit more severe: the dummy device isn't just useless, it stands in the way of the real device later on. Thanks, Roman.
On 8/30/19 2:30 PM, Roman Kagan wrote:
On Fri, Aug 30, 2019 at 10:09:06AM +0100, Daniel P. Berrangé wrote:
On Fri, Aug 30, 2019 at 08:44:03AM +0000, Nikolay Shirokovskiy wrote:
Hi, all!
We use an interesting approach when starting/migrating/etc domain with usb hostdev with startupPolicy=optional. We add qemu usb-host device with missing hostaddr/hostbus parameters (dummy device). I guess there are 2 reasons why we do it. First without dummy device migration will fail as described in [1]. Second is an interesting property of dummy device that qemu starts to monitor for attaching of usb devices and binds the first attached to node to the dummy device. So one can start a domain with missing hostdev and attach it later or migrate a domain then detach hostdev on source and attach it on destination. But as qemu binds the first attached device this is not reliable, to say the least. And after all this does not work if domain uses distinct mount namespace which is default.
Even without mount namespaces, it should fail as QEMU is running non-root and libvirt won't have granted access to any host USB devices in /dev, and also SELinux policy will forbid this.
Right, but the case with mount namespaces is particularly problematic: if the device open fails due to missing device node, libusb removes the device from its internal device list. This results in the following scenario:
- libvirt adds a dummy usb-host device to QEMU in place of a missing device
- QEMU (via libusb) installs a watch for udev add events
- the physical device is plugged into the host
- QEMU detects the addition of the device and, since the dummy device matches everything, tries to open it
- by this time libvirt may have not created a device node in QEMU's mount namespace, so the open fails due to missing device node, and libusb removes the device from its internal list
- libvirt removes the dummy usb-host device and adds the actual usb-host device
- QEMU fails to open it because it's no longer seen by libusb
There is a bug filed against libusb exactly for this: https://bugzilla.redhat.com/show_bug.cgi?id=1595525 BTW: you don't have to migrate, it's sufficient to start a domain with a missing USB and startupPolicy='optional' and then physically plug it into the host and then try to hotplug it into the domain. Michal
On Fri, Aug 30, 2019 at 05:08:52PM +0200, Michal Privoznik wrote:
On 8/30/19 2:30 PM, Roman Kagan wrote:
On Fri, Aug 30, 2019 at 10:09:06AM +0100, Daniel P. Berrangé wrote:
On Fri, Aug 30, 2019 at 08:44:03AM +0000, Nikolay Shirokovskiy wrote:
Hi, all!
We use an interesting approach when starting/migrating/etc domain with usb hostdev with startupPolicy=optional. We add qemu usb-host device with missing hostaddr/hostbus parameters (dummy device). I guess there are 2 reasons why we do it. First without dummy device migration will fail as described in [1]. Second is an interesting property of dummy device that qemu starts to monitor for attaching of usb devices and binds the first attached to node to the dummy device. So one can start a domain with missing hostdev and attach it later or migrate a domain then detach hostdev on source and attach it on destination. But as qemu binds the first attached device this is not reliable, to say the least. And after all this does not work if domain uses distinct mount namespace which is default.
Even without mount namespaces, it should fail as QEMU is running non-root and libvirt won't have granted access to any host USB devices in /dev, and also SELinux policy will forbid this.
Right, but the case with mount namespaces is particularly problematic: if the device open fails due to missing device node, libusb removes the device from its internal device list. This results in the following scenario:
- libvirt adds a dummy usb-host device to QEMU in place of a missing device
- QEMU (via libusb) installs a watch for udev add events
- the physical device is plugged into the host
- QEMU detects the addition of the device and, since the dummy device matches everything, tries to open it
- by this time libvirt may have not created a device node in QEMU's mount namespace, so the open fails due to missing device node, and libusb removes the device from its internal list
- libvirt removes the dummy usb-host device and adds the actual usb-host device
- QEMU fails to open it because it's no longer seen by libusb
There is a bug filed against libusb exactly for this:
https://bugzilla.redhat.com/show_bug.cgi?id=1595525
BTW: you don't have to migrate, it's sufficient to start a domain with a missing USB and startupPolicy='optional' and then physically plug it into the host and then try to hotplug it into the domain.
AFAIR, the startupPolicy=optional was never really intended to allow for the device to be dynamically added after startup. It was only trying drop the device if it didn't exist at startup. If we do want a way to dynamically add after startup, then libvirt itself would have to monitor the USB devices on the host, and then perform QMP commands to hot-add to QEMU. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 8/30/19 5:19 PM, Daniel P. Berrangé wrote:
On Fri, Aug 30, 2019 at 05:08:52PM +0200, Michal Privoznik wrote:
On 8/30/19 2:30 PM, Roman Kagan wrote:
On Fri, Aug 30, 2019 at 10:09:06AM +0100, Daniel P. Berrangé wrote:
On Fri, Aug 30, 2019 at 08:44:03AM +0000, Nikolay Shirokovskiy wrote:
Hi, all!
We use an interesting approach when starting/migrating/etc domain with usb hostdev with startupPolicy=optional. We add qemu usb-host device with missing hostaddr/hostbus parameters (dummy device). I guess there are 2 reasons why we do it. First without dummy device migration will fail as described in [1]. Second is an interesting property of dummy device that qemu starts to monitor for attaching of usb devices and binds the first attached to node to the dummy device. So one can start a domain with missing hostdev and attach it later or migrate a domain then detach hostdev on source and attach it on destination. But as qemu binds the first attached device this is not reliable, to say the least. And after all this does not work if domain uses distinct mount namespace which is default.
Even without mount namespaces, it should fail as QEMU is running non-root and libvirt won't have granted access to any host USB devices in /dev, and also SELinux policy will forbid this.
Right, but the case with mount namespaces is particularly problematic: if the device open fails due to missing device node, libusb removes the device from its internal device list. This results in the following scenario:
- libvirt adds a dummy usb-host device to QEMU in place of a missing device
- QEMU (via libusb) installs a watch for udev add events
- the physical device is plugged into the host
- QEMU detects the addition of the device and, since the dummy device matches everything, tries to open it
- by this time libvirt may have not created a device node in QEMU's mount namespace, so the open fails due to missing device node, and libusb removes the device from its internal list
- libvirt removes the dummy usb-host device and adds the actual usb-host device
- QEMU fails to open it because it's no longer seen by libusb
There is a bug filed against libusb exactly for this:
https://bugzilla.redhat.com/show_bug.cgi?id=1595525
BTW: you don't have to migrate, it's sufficient to start a domain with a missing USB and startupPolicy='optional' and then physically plug it into the host and then try to hotplug it into the domain.
AFAIR, the startupPolicy=optional was never really intended to allow for the device to be dynamically added after startup. It was only trying drop the device if it didn't exist at startup.
Sure, I'm not saying that. That's why I'm saying explicitly you need to hotplug the device that was misssing. I'm only supporting our case of not adding dummy device to libvirt since there's a bug filed against libusb that exhibits same symptoms. Perhaps I should have said that more clearly. Michal
On Fri, Aug 30, 2019 at 04:19:23PM +0100, Daniel P. Berrangé wrote:
On Fri, Aug 30, 2019 at 05:08:52PM +0200, Michal Privoznik wrote:
On 8/30/19 2:30 PM, Roman Kagan wrote:
On Fri, Aug 30, 2019 at 10:09:06AM +0100, Daniel P. Berrangé wrote:
On Fri, Aug 30, 2019 at 08:44:03AM +0000, Nikolay Shirokovskiy wrote:
Hi, all!
We use an interesting approach when starting/migrating/etc domain with usb hostdev with startupPolicy=optional. We add qemu usb-host device with missing hostaddr/hostbus parameters (dummy device). I guess there are 2 reasons why we do it. First without dummy device migration will fail as described in [1]. Second is an interesting property of dummy device that qemu starts to monitor for attaching of usb devices and binds the first attached to node to the dummy device. So one can start a domain with missing hostdev and attach it later or migrate a domain then detach hostdev on source and attach it on destination. But as qemu binds the first attached device this is not reliable, to say the least. And after all this does not work if domain uses distinct mount namespace which is default.
Even without mount namespaces, it should fail as QEMU is running non-root and libvirt won't have granted access to any host USB devices in /dev, and also SELinux policy will forbid this.
Right, but the case with mount namespaces is particularly problematic: if the device open fails due to missing device node, libusb removes the device from its internal device list. This results in the following scenario:
- libvirt adds a dummy usb-host device to QEMU in place of a missing device
- QEMU (via libusb) installs a watch for udev add events
- the physical device is plugged into the host
- QEMU detects the addition of the device and, since the dummy device matches everything, tries to open it
- by this time libvirt may have not created a device node in QEMU's mount namespace, so the open fails due to missing device node, and libusb removes the device from its internal list
- libvirt removes the dummy usb-host device and adds the actual usb-host device
- QEMU fails to open it because it's no longer seen by libusb
There is a bug filed against libusb exactly for this:
https://bugzilla.redhat.com/show_bug.cgi?id=1595525
BTW: you don't have to migrate, it's sufficient to start a domain with a missing USB and startupPolicy='optional' and then physically plug it into the host and then try to hotplug it into the domain.
AFAIR, the startupPolicy=optional was never really intended to allow for the device to be dynamically added after startup. It was only trying drop the device if it didn't exist at startup.
If we do want a way to dynamically add after startup, then libvirt itself would have to monitor the USB devices on the host, and then perform QMP commands to hot-add to QEMU.
This is actually what Nikolay was trying to achieve with his patchset (https://www.redhat.com/archives/libvir-list/2019-August/msg01413.html). Is this a NAK to it, with a suggestion that it's the upper layer's responsibility? Thanks, Roman.
On Mon, Sep 02, 2019 at 02:11:01PM +0000, Roman Kagan wrote:
On Fri, Aug 30, 2019 at 04:19:23PM +0100, Daniel P. Berrangé wrote:
On Fri, Aug 30, 2019 at 05:08:52PM +0200, Michal Privoznik wrote:
On 8/30/19 2:30 PM, Roman Kagan wrote:
On Fri, Aug 30, 2019 at 10:09:06AM +0100, Daniel P. Berrangé wrote:
On Fri, Aug 30, 2019 at 08:44:03AM +0000, Nikolay Shirokovskiy wrote:
Hi, all!
We use an interesting approach when starting/migrating/etc domain with usb hostdev with startupPolicy=optional. We add qemu usb-host device with missing hostaddr/hostbus parameters (dummy device). I guess there are 2 reasons why we do it. First without dummy device migration will fail as described in [1]. Second is an interesting property of dummy device that qemu starts to monitor for attaching of usb devices and binds the first attached to node to the dummy device. So one can start a domain with missing hostdev and attach it later or migrate a domain then detach hostdev on source and attach it on destination. But as qemu binds the first attached device this is not reliable, to say the least. And after all this does not work if domain uses distinct mount namespace which is default.
Even without mount namespaces, it should fail as QEMU is running non-root and libvirt won't have granted access to any host USB devices in /dev, and also SELinux policy will forbid this.
Right, but the case with mount namespaces is particularly problematic: if the device open fails due to missing device node, libusb removes the device from its internal device list. This results in the following scenario:
- libvirt adds a dummy usb-host device to QEMU in place of a missing device
- QEMU (via libusb) installs a watch for udev add events
- the physical device is plugged into the host
- QEMU detects the addition of the device and, since the dummy device matches everything, tries to open it
- by this time libvirt may have not created a device node in QEMU's mount namespace, so the open fails due to missing device node, and libusb removes the device from its internal list
- libvirt removes the dummy usb-host device and adds the actual usb-host device
- QEMU fails to open it because it's no longer seen by libusb
There is a bug filed against libusb exactly for this:
https://bugzilla.redhat.com/show_bug.cgi?id=1595525
BTW: you don't have to migrate, it's sufficient to start a domain with a missing USB and startupPolicy='optional' and then physically plug it into the host and then try to hotplug it into the domain.
AFAIR, the startupPolicy=optional was never really intended to allow for the device to be dynamically added after startup. It was only trying drop the device if it didn't exist at startup.
If we do want a way to dynamically add after startup, then libvirt itself would have to monitor the USB devices on the host, and then perform QMP commands to hot-add to QEMU.
This is actually what Nikolay was trying to achieve with his patchset (https://www.redhat.com/archives/libvir-list/2019-August/msg01413.html). Is this a NAK to it, with a suggestion that it's the upper layer's responsibility?
I think so. The error handling around hostdevs and migration has some quite nasty edge cases. In particular if you unplug before migrating and migrate fails, and you try to restart the source VM, but then have failures re-adding the USB devices. The mgmt app has to interogate libvirt to see what USB devices were succesfully attached and which failed, so they can account for their usage. At that point you might as well just have the mgmt app own the whole process IMHO. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (5)
-
Daniel P. Berrangé -
Michal Privoznik -
Michal Prívozník
-
Nikolay Shirokovskiy -
Roman Kagan