Re: [libvirt] [RFC PATCH] hostdev: add support for "managed='detach'"

> Could this be controlled by a kernel parameter? So you > can just add something like >  >   vfio-pci.devices=0000:02:00.0,0000:03:00.0 >  > to your bootloader configuration and be sure that the > devices you've listed will never be touched by the host. >  > Note that I have no idea how much work implementing > something like the above would be, or whether it would > even be possible :)   vfio-pci is typically a module, device assignment and userspace drivers are very niche as far as the kernel is concerned and the driver is complex enough that building it statically into the kernel has some disadvantages.  So adding module options to vfio-pci doesn't really help.  We sort of tried this with adding the vfio-pci.ids module option so we'd have parity with pci-stub, but it didn't really work because users need to go through enough hoops to get vfio-pci loaded early that they might as well just use the driver_override option. Live and learn, pci-stub.ids is still generally more useful than the equivalent vfio-pci option because pci-stub is (or should be) built statically into the kernel.

Also, a fun property that we get to deal with in PCI space is that bus numbers are under the control of the OS, so if you boot with pci=assign-buses, your device address may change.  That means that specifying a device by address is not as clean of a solution as you might think.

It seems like this quickly devolves into a question of whether the kernel is even the right place to do this, if you have a properly modular kernel then userspace really has the control of how modules are loaded and can intervene to exclude certain devices. The kink is that we need to do that early in boot, which circles right back around to the initramfs scripts, but afaik those are generally managed by each distro with only loose similarities, so we'd need to hope that we contribute something that becomes a defacto standard.

> > The > > problem is more that this is not an uncommon scenario.  In one of my > > previous replies I listed all of the driver issues that I know about. > > In some cases we can't fix them because the driver is proprietary, in > > others we just don't have the bandwidth.  Users have far more devices > > at their disposal to test than we do, so they're likely going to > > continue to run into these issues.   > Summing up the issues you listed in that message, along with > my thoughts: >  >   * stealing audio device from host when running a game in a >     guest, giving it back afterwards >  >     - desktop use case >     - seems to be working fine with managed='yes' >     - if something goes wrong when reattaching to the host, >       the user can probably still go about his business; >       rebooting the host is going to be annoying but not a >       huge deal >     - limited number of in-kernel slots means you can only >       do this a number of time before it stops working; >       again, having to reboot the host once every so-many >       guest boots is probably not a deal breaker >     - managed='detach' is not helpful here, and neither is >       the proposal above >  >   * NIC reserved for guest use >  >     - both server and desktop use case >     - may or may not handle going back and forth between >       the host and the guest >     - managed='detach' would still prevent most issues, >       assuming the host driver detaches cleanly >     - best would be if the host never touched the device >     - a solution like the one outlined above seems perfect >     - for oVirt / OpenStack, the management layer could >       take care of updating the bootloader configuration >     - for pure libvirt, that responsability would fall on >       the user >     - should virt-manager etc. rather default to >       managed='detach' when assigning a NIC to a guest? >  >   * GPU with binary driver >  >     - both server and desktop use case >     - doesn't handle dynamic unbind gracefully >     - best would be if the host never touched the device >     - a solution like the one outlined above seems perfect >     - for oVirt / OpenStack, the management layer could >       take care of updating the bootloader configuration >     - for pure libvirt, that responsability would fall on >       the user >     - managed='detach' can prevent some failures, but not >       those that happen on host driver unbind >  >   * Primary Intel GPU >  >     - desktop use case (any reason to do this for servers?) >     - will probably fail on reattach >     - mananged='detach' would prevent host crashes and the >       like >     - must assume the user has some other way to connect to >       the host, because at some point they're going to be >       unable to use the GPU from the host anyway >  > Have I missed anything?   I think the only thing missing is that libvirt sort of becomes a natural management point for an assigned device because it's specified with some management characteristics.  Users set managed='yes', assuming libvirt will do the right thing

and because the managed='no' path requires them to come up with their own solution.  So maybe we just need to create some framework for users to be able to take that managed='no' path with feeling like they're stepping off a cliff into their own adhoc management scripts.