[libvirt] RFC: Creating mediated devices with libvirt

Hi all, so there's been an off-list discussion about finally implementing creation of mediated devices with libvirt and it's more than desired to get as many opinions on that as possible, so please do share your ideas. This did come up already as part of some older threads ([1] for example), so this will be a respin of the discussions. Long story short, we decided to put device creation off and focus on the introduction of the framework as such first and build upon that later, i.e. now. [1] https://www.redhat.com/archives/libvir-list/2017-February/msg00177.html ======================================== PART 1: NODEDEV-DRIVER ======================================== API-wise, device creation through the nodedev driver should be pretty straightforward and without any issues, since virNodeDevCreateXML takes an XML and does support flags. Looking at the current device XML: <device> <name>mdev_0cce8709_0640_46ef_bd14_962c7f73cc6f</name> <path>/sys/devices/pci0000:00/.../0cce8709-0640-46ef-bd14-962c7f73cc6f</path> <parent>pci_0000_03_00_0</parent> <driver> <name>vfio_mdev</name> </driver> <capability type='mdev'> <type id='nvidia-11'/> <iommuGroup number='13'/> <uuid>UUID<uuid> <!-- optional enhancement, see below --> </capability> </device> We can ignore <path>,<driver>,<iommugroup> elements, since these are useless during creation. We also cannot use <name> since we don't support arbitrary names and we also can't rely on users providing a name in correct form which we would need to further parse in order to get the UUID. So since the only thing missing to successfully use create an mdev using XML is the UUID (if user doesn't want it to be generated automatically), how about having a <uuid> subelement under <capability> just like PCIs have <domain> and friends, USBs have <bus> & <device>, interfaces have <address> to uniquely identify the device even if the name itself is unique. Removal of a device should work as well, although we might want to consider creating a *Flags version of the API. ============================================================= PART 2: DOMAIN XML & DEVICE AUTO-CREATION, NO POLICY INVOLVED! ============================================================= There were some doubts about auto-creation mentioned in [1], although they weren't specified further. So hopefully, we'll get further in the discussion this time. 1) Convenience For apps like virt-manager, user will want to add a host device transparently, "hey libvirt, I want an mdev assigned to my VM, can you do that". Even for higher management apps, like oVirt, even they might not care about the parent device at all times and considering that they would need to enumerate the parents, pick one, create the device XML and pass it to the nodedev driver, IMHO it would actually be easier and faster to just do it directly through sysfs, bypassing libvirt once again.... 2) Future domain migration Suppose now that the mdev backing physical devices support state dump and reload. Chances are, that the corresponding mdev doesn't even exist or has a different UUID on the destination, so libvirt would do its best to handle this before the domain could be resumed. Following what we already have: <devices> <hostdev mode='subsystem' type='mdev' model='vfio-pci'> <source> <address uuid='c2177883-f1bb-47f0-914d-32a22e3a8804'> </source> </hostdev> </devices> Instead of trying to somehow extend the <address> element using more attributes like 'domain', 'slot', 'function', etc. that would render the whole element ambiguous, I was thinking about creating a <parent> element nested under <source> that would be basically just a nested definition of another host device re-using all the element we already know, i.e. <address> for PCI, and of course others if there happens to be a need for devices other than PCI. So speaking about XML, we'd end up with something like: <devices> <hostdev mode='subsystem' type='mdev' model='vfio-pci'> <source> <parent> <!-- possibly another <source> element - do we really want that? --> <address domain='0x0000' bus='0x00' slot='0x00' function='0x00'> <type id='foo'/> <!-- end of potential <source> element --> </parent> <!-- this one takes precedence if it exists, ignoring the parent --> <address uuid='c2177883-f1bb-47f0-914d-32a22e3a8804'> </source> </hostdev> </devices> So, this was the first idea off the top of my head, so I'd appreciate any suggestions, comments, especially from people who have got the 'legacy' insight into libvirt and can predict potential pitfalls based on experience :). Thanks, Erik