Re: libvirt-devaddr: a new library for device address assignment

On Wed, 4 Mar 2020, 14:51 Daniel P. Berrangé, <berrange(a)redhat.com&gt; wrote: > > We've been doing alot of refactoring of code in recent times, and also > have plans for significant infrastructure changes. We still need to > spend time delivering interesting features to users / applications. > This mail is to introduce an idea for a solution to an specific > area applications have had long term pain with libvirt's current > "mechanism, not policy" approach - device addressing. This is a way > for us to show brand new ideas & approaches for what the libvirt > project can deliver in terms of management APIs. > > To set expectations straight: I have written no code for this yet, > merely identified the gap & conceptual solution. > > > The device addressing problem > ============================= > > One of the key jobs libvirt does when processing a new domain XML > configuration is to assign addresses to all devices that are present. > This involves adding various device controllers (PCI bridges, PCI root > ports, IDE/SCSI buses, USB controllers, etc) if they are not already > present, and then assigning PCI, USB, IDE, SCSI, etc, addresses to each > device so they are associated with controllers. When libvirt spawns a > QEMU guest, it will pass full address information to QEMU. > > Libvirt, as a general rule, aims to avoid defining and implementing > policy around expansion of guest configuration / defaults, however, it > is inescapable in the case of device addressing due to the need to > guarantee a stable hardware ABI to make live migration and save/restore > to disk work. The policy that libvirt has implemented for device > addressing is, as much as possible, the same as the addressing scheme > QEMU would apply itself. > > While libvirt succeeds in its goal of providing a stable hardware API, > the addressing scheme used is not well suited to all deployment > scenarios of QEMU. This is an inevitable result of having a specific > assignment policy implemented in libvirt which has to trade off mutually > incompatible use cases/goals. > > When the libvirt addressing policy is not been sufficient, management > applications are forced to take on address assignment themselves, > which is a massive non-trivial job with many subtle problems to > consider. > > Places where libvirt's addressing is insufficient for PCI include > > * Setting up multiple guest NUMA nodes and associating devices to > specific nodes > * Pre-emptive creation of extra PCIe root ports, to allow for later > device hotplug on PCIe topologies > * Determining whether to place a device on a PCI or PCIe bridge > * Controlling whether a device is placed into a hotpluggable slot > * Controlling whether a PCIe root port supports hotplug or not > * Determining whether to places all devices on distinct slots or > buses, vs grouping them all into functions on the same slot > * Ability to expand the device addressing without being on the > hypervisor host (I don't understand the last bullet point)

> Libvirt wishes to avoid implementing many different address assignment > policies. It also wishes to keep the domain XML as a representation > of the virtual hardware, not add a bunch of properties to it which > merely serve as tunable input parameters for device addressing > algorithms. > > There is thus a dilemma here. Management applications increasingly > need fine grained control over device addressing, while libvirt > doesn't want to expose fine grained policy controls via the XML. > > > The new libvirt-devaddr API > =========================== > > The way out of this is to define a brand new virt management API > which tackles this specific problem in a way that addresses all the > problems mgmt apps have with device addressing and explicitly > provides a variety of policy impls with tunable behaviour. > > By "new API", I actually mean an entirely new library, completely > distinct from libvirt.so, or anything else we've delivered so > far. The closest we've come to delivering something at this kind > of conceptual level, would be the abortive attempt we made with > "libvirt-builder" to deliver a policy-driven API instead of mechanism > based. This proposal is still quite different from that attempt. > > At a high level > > * The new API is "libvirt-devaddr" - short for "libvirt device addressing" > > * As input it will take > > 1. The guest CPU architecture and machine type > 2. A list of global tunables specifying desired behaviour of the > address assignment policy > 3. A minimal list of devices needed in the virtual machine, with > optional addresses and optional per-device tunables to override > the global tunables > > * As output it will emit > > 1. fully expanded list of devices needed in the virtual machine, > with addressing information sufficient to ensure stable hardware ABI > > Initially the API would implement something that behaves the same > way as libvirt's current address assignment API. > > The intended usage would be > > * Mgmt application makes a minimal list of devices they want in > their guest > * List of devices is fed into libvirt-devaddr API > * Mgmt application gets back a full list of devices & addresses > * Mgmt application writes a libvirt XML doc using this full list & > addresses > * Mgmt application creates the guest in libvirt > > IOW, this new "libvirt-devaddr" API is intended to be used prior to > creating the XML that is used by libvirt. The API could also be used > prior to needing to hotplug a new device to an existing guest. > This API is intended to be a deliverable of the libvirt project, but > it would be completely independent of the current libvirt API. Most > especially note that it would NOT use the domain XML in any way. > This gives applications maximum flexibility in how they consume this > functionality, not trying to force a way to build domain XML. This procedure forces Mgmt to learn a new language to describe device placement. Mgmt (or should I just say "we"?) currently expresses the "minimal list of devices" in XML form and pass it to libvirt. Here we are asked to pass it once to libvirt-devaddr, parse its output, and feed it as XML to libvirt.

I believe it would be easier to use the domxml as the base language for the new library, too. libvirt-devaddr would accept it with various hints (expressed as its own extension to the XML?) such as "place all of these devices in the same NUMA node", "keep on root bus" or "separate these two chattering devices to their own bus". The output of libvirt-devaddr would be a domxml with <devices> filled with controllers and addresses, readily available for consumption by libvirt.