On Mon, 2019-11-18 at 19:00 +0000, Daniel P. Berrangé wrote:
On Mon, Nov 18, 2019 at 10:06:34AM -0700, Alex Williamson wrote:
Hey folks,
We had some discussions at KVM Forum around mdev live migration and what that might mean for libvirt handling of mdev devices and potential libvirt/mdevctl[1] flows. I believe the current situation is that libvirt knows nothing about an mdev beyond the UUID in the XML. It expects the mdev to exist on the system prior to starting the VM. The intention is for mdevctl to step in here by providing persistence for mdev devices such that these pre-defined mdevs are potentially not just ephemeral, for example, we can tag specific mdevs for automatic startup on each boot.
It seems the next step in this journey is to figure out if libvirt can interact with mdevctl to "manage" a device. I believe we've avoided defining managed='yes' behavior for mdev hostdevs up to this point because creating an mdev device involves policy decisions. For example, which parent device hosts the mdev, are there optimal NUMA considerations, are there performance versus power considerations, what is the nature of the mdev, etc. mdevctl doesn't necessarily want to make placement decisions either, but it does understand how to create and remove an mdev, what it's type is, associate it to a fixed parent, apply attributes, etc. So would it be reasonable that for a manage='yes' mdev hostdev device, libvirt might attempt to use mdevctl to start an mdev by UUID and stop it when the VM is shutdown? This assumes the mdev referenced by the UUID is already defined and known to mdevct. I'd expect semantics much like managed='yes' around vfio- pci binding, ex. start/stop if it doesn't exist, leave it alone if it already exists.
If that much seems reasonable, and someone is willing to invest some development time to support it, what are then the next steps to enable migration?
The first step is to deal with our virNodeDevice APIs.
Currently we have
- Listing devices via ( virConnectListAllNodeDevices ) - Create transient device ( virNodeDeviceCreateXML ) - Delete transient device ( virNodeDeviceDestroy )
The create/delete APIs only deal with NPIV HBAs right now, so we need to extend that to deal with mdevs as first step.
This entails defining an XML format that can represent the information we need about an mdev.
So, there is already an XML format that represents information about an mdev device [1]. Do you mean extending that to add any additional properties needed for mdevctl? or defining something new? [1] https://libvirt.org/drvnodedev.html#MDEV To define and create an mdev, mdevctl needs a UUID, a parent device, and a type. These properties all appear to be supported via the existing XML format. mdevctl also supports assigning arbitrary sysfs attributes to a device. These attributes have an explicit ordering and are written to sysfs in the specified order when a device is started. This might be the only thing that doesn't fit into the current xml format. Jonathon