Re: [libvirt] libvirt mdev migration, mdevctl integration

On Mon, 2019-11-18 at 19:00 +0000, Daniel P. Berrangé wrote: > On Mon, Nov 18, 2019 at 10:06:34AM -0700, Alex Williamson wrote: > > Hey folks, > > > > We had some discussions at KVM Forum around mdev live migration and > > what that might mean for libvirt handling of mdev devices and > > potential libvirt/mdevctl[1] flows. I believe the current > > situation is > > that libvirt knows nothing about an mdev beyond the UUID in the > > XML. > > It expects the mdev to exist on the system prior to starting the > > VM. > > The intention is for mdevctl to step in here by providing > > persistence > > for mdev devices such that these pre-defined mdevs are potentially > > not > > just ephemeral, for example, we can tag specific mdevs for > > automatic > > startup on each boot. > > > > It seems the next step in this journey is to figure out if libvirt > > can > > interact with mdevctl to "manage" a device. I believe we've > > avoided > > defining managed='yes' behavior for mdev hostdevs up to this point > > because creating an mdev device involves policy decisions. For > > example, which parent device hosts the mdev, are there optimal NUMA > > considerations, are there performance versus power considerations, > > what > > is the nature of the mdev, etc. mdevctl doesn't necessarily want > > to > > make placement decisions either, but it does understand how to > > create > > and remove an mdev, what it's type is, associate it to a fixed > > parent, apply attributes, etc. So would it be reasonable that for > > a > > manage='yes' mdev hostdev device, libvirt might attempt to use > > mdevctl > > to start an mdev by UUID and stop it when the VM is shutdown? This > > assumes the mdev referenced by the UUID is already defined and > > known to > > mdevct. I'd expect semantics much like managed='yes' around vfio- > > pci > > binding, ex. start/stop if it doesn't exist, leave it alone if it > > already exists. > > > > If that much seems reasonable, and someone is willing to invest > > some > > development time to support it, what are then the next steps to > > enable > > migration? > > The first step is to deal with our virNodeDevice APIs. > > Currently we have > > - Listing devices via ( virConnectListAllNodeDevices ) > - Create transient device ( virNodeDeviceCreateXML ) > - Delete transient device ( virNodeDeviceDestroy ) > > The create/delete APIs only deal with NPIV HBAs right now, so we need > to extend that to deal with mdevs as first step. > > This entails defining an XML format that can represent the > information > we need about an mdev. So, there is already an XML format that represents information about an mdev device [1]. Do you mean extending that to add any additional properties needed for mdevctl? or defining something new? [1] https://libvirt.org/drvnodedev.html#MDEV

To define and create an mdev, mdevctl needs a UUID, a parent device, and a type. These properties all appear to be supported via the existing XML format. mdevctl also supports assigning arbitrary sysfs attributes to a device. These attributes have an explicit ordering and are written to sysfs in the specified order when a device is started. This might be the only thing that doesn't fit into the current xml format.