Re: [libvirt] Matching the type of mediated devices in the migration
On Sun, Jul 29, 2018 at 09:19:41PM +0000, Wang, Zhi A wrote:
BACKGROUND
As the live migration of mdev is going to be supported in VFIO, a scheme of deciding if a
mdev could be migratable between the source machine and the destination machine is needed.
Mostly, this email is going to discuss a possible solution which needs fewer modifications
of libvirt/VFIO.
The configuration of a mdev is located in the domain XML, which guides libvirt how to
find the mdev and generating the command line for QEMU. It basically only includes the
UUID of a mdev. The domain XML of the source machine and destination machine are going to
be compared before the migration really happens. Each configuration item would be compared
and checked by libvirt. If one item of the source machine is different from the item of
destination machine, the migration fails. For mdev, there is no any check/match before the
migration happens yet.
The user could use the node device list of libvirt to list the host devices and see the
capabilities of those devices. The current node device code of libvirt has already been
able to extract the supported mdev types from a host PCI device, plus some basic
information, like max supported mdev instance of a host PCI device.
THE SOLUTION
To strictly check the mdev type and make sure the migration happens between the
compatible mediated devices, three new mandatory elements in the domain XML below the
hostdev element would be introduced:
vendorid: The vendor ID of the mdev, which comes from the host PCI device. A user could
obtain this information from the host PCI device which supports mdev in the node device
list.
productid: The product ID of the mdev, which also comes from the host PCI device. A user
could obtain this information from the same approach above.
mdevtype: The type of the mdev. As the creation of the mdev is managed by the user, the
user knows the type of the mdev and would be responsible for filling out this information.
As you pointed out, we have this information, we therefore shouldn't duplicate
it within the domain XML. AFAIK we can probe that information from the
node-device driver before starting migration, put it into the migration cookie,
send the cookie over to the destination, retrieve the info from the cookie,
perform some checks and decide whether we should continue or abort the
migration. Or is there something I'm missing ? (this can very much be the case
as I'm not very familiar with the migration code)
These three elements are only needed when the device API of a mdev is
"vfio-PCI". Take the example of mdev configuration from
https://libvirt.org/formatdomain.html to illustrate the modification:
<devices>
<hostdev mode='subsystem' type='mdev'
model='vfio-pci'>
<source>
<address uuid='c2177883-f1bb-47f0-914d-32a22e3a8804'/>
<vendorid>0xdead</vendorid> <!-- The VID of the host PCI device
which supports this mdev -->
<productid>0xbeef</productid> <!-- The PID of the host PCI device
which supports this mdev -->
<mdevtype>type</mdevtype> <!-- The vendor-specific mdev type string
-->
</source>
</hostdev>
With the newly introduced elements above, the flow of the creation of a domain XML with
mdev will be like:
1. The user obtains the vendorid/productid from node device list
2. The user fills the vendorid/productid/mdevtype in the domain XML
3. When a migration happens, libvirt check these elements. If one item is different
between two domain XML, then migration fails.
What kind of checks are we talking about? Speaking of vendor/product ids,
simple string comparison doesn't scale, as libvirt would have to compensate for
every future updates to the vendor driver, IOW if <vendor> decides that in
driver version A, only matching product IDs were allowed in migration, but a
fresh new driver version B allows certain product IDs to be cross compatible in
terms of migration, libvirt must have access to this kind of information,
otherwise we're just going to end up being a dumping ground holding a massive
database of all the compatible combinations. The same goes for mdevtype,
ensuring compatibility between types is the vendors responsibility and may be a
subject to change which is out of libvirt's hands, thus if libvirt is the one
ultimately making a qualified decision about migration, we need to be able to
query this kind of data ad-hoc rather than having it as part of libvirt.
Erik