Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

On Wed, Feb 01, 2017 at 12:33:22PM +0100, Maxime Coquelin wrote: > > > On 02/01/2017 10:43 AM, Daniel P. Berrange wrote: >> On Wed, Feb 01, 2017 at 10:14:54AM +0100, Michal Privoznik wrote: >>> On 02/01/2017 09:35 AM, Maxime Coquelin wrote: >> >>>> Solution 3: Libvirt queries OVS for vhost backend version string: *OK* >>>> ====================================================================== >>>> >>>> >>>> The idea is to have a table of supported versions, associated to >>>> key/value pairs. Libvirt could query the list of supported versions >>>> strings for each hosts, and select the first common one among all hosts. >>> >>> How does libvirt know what hosts to ask? Libvirt aims on managing a >>> single host. It has no knowledge of other hosts on the network. That's >>> task for upper layers like RHEV, OpenStack, etc. >>> >>>> >>>> Then, libvirt would ask OVS to probe the vhost-user interfaces in the >>>> selected version (compatibility mode). For example host A runs OVS-2.7, >>>> and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode >>>> (e.g. with indirect descriptors disabled), which should be selected at >>>> vhost-user interface probe time. >>>> >>>> Advantage of doing so is that libvirt does not need any update if new >>>> keys are introduced (i.e. it does not need to know how the new keys have >>>> to be handled), all these checks remain in OVS's vhost-user implementation. >>> >>> And that's where they should stay. Duplicating code between projects >>> will inevitably lead to a divergence. >>> >>>> >>>> Ideally, we would support per vhost-user interface compatibility mode, >>>> which may have an impact also on DPDK API, as the Virtio feature update >>>> API is global, and not per port. >>> >>> In general, I don't think we want any kind of this logic in libvirt. Either: >>> >>> a) fallback logic should be implemented in qemu (e.g. upon migration it >>> should detect that the migrated guest uses certain version and thus set >>> backend to use that version or error out and cancel migration), or >>> >>> b) libvirt would grew another element/attribute to specify version of >>> vhost-user backend in use and do nothing more than pass it to qemu. At >>> the same time, we can provide an API (or extend and existing one, e.g. >>> virsh domcapabilities) to list all available versions on given host. >>> Upper layer, which knows what are the possible hosts suitable for >>> virtualization, can then use this API to ask all the hosts, construct >>> the matrix, select preferred version and put it into libvirt's domain XML. >>> >>> But frankly, I don't like b) that much. Lets put the fact this is OVS >>> aside for a moment. Just pretend this is a generic device in qemu. Would >>> we do the same magic with it? No! Or lets talk about machine types. You >>> spawn -M type$((X+1)) guest and then decide to migrate it to a host with >>> older qemu wich supports just typeX. Well, you get an error. Do we care? >>> Not at all! It's your responsibility (as user/admin) to upgrade the qemu >>> so that it supports new machine type. I think the same applies to OVS. >> >> With machine types, if the latest machine type is X, libvirt allows >> the mgmt app to spawn a guest with mcahine type X-1, so that you can >> later migrate the VM to a host with older QEMU. >> >> With the vhost user device, the VM will always be launched with version >> Y. There's currently no way to request launching the vhost user device >> wtih version Y-1. So even if you set your machine type to X-1 for >> compat with older host, vhost user will be stuck at version Y preventing >> the migration. >> >> One argument would be to say that the vhost user featureset should be >> determined by the machine type version, instead of introducing a new >> version. The complexity is that vhost-user is a pretty dumb device >> from QEMUs POV - most of the interesting logic & the features that >> need to be constrained lie in code outside of QEMU, in whatever >> server is connected to the vhost user socket. >> >> So I can see the value in allowing a simple version string to be >> associated with the vhost-user NIC. >> >> What I'm unclear about is how we would be able to report capabilities >> for a host to enumerate what versions were possible. Libvirt doesn't >> interact with any of the 3rd party vhost user servers, so it is probably >> out of scope - it'd be upto the higher level mgmt app to talk to DPDK >> and figure out what versions it supports. >> >> That does make me wonder though if libvirt & QEMU need to be involved >> at all in any part. > > Indeed, if the higher level management tool decides for the VM's machine > type, it is where it should also be done for the vhost-user backend. I > now understand this does not make much sense to have libvirt being > involved in this, all (querying/selecting/setting compat mode) should be > manageable in the upper layer. > > I'm not familiar with these layers, so your inputs are really helpful. > >> >> When provisioning a new guest, the mgmt app presumably has to talk to >> DPDK to setup the NIC port, so DPDK is ready when QEMU launches and >> connects. Surely as part of that NIC port setup, it could directly >> tell DPDK which version or featureset to permit on the port ? It is >> not obvious why the version string has to be fed in via QEMU. > No it is not, my proposal was that libvirt set the version string in > OVS, QEMU was not involved. > > From these inputs, the idea remains valid, except that libvirt is not > the right place to manage this. Instead, RHEV, Openstack or any other > management tool should handle the compat mode selection. It depends where / how in OVS it needs to be set. The only stuff libvirt does with OVS is to run 'add-port' and 'del-port' commands via the ovs cli tool. We pass through arguments from the port profile stored in the XML config. <interface type='bridge'> <source bridge='ovsbr'/> <virtualport type='openvswitch'> <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> </virtualport> </interface> eg those things in <parameters/> get passed as cli args to the 'add-port' command. Soo if add-port needs this new version string, then we'd need to add the version to the openvswitch virtualport XML. If the version is provided to OVS in a different command, then it would probably be outside scope of libvirt.