Re: [libvirt] [RFC] Vhost-user backends cross-version migration support

On Fri, Feb 03, 2017 at 10:41:12AM +0100, Maxime Coquelin wrote: > > > On 02/03/2017 10:27 AM, Daniel P. Berrange wrote: >> On Thu, Feb 02, 2017 at 08:27:18PM +0200, Michael S. Tsirkin wrote: >>> On Thu, Feb 02, 2017 at 06:21:55PM +0000, Daniel P. Berrange wrote: >>>> On Thu, Feb 02, 2017 at 07:31:49PM +0200, Michael S. Tsirkin wrote: >>>>> On Thu, Feb 02, 2017 at 05:29:08PM +0000, Daniel P. Berrange wrote: >>>>>> On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote: >>>>>>> On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote: >>>>>>>> On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote: >>>>>>>>> On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote: >>>>>>>>>> On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: >>>>>>>>>>>> >>>>>>>>>>>> It depends where / how in OVS it needs to be set. The only stuff libvirt >>>>>>>>>>>> does with OVS is to run 'add-port' and 'del-port' commands via the ovs >>>>>>>>>>>> cli tool. We pass through arguments from the port profile stored in the >>>>>>>>>>>> XML config. >>>>>>>>>>>> >>>>>>>>>>>> <interface type='bridge'> >>>>>>>>>>>> <source bridge='ovsbr'/> >>>>>>>>>>>> <virtualport type='openvswitch'> >>>>>>>>>>>> <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> >>>>>>>>>>>> </virtualport> >>>>>>>>>>>> </interface> >>>>>>>>>>>> >>>>>>>>>>>> eg those things in <parameters/> get passed as cli args to the 'add-port' >>>>>>>>>>>> command. Soo if add-port needs this new version string, then we'd need >>>>>>>>>>>> to add the version to the openvswitch virtualport XML. >>>>>>>>>>>> >>>>>>>>>>>> If the version is provided to OVS in a different command, then it would >>>>>>>>>>>> probably be outside scope of libvirt. >>>>>>>>>>> >>>>>>>>>>> I think it would make sense to be a parameter of the add-port command. >>>>>>>>>>> But it would be for vhost-user related add-port command, I didn't find >>>>>>>>>>> where/if this is managed in libvirt XML. >>>>>>>>>> >>>>>>>>>> For vhost-user, libvirt does not have any interaction with OVS at >>>>>>>>>> all. If the thing that's using the vhost-user UNIX socket, in turn >>>>>>>>>> connects to OVS, that's outside scope of libvirt. IOW, for vhost-user >>>>>>>>>> OVS it seems like that job is for Nova / os-vif to solve. >>>>>>>>> >>>>>>>>> I don't think they currently understand the issues involved in >>>>>>>>> cross-version migration though. This is a complex subject and easy to >>>>>>>>> get wrong. It would be significantly better to figure it out at libvirt >>>>>>>>> level since it already deals with cross-version migration issues. >>>>>>>> >>>>>>>> Libvirt considers vhost-user to be a blackbox though - it just exposes >>>>>>>> a UNIX socket, and whatever is on the other end is completely opaque. >>>>>>>> The fact that the other end might plumb the data stream into openvswitch >>>>>>>> is not something libvirt should know, as we don't want to end up having >>>>>>>> to add custom code to libvirt for every different vhost-user server >>>>>>>> impl. >>>>>>>> >>>>>>>> IOW, if the version str can be passed to QEMU, and then onto vhost-user >>>>>>>> backend in QEMU, then libvirt can be involved. If the version str has >>>>>>>> to be given to openvswitch that's not for libvirt to deall with. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Daniel >>>>>>> >>>>>>> It's not just about passing it to QEMU. The following is needed: >>>>>>> - need to query version when creating VM/device >>>>>>> - need to query supported versions when migrating VM >>>>>> >>>>>> Those are both things that nova can do, since it knows what the vhost-user >>>>>> device in question is connected to, and so can query the versions, and check >>>>>> versions before triggering migration with libvirt >>>>> >>>>> It can, but then it will need to query libvirt or source for the version >>>>> string since it's in the XML. >>>> >>>> No, it wouldn't be in the XML at all. Nova on the source queries what >>>> vhostuser version it has and what the target host has, and can prevent >>>> the migration if they're incompatible. >>> >>> This is not sufficient. Exactly the same as qemu machine type, >>> this must be preserved from time of install and moved >>> wherever VM goes. >> >> Nova has the ability todo that. >> >>>> I dont think libvirt has to be >>>> involved at all for this, as all the info can be obtained by nova/os-vif >>>> from the vhostuser impl it has configured. >>> >>> Given we are already confused at libvirt level, I find it >>> highly unlikely upper levels will do the right thing. >> >> The only confusion is about what must consume the data. I mistakenly >> thought it was being proposed that the qemu vhostuser backend was >> being given a version string. Now it seems the version info must be >> set against the 3rd party component that vhostuser connects to, and >> that is outside scope of libvirt. That is entirely managed by Nova >> today, so Nova is the right thing to manage this versioning info. > > This is my understanding, with as example using Nova: > 1. Before starting the VM, Nova queries source host's OVS and all > possible destination hosts' OVS to retrieve supported versions > 2. Nova chooses the first common version supported by all hosts > 3. Nova create the dpdkvhostuser interfaces in OVS with passing the > selected version as parameter > > That should be enough, right? Or maybe I miss something? It isn't realistic to check all possible destination hosts when first starting QEMU, not least becasue they may not yet exist. New hardware may be deployed *after* QEMU is first started. When starting the VM, it should just default to using whatever the latest version is on the host in question. There can be an optional host configuration parameter that admins can set to force use of an older version. This is how Nova deals with machine type - it defaults to latest, but you can set hw_machine_type in nova.conf to force an older version if you need migration compatibility between hosts with varying versions.

At time of migration, Nova need to request the version of the target host. If this version is not compatible with what the VM is currently using, then Nova should *not* start the migration, it should raise an error such that the scheduler retries and picks a different target host