Re: [libvirt] dpdk/vpp and cross-version migration for vhost

On Thu, Nov 17, 2016 at 09:47:09AM +0100, Maxime Coquelin wrote: > > > On 11/17/2016 09:29 AM, Yuanhan Liu wrote: > >As usaual, sorry for late response :/ > > > >On Thu, Oct 13, 2016 at 08:50:52PM +0300, Michael S. Tsirkin wrote: > >>Hi! > >>So it looks like we face a problem with cross-version > >>migration when using vhost. It's not new but became more > >>acute with the advent of vhost user. > >> > >>For users to be able to migrate between different versions > >>of the hypervisor the interface exposed to guests > >>by hypervisor must stay unchanged. > >> > >>The problem is that a qemu device is connected > >>to a backend in another process, so the interface > >>exposed to guests depends on the capabilities of that > >>process. > >> > >>Specifically, for vhost user interface based on virtio, this includes > >>the "host features" bitmap that defines the interface, as well as more > >>host values such as the max ring size. Adding new features/changing > >>values to this interface is required to make progress, but on the other > >>hand we need ability to get the old host features to be compatible. > > > >It looks like to the same issue of vhost-user reconnect to me. For example, > > > >- start dpdk 16.07 & qemu 2.5 > >- kill dpdk > >- start dpdk 16.11 > > > >Though DPDK 16.11 has more features comparing to dpdk 16.07 (say, indirect), > >above should work. Because qemu saves the negotiated features before the > >disconnect and stores it back after the reconnection. > > > > commit a463215b087c41d7ca94e51aa347cde523831873 > > Author: Marc-André Lureau <marcandre.lureau(a)redhat.com&gt; > > Date: Mon Jun 6 18:45:05 2016 +0200 > > > > vhost-net: save & restore vhost-user acked features > > > > The initial vhost-user connection sets the features to be negotiated > > with the driver. Renegotiation isn't possible without device reset. > > > > To handle reconnection of vhost-user backend, ensure the same set of > > features are provided, and reuse already acked features. > > > > Signed-off-by: Marc-André Lureau <marcandre.lureau(a)redhat.com&gt; > > > > > >So we could do similar to vhost-user? I mean, save the acked features > >before migration and store it back after it. This should be able to > >keep the compatibility. If user downgrades DPDK version, it also could > >be easily detected, and then exit with an error to user: migration > >failed due to un-compatible vhost features. > > > >Just some rough thoughts. Makes tiny sense? > > My understanding is that the management tool has to know whether > versions are compatible before initiating the migration: Makes sense. How about getting and restoring the acked features through qemu command lines then, say, through the monitor interface? With that, it would be something like: - start vhost-user backend (DPDK, VPP, or whatever) & qemu in the src host - read the acked features (through monitor interface) - start vhost-user backend in the dst host - start qemu in the dst host with the just queried acked features QEMU then is expected to use this feature set for the later vhost-user feature negotitation. Exit if features compatibility is broken. Thoughts? --yliu

> 1. The downtime could be unpredictable if a VM has to move from hosts > to hosts multiple times, which is problematic, especially for NFV. > 2. If migration is not possible, maybe the management tool would > prefer not to interrupt the VM on current host. > > I have little experience with migration though, so I could be mistaken. > > Thanks, > Maxime > > > > > --yliu > >> > >>To solve this problem within qemu, qemu has a versioning system based on > >>a machine type concept which fundamentally is a version string, by > >>specifying that string one can get hardware compatible with a previous > >>qemu version. QEMU also reports the latest version and list of versions > >>supported so libvirt records the version at VM creation and then is > >>careful to use this machine version whenever it migrates a VM. > >> > >>One might wonder how is this solved with a kernel vhost backend. The > >>answer is that it mostly isn't - instead an assumption is made, that > >>qemu versions are deployed together with the kernel - this is generally > >>true for downstreams. Thus whenever qemu gains a new feature, it is > >>already supported by the kernel as well. However, if one attempts > >>migration with a new qemu from a system with a new to old kernel, one > >>would get a failure. > >> > >>In the world where we have multiple userspace backends, with some of > >>these supplied by ISVs, this seems non-realistic. > >> > >>IMO we need to support vhost backend versioning, ideally > >>in a way that will also work for vhost kernel backends. > >> > >>So I'd like to get some input from both backend and management > >>developers on what a good solution would look like. > >> > >>If we want to emulate the qemu solution, this involves adding the > >>concept of interface versions to dpdk. For example, dpdk could supply a > >>file (or utility printing?) with list of versions: latest and versions > >>supported. libvirt could read that and > >>- store latest version at vm creation > >>- pass it around with the vm > >>- pass it to qemu > >> > >>>From here, qemu could pass this over the vhost-user channel, > >>thus making sure it's initialized with the correct > >>compatible interface. > >> > >>As version here is an opaque string for libvirt and qemu, > >>anything can be used - but I suggest either a list > >>of values defining the interface, e.g. > >>any_layout=on,max_ring=256 > >>or a version including the name and vendor of the backend, > >>e.g. "org.dpdk.v4.5.6". > >> > >>Note that typically the list of supported versions can only be > >>extended, not shrunk. Also, if the host/guest interface > >>does not change, don't change the current version as > >>this just creates work for everyone. > >> > >>Thoughts? Would this work well for management? dpdk? vpp? > >> > >>Thanks! > >> > >>-- > >>MST