[libvirt] [RFC] Vhost-user backends cross-version migration support
Hi, Few months ago, Michael reported a problem about migrating VMs relying on vhost-user between hosts supporting different backend versions: - Message-Id: <20161011173526-mutt-send-email-mst@kernel.org> - https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03026.html The goal of this thread is to draft a proposal based on the outcomes of discussions with contributors of the different parties (DPDK/OVS /libvirt/...). Problem statement: ================== When migrating a VM from one host to another, the interfaces exposed by QEMU must stay unchanged in order to guarantee a successful migration. In the case of vhost-user interface, parameters like supported Virtio feature set, max number of queues, max vring sizes,... must remain compatible. Indeed, the frontend not being re-initialized, no renegotiation happens at migration time. For example, we have a VM that runs on host A, which has its vhost-user backend advertising VIRTIO_F_RING_INDIRECT_DESC feature. Since the Guest also support this feature, it is successfully negotiated, and guest transmit packets using indirect descriptor tables, that the backend knows to handle. At some point, the VM is being migrated to host B, which runs an older version of the backend not supporting this VIRTIO_F_RING_INDIRECT_DESC feature. The migration would break, because the Guest still have the VIRTIO_F_RING_INDIRECT_DESC bit sets, and the virtqueue contains some decriptors pointing to indirect tables, that backend B doesn't know to handle. This is just an example about Virtio features compatibility, but other backend implementation details could cause other failures. What we need is to be able to query the destination host's backend to ensure migration is possible. Also, we would need to query this statically, even before the VM is started, to be sure it could be migrated elsewhere for any reason. Solution 1: Libvirt queries DPDK vhost lib: *KO* ================================================ Initial idea was to have the management tool (libvirt) to query DPDK vhost lib and get key/value pairs and check whether migration is possible. This solution doesn't work for several reasons: 1. Vhost lib API provide a way for the application to disable features at runtime (understand, not at build time). So coming back to previous example, DPDK v16.11 supports indirect descriptor features, but it could be disabled by OVS. We had a look at whether this API was mandatory, and it turns out to be, as TSO feature is supported on DPDK but not in OVS. So we cannot rely on DPDK only. 2. Some parameter may be not only DPDK specific, such as the maximum number of queues for example. Solution 2: Libvirt queries OVS for vhost backend key/value pairs: *KO* ======================================================================= Second idea was for OVS to expose its vhost backend implementation parameters as key/value pairs, for example in the DB or by a dedicated tool. For example, you could have this kind of information: - virtio-features: 0x12045694 - max-rx-queues: 1024 - max-rx-ring-size: 512 Doing this, Libvirt has the information to take decision whether migration is possible or not. The problem is that Libvirt doesn't know (and want) to interpret these values (should it be equal/lower/greater/...?), and each time a new key is introduced in OVS, Libvirt will have to be updated to handle it, adding an unwanted synchronization constraint between the projects. Solution 3: Libvirt queries OVS for vhost backend version string: *OK* ====================================================================== The idea is to have a table of supported versions, associated to key/value pairs. Libvirt could query the list of supported versions strings for each hosts, and select the first common one among all hosts. Then, libvirt would ask OVS to probe the vhost-user interfaces in the selected version (compatibility mode). For example host A runs OVS-2.7, and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode (e.g. with indirect descriptors disabled), which should be selected at vhost-user interface probe time. Advantage of doing so is that libvirt does not need any update if new keys are introduced (i.e. it does not need to know how the new keys have to be handled), all these checks remain in OVS's vhost-user implementation. Ideally, we would support per vhost-user interface compatibility mode, which may have an impact also on DPDK API, as the Virtio feature update API is global, and not per port. - Implementation: ----------------- Goal here is just to illustrate this proposal, I'm sure you will have good suggestion to improve it. In OVS vhost-user library, we would introduce a new structure, for example (neither compiled nor tested): struct vhostuser_compat { char *version; uint64_t virtio_features; uint32_t max_rx_queue_sz; uint32_t max_nr_queues; }; *version* field is the compatibility version string. It could be something like: "upstream.ovs-dpdk.v2.6" In case for example Fedora adds some more patches to its package that would break migration to upstream version, it could have a dedicated compatibility string: "fc26.ovs-dpdk.v2.6". In case OVS-v2.7 does not break compatibility with previous OVS-v2.6 version, then no need to create a new compatibility entry, just keep v2.6 one. *virtio_features* field is the Virtio features set for a given compatibility version. When an OVS tag is to be created, it would be associated to a DPDK version. The Virtio features for these version would be stored in this field. It would allow to upgrade the DPDK package for example from v16.07 to v16.11 without breaking migration. In case the distribution wants to benefit from latests Virtio features, it would have to create a new entry to ensure migration won't be broken. *max_rx_queue_sz* *max_nr_queues* fields are just here for example, don't think this is needed today. I just want to illustrate that we have to anticipate other parameters than the Virtio feature set, even if not necessary at the moment. We create a table with different compatibility versions in OVS vhost-user lib: static struct vhostuser_compat vu_compat[] = { { .version = "upstream.ovs-dpdk.v2.7", .virtio_features = 0x12045694, .max_rx_queue_sz = 512, }, { .version = "upstream.ovs-dpdk.v2.6", .virtio_features = 0x10045694, .max_rx_queue_sz = 1024, }, } At some time during installation, or system init, the table would be parsed, and compatibility version strings would be stored into the OVS database, or a new tool would be created to list these strings. Before launching the VM, libvirt will query the version strings for each hosts using for example the JSON RPC API of OVS (maybe not the best solution, looking forward for your comments on this). Libvirt would then select the first common supported version, and insert this string into the vhost-user interfaces parameters in the OVS DBs of each host. When the vhost-user connection is initiated, OVS would know in which compatibility mode to init the interface, for example by restricting the support Virtio features of the interface. Do you think this is reasonable? Or maybe you have alternative ideas that would be best fit to ensure successful migration? Cheers, Maxime
On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
Hi,
Few months ago, Michael reported a problem about migrating VMs relying on vhost-user between hosts supporting different backend versions: - Message-Id: <20161011173526-mutt-send-email-mst@kernel.org> - https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03026.html
The goal of this thread is to draft a proposal based on the outcomes of discussions with contributors of the different parties (DPDK/OVS /libvirt/...).
Problem statement: ==================
When migrating a VM from one host to another, the interfaces exposed by QEMU must stay unchanged in order to guarantee a successful migration. In the case of vhost-user interface, parameters like supported Virtio feature set, max number of queues, max vring sizes,... must remain compatible. Indeed, the frontend not being re-initialized, no renegotiation happens at migration time.
For example, we have a VM that runs on host A, which has its vhost-user backend advertising VIRTIO_F_RING_INDIRECT_DESC feature. Since the Guest also support this feature, it is successfully negotiated, and guest transmit packets using indirect descriptor tables, that the backend knows to handle. At some point, the VM is being migrated to host B, which runs an older version of the backend not supporting this VIRTIO_F_RING_INDIRECT_DESC feature. The migration would break, because the Guest still have the VIRTIO_F_RING_INDIRECT_DESC bit sets, and the virtqueue contains some decriptors pointing to indirect tables, that backend B doesn't know to handle. This is just an example about Virtio features compatibility, but other backend implementation details could cause other failures.
Exactly. Libvirt can't possibly know which virtio features has guest negotiated to use. Therefore I don't think this falls into libvirt scope.
What we need is to be able to query the destination host's backend to ensure migration is possible. Also, we would need to query this statically, even before the VM is started, to be sure it could be migrated elsewhere for any reason.
Again, if you have more than two hosts, say A-Z, I don't see how libvirt could know what hosts to asks (where you will migrate your guest), and what combination of virtio features is okay and which is a deal breaker.
Solution 1: Libvirt queries DPDK vhost lib: *KO* ================================================
Initial idea was to have the management tool (libvirt) to query DPDK vhost lib and get key/value pairs and check whether migration is possible. This solution doesn't work for several reasons: 1. Vhost lib API provide a way for the application to disable features at runtime (understand, not at build time). So coming back to previous example, DPDK v16.11 supports indirect descriptor features, but it could be disabled by OVS. We had a look at whether this API was mandatory, and it turns out to be, as TSO feature is supported on DPDK but not in OVS. So we cannot rely on DPDK only. 2. Some parameter may be not only DPDK specific, such as the maximum number of queues for example.
Solution 2: Libvirt queries OVS for vhost backend key/value pairs: *KO* =======================================================================
Second idea was for OVS to expose its vhost backend implementation parameters as key/value pairs, for example in the DB or by a dedicated tool. For example, you could have this kind of information: - virtio-features: 0x12045694 - max-rx-queues: 1024 - max-rx-ring-size: 512 Doing this, Libvirt has the information to take decision whether migration is possible or not. The problem is that Libvirt doesn't know (and want) to interpret these values (should it be equal/lower/greater/...?), and each time a new key is introduced in OVS, Libvirt will have to be updated to handle it, adding an unwanted synchronization constraint between the projects.
Solution 3: Libvirt queries OVS for vhost backend version string: *OK* ======================================================================
The idea is to have a table of supported versions, associated to key/value pairs. Libvirt could query the list of supported versions strings for each hosts, and select the first common one among all hosts.
How does libvirt know what hosts to ask? Libvirt aims on managing a single host. It has no knowledge of other hosts on the network. That's task for upper layers like RHEV, OpenStack, etc.
Then, libvirt would ask OVS to probe the vhost-user interfaces in the selected version (compatibility mode). For example host A runs OVS-2.7, and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode (e.g. with indirect descriptors disabled), which should be selected at vhost-user interface probe time.
Advantage of doing so is that libvirt does not need any update if new keys are introduced (i.e. it does not need to know how the new keys have to be handled), all these checks remain in OVS's vhost-user implementation.
And that's where they should stay. Duplicating code between projects will inevitably lead to a divergence.
Ideally, we would support per vhost-user interface compatibility mode, which may have an impact also on DPDK API, as the Virtio feature update API is global, and not per port.
In general, I don't think we want any kind of this logic in libvirt. Either: a) fallback logic should be implemented in qemu (e.g. upon migration it should detect that the migrated guest uses certain version and thus set backend to use that version or error out and cancel migration), or b) libvirt would grew another element/attribute to specify version of vhost-user backend in use and do nothing more than pass it to qemu. At the same time, we can provide an API (or extend and existing one, e.g. virsh domcapabilities) to list all available versions on given host. Upper layer, which knows what are the possible hosts suitable for virtualization, can then use this API to ask all the hosts, construct the matrix, select preferred version and put it into libvirt's domain XML. But frankly, I don't like b) that much. Lets put the fact this is OVS aside for a moment. Just pretend this is a generic device in qemu. Would we do the same magic with it? No! Or lets talk about machine types. You spawn -M type$((X+1)) guest and then decide to migrate it to a host with older qemu wich supports just typeX. Well, you get an error. Do we care? Not at all! It's your responsibility (as user/admin) to upgrade the qemu so that it supports new machine type. I think the same applies to OVS. Sorry. Michal
On Wed, Feb 01, 2017 at 10:14:54AM +0100, Michal Privoznik wrote:
On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
Solution 3: Libvirt queries OVS for vhost backend version string: *OK* ======================================================================
The idea is to have a table of supported versions, associated to key/value pairs. Libvirt could query the list of supported versions strings for each hosts, and select the first common one among all hosts.
How does libvirt know what hosts to ask? Libvirt aims on managing a single host. It has no knowledge of other hosts on the network. That's task for upper layers like RHEV, OpenStack, etc.
Then, libvirt would ask OVS to probe the vhost-user interfaces in the selected version (compatibility mode). For example host A runs OVS-2.7, and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode (e.g. with indirect descriptors disabled), which should be selected at vhost-user interface probe time.
Advantage of doing so is that libvirt does not need any update if new keys are introduced (i.e. it does not need to know how the new keys have to be handled), all these checks remain in OVS's vhost-user implementation.
And that's where they should stay. Duplicating code between projects will inevitably lead to a divergence.
Ideally, we would support per vhost-user interface compatibility mode, which may have an impact also on DPDK API, as the Virtio feature update API is global, and not per port.
In general, I don't think we want any kind of this logic in libvirt. Either:
a) fallback logic should be implemented in qemu (e.g. upon migration it should detect that the migrated guest uses certain version and thus set backend to use that version or error out and cancel migration), or
b) libvirt would grew another element/attribute to specify version of vhost-user backend in use and do nothing more than pass it to qemu. At the same time, we can provide an API (or extend and existing one, e.g. virsh domcapabilities) to list all available versions on given host. Upper layer, which knows what are the possible hosts suitable for virtualization, can then use this API to ask all the hosts, construct the matrix, select preferred version and put it into libvirt's domain XML.
But frankly, I don't like b) that much. Lets put the fact this is OVS aside for a moment. Just pretend this is a generic device in qemu. Would we do the same magic with it? No! Or lets talk about machine types. You spawn -M type$((X+1)) guest and then decide to migrate it to a host with older qemu wich supports just typeX. Well, you get an error. Do we care? Not at all! It's your responsibility (as user/admin) to upgrade the qemu so that it supports new machine type. I think the same applies to OVS.
With machine types, if the latest machine type is X, libvirt allows the mgmt app to spawn a guest with mcahine type X-1, so that you can later migrate the VM to a host with older QEMU. With the vhost user device, the VM will always be launched with version Y. There's currently no way to request launching the vhost user device wtih version Y-1. So even if you set your machine type to X-1 for compat with older host, vhost user will be stuck at version Y preventing the migration. One argument would be to say that the vhost user featureset should be determined by the machine type version, instead of introducing a new version. The complexity is that vhost-user is a pretty dumb device from QEMUs POV - most of the interesting logic & the features that need to be constrained lie in code outside of QEMU, in whatever server is connected to the vhost user socket. So I can see the value in allowing a simple version string to be associated with the vhost-user NIC. What I'm unclear about is how we would be able to report capabilities for a host to enumerate what versions were possible. Libvirt doesn't interact with any of the 3rd party vhost user servers, so it is probably out of scope - it'd be upto the higher level mgmt app to talk to DPDK and figure out what versions it supports. That does make me wonder though if libvirt & QEMU need to be involved at all in any part. When provisioning a new guest, the mgmt app presumably has to talk to DPDK to setup the NIC port, so DPDK is ready when QEMU launches and connects. Surely as part of that NIC port setup, it could directly tell DPDK which version or featureset to permit on the port ? It is not obvious why the version string has to be fed in via QEMU. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
On 02/01/2017 10:43 AM, Daniel P. Berrange wrote:
On Wed, Feb 01, 2017 at 10:14:54AM +0100, Michal Privoznik wrote:
On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
Solution 3: Libvirt queries OVS for vhost backend version string: *OK* ======================================================================
The idea is to have a table of supported versions, associated to key/value pairs. Libvirt could query the list of supported versions strings for each hosts, and select the first common one among all hosts.
How does libvirt know what hosts to ask? Libvirt aims on managing a single host. It has no knowledge of other hosts on the network. That's task for upper layers like RHEV, OpenStack, etc.
Then, libvirt would ask OVS to probe the vhost-user interfaces in the selected version (compatibility mode). For example host A runs OVS-2.7, and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode (e.g. with indirect descriptors disabled), which should be selected at vhost-user interface probe time.
Advantage of doing so is that libvirt does not need any update if new keys are introduced (i.e. it does not need to know how the new keys have to be handled), all these checks remain in OVS's vhost-user implementation.
And that's where they should stay. Duplicating code between projects will inevitably lead to a divergence.
Ideally, we would support per vhost-user interface compatibility mode, which may have an impact also on DPDK API, as the Virtio feature update API is global, and not per port.
In general, I don't think we want any kind of this logic in libvirt. Either:
a) fallback logic should be implemented in qemu (e.g. upon migration it should detect that the migrated guest uses certain version and thus set backend to use that version or error out and cancel migration), or
b) libvirt would grew another element/attribute to specify version of vhost-user backend in use and do nothing more than pass it to qemu. At the same time, we can provide an API (or extend and existing one, e.g. virsh domcapabilities) to list all available versions on given host. Upper layer, which knows what are the possible hosts suitable for virtualization, can then use this API to ask all the hosts, construct the matrix, select preferred version and put it into libvirt's domain XML.
But frankly, I don't like b) that much. Lets put the fact this is OVS aside for a moment. Just pretend this is a generic device in qemu. Would we do the same magic with it? No! Or lets talk about machine types. You spawn -M type$((X+1)) guest and then decide to migrate it to a host with older qemu wich supports just typeX. Well, you get an error. Do we care? Not at all! It's your responsibility (as user/admin) to upgrade the qemu so that it supports new machine type. I think the same applies to OVS.
With machine types, if the latest machine type is X, libvirt allows the mgmt app to spawn a guest with mcahine type X-1, so that you can later migrate the VM to a host with older QEMU.
With the vhost user device, the VM will always be launched with version Y. There's currently no way to request launching the vhost user device wtih version Y-1. So even if you set your machine type to X-1 for compat with older host, vhost user will be stuck at version Y preventing the migration.
One argument would be to say that the vhost user featureset should be determined by the machine type version, instead of introducing a new version. The complexity is that vhost-user is a pretty dumb device from QEMUs POV - most of the interesting logic & the features that need to be constrained lie in code outside of QEMU, in whatever server is connected to the vhost user socket.
So I can see the value in allowing a simple version string to be associated with the vhost-user NIC.
What I'm unclear about is how we would be able to report capabilities for a host to enumerate what versions were possible. Libvirt doesn't interact with any of the 3rd party vhost user servers, so it is probably out of scope - it'd be upto the higher level mgmt app to talk to DPDK and figure out what versions it supports.
That does make me wonder though if libvirt & QEMU need to be involved at all in any part.
Indeed, if the higher level management tool decides for the VM's machine type, it is where it should also be done for the vhost-user backend. I now understand this does not make much sense to have libvirt being involved in this, all (querying/selecting/setting compat mode) should be manageable in the upper layer. I'm not familiar with these layers, so your inputs are really helpful.
When provisioning a new guest, the mgmt app presumably has to talk to DPDK to setup the NIC port, so DPDK is ready when QEMU launches and connects. Surely as part of that NIC port setup, it could directly tell DPDK which version or featureset to permit on the port ? It is not obvious why the version string has to be fed in via QEMU.
No it is not, my proposal was that libvirt set the version string in OVS, QEMU was not involved. From these inputs, the idea remains valid, except that libvirt is not the right place to manage this. Instead, RHEV, Openstack or any other management tool should handle the compat mode selection. Do you agree? Thanks, Maxime
Regards, Daniel
On Wed, Feb 01, 2017 at 12:33:22PM +0100, Maxime Coquelin wrote:
On 02/01/2017 10:43 AM, Daniel P. Berrange wrote:
On Wed, Feb 01, 2017 at 10:14:54AM +0100, Michal Privoznik wrote:
On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
Solution 3: Libvirt queries OVS for vhost backend version string: *OK* ======================================================================
The idea is to have a table of supported versions, associated to key/value pairs. Libvirt could query the list of supported versions strings for each hosts, and select the first common one among all hosts.
How does libvirt know what hosts to ask? Libvirt aims on managing a single host. It has no knowledge of other hosts on the network. That's task for upper layers like RHEV, OpenStack, etc.
Then, libvirt would ask OVS to probe the vhost-user interfaces in the selected version (compatibility mode). For example host A runs OVS-2.7, and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode (e.g. with indirect descriptors disabled), which should be selected at vhost-user interface probe time.
Advantage of doing so is that libvirt does not need any update if new keys are introduced (i.e. it does not need to know how the new keys have to be handled), all these checks remain in OVS's vhost-user implementation.
And that's where they should stay. Duplicating code between projects will inevitably lead to a divergence.
Ideally, we would support per vhost-user interface compatibility mode, which may have an impact also on DPDK API, as the Virtio feature update API is global, and not per port.
In general, I don't think we want any kind of this logic in libvirt. Either:
a) fallback logic should be implemented in qemu (e.g. upon migration it should detect that the migrated guest uses certain version and thus set backend to use that version or error out and cancel migration), or
b) libvirt would grew another element/attribute to specify version of vhost-user backend in use and do nothing more than pass it to qemu. At the same time, we can provide an API (or extend and existing one, e.g. virsh domcapabilities) to list all available versions on given host. Upper layer, which knows what are the possible hosts suitable for virtualization, can then use this API to ask all the hosts, construct the matrix, select preferred version and put it into libvirt's domain XML.
But frankly, I don't like b) that much. Lets put the fact this is OVS aside for a moment. Just pretend this is a generic device in qemu. Would we do the same magic with it? No! Or lets talk about machine types. You spawn -M type$((X+1)) guest and then decide to migrate it to a host with older qemu wich supports just typeX. Well, you get an error. Do we care? Not at all! It's your responsibility (as user/admin) to upgrade the qemu so that it supports new machine type. I think the same applies to OVS.
With machine types, if the latest machine type is X, libvirt allows the mgmt app to spawn a guest with mcahine type X-1, so that you can later migrate the VM to a host with older QEMU.
With the vhost user device, the VM will always be launched with version Y. There's currently no way to request launching the vhost user device wtih version Y-1. So even if you set your machine type to X-1 for compat with older host, vhost user will be stuck at version Y preventing the migration.
One argument would be to say that the vhost user featureset should be determined by the machine type version, instead of introducing a new version. The complexity is that vhost-user is a pretty dumb device from QEMUs POV - most of the interesting logic & the features that need to be constrained lie in code outside of QEMU, in whatever server is connected to the vhost user socket.
So I can see the value in allowing a simple version string to be associated with the vhost-user NIC.
What I'm unclear about is how we would be able to report capabilities for a host to enumerate what versions were possible. Libvirt doesn't interact with any of the 3rd party vhost user servers, so it is probably out of scope - it'd be upto the higher level mgmt app to talk to DPDK and figure out what versions it supports.
That does make me wonder though if libvirt & QEMU need to be involved at all in any part.
Indeed, if the higher level management tool decides for the VM's machine type, it is where it should also be done for the vhost-user backend. I now understand this does not make much sense to have libvirt being involved in this, all (querying/selecting/setting compat mode) should be manageable in the upper layer.
I'm not familiar with these layers, so your inputs are really helpful.
When provisioning a new guest, the mgmt app presumably has to talk to DPDK to setup the NIC port, so DPDK is ready when QEMU launches and connects. Surely as part of that NIC port setup, it could directly tell DPDK which version or featureset to permit on the port ? It is not obvious why the version string has to be fed in via QEMU.
No it is not, my proposal was that libvirt set the version string in OVS, QEMU was not involved.
From these inputs, the idea remains valid, except that libvirt is not the right place to manage this. Instead, RHEV, Openstack or any other management tool should handle the compat mode selection.
It depends where / how in OVS it needs to be set. The only stuff libvirt does with OVS is to run 'add-port' and 'del-port' commands via the ovs cli tool. We pass through arguments from the port profile stored in the XML config. <interface type='bridge'> <source bridge='ovsbr'/> <virtualport type='openvswitch'> <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> </virtualport> </interface> eg those things in <parameters/> get passed as cli args to the 'add-port' command. Soo if add-port needs this new version string, then we'd need to add the version to the openvswitch virtualport XML. If the version is provided to OVS in a different command, then it would probably be outside scope of libvirt. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
On Wed, Feb 01, 2017 at 11:41:19AM +0000, Daniel P. Berrange wrote:
On Wed, Feb 01, 2017 at 12:33:22PM +0100, Maxime Coquelin wrote:
On 02/01/2017 10:43 AM, Daniel P. Berrange wrote:
On Wed, Feb 01, 2017 at 10:14:54AM +0100, Michal Privoznik wrote:
On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
Solution 3: Libvirt queries OVS for vhost backend version string: *OK* ======================================================================
The idea is to have a table of supported versions, associated to key/value pairs. Libvirt could query the list of supported versions strings for each hosts, and select the first common one among all hosts.
How does libvirt know what hosts to ask? Libvirt aims on managing a single host. It has no knowledge of other hosts on the network. That's task for upper layers like RHEV, OpenStack, etc.
Then, libvirt would ask OVS to probe the vhost-user interfaces in the selected version (compatibility mode). For example host A runs OVS-2.7, and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode (e.g. with indirect descriptors disabled), which should be selected at vhost-user interface probe time.
Advantage of doing so is that libvirt does not need any update if new keys are introduced (i.e. it does not need to know how the new keys have to be handled), all these checks remain in OVS's vhost-user implementation.
And that's where they should stay. Duplicating code between projects will inevitably lead to a divergence.
Ideally, we would support per vhost-user interface compatibility mode, which may have an impact also on DPDK API, as the Virtio feature update API is global, and not per port.
In general, I don't think we want any kind of this logic in libvirt. Either:
a) fallback logic should be implemented in qemu (e.g. upon migration it should detect that the migrated guest uses certain version and thus set backend to use that version or error out and cancel migration), or
b) libvirt would grew another element/attribute to specify version of vhost-user backend in use and do nothing more than pass it to qemu. At the same time, we can provide an API (or extend and existing one, e.g. virsh domcapabilities) to list all available versions on given host. Upper layer, which knows what are the possible hosts suitable for virtualization, can then use this API to ask all the hosts, construct the matrix, select preferred version and put it into libvirt's domain XML.
But frankly, I don't like b) that much. Lets put the fact this is OVS aside for a moment. Just pretend this is a generic device in qemu. Would we do the same magic with it? No! Or lets talk about machine types. You spawn -M type$((X+1)) guest and then decide to migrate it to a host with older qemu wich supports just typeX. Well, you get an error. Do we care? Not at all! It's your responsibility (as user/admin) to upgrade the qemu so that it supports new machine type. I think the same applies to OVS.
With machine types, if the latest machine type is X, libvirt allows the mgmt app to spawn a guest with mcahine type X-1, so that you can later migrate the VM to a host with older QEMU.
With the vhost user device, the VM will always be launched with version Y. There's currently no way to request launching the vhost user device wtih version Y-1. So even if you set your machine type to X-1 for compat with older host, vhost user will be stuck at version Y preventing the migration.
One argument would be to say that the vhost user featureset should be determined by the machine type version, instead of introducing a new version. The complexity is that vhost-user is a pretty dumb device from QEMUs POV - most of the interesting logic & the features that need to be constrained lie in code outside of QEMU, in whatever server is connected to the vhost user socket.
So I can see the value in allowing a simple version string to be associated with the vhost-user NIC.
What I'm unclear about is how we would be able to report capabilities for a host to enumerate what versions were possible. Libvirt doesn't interact with any of the 3rd party vhost user servers, so it is probably out of scope - it'd be upto the higher level mgmt app to talk to DPDK and figure out what versions it supports.
That does make me wonder though if libvirt & QEMU need to be involved at all in any part.
Indeed, if the higher level management tool decides for the VM's machine type, it is where it should also be done for the vhost-user backend. I now understand this does not make much sense to have libvirt being involved in this, all (querying/selecting/setting compat mode) should be manageable in the upper layer.
I'm not familiar with these layers, so your inputs are really helpful.
When provisioning a new guest, the mgmt app presumably has to talk to DPDK to setup the NIC port, so DPDK is ready when QEMU launches and connects. Surely as part of that NIC port setup, it could directly tell DPDK which version or featureset to permit on the port ? It is not obvious why the version string has to be fed in via QEMU.
No it is not, my proposal was that libvirt set the version string in OVS, QEMU was not involved.
From these inputs, the idea remains valid, except that libvirt is not the right place to manage this. Instead, RHEV, Openstack or any other management tool should handle the compat mode selection.
It depends where / how in OVS it needs to be set. The only stuff libvirt does with OVS is to run 'add-port' and 'del-port' commands via the ovs cli tool. We pass through arguments from the port profile stored in the XML config.
<interface type='bridge'> <source bridge='ovsbr'/> <virtualport type='openvswitch'> <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> </virtualport> </interface>
eg those things in <parameters/> get passed as cli args to the 'add-port' command. Soo if add-port needs this new version string, then we'd need to add the version to the openvswitch virtualport XML.
If the version is provided to OVS in a different command, then it would probably be outside scope of libvirt.
Regards, Daniel
OK but how does it end up there in the 1st place? Something like we do for qemu, where there's a query for "pc" alias, and that is stored, seems called for. Further, it seems ugly to assume add will fail if it can not support what is required. Better to query what is supported I think.
-- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
On Wed, Feb 01, 2017 at 12:33:22PM +0100, Maxime Coquelin wrote:
On 02/01/2017 10:43 AM, Daniel P. Berrange wrote:
On Wed, Feb 01, 2017 at 10:14:54AM +0100, Michal Privoznik wrote:
On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
Solution 3: Libvirt queries OVS for vhost backend version string: *OK* ======================================================================
The idea is to have a table of supported versions, associated to key/value pairs. Libvirt could query the list of supported versions strings for each hosts, and select the first common one among all hosts.
How does libvirt know what hosts to ask? Libvirt aims on managing a single host. It has no knowledge of other hosts on the network. That's task for upper layers like RHEV, OpenStack, etc.
Then, libvirt would ask OVS to probe the vhost-user interfaces in the selected version (compatibility mode). For example host A runs OVS-2.7, and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode (e.g. with indirect descriptors disabled), which should be selected at vhost-user interface probe time.
Advantage of doing so is that libvirt does not need any update if new keys are introduced (i.e. it does not need to know how the new keys have to be handled), all these checks remain in OVS's vhost-user implementation.
And that's where they should stay. Duplicating code between projects will inevitably lead to a divergence.
Ideally, we would support per vhost-user interface compatibility mode, which may have an impact also on DPDK API, as the Virtio feature update API is global, and not per port.
In general, I don't think we want any kind of this logic in libvirt. Either:
a) fallback logic should be implemented in qemu (e.g. upon migration it should detect that the migrated guest uses certain version and thus set backend to use that version or error out and cancel migration), or
b) libvirt would grew another element/attribute to specify version of vhost-user backend in use and do nothing more than pass it to qemu. At the same time, we can provide an API (or extend and existing one, e.g. virsh domcapabilities) to list all available versions on given host. Upper layer, which knows what are the possible hosts suitable for virtualization, can then use this API to ask all the hosts, construct the matrix, select preferred version and put it into libvirt's domain XML.
But frankly, I don't like b) that much. Lets put the fact this is OVS aside for a moment. Just pretend this is a generic device in qemu. Would we do the same magic with it? No! Or lets talk about machine types. You spawn -M type$((X+1)) guest and then decide to migrate it to a host with older qemu wich supports just typeX. Well, you get an error. Do we care? Not at all! It's your responsibility (as user/admin) to upgrade the qemu so that it supports new machine type. I think the same applies to OVS.
With machine types, if the latest machine type is X, libvirt allows the mgmt app to spawn a guest with mcahine type X-1, so that you can later migrate the VM to a host with older QEMU.
With the vhost user device, the VM will always be launched with version Y. There's currently no way to request launching the vhost user device wtih version Y-1. So even if you set your machine type to X-1 for compat with older host, vhost user will be stuck at version Y preventing the migration.
One argument would be to say that the vhost user featureset should be determined by the machine type version, instead of introducing a new version. The complexity is that vhost-user is a pretty dumb device from QEMUs POV - most of the interesting logic & the features that need to be constrained lie in code outside of QEMU, in whatever server is connected to the vhost user socket.
So I can see the value in allowing a simple version string to be associated with the vhost-user NIC.
What I'm unclear about is how we would be able to report capabilities for a host to enumerate what versions were possible. Libvirt doesn't interact with any of the 3rd party vhost user servers, so it is probably out of scope - it'd be upto the higher level mgmt app to talk to DPDK and figure out what versions it supports.
That does make me wonder though if libvirt & QEMU need to be involved at all in any part.
Indeed, if the higher level management tool decides for the VM's machine type, it is where it should also be done for the vhost-user backend. I now understand this does not make much sense to have libvirt being involved in this, all (querying/selecting/setting compat mode) should be manageable in the upper layer.
I'm not familiar with these layers, so your inputs are really helpful.
When provisioning a new guest, the mgmt app presumably has to talk to DPDK to setup the NIC port, so DPDK is ready when QEMU launches and connects. Surely as part of that NIC port setup, it could directly tell DPDK which version or featureset to permit on the port ? It is not obvious why the version string has to be fed in via QEMU.
No it is not, my proposal was that libvirt set the version string in OVS, QEMU was not involved.
From these inputs, the idea remains valid, except that libvirt is not the right place to manage this. Instead, RHEV, Openstack or any other management tool should handle the compat mode selection.
It depends where / how in OVS it needs to be set. The only stuff libvirt does with OVS is to run 'add-port' and 'del-port' commands via the ovs cli tool. We pass through arguments from the port profile stored in the XML config.
<interface type='bridge'> <source bridge='ovsbr'/> <virtualport type='openvswitch'> <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> </virtualport> </interface>
eg those things in <parameters/> get passed as cli args to the 'add-port' command. Soo if add-port needs this new version string, then we'd need to add the version to the openvswitch virtualport XML.
If the version is provided to OVS in a different command, then it would probably be outside scope of libvirt.
I think it would make sense to be a parameter of the add-port command. But it would be for vhost-user related add-port command, I didn't find where/if this is managed in libvirt XML. Regards, Maxime
On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
It depends where / how in OVS it needs to be set. The only stuff libvirt does with OVS is to run 'add-port' and 'del-port' commands via the ovs cli tool. We pass through arguments from the port profile stored in the XML config.
<interface type='bridge'> <source bridge='ovsbr'/> <virtualport type='openvswitch'> <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> </virtualport> </interface>
eg those things in <parameters/> get passed as cli args to the 'add-port' command. Soo if add-port needs this new version string, then we'd need to add the version to the openvswitch virtualport XML.
If the version is provided to OVS in a different command, then it would probably be outside scope of libvirt.
I think it would make sense to be a parameter of the add-port command. But it would be for vhost-user related add-port command, I didn't find where/if this is managed in libvirt XML.
For vhost-user, libvirt does not have any interaction with OVS at all. If the thing that's using the vhost-user UNIX socket, in turn connects to OVS, that's outside scope of libvirt. IOW, for vhost-user OVS it seems like that job is for Nova / os-vif to solve. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
It depends where / how in OVS it needs to be set. The only stuff libvirt does with OVS is to run 'add-port' and 'del-port' commands via the ovs cli tool. We pass through arguments from the port profile stored in the XML config.
<interface type='bridge'> <source bridge='ovsbr'/> <virtualport type='openvswitch'> <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> </virtualport> </interface>
eg those things in <parameters/> get passed as cli args to the 'add-port' command. Soo if add-port needs this new version string, then we'd need to add the version to the openvswitch virtualport XML.
If the version is provided to OVS in a different command, then it would probably be outside scope of libvirt.
I think it would make sense to be a parameter of the add-port command. But it would be for vhost-user related add-port command, I didn't find where/if this is managed in libvirt XML.
For vhost-user, libvirt does not have any interaction with OVS at all. If the thing that's using the vhost-user UNIX socket, in turn connects to OVS, that's outside scope of libvirt. IOW, for vhost-user OVS it seems like that job is for Nova / os-vif to solve.
Regards, Daniel
I don't think they currently understand the issues involved in cross-version migration though. This is a complex subject and easy to get wrong. It would be significantly better to figure it out at libvirt level since it already deals with cross-version migration issues.
-- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
It depends where / how in OVS it needs to be set. The only stuff libvirt does with OVS is to run 'add-port' and 'del-port' commands via the ovs cli tool. We pass through arguments from the port profile stored in the XML config.
<interface type='bridge'> <source bridge='ovsbr'/> <virtualport type='openvswitch'> <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> </virtualport> </interface>
eg those things in <parameters/> get passed as cli args to the 'add-port' command. Soo if add-port needs this new version string, then we'd need to add the version to the openvswitch virtualport XML.
If the version is provided to OVS in a different command, then it would probably be outside scope of libvirt.
I think it would make sense to be a parameter of the add-port command. But it would be for vhost-user related add-port command, I didn't find where/if this is managed in libvirt XML.
For vhost-user, libvirt does not have any interaction with OVS at all. If the thing that's using the vhost-user UNIX socket, in turn connects to OVS, that's outside scope of libvirt. IOW, for vhost-user OVS it seems like that job is for Nova / os-vif to solve.
I don't think they currently understand the issues involved in cross-version migration though. This is a complex subject and easy to get wrong. It would be significantly better to figure it out at libvirt level since it already deals with cross-version migration issues.
Libvirt considers vhost-user to be a blackbox though - it just exposes a UNIX socket, and whatever is on the other end is completely opaque. The fact that the other end might plumb the data stream into openvswitch is not something libvirt should know, as we don't want to end up having to add custom code to libvirt for every different vhost-user server impl. IOW, if the version str can be passed to QEMU, and then onto vhost-user backend in QEMU, then libvirt can be involved. If the version str has to be given to openvswitch that's not for libvirt to deall with. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
It depends where / how in OVS it needs to be set. The only stuff libvirt does with OVS is to run 'add-port' and 'del-port' commands via the ovs cli tool. We pass through arguments from the port profile stored in the XML config.
<interface type='bridge'> <source bridge='ovsbr'/> <virtualport type='openvswitch'> <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> </virtualport> </interface>
eg those things in <parameters/> get passed as cli args to the 'add-port' command. Soo if add-port needs this new version string, then we'd need to add the version to the openvswitch virtualport XML.
If the version is provided to OVS in a different command, then it would probably be outside scope of libvirt.
I think it would make sense to be a parameter of the add-port command. But it would be for vhost-user related add-port command, I didn't find where/if this is managed in libvirt XML.
For vhost-user, libvirt does not have any interaction with OVS at all. If the thing that's using the vhost-user UNIX socket, in turn connects to OVS, that's outside scope of libvirt. IOW, for vhost-user OVS it seems like that job is for Nova / os-vif to solve.
I don't think they currently understand the issues involved in cross-version migration though. This is a complex subject and easy to get wrong. It would be significantly better to figure it out at libvirt level since it already deals with cross-version migration issues.
Libvirt considers vhost-user to be a blackbox though - it just exposes a UNIX socket, and whatever is on the other end is completely opaque. The fact that the other end might plumb the data stream into openvswitch is not something libvirt should know, as we don't want to end up having to add custom code to libvirt for every different vhost-user server impl.
IOW, if the version str can be passed to QEMU, and then onto vhost-user backend in QEMU, then libvirt can be involved. If the version str has to be given to openvswitch that's not for libvirt to deall with.
Regards, Daniel
It's not just about passing it to QEMU. The following is needed: - need to query version when creating VM/device - need to query supported versions when migrating VM
-- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
It depends where / how in OVS it needs to be set. The only stuff libvirt does with OVS is to run 'add-port' and 'del-port' commands via the ovs cli tool. We pass through arguments from the port profile stored in the XML config.
<interface type='bridge'> <source bridge='ovsbr'/> <virtualport type='openvswitch'> <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> </virtualport> </interface>
eg those things in <parameters/> get passed as cli args to the 'add-port' command. Soo if add-port needs this new version string, then we'd need to add the version to the openvswitch virtualport XML.
If the version is provided to OVS in a different command, then it would probably be outside scope of libvirt.
I think it would make sense to be a parameter of the add-port command. But it would be for vhost-user related add-port command, I didn't find where/if this is managed in libvirt XML.
For vhost-user, libvirt does not have any interaction with OVS at all. If the thing that's using the vhost-user UNIX socket, in turn connects to OVS, that's outside scope of libvirt. IOW, for vhost-user OVS it seems like that job is for Nova / os-vif to solve.
I don't think they currently understand the issues involved in cross-version migration though. This is a complex subject and easy to get wrong. It would be significantly better to figure it out at libvirt level since it already deals with cross-version migration issues.
Libvirt considers vhost-user to be a blackbox though - it just exposes a UNIX socket, and whatever is on the other end is completely opaque. The fact that the other end might plumb the data stream into openvswitch is not something libvirt should know, as we don't want to end up having to add custom code to libvirt for every different vhost-user server impl.
IOW, if the version str can be passed to QEMU, and then onto vhost-user backend in QEMU, then libvirt can be involved. If the version str has to be given to openvswitch that's not for libvirt to deall with.
Regards, Daniel
It's not just about passing it to QEMU. The following is needed: - need to query version when creating VM/device - need to query supported versions when migrating VM
Those are both things that nova can do, since it knows what the vhost-user device in question is connected to, and so can query the versions, and check versions before triggering migration with libvirt Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
On Thu, Feb 02, 2017 at 05:29:08PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: > > It depends where / how in OVS it needs to be set. The only stuff libvirt > does with OVS is to run 'add-port' and 'del-port' commands via the ovs > cli tool. We pass through arguments from the port profile stored in the > XML config. > > <interface type='bridge'> > <source bridge='ovsbr'/> > <virtualport type='openvswitch'> > <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> > </virtualport> > </interface> > > eg those things in <parameters/> get passed as cli args to the 'add-port' > command. Soo if add-port needs this new version string, then we'd need > to add the version to the openvswitch virtualport XML. > > If the version is provided to OVS in a different command, then it would > probably be outside scope of libvirt.
I think it would make sense to be a parameter of the add-port command. But it would be for vhost-user related add-port command, I didn't find where/if this is managed in libvirt XML.
For vhost-user, libvirt does not have any interaction with OVS at all. If the thing that's using the vhost-user UNIX socket, in turn connects to OVS, that's outside scope of libvirt. IOW, for vhost-user OVS it seems like that job is for Nova / os-vif to solve.
I don't think they currently understand the issues involved in cross-version migration though. This is a complex subject and easy to get wrong. It would be significantly better to figure it out at libvirt level since it already deals with cross-version migration issues.
Libvirt considers vhost-user to be a blackbox though - it just exposes a UNIX socket, and whatever is on the other end is completely opaque. The fact that the other end might plumb the data stream into openvswitch is not something libvirt should know, as we don't want to end up having to add custom code to libvirt for every different vhost-user server impl.
IOW, if the version str can be passed to QEMU, and then onto vhost-user backend in QEMU, then libvirt can be involved. If the version str has to be given to openvswitch that's not for libvirt to deall with.
Regards, Daniel
It's not just about passing it to QEMU. The following is needed: - need to query version when creating VM/device - need to query supported versions when migrating VM
Those are both things that nova can do, since it knows what the vhost-user device in question is connected to, and so can query the versions, and check versions before triggering migration with libvirt
Regards, Daniel
It can, but then it will need to query libvirt or source for the version string since it's in the XML.
-- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
On Thu, Feb 02, 2017 at 07:31:49PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 05:29:08PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote: > > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: > > > > It depends where / how in OVS it needs to be set. The only stuff libvirt > > does with OVS is to run 'add-port' and 'del-port' commands via the ovs > > cli tool. We pass through arguments from the port profile stored in the > > XML config. > > > > <interface type='bridge'> > > <source bridge='ovsbr'/> > > <virtualport type='openvswitch'> > > <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> > > </virtualport> > > </interface> > > > > eg those things in <parameters/> get passed as cli args to the 'add-port' > > command. Soo if add-port needs this new version string, then we'd need > > to add the version to the openvswitch virtualport XML. > > > > If the version is provided to OVS in a different command, then it would > > probably be outside scope of libvirt. > > I think it would make sense to be a parameter of the add-port command. > But it would be for vhost-user related add-port command, I didn't find > where/if this is managed in libvirt XML.
For vhost-user, libvirt does not have any interaction with OVS at all. If the thing that's using the vhost-user UNIX socket, in turn connects to OVS, that's outside scope of libvirt. IOW, for vhost-user OVS it seems like that job is for Nova / os-vif to solve.
I don't think they currently understand the issues involved in cross-version migration though. This is a complex subject and easy to get wrong. It would be significantly better to figure it out at libvirt level since it already deals with cross-version migration issues.
Libvirt considers vhost-user to be a blackbox though - it just exposes a UNIX socket, and whatever is on the other end is completely opaque. The fact that the other end might plumb the data stream into openvswitch is not something libvirt should know, as we don't want to end up having to add custom code to libvirt for every different vhost-user server impl.
IOW, if the version str can be passed to QEMU, and then onto vhost-user backend in QEMU, then libvirt can be involved. If the version str has to be given to openvswitch that's not for libvirt to deall with.
Regards, Daniel
It's not just about passing it to QEMU. The following is needed: - need to query version when creating VM/device - need to query supported versions when migrating VM
Those are both things that nova can do, since it knows what the vhost-user device in question is connected to, and so can query the versions, and check versions before triggering migration with libvirt
It can, but then it will need to query libvirt or source for the version string since it's in the XML.
No, it wouldn't be in the XML at all. Nova on the source queries what vhostuser version it has and what the target host has, and can prevent the migration if they're incompatible. I dont think libvirt has to be involved at all for this, as all the info can be obtained by nova/os-vif from the vhostuser impl it has configured. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
On Thu, Feb 02, 2017 at 06:21:55PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 07:31:49PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 05:29:08PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote: > On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote: > > > > > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: > > > > > > It depends where / how in OVS it needs to be set. The only stuff libvirt > > > does with OVS is to run 'add-port' and 'del-port' commands via the ovs > > > cli tool. We pass through arguments from the port profile stored in the > > > XML config. > > > > > > <interface type='bridge'> > > > <source bridge='ovsbr'/> > > > <virtualport type='openvswitch'> > > > <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> > > > </virtualport> > > > </interface> > > > > > > eg those things in <parameters/> get passed as cli args to the 'add-port' > > > command. Soo if add-port needs this new version string, then we'd need > > > to add the version to the openvswitch virtualport XML. > > > > > > If the version is provided to OVS in a different command, then it would > > > probably be outside scope of libvirt. > > > > I think it would make sense to be a parameter of the add-port command. > > But it would be for vhost-user related add-port command, I didn't find > > where/if this is managed in libvirt XML. > > For vhost-user, libvirt does not have any interaction with OVS at > all. If the thing that's using the vhost-user UNIX socket, in turn > connects to OVS, that's outside scope of libvirt. IOW, for vhost-user > OVS it seems like that job is for Nova / os-vif to solve.
I don't think they currently understand the issues involved in cross-version migration though. This is a complex subject and easy to get wrong. It would be significantly better to figure it out at libvirt level since it already deals with cross-version migration issues.
Libvirt considers vhost-user to be a blackbox though - it just exposes a UNIX socket, and whatever is on the other end is completely opaque. The fact that the other end might plumb the data stream into openvswitch is not something libvirt should know, as we don't want to end up having to add custom code to libvirt for every different vhost-user server impl.
IOW, if the version str can be passed to QEMU, and then onto vhost-user backend in QEMU, then libvirt can be involved. If the version str has to be given to openvswitch that's not for libvirt to deall with.
Regards, Daniel
It's not just about passing it to QEMU. The following is needed: - need to query version when creating VM/device - need to query supported versions when migrating VM
Those are both things that nova can do, since it knows what the vhost-user device in question is connected to, and so can query the versions, and check versions before triggering migration with libvirt
It can, but then it will need to query libvirt or source for the version string since it's in the XML.
No, it wouldn't be in the XML at all. Nova on the source queries what vhostuser version it has and what the target host has, and can prevent the migration if they're incompatible.
This is not sufficient. Exactly the same as qemu machine type, this must be preserved from time of install and moved wherever VM goes.
I dont think libvirt has to be involved at all for this, as all the info can be obtained by nova/os-vif from the vhostuser impl it has configured.
Regards, Daniel
Given we are already confused at libvirt level, I find it highly unlikely upper levels will do the right thing. Generally I simply don't understand why would we expose to higher levels something which is fundamentally not a policy decision.
-- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
On Thu, Feb 02, 2017 at 08:27:18PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 06:21:55PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 07:31:49PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 05:29:08PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote: > On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote: > > On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote: > > > > > > > > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: > > > > > > > > It depends where / how in OVS it needs to be set. The only stuff libvirt > > > > does with OVS is to run 'add-port' and 'del-port' commands via the ovs > > > > cli tool. We pass through arguments from the port profile stored in the > > > > XML config. > > > > > > > > <interface type='bridge'> > > > > <source bridge='ovsbr'/> > > > > <virtualport type='openvswitch'> > > > > <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> > > > > </virtualport> > > > > </interface> > > > > > > > > eg those things in <parameters/> get passed as cli args to the 'add-port' > > > > command. Soo if add-port needs this new version string, then we'd need > > > > to add the version to the openvswitch virtualport XML. > > > > > > > > If the version is provided to OVS in a different command, then it would > > > > probably be outside scope of libvirt. > > > > > > I think it would make sense to be a parameter of the add-port command. > > > But it would be for vhost-user related add-port command, I didn't find > > > where/if this is managed in libvirt XML. > > > > For vhost-user, libvirt does not have any interaction with OVS at > > all. If the thing that's using the vhost-user UNIX socket, in turn > > connects to OVS, that's outside scope of libvirt. IOW, for vhost-user > > OVS it seems like that job is for Nova / os-vif to solve. > > I don't think they currently understand the issues involved in > cross-version migration though. This is a complex subject and easy to > get wrong. It would be significantly better to figure it out at libvirt > level since it already deals with cross-version migration issues.
Libvirt considers vhost-user to be a blackbox though - it just exposes a UNIX socket, and whatever is on the other end is completely opaque. The fact that the other end might plumb the data stream into openvswitch is not something libvirt should know, as we don't want to end up having to add custom code to libvirt for every different vhost-user server impl.
IOW, if the version str can be passed to QEMU, and then onto vhost-user backend in QEMU, then libvirt can be involved. If the version str has to be given to openvswitch that's not for libvirt to deall with.
Regards, Daniel
It's not just about passing it to QEMU. The following is needed: - need to query version when creating VM/device - need to query supported versions when migrating VM
Those are both things that nova can do, since it knows what the vhost-user device in question is connected to, and so can query the versions, and check versions before triggering migration with libvirt
It can, but then it will need to query libvirt or source for the version string since it's in the XML.
No, it wouldn't be in the XML at all. Nova on the source queries what vhostuser version it has and what the target host has, and can prevent the migration if they're incompatible.
This is not sufficient. Exactly the same as qemu machine type, this must be preserved from time of install and moved wherever VM goes.
Nova has the ability todo that.
I dont think libvirt has to be involved at all for this, as all the info can be obtained by nova/os-vif from the vhostuser impl it has configured.
Given we are already confused at libvirt level, I find it highly unlikely upper levels will do the right thing.
The only confusion is about what must consume the data. I mistakenly thought it was being proposed that the qemu vhostuser backend was being given a version string. Now it seems the version info must be set against the 3rd party component that vhostuser connects to, and that is outside scope of libvirt. That is entirely managed by Nova today, so Nova is the right thing to manage this versioning info. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
On 02/03/2017 10:27 AM, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 08:27:18PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 06:21:55PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 07:31:49PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 05:29:08PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote: > On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote: >> On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote: >>> On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote: >>>> >>>> >>>> On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: >>>>> >>>>> It depends where / how in OVS it needs to be set. The only stuff libvirt >>>>> does with OVS is to run 'add-port' and 'del-port' commands via the ovs >>>>> cli tool. We pass through arguments from the port profile stored in the >>>>> XML config. >>>>> >>>>> <interface type='bridge'> >>>>> <source bridge='ovsbr'/> >>>>> <virtualport type='openvswitch'> >>>>> <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> >>>>> </virtualport> >>>>> </interface> >>>>> >>>>> eg those things in <parameters/> get passed as cli args to the 'add-port' >>>>> command. Soo if add-port needs this new version string, then we'd need >>>>> to add the version to the openvswitch virtualport XML. >>>>> >>>>> If the version is provided to OVS in a different command, then it would >>>>> probably be outside scope of libvirt. >>>> >>>> I think it would make sense to be a parameter of the add-port command. >>>> But it would be for vhost-user related add-port command, I didn't find >>>> where/if this is managed in libvirt XML. >>> >>> For vhost-user, libvirt does not have any interaction with OVS at >>> all. If the thing that's using the vhost-user UNIX socket, in turn >>> connects to OVS, that's outside scope of libvirt. IOW, for vhost-user >>> OVS it seems like that job is for Nova / os-vif to solve. >> >> I don't think they currently understand the issues involved in >> cross-version migration though. This is a complex subject and easy to >> get wrong. It would be significantly better to figure it out at libvirt >> level since it already deals with cross-version migration issues. > > Libvirt considers vhost-user to be a blackbox though - it just exposes > a UNIX socket, and whatever is on the other end is completely opaque. > The fact that the other end might plumb the data stream into openvswitch > is not something libvirt should know, as we don't want to end up having > to add custom code to libvirt for every different vhost-user server > impl. > > IOW, if the version str can be passed to QEMU, and then onto vhost-user > backend in QEMU, then libvirt can be involved. If the version str has > to be given to openvswitch that's not for libvirt to deall with. > > Regards, > Daniel
It's not just about passing it to QEMU. The following is needed: - need to query version when creating VM/device - need to query supported versions when migrating VM
Those are both things that nova can do, since it knows what the vhost-user device in question is connected to, and so can query the versions, and check versions before triggering migration with libvirt
It can, but then it will need to query libvirt or source for the version string since it's in the XML.
No, it wouldn't be in the XML at all. Nova on the source queries what vhostuser version it has and what the target host has, and can prevent the migration if they're incompatible.
This is not sufficient. Exactly the same as qemu machine type, this must be preserved from time of install and moved wherever VM goes.
Nova has the ability todo that.
I dont think libvirt has to be involved at all for this, as all the info can be obtained by nova/os-vif from the vhostuser impl it has configured.
Given we are already confused at libvirt level, I find it highly unlikely upper levels will do the right thing.
The only confusion is about what must consume the data. I mistakenly thought it was being proposed that the qemu vhostuser backend was being given a version string. Now it seems the version info must be set against the 3rd party component that vhostuser connects to, and that is outside scope of libvirt. That is entirely managed by Nova today, so Nova is the right thing to manage this versioning info.
This is my understanding, with as example using Nova: 1. Before starting the VM, Nova queries source host's OVS and all possible destination hosts' OVS to retrieve supported versions 2. Nova chooses the first common version supported by all hosts 3. Nova create the dpdkvhostuser interfaces in OVS with passing the selected version as parameter That should be enough, right? Or maybe I miss something? Thanks, Maxime
On Fri, Feb 03, 2017 at 10:41:12AM +0100, Maxime Coquelin wrote:
On 02/03/2017 10:27 AM, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 08:27:18PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 06:21:55PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 07:31:49PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 05:29:08PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote: > On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote: > > On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote: > > > On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote: > > > > On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote: > > > > > > > > > > > > > > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: > > > > > > > > > > > > It depends where / how in OVS it needs to be set. The only stuff libvirt > > > > > > does with OVS is to run 'add-port' and 'del-port' commands via the ovs > > > > > > cli tool. We pass through arguments from the port profile stored in the > > > > > > XML config. > > > > > > > > > > > > <interface type='bridge'> > > > > > > <source bridge='ovsbr'/> > > > > > > <virtualport type='openvswitch'> > > > > > > <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> > > > > > > </virtualport> > > > > > > </interface> > > > > > > > > > > > > eg those things in <parameters/> get passed as cli args to the 'add-port' > > > > > > command. Soo if add-port needs this new version string, then we'd need > > > > > > to add the version to the openvswitch virtualport XML. > > > > > > > > > > > > If the version is provided to OVS in a different command, then it would > > > > > > probably be outside scope of libvirt. > > > > > > > > > > I think it would make sense to be a parameter of the add-port command. > > > > > But it would be for vhost-user related add-port command, I didn't find > > > > > where/if this is managed in libvirt XML. > > > > > > > > For vhost-user, libvirt does not have any interaction with OVS at > > > > all. If the thing that's using the vhost-user UNIX socket, in turn > > > > connects to OVS, that's outside scope of libvirt. IOW, for vhost-user > > > > OVS it seems like that job is for Nova / os-vif to solve. > > > > > > I don't think they currently understand the issues involved in > > > cross-version migration though. This is a complex subject and easy to > > > get wrong. It would be significantly better to figure it out at libvirt > > > level since it already deals with cross-version migration issues. > > > > Libvirt considers vhost-user to be a blackbox though - it just exposes > > a UNIX socket, and whatever is on the other end is completely opaque. > > The fact that the other end might plumb the data stream into openvswitch > > is not something libvirt should know, as we don't want to end up having > > to add custom code to libvirt for every different vhost-user server > > impl. > > > > IOW, if the version str can be passed to QEMU, and then onto vhost-user > > backend in QEMU, then libvirt can be involved. If the version str has > > to be given to openvswitch that's not for libvirt to deall with. > > > > Regards, > > Daniel > > It's not just about passing it to QEMU. The following is needed: > - need to query version when creating VM/device > - need to query supported versions when migrating VM
Those are both things that nova can do, since it knows what the vhost-user device in question is connected to, and so can query the versions, and check versions before triggering migration with libvirt
It can, but then it will need to query libvirt or source for the version string since it's in the XML.
No, it wouldn't be in the XML at all. Nova on the source queries what vhostuser version it has and what the target host has, and can prevent the migration if they're incompatible.
This is not sufficient. Exactly the same as qemu machine type, this must be preserved from time of install and moved wherever VM goes.
Nova has the ability todo that.
I dont think libvirt has to be involved at all for this, as all the info can be obtained by nova/os-vif from the vhostuser impl it has configured.
Given we are already confused at libvirt level, I find it highly unlikely upper levels will do the right thing.
The only confusion is about what must consume the data. I mistakenly thought it was being proposed that the qemu vhostuser backend was being given a version string. Now it seems the version info must be set against the 3rd party component that vhostuser connects to, and that is outside scope of libvirt. That is entirely managed by Nova today, so Nova is the right thing to manage this versioning info.
This is my understanding, with as example using Nova: 1. Before starting the VM, Nova queries source host's OVS and all possible destination hosts' OVS to retrieve supported versions 2. Nova chooses the first common version supported by all hosts 3. Nova create the dpdkvhostuser interfaces in OVS with passing the selected version as parameter
That should be enough, right? Or maybe I miss something?
It isn't realistic to check all possible destination hosts when first starting QEMU, not least becasue they may not yet exist. New hardware may be deployed *after* QEMU is first started. When starting the VM, it should just default to using whatever the latest version is on the host in question. There can be an optional host configuration parameter that admins can set to force use of an older version. This is how Nova deals with machine type - it defaults to latest, but you can set hw_machine_type in nova.conf to force an older version if you need migration compatibility between hosts with varying versions. At time of migration, Nova need to request the version of the target host. If this version is not compatible with what the VM is currently using, then Nova should *not* start the migration, it should raise an error such that the scheduler retries and picks a different target host Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
On 02/03/2017 11:11 AM, Daniel P. Berrange wrote:
On Fri, Feb 03, 2017 at 10:41:12AM +0100, Maxime Coquelin wrote:
On 02/03/2017 10:27 AM, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 08:27:18PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 06:21:55PM +0000, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 07:31:49PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 05:29:08PM +0000, Daniel P. Berrange wrote: > On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote: >> On Thu, Feb 02, 2017 at 05:10:28PM +0000, Daniel P. Berrange wrote: >>> On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote: >>>> On Thu, Feb 02, 2017 at 03:06:21PM +0000, Daniel P. Berrange wrote: >>>>> On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote: >>>>>> >>>>>> >>>>>> On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: >>>>>>> >>>>>>> It depends where / how in OVS it needs to be set. The only stuff libvirt >>>>>>> does with OVS is to run 'add-port' and 'del-port' commands via the ovs >>>>>>> cli tool. We pass through arguments from the port profile stored in the >>>>>>> XML config. >>>>>>> >>>>>>> <interface type='bridge'> >>>>>>> <source bridge='ovsbr'/> >>>>>>> <virtualport type='openvswitch'> >>>>>>> <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> >>>>>>> </virtualport> >>>>>>> </interface> >>>>>>> >>>>>>> eg those things in <parameters/> get passed as cli args to the 'add-port' >>>>>>> command. Soo if add-port needs this new version string, then we'd need >>>>>>> to add the version to the openvswitch virtualport XML. >>>>>>> >>>>>>> If the version is provided to OVS in a different command, then it would >>>>>>> probably be outside scope of libvirt. >>>>>> >>>>>> I think it would make sense to be a parameter of the add-port command. >>>>>> But it would be for vhost-user related add-port command, I didn't find >>>>>> where/if this is managed in libvirt XML. >>>>> >>>>> For vhost-user, libvirt does not have any interaction with OVS at >>>>> all. If the thing that's using the vhost-user UNIX socket, in turn >>>>> connects to OVS, that's outside scope of libvirt. IOW, for vhost-user >>>>> OVS it seems like that job is for Nova / os-vif to solve. >>>> >>>> I don't think they currently understand the issues involved in >>>> cross-version migration though. This is a complex subject and easy to >>>> get wrong. It would be significantly better to figure it out at libvirt >>>> level since it already deals with cross-version migration issues. >>> >>> Libvirt considers vhost-user to be a blackbox though - it just exposes >>> a UNIX socket, and whatever is on the other end is completely opaque. >>> The fact that the other end might plumb the data stream into openvswitch >>> is not something libvirt should know, as we don't want to end up having >>> to add custom code to libvirt for every different vhost-user server >>> impl. >>> >>> IOW, if the version str can be passed to QEMU, and then onto vhost-user >>> backend in QEMU, then libvirt can be involved. If the version str has >>> to be given to openvswitch that's not for libvirt to deall with. >>> >>> Regards, >>> Daniel >> >> It's not just about passing it to QEMU. The following is needed: >> - need to query version when creating VM/device >> - need to query supported versions when migrating VM > > Those are both things that nova can do, since it knows what the vhost-user > device in question is connected to, and so can query the versions, and check > versions before triggering migration with libvirt
It can, but then it will need to query libvirt or source for the version string since it's in the XML.
No, it wouldn't be in the XML at all. Nova on the source queries what vhostuser version it has and what the target host has, and can prevent the migration if they're incompatible.
This is not sufficient. Exactly the same as qemu machine type, this must be preserved from time of install and moved wherever VM goes.
Nova has the ability todo that.
I dont think libvirt has to be involved at all for this, as all the info can be obtained by nova/os-vif from the vhostuser impl it has configured.
Given we are already confused at libvirt level, I find it highly unlikely upper levels will do the right thing.
The only confusion is about what must consume the data. I mistakenly thought it was being proposed that the qemu vhostuser backend was being given a version string. Now it seems the version info must be set against the 3rd party component that vhostuser connects to, and that is outside scope of libvirt. That is entirely managed by Nova today, so Nova is the right thing to manage this versioning info.
This is my understanding, with as example using Nova: 1. Before starting the VM, Nova queries source host's OVS and all possible destination hosts' OVS to retrieve supported versions 2. Nova chooses the first common version supported by all hosts 3. Nova create the dpdkvhostuser interfaces in OVS with passing the selected version as parameter
That should be enough, right? Or maybe I miss something?
It isn't realistic to check all possible destination hosts when first starting QEMU, not least becasue they may not yet exist. New hardware may be deployed *after* QEMU is first started.
When starting the VM, it should just default to using whatever the latest version is on the host in question.
There can be an optional host configuration parameter that admins can set to force use of an older version.
This is how Nova deals with machine type - it defaults to latest, but you can set hw_machine_type in nova.conf to force an older version if you need migration compatibility between hosts with varying versions.
Ok, I see. I thought it could be automated somehow if a list of available host is already present.
At time of migration, Nova need to request the version of the target host. If this version is not compatible with what the VM is currently using, then Nova should *not* start the migration, it should raise an error such that the scheduler retries and picks a different target host
Thanks for the clarification. Maxime
On 02/02/2017 10:06 AM, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
It depends where / how in OVS it needs to be set. The only stuff libvirt does with OVS is to run 'add-port' and 'del-port' commands via the ovs cli tool.
(aside note: the code that exec's ovs-vsctl was written back when there was no standardized API for performing such operations. libvirt would prefer to not exec external programs though, and I've heard that OVS may now have an official API of some sort for doing things like this (maybe via netlink or dbus or something?) If that's the case, can someone point me in the right direction?)
We pass through arguments from the port profile stored in the XML config.
<interface type='bridge'> <source bridge='ovsbr'/> <virtualport type='openvswitch'> <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> </virtualport> </interface>
eg those things in <parameters/> get passed as cli args to the 'add-port' command. Soo if add-port needs this new version string, then we'd need to add the version to the openvswitch virtualport XML.
If the version is provided to OVS in a different command, then it would probably be outside scope of libvirt. I think it would make sense to be a parameter of the add-port command. But it would be for vhost-user related add-port command, I didn't find where/if this is managed in libvirt XML. For vhost-user, libvirt does not have any interaction with OVS at all. If the thing that's using the vhost-user UNIX socket, in turn connects to OVS, that's outside scope of libvirt. IOW, for vhost-user OVS it seems like that job is for Nova / os-vif to solve.
This brings up another tangentially related question I came up against last night - qemu now has an option to report the host's MTU to the guest for virtio and vhost-user interfaces, and Michal Privoznik recently pushed patches to set the MTU sent to the guest via an explicit <mtu size='n'/> in libvirt's interface config: https://bugzilla.redhat.com/1408701 But it would be much nicer if libvirt could learn the MTU of [that stuff at the other end of the unix socket] without requiring intervention in libvirt's config. For example, I'm just now testing patches for tap-based interfaces (connecting to Linux host bridges or OVS switches) that query the current MTU of the bridge and report that to qemu; this eliminates the burden of configuring each interface of each guest individually (and changing that config in all those places if someone ever wants to change the MTU of the bridge). As Dan says, though, libvirt's only interaction in the case of vhost-user is with the unix socket. Is there any way to learn what is the appropriate MTU from OVS in these cases? Or must Nova (or ovirt or some poor user) set that up in the libvirt config for every single interface?
On Thu, Feb 02, 2017 at 11:47:57AM -0500, Laine Stump wrote:
On 02/02/2017 10:06 AM, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
It depends where / how in OVS it needs to be set. The only stuff libvirt does with OVS is to run 'add-port' and 'del-port' commands via the ovs cli tool.
(aside note: the code that exec's ovs-vsctl was written back when there was no standardized API for performing such operations. libvirt would prefer to not exec external programs though, and I've heard that OVS may now have an official API of some sort for doing things like this (maybe via netlink or dbus or something?) If that's the case, can someone point me in the right direction?)
We pass through arguments from the port profile stored in the XML config.
<interface type='bridge'> <source bridge='ovsbr'/> <virtualport type='openvswitch'> <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> </virtualport> </interface>
eg those things in <parameters/> get passed as cli args to the 'add-port' command. Soo if add-port needs this new version string, then we'd need to add the version to the openvswitch virtualport XML.
If the version is provided to OVS in a different command, then it would probably be outside scope of libvirt. I think it would make sense to be a parameter of the add-port command. But it would be for vhost-user related add-port command, I didn't find where/if this is managed in libvirt XML. For vhost-user, libvirt does not have any interaction with OVS at all. If the thing that's using the vhost-user UNIX socket, in turn connects to OVS, that's outside scope of libvirt. IOW, for vhost-user OVS it seems like that job is for Nova / os-vif to solve.
This brings up another tangentially related question I came up against last night - qemu now has an option to report the host's MTU to the guest for virtio and vhost-user interfaces, and Michal Privoznik recently pushed patches to set the MTU sent to the guest via an explicit <mtu size='n'/> in libvirt's interface config:
https://bugzilla.redhat.com/1408701
But it would be much nicer if libvirt could learn the MTU of [that stuff at the other end of the unix socket] without requiring intervention in libvirt's config. For example, I'm just now testing patches for tap-based interfaces (connecting to Linux host bridges or OVS switches) that query the current MTU of the bridge and report that to qemu; this eliminates the burden of configuring each interface of each guest individually (and changing that config in all those places if someone ever wants to change the MTU of the bridge).
As Dan says, though, libvirt's only interaction in the case of vhost-user is with the unix socket. Is there any way to learn what is the appropriate MTU from OVS in these cases? Or must Nova (or ovirt or some poor user) set that up in the libvirt config for every single interface?
We could add commands for all kind of queries to the vhost-user protocol. libvirt would have to learn the vhost-user protocol though. Interested? -- MST
On Thu, Feb 02, 2017 at 07:09:24PM +0200, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 11:47:57AM -0500, Laine Stump wrote:
On 02/02/2017 10:06 AM, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
It depends where / how in OVS it needs to be set. The only stuff libvirt does with OVS is to run 'add-port' and 'del-port' commands via the ovs cli tool.
(aside note: the code that exec's ovs-vsctl was written back when there was no standardized API for performing such operations. libvirt would prefer to not exec external programs though, and I've heard that OVS may now have an official API of some sort for doing things like this (maybe via netlink or dbus or something?) If that's the case, can someone point me in the right direction?)
We pass through arguments from the port profile stored in the XML config.
<interface type='bridge'> <source bridge='ovsbr'/> <virtualport type='openvswitch'> <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> </virtualport> </interface>
eg those things in <parameters/> get passed as cli args to the 'add-port' command. Soo if add-port needs this new version string, then we'd need to add the version to the openvswitch virtualport XML.
If the version is provided to OVS in a different command, then it would probably be outside scope of libvirt. I think it would make sense to be a parameter of the add-port command. But it would be for vhost-user related add-port command, I didn't find where/if this is managed in libvirt XML. For vhost-user, libvirt does not have any interaction with OVS at all. If the thing that's using the vhost-user UNIX socket, in turn connects to OVS, that's outside scope of libvirt. IOW, for vhost-user OVS it seems like that job is for Nova / os-vif to solve.
This brings up another tangentially related question I came up against last night - qemu now has an option to report the host's MTU to the guest for virtio and vhost-user interfaces, and Michal Privoznik recently pushed patches to set the MTU sent to the guest via an explicit <mtu size='n'/> in libvirt's interface config:
https://bugzilla.redhat.com/1408701
But it would be much nicer if libvirt could learn the MTU of [that stuff at the other end of the unix socket] without requiring intervention in libvirt's config. For example, I'm just now testing patches for tap-based interfaces (connecting to Linux host bridges or OVS switches) that query the current MTU of the bridge and report that to qemu; this eliminates the burden of configuring each interface of each guest individually (and changing that config in all those places if someone ever wants to change the MTU of the bridge).
As Dan says, though, libvirt's only interaction in the case of vhost-user is with the unix socket. Is there any way to learn what is the appropriate MTU from OVS in these cases? Or must Nova (or ovirt or some poor user) set that up in the libvirt config for every single interface?
We could add commands for all kind of queries to the vhost-user protocol. libvirt would have to learn the vhost-user protocol though. Interested?
No, I really don't think libvirt should implement the vhost-user protocol Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
On 02/02/2017 06:09 PM, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 11:47:57AM -0500, Laine Stump wrote:
On 02/02/2017 10:06 AM, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
It depends where / how in OVS it needs to be set. The only stuff libvirt does with OVS is to run 'add-port' and 'del-port' commands via the ovs cli tool.
(aside note: the code that exec's ovs-vsctl was written back when there was no standardized API for performing such operations. libvirt would prefer to not exec external programs though, and I've heard that OVS may now have an official API of some sort for doing things like this (maybe via netlink or dbus or something?) If that's the case, can someone point me in the right direction?)
We pass through arguments from the port profile stored in the XML config.
<interface type='bridge'> <source bridge='ovsbr'/> <virtualport type='openvswitch'> <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> </virtualport> </interface>
eg those things in <parameters/> get passed as cli args to the 'add-port' command. Soo if add-port needs this new version string, then we'd need to add the version to the openvswitch virtualport XML.
If the version is provided to OVS in a different command, then it would probably be outside scope of libvirt. I think it would make sense to be a parameter of the add-port command. But it would be for vhost-user related add-port command, I didn't find where/if this is managed in libvirt XML. For vhost-user, libvirt does not have any interaction with OVS at all. If the thing that's using the vhost-user UNIX socket, in turn connects to OVS, that's outside scope of libvirt. IOW, for vhost-user OVS it seems like that job is for Nova / os-vif to solve.
This brings up another tangentially related question I came up against last night - qemu now has an option to report the host's MTU to the guest for virtio and vhost-user interfaces, and Michal Privoznik recently pushed patches to set the MTU sent to the guest via an explicit <mtu size='n'/> in libvirt's interface config:
https://bugzilla.redhat.com/1408701
But it would be much nicer if libvirt could learn the MTU of [that stuff at the other end of the unix socket] without requiring intervention in libvirt's config. For example, I'm just now testing patches for tap-based interfaces (connecting to Linux host bridges or OVS switches) that query the current MTU of the bridge and report that to qemu; this eliminates the burden of configuring each interface of each guest individually (and changing that config in all those places if someone ever wants to change the MTU of the bridge).
As Dan says, though, libvirt's only interaction in the case of vhost-user is with the unix socket. Is there any way to learn what is the appropriate MTU from OVS in these cases? Or must Nova (or ovirt or some poor user) set that up in the libvirt config for every single interface?
We could add commands for all kind of queries to the vhost-user protocol. libvirt would have to learn the vhost-user protocol though. Interested?
I think it could be possible to query the MTU value from the OVS DB using its JSON RPC-like API, but this is something I haven't tried. I guess it would need to resolve the ovs interface from the vhost-user socket path. Can people familiar with OVS confirm this is something possible? Regards, Maxime
On 02/02/2017 06:16 PM, Maxime Coquelin wrote:
On 02/02/2017 06:09 PM, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 11:47:57AM -0500, Laine Stump wrote:
On 02/02/2017 10:06 AM, Daniel P. Berrange wrote:
On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
On 02/01/2017 12:41 PM, Daniel P. Berrange wrote:
It depends where / how in OVS it needs to be set. The only stuff libvirt does with OVS is to run 'add-port' and 'del-port' commands via the ovs cli tool.
(aside note: the code that exec's ovs-vsctl was written back when there was no standardized API for performing such operations. libvirt would prefer to not exec external programs though, and I've heard that OVS may now have an official API of some sort for doing things like this (maybe via netlink or dbus or something?) If that's the case, can someone point me in the right direction?)
We pass through arguments from the port profile stored in the XML config.
<interface type='bridge'> <source bridge='ovsbr'/> <virtualport type='openvswitch'> <parameters profileid='menial' interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> </virtualport> </interface>
eg those things in <parameters/> get passed as cli args to the 'add-port' command. Soo if add-port needs this new version string, then we'd need to add the version to the openvswitch virtualport XML.
If the version is provided to OVS in a different command, then it would probably be outside scope of libvirt. I think it would make sense to be a parameter of the add-port command. But it would be for vhost-user related add-port command, I didn't find where/if this is managed in libvirt XML. For vhost-user, libvirt does not have any interaction with OVS at all. If the thing that's using the vhost-user UNIX socket, in turn connects to OVS, that's outside scope of libvirt. IOW, for vhost-user OVS it seems like that job is for Nova / os-vif to solve.
This brings up another tangentially related question I came up against last night - qemu now has an option to report the host's MTU to the guest for virtio and vhost-user interfaces, and Michal Privoznik recently pushed patches to set the MTU sent to the guest via an explicit <mtu size='n'/> in libvirt's interface config:
https://bugzilla.redhat.com/1408701
But it would be much nicer if libvirt could learn the MTU of [that stuff at the other end of the unix socket] without requiring intervention in libvirt's config. For example, I'm just now testing patches for tap-based interfaces (connecting to Linux host bridges or OVS switches) that query the current MTU of the bridge and report that to qemu; this eliminates the burden of configuring each interface of each guest individually (and changing that config in all those places if someone ever wants to change the MTU of the bridge).
As Dan says, though, libvirt's only interaction in the case of vhost-user is with the unix socket. Is there any way to learn what is the appropriate MTU from OVS in these cases? Or must Nova (or ovirt or some poor user) set that up in the libvirt config for every single interface?
We could add commands for all kind of queries to the vhost-user protocol. libvirt would have to learn the vhost-user protocol though. Interested?
I think it could be possible to query the MTU value from the OVS DB using its JSON RPC-like API, but this is something I haven't tried. I guess it would need to resolve the ovs interface from the vhost-user socket path.
Can people familiar with OVS confirm this is something possible?
Libvirt does not use OVS' JSON RPC yet (it'll be long way to go anyway). Therefore, for any OVS interaction it uses ovs-vsctl directly. Therefore: ovs-vsctl get Interface ovsbr0 mtu is what Laine is looking for. However, something fishy is happening here: lisa ~ # ovs-vsctl set Interface ovsbr0 mtu=9000 lisa ~ # ovs-vsctl get Interface ovsbr0 mtu 1500 lisa ~ # ifconfig ovsbr0 mtu 9000 lisa ~ # ovs-vsctl get Interface ovsbr0 mtu 9000 (yes, Lisa Simpson) But since we just want to query the bridge's MTU and not set it, we should be safe. Michal
On 02/03/2017 04:12 AM, Michal Privoznik wrote:
On 02/02/2017 06:16 PM, Maxime Coquelin wrote:
On 02/02/2017 06:09 PM, Michael S. Tsirkin wrote:
On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote:
On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: > It depends where / how in OVS it needs to be set. The only stuff > libvirt > does with OVS is to run 'add-port' and 'del-port' commands via the > ovs > cli tool. (aside note: the code that exec's ovs-vsctl was written back when
On 02/02/2017 10:06 AM, Daniel P. Berrange wrote: there was no standardized API for performing such operations. libvirt would prefer to not exec external programs though, and I've heard that OVS may now have an official API of some sort for doing things like this (maybe via netlink or dbus or something?) If that's the case, can someone point me in the right direction?)
> We pass through arguments from the port profile stored in the > XML config. > > <interface type='bridge'> > <source bridge='ovsbr'/> > <virtualport type='openvswitch'> > <parameters profileid='menial' > interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> > </virtualport> > </interface> > > eg those things in <parameters/> get passed as cli args to the > 'add-port' > command. Soo if add-port needs this new version string, then we'd > need > to add the version to the openvswitch virtualport XML. > > If the version is provided to OVS in a different command, then it > would > probably be outside scope of libvirt. I think it would make sense to be a parameter of the add-port command. But it would be for vhost-user related add-port command, I didn't find where/if this is managed in libvirt XML. For vhost-user, libvirt does not have any interaction with OVS at all. If the thing that's using the vhost-user UNIX socket, in turn connects to OVS, that's outside scope of libvirt. IOW, for vhost-user OVS it seems like that job is for Nova / os-vif to solve. This brings up another tangentially related question I came up against last night - qemu now has an option to report the host's MTU to the guest for virtio and vhost-user interfaces, and Michal Privoznik recently pushed patches to set the MTU sent to the guest via an explicit <mtu size='n'/> in libvirt's interface config:
https://bugzilla.redhat.com/1408701
But it would be much nicer if libvirt could learn the MTU of [that stuff at the other end of the unix socket] without requiring intervention in libvirt's config. For example, I'm just now testing patches for tap-based interfaces (connecting to Linux host bridges or OVS switches) that query the current MTU of the bridge and report that to qemu; this eliminates the burden of configuring each interface of each guest individually (and changing that config in all those places if someone ever wants to change the MTU of the bridge).
As Dan says, though, libvirt's only interaction in the case of vhost-user is with the unix socket. Is there any way to learn what is the appropriate MTU from OVS in these cases? Or must Nova (or ovirt or some poor user) set that up in the libvirt config for every single interface? We could add commands for all kind of queries to the vhost-user
On Thu, Feb 02, 2017 at 11:47:57AM -0500, Laine Stump wrote: protocol. libvirt would have to learn the vhost-user protocol though. Interested?
I think it could be possible to query the MTU value from the OVS DB using its JSON RPC-like API, but this is something I haven't tried. I guess it would need to resolve the ovs interface from the vhost-user socket path.
Can people familiar with OVS confirm this is something possible?
Libvirt does not use OVS' JSON RPC yet (it'll be long way to go anyway). Therefore, for any OVS interaction it uses ovs-vsctl directly. Therefore:
ovs-vsctl get Interface ovsbr0 mtu
is what Laine is looking for. However, something fishy is happening here:
lisa ~ # ovs-vsctl set Interface ovsbr0 mtu=9000 lisa ~ # ovs-vsctl get Interface ovsbr0 mtu 1500 lisa ~ # ifconfig ovsbr0 mtu 9000 lisa ~ # ovs-vsctl get Interface ovsbr0 mtu 9000
(yes, Lisa Simpson)
But since we just want to query the bridge's MTU and not set it, we should be safe.
Except that (as I just noted in the BZ) libvirt doesn't know the name of the bridge in the case of vhost-user. It only knows the name of the socket. So maybe in this case the only solution is for Nova (which *does* know about the bridge) needs to query the MTU and populate libvirt's XML appropriately.
Hi, On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
Hi,
Few months ago, Michael reported a problem about migrating VMs relying on vhost-user between hosts supporting different backend versions: - Message-Id: <20161011173526-mutt-send-email-mst@kernel.org> - https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03026.html
The goal of this thread is to draft a proposal based on the outcomes of discussions with contributors of the different parties (DPDK/OVS /libvirt/...).
Thanks the first feedback. It seems to converge that this is Nova's role, but not Libvirt one to manage these versions from management tool layer. This change has has no impact from OVS perspective, same requirements apply. I am interested on OVS contributors feedback on the below design proposal. Especially, I would like to have your opinion on the best way for OVS to expose its supported versions: - Static file generated at build time from version table described below - Entries in the OVS DB - Dedicated tool listing strings from the version table described below For selecting the right version of the vhost-user backend, do you agree it should be done via a new parameter of the ovs-vsctl add-port command for dpdkvhostuser ports?
Problem statement: ==================
When migrating a VM from one host to another, the interfaces exposed by QEMU must stay unchanged in order to guarantee a successful migration. In the case of vhost-user interface, parameters like supported Virtio feature set, max number of queues, max vring sizes,... must remain compatible. Indeed, the frontend not being re-initialized, no renegotiation happens at migration time.
For example, we have a VM that runs on host A, which has its vhost-user backend advertising VIRTIO_F_RING_INDIRECT_DESC feature. Since the Guest also support this feature, it is successfully negotiated, and guest transmit packets using indirect descriptor tables, that the backend knows to handle. At some point, the VM is being migrated to host B, which runs an older version of the backend not supporting this VIRTIO_F_RING_INDIRECT_DESC feature. The migration would break, because the Guest still have the VIRTIO_F_RING_INDIRECT_DESC bit sets, and the virtqueue contains some decriptors pointing to indirect tables, that backend B doesn't know to handle. This is just an example about Virtio features compatibility, but other backend implementation details could cause other failures.
What we need is to be able to query the destination host's backend to ensure migration is possible. Also, we would need to query this statically, even before the VM is started, to be sure it could be migrated elsewhere for any reason.
...
Solution 3: Libvirt queries OVS for vhost backend version string: *OK* ======================================================================
The idea is to have a table of supported versions, associated to key/value pairs. Libvirt could query the list of supported versions strings for each hosts, and select the first common one among all hosts.
Then, libvirt would ask OVS to probe the vhost-user interfaces in the selected version (compatibility mode). For example host A runs OVS-2.7, and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode (e.g. with indirect descriptors disabled), which should be selected at vhost-user interface probe time.
Advantage of doing so is that libvirt does not need any update if new keys are introduced (i.e. it does not need to know how the new keys have to be handled), all these checks remain in OVS's vhost-user implementation.
Ideally, we would support per vhost-user interface compatibility mode, which may have an impact also on DPDK API, as the Virtio feature update API is global, and not per port.
- Implementation: -----------------
Goal here is just to illustrate this proposal, I'm sure you will have good suggestion to improve it. In OVS vhost-user library, we would introduce a new structure, for example (neither compiled nor tested):
struct vhostuser_compat { char *version; uint64_t virtio_features; uint32_t max_rx_queue_sz; uint32_t max_nr_queues; };
*version* field is the compatibility version string. It could be something like: "upstream.ovs-dpdk.v2.6" In case for example Fedora adds some more patches to its package that would break migration to upstream version, it could have a dedicated compatibility string: "fc26.ovs-dpdk.v2.6". In case OVS-v2.7 does not break compatibility with previous OVS-v2.6 version, then no need to create a new compatibility entry, just keep v2.6 one.
*virtio_features* field is the Virtio features set for a given compatibility version. When an OVS tag is to be created, it would be associated to a DPDK version. The Virtio features for these version would be stored in this field. It would allow to upgrade the DPDK package for example from v16.07 to v16.11 without breaking migration. In case the distribution wants to benefit from latests Virtio features, it would have to create a new entry to ensure migration won't be broken.
*max_rx_queue_sz* *max_nr_queues* fields are just here for example, don't think this is needed today. I just want to illustrate that we have to anticipate other parameters than the Virtio feature set, even if not necessary at the moment.
We create a table with different compatibility versions in OVS vhost-user lib:
static struct vhostuser_compat vu_compat[] = { { .version = "upstream.ovs-dpdk.v2.7", .virtio_features = 0x12045694, .max_rx_queue_sz = 512, }, { .version = "upstream.ovs-dpdk.v2.6", .virtio_features = 0x10045694, .max_rx_queue_sz = 1024, }, }
At some time during installation, or system init, the table would be parsed, and compatibility version strings would be stored into the OVS database, or a new tool would be created to list these strings.
Before launching the VM, libvirt will query the version strings for each hosts using for example the JSON RPC API of OVS (maybe not the best solution, looking forward for your comments on this). Libvirt would then select the first common supported version, and insert this string into the vhost-user interfaces parameters in the OVS DBs of each host.
When the vhost-user connection is initiated, OVS would know in which compatibility mode to init the interface, for example by restricting the support Virtio features of the interface.
Do you think this is reasonable? Or maybe you have alternative ideas that would be best fit to ensure successful migration?
Thanks, Maxime
On Fri, Feb 03, 2017 at 03:11:10PM +0100, Maxime Coquelin wrote:
Hi,
On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
Hi,
Few months ago, Michael reported a problem about migrating VMs relying on vhost-user between hosts supporting different backend versions: - Message-Id: <20161011173526-mutt-send-email-mst@kernel.org> - https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03026.html
The goal of this thread is to draft a proposal based on the outcomes of discussions with contributors of the different parties (DPDK/OVS /libvirt/...).
Thanks the first feedback. It seems to converge that this is Nova's role, but not Libvirt one to manage these versions from management tool layer.
I think the conclusion is not that it should go up the stack. I think this will just get broken all the time. No one understands versions and stuff. Even QEMU developers get confused and break compatibility once in a while. My conclusion is that doing it from OVS side is wrong. Migration is not an OVS thing, it's a QEMU thing, and libvirt abstracts QEMU. People just want migration to work, ok? It's our job to do it, we do not really need a "make things work" flag. If libvirt does not want to use the vhost-user protocol (which sounds reasonable, it's rather complex) how about qemu providing a small utility to query the port? We could output json or whatever. This can help with MTU as well. And maybe it will help with nowait support - if someone uses the utility to dump backend config once, QEMU can later start the device without feature queries.
This change has has no impact from OVS perspective, same requirements apply. I am interested on OVS contributors feedback on the below design proposal.
Especially, I would like to have your opinion on the best way for OVS to expose its supported versions: - Static file generated at build time from version table described below - Entries in the OVS DB - Dedicated tool listing strings from the version table described below
For selecting the right version of the vhost-user backend, do you agree it should be done via a new parameter of the ovs-vsctl add-port command for dpdkvhostuser ports?
Problem statement: ==================
When migrating a VM from one host to another, the interfaces exposed by QEMU must stay unchanged in order to guarantee a successful migration. In the case of vhost-user interface, parameters like supported Virtio feature set, max number of queues, max vring sizes,... must remain compatible. Indeed, the frontend not being re-initialized, no renegotiation happens at migration time.
For example, we have a VM that runs on host A, which has its vhost-user backend advertising VIRTIO_F_RING_INDIRECT_DESC feature. Since the Guest also support this feature, it is successfully negotiated, and guest transmit packets using indirect descriptor tables, that the backend knows to handle. At some point, the VM is being migrated to host B, which runs an older version of the backend not supporting this VIRTIO_F_RING_INDIRECT_DESC feature. The migration would break, because the Guest still have the VIRTIO_F_RING_INDIRECT_DESC bit sets, and the virtqueue contains some decriptors pointing to indirect tables, that backend B doesn't know to handle. This is just an example about Virtio features compatibility, but other backend implementation details could cause other failures.
What we need is to be able to query the destination host's backend to ensure migration is possible. Also, we would need to query this statically, even before the VM is started, to be sure it could be migrated elsewhere for any reason.
...
Solution 3: Libvirt queries OVS for vhost backend version string: *OK* ======================================================================
The idea is to have a table of supported versions, associated to key/value pairs. Libvirt could query the list of supported versions strings for each hosts, and select the first common one among all hosts.
Then, libvirt would ask OVS to probe the vhost-user interfaces in the selected version (compatibility mode). For example host A runs OVS-2.7, and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode (e.g. with indirect descriptors disabled), which should be selected at vhost-user interface probe time.
Advantage of doing so is that libvirt does not need any update if new keys are introduced (i.e. it does not need to know how the new keys have to be handled), all these checks remain in OVS's vhost-user implementation.
Ideally, we would support per vhost-user interface compatibility mode, which may have an impact also on DPDK API, as the Virtio feature update API is global, and not per port.
- Implementation: -----------------
Goal here is just to illustrate this proposal, I'm sure you will have good suggestion to improve it. In OVS vhost-user library, we would introduce a new structure, for example (neither compiled nor tested):
struct vhostuser_compat { char *version; uint64_t virtio_features; uint32_t max_rx_queue_sz; uint32_t max_nr_queues; };
*version* field is the compatibility version string. It could be something like: "upstream.ovs-dpdk.v2.6" In case for example Fedora adds some more patches to its package that would break migration to upstream version, it could have a dedicated compatibility string: "fc26.ovs-dpdk.v2.6". In case OVS-v2.7 does not break compatibility with previous OVS-v2.6 version, then no need to create a new compatibility entry, just keep v2.6 one.
*virtio_features* field is the Virtio features set for a given compatibility version. When an OVS tag is to be created, it would be associated to a DPDK version. The Virtio features for these version would be stored in this field. It would allow to upgrade the DPDK package for example from v16.07 to v16.11 without breaking migration. In case the distribution wants to benefit from latests Virtio features, it would have to create a new entry to ensure migration won't be broken.
*max_rx_queue_sz* *max_nr_queues* fields are just here for example, don't think this is needed today. I just want to illustrate that we have to anticipate other parameters than the Virtio feature set, even if not necessary at the moment.
We create a table with different compatibility versions in OVS vhost-user lib:
static struct vhostuser_compat vu_compat[] = { { .version = "upstream.ovs-dpdk.v2.7", .virtio_features = 0x12045694, .max_rx_queue_sz = 512, }, { .version = "upstream.ovs-dpdk.v2.6", .virtio_features = 0x10045694, .max_rx_queue_sz = 1024, }, }
At some time during installation, or system init, the table would be parsed, and compatibility version strings would be stored into the OVS database, or a new tool would be created to list these strings.
Before launching the VM, libvirt will query the version strings for each hosts using for example the JSON RPC API of OVS (maybe not the best solution, looking forward for your comments on this). Libvirt would then select the first common supported version, and insert this string into the vhost-user interfaces parameters in the OVS DBs of each host.
When the vhost-user connection is initiated, OVS would know in which compatibility mode to init the interface, for example by restricting the support Virtio features of the interface.
Do you think this is reasonable? Or maybe you have alternative ideas that would be best fit to ensure successful migration?
Thanks, Maxime
On Fri, Feb 03, 2017 at 05:34:07PM +0200, Michael S. Tsirkin wrote:
On Fri, Feb 03, 2017 at 03:11:10PM +0100, Maxime Coquelin wrote:
Hi,
On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
Hi,
Few months ago, Michael reported a problem about migrating VMs relying on vhost-user between hosts supporting different backend versions: - Message-Id: <20161011173526-mutt-send-email-mst@kernel.org> - https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03026.html
The goal of this thread is to draft a proposal based on the outcomes of discussions with contributors of the different parties (DPDK/OVS /libvirt/...).
Thanks the first feedback. It seems to converge that this is Nova's role, but not Libvirt one to manage these versions from management tool layer.
I think the conclusion is not that it should go up the stack. I think this will just get broken all the time. No one understands versions and stuff. Even QEMU developers get confused and break compatibility once in a while.
I don't think it is really true, nor is it really specific to migration, it can hit any time you have a network connection that is re-opened even without migration. Version compatibility negotiations are an inherant part of any network protocol that is expected to evolve over time.
My conclusion is that doing it from OVS side is wrong. Migration is not an OVS thing, it's a QEMU thing, and libvirt abstracts QEMU. People just want migration to work, ok? It's our job to do it, we do not really need a "make things work" flag.
If libvirt does not want to use the vhost-user protocol (which sounds reasonable, it's rather complex) how about qemu providing a small utility to query the port? We could output json or whatever.
This can help with MTU as well.
And maybe it will help with nowait support - if someone uses the utility to dump backend config once, QEMU can later start the device without feature queries.
I don't think it is QEMU's job to deal with external components in this way and don't think QEMU vhost-user should be treated as special. This general scenario arises any time there is a QEMU backend is talking to an external service/process. For example, a virtio-serial talking to a daemon in the host. This daemon can support different versions of the protocol being spoken, so we have the same compat issue on migration. Or a traditional serial device, which the guest is using to talk to an external daemon the host. In a few cases we might know what the protocol is, but in general the data stream is opaque to QEMU. QEMU should not have need to learn about every single protocol that might be used to communicate with some external service. This is an unbounded set of possibilities to deal with, some of which will not even be open source. This all needs to be delegated to whatever mgmt app is responsible for setting up these external services, as it has the right domain knowledge about the applications being used, to make the policy decisions that are suitable for its usage scenario. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
On Fri, Feb 03, 2017 at 03:54:52PM +0000, Daniel P. Berrange wrote:
On Fri, Feb 03, 2017 at 05:34:07PM +0200, Michael S. Tsirkin wrote:
On Fri, Feb 03, 2017 at 03:11:10PM +0100, Maxime Coquelin wrote:
Hi,
On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
Hi,
Few months ago, Michael reported a problem about migrating VMs relying on vhost-user between hosts supporting different backend versions: - Message-Id: <20161011173526-mutt-send-email-mst@kernel.org> - https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03026.html
The goal of this thread is to draft a proposal based on the outcomes of discussions with contributors of the different parties (DPDK/OVS /libvirt/...).
Thanks the first feedback. It seems to converge that this is Nova's role, but not Libvirt one to manage these versions from management tool layer.
I think the conclusion is not that it should go up the stack. I think this will just get broken all the time. No one understands versions and stuff. Even QEMU developers get confused and break compatibility once in a while.
I don't think it is really true, nor is it really specific to migration, it can hit any time you have a network connection that is re-opened even without migration. Version compatibility negotiations are an inherant part of any network protocol that is expected to evolve over time.
My conclusion is that doing it from OVS side is wrong. Migration is not an OVS thing, it's a QEMU thing, and libvirt abstracts QEMU. People just want migration to work, ok? It's our job to do it, we do not really need a "make things work" flag.
If libvirt does not want to use the vhost-user protocol (which sounds reasonable, it's rather complex) how about qemu providing a small utility to query the port? We could output json or whatever.
This can help with MTU as well.
And maybe it will help with nowait support - if someone uses the utility to dump backend config once, QEMU can later start the device without feature queries.
I don't think it is QEMU's job to deal with external components in this way and don't think QEMU vhost-user should be treated as special.
It's very special because it's the only time we ship parts of device emulation out to an external service. So they need to be versioned together with QEMU.
This general scenario arises any time there is a QEMU backend is talking to an external service/process.
For example, a virtio-serial talking to a daemon in the host. This daemon can support different versions of the protocol being spoken, so we have the same compat issue on migration. Or a traditional serial device, which the guest is using to talk to an external daemon the host. In a few cases we might know what the protocol is, but in general the data stream is opaque to QEMU.
Are there any examples of people actually managing this across versions and getting it right? AFAIK there are cases where data stream is opaque to QEMU like -cpu host and in every such case VM is unmigrateable. OTOH vhost-user is not opaque to QEMU at all, and that is intentional so we can support migration.
QEMU should not have need to learn about every single protocol that might be used to communicate with some external service. This is an unbounded set of possibilities to deal with, some of which will not even be open source.
This all needs to be delegated to whatever mgmt app is responsible for setting up these external services, as it has the right domain knowledge about the applications being used, to make the policy decisions that are suitable for its usage scenario.
Regards, Daniel
I disagree it should not and in fact this is exactly what QEMU often does. This is how in the past we were mostly able to emulate any missing backend features in QEMU to make things transparent to guest. E.g. it can actually support the same virtio net device with userspace and kernel backends and migrate between them.
-- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
On 02/03/2017 04:34 PM, Michael S. Tsirkin wrote:
On Fri, Feb 03, 2017 at 03:11:10PM +0100, Maxime Coquelin wrote:
Hi,
On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
Hi,
Few months ago, Michael reported a problem about migrating VMs relying on vhost-user between hosts supporting different backend versions: - Message-Id: <20161011173526-mutt-send-email-mst@kernel.org> - https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03026.html
The goal of this thread is to draft a proposal based on the outcomes of discussions with contributors of the different parties (DPDK/OVS /libvirt/...).
Thanks the first feedback. It seems to converge that this is Nova's role, but not Libvirt one to manage these versions from management tool layer.
I think the conclusion is not that it should go up the stack. I think this will just get broken all the time. No one understands versions and stuff. Even QEMU developers get confused and break compatibility once in a while.
Is it that difficult for OVS to know with which versions of OVS it is compatible with, and propose a way to be compatible with older versions? From Nova perspective, it just has to collect supported versions strings, and select the most recent common one.
My conclusion is that doing it from OVS side is wrong. Migration is not an OVS thing, it's a QEMU thing, and libvirt abstracts QEMU. People just want migration to work, ok? It's our job to do it, we do not really need a "make things work" flag.
If libvirt does not want to use the vhost-user protocol (which sounds reasonable, it's rather complex) how about qemu providing a small utility to query the port? We could output json or whatever.
Doing this, it would be libvirt role to compare key/value pairs in the json output, right? If so, it would mean teaching libvirt to compare things it knows nothing about, that are likely to evolve over time. Also, it means the port will have already been created, so I'm not clear how we would manage compatibility version.
This can help with MTU as well.
And maybe it will help with nowait support - if someone uses the utility to dump backend config once, QEMU can later start the device without feature queries.
Regards, Maxime
participants (6)
-
Daniel P. Berrange -
Laine Stump -
Laine Stump -
Maxime Coquelin -
Michael S. Tsirkin -
Michal Privoznik