[libvirt] Question about parallel migration connections
Are they supported with tunneled migration? The feature seems limited to native migration, in which case I can send a patch prohibiting parallel migration connections with the tunnel. Regards, Jim
On Thu, Jan 09, 2020 at 23:04:19 +0000, Jim Fehlig wrote:
Are they supported with tunneled migration? The feature seems limited to native migration, in which case I can send a patch prohibiting parallel migration connections with the tunnel.
That is true. Libvirt's stream which is used for tunelling the connection can carry only one connection and the APIs driving tunelled migration allow only one stream in arguments. It's the same reason why tunelled migration doesn't support migration of non-shared storage using NBD.
On 1/10/20 12:16 AM, Peter Krempa wrote:
On Thu, Jan 09, 2020 at 23:04:19 +0000, Jim Fehlig wrote:
Are they supported with tunneled migration? The feature seems limited to native migration, in which case I can send a patch prohibiting parallel migration connections with the tunnel.
That is true. Libvirt's stream which is used for tunelling the connection can carry only one connection and the APIs driving tunelled migration allow only one stream in arguments.
So should we check for and reject parallel+tunneled? I don't see any similar checks in the migration entry point, but suppose it would look something like the below patch (which I can formally send to the list if folks agree to such a check). Regards, Jim diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c index eb66999f07..a98287551e 100644 --- a/src/libvirt-domain.c +++ b/src/libvirt-domain.c @@ -3011,6 +3011,8 @@ virDomainMigrateVersion3Full(virDomainPtr domain, return NULL; } + VIR_EXCLUSIVE_FLAGS_RET(VIR_MIGRATE_TUNNELLED, VIR_MIGRATE_PARALLEL, NULL); + if (virTypedParamsCopy(&tmp, params, nparams) < 0) return NULL; params = tmp;
On Thu, Jan 09, 2020 at 11:04:19PM +0000, Jim Fehlig wrote:
Are they supported with tunneled migration? The feature seems limited to native migration, in which case I can send a patch prohibiting parallel migration connections with the tunnel.
Native migration should be preferred over tunneled migration these days. The tunneled migration feature was primarily a workaround for the lack of TLS support in QEMU, in order to leverage libvirtd's TLS connection. QEMU has support for TLS directly in its native migration protocol these days. That should be preserved as it provides a better performing data channel than tunnelling. This will especially be seen with parallel migration. Even if libvirt enabled parallel migration with tunnelling, libvirtd does all I/O in a single thread, so you wouldn't see any performance benefit from it, especially when TLS is used. This is actually true whether you've got a single QEMU with multiple TCP connections for migration, or multiple QEMU's migrating concurrently. Both situations will be limited by libvirt's single thread for I/O. With QEMU's native TLS support and parallell migration you'll be able to max out performance of many CPUs to maximise data throughput. The only real interesting benefit of tunnelled migration that remains is the fact that everything happens over a single TCP port, so there is less to open in the firewall. IMHO this is not compelling enough to offset the serious performance downsides of tunnelling, now that QEMJU has native TLS support. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 1/10/20 2:26 AM, Daniel P. Berrangé wrote:
On Thu, Jan 09, 2020 at 11:04:19PM +0000, Jim Fehlig wrote:
Are they supported with tunneled migration? The feature seems limited to native migration, in which case I can send a patch prohibiting parallel migration connections with the tunnel.
Native migration should be preferred over tunneled migration these days. The tunneled migration feature was primarily a workaround for the lack of TLS support in QEMU, in order to leverage libvirtd's TLS connection.
Tunneled serves the same purpose in the xen driver.
QEMU has support for TLS directly in its native migration protocol these days. That should be preserved as it provides a better performing data channel than tunnelling. This will especially be seen with parallel migration. Even if libvirt enabled parallel migration with tunnelling, libvirtd does all I/O in a single thread, so you wouldn't see any performance benefit from it, especially when TLS is used. This is actually true whether you've got a single QEMU with multiple TCP connections for migration, or multiple QEMU's migrating concurrently. Both situations will be limited by libvirt's single thread for I/O.
Nod.
With QEMU's native TLS support and parallell migration you'll be able to max out performance of many CPUs to maximise data throughput.
The docs on parallel migration are slim. What guidance should we provide wrt selecting a reasonable number of connections for parallel migration? Should users experiment to find a number that saturates the network link used for migration? AFAICT there are currently no bounds checks on the number. E.g. there is nothing preventing 'virsh migrate --parallel --parallel-connections 1000 ...'. Regards, Jim
The only real interesting benefit of tunnelled migration that remains is the fact that everything happens over a single TCP port, so there is less to open in the firewall. IMHO this is not compelling enough to offset the serious performance downsides of tunnelling, now that QEMJU has native TLS support.
Regards, Daniel
On Fri, Jan 10, 2020 at 02:22:00PM +0000, Jim Fehlig wrote:
On 1/10/20 2:26 AM, Daniel P. Berrangé wrote:
On Thu, Jan 09, 2020 at 11:04:19PM +0000, Jim Fehlig wrote:
Are they supported with tunneled migration? The feature seems limited to native migration, in which case I can send a patch prohibiting parallel migration connections with the tunnel.
Native migration should be preferred over tunneled migration these days. The tunneled migration feature was primarily a workaround for the lack of TLS support in QEMU, in order to leverage libvirtd's TLS connection.
Tunneled serves the same purpose in the xen driver.
QEMU has support for TLS directly in its native migration protocol these days. That should be preserved as it provides a better performing data channel than tunnelling. This will especially be seen with parallel migration. Even if libvirt enabled parallel migration with tunnelling, libvirtd does all I/O in a single thread, so you wouldn't see any performance benefit from it, especially when TLS is used. This is actually true whether you've got a single QEMU with multiple TCP connections for migration, or multiple QEMU's migrating concurrently. Both situations will be limited by libvirt's single thread for I/O.
Nod.
With QEMU's native TLS support and parallell migration you'll be able to max out performance of many CPUs to maximise data throughput.
The docs on parallel migration are slim. What guidance should we provide wrt selecting a reasonable number of connections for parallel migration? Should users experiment to find a number that saturates the network link used for migration? AFAICT there are currently no bounds checks on the number. E.g. there is nothing preventing 'virsh migrate --parallel --parallel-connections 1000 ...'.
This is a topical subject right now: https://lists.gnu.org/archive/html/qemu-devel/2020-01/msg00914.html If you are using TLS, then the upper limit on the number of connections you want is equal to the number of logical host CPUs. This is because when using TLS, you are probably going to max out CPU time on AES encryption, even when using AESNI acceleration. Of course if you have other guests running on the host, you likely dont want all CPUs being used for migration AES work. You also may not want to max out the network link, if you have other guests using that same link. So a real world upper limit on connections is going to be some subset of host CPUs TBD. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (3)
-
Daniel P. Berrangé -
Jim Fehlig -
Peter Krempa