5:04 p.m.
Are they supported with tunneled migration? The feature seems limited to native
migration, in which case I can send a patch prohibiting parallel migration
connections with the tunnel.
Regards,
Jim
1:16 a.m.
On Thu, Jan 09, 2020 at 23:04:19 +0000, Jim Fehlig wrote:
Are they supported with tunneled migration? The feature seems limited
to native
migration, in which case I can send a patch prohibiting parallel migration
connections with the tunnel.
That is true. Libvirt's stream which is used for tunelling the
connection can carry only one connection and the APIs driving tunelled
migration allow only one stream in arguments.
It's the same reason why tunelled migration doesn't support migration of
non-shared storage using NBD.
5:42 p.m.
On 1/10/20 12:16 AM, Peter Krempa wrote:
On Thu, Jan 09, 2020 at 23:04:19 +0000, Jim Fehlig wrote:
> Are they supported with tunneled migration? The feature seems limited to native
> migration, in which case I can send a patch prohibiting parallel migration
> connections with the tunnel.
That is true. Libvirt's stream which is used for tunelling the
connection can carry only one connection and the APIs driving tunelled
migration allow only one stream in arguments.
So should we check for and reject parallel+tunneled? I don't see any similar
checks in the migration entry point, but suppose it would look something like
the below patch (which I can formally send to the list if folks agree to such a
check).
Regards,
Jim
diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c
index eb66999f07..a98287551e 100644
--- a/src/libvirt-domain.c
+++ b/src/libvirt-domain.c
@@ -3011,6 +3011,8 @@ virDomainMigrateVersion3Full(virDomainPtr domain,
return NULL;
}
+ VIR_EXCLUSIVE_FLAGS_RET(VIR_MIGRATE_TUNNELLED, VIR_MIGRATE_PARALLEL, NULL);
+
if (virTypedParamsCopy(&tmp, params, nparams) < 0)
return NULL;
params = tmp;
3:26 a.m.
On Thu, Jan 09, 2020 at 11:04:19PM +0000, Jim Fehlig wrote:
Are they supported with tunneled migration? The feature seems limited
to native
migration, in which case I can send a patch prohibiting parallel migration
connections with the tunnel.
Native migration should be preferred over tunneled migration these days.
The tunneled migration feature was primarily a workaround for the lack
of TLS support in QEMU, in order to leverage libvirtd's TLS connection.
QEMU has support for TLS directly in its native migration protocol these
days. That should be preserved as it provides a better performing data
channel than tunnelling. This will especially be seen with parallel
migration. Even if libvirt enabled parallel migration with tunnelling,
libvirtd does all I/O in a single thread, so you wouldn't see any
performance benefit from it, especially when TLS is used. This is
actually true whether you've got a single QEMU with multiple TCP
connections for migration, or multiple QEMU's migrating concurrently.
Both situations will be limited by libvirt's single thread for I/O.
With QEMU's native TLS support and parallell migration you'll be able
to max out performance of many CPUs to maximise data throughput.
The only real interesting benefit of tunnelled migration that remains
is the fact that everything happens over a single TCP port, so there
is less to open in the firewall. IMHO this is not compelling enough
to offset the serious performance downsides of tunnelling, now that
QEMJU has native TLS support.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
8:22 a.m.
On 1/10/20 2:26 AM, Daniel P. Berrangé wrote:
On Thu, Jan 09, 2020 at 11:04:19PM +0000, Jim Fehlig wrote:
> Are they supported with tunneled migration? The feature seems limited to native
> migration, in which case I can send a patch prohibiting parallel migration
> connections with the tunnel.
Native migration should be preferred over tunneled migration these days.
The tunneled migration feature was primarily a workaround for the lack
of TLS support in QEMU, in order to leverage libvirtd's TLS connection.
Tunneled serves the same purpose in the xen driver.
QEMU has support for TLS directly in its native migration protocol
these
days. That should be preserved as it provides a better performing data
channel than tunnelling. This will especially be seen with parallel
migration. Even if libvirt enabled parallel migration with tunnelling,
libvirtd does all I/O in a single thread, so you wouldn't see any
performance benefit from it, especially when TLS is used. This is
actually true whether you've got a single QEMU with multiple TCP
connections for migration, or multiple QEMU's migrating concurrently.
Both situations will be limited by libvirt's single thread for I/O.
Nod.
With QEMU's native TLS support and parallell migration you'll
be able
to max out performance of many CPUs to maximise data throughput.
The docs on parallel migration are slim. What guidance should we provide wrt
selecting a reasonable number of connections for parallel migration? Should
users experiment to find a number that saturates the network link used for
migration? AFAICT there are currently no bounds checks on the number. E.g. there
is nothing preventing 'virsh migrate --parallel --parallel-connections 1000 ...'.
Regards,
Jim
The only real interesting benefit of tunnelled migration that
remains
is the fact that everything happens over a single TCP port, so there
is less to open in the firewall. IMHO this is not compelling enough
to offset the serious performance downsides of tunnelling, now that
QEMJU has native TLS support.
Regards,
Daniel
8:48 a.m.
On Fri, Jan 10, 2020 at 02:22:00PM +0000, Jim Fehlig wrote:
On 1/10/20 2:26 AM, Daniel P. Berrangé wrote:
> On Thu, Jan 09, 2020 at 11:04:19PM +0000, Jim Fehlig wrote:
>> Are they supported with tunneled migration? The feature seems limited to native
>> migration, in which case I can send a patch prohibiting parallel migration
>> connections with the tunnel.
>
> Native migration should be preferred over tunneled migration these days.
> The tunneled migration feature was primarily a workaround for the lack
> of TLS support in QEMU, in order to leverage libvirtd's TLS connection.
Tunneled serves the same purpose in the xen driver.
> QEMU has support for TLS directly in its native migration protocol these
> days. That should be preserved as it provides a better performing data
> channel than tunnelling. This will especially be seen with parallel
> migration. Even if libvirt enabled parallel migration with tunnelling,
> libvirtd does all I/O in a single thread, so you wouldn't see any
> performance benefit from it, especially when TLS is used. This is
> actually true whether you've got a single QEMU with multiple TCP
> connections for migration, or multiple QEMU's migrating concurrently.
> Both situations will be limited by libvirt's single thread for I/O.
Nod.
> With QEMU's native TLS support and parallell migration you'll be able
> to max out performance of many CPUs to maximise data throughput.
The docs on parallel migration are slim. What guidance should we provide wrt
selecting a reasonable number of connections for parallel migration? Should
users experiment to find a number that saturates the network link used for
migration? AFAICT there are currently no bounds checks on the number. E.g. there
is nothing preventing 'virsh migrate --parallel --parallel-connections 1000 ...'.
This is a topical subject right now:
https://lists.gnu.org/archive/html/qemu-devel/2020-01/msg00914.html
If you are using TLS, then the upper limit on the number of connections
you want is equal to the number of logical host CPUs. This is because
when using TLS, you are probably going to max out CPU time on AES
encryption, even when using AESNI acceleration.
Of course if you have other guests running on the host, you likely dont
want all CPUs being used for migration AES work. You also may not want
to max out the network link, if you have other guests using that same
link.
So a real world upper limit on connections is going to be some subset of
host CPUs TBD.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
1782
days inactive
1783
days old
5 comments
3 participants
participants (3)
-
Daniel P. Berrangé -
Jim Fehlig -
Peter Krempa