On Mon, May 07, 2018 at 11:55:14AM -0400, Shawn Q wrote: > Hi folk, we are using 10gb NICs with multithreaded compression. > > We're finding that the standard `virsh migrate` gets at most ~1.2gbps, > similar to a single scp session. Hmm, I didn't actively measure the throughput when I tested, but looking at the results and trying to infer bandwidth, I'm fairly sure I saw in excess of 1.2gbps. With scp I would expect transfer rate to be limited by the speed your CPUS can perform AES encryption. Your migration command is just transferring plaintext so should not be limited in that way and should thus get better results. I wonder if the compression code is harming you because that burns massive amounts of CPU time, often for minimal compression benefit. > When we do a multipart upload with multiple scp connections we can squeeze > as high as 9.6gbps. > > Is there was a way to get `virsh migrate` to perform multiple connections > as well when transferring? There is work underway in QEMU to add this feature, but it will be a while before it is ready. > Would be useful to be able to migrate big guests using the full capacity of > the 10gb nics. > > Our example command to migrate: > > # virsh migrate --compressed --comp-methods mt --comp-mt-level 9 > --comp-mt-threads 16 --comp-mt-dthreads 16 --verbose --live > --copy-storage-all --undefinesource --unsafe --persistent > dev-testvm790.mydomain.net qemu+tcp://myhost-10g.mydomain.net/system I would caution that the compression code should not be assumed to be beneficial. When I tested compression against extreme guest workloads, it was actually harmful[1] - it burns a lot of CPU time, which takes significant time away from the guest OS vCPUs, while not getting very good compression. If you have trouble with migration not completing, the best advice at this time is to use the post-copy migration method. This will guarantee that migration can complete in finite time, while having the lower impact on guest vCPUs, assuming you let it run pre-copy to copy bulk of RAM, before flipping to post-copy. Regards, Daniel [1] https://www.berrange.com/posts/2016/05/12/analysis-of- techniques-for-ensuring-migration-completion-with-kvm/ -- |: https://berrange.com -o- https://www.flickr.com/photos/ dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/ dberrange :|