[libvirt] libvirk KVM save (migrate) terribly slow
Hi, I noticed that using libvirt to save running domain is terribly slow, I mean it can take even 10 minutes to save VM with 2GB RAM, although the host is almost idle (8CPU, 16GB RAM). If I understand correctly, libvirt first pauses the vm and then migrates it to file (optionally through gzip). Restoring it back to running state is almost instant. I guess this is not what should be expected, is there something particular I should check? Thanks a lot in advance with best regards nik -- ------------------------------------- Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis@linuxbox.cz -------------------------------------
On Thu, Dec 10, 2009 at 08:05:12AM +0100, Nikola Ciprich wrote:
Hi, I noticed that using libvirt to save running domain is terribly slow, I mean it can take even 10 minutes to save VM with 2GB RAM, although the host is almost idle (8CPU, 16GB RAM). If I understand correctly, libvirt first pauses the vm and then migrates it to file (optionally through gzip). Restoring it back to running state is almost instant. I guess this is not what should be expected, is there something particular I should check?
This matches behaviour that I see when saving VMs. QEMU takes a really very long time to save itself using migrate exec:. It is not CPU limited even in gzip mode - there is near zero CPU usage while this is going on. QEMU just seems to be very slow at sending the data. I've not had time to investigate whether this is a flaw in QEMU's exec: migration handling, or whether it is being hurt by too small pipe buffers, or something else entirely. I must say I'm not entirely convinced that it is a good idea for QEMU to be mixing use of FILE * / popen, with non-blocking I/O, but that could be a red-herring. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On 10 déc. 2009, at 12:44, Daniel P. Berrange wrote:
On Thu, Dec 10, 2009 at 08:05:12AM +0100, Nikola Ciprich wrote:
Hi, I noticed that using libvirt to save running domain is terribly slow, I mean it can take even 10 minutes to save VM with 2GB RAM, although the host is almost idle (8CPU, 16GB RAM). If I understand correctly, libvirt first pauses the vm and then migrates it to file (optionally through gzip). Restoring it back to running state is almost instant. I guess this is not what should be expected, is there something particular I should check?
This matches behaviour that I see when saving VMs. QEMU takes a really very long time to save itself using migrate exec:. It is not CPU limited even in gzip mode - there is near zero CPU usage while this is going on. QEMU just seems to be very slow at sending the data. I've not had time to investigate whether this is a flaw in QEMU's exec: migration handling, or whether it is being hurt by too small pipe buffers, or something else entirely. I must say I'm not entirely convinced that it is a good idea for QEMU to be mixing use of FILE * / popen, with non-blocking I/O, but that could be a red-herring.
Daniel
I've reported this issue back when the regression was introduced in qemu. Anthony had the same idea (not mixing popen with non-blocking I/O), but no solution was found at the time. I haven't tried recently (I'm using TCP migration only) but the problem must still be here. The thread on qemu-devel: http://lists.gnu.org/archive/html/qemu-devel/2009-08/msg01557.html http://lists.gnu.org/archive/html/qemu-devel/2009-09/msg00020.html -- Pierre Riteau -- http://perso.univ-rennes1.fr/pierre.riteau/
participants (3)
-
Daniel P. Berrange -
Nikola Ciprich -
Pierre Riteau