I've opened a bug to track this issue:
https://bugzilla.redhat.com/show_bug.cgi?id=666266
I have tried to work around the issue by making my migrations non-live,
hoping that the shorter downtime will make this less of a big deal. But
migrating busy domains with lots of RAM still takes a surprisingly long
time even if the domain is paused during migration -- something I did
not expect.
For instance, a domain with 8 GB of ram migrating over a gigabit network
takes about 20 seconds if the domain is idle and not using much ram.
However, if I start a run of 'stress' inside the VM which uses 6 gigs of
RAM in random malloc/free calls, the same domain might take 20 minutes
to migrate.
Has anyone else experienced this? Is there some kind of compression
going on when doing tunneled migrations which breaks down when the pages
are dirty?
--Igor
On Thu, Dec 23, 2010 at 06:13:35AM +1100, Justin Clift wrote:
On 23/12/2010, at 5:44 AM, Igor Serebryany wrote:
> On Mon, Dec 20, 2010 at 02:14:42PM +0100, Jiri Denemark wrote:
>> Ouch, I wonder if that could be the reason...
>
> Just to make sure that my strange compiling/packages aren't causing the
> problem, I've set up two servers running fedora 14 with the official
> libvirt packages. i experience the same problem -- while a vm is in
> migration, nothing else works. my traces reveal libvirt getting stuck at
> the same exact spot as on my debian machines.
This sounded familiar, so went looking in git history. I *think* this is
what was sounding familiar, from ~ libvirt 0.8.2:
commit 755b53f946107539ba6faf3022450b3d0bbe1d78
Author: Daniel P. Berrange <berrange(a)redhat.com>
Date: Thu Jun 24 19:15:27 2010 +0100
Avoid blocking all APIs during incoming migration
During incoming migration the QEMU monitor is not able to be
used. The incoming migration code did not keep hold of the
job lock because migration is split across multiple API calls.
This meant that further monitor commands on the guest would
hang until migration finished with no timeout.
In this change the qemuDomainMigratePrepare method sets the
job flag just before it returns. The qemuDomainMigrateFinish
method checks for this job flag & clears it once done. This
prevents any use of the monitor between prepare+finish steps.
The qemuDomainGetJobInfo method is also updated to refresh
the job elapsed time. This means that virsh domjobinfo can
return time data during incoming migration
* src/qemu/qemu_driver.c: Keep a job active during incoming
migration. Refresh job elapsed time when returning job info
Anyone know if this is actually relevant?
> i am desperate to get this bug fixed, and i don't think i have the
> resources to try to patch it myself. is there anything i can do to help
> you guys? more debugging info? foot massage?
Good idea. Foot massages would definitely be an incentive I
could be convinced with. :)
(it would probably help if I was the right kind of coder for this,
but I'm not)