Re: [libvirt-users] libvirt unavailable while a VM is in migration?]
On Mon, Dec 20, 2010 at 02:14:42PM +0100, Jiri Denemark wrote:
Ouch, I wonder if that could be the reason...
Just to make sure that my strange compiling/packages aren't causing the problem, I've set up two servers running fedora 14 with the official libvirt packages. i experience the same problem -- while a vm is in migration, nothing else works. my traces reveal libvirt getting stuck at the same exact spot as on my debian machines. at this point, i'm pretty sure that this is a bug in libvirt. Has libvirt ever had the ability to perform other operations while a qemu migration is in progress? If so than this is a regression, and a pretty serious one. For me, this makes live-migrations totally unusable, and since I cannot run my project without live-migrations and I cannot use any outside tools to do live-migrations when I'm using libvirt this makes libvirt totally unusable. i am desperate to get this bug fixed, and i don't think i have the resources to try to patch it myself. is there anything i can do to help you guys? more debugging info? foot massage? --Igor
On 23/12/2010, at 5:44 AM, Igor Serebryany wrote:
On Mon, Dec 20, 2010 at 02:14:42PM +0100, Jiri Denemark wrote:
Ouch, I wonder if that could be the reason...
Just to make sure that my strange compiling/packages aren't causing the problem, I've set up two servers running fedora 14 with the official libvirt packages. i experience the same problem -- while a vm is in migration, nothing else works. my traces reveal libvirt getting stuck at the same exact spot as on my debian machines.
This sounded familiar, so went looking in git history. I *think* this is what was sounding familiar, from ~ libvirt 0.8.2: commit 755b53f946107539ba6faf3022450b3d0bbe1d78 Author: Daniel P. Berrange <berrange@redhat.com> Date: Thu Jun 24 19:15:27 2010 +0100 Avoid blocking all APIs during incoming migration During incoming migration the QEMU monitor is not able to be used. The incoming migration code did not keep hold of the job lock because migration is split across multiple API calls. This meant that further monitor commands on the guest would hang until migration finished with no timeout. In this change the qemuDomainMigratePrepare method sets the job flag just before it returns. The qemuDomainMigrateFinish method checks for this job flag & clears it once done. This prevents any use of the monitor between prepare+finish steps. The qemuDomainGetJobInfo method is also updated to refresh the job elapsed time. This means that virsh domjobinfo can return time data during incoming migration * src/qemu/qemu_driver.c: Keep a job active during incoming migration. Refresh job elapsed time when returning job info Anyone know if this is actually relevant?
i am desperate to get this bug fixed, and i don't think i have the resources to try to patch it myself. is there anything i can do to help you guys? more debugging info? foot massage?
Good idea. Foot massages would definitely be an incentive I could be convinced with. :) (it would probably help if I was the right kind of coder for this, but I'm not)
I've opened a bug to track this issue: https://bugzilla.redhat.com/show_bug.cgi?id=666266 I have tried to work around the issue by making my migrations non-live, hoping that the shorter downtime will make this less of a big deal. But migrating busy domains with lots of RAM still takes a surprisingly long time even if the domain is paused during migration -- something I did not expect. For instance, a domain with 8 GB of ram migrating over a gigabit network takes about 20 seconds if the domain is idle and not using much ram. However, if I start a run of 'stress' inside the VM which uses 6 gigs of RAM in random malloc/free calls, the same domain might take 20 minutes to migrate. Has anyone else experienced this? Is there some kind of compression going on when doing tunneled migrations which breaks down when the pages are dirty? --Igor On Thu, Dec 23, 2010 at 06:13:35AM +1100, Justin Clift wrote:
On 23/12/2010, at 5:44 AM, Igor Serebryany wrote:
On Mon, Dec 20, 2010 at 02:14:42PM +0100, Jiri Denemark wrote:
Ouch, I wonder if that could be the reason...
Just to make sure that my strange compiling/packages aren't causing the problem, I've set up two servers running fedora 14 with the official libvirt packages. i experience the same problem -- while a vm is in migration, nothing else works. my traces reveal libvirt getting stuck at the same exact spot as on my debian machines.
This sounded familiar, so went looking in git history. I *think* this is what was sounding familiar, from ~ libvirt 0.8.2:
commit 755b53f946107539ba6faf3022450b3d0bbe1d78 Author: Daniel P. Berrange <berrange@redhat.com> Date: Thu Jun 24 19:15:27 2010 +0100
Avoid blocking all APIs during incoming migration
During incoming migration the QEMU monitor is not able to be used. The incoming migration code did not keep hold of the job lock because migration is split across multiple API calls. This meant that further monitor commands on the guest would hang until migration finished with no timeout.
In this change the qemuDomainMigratePrepare method sets the job flag just before it returns. The qemuDomainMigrateFinish method checks for this job flag & clears it once done. This prevents any use of the monitor between prepare+finish steps.
The qemuDomainGetJobInfo method is also updated to refresh the job elapsed time. This means that virsh domjobinfo can return time data during incoming migration
* src/qemu/qemu_driver.c: Keep a job active during incoming migration. Refresh job elapsed time when returning job info
Anyone know if this is actually relevant?
i am desperate to get this bug fixed, and i don't think i have the resources to try to patch it myself. is there anything i can do to help you guys? more debugging info? foot massage?
Good idea. Foot massages would definitely be an incentive I could be convinced with. :)
(it would probably help if I was the right kind of coder for this, but I'm not)
I recently had a chance to come back to this problem, and I'm sorry to say that it hasn't gotten any better. I tested on debian sid with a stock install of kvm 0.14 and libvirt 0.8.8-3 -- i'm still completely unable to interact with libvirt while an outgoing migration is in progress, although it seems as though actual migrations have gotten much faster. just an fyi. --Igor On Wed, Dec 29, 2010 at 05:06:12PM -0600, Igor Serebryany wrote:
I've opened a bug to track this issue: https://bugzilla.redhat.com/show_bug.cgi?id=666266
I have tried to work around the issue by making my migrations non-live, hoping that the shorter downtime will make this less of a big deal. But migrating busy domains with lots of RAM still takes a surprisingly long time even if the domain is paused during migration -- something I did not expect.
For instance, a domain with 8 GB of ram migrating over a gigabit network takes about 20 seconds if the domain is idle and not using much ram. However, if I start a run of 'stress' inside the VM which uses 6 gigs of RAM in random malloc/free calls, the same domain might take 20 minutes to migrate.
Has anyone else experienced this? Is there some kind of compression going on when doing tunneled migrations which breaks down when the pages are dirty?
--Igor
On Thu, Dec 23, 2010 at 06:13:35AM +1100, Justin Clift wrote:
On 23/12/2010, at 5:44 AM, Igor Serebryany wrote:
On Mon, Dec 20, 2010 at 02:14:42PM +0100, Jiri Denemark wrote:
Ouch, I wonder if that could be the reason...
Just to make sure that my strange compiling/packages aren't causing the problem, I've set up two servers running fedora 14 with the official libvirt packages. i experience the same problem -- while a vm is in migration, nothing else works. my traces reveal libvirt getting stuck at the same exact spot as on my debian machines.
This sounded familiar, so went looking in git history. I *think* this is what was sounding familiar, from ~ libvirt 0.8.2:
commit 755b53f946107539ba6faf3022450b3d0bbe1d78 Author: Daniel P. Berrange <berrange@redhat.com> Date: Thu Jun 24 19:15:27 2010 +0100
Avoid blocking all APIs during incoming migration
During incoming migration the QEMU monitor is not able to be used. The incoming migration code did not keep hold of the job lock because migration is split across multiple API calls. This meant that further monitor commands on the guest would hang until migration finished with no timeout.
In this change the qemuDomainMigratePrepare method sets the job flag just before it returns. The qemuDomainMigrateFinish method checks for this job flag & clears it once done. This prevents any use of the monitor between prepare+finish steps.
The qemuDomainGetJobInfo method is also updated to refresh the job elapsed time. This means that virsh domjobinfo can return time data during incoming migration
* src/qemu/qemu_driver.c: Keep a job active during incoming migration. Refresh job elapsed time when returning job info
Anyone know if this is actually relevant?
i am desperate to get this bug fixed, and i don't think i have the resources to try to patch it myself. is there anything i can do to help you guys? more debugging info? foot massage?
Good idea. Foot massages would definitely be an incentive I could be convinced with. :)
(it would probably help if I was the right kind of coder for this, but I'm not)
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
participants (2)
-
Igor Serebryany -
Justin Clift