[libvirt] libvirt(-java): virDomainMigrateSetMaxDowntime
Hi, I'm facing some troubles with virDomainMigrate & virDomainMigrateSetMaxDowntime. The core problem is that KVM's default value for the maximum allowed downtime is 30ms (max_downtime in migration.c, it's nanoseconds there; 0.12.3) which is too low for my VMs when they're busy (~50% CPU util and above). Migrations then take literally forever, I had to abort them after 15 minutes or so. I'm using GBit Ethernet, so plenty bandwidth should be available. Increasing the allowed downtime to 50ms seems to help, but I have not tested situations where the VM is completely utilized. Anyways, the default value is too low for me, so I tried virDomainMigrateSetMaxDowntime resp. the Java wrapper function. Here I'm facing a problem I can overcome only with a quite crude hack: org.libvirt.Domain.migrate(..) blocks until the migration is done, which is of course reasonable. So I tried calling migrateSetMaxDowntime(..) before migrating, causing an error: "Requested operation is not valid: domain is not being migrated" This tells me that calling migrateSetMaxDowntime is only allowed during migrations. As I'm migrating VMs automatically and without any user intervention I'd need to create some glue code that runs in an extra thread, waiting "some time" hoping that the migration was kicked off in the main thread yet and then calling migrateSetMaxDowntime. I'd like to avoid such quirks in the long run, if possible. So my question: Would it be possible to extend the migrate() method resp. virDomainMigrate() function with an optional maxDowntime parameter that is passed down as QEMU_JOB_SIGNAL_MIGRATE_DOWNTIME so that qemuDomainWaitForMigrationComplete would set the value? Or are there easier ways? Thanks and regards, -t
On Tue, Jul 13, 2010 at 06:56:53PM +0200, Thomas Treutner wrote:
Hi,
I'm facing some troubles with virDomainMigrate & virDomainMigrateSetMaxDowntime. The core problem is that KVM's default value for the maximum allowed downtime is 30ms (max_downtime in migration.c, it's nanoseconds there; 0.12.3) which is too low for my VMs when they're busy (~50% CPU util and above). Migrations then take literally forever, I had to abort them after 15 minutes or so. I'm using GBit Ethernet, so plenty bandwidth should be available. Increasing the allowed downtime to 50ms seems to help, but I have not tested situations where the VM is completely utilized. Anyways, the default value is too low for me, so I tried virDomainMigrateSetMaxDowntime resp. the Java wrapper function.
Here I'm facing a problem I can overcome only with a quite crude hack: org.libvirt.Domain.migrate(..) blocks until the migration is done, which is of course reasonable. So I tried calling migrateSetMaxDowntime(..) before migrating, causing an error:
"Requested operation is not valid: domain is not being migrated"
This tells me that calling migrateSetMaxDowntime is only allowed during migrations. As I'm migrating VMs automatically and without any user intervention I'd need to create some glue code that runs in an extra thread, waiting "some time" hoping that the migration was kicked off in the main thread yet and then calling migrateSetMaxDowntime. I'd like to avoid such quirks in the long run, if possible.
Multiple threads is our recommended approach to the problem, since it is a general solution. eg you can call virDomainSuspend to pause the guest during migration & thus let it complete non-live. And virDomainGetJobInfo to check progress. And virDomainAbortJob to cancel.
So my question: Would it be possible to extend the migrate() method resp. virDomainMigrate() function with an optional maxDowntime parameter that is passed down as QEMU_JOB_SIGNAL_MIGRATE_DOWNTIME so that qemuDomainWaitForMigrationComplete would set the value? Or are there easier ways?
That approach really desirable IMHO, because it is already possible todo this using threads, which is already neccessary for the other APIs you can invoke during migration. If you care about the max downtime parameter, then you almost certainly need to care about calling virDomainGetJobInfo() in order to determine whether the guest is actually progressing during migration or not. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On 07/13/2010 01:12 PM, Daniel P. Berrange wrote:
On Tue, Jul 13, 2010 at 06:56:53PM +0200, Thomas Treutner wrote:
Hi,
I'm facing some troubles with virDomainMigrate & virDomainMigrateSetMaxDowntime. The core problem is that KVM's default value for the maximum allowed downtime is 30ms (max_downtime in migration.c, it's nanoseconds there; 0.12.3) which is too low for my VMs when they're busy (~50% CPU util and above). Migrations then take literally forever, I had to abort them after 15 minutes or so. I'm using GBit Ethernet, so plenty bandwidth should be available. Increasing the allowed downtime to 50ms seems to help, but I have not tested situations where the VM is completely utilized. Anyways, the default value is too low for me, so I tried virDomainMigrateSetMaxDowntime resp. the Java wrapper function.
Here I'm facing a problem I can overcome only with a quite crude hack: org.libvirt.Domain.migrate(..) blocks until the migration is done, which is of course reasonable. So I tried calling migrateSetMaxDowntime(..) before migrating, causing an error:
"Requested operation is not valid: domain is not being migrated"
This tells me that calling migrateSetMaxDowntime is only allowed during migrations. As I'm migrating VMs automatically and without any user intervention I'd need to create some glue code that runs in an extra thread, waiting "some time" hoping that the migration was kicked off in the main thread yet and then calling migrateSetMaxDowntime. I'd like to avoid such quirks in the long run, if possible.
Multiple threads is our recommended approach to the problem, since it is a general solution. eg you can call virDomainSuspend to pause the guest during migration & thus let it complete non-live. And virDomainGetJobInfo to check progress. And virDomainAbortJob to cancel.
So my question: Would it be possible to extend the migrate() method resp. virDomainMigrate() function with an optional maxDowntime parameter that is passed down as QEMU_JOB_SIGNAL_MIGRATE_DOWNTIME so that qemuDomainWaitForMigrationComplete would set the value? Or are there easier ways?
That approach really desirable IMHO, because it is already possible todo this using threads, which is already neccessary for the other APIs you can invoke during migration. If you care about the max downtime parameter, then you almost certainly need to care about calling virDomainGetJobInfo() in order to determine whether the guest is actually progressing during migration or not.
Also sounds like it would be handy to allow globally configuring the default migration downtime in /etc/libvirt/qemu.conf - Cole
[forgot list] On 07/13/2010 08:21 PM, Cole Robinson wrote:
On 07/13/2010 01:12 PM, Daniel P. Berrange wrote:
On Tue, Jul 13, 2010 at 06:56:53PM +0200, Thomas Treutner wrote:
So my question: Would it be possible to extend the migrate() method resp. virDomainMigrate() function with an optional maxDowntime parameter that is passed down as QEMU_JOB_SIGNAL_MIGRATE_DOWNTIME so that qemuDomainWaitForMigrationComplete would set the value? Or are there easier ways?
That approach really desirable IMHO, because it is already possible todo this using threads, which is already neccessary for the other APIs you can invoke during migration. If you care about the max downtime parameter, then you almost certainly need to care about calling virDomainGetJobInfo() in order to determine whether the guest is actually progressing during migration or not.
Also sounds like it would be handy to allow globally configuring the default migration downtime in /etc/libvirt/qemu.conf
Yep, that would be at least something. I'd even doubt that the qemu default value makes sense at all, in more than one way: 1) When I set 100ms, the last jobinfo I see says approx. 80MByte remaining. Over Gbit Ethernet, it would take about 0.5s to transfer that, in the ideal case. A flooding ping says about 800ms max delay. Maybe there's a bit/Byte-bug lurking around somewhere in qemu? (100ms vs. 800ms) 2) Even if there's no bug, the 30ms look far too optimistic to me. As I said, unless the VM is only moderately loaded (<50%), migrations never finish and would run forever, I assume. I suggest that the default value should be higher and/or some action should be taken if the migration can not be done within some time or iteration limit (IIRC Xen takes the iteration approach). One can argue whether the action should be abortion or just-do-it, but I think there should be definitively *some* action to have a sane default without tracking a migration process in each and every software built on libvirt/qemu. (Yes, it's primarily a qemu issue, but there are lots of other optional, driver-specific things in libvirt too). I'd find it even more interesting to have the downtime as a part of the domain config, with choices/values for timeouts and the respective action, but that's a lot of wishes and would need hypervisor support. regards, -t
On Tue, Jul 13, 2010 at 09:23:37PM +0200, Thomas Treutner wrote:
[forgot list]
On 07/13/2010 08:21 PM, Cole Robinson wrote:
On 07/13/2010 01:12 PM, Daniel P. Berrange wrote:
On Tue, Jul 13, 2010 at 06:56:53PM +0200, Thomas Treutner wrote:
So my question: Would it be possible to extend the migrate() method resp. virDomainMigrate() function with an optional maxDowntime parameter that is passed down as QEMU_JOB_SIGNAL_MIGRATE_DOWNTIME so that qemuDomainWaitForMigrationComplete would set the value? Or are there easier ways?
That approach really desirable IMHO, because it is already possible todo this using threads, which is already neccessary for the other APIs you can invoke during migration. If you care about the max downtime parameter, then you almost certainly need to care about calling virDomainGetJobInfo() in order to determine whether the guest is actually progressing during migration or not.
Also sounds like it would be handy to allow globally configuring the default migration downtime in /etc/libvirt/qemu.conf
Yep, that would be at least something. I'd even doubt that the qemu default value makes sense at all, in more than one way:
1) When I set 100ms, the last jobinfo I see says approx. 80MByte remaining. Over Gbit Ethernet, it would take about 0.5s to transfer that, in the ideal case. A flooding ping says about 800ms max delay. Maybe there's a bit/Byte-bug lurking around somewhere in qemu? (100ms vs. 800ms)
2) Even if there's no bug, the 30ms look far too optimistic to me. As I said, unless the VM is only moderately loaded (<50%), migrations never finish and would run forever, I assume. I suggest that the default value should be higher and/or some action should be taken if the migration can not be done within some time or iteration limit (IIRC Xen takes the iteration approach). One can argue whether the action should be abortion or just-do-it, but I think there should be definitively *some* action to have a sane default without tracking a migration process in each and every software built on libvirt/qemu. (Yes, it's primarily a qemu issue, but there are lots of other optional, driver-specific things in libvirt too).
People have argued for the iterative + adaptive approach Xen uses to be applied in upstream QEMU, but the community has rejected it as a policy decision that belongs in the mgmt apps. Thus they only provide the progress info, migrate downtime tunable & the pause option and the ability to cancel a migration. libvirt then exposes these & its upto apps to dynamically manage it. IMHO this rather sucks for apps which don't care about this level of flexibility & would rather it 'just works' like Xen, but this discussion has been had & lost many times on qemu-devel now :-( Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On 07/13/2010 07:12 PM, Daniel P. Berrange wrote:
This tells me that calling migrateSetMaxDowntime is only allowed during migrations. As I'm migrating VMs automatically and without any user intervention I'd need to create some glue code that runs in an extra thread, waiting "some time" hoping that the migration was kicked off in the main thread yet and then calling migrateSetMaxDowntime. I'd like to avoid such quirks in the long run, if possible.
Multiple threads is our recommended approach to the problem, since it is a general solution. eg you can call virDomainSuspend to pause the guest during migration& thus let it complete non-live. And virDomainGetJobInfo to check progress. And virDomainAbortJob to cancel.
Sorry to reanimate this zombie, but I think this approach isn't really doable in a clean way, but maybe I'm missing something. I've started a new thread with Message-ID: <4E709C1E.1030408@scripty.at> here. Maybe you could give your highly appreciated 0.02 $/€ ... I've also sent a fix to the under-underlying issue, which was much more easy for me both than I thought and than getting multiple threads done right here (I think one can't, but see above etc.). Anyways, chances of acceptance are not too high, eventually. One can't argue with politics. http://lists.gnu.org/archive/html/qemu-devel/2011-09/msg01754.html regards, thomas
participants (3)
-
Cole Robinson -
Daniel P. Berrange -
Thomas Treutner