[libvirt] [PATCH] Set qemu migration speed unlimited when migrating to file
Discussed previously: https://www.redhat.com/archives/libvir-list/2011-August/msg00166.html The qemu migration speed default is 32MiB/s as defined in migration.c /* Migration speed throttling */ static int64_t max_throttle = (32 << 20); There is no reason to throttle migration when targeting a file. For dump and save operations, set migration speed to unlimited prior to migration and restore to default value after migration. Default units is MB for migrate_set_speed monitor command, so (INT64_MAX / (1024 * 1024)) is used for unlimited migration speed. Tested with both json and text monitors. --- src/qemu/qemu_migration.c | 12 ++++++++++++ 1 files changed, 12 insertions(+), 0 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 7aeea69..4542289 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2676,6 +2676,13 @@ qemuMigrationToFile(struct qemud_driver *driver, virDomainObjPtr vm, virCommandPtr cmd = NULL; int pipeFD[2] = { -1, -1 }; + /* No need for qemu default of 32MiB/s when migrating to a file. + Default speed unit is MB, so set to unlimited with INT64_MAX / 1M. + Failure to change migration speed is not fatal. */ + qemuDomainObjEnterMonitor(driver, vm); + qemuMonitorSetMigrationSpeed(priv->mon, INT64_MAX / (1024 * 1024)); + qemuDomainObjExitMonitor(driver, vm); + if (qemuCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATE_QEMU_FD) && (!compressor || pipe(pipeFD) == 0)) { /* All right! We can use fd migration, which means that qemu @@ -2783,6 +2790,11 @@ qemuMigrationToFile(struct qemud_driver *driver, virDomainObjPtr vm, ret = 0; cleanup: + /* Restore migration speed to 32MiB/s default */ + qemuDomainObjEnterMonitor(driver, vm); + qemuMonitorSetMigrationSpeed(priv->mon, (32 << 20)); + qemuDomainObjExitMonitor(driver, vm); + VIR_FORCE_CLOSE(pipeFD[0]); VIR_FORCE_CLOSE(pipeFD[1]); virCommandFree(cmd); -- 1.7.5.4
On 08/04/2011 10:22 AM, Jim Fehlig wrote:
Discussed previously:
https://www.redhat.com/archives/libvir-list/2011-August/msg00166.html
The qemu migration speed default is 32MiB/s as defined in migration.c
/* Migration speed throttling */ static int64_t max_throttle = (32<< 20);
There is no reason to throttle migration when targeting a file. For dump and save operations, set migration speed to unlimited prior to migration and restore to default value after migration. Default units is MB for migrate_set_speed monitor command, so (INT64_MAX / (1024 * 1024)) is used for unlimited migration speed.
Tested with both json and text monitors. --- src/qemu/qemu_migration.c | 12 ++++++++++++ 1 files changed, 12 insertions(+), 0 deletions(-)
Nice idea, but incomplete patch.
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 7aeea69..4542289 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2676,6 +2676,13 @@ qemuMigrationToFile(struct qemud_driver *driver, virDomainObjPtr vm, virCommandPtr cmd = NULL; int pipeFD[2] = { -1, -1 };
+ /* No need for qemu default of 32MiB/s when migrating to a file. + Default speed unit is MB, so set to unlimited with INT64_MAX / 1M. + Failure to change migration speed is not fatal. */ + qemuDomainObjEnterMonitor(driver, vm); + qemuMonitorSetMigrationSpeed(priv->mon, INT64_MAX / (1024 * 1024)); + qemuDomainObjExitMonitor(driver, vm); + if (qemuCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATE_QEMU_FD)&&
Missing a check that the domain is still active after you exit the monitor (in both places). Furthermore, since migrationToFile is only used inside async jobs, we should be using qemuDomainObjEnterMonitorAsync, not qemuDomainObjEnterMonitor (fixing that will add in the check for a still-active vm, but then you have to be careful to only call qemuDomainObjExitMonitorWithDriver if you successfully entered the monitor). -- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org
Discussed previously: https://www.redhat.com/archives/libvir-list/2011-August/msg00166.html The qemu migration speed default is 32MiB/s as defined in migration.c /* Migration speed throttling */ static int64_t max_throttle = (32 << 20); There is no reason to throttle migration when targeting a file. For dump and save operations, set migration speed to unlimited prior to migration and restore to default value after migration. Default units is MB for migrate_set_speed monitor command, so (INT64_MAX / (1024 * 1024)) is used for unlimited migration speed. Tested with both json and text monitors. V2: - Use qemuDomainObjEnterMonitorAsync() instead of qemuDomainObjEnterMonitor() - Check return status of qemuDomainObjEnterMonitorAsync() --- src/qemu/qemu_migration.c | 14 ++++++++++++++ 1 files changed, 14 insertions(+), 0 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 7aeea69..fa05ccf 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2676,6 +2676,14 @@ qemuMigrationToFile(struct qemud_driver *driver, virDomainObjPtr vm, virCommandPtr cmd = NULL; int pipeFD[2] = { -1, -1 }; + /* No need for qemu default of 32MiB/s when migrating to a file. + Default speed unit is MB, so set to unlimited with INT64_MAX / 1M. + Failure to change migration speed is not fatal. */ + if (qemuDomainObjEnterMonitorAsync(driver, vm, asyncJob) == 0) { + qemuMonitorSetMigrationSpeed(priv->mon, INT64_MAX / (1024 * 1024)); + qemuDomainObjExitMonitorWithDriver(driver, vm); + } + if (qemuCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATE_QEMU_FD) && (!compressor || pipe(pipeFD) == 0)) { /* All right! We can use fd migration, which means that qemu @@ -2783,6 +2791,12 @@ qemuMigrationToFile(struct qemud_driver *driver, virDomainObjPtr vm, ret = 0; cleanup: + /* Restore migration speed to 32MiB/s default */ + if (qemuDomainObjEnterMonitorAsync(driver, vm, asyncJob) == 0) { + qemuMonitorSetMigrationSpeed(priv->mon, (32 << 20)); + qemuDomainObjExitMonitorWithDriver(driver, vm); + } + VIR_FORCE_CLOSE(pipeFD[0]); VIR_FORCE_CLOSE(pipeFD[1]); virCommandFree(cmd); -- 1.7.5.4
On 08/04/2011 11:35 AM, Jim Fehlig wrote:
Discussed previously:
https://www.redhat.com/archives/libvir-list/2011-August/msg00166.html
The qemu migration speed default is 32MiB/s as defined in migration.c
/* Migration speed throttling */ static int64_t max_throttle = (32<< 20);
Too bad it's not publicly exposed in a header file, in case it ever changes.
There is no reason to throttle migration when targeting a file. For dump and save operations, set migration speed to unlimited prior to migration and restore to default value after migration. Default units is MB for migrate_set_speed monitor command, so (INT64_MAX / (1024 * 1024)) is used for unlimited migration speed.
Tested with both json and text monitors.
+++ b/src/qemu/qemu_migration.c @@ -2676,6 +2676,14 @@ qemuMigrationToFile(struct qemud_driver *driver, virDomainObjPtr vm, virCommandPtr cmd = NULL; int pipeFD[2] = { -1, -1 };
+ /* No need for qemu default of 32MiB/s when migrating to a file. + Default speed unit is MB, so set to unlimited with INT64_MAX / 1M. + Failure to change migration speed is not fatal. */ + if (qemuDomainObjEnterMonitorAsync(driver, vm, asyncJob) == 0) { + qemuMonitorSetMigrationSpeed(priv->mon, INT64_MAX / (1024 * 1024)); + qemuDomainObjExitMonitorWithDriver(driver, vm); + } +
This part is now fine.
if (qemuCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATE_QEMU_FD)&& (!compressor || pipe(pipeFD) == 0)) { /* All right! We can use fd migration, which means that qemu @@ -2783,6 +2791,12 @@ qemuMigrationToFile(struct qemud_driver *driver, virDomainObjPtr vm, ret = 0;
cleanup: + /* Restore migration speed to 32MiB/s default */ + if (qemuDomainObjEnterMonitorAsync(driver, vm, asyncJob) == 0) { + qemuMonitorSetMigrationSpeed(priv->mon, (32<< 20));
If we keep this hunk, I'd like to see this magic number as a #define earlier in the file, with the same justification as you gave in your commit comment about where it comes from (32Mbps in qemu's migration.c), so it becomes easier to replace the number and/or consistently use it elsewhere in the code if qemu ever changes.
+ qemuDomainObjExitMonitorWithDriver(driver, vm); + }
I was first worried that this part means we have more issues. That is, the user can independently call virDomainMigrateSetMaxSpeed, and it seems like we should revert back to that value, rather than to the qemu default. But on further thought - guess what? After you complete migration to a file, the qemu process is (supposed to have) ended! There's no reason to restore migration speed after this point, because there's nothing further you can do with the qemu process; once the domain is later restored from the save file, you have created a new qemu process which is once again back at the default migration speed. That means we can effectively drop this hunk with no change in behavior. Meanwhile, it raises independent issues - why do we have a write-only interface in virDomainMigrateSetMaxSpeed? Shouldn't we also be able to query the speed currently in use, and shouldn't the domain XML track the current migration speed? -- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org
Eric Blake wrote:
On 08/04/2011 11:35 AM, Jim Fehlig wrote:
Discussed previously:
https://www.redhat.com/archives/libvir-list/2011-August/msg00166.html
The qemu migration speed default is 32MiB/s as defined in migration.c
/* Migration speed throttling */ static int64_t max_throttle = (32<< 20);
Too bad it's not publicly exposed in a header file, in case it ever changes.
There is no reason to throttle migration when targeting a file. For dump and save operations, set migration speed to unlimited prior to migration and restore to default value after migration. Default units is MB for migrate_set_speed monitor command, so (INT64_MAX / (1024 * 1024)) is used for unlimited migration speed.
Tested with both json and text monitors.
+++ b/src/qemu/qemu_migration.c @@ -2676,6 +2676,14 @@ qemuMigrationToFile(struct qemud_driver *driver, virDomainObjPtr vm, virCommandPtr cmd = NULL; int pipeFD[2] = { -1, -1 };
+ /* No need for qemu default of 32MiB/s when migrating to a file. + Default speed unit is MB, so set to unlimited with INT64_MAX / 1M. + Failure to change migration speed is not fatal. */ + if (qemuDomainObjEnterMonitorAsync(driver, vm, asyncJob) == 0) { + qemuMonitorSetMigrationSpeed(priv->mon, INT64_MAX / (1024 * 1024)); + qemuDomainObjExitMonitorWithDriver(driver, vm); + } +
This part is now fine.
if (qemuCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATE_QEMU_FD)&& (!compressor || pipe(pipeFD) == 0)) { /* All right! We can use fd migration, which means that qemu @@ -2783,6 +2791,12 @@ qemuMigrationToFile(struct qemud_driver *driver, virDomainObjPtr vm, ret = 0;
cleanup: + /* Restore migration speed to 32MiB/s default */ + if (qemuDomainObjEnterMonitorAsync(driver, vm, asyncJob) == 0) { + qemuMonitorSetMigrationSpeed(priv->mon, (32<< 20));
If we keep this hunk, I'd like to see this magic number as a #define earlier in the file, with the same justification as you gave in your commit comment about where it comes from (32Mbps in qemu's migration.c), so it becomes easier to replace the number and/or consistently use it elsewhere in the code if qemu ever changes.
Ok.
+ qemuDomainObjExitMonitorWithDriver(driver, vm); + }
I was first worried that this part means we have more issues. That is, the user can independently call virDomainMigrateSetMaxSpeed, and it seems like we should revert back to that value, rather than to the qemu default. But on further thought - guess what? After you complete migration to a file, the qemu process is (supposed to have) ended! There's no reason to restore migration speed after this point, because there's nothing further you can do with the qemu process; once the domain is later restored from the save file, you have created a new qemu process which is once again back at the default migration speed. That means we can effectively drop this hunk with no change in behavior.
Yeah, I debated about whether to even add this hunk, but was considering 'virsh dump --live ...' where the process still exists and could potentially be migrated to another host later.
Meanwhile, it raises independent issues - why do we have a write-only interface in virDomainMigrateSetMaxSpeed? Shouldn't we also be able to query the speed currently in use, and shouldn't the domain XML track the current migration speed?
AFAICT, qemu doesn't provide a way to query the speed. The monitor would need this addition before we could plumb it in libvirt. How would you like to proceed? Regards, Jim
On 08/04/2011 12:41 PM, Jim Fehlig wrote:
Yeah, I debated about whether to even add this hunk, but was considering 'virsh dump --live ...' where the process still exists and could potentially be migrated to another host later.
Oh, I forgot about that. I guess we do have a case where migration to file does not imply the death of the qemu process.
Meanwhile, it raises independent issues - why do we have a write-only interface in virDomainMigrateSetMaxSpeed? Shouldn't we also be able to query the speed currently in use, and shouldn't the domain XML track the current migration speed?
AFAICT, qemu doesn't provide a way to query the speed. The monitor would need this addition before we could plumb it in libvirt.
Not necessarily true. If we track it in the domain xml, then libvirt can update that xml any time it makes a change, and set qemu to match the xml any time that it is changed or qemu started, and answer queries from the xml rather than by querying the monitor.
How would you like to proceed?
Are you in the mood to add virDomainMigrateGetMaxSpeed and the XML change necessary to track it? If so, then I'd favor tackling that first, at which point we'll have answered our questions about what speed to restore here. I would even be okay with pushing this patch first (with the tweak of adding a #define for the 32<<20 magic number), if you can get more feedback from others agreeing on the points raised here; cleaning things up to restore the saved value rather than the default value would be an appropriate part of the series adding the ability to track the saved value. -- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org
Eric Blake wrote:
On 08/04/2011 12:41 PM, Jim Fehlig wrote:
Yeah, I debated about whether to even add this hunk, but was considering 'virsh dump --live ...' where the process still exists and could potentially be migrated to another host later.
Oh, I forgot about that. I guess we do have a case where migration to file does not imply the death of the qemu process.
Meanwhile, it raises independent issues - why do we have a write-only interface in virDomainMigrateSetMaxSpeed? Shouldn't we also be able to query the speed currently in use, and shouldn't the domain XML track the current migration speed?
AFAICT, qemu doesn't provide a way to query the speed. The monitor would need this addition before we could plumb it in libvirt.
Not necessarily true. If we track it in the domain xml, then libvirt can update that xml any time it makes a change, and set qemu to match the xml any time that it is changed or qemu started, and answer queries from the xml rather than by querying the monitor.
Ah, right. Good point.
How would you like to proceed?
Are you in the mood to add virDomainMigrateGetMaxSpeed and the XML change necessary to track it? If so, then I'd favor tackling that first, at which point we'll have answered our questions about what speed to restore here.
I don't have many free cycles ATM, but could add this to my todo list.
I would even be okay with pushing this patch first (with the tweak of adding a #define for the 32<<20 magic number), if you can get more feedback from others agreeing on the points raised here; cleaning things up to restore the saved value rather than the default value would be an appropriate part of the series adding the ability to track the saved value.
Other opinions? Is this patch useful as is (after fixing magic number) or should Eric's suggestion of adding virDomainMigrateGetMaxSpeed() be done first? In most cases, the speed is changed when actually invoking the virDomainMigrate* APIs, in which case the qemu process is terminated after successful migration and the value is lost anyhow. I guess the only interesting case is apps that call virDomainMigrateSetMaxSpeed() and expect the value to persist during life of associated qemu process. Regards, Jim
participants (2)
-
Eric Blake -
Jim Fehlig