Re: [libvirt] [RFC] libvirt support multi-thread compression for live migration
[correcting list address: libvirt-list, not libver-list] On 06/02/2015 05:58 AM, Feng, Shaohe wrote:
Hi folks: I'd like to request some comments on enabling multi-thread compression in libvirt.
Currently, qemu upstream has supported multi-thread compression for live migration. $ git log 263170e~1..362ba4e -oneline This can preserve bandwidth and speed up the live migration process, so this is an import feature we need to enable so for other high level such as openstack will be benefit.
We plan to add feature of multi-thread compression and actually the patch is working in process.
Now some things need for comments.
1. Expose new set/get multi-thread compression parameters API for live migration. Now qemu only supports zlib compression. Maybe LZ4 and gzip will be supported later. but there are 3 parameters needed by qemu currently. "compress-level": compression level, default is 1. For zlib compression, it is from 0-9, 0 means use default level, 9 means high compressed. "compress-threads": compression thread count for multi-thread migration, default is 8 in qemu. "decompress-threads": decompression thread count for multi-thread migration, default is 2 in qemu
So in libvirt we will expose the public symbols as follow, we only 3 parameters, missing compression method(zlib, LZ4 and gzip) parameters, for qemu do not support it at present. index d851225..103b3b9 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -798,6 +798,17 @@ int virDomainMigrateGetMaxSpeed(virDomainPtr domain, unsigned long *bandwidth, unsigned int flags);
+int virdomainMigrateSetParameters(virDomainPtr domain, + int level, + int threads, + int dthreads, + int flags) +int virdomainMigrateGetParameters(virDomainPtr domain, + int *level, + int *threads, + int *dthreads, + int flags) +
I'd rather we used virTypedParameters, to make it easier to use the same API to pass ALL future possible tuning parameters, rather than just hard-coding to only the parameters of this one feature.
For virdomainMigrateSetParameters, if specifying a negative value to level, threads, and dthreads parameters, such as -1, means do not set the parameters. The underlying code will not pass the corresponding parameters to qemu monitor. Such as threads and dthreads is -1, then pass the following json streaming to qemu. { "execute": "migrate-set-parameters" , "arguments": { "compress-level": 1 } }
The virsh will expose the two commands: # virsh migrate-setparameters <domain> [--level level] [--threads threads] [--dthreads dthread] # virsh migrate-getparameters <domain>
2. How to enable multi-thread compression in application interface? There is not a special interface for setting migration capabilities. But we can set the capabilities when execute an virsh command as following: $ migrate --live] [--offline] [--p2p] [--direct] [--tunnelled] [--persistent] [--undefinesource] [--suspend] [--copy-storage-all] [--copy-storage-inc] [--change-protection] [--unsafe] [--verbose] [--compressed] [--abort-on-error] <domain> <desturi> [<migrateuri>] [<graphicsuri>] [<listen-address>] [<dname>] [--timeout <number>] [--xml <string>]
There is already a "compressed" option, here "compressed" means "xbzrle" not "multi- compressed". can I rename the "compressed" to "xbzrle"?
If we think that it is worth always turning on both compression styles simultaneously, then reuse the bit. Otherwise, we need a different bit, and users can choose which (or both) of the two compression styles as desired.
so we add another option for multi-thread compression, which is better option name? "-multi- compressed" ? Any way this is confuse to user. We need to distinguish<http://dict.youdao.com/w/distinguish/> these two.
So I wonder should we expose new API to enable the migrate multi-thread compress. +int virdomainMigrateEnableMultiThreadCompress (virDomainPtr domain, int flags) It will pass the following command to qemu monitor. { "execute": "migrate-set-capabilities" , "arguments": { "capabilities": [ { "capability": " compress", "state": true } ] } }
NOTE: Now, multiple thread compression can co-work with xbzrle. When xbzrle is on, multiple thread compression will only work at the first round of RAM data sync. Qemu, $ git show 98f1138902195bd9ab8a753d0ee2cf2a0a88b6e8 if compress and xbzrle are both on, compress only takes effect in the ram bulk stage, after that, it will be disabled and only xbzrle takes effect, this can help to minimize migration traffic.
3. Migration between different livirt/qemu version We can support to expose 2 new API to set/get migrate capabilities. + virdomainMigrateSetCapabilities + virdomainMigrateGetCapabilities And we can suggest the user application to probe the capabilities before execute live migration in our doc. This need discussion is it necessary support these two in libvirt?
Without the above GetCapabilities API, user application can probe multi-thread compress capabilities by virdomainMigrateGetParameters.
Or return error directly when user application execute live migration with multi-thread flag specifying, but do not any Capabilities probe.
INFO: qmeu command {"execute": "query-migrate-capabilities"} {"return": [{"state": false, "capability": "xbzrle"}, {"state": false, "capability": "rdma-pin-all"}, {"state": false, "capability": "auto-converge"}, {"state": false, "capability": "zero-blocks"}, {"state": false, "capability": "compress"}]}
-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Hi Eric Thanks for replying the mail, replied in lines. BR, Eli(Li Yong)Qiao -----Original Message----- From: Eric Blake [mailto:eblake@redhat.com] Sent: Tuesday, June 02, 2015 8:21 PM To: Feng, Shaohe; libvir-list@redhat.com Cc: Qiao, Liyong; Bhandaru, Malini K; Ding, Jian-feng; Chylinski, Arek; Koniszewski, Pawel; Li, Liang Z; berrange@redhat.com; veillard@redhat.com Subject: Re: [RFC] libvirt support multi-thread compression for live migration [correcting list address: libvirt-list, not libver-list] On 06/02/2015 05:58 AM, Feng, Shaohe wrote:
Hi folks: I'd like to request some comments on enabling multi-thread compression in libvirt.
Currently, qemu upstream has supported multi-thread compression for live migration. $ git log 263170e~1..362ba4e -oneline This can preserve bandwidth and speed up the live migration process, so this is an import feature we need to enable so for other high level such as openstack will be benefit.
We plan to add feature of multi-thread compression and actually the patch is working in process.
Now some things need for comments.
1. Expose new set/get multi-thread compression parameters API for live migration. Now qemu only supports zlib compression. Maybe LZ4 and gzip will be supported later. but there are 3 parameters needed by qemu currently. "compress-level": compression level, default is 1. For zlib compression, it is from 0-9, 0 means use default level, 9 means high compressed. "compress-threads": compression thread count for multi-thread migration, default is 8 in qemu. "decompress-threads": decompression thread count for multi-thread migration, default is 2 in qemu
So in libvirt we will expose the public symbols as follow, we only 3 parameters, missing compression method(zlib, LZ4 and gzip) parameters, for qemu do not support it at present. index d851225..103b3b9 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -798,6 +798,17 @@ int virDomainMigrateGetMaxSpeed(virDomainPtr domain, unsigned long *bandwidth, unsigned int flags);
+int virdomainMigrateSetParameters(virDomainPtr domain, + int level, + int threads, + int dthreads, + int flags) int +virdomainMigrateGetParameters(virDomainPtr domain, + int *level, + int *threads, + int *dthreads, + int flags) +
I'd rather we used virTypedParameters, to make it easier to use the same API to pass ALL future possible tuning parameters, rather than just hard-coding to only the parameters of this one feature. Okay ,that sound good, but if virTypedParameters can be passed to qemu_driver such as qemu_monitor_json.c qemu_monitor_text.c ?
For virdomainMigrateSetParameters, if specifying a negative value to level, threads, and dthreads parameters, such as -1, means do not set the parameters. The underlying code will not pass the corresponding parameters to qemu monitor. Such as threads and dthreads is -1, then pass the following json streaming to qemu. { "execute": "migrate-set-parameters" , "arguments": { "compress-level": 1 } }
The virsh will expose the two commands: # virsh migrate-setparameters <domain> [--level level] [--threads threads] [--dthreads dthread] # virsh migrate-getparameters <domain>
2. How to enable multi-thread compression in application interface? There is not a special interface for setting migration capabilities. But we can set the capabilities when execute an virsh command as following: $ migrate --live] [--offline] [--p2p] [--direct] [--tunnelled] [--persistent] [--undefinesource] [--suspend] [--copy-storage-all] [--copy-storage-inc] [--change-protection] [--unsafe] [--verbose] [--compressed] [--abort-on-error] <domain> <desturi> [<migrateuri>] [<graphicsuri>] [<listen-address>] [<dname>] [--timeout <number>] [--xml <string>]
There is already a "compressed" option, here "compressed" means "xbzrle" not "multi- compressed". can I rename the "compressed" to "xbzrle"?
If we think that it is worth always turning on both compression styles simultaneously, then reuse the bit. Otherwise, we need a different bit, and users can choose which (or both) of the two compression styles as desired. +1 for reuse compressed, and we can set compress-level, compress-threads, compress-dthreads by virdomainMigrateSetParameters(maybe some new virsh command--- migrate-setparameter) But how can we passing these parameter when we using `virsh migrate `, is there any parameter we can use in 'virsh migrate' command ? Can you point me out ?
so we add another option for multi-thread compression, which is better option name? "-multi- compressed" ? Any way this is confuse to user. We need to distinguish<http://dict.youdao.com/w/distinguish/> these two.
So I wonder should we expose new API to enable the migrate multi-thread compress. +int virdomainMigrateEnableMultiThreadCompress (virDomainPtr domain, +int flags) It will pass the following command to qemu monitor. { "execute": "migrate-set-capabilities" , "arguments": { "capabilities": [ { "capability": " compress", "state": true } ] } }
NOTE: Now, multiple thread compression can co-work with xbzrle. When xbzrle is on, multiple thread compression will only work at the first round of RAM data sync. Qemu, $ git show 98f1138902195bd9ab8a753d0ee2cf2a0a88b6e8 if compress and xbzrle are both on, compress only takes effect in the ram bulk stage, after that, it will be disabled and only xbzrle takes effect, this can help to minimize migration traffic.
3. Migration between different livirt/qemu version We can support to expose 2 new API to set/get migrate capabilities. + virdomainMigrateSetCapabilities + virdomainMigrateGetCapabilities And we can suggest the user application to probe the capabilities before execute live migration in our doc. This need discussion is it necessary support these two in libvirt?
Without the above GetCapabilities API, user application can probe multi-thread compress capabilities by virdomainMigrateGetParameters.
Or return error directly when user application execute live migration with multi-thread flag specifying, but do not any Capabilities probe.
INFO: qmeu command {"execute": "query-migrate-capabilities"} {"return": [{"state": false, "capability": "xbzrle"}, {"state": false, "capability": "rdma-pin-all"}, {"state": false, "capability": "auto-converge"}, {"state": false, "capability": "zero-blocks"}, {"state": false, "capability": "compress"}]}
-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On 06/02/2015 07:56 AM, Qiao, Liyong wrote:
Hi Eric Thanks for replying the mail, replied in lines.
+virdomainMigrateGetParameters(virDomainPtr domain, + int *level, + int *threads, + int *dthreads, + int flags) +
I'd rather we used virTypedParameters, to make it easier to use the same API to pass ALL future possible tuning parameters, rather than just hard-coding to only the parameters of this one feature.
Okay ,that sound good, but if virTypedParameters can be passed to qemu_driver such as qemu_monitor_json.c qemu_monitor_text.c ?
[Your quoting style makes it very hard to distinguish original text from added text. Please consider changing your outgoing mail process to ensure that you add another level of quoting to all previous text so that your added text is the only unquoted text. Also, configure your mailer to wrap long lines.] Use existing API for a guide - for example, virDomainSetBlockIoTune takes virTypedParamters, as well as defines a specific set of parameters that it will understand. The set of parameters can be enlarged without needing a new API (good for backporting features without requiring a .so version bump), but for any given libvirt version, the set of features understood is finite and limited to what you can translate to the QMP call. So qemu_driver.c would take the virTypedParameters, reject the ones it does not understand, and convert the ones it does understand into the 'int level, int threads, int dthreads' parameters used in qemu_monitor_json.c where you drive the actual QMP command with fixed parameters and types.
If we think that it is worth always turning on both compression styles simultaneously, then reuse the bit. Otherwise, we need a different bit, and users can choose which (or both) of the two compression styles as desired.
+1 for reuse compressed, and we can set compress-level, compress-threads, compress-dthreads by virdomainMigrateSetParameters(maybe some new virsh command--- migrate-setparameter) But how can we passing these parameter when we using `virsh migrate `, is there any parameter we can use in 'virsh migrate' command ? Can you point me out ?
The underlying API already has a form that takes virTypedParameters (see virDomainMigrate3()), so your first task is to figure out how to extend the API to expose new typed parameters for your new migration tunings. Once that is done, then you can worry about how to tweak the 'virsh migrate' client to pass in those new parameters. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Hi Eric Thanks for replying the mail, replied in lines.
+virdomainMigrateGetParameters(virDomainPtr domain, + int *level, + int *threads, + int *dthreads, + int flags) + I'd rather we used virTypedParameters, to make it easier to use the same API to pass ALL future possible tuning parameters, rather than just hard-coding to only the parameters of this one feature.
Okay ,that sound good, but if virTypedParameters can be passed to qemu_driver such as qemu_monitor_json.c qemu_monitor_text.c ? [Your quoting style makes it very hard to distinguish original text from added text. Please consider changing your outgoing mail process to ensure that you add another level of quoting to all previous text so
On 06/02/2015 07:56 AM, Qiao, Liyong wrote: that your added text is the only unquoted text. Also, configure your mailer to wrap long lines.] hi Eric, sorry about the inconvenient mail client, I switch outlook to
On 2015年06月03日 01:02, Eric Blake wrote: thunder bird, hopes it will be better.
Use existing API for a guide - for example, virDomainSetBlockIoTune takes virTypedParamters, as well as defines a specific set of parameters that it will understand. The set of parameters can be enlarged without needing a new API (good for backporting features without requiring a .so version bump), but for any given libvirt version, the set of features understood is finite and limited to what you can translate to the QMP call. So qemu_driver.c would take the virTypedParameters, reject the ones it does not understand, and convert the ones it does understand into the 'int level, int threads, int dthreads' parameters used in qemu_monitor_json.c where you drive the actual QMP command with fixed parameters and types. ah, that helps, thanks for kindly supporting, we will think a bit more to how implement this API (taken virTypedParamters as parameter)
If we think that it is worth always turning on both compression styles simultaneously, then reuse the bit. Otherwise, we need a different bit, and users can choose which (or both) of the two compression styles as desired.
+1 for reuse compressed, and we can set compress-level, compress-threads, compress-dthreads by virdomainMigrateSetParameters(maybe some new virsh command--- migrate-setparameter) But how can we passing these parameter when we using `virsh migrate `, is there any parameter we can use in 'virsh migrate' command ? Can you point me out ? The underlying API already has a form that takes virTypedParameters (see virDomainMigrate3()), so your first task is to figure out how to extend the API to expose new typed parameters for your new migration tunings. Once that is done, then you can worry about how to tweak the 'virsh migrate' client to pass in those new parameters.
-- BR, Eli(Li Yong)Qiao
Eric, Thank you for reply. Is it necessary to expose these two APIs to user application? + virdomainMigrateSetCapabilities + virdomainMigrateGetCapabilities For qemu , the migration capabilities are "xbzrle", "rdma-pin-all", "auto-converge", "zero-blocks" and "compress". Sorry , I use outlook. Will change to thunderbird. -----Original Message----- From: Qiao, Liyong Sent: Wednesday, June 3, 2015 7:49 AM To: Eric Blake; Feng, Shaohe; libvir-list@redhat.com Cc: Bhandaru, Malini K; Ding, Jian-feng; Chylinski, Arek; Koniszewski, Pawel; Li, Liang Z; berrange@redhat.com; veillard@redhat.com Subject: Re: [RFC] libvirt support multi-thread compression for live migration On 2015年06月03日 01:02, Eric Blake wrote:
Hi Eric Thanks for replying the mail, replied in lines.
+virdomainMigrateGetParameters(virDomainPtr domain, + int *level, + int *threads, + int *dthreads, + int flags) + I'd rather we used virTypedParameters, to make it easier to use the same API to pass ALL future possible tuning parameters, rather than just hard-coding to only the parameters of this one feature.
Okay ,that sound good, but if virTypedParameters can be passed to qemu_driver such as qemu_monitor_json.c qemu_monitor_text.c ? [Your quoting style makes it very hard to distinguish original text from added text. Please consider changing your outgoing mail process to ensure that you add another level of quoting to all previous text so that your added text is the only unquoted text. Also, configure your mailer to wrap long lines.] hi Eric, sorry about the inconvenient mail client, I switch outlook to thunder bird, hopes it will be better. Use existing API for a guide - for example, virDomainSetBlockIoTune takes virTypedParamters, as well as defines a specific set of
On 06/02/2015 07:56 AM, Qiao, Liyong wrote: parameters that it will understand. The set of parameters can be enlarged without needing a new API (good for backporting features without requiring a .so version bump), but for any given libvirt version, the set of features understood is finite and limited to what you can translate to the QMP call. So qemu_driver.c would take the virTypedParameters, reject the ones it does not understand, and convert the ones it does understand into the 'int level, int threads, int dthreads' parameters used in qemu_monitor_json.c where you drive the actual QMP command with fixed parameters and types. ah, that helps, thanks for kindly supporting, we will think a bit more to how implement this API (taken virTypedParamters as parameter)
If we think that it is worth always turning on both compression styles simultaneously, then reuse the bit. Otherwise, we need a different bit, and users can choose which (or both) of the two compression styles as desired.
+1 for reuse compressed, and we can set compress-level, +compress-threads, compress-dthreads by +virdomainMigrateSetParameters(maybe some new virsh command--- +migrate-setparameter) But how can we passing these parameter when we using `virsh migrate `, is there any parameter we can use in 'virsh migrate' command ? Can you point me out ? The underlying API already has a form that takes virTypedParameters (see virDomainMigrate3()), so your first task is to figure out how to extend the API to expose new typed parameters for your new migration tunings. Once that is done, then you can worry about how to tweak the 'virsh migrate' client to pass in those new parameters.
-- BR, Eli(Li Yong)Qiao
On Wed, Jun 03, 2015 at 15:13:17 +0000, Feng, Shaohe wrote:
Eric, Thank you for reply.
Is it necessary to expose these two APIs to user application? + virdomainMigrateSetCapabilities + virdomainMigrateGetCapabilities
For qemu , the migration capabilities are "xbzrle", "rdma-pin-all", "auto-converge", "zero-blocks" and "compress".
No, these capabilities are either turned on/off based on flags passed to virDomainMigrate3 API. Jirka
On 2015年06月04日 20:01, Jiri Denemark wrote:
On Wed, Jun 03, 2015 at 15:13:17 +0000, Feng, Shaohe wrote:
Eric, Thank you for reply.
Is it necessary to expose these two APIs to user application? + virdomainMigrateSetCapabilities + virdomainMigrateGetCapabilities
For qemu , the migration capabilities are "xbzrle", "rdma-pin-all", "auto-converge", "zero-blocks" and "compress". No, these capabilities are either turned on/off based on flags passed to virDomainMigrate3 API. I think we need to call qemuMonitorGetMigrationCapability when qemuMigrationSetCompression if the source/dest node doesn't support 'compress' capability, we need to stop migration if flags contain VIR_MIGRATE_COMPRESSED (currently it stands for xbzrle, and Eric's previous mail suggest we share this flag with 'compress')
-Eli.
Jirka
-- BR, Eli(Li Yong)Qiao
On 06/04/2015 06:31 PM, Qiao,Liyong wrote:
On 2015年06月04日 20:01, Jiri Denemark wrote:
On Wed, Jun 03, 2015 at 15:13:17 +0000, Feng, Shaohe wrote:
Eric, Thank you for reply.
Is it necessary to expose these two APIs to user application? + virdomainMigrateSetCapabilities + virdomainMigrateGetCapabilities
For qemu , the migration capabilities are "xbzrle", "rdma-pin-all", "auto-converge", "zero-blocks" and "compress". No, these capabilities are either turned on/off based on flags passed to virDomainMigrate3 API. I think we need to call qemuMonitorGetMigrationCapability when
[It's hard to read replies that aren't visually distinct from the rest of the text; you'll notice that most posters on this list leave a blank line before and after any text they add, so that a scan of the left-most column can easily spot blanks as a key for where new content begins]
qemuMigrationSetCompression if the source/dest node doesn't support 'compress' capability, we need to stop migration if flags contain VIR_MIGRATE_COMPRESSED (currently it stands for xbzrle, and Eric's previous mail suggest we share this flag with 'compress')
No, I was asking whether libvirt's 'compress' flag should imply "all possible compression at once", or whether there are cases where xbzrle and multi-thread are independently useful and should not both be on at once. If it is safe to always use both, then we don't need a new flag, but I don't know if it is safe. The migration handshake is already set up to negotiate capabilities between source and destination, whether it takes one flag (turning on both compressions, or gracefully falling back to xbzrle alone) or two (one per compression type, and erroring out if a request is made for an unsupported compression). -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On 2015年06月04日 20:01, Jiri Denemark wrote:
On Wed, Jun 03, 2015 at 15:13:17 +0000, Feng, Shaohe wrote:
Eric, Thank you for reply.
Is it necessary to expose these two APIs to user application? + virdomainMigrateSetCapabilities + virdomainMigrateGetCapabilities
For qemu , the migration capabilities are "xbzrle", "rdma-pin-all", "auto-converge", "zero-blocks" and "compress". No, these capabilities are either turned on/off based on flags passed to virDomainMigrate3 API. I think we need to call qemuMonitorGetMigrationCapability when
[It's hard to read replies that aren't visually distinct from the rest of the text; you'll notice that most posters on this list leave a blank line before and after any text they add, so that a scan of the left-most column can easily spot blanks as a key for where new content begins]
qemuMigrationSetCompression if the source/dest node doesn't support 'compress' capability, we need to stop migration if flags contain VIR_MIGRATE_COMPRESSED (currently it stands for xbzrle, and Eric's previous mail suggest we share this flag with 'compress')
No, I was asking whether libvirt's 'compress' flag should imply "all possible compression at once", or whether there are cases where xbzrle and multi- thread are independently useful and should not both be on at once. If it is safe to always use both, then we don't need a new flag, but I don't know if it is safe.
Multiple thread compression is CPU-intensive, while xbzrle is RAM-intensive. Although they can co-work correctly, there is no evidence show that turning both on will achieve better performance than using xbzrle or multi-thread independently. May be it's better to let the users decide which one to use depend on their specific use case.
The migration handshake is already set up to negotiate capabilities between source and destination, whether it takes one flag (turning on both compressions, or gracefully falling back to xbzrle alone) or two (one per compression type, and erroring out if a request is made for an unsupported compression).
-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
-----Original Message----- From: Qiao, Liyong Sent: Wednesday, June 3, 2015 7:49 AM To: Eric Blake; Feng, Shaohe; libvir-list@redhat.com Cc: Bhandaru, Malini K; Ding, Jian-feng; Chylinski, Arek; Koniszewski, Pawel; Li, Liang Z; berrange@redhat.com; veillard@redhat.com Subject: Re: [RFC] libvirt support multi-thread compression for live migration
On 2015年06月03日 01:02, Eric Blake wrote:
Hi Eric Thanks for replying the mail, replied in lines.
+virdomainMigrateGetParameters(virDomainPtr domain, + int *level, + int *threads, + int *dthreads, + int flags) + I'd rather we used virTypedParameters, to make it easier to use the same API to pass ALL future possible tuning parameters, rather than just hard-coding to only the parameters of this one feature.
Okay ,that sound good, but if virTypedParameters can be passed to qemu_driver such as qemu_monitor_json.c qemu_monitor_text.c ? [Your quoting style makes it very hard to distinguish original text from added text. Please consider changing your outgoing mail process to ensure that you add another level of quoting to all previous text so that your added text is the only unquoted text. Also, configure your mailer to wrap long lines.] hi Eric, sorry about the inconvenient mail client, I switch outlook to thunder bird, hopes it will be better. Use existing API for a guide - for example, virDomainSetBlockIoTune takes virTypedParamters, as well as defines a specific set of
On 06/02/2015 07:56 AM, Qiao, Liyong wrote: parameters that it will understand. The set of parameters can be enlarged without needing a new API (good for backporting features without requiring a .so version bump), but for any given libvirt version, the set of features understood is finite and limited to what you can translate to the QMP call. So qemu_driver.c would take the virTypedParameters, reject the ones it does not understand, and convert the ones it does understand into the 'int level, int threads, int dthreads' parameters used in qemu_monitor_json.c where you drive the actual QMP command with fixed parameters and types. ah, that helps, thanks for kindly supporting, we will think a bit more to how implement this API (taken virTypedParamters as parameter)
If we think that it is worth always turning on both compression styles simultaneously, then reuse the bit. Otherwise, we need a different bit, and users can choose which (or both) of the two compression styles as desired.
+1 for reuse compressed, and we can set compress-level,
Consider this scenario : one of the host/hypervisor are high cpu/memory usage, Cloud Tool, such as Openstack can make a strategy that do not compression by multi-thread, for high cpu usage, they just want to use "xbzrle".
Is this scenario reasonable?
+compress-threads, compress-dthreads by +virdomainMigrateSetParameters(maybe some new virsh command--- +migrate-setparameter) But how can we passing these parameter when we using `virsh migrate `, is there any parameter we can use in 'virsh migrate' command ? Can you point me out ? The underlying API already has a form that takes virTypedParameters (see virDomainMigrate3()), so your first task is to figure out how to extend the API to expose new typed parameters for your new migration tunings. Once that is done, then you can worry about how to tweak the 'virsh migrate' client to pass in those new parameters.
-- BR, Eli(Li Yong)Qiao
On Tue, Jun 02, 2015 at 11:02:27 -0600, Eric Blake wrote:
On 06/02/2015 07:56 AM, Qiao, Liyong wrote:
Hi Eric Thanks for replying the mail, replied in lines.
+virdomainMigrateGetParameters(virDomainPtr domain, + int *level, + int *threads, + int *dthreads, + int flags) +
I'd rather we used virTypedParameters, to make it easier to use the same API to pass ALL future possible tuning parameters, rather than just hard-coding to only the parameters of this one feature.
Okay ,that sound good, but if virTypedParameters can be passed to qemu_driver such as qemu_monitor_json.c qemu_monitor_text.c ?
[Your quoting style makes it very hard to distinguish original text from added text. Please consider changing your outgoing mail process to ensure that you add another level of quoting to all previous text so that your added text is the only unquoted text. Also, configure your mailer to wrap long lines.]
Use existing API for a guide - for example, virDomainSetBlockIoTune takes virTypedParamters, as well as defines a specific set of parameters that it will understand.
And do we actually need to changed these migration parameters on the fly when migration is already running. I can imagine we would need to do so for some parameters but multithreaded compression sounds like something that needs to be configured when migration starts and there's nothing to be tuned on the fly. If this is the case, I think we should just add these new parameters to virDomainMigrate3 instead of requiring another API to be called to actually configure multithreaded compression. The separate API makes it a bit harder to repeat migration (which previously failed) with a different parameters (e.g., without multithreaded compression). Jirka
Hi Eric Thanks for replying the mail, replied in lines.
+virdomainMigrateGetParameters(virDomainPtr domain, + int *level, + int *threads, + int *dthreads, + int flags) + I'd rather we used virTypedParameters, to make it easier to use the same API to pass ALL future possible tuning parameters, rather than just hard-coding to only the parameters of this one feature.
Okay ,that sound good, but if virTypedParameters can be passed to qemu_driver such as qemu_monitor_json.c qemu_monitor_text.c ? [Your quoting style makes it very hard to distinguish original text from added text. Please consider changing your outgoing mail process to ensure that you add another level of quoting to all previous text so
On 06/02/2015 07:56 AM, Qiao, Liyong wrote: that your added text is the only unquoted text. Also, configure your mailer to wrap long lines.]
Use existing API for a guide - for example, virDomainSetBlockIoTune takes virTypedParamters, as well as defines a specific set of parameters that it will understand. And do we actually need to changed these migration parameters on the fly when migration is already running. I can imagine we would need to do so for some parameters but multithreaded compression sounds like something
On Tue, Jun 02, 2015 at 11:02:27 -0600, Eric Blake wrote: that needs to be configured when migration starts and there's nothing to be tuned on the fly. If this is the case, I think we should just add these new parameters to virDomainMigrate3 instead of requiring another API to be called to actually configure multithreaded compression.
The separate API makes it a bit harder to repeat migration (which previously failed) with a different parameters (e.g., without multithreaded compression). hi jirka
On 2015年06月04日 20:09, Jiri Denemark wrote: thanks for your advice, that make sense for me. if I understanding you correctly, then we need only do this virsh migrate --live --compressed --compress-level 1--compress-threads 8 --decompress-threads 2 and all compressed parameters (compress-level ,compress-threads, decompress-threads) only be specified when we do call virDomainMigrate3 instead of expose to user by 'virsh migrate-setparameters' ? I think that's good for me, and make user easy to start LM. thanks, Eli.
Jirka
-- BR, Eli(Li Yong)Qiao
On Fri, Jun 05, 2015 at 08:22:31 +0800, Qiao,Liyong wrote:
Hi Eric Thanks for replying the mail, replied in lines.
+virdomainMigrateGetParameters(virDomainPtr domain, + int *level, + int *threads, + int *dthreads, + int flags) + I'd rather we used virTypedParameters, to make it easier to use the same API to pass ALL future possible tuning parameters, rather than just hard-coding to only the parameters of this one feature.
Okay ,that sound good, but if virTypedParameters can be passed to qemu_driver such as qemu_monitor_json.c qemu_monitor_text.c ? [Your quoting style makes it very hard to distinguish original text from added text. Please consider changing your outgoing mail process to ensure that you add another level of quoting to all previous text so
On 06/02/2015 07:56 AM, Qiao, Liyong wrote: that your added text is the only unquoted text. Also, configure your mailer to wrap long lines.]
Use existing API for a guide - for example, virDomainSetBlockIoTune takes virTypedParamters, as well as defines a specific set of parameters that it will understand. And do we actually need to changed these migration parameters on the fly when migration is already running. I can imagine we would need to do so for some parameters but multithreaded compression sounds like something
On Tue, Jun 02, 2015 at 11:02:27 -0600, Eric Blake wrote: that needs to be configured when migration starts and there's nothing to be tuned on the fly. If this is the case, I think we should just add these new parameters to virDomainMigrate3 instead of requiring another API to be called to actually configure multithreaded compression.
The separate API makes it a bit harder to repeat migration (which previously failed) with a different parameters (e.g., without multithreaded compression). hi jirka
On 2015年06月04日 20:09, Jiri Denemark wrote: thanks for your advice, that make sense for me.
if I understanding you correctly, then we need only do this
virsh migrate --live --compressed --compress-level 1--compress-threads 8 --decompress-threads 2
and all compressed parameters (compress-level ,compress-threads, decompress-threads) only be specified when we do call virDomainMigrate3 instead of expose to user by 'virsh migrate-setparameters' ?
Right, although there still seems to be an open question whether we can reuse --compressed or we need to come up with another option so that we can independently turn on XBZRLE and multithreaded compression. Jirka
participants (6)
-
Eric Blake -
Feng, Shaohe -
Jiri Denemark -
Li, Liang Z -
Qiao, Liyong -
Qiao,Liyong