2:27 p.m.
On 08/10/2011 11:40 AM, Avi Kivity wrote:
On 08/10/2011 07:23 PM, Anthony Liguori wrote:
>> Right now we have capabilties in the form of -help output.
> >> If -help says
> >> -no-xzbrle disable xzbrle support
> >> (or -migration-compression xzbrle=off, or something)
that's sufficient
>> for management tools.
> This is static, not dynamic.
You may attempt to migrate to another
> host that supports it and then migrate to a second host that doesn't
> support it after the first migration fails.
This may be acceptable, wait until the entire migration cluster is
xzbrle capable before enabling it. If not, add a monitor command.
1) xzbrle needs to be disabled by default. That way management tools
don't unknowingly enable it by not passing -no-xzbrle.
2) there needs to be a mechanism for the management tool to query
whether qemu supports xzbrle.
3) a management tool should be able to query the source and destination,
and then enable xzbrle if both sides support it.
You can argue that (3) could be static. A command could be added to
toggle it dynamically through the monitor.
But no matter what, someone has to touch libvirt and any other tool that
works with QEMU to make this thing work. But this is a general problem.
Any optional change to the migration protocol has exactly the same
characteristics whether it's XZBRLE, XZBRLE v2 (if there is a v2),
ASN.1, or any other form of compression that rolls around.
Instead of teaching management tools how to deal with all of these
things, let's just fix this problem once. It just takes:
a) A query-migration-caps command that returns a dict with two lists of
strings. Something like:
{ 'execute': 'query-migration-caps' }
{ 'return' : { 'capabilities': [ 'xbzrle' ], 'current': []
} }
b) A set-migration-caps command that takes a list of strings. It simply
takes the intersection of the capabilities set with the argument and
sets the current set to the result. Something like:
{ 'execute': 'set-migration-caps', 'arguments': { 'set': [
'xbzrle' ] }}
{ 'return' : {} }
c) An internal interface to register a capability and an internal
interface to check if a capability is currently enabled. The xzbrle
code just needs to disable itself if the capability isn't set.
Then we teach libvirt (and other tools) to query the caps list on the
source, set the destination, query the current set on the destination,
and then set that set on the source.
As we introduce new things, like the next great compression protocol, or
ASN.1, we don't need to touch libvirt again. libvirt can still know
about the caps and selectively override QEMU if it's so inclined but it
prevents us from reinventing the same mechanisms over and over again.
>> We shouldn't block this feature just because some monitor
facility is
>> not yet implemented.
> We shouldn't make *any* changes to the migration
protocol before we
> have a feature negotiation capability. I only want to do a hard break
> of the protocol once.
Didn't we agree that management tool mediated feature negotiation (that
is, outside the migration protocol itself) is acceptable?
Yes. But that negotiation needs to become part of the "protocol" for
migration. In the absence of that negotiation, we need to use the wire
protocol we use today. We cannot have ad-hoc feature negotiation for
every change we make to the wire protocol.
Regards,
Anthony Liguori
3:03 a.m.
New subject: [libvirt] [Qemu-devel] [PATCH v4] XBZRLE delta for live migration of large memory apps
From: Anthony Liguori [mailto:anthony@codemonkey.ws]
Sent: Wednesday, August 10, 2011 10:28 PM
To: Avi Kivity
Cc: Blue Swirl; Stefan Hajnoczi; Shribman, Aidan; qemu-devel
Developers; libvir-list(a)redhat.com
Subject: Re: [Qemu-devel] [PATCH v4] XBZRLE delta for live migration of
large memory apps
a) A query-migration-caps command that returns a dict with two lists
of
strings. Something like:
{ 'execute': 'query-migration-caps' }
{ 'return' : { 'capabilities': [ 'xbzrle' ], 'current':
[] } }
b) A set-migration-caps command that takes a list of strings. It
simply
takes the intersection of the capabilities set with the argument and
sets the current set to the result. Something like:
{ 'execute': 'set-migration-caps', 'arguments': { 'set':
[ 'xbzrle' ]
}}
{ 'return' : {} }
We may want to further sub-divide capabilities into categories:
{ 'execute': 'query-migration-caps' }
{ 'return' :
{ 'encoding' : { 'current', 'asn.1', 'proto2',
'thrift', etc. } },
{ 'delta' : { 'xbzrle', "xdelta", ...} },
{ 'compression' : { 'snappy', 'lzo' } } }
This would help libvirt/management to select features automatically or manually (via UI)
without having to 'understand' the any given capability meaning.
Yes. But that negotiation needs to become part of the
"protocol" for
migration. In the absence of that negotiation, we need to use the wire
protocol we use today. We cannot have ad-hoc feature negotiation for
every change we make to the wire protocol.
Agreed. Therefore caps plus xbzrle could be added before ASN.1/v1.0 without breaking
anything as long as when 'set-migration-caps' is not issued Qemu uses the current
protocol.
Aidan
8 a.m.
New subject: [libvirt] [Qemu-devel] [PATCH v4] XBZRLE delta for live migration of large memory apps
On 08/11/2011 03:03 AM, Shribman, Aidan wrote:
> From: Anthony Liguori [mailto:anthony@codemonkey.ws]
> Sent: Wednesday, August 10, 2011 10:28 PM
> To: Avi Kivity
> Cc: Blue Swirl; Stefan Hajnoczi; Shribman, Aidan; qemu-devel
> Developers; libvir-list(a)redhat.com
> Subject: Re: [Qemu-devel] [PATCH v4] XBZRLE delta for live migration of
> large memory apps
> a) A query-migration-caps command that returns a dict with two lists of
> strings. Something like:
>
> { 'execute': 'query-migration-caps' }
> { 'return' : { 'capabilities': [ 'xbzrle' ],
'current': [] } }
>
> b) A set-migration-caps command that takes a list of strings. It
> simply
> takes the intersection of the capabilities set with the argument and
> sets the current set to the result. Something like:
>
> { 'execute': 'set-migration-caps', 'arguments': {
'set': [ 'xbzrle' ]
> }}
> { 'return' : {} }
We may want to further sub-divide capabilities into categories:
{ 'execute': 'query-migration-caps' }
{ 'return' :
{ 'encoding' : { 'current', 'asn.1', 'proto2',
'thrift', etc. } },
{ 'delta' : { 'xbzrle', "xdelta", ...} },
{ 'compression' : { 'snappy', 'lzo' } } }
This would help libvirt/management to select features automatically or manually (via UI)
without having to 'understand' the any given capability meaning.
I would prefer caps to be mostly transparent to libvirt. In fact, I'd
like to see exactly three caps: xbzrle, asn1, and autonegotiate.
I'd like to move the caps negotation into the protocol itself.
> Yes. But that negotiation needs to become part of the
"protocol" for
> migration. In the absence of that negotiation, we need to use the wire
> protocol we use today. We cannot have ad-hoc feature negotiation for
> every change we make to the wire protocol.
Agreed. Therefore caps plus xbzrle could be added before ASN.1/v1.0 without breaking
anything as long as when 'set-migration-caps' is not issued Qemu uses the current
protocol.
Exactly.
Regards,
Anthony Liguori
Aidan
3:17 a.m.
New subject: [libvirt] [Qemu-devel] [PATCH v4] XBZRLE delta for live migration of large memory apps
On 08/10/2011 10:27 PM, Anthony Liguori wrote:
> This may be acceptable, wait until the entire migration cluster
is
> xzbrle capable before enabling it. If not, add a monitor command.
1) xzbrle needs to be disabled by default. That way management tools
don't unknowingly enable it by not passing -no-xzbrle.
We could hook it to -M, though it's a bit gross. Otherwise we need to
document this clearly in the management tool author's guide.
3) a management tool should be able to query the source and
destination, and then enable xzbrle if both sides support it.
You can argue that (3) could be static. A command could be added to
toggle it dynamically through the monitor.
But no matter what, someone has to touch libvirt and any other tool
that works with QEMU to make this thing work. But this is a general
problem. Any optional change to the migration protocol has exactly
the same characteristics whether it's XZBRLE, XZBRLE v2 (if there is a
v2), ASN.1, or any other form of compression that rolls around.
If we have two-way communication we can do this transparently in the
protocol itself.
Instead of teaching management tools how to deal with all of these
things, let's just fix this problem once. It just takes:
a) A query-migration-caps command that returns a dict with two lists
of strings. Something like:
{ 'execute': 'query-migration-caps' }
{ 'return' : { 'capabilities': [ 'xbzrle' ], 'current':
[] } }
b) A set-migration-caps command that takes a list of strings. It
simply takes the intersection of the capabilities set with the
argument and sets the current set to the result. Something like:
{ 'execute': 'set-migration-caps', 'arguments': { 'set':
[ 'xbzrle' ] }}
{ 'return' : {} }
c) An internal interface to register a capability and an internal
interface to check if a capability is currently enabled. The xzbrle
code just needs to disable itself if the capability isn't set.
Then we teach libvirt (and other tools) to query the caps list on the
source, set the destination, query the current set on the destination,
and then set that set on the source.
This is only if the capability has no side effect.
As we introduce new things, like the next great compression protocol,
or ASN.1, we don't need to touch libvirt again. libvirt can still
know about the caps and selectively override QEMU if it's so inclined
but it prevents us from reinventing the same mechanisms over and over
again.
Right.
Yes. But that negotiation needs to become part of the "protocol" for
migration. In the absence of that negotiation, we need to use the
wire protocol we use today. We cannot have ad-hoc feature negotiation
for every change we make to the wire protocol.
Okay, as long as we have someone willing to implement it.
--
error compiling committee.c: too many arguments to function
4:16 a.m.
New subject: [libvirt] [Qemu-devel] [PATCH v4] XBZRLE delta for live migration of large memory apps
On Thu, Aug 11, 2011 at 11:17:09AM +0300, Avi Kivity wrote:
On 08/10/2011 10:27 PM, Anthony Liguori wrote:
>>This may be acceptable, wait until the entire migration cluster is
>>xzbrle capable before enabling it. If not, add a monitor command.
>
>
>1) xzbrle needs to be disabled by default. That way management
>tools don't unknowingly enable it by not passing -no-xzbrle.
We could hook it to -M, though it's a bit gross.
That would needlessly prevent its use for any existing installed
guests with a older machine type, which are running in a new QEMU
Some kind of monitor capabilities seems good to me.
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
4:20 a.m.
New subject: [libvirt] [Qemu-devel] [PATCH v4] XBZRLE delta for live migration of large memory apps
On 08/11/2011 12:16 PM, Daniel P. Berrange wrote:
On Thu, Aug 11, 2011 at 11:17:09AM +0300, Avi Kivity wrote:
> On 08/10/2011 10:27 PM, Anthony Liguori wrote:
> >>This may be acceptable, wait until the entire migration cluster is
> >>xzbrle capable before enabling it. If not, add a monitor command.
> >
> >
> >1) xzbrle needs to be disabled by default. That way management
> >tools don't unknowingly enable it by not passing -no-xzbrle.
>
> We could hook it to -M, though it's a bit gross.
That would needlessly prevent its use for any existing installed
guests with a older machine type, which are running in a new QEMU
You could still enable it explicitly; I'm just trying to get it to be
enabled by default.
Some kind of monitor capabilities seems good to me.
Live migration is probably mostly done in managed environments, so I
think you're right.
--
error compiling committee.c: too many arguments to function
8:03 a.m.
New subject: [libvirt] [Qemu-devel] [PATCH v4] XBZRLE delta for live migration of large memory apps
On 08/11/2011 03:17 AM, Avi Kivity wrote:
> 3) a management tool should be able to query the source and
> destination, and then enable xzbrle if both sides support it.
>
> You can argue that (3) could be static. A command could be added to
> toggle it dynamically through the monitor.
>
> But no matter what, someone has to touch libvirt and any other tool
> that works with QEMU to make this thing work. But this is a general
> problem. Any optional change to the migration protocol has exactly the
> same characteristics whether it's XZBRLE, XZBRLE v2 (if there is a
> v2), ASN.1, or any other form of compression that rolls around.
If we have two-way communication we can do this transparently in the
protocol itself.
Yes. This should be one of the initial caps to introduce.
> Instead of teaching management tools how to deal with all of
these
> things, let's just fix this problem once. It just takes:
>
> a) A query-migration-caps command that returns a dict with two lists
> of strings. Something like:
>
> { 'execute': 'query-migration-caps' }
> { 'return' : { 'capabilities': [ 'xbzrle' ],
'current': [] } }
>
> b) A set-migration-caps command that takes a list of strings. It
> simply takes the intersection of the capabilities set with the
> argument and sets the current set to the result. Something like:
>
> { 'execute': 'set-migration-caps', 'arguments': {
'set': [ 'xbzrle' ] }}
> { 'return' : {} }
>
> c) An internal interface to register a capability and an internal
> interface to check if a capability is currently enabled. The xzbrle
> code just needs to disable itself if the capability isn't set.
>
> Then we teach libvirt (and other tools) to query the caps list on the
> source, set the destination, query the current set on the destination,
> and then set that set on the source.
This is only if the capability has no side effect.
Right, it can't change the output of any monitor commands or anything
like that. It's strictly about the encoding of the wire protocol which
ought to be transparent to libvirt.
> As we introduce new things, like the next great compression
protocol,
> or ASN.1, we don't need to touch libvirt again. libvirt can still know
> about the caps and selectively override QEMU if it's so inclined but
> it prevents us from reinventing the same mechanisms over and over again.
Right.
>
> Yes. But that negotiation needs to become part of the "protocol" for
> migration. In the absence of that negotiation, we need to use the wire
> protocol we use today. We cannot have ad-hoc feature negotiation for
> every change we make to the wire protocol.
Okay, as long as we have someone willing to implement it.
Sounds like a good hackathon project :-)
Regards,
Anthony Liguori
4:24 a.m.
New subject: [libvirt] [Qemu-devel] [PATCH v4] XBZRLE delta for live migration of large memory apps
On Wed, Aug 10, 2011 at 02:27:41PM -0500, Anthony Liguori wrote:
On 08/10/2011 11:40 AM, Avi Kivity wrote:
Instead of teaching management tools how to deal with all of these
things, let's just fix this problem once. It just takes:
a) A query-migration-caps command that returns a dict with two lists
of strings. Something like:
{ 'execute': 'query-migration-caps' }
{ 'return' : { 'capabilities': [ 'xbzrle' ], 'current':
[] } }
b) A set-migration-caps command that takes a list of strings. It
simply takes the intersection of the capabilities set with the
argument and sets the current set to the result. Something like:
{ 'execute': 'set-migration-caps', 'arguments': { 'set':
[ 'xbzrle' ] }}
{ 'return' : {} }
We have a number of commands to set migration parameters (bandwidth,
max downtime, etc). One thing that has always troubled me with this,
is that they are a global setting for QEMU, not simply the next
migration command. This is fine if you expect that only a single
'migrate' command will ever be invoked during the lifetime of QEMU,
but this doesn't hold true when we use 'migrate' as a means to
implement saving of snapshots/save-to-file, or "core" dumps.
For example, when libvirt does 'save to file', we want to set the
max bandwidth to unlimited for that, but we don't want that 'unlimited'
setting to apply to a future cross-host migration attempt. This means
we have to send three commands
migrate_set_speed 9223372036854775808
migrate file:/some/file
migrate_set_speed 33554432
it doubly sucks because there is no way to reset the migration speed
to the QEMU default, so we have to hardcoded (32 << 20) which is what
QEMU currently uses.
It would be more desirable if we could simply pass in the desired
speed, compression algorithm, max downtime, etc, as parameters to
the 'migrate' command. And then have 'migrate_set_speed' only
affect the current active migration, not any future ones.
So a 'query-migration-caps' command is nice, but I think having a
set-migration-caps command is wrong. There should just be a 'caps'
parameter for 'migrate'.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
4852
days inactive
4853
days old
7 comments
4 participants
participants (4)
-
Anthony Liguori -
Avi Kivity -
Daniel P. Berrange -
Shribman, Aidan