[libvirt] Enhancing block/disk migration in libvirt
Hello all, I'm new to both openstack and libvirt so I may get some of this slightly wrong[1]. Here is some context form the openstack world (which at least some of you are aware of). There are at least 2 open bug against openstack (nova) in the area of block/disk migration. 1) Live migration fails when the instance has a config-drive[2] Here openstack(nova) fails because a drive that nova expects to be migrated isn't migrated. 2) libvirt live_snapshot periodically explodes on libvirt 1.2.2 in the gate[3] Here openstack(nova) fails because a drive that nova expects NOT to be migrated is migrated. To me these are essentially the same bug/issue. There is no way to communicate with libvirt the users expectations around block/disk mirgration. My idea so far would be to add an options element to the 'disk' XML node. This element could start with 3 possible states block_migration="default": Let libvirt decide block_migration="yes": This device should be block migrated block_migration="no": This device should *NOT* be block migrated The absence of this element would be treated as "default" above. This would mean that all existing domain XML would still be valid and have the expected behaviour and users (such as opensatck) can be explicit about deviced that do/do not need to be block migrated. While I'm certainly open to discussing the finer points of the implementation, right now I'm interested in getting a feel for is this idea generally ok? Yours Tony. [1] I'm happy to be corrected / pointed at community guidelines that I may have missed. [2] https://bugs.launchpad.net/nova/+bug/1246201 [3] https://bugs.launchpad.net/nova/+bug/1334398
On Mon, Feb 16, 2015 at 01:42:13PM +1100, Tony Breeds wrote:
Hello all, I'm new to both openstack and libvirt so I may get some of this slightly wrong[1].
Here is some context form the openstack world (which at least some of you are aware of). There are at least 2 open bug against openstack (nova) in the area of block/disk migration.
1) Live migration fails when the instance has a config-drive[2] Here openstack(nova) fails because a drive that nova expects to be migrated isn't migrated.
2) libvirt live_snapshot periodically explodes on libvirt 1.2.2 in the gate[3] Here openstack(nova) fails because a drive that nova expects NOT to be migrated is migrated.
On a related note, some weeks ago, I captured some context and analysis (including some debugging done by Dan Berrange and Eric Blake on public IRC channel mid-last year) for the above bug, since the actual bug has too many comments: https://kashyapc.fedorapeople.org/virt/openstack/nova-live-snapshot-bug-1334... The cause of it is almost narrowed down here by Dan in this comment https://bugs.launchpad.net/nova/+bug/1334398/comments/27 -- /kashyap
On Mon, Feb 16, 2015 at 11:12:14AM +0100, Kashyap Chamarthy wrote:
On a related note, some weeks ago, I captured some context and analysis (including some debugging done by Dan Berrange and Eric Blake on public IRC channel mid-last year) for the above bug, since the actual bug has too many comments:
https://kashyapc.fedorapeople.org/virt/openstack/nova-live-snapshot-bug-1334...
The cause of it is almost narrowed down here by Dan in this comment
Great thanks. Yours Tony.
On Mon, Feb 16, 2015 at 01:42:13PM +1100, Tony Breeds wrote:
Hello all, I'm new to both openstack and libvirt so I may get some of this slightly wrong[1].
Here is some context form the openstack world (which at least some of you are aware of). There are at least 2 open bug against openstack (nova) in the area of block/disk migration.
1) Live migration fails when the instance has a config-drive[2] Here openstack(nova) fails because a drive that nova expects to be migrated isn't migrated.
2) libvirt live_snapshot periodically explodes on libvirt 1.2.2 in the gate[3] Here openstack(nova) fails because a drive that nova expects NOT to be migrated is migrated.
To me these are essentially the same bug/issue. There is no way to communicate with libvirt the users expectations around block/disk mirgration.
My idea so far would be to add an options element to the 'disk' XML node. This element could start with 3 possible states
block_migration="default": Let libvirt decide block_migration="yes": This device should be block migrated block_migration="no": This device should *NOT* be block migrated
The absence of this element would be treated as "default" above.
This would mean that all existing domain XML would still be valid and have the expected behaviour and users (such as opensatck) can be explicit about deviced that do/do not need to be block migrated.
While I'm certainly open to discussing the finer points of the implementation, right now I'm interested in getting a feel for is this idea generally ok?
This doesn't really belong in the XML. The XML configs are for controlling the guest configuration, not functional behaviour of the APIs. So we need to fit any neccessary information into the migration API parameters instead, or define a new migration API for it. At minimum we just want a list of disks to be migrated. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 16.02.2015 11:16, Daniel P. Berrange wrote:
On Mon, Feb 16, 2015 at 01:42:13PM +1100, Tony Breeds wrote:
Hello all, I'm new to both openstack and libvirt so I may get some of this slightly wrong[1].
Here is some context form the openstack world (which at least some of you are aware of). There are at least 2 open bug against openstack (nova) in the area of block/disk migration.
1) Live migration fails when the instance has a config-drive[2] Here openstack(nova) fails because a drive that nova expects to be migrated isn't migrated.
2) libvirt live_snapshot periodically explodes on libvirt 1.2.2 in the gate[3] Here openstack(nova) fails because a drive that nova expects NOT to be migrated is migrated.
To me these are essentially the same bug/issue. There is no way to communicate with libvirt the users expectations around block/disk mirgration.
My idea so far would be to add an options element to the 'disk' XML node. This element could start with 3 possible states
block_migration="default": Let libvirt decide block_migration="yes": This device should be block migrated block_migration="no": This device should *NOT* be block migrated
The absence of this element would be treated as "default" above.
This would mean that all existing domain XML would still be valid and have the expected behaviour and users (such as opensatck) can be explicit about deviced that do/do not need to be block migrated.
While I'm certainly open to discussing the finer points of the implementation, right now I'm interested in getting a feel for is this idea generally ok?
This doesn't really belong in the XML. The XML configs are for controlling the guest configuration, not functional behaviour of the APIs. So we need to fit any neccessary information into the migration API parameters instead, or define a new migration API for it. At minimum we just want a list of disks to be migrated.
We have a migrate API that takes an array of virTypedParameter. We can use that to let users pass a list of disks to migrate. Michal
On Mon, Feb 16, 2015 at 11:34:51AM +0100, Michal Privoznik wrote:
On 16.02.2015 11:16, Daniel P. Berrange wrote:
This doesn't really belong in the XML. The XML configs are for controlling the guest configuration, not functional behaviour of the APIs. So we need to fit any neccessary information into the migration API parameters instead, or define a new migration API for it. At minimum we just want a list of disks to be migrated.
We have a migrate API that takes an array of virTypedParameter. We can use that to let users pass a list of disks to migrate.
Thanks I'll look at adding that to the API. Yours Tony.
On 17.02.2015 08:52, Tony Breeds wrote:
On Mon, Feb 16, 2015 at 11:34:51AM +0100, Michal Privoznik wrote:
On 16.02.2015 11:16, Daniel P. Berrange wrote:
This doesn't really belong in the XML. The XML configs are for controlling the guest configuration, not functional behaviour of the APIs. So we need to fit any neccessary information into the migration API parameters instead, or define a new migration API for it. At minimum we just want a list of disks to be migrated.
We have a migrate API that takes an array of virTypedParameter. We can use that to let users pass a list of disks to migrate.
Thanks I'll look at adding that to the API.
Oh, I forgot to paste the API name: virDomainMigrate3 Michal
On Tue, Feb 17, 2015 at 07:05:13PM +1100, Tony Breeds wrote:
On Tue, Feb 17, 2015 at 09:02:57AM +0100, Michal Privoznik wrote:
Oh, I forgot to paste the API name: virDomainMigrate3
That'll narrow the search! Thanks!
Is there an upstream libvirt bug for this work? If not, can it be filed with the context from this thread for tracking purpose? I'm asking because, a few upstream OpenStack Nova folks expressed interest on IRC to track this work. URL to file: https://bugzilla.redhat.com/bugzilla/enter_bug.cgi?product=Virtualization%20Tools&component=libvirt -- /kashyap
On Thu, Mar 12, 2015 at 01:05:21PM +0100, Kashyap Chamarthy wrote:
On Tue, Feb 17, 2015 at 07:05:13PM +1100, Tony Breeds wrote:
On Tue, Feb 17, 2015 at 09:02:57AM +0100, Michal Privoznik wrote:
Oh, I forgot to paste the API name: virDomainMigrate3
That'll narrow the search! Thanks!
Is there an upstream libvirt bug for this work? If not, can it be filed with the context from this thread for tracking purpose? I'm asking because, a few upstream OpenStack Nova folks expressed interest on IRC to track this work.
URL to file:
https://bugzilla.redhat.com/bugzilla/enter_bug.cgi?product=Virtualization%20Tools&component=libvirt
Done. https://bugzilla.redhat.com/show_bug.cgi?id=1203032 I called it out as a feature rather than bug but there is a place to track it. This work is still important to me (and openstack) timing is such that I got pulled into something that has delayed my start date on this work. I've been contacted by a couple of people saying they'd benefit form the work even though they do not use openstack. Yours Tony.
participants (4)
-
Daniel P. Berrange -
Kashyap Chamarthy -
Michal Privoznik -
Tony Breeds