Re: [libvirt] QEMU interfaces for image streaming and post-copy block migration
On 09/07/2010 10:03 AM, Daniel P. Berrange wrote:
On Tue, Sep 07, 2010 at 08:41:44AM -0500, Anthony Liguori wrote:
> Hi,
>
> We've got copy-on-read and image streaming working in QED and before
> going much further, I wanted to bounce some interfaces off of the
> libvirt folks to make sure our final interface makes sense.
>
> Here's the basic idea:
>
[snip]
> A related topic is block migration. Today we support pre-copy migration
> which means we transfer the block device and then do a live migration.
> Another approach is to do a live migration, and on the source, run a
> block server using image streaming on the destination to move the device.
>
> With QED, to implement this one would:
>
> 1) launch qemu-nbd on the source while the guest is running
> 2) create a qed file on the destination with copy-on-read enabled and a
> backing file using nbd: to point to the source qemu-nbd
> 3) run qemu -incoming on the destination with the qed file
> 4) execute the migration
> 5) when migration completes, begin streaming on the destination to
> complete the copy
> 6) when the streaming is complete, shut down the qemu-nbd instance on
> the source
>
IMHO, adding further network sockets is the one thing we absolutely
don't want to do to migration. I don't much like the idea of launching
extra daemons either.
One of the use cases I'm trying to accommodate is migration to free
resources. By launching a qemu-nbd daemon, we can kill the source qemu
process and free up all of the associated memory.
> This is a bit involved and we could potentially automate some of
this in
> qemu by launching qemu-nbd and providing commands to do some of this.
> Again though, I think the question is what type of interfaces would
> libvirt prefer? Low level interfaces + recipes on how to do high level
> things or higher level interfaces?
>
I think it should be done entirely within the main QEMU migration
socket. I know this isn't possible with the current impl, since it
is unidirectional, preventing the target sending the source requests
for specific data blocks. If we made migration socket bi-directional
I think we could do it all within qemu with no external helpers
or extra sockets
1. Create empty qed file on the destination with copy on read
enable backing file pointing to a special 'migrate:' protocol
Why not just point migration and nbd to a unix domain socket and then
multiplex the two protocols at a higher level?
2. Run qemu -incoming on the destination with with the qed file
3. execute the migration
4. when migration completes, target QEMU continues streaming blocks
from the soruce qemu.
5. when streaming is complete, source qemu can shutdown.
Both your original proposal and mine here seem to have a pretty
bad failure scenario though. After the cut-over point where the
VM cpus start running on the destination QEMU, AFAICT, any failure
on the source before block streaming complete leaves you dead in
the water. The source VM no longer has up2date RAM contents and
the destination VM does not yet have a complete disk image.
Yes. It's a trade off. However, pre-copy doesn't really change your
likelihood of catastrophic failure because if you were going to fail in
the source, it was going to happen before you completed the block
transfer anyway.
The advantage of post-copy is that you immediately free resources on the
source so as a reaction to pressure from overcommit, it's tremendously
useful.
I still think pre-copy has it's place though.
Regards,
Anthony Liguori
Regards,
Daniel