Re: [libvirt] Tunnelled post-copy live migration - support of bidirectional communication over the tunnel
Hello,
thank you for the clarification.
However, right now I still cannot see any traffic going back from the
stream (stream doesn't become readable on the source side). I still
struggle with the overall tunnelling design - especially on the
destination side. I suppose following scheme where unidirectional pipes
were replaced by sockets:
(souce QEMU) sock1 ----- sock2 === A TUNNEL (STREAM) B === sock3 -----
sock4(destination QEMU)
I think the source side is quite clear Sock1 and sock2 are socket pair
on the source host. Data from src to dest are sent just by reading from
sock2 and and sending to the A-end of the stream.
Sock3 and sock4 is the socketpair on the destination side. The part that
confuses me is the destination side. How is the traffic handled there?
In the 'qemuMigrationPrepareAny' around here [1]. The fd of sock3 is
given to the stream and the fd of the sock4 to the destination QEMU .
Does it mean, that All traffic sent to the A-end of the stream goes all
the way to the destination QEMU (sock4) with any other handling? (At
least it seems to be that case.)
And the same question for the other direction. Does it mean that all
data written by the destination QEMU to the sock4 should automatically
appear on the A-end of the tunnel? Or is there a need to add a logic to
add code that reads from sock3 and send it to B-end of the stream?
Overally I am not very clear with the relationship of B-end of the
tunnel and fd of sock(3).
[1]http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration....
Thank you very much for any suggestions.
Regards,
Vojtech
On 15.01.2015 14:56, Daniel P. Berrange wrote:
On Thu, Jan 15, 2015 at 02:45:13PM +0100, Vojtech Cima wrote:
> Hello everyone,
>
> we are very interested about the post-copy live migration mechanism in
> libvirt [1] using QEMU. Mentioning post-copy in this context means the setup
> with the first full iteration of standard pre-copy and then switching to the
> post-copy. VIR_MIGRATE_ENABLE_POSTCOPY, VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY
> together with VIR_MIGRATE_TUNNELLED flag.
>
> Recently we have noticed the issue that post-copy live migration doesn't
> work over the tunnel since since current tunnel implementation in libvirt
> supports only uni-directional communication. I spent some time trying to
> enable bidirectional communication over the tunnel but still facing several
> issues and I would appreciate any hints how to move forward.
>
> The idea is to add a function (probably a callback function) responsible for
> handling the traffic in the opposite direction (dest->src) reading from the
> virStream and writing to the QEMU [2]. To be able to do so I replaced the
> current 'pipe' calls with the 'socketpair' calls to create
bidirectional
> paths.
>
> Now, trying to use tunnelled post-copy live migration, it gets initiated,
> there is some traffic going from source to destination (obviously the
> pre-copy phase) but at some point it gets stuck on the 'saferead' function
> [3] which is now blocking. Can this issue issue be linked with the default
> socket blocking policy?
> Trying to use standard 'read' function instead of 'saferead' results
that
> the lower data transfers and it seems it's missing significant part of
> information.
Since the code was historically unidirectional, the qemuMigrationIOFunc()
method used the virStream object in blocking I/O mode, and used a simple
loop to read from the pipe / write to the stream. If you want to do
bi-directional transfer that won't be possible - you'll need to put the
stream into fully non-blocking mode and use event callbacks to deal with
all I/O. This also means you can't use saferead either. The code in
tools/virsh-console.c is a reasonably good example of how to use a
stream in non-blocking mode doing bi-directional I/O between the stream
and regular file descriptors. The virsh-console uses stdin/stdout but
you'd just use your socketpair instead.
> Another point is that we suppose that the virStream on its own supports
> bidirectional communication, is that correct?
Yes, absolutely.
> [1] https://github.com/orbitfp7/qemu/tree/wp3-postcopy
>
> [2]
http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;...
>
> [3]
http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;...
>
> Thank you very much for any ideas on this topic.
Regards,
Daniel