[libvirt] Tunnelled post-copy live migration - support of bidirectional communication over the tunnel
7:45 a.m.
Hello everyone,
we are very interested about the post-copy live migration mechanism in
libvirt [1] using QEMU. Mentioning post-copy in this context means the
setup with the first full iteration of standard pre-copy and then
switching to the post-copy. VIR_MIGRATE_ENABLE_POSTCOPY,
VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY together with VIR_MIGRATE_TUNNELLED flag.
Recently we have noticed the issue that post-copy live migration doesn't
work over the tunnel since since current tunnel implementation in
libvirt supports only uni-directional communication. I spent some time
trying to enable bidirectional communication over the tunnel but still
facing several issues and I would appreciate any hints how to move forward.
The idea is to add a function (probably a callback function) responsible
for handling the traffic in the opposite direction (dest->src) reading
from the virStream and writing to the QEMU [2]. To be able to do so I
replaced the current 'pipe' calls with the 'socketpair' calls to create
bidirectional paths.
Now, trying to use tunnelled post-copy live migration, it gets
initiated, there is some traffic going from source to destination
(obviously the pre-copy phase) but at some point it gets stuck on the
'saferead' function [3] which is now blocking. Can this issue issue be
linked with the default socket blocking policy?
Trying to use standard 'read' function instead of 'saferead' results
that the lower data transfers and it seems it's missing significant part
of information.
Another point is that we suppose that the virStream on its own supports
bidirectional communication, is that correct?
[1] https://github.com/orbitfp7/qemu/tree/wp3-postcopy
[2]
http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;...
[3]
http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;...
Thank you very much for any ideas on this topic.
Regards,
Vojtech
7:56 a.m.
On Thu, Jan 15, 2015 at 02:45:13PM +0100, Vojtech Cima wrote:
Hello everyone,
we are very interested about the post-copy live migration mechanism in
libvirt [1] using QEMU. Mentioning post-copy in this context means the setup
with the first full iteration of standard pre-copy and then switching to the
post-copy. VIR_MIGRATE_ENABLE_POSTCOPY, VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY
together with VIR_MIGRATE_TUNNELLED flag.
Recently we have noticed the issue that post-copy live migration doesn't
work over the tunnel since since current tunnel implementation in libvirt
supports only uni-directional communication. I spent some time trying to
enable bidirectional communication over the tunnel but still facing several
issues and I would appreciate any hints how to move forward.
The idea is to add a function (probably a callback function) responsible for
handling the traffic in the opposite direction (dest->src) reading from the
virStream and writing to the QEMU [2]. To be able to do so I replaced the
current 'pipe' calls with the 'socketpair' calls to create bidirectional
paths.
Now, trying to use tunnelled post-copy live migration, it gets initiated,
there is some traffic going from source to destination (obviously the
pre-copy phase) but at some point it gets stuck on the 'saferead' function
[3] which is now blocking. Can this issue issue be linked with the default
socket blocking policy?
Trying to use standard 'read' function instead of 'saferead' results
that
the lower data transfers and it seems it's missing significant part of
information.
Since the code was historically unidirectional, the qemuMigrationIOFunc()
method used the virStream object in blocking I/O mode, and used a simple
loop to read from the pipe / write to the stream. If you want to do
bi-directional transfer that won't be possible - you'll need to put the
stream into fully non-blocking mode and use event callbacks to deal with
all I/O. This also means you can't use saferead either. The code in
tools/virsh-console.c is a reasonably good example of how to use a
stream in non-blocking mode doing bi-directional I/O between the stream
and regular file descriptors. The virsh-console uses stdin/stdout but
you'd just use your socketpair instead.
Another point is that we suppose that the virStream on its own
supports
bidirectional communication, is that correct?
Yes, absolutely.
[1] https://github.com/orbitfp7/qemu/tree/wp3-postcopy
[2]
http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;...
[3]
http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;...
Thank you very much for any ideas on this topic.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
3:58 a.m.
Hello,
thank you for the clarification.
However, right now I still cannot see any traffic going back from the
stream (stream doesn't become readable on the source side). I still
struggle with the overall tunnelling design - especially on the
destination side. I suppose following scheme where unidirectional pipes
were replaced by sockets:
(souce QEMU) sock1 ----- sock2 === A TUNNEL (STREAM) B === sock3 -----
sock4(destination QEMU)
I think the source side is quite clear Sock1 and sock2 are socket pair
on the source host. Data from src to dest are sent just by reading from
sock2 and and sending to the A-end of the stream.
Sock3 and sock4 is the socketpair on the destination side. The part that
confuses me is the destination side. How is the traffic handled there?
In the 'qemuMigrationPrepareAny' around here [1]. The fd of sock3 is
given to the stream and the fd of the sock4 to the destination QEMU .
Does it mean, that All traffic sent to the A-end of the stream goes all
the way to the destination QEMU (sock4) with any other handling? (At
least it seems to be that case.)
And the same question for the other direction. Does it mean that all
data written by the destination QEMU to the sock4 should automatically
appear on the A-end of the tunnel? Or is there a need to add a logic to
add code that reads from sock3 and send it to B-end of the stream?
Overally I am not very clear with the relationship of B-end of the
tunnel and fd of sock(3).
[1]http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration....
Thank you very much for any suggestions.
Regards,
Vojtech
On 15.01.2015 14:56, Daniel P. Berrange wrote:
On Thu, Jan 15, 2015 at 02:45:13PM +0100, Vojtech Cima wrote:
> Hello everyone,
>
> we are very interested about the post-copy live migration mechanism in
> libvirt [1] using QEMU. Mentioning post-copy in this context means the setup
> with the first full iteration of standard pre-copy and then switching to the
> post-copy. VIR_MIGRATE_ENABLE_POSTCOPY, VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY
> together with VIR_MIGRATE_TUNNELLED flag.
>
> Recently we have noticed the issue that post-copy live migration doesn't
> work over the tunnel since since current tunnel implementation in libvirt
> supports only uni-directional communication. I spent some time trying to
> enable bidirectional communication over the tunnel but still facing several
> issues and I would appreciate any hints how to move forward.
>
> The idea is to add a function (probably a callback function) responsible for
> handling the traffic in the opposite direction (dest->src) reading from the
> virStream and writing to the QEMU [2]. To be able to do so I replaced the
> current 'pipe' calls with the 'socketpair' calls to create
bidirectional
> paths.
>
> Now, trying to use tunnelled post-copy live migration, it gets initiated,
> there is some traffic going from source to destination (obviously the
> pre-copy phase) but at some point it gets stuck on the 'saferead' function
> [3] which is now blocking. Can this issue issue be linked with the default
> socket blocking policy?
> Trying to use standard 'read' function instead of 'saferead' results
that
> the lower data transfers and it seems it's missing significant part of
> information.
Since the code was historically unidirectional, the qemuMigrationIOFunc()
method used the virStream object in blocking I/O mode, and used a simple
loop to read from the pipe / write to the stream. If you want to do
bi-directional transfer that won't be possible - you'll need to put the
stream into fully non-blocking mode and use event callbacks to deal with
all I/O. This also means you can't use saferead either. The code in
tools/virsh-console.c is a reasonably good example of how to use a
stream in non-blocking mode doing bi-directional I/O between the stream
and regular file descriptors. The virsh-console uses stdin/stdout but
you'd just use your socketpair instead.
> Another point is that we suppose that the virStream on its own supports
> bidirectional communication, is that correct?
Yes, absolutely.
> [1] https://github.com/orbitfp7/qemu/tree/wp3-postcopy
>
> [2]
http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;...
>
> [3]
http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;...
>
> Thank you very much for any ideas on this topic.
Regards,
Daniel
11:29 a.m.
On 15 Jan 2015, at 14:56, Daniel P. Berrange
<berrange(a)redhat.com> wrote:
On Thu, Jan 15, 2015 at 02:45:13PM +0100, Vojtech Cima wrote:
> Hello everyone,
>
> we are very interested about the post-copy live migration mechanism in
> libvirt [1] using QEMU. Mentioning post-copy in this context means the setup
> with the first full iteration of standard pre-copy and then switching to the
> post-copy. VIR_MIGRATE_ENABLE_POSTCOPY, VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY
> together with VIR_MIGRATE_TUNNELLED flag.
>
> Recently we have noticed the issue that post-copy live migration doesn't
> work over the tunnel since since current tunnel implementation in libvirt
> supports only uni-directional communication. I spent some time trying to
> enable bidirectional communication over the tunnel but still facing several
> issues and I would appreciate any hints how to move forward.
>
> The idea is to add a function (probably a callback function) responsible for
> handling the traffic in the opposite direction (dest->src) reading from the
> virStream and writing to the QEMU [2]. To be able to do so I replaced the
> current 'pipe' calls with the 'socketpair' calls to create
bidirectional
> paths.
>
> Now, trying to use tunnelled post-copy live migration, it gets initiated,
> there is some traffic going from source to destination (obviously the
> pre-copy phase) but at some point it gets stuck on the 'saferead' function
> [3] which is now blocking. Can this issue issue be linked with the default
> socket blocking policy?
> Trying to use standard 'read' function instead of 'saferead' results
that
> the lower data transfers and it seems it's missing significant part of
> information.
Since the code was historically unidirectional, the qemuMigrationIOFunc()
method used the virStream object in blocking I/O mode, and used a simple
loop to read from the pipe / write to the stream. If you want to do
bi-directional transfer that won't be possible - you'll need to put the
stream into fully non-blocking mode and use event callbacks to deal with
all I/O. This also means you can't use saferead either. The code in
tools/virsh-console.c is a reasonably good example of how to use a
stream in non-blocking mode doing bi-directional I/O between the stream
and regular file descriptors. The virsh-console uses stdin/stdout but
you'd just use your socketpair instead.
Thanks for your hints. I have created a patch handling the bi-directional communication
using callbacks in 'src/qemu/qemu_migration.c' and it seems to be working fine
although it handles migration traffic only on the source host.
The migration traffic on the destination host is handled in 'daemonStreamEvent'
callback in ‘daemon/stream.c’. There is a stream object representing connection to/from
destination QEMU client. The problem in current implementation is that callback is setup
to check for VIR_STREAM_EVENT_WRITABLE most of the time (only if the transmit parameter
allowed the callback listens for VIR_STREAM_EVENT_READABLE but just once. Since
immediately after it reads one msg from QEMU callback is updated back to trigger only on
WRITABLE event again).
Do you think there can be any problem if this callback is setup to be triggered on both
events all the time in order to handle migration I/O equally?
> Another point is that we suppose that the virStream on its own supports
> bidirectional communication, is that correct?
Yes, absolutely.
>
> [1] https://github.com/orbitfp7/qemu/tree/wp3-postcopy
>
> [2]
http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;...
>
> [3]
http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;...
>
> Thank you very much for any ideas on this topic.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Regards,
Vojtech
3599
days inactive
3629
days old
3 comments
2 participants
participants (2)
-
Daniel P. Berrange -
Vojtech Cima