[libvirt] Tunnelled post-copy live migration - support of bidirectional communication over the tunnel
Hello everyone, we are very interested about the post-copy live migration mechanism in libvirt [1] using QEMU. Mentioning post-copy in this context means the setup with the first full iteration of standard pre-copy and then switching to the post-copy. VIR_MIGRATE_ENABLE_POSTCOPY, VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY together with VIR_MIGRATE_TUNNELLED flag. Recently we have noticed the issue that post-copy live migration doesn't work over the tunnel since since current tunnel implementation in libvirt supports only uni-directional communication. I spent some time trying to enable bidirectional communication over the tunnel but still facing several issues and I would appreciate any hints how to move forward. The idea is to add a function (probably a callback function) responsible for handling the traffic in the opposite direction (dest->src) reading from the virStream and writing to the QEMU [2]. To be able to do so I replaced the current 'pipe' calls with the 'socketpair' calls to create bidirectional paths. Now, trying to use tunnelled post-copy live migration, it gets initiated, there is some traffic going from source to destination (obviously the pre-copy phase) but at some point it gets stuck on the 'saferead' function [3] which is now blocking. Can this issue issue be linked with the default socket blocking policy? Trying to use standard 'read' function instead of 'saferead' results that the lower data transfers and it seems it's missing significant part of information. Another point is that we suppose that the virStream on its own supports bidirectional communication, is that correct? [1] https://github.com/orbitfp7/qemu/tree/wp3-postcopy [2] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;h=0... [3] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;h=0... Thank you very much for any ideas on this topic. Regards, Vojtech
On Thu, Jan 15, 2015 at 02:45:13PM +0100, Vojtech Cima wrote:
Hello everyone,
we are very interested about the post-copy live migration mechanism in libvirt [1] using QEMU. Mentioning post-copy in this context means the setup with the first full iteration of standard pre-copy and then switching to the post-copy. VIR_MIGRATE_ENABLE_POSTCOPY, VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY together with VIR_MIGRATE_TUNNELLED flag.
Recently we have noticed the issue that post-copy live migration doesn't work over the tunnel since since current tunnel implementation in libvirt supports only uni-directional communication. I spent some time trying to enable bidirectional communication over the tunnel but still facing several issues and I would appreciate any hints how to move forward.
The idea is to add a function (probably a callback function) responsible for handling the traffic in the opposite direction (dest->src) reading from the virStream and writing to the QEMU [2]. To be able to do so I replaced the current 'pipe' calls with the 'socketpair' calls to create bidirectional paths.
Now, trying to use tunnelled post-copy live migration, it gets initiated, there is some traffic going from source to destination (obviously the pre-copy phase) but at some point it gets stuck on the 'saferead' function [3] which is now blocking. Can this issue issue be linked with the default socket blocking policy? Trying to use standard 'read' function instead of 'saferead' results that the lower data transfers and it seems it's missing significant part of information.
Since the code was historically unidirectional, the qemuMigrationIOFunc() method used the virStream object in blocking I/O mode, and used a simple loop to read from the pipe / write to the stream. If you want to do bi-directional transfer that won't be possible - you'll need to put the stream into fully non-blocking mode and use event callbacks to deal with all I/O. This also means you can't use saferead either. The code in tools/virsh-console.c is a reasonably good example of how to use a stream in non-blocking mode doing bi-directional I/O between the stream and regular file descriptors. The virsh-console uses stdin/stdout but you'd just use your socketpair instead.
Another point is that we suppose that the virStream on its own supports bidirectional communication, is that correct?
Yes, absolutely.
[1] https://github.com/orbitfp7/qemu/tree/wp3-postcopy
[2] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;h=0...
[3] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;h=0...
Thank you very much for any ideas on this topic.
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Hello, thank you for the clarification. However, right now I still cannot see any traffic going back from the stream (stream doesn't become readable on the source side). I still struggle with the overall tunnelling design - especially on the destination side. I suppose following scheme where unidirectional pipes were replaced by sockets: (souce QEMU) sock1 ----- sock2 === A TUNNEL (STREAM) B === sock3 ----- sock4(destination QEMU) I think the source side is quite clear Sock1 and sock2 are socket pair on the source host. Data from src to dest are sent just by reading from sock2 and and sending to the A-end of the stream. Sock3 and sock4 is the socketpair on the destination side. The part that confuses me is the destination side. How is the traffic handled there? In the 'qemuMigrationPrepareAny' around here [1]. The fd of sock3 is given to the stream and the fd of the sock4 to the destination QEMU . Does it mean, that All traffic sent to the A-end of the stream goes all the way to the destination QEMU (sock4) with any other handling? (At least it seems to be that case.) And the same question for the other direction. Does it mean that all data written by the destination QEMU to the sock4 should automatically appear on the A-end of the tunnel? Or is there a need to add a logic to add code that reads from sock3 and send it to B-end of the stream? Overally I am not very clear with the relationship of B-end of the tunnel and fd of sock(3). [1]http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;h=0... Thank you very much for any suggestions. Regards, Vojtech On 15.01.2015 14:56, Daniel P. Berrange wrote:
On Thu, Jan 15, 2015 at 02:45:13PM +0100, Vojtech Cima wrote:
Hello everyone,
we are very interested about the post-copy live migration mechanism in libvirt [1] using QEMU. Mentioning post-copy in this context means the setup with the first full iteration of standard pre-copy and then switching to the post-copy. VIR_MIGRATE_ENABLE_POSTCOPY, VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY together with VIR_MIGRATE_TUNNELLED flag.
Recently we have noticed the issue that post-copy live migration doesn't work over the tunnel since since current tunnel implementation in libvirt supports only uni-directional communication. I spent some time trying to enable bidirectional communication over the tunnel but still facing several issues and I would appreciate any hints how to move forward.
The idea is to add a function (probably a callback function) responsible for handling the traffic in the opposite direction (dest->src) reading from the virStream and writing to the QEMU [2]. To be able to do so I replaced the current 'pipe' calls with the 'socketpair' calls to create bidirectional paths.
Now, trying to use tunnelled post-copy live migration, it gets initiated, there is some traffic going from source to destination (obviously the pre-copy phase) but at some point it gets stuck on the 'saferead' function [3] which is now blocking. Can this issue issue be linked with the default socket blocking policy? Trying to use standard 'read' function instead of 'saferead' results that the lower data transfers and it seems it's missing significant part of information. Since the code was historically unidirectional, the qemuMigrationIOFunc() method used the virStream object in blocking I/O mode, and used a simple loop to read from the pipe / write to the stream. If you want to do bi-directional transfer that won't be possible - you'll need to put the stream into fully non-blocking mode and use event callbacks to deal with all I/O. This also means you can't use saferead either. The code in tools/virsh-console.c is a reasonably good example of how to use a stream in non-blocking mode doing bi-directional I/O between the stream and regular file descriptors. The virsh-console uses stdin/stdout but you'd just use your socketpair instead.
Another point is that we suppose that the virStream on its own supports bidirectional communication, is that correct? Yes, absolutely.
[1] https://github.com/orbitfp7/qemu/tree/wp3-postcopy
[2] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;h=0...
[3] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;h=0...
Thank you very much for any ideas on this topic. Regards, Daniel
On 15 Jan 2015, at 14:56, Daniel P. Berrange <berrange@redhat.com> wrote:
On Thu, Jan 15, 2015 at 02:45:13PM +0100, Vojtech Cima wrote:
Hello everyone,
we are very interested about the post-copy live migration mechanism in libvirt [1] using QEMU. Mentioning post-copy in this context means the setup with the first full iteration of standard pre-copy and then switching to the post-copy. VIR_MIGRATE_ENABLE_POSTCOPY, VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY together with VIR_MIGRATE_TUNNELLED flag.
Recently we have noticed the issue that post-copy live migration doesn't work over the tunnel since since current tunnel implementation in libvirt supports only uni-directional communication. I spent some time trying to enable bidirectional communication over the tunnel but still facing several issues and I would appreciate any hints how to move forward.
The idea is to add a function (probably a callback function) responsible for handling the traffic in the opposite direction (dest->src) reading from the virStream and writing to the QEMU [2]. To be able to do so I replaced the current 'pipe' calls with the 'socketpair' calls to create bidirectional paths.
Now, trying to use tunnelled post-copy live migration, it gets initiated, there is some traffic going from source to destination (obviously the pre-copy phase) but at some point it gets stuck on the 'saferead' function [3] which is now blocking. Can this issue issue be linked with the default socket blocking policy? Trying to use standard 'read' function instead of 'saferead' results that the lower data transfers and it seems it's missing significant part of information.
Since the code was historically unidirectional, the qemuMigrationIOFunc() method used the virStream object in blocking I/O mode, and used a simple loop to read from the pipe / write to the stream. If you want to do bi-directional transfer that won't be possible - you'll need to put the stream into fully non-blocking mode and use event callbacks to deal with all I/O. This also means you can't use saferead either. The code in tools/virsh-console.c is a reasonably good example of how to use a stream in non-blocking mode doing bi-directional I/O between the stream and regular file descriptors. The virsh-console uses stdin/stdout but you'd just use your socketpair instead.
Thanks for your hints. I have created a patch handling the bi-directional communication using callbacks in 'src/qemu/qemu_migration.c' and it seems to be working fine although it handles migration traffic only on the source host. The migration traffic on the destination host is handled in 'daemonStreamEvent' callback in ‘daemon/stream.c’. There is a stream object representing connection to/from destination QEMU client. The problem in current implementation is that callback is setup to check for VIR_STREAM_EVENT_WRITABLE most of the time (only if the transmit parameter allowed the callback listens for VIR_STREAM_EVENT_READABLE but just once. Since immediately after it reads one msg from QEMU callback is updated back to trigger only on WRITABLE event again). Do you think there can be any problem if this callback is setup to be triggered on both events all the time in order to handle migration I/O equally?
Another point is that we suppose that the virStream on its own supports bidirectional communication, is that correct?
Yes, absolutely.
[1] https://github.com/orbitfp7/qemu/tree/wp3-postcopy
[2] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;h=0...
[3] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_migration.c;h=0...
Thank you very much for any ideas on this topic.
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Regards, Vojtech
participants (2)
-
Daniel P. Berrange -
Vojtech Cima