The provided patchset implements NBD disk migration over a tunnelled
connection provided by libvirt.
The migration source instructs QEMU to NBD mirror drives into the provided
UNIX socket. These connections and all the data are then tunnelled to the
destination using newly introduced RPC call. The migration destination
implements a driver method that connects the tunnelled stream to the QEMU's
NBD destination.
The detailed scheme is the following:
PREPARE
1. Migration destination starts QEMU's NBD server listening on a UNIX socket
using the `nbd-server-add` monitor command and tells NBD to accept listed
disks via code added to qemuMigrationStartNBDServer that calls introduced
qemuMonitorNBDServerStartUnix monitor function.
PERFORM
2. Migration source creates a UNIX socket that is later used as NBDs
destination in `drive-mirror` monitor command.
This is implemented as a call to virNetSocketNewListenUnix from
doTunnelMigrate.
3. Source starts IOThread that polls on the UNIX socket, accepting every
incoming QEMU connection.
This is done by adding a new pollfd in the poll(2) call in
qemuMigrationIOFunc that calls introduced qemuNBDTunnelAcceptAndPipe
function.
4. The qemuNBDTunnelAcceptAndPipe function accepts the connection and creates
two virStream's. One is `local` that is later associated with just accepted
connection using virFDStreamOpen. Second is `remote` that is later
tunnelled to the remote destination stream.
The `local` stream is converted to a virFDStreamDrv stream using the
virFDStreamOpen call on the fd returned by accept(2).
The `remote` stream is associated with a stream on the destination in
the way similar to used by PrepareTunnel3* function. That is, the
virDomainMigrateOpenTunnel function called on the destination
connection object. The virDomainMigrateOpenTunnel calls remote driver's
handler remoteDomainMigrateOpenTunnel that makes DOMAIN_MIGRATE_OPEN_TUNNEL
call to the destination host. The code in remoteDomainMigrateOpenTunnel
ties passed virStream object to a virStream on the destination host via
remoteStreamDrv driver. The remote driver handles stream's IO by tunnelling
data through the RPC connection.
The qemuNBDTunnelAcceptAndPipe at last assigns both streams the same event
callback qemuMigrationPipeEvent. Its job is to track statuses of the
streams doing IO whenever it is necessary.
5. Source starts the drive mirroring using the qemuMigrationDriveMirror func.
The function instructs QEMU to mirror drives to the UNIX socket that thread
listens on.
Since it is necessary for the mirror driving to get into the 'synchronized'
state, where writes go to both destinations simultaneously, before
continuing VM migration, the thread serving the connections must be
started earlier.
6. When the connection to a UNIX socket on the migration source is made
the DOMAIN_MIGRATE_OPEN_TUNNEL proc is called on the migration destination.
The handler of this code calls virDomainMigrateOpenTunnel which calls
qemuMigrationOpenNBDTunnel by the means of qemuDomainMigrateOpenTunnel.
The qemuMigrationOpenNBDTunnel connects the stream linked to a source's
stream to the NBD's UNIX socket on the migration destination side.
7. The rest of the disk migration occurs semimagically: virStream* APIs tunnel
data in both directions. This is done by qemuMigrationPipeEvent event
callback set for both streams.
The order of the patches is roughly the following:
* First, the RPC machinery and remote driver's virDrvDomainMigrateOpenTunnel
implementation are added.
* Then, the source-side of the protocol is implemented: code listening
on a UNIX socket is added, DriveMirror is enhanced to instruct QEMU to
`drive-mirror` here and starting IOThread driving the tunneling sooner.
* After that, the destination-side of the protocol is implemented:
the qemuMonitorNBDServerStartUnix added and qemuMigrationStartNBDServer
enhanced to call it. The qemuDomainMigrateOpenTunnel is implemented
along with qemuMigrationOpenNBDTunnel that does the real job.
* Finally, the code blocking NBD migration for tunnelled migration is
removed.
Pavel Boldin (21):
rpc: add DOMAIN_MIGRATE_OPEN_TUNNEL proc
driver: add virDrvDomainMigrateOpenTunnel
remote_driver: introduce virRemoteClientNew
remote_driver: add remoteDomainMigrateOpenTunnel
domain: add virDomainMigrateOpenTunnel
domain: add virDomainMigrateTunnelFlags
remote: impl remoteDispatchDomainMigrateOpenTunnel
qemu: migration: src: add nbd tunnel socket data
qemu: migration: src: nbdtunnel unix socket
qemu: migration: src: qemu `drive-mirror` to UNIX
qemu: migration: src: qemuSock for running thread
qemu: migration: src: add NBD unixSock to iothread
qemu: migration: src: qemuNBDTunnelAcceptAndPipe
qemu: migration: src: stream piping
qemu: monitor: add qemuMonitorNBDServerStartUnix
qemu: migration: dest: nbd-server to UNIX sock
qemu: migration: dest: qemuMigrationOpenTunnel
qemu: driver: add qemuDomainMigrateOpenTunnel
qemu: migration: dest: qemuMigrationOpenNBDTunnel
qemu: migration: allow NBD tunneling migration
apparmor: fix tunnelmigrate permissions
daemon/remote.c | 50 ++++
docs/apibuild.py | 1 +
docs/hvsupport.pl | 1 +
include/libvirt/libvirt-domain.h | 3 +
src/driver-hypervisor.h | 8 +
src/libvirt-domain.c | 43 ++++
src/libvirt_internal.h | 6 +
src/libvirt_private.syms | 1 +
src/qemu/qemu_driver.c | 24 ++
src/qemu/qemu_migration.c | 495 +++++++++++++++++++++++++++++++++------
src/qemu/qemu_migration.h | 6 +
src/qemu/qemu_monitor.c | 12 +
src/qemu/qemu_monitor.h | 2 +
src/qemu/qemu_monitor_json.c | 35 +++
src/qemu/qemu_monitor_json.h | 2 +
src/remote/remote_driver.c | 91 +++++--
src/remote/remote_protocol.x | 19 +-
src/remote_protocol-structs | 8 +
src/security/virt-aa-helper.c | 4 +-
19 files changed, 719 insertions(+), 92 deletions(-)
--
1.9.1