[libvirt] [RFC PATCHv2 0/3] outgoing migration via fd: rather than exec:
Posted as an RFC because there are still two open bugs to be solved. I've tested that with this installed, I was able to do 'virsh save domain file' without compression to a local file using qemu fd: migration. 1. I'm working on making virFileOperation use SCM_RIGHTS so that we can pass an arbitrary fd from the child back to the parent. This is necessary for supporting root-squash NFS domain save via fd, since a child cannot consistently modify the appropriate locks necessary to issue a monitor command (the current virFileOperation implementation guarantees that the hook will execute on the right fd, but does not guarantee whether the operation is in the parent or a child, and thus the operation is not permitted to modify any state in the parent other than by its return value). The current state of the patch just falls back to exec: migration if the parent process can't reopen the fd. 2. I also need to test for qemu capabilities, as qemu 0.11.0 does _not_ support fd: migration, so the old exec: migration path must remain. But checking for capabilities is a bear in itself, since we should be caching qemu capabilities when a domain is started, and not calling 'qemu -help' for every subsequent command that needs to know. Eric Blake (3): qemu: use lighter-weight fd:n on incoming tunneled migration qemu: support migration to fd qemu: allow simple domain save to use fd: protocol src/qemu/qemu_driver.c | 16 ++++++++++++-- src/qemu/qemu_migration.c | 44 +++++++++++------------------------------ src/qemu/qemu_monitor.c | 22 +++++++++++++++++++++ src/qemu/qemu_monitor.h | 4 +++ src/qemu/qemu_monitor_json.c | 21 +++++++++++++++++++- src/qemu/qemu_monitor_json.h | 6 ++++- src/qemu/qemu_monitor_text.c | 24 +++++++++++++++++++++- src/qemu/qemu_monitor_text.h | 6 ++++- 8 files changed, 103 insertions(+), 40 deletions(-) -- 1.7.4
Outgoing migration still uses a Unix socket and or exec netcat until the next patch. * src/qemu/qemu_migration.c (qemuMigrationPrepareTunnel): Replace Unix socket with simpler pipe. Suggested by Paolo Bonzini. --- v1 was ack'd here but not yet applied: https://www.redhat.com/archives/libvir-list/2011-January/msg00742.html changes in v2: rebase to latest upstream src/qemu/qemu_migration.c | 44 ++++++++++++-------------------------------- 1 files changed, 12 insertions(+), 32 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 822cb18..dc51bc8 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -239,11 +239,10 @@ qemuMigrationPrepareTunnel(struct qemud_driver *driver, { virDomainDefPtr def = NULL; virDomainObjPtr vm = NULL; - char *migrateFrom; virDomainEventPtr event = NULL; int ret = -1; int internalret; - char *unixfile = NULL; + int dataFD[2] = { -1, -1 }; virBitmapPtr qemuCaps = NULL; qemuDomainObjPrivatePtr priv = NULL; struct timeval now; @@ -289,12 +288,11 @@ qemuMigrationPrepareTunnel(struct qemud_driver *driver, /* Domain starts inactive, even if the domain XML had an id field. */ vm->def->id = -1; - if (virAsprintf(&unixfile, "%s/qemu.tunnelmigrate.dest.%s", - driver->libDir, vm->def->name) < 0) { - virReportOOMError(); + if (pipe(dataFD) < 0) { + virReportSystemError(errno, "%s", + _("cannot create pipe for tunnelled migration")); goto endjob; } - unlink(unixfile); /* check that this qemu version supports the interactive exec */ if (qemuCapsExtractVersionInfo(vm->def->emulator, vm->def->os.arch, @@ -304,25 +302,11 @@ qemuMigrationPrepareTunnel(struct qemud_driver *driver, vm->def->emulator); goto endjob; } - if (qemuCapsGet(qemuCaps, QEMU_CAPS_MIGRATE_QEMU_UNIX)) - internalret = virAsprintf(&migrateFrom, "unix:%s", unixfile); - else if (qemuCapsGet(qemuCaps, QEMU_CAPS_MIGRATE_QEMU_EXEC)) - internalret = virAsprintf(&migrateFrom, "exec:nc -U -l %s", unixfile); - else { - qemuReportError(VIR_ERR_OPERATION_FAILED, - "%s", _("Destination qemu is too old to support tunnelled migration")); - goto endjob; - } - if (internalret < 0) { - virReportOOMError(); - goto endjob; - } /* Start the QEMU daemon, with the same command-line arguments plus - * -incoming unix:/path/to/file or exec:nc -U /path/to/file + * -incoming stdin (which qemu_command might convert to exec:cat or fd:n) */ - internalret = qemuProcessStart(dconn, driver, vm, migrateFrom, true, - -1, NULL, VIR_VM_OP_MIGRATE_IN_START); - VIR_FREE(migrateFrom); + internalret = qemuProcessStart(dconn, driver, vm, "stdin", true, dataFD[1], + NULL, VIR_VM_OP_MIGRATE_IN_START); if (internalret < 0) { qemuDomainStartAudit(vm, "migrated", false); /* Note that we don't set an error here because qemuProcessStart @@ -335,9 +319,7 @@ qemuMigrationPrepareTunnel(struct qemud_driver *driver, goto endjob; } - if (virFDStreamConnectUNIX(st, - unixfile, - false) < 0) { + if (virFDStreamOpen(st, dataFD[0]) < 0) { qemuDomainStartAudit(vm, "migrated", false); qemuProcessStop(driver, vm, 0); if (!vm->persistent) { @@ -345,9 +327,8 @@ qemuMigrationPrepareTunnel(struct qemud_driver *driver, virDomainRemoveInactive(&driver->domains, vm); vm = NULL; } - virReportSystemError(errno, - _("cannot open unix socket '%s' for tunnelled migration"), - unixfile); + virReportSystemError(errno, "%s", + _("cannot pass pipe for tunnelled migration")); goto endjob; } @@ -378,9 +359,8 @@ endjob: cleanup: qemuCapsFree(qemuCaps); virDomainDefFree(def); - if (unixfile) - unlink(unixfile); - VIR_FREE(unixfile); + VIR_FORCE_CLOSE(dataFD[0]); + VIR_FORCE_CLOSE(dataFD[1]); if (vm) virDomainObjUnlock(vm); if (event) -- 1.7.4
On Wed, Mar 02, 2011 at 03:07:04PM -0700, Eric Blake wrote:
Outgoing migration still uses a Unix socket and or exec netcat until the next patch.
* src/qemu/qemu_migration.c (qemuMigrationPrepareTunnel): Replace Unix socket with simpler pipe. Suggested by Paolo Bonzini. ---
v1 was ack'd here but not yet applied: https://www.redhat.com/archives/libvir-list/2011-January/msg00742.html changes in v2: rebase to latest upstream
src/qemu/qemu_migration.c | 44 ++++++++++++-------------------------------- 1 files changed, 12 insertions(+), 32 deletions(-)
ACK Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 03/03/2011 04:54 AM, Daniel P. Berrange wrote:
On Wed, Mar 02, 2011 at 03:07:04PM -0700, Eric Blake wrote:
Outgoing migration still uses a Unix socket and or exec netcat until the next patch.
* src/qemu/qemu_migration.c (qemuMigrationPrepareTunnel): Replace Unix socket with simpler pipe. Suggested by Paolo Bonzini. ---
v1 was ack'd here but not yet applied: https://www.redhat.com/archives/libvir-list/2011-January/msg00742.html changes in v2: rebase to latest upstream
src/qemu/qemu_migration.c | 44 ++++++++++++-------------------------------- 1 files changed, 12 insertions(+), 32 deletions(-)
ACK
Self-NACK - I forgot to FD_CLOEXEC the write side of the pipe (virCommand explicitly closes it anyways, but we might as well be clean on our intentions). I'll fix that in v3. Patch 2/3 is fine as is, but no point in pushing it until there's a client. And patch 3/3 will be heavily reworked in v3 (I'm up to at least 5 separate patches, and counting). -- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org
* src/qemu/qemu_monitor.h (qemuMonitorMigrateToFd): New prototype. * src/qemu/qemu_monitor_text.h (qemuMonitorTextMigrateToFd): Likewise. * src/qemu/qemu_monitor_json.h (qemuMonitorJSONMigrateToFd): Likewise. * src/qemu/qemu_monitor.c (qemuMonitorMigrateToFd): New function. * src/qemu/qemu_monitor_text.c (qemuMonitorTextMigrateToFd): Likewise. * src/qemu/qemu_monitor_json.c (qemuMonitorJSONMigrateToFd): Likewise. --- v2: new patch src/qemu/qemu_monitor.c | 22 ++++++++++++++++++++++ src/qemu/qemu_monitor.h | 4 ++++ src/qemu/qemu_monitor_json.c | 21 ++++++++++++++++++++- src/qemu/qemu_monitor_json.h | 6 +++++- src/qemu/qemu_monitor_text.c | 24 ++++++++++++++++++++++-- src/qemu/qemu_monitor_text.h | 6 +++++- 6 files changed, 78 insertions(+), 5 deletions(-) diff --git a/src/qemu/qemu_monitor.c b/src/qemu/qemu_monitor.c index dfb1aad..312e797 100644 --- a/src/qemu/qemu_monitor.c +++ b/src/qemu/qemu_monitor.c @@ -1384,6 +1384,28 @@ int qemuMonitorGetMigrationStatus(qemuMonitorPtr mon, } +int qemuMonitorMigrateToFd(qemuMonitorPtr mon, + unsigned int flags, + int fd) +{ + int ret; + VIR_DEBUG("mon=%p fd=%d flags=%u", + mon, fd, flags); + + if (!mon) { + qemuReportError(VIR_ERR_INVALID_ARG, "%s", + _("monitor must not be NULL")); + return -1; + } + + if (mon->json) + ret = qemuMonitorJSONMigrateToFd(mon, flags, fd); + else + ret = qemuMonitorTextMigrateToFd(mon, flags, fd); + return ret; +} + + int qemuMonitorMigrateToHost(qemuMonitorPtr mon, unsigned int flags, const char *hostname, diff --git a/src/qemu/qemu_monitor.h b/src/qemu/qemu_monitor.h index 0ea1330..1a64ac0 100644 --- a/src/qemu/qemu_monitor.h +++ b/src/qemu/qemu_monitor.h @@ -260,6 +260,10 @@ typedef enum { QEMU_MONITOR_MIGRATION_FLAGS_LAST } QEMU_MONITOR_MIGRATE; +int qemuMonitorMigrateToFd(qemuMonitorPtr mon, + unsigned int flags, + int fd); + int qemuMonitorMigrateToHost(qemuMonitorPtr mon, unsigned int flags, const char *hostname, diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index e6903a1..81201ff 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -1,7 +1,7 @@ /* * qemu_monitor_json.c: interaction with QEMU monitor console * - * Copyright (C) 2006-2010 Red Hat, Inc. + * Copyright (C) 2006-2011 Red Hat, Inc. * Copyright (C) 2006 Daniel P. Berrange * * This library is free software; you can redistribute it and/or @@ -1697,6 +1697,25 @@ static int qemuMonitorJSONMigrate(qemuMonitorPtr mon, } +int qemuMonitorJSONMigrateToFd(qemuMonitorPtr mon, + unsigned int flags, + int fd) +{ + int ret; + + if (qemuMonitorJSONSendFileHandle(mon, "migrate", fd) < 0) + return -1; + + ret = qemuMonitorJSONMigrate(mon, flags, "fd:migrate"); + + if (ret < 0) { + if (qemuMonitorJSONCloseFileHandle(mon, "migrate") < 0) + VIR_WARN0("failed to close migration handle"); + } + + return ret; +} + int qemuMonitorJSONMigrateToHost(qemuMonitorPtr mon, unsigned int flags, const char *hostname, diff --git a/src/qemu/qemu_monitor_json.h b/src/qemu/qemu_monitor_json.h index 4ae472a..c2b45f3 100644 --- a/src/qemu/qemu_monitor_json.h +++ b/src/qemu/qemu_monitor_json.h @@ -1,7 +1,7 @@ /* * qemu_monitor_json.h: interaction with QEMU monitor console * - * Copyright (C) 2006-2009 Red Hat, Inc. + * Copyright (C) 2006-2009, 2011 Red Hat, Inc. * Copyright (C) 2006 Daniel P. Berrange * * This library is free software; you can redistribute it and/or @@ -104,6 +104,10 @@ int qemuMonitorJSONGetMigrationStatus(qemuMonitorPtr mon, unsigned long long *remaining, unsigned long long *total); +int qemuMonitorJSONMigrateToFd(qemuMonitorPtr mon, + unsigned int flags, + int fd); + int qemuMonitorJSONMigrateToHost(qemuMonitorPtr mon, unsigned int flags, const char *hostname, diff --git a/src/qemu/qemu_monitor_text.c b/src/qemu/qemu_monitor_text.c index 0aed690..76e0ec0 100644 --- a/src/qemu/qemu_monitor_text.c +++ b/src/qemu/qemu_monitor_text.c @@ -1,7 +1,7 @@ /* * qemu_monitor_text.c: interaction with QEMU monitor console * - * Copyright (C) 2006-2010 Red Hat, Inc. + * Copyright (C) 2006-2011 Red Hat, Inc. * Copyright (C) 2006 Daniel P. Berrange * * This library is free software; you can redistribute it and/or @@ -1256,7 +1256,8 @@ static int qemuMonitorTextMigrate(qemuMonitorPtr mon, * unknown command: migrate" */ if (strstr(info, "unknown command:")) { qemuReportError(VIR_ERR_NO_SUPPORT, - _("migration to '%s' not supported by this qemu: %s"), dest, info); + _("migration to '%s' not supported by this qemu: %s"), + dest, info); goto cleanup; } @@ -1271,6 +1272,25 @@ cleanup: return ret; } +int qemuMonitorTextMigrateToFd(qemuMonitorPtr mon, + unsigned int flags, + int fd) +{ + int ret; + + if (qemuMonitorTextSendFileHandle(mon, "migrate", fd) < 0) + return -1; + + ret = qemuMonitorTextMigrate(mon, flags, "fd:migrate"); + + if (ret < 0) { + if (qemuMonitorTextCloseFileHandle(mon, "migrate") < 0) + VIR_WARN0("failed to close migration handle"); + } + + return ret; +} + int qemuMonitorTextMigrateToHost(qemuMonitorPtr mon, unsigned int flags, const char *hostname, diff --git a/src/qemu/qemu_monitor_text.h b/src/qemu/qemu_monitor_text.h index b29dbcc..f1b53fc 100644 --- a/src/qemu/qemu_monitor_text.h +++ b/src/qemu/qemu_monitor_text.h @@ -1,7 +1,7 @@ /* * qemu_monitor_text.h: interaction with QEMU monitor console * - * Copyright (C) 2006-2009 Red Hat, Inc. + * Copyright (C) 2006-2009, 2011 Red Hat, Inc. * Copyright (C) 2006 Daniel P. Berrange * * This library is free software; you can redistribute it and/or @@ -102,6 +102,10 @@ int qemuMonitorTextGetMigrationStatus(qemuMonitorPtr mon, unsigned long long *remaining, unsigned long long *total); +int qemuMonitorTextMigrateToFd(qemuMonitorPtr mon, + unsigned int flags, + int fd); + int qemuMonitorTextMigrateToHost(qemuMonitorPtr mon, unsigned int flags, const char *hostname, -- 1.7.4
On Wed, Mar 02, 2011 at 03:07:05PM -0700, Eric Blake wrote:
* src/qemu/qemu_monitor.h (qemuMonitorMigrateToFd): New prototype. * src/qemu/qemu_monitor_text.h (qemuMonitorTextMigrateToFd): Likewise. * src/qemu/qemu_monitor_json.h (qemuMonitorJSONMigrateToFd): Likewise. * src/qemu/qemu_monitor.c (qemuMonitorMigrateToFd): New function. * src/qemu/qemu_monitor_text.c (qemuMonitorTextMigrateToFd): Likewise. * src/qemu/qemu_monitor_json.c (qemuMonitorJSONMigrateToFd): Likewise. ---
v2: new patch
src/qemu/qemu_monitor.c | 22 ++++++++++++++++++++++ src/qemu/qemu_monitor.h | 4 ++++ src/qemu/qemu_monitor_json.c | 21 ++++++++++++++++++++- src/qemu/qemu_monitor_json.h | 6 +++++- src/qemu/qemu_monitor_text.c | 24 ++++++++++++++++++++++-- src/qemu/qemu_monitor_text.h | 6 +++++- 6 files changed, 78 insertions(+), 5 deletions(-)
ACK Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Work in progress - this fails for root-squash NFS, as well as for older qemu that don't support fd: migration, and doesn't cover compression coupled with migration. All of those need to be provided by a final solution. * src/qemu/qemu_driver.c (qemudDomainSaveFlag): Use new function when there is no compression. --- v2: new patch, provided for getting feedback on the approach src/qemu/qemu_driver.c | 16 +++++++++++++--- 1 files changed, 13 insertions(+), 3 deletions(-) diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index c9095bb..54795ac 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -1981,10 +1981,20 @@ static int qemudDomainSaveFlag(struct qemud_driver *driver, virDomainPtr dom, if (header.compressed == QEMUD_SAVE_FORMAT_RAW) { const char *args[] = { "cat", NULL }; + /* XXX gross - why don't we reuse the fd already opened earlier */ + int fd = open(path, O_WRONLY); qemuDomainObjEnterMonitorWithDriver(driver, vm); - rc = qemuMonitorMigrateToFile(priv->mon, - QEMU_MONITOR_MIGRATE_BACKGROUND, - args, path, offset); + /* XXX needs to depend on QEMU_CAPS_MIGRATE_QEMU_FD */ + if (fd >= 0 && lseek(fd, offset, SEEK_SET) == offset) { + rc = qemuMonitorMigrateToFd(priv->mon, + QEMU_MONITOR_MIGRATE_BACKGROUND, + fd); + } else { + rc = qemuMonitorMigrateToFile(priv->mon, + QEMU_MONITOR_MIGRATE_BACKGROUND, + args, path, offset); + } + VIR_FORCE_CLOSE(fd); qemuDomainObjExitMonitorWithDriver(driver, vm); } else { const char *prog = qemudSaveCompressionTypeToString(header.compressed); -- 1.7.4
participants (2)
-
Daniel P. Berrange -
Eric Blake