4:07 p.m.
Posted as an RFC because there are still two open bugs to be solved.
I've tested that with this installed, I was able to do 'virsh save
domain file' without compression to a local file using qemu fd:
migration.
1. I'm working on making virFileOperation use SCM_RIGHTS so that we
can pass an arbitrary fd from the child back to the parent. This is
necessary for supporting root-squash NFS domain save via fd, since a
child cannot consistently modify the appropriate locks necessary to
issue a monitor command (the current virFileOperation implementation
guarantees that the hook will execute on the right fd, but does not
guarantee whether the operation is in the parent or a child, and thus
the operation is not permitted to modify any state in the parent other
than by its return value). The current state of the patch just falls
back to exec: migration if the parent process can't reopen the fd.
2. I also need to test for qemu capabilities, as qemu 0.11.0 does
_not_ support fd: migration, so the old exec: migration path must
remain. But checking for capabilities is a bear in itself, since we
should be caching qemu capabilities when a domain is started, and not
calling 'qemu -help' for every subsequent command that needs to know.
Eric Blake (3):
qemu: use lighter-weight fd:n on incoming tunneled migration
qemu: support migration to fd
qemu: allow simple domain save to use fd: protocol
src/qemu/qemu_driver.c | 16 ++++++++++++--
src/qemu/qemu_migration.c | 44 +++++++++++------------------------------
src/qemu/qemu_monitor.c | 22 +++++++++++++++++++++
src/qemu/qemu_monitor.h | 4 +++
src/qemu/qemu_monitor_json.c | 21 +++++++++++++++++++-
src/qemu/qemu_monitor_json.h | 6 ++++-
src/qemu/qemu_monitor_text.c | 24 +++++++++++++++++++++-
src/qemu/qemu_monitor_text.h | 6 ++++-
8 files changed, 103 insertions(+), 40 deletions(-)
--
1.7.4
4:07 p.m.
New subject: [libvirt] [RFC PATCHv2 1/3] qemu: use lighter-weight fd:n on incoming tunneled migration
Outgoing migration still uses a Unix socket and or exec netcat until
the next patch.
* src/qemu/qemu_migration.c (qemuMigrationPrepareTunnel):
Replace Unix socket with simpler pipe.
Suggested by Paolo Bonzini.
---
v1 was ack'd here but not yet applied:
https://www.redhat.com/archives/libvir-list/2011-January/msg00742.html
changes in v2: rebase to latest upstream
src/qemu/qemu_migration.c | 44 ++++++++++++--------------------------------
1 files changed, 12 insertions(+), 32 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c
index 822cb18..dc51bc8 100644
--- a/src/qemu/qemu_migration.c
+++ b/src/qemu/qemu_migration.c
@@ -239,11 +239,10 @@ qemuMigrationPrepareTunnel(struct qemud_driver *driver,
{
virDomainDefPtr def = NULL;
virDomainObjPtr vm = NULL;
- char *migrateFrom;
virDomainEventPtr event = NULL;
int ret = -1;
int internalret;
- char *unixfile = NULL;
+ int dataFD[2] = { -1, -1 };
virBitmapPtr qemuCaps = NULL;
qemuDomainObjPrivatePtr priv = NULL;
struct timeval now;
@@ -289,12 +288,11 @@ qemuMigrationPrepareTunnel(struct qemud_driver *driver,
/* Domain starts inactive, even if the domain XML had an id field. */
vm->def->id = -1;
- if (virAsprintf(&unixfile, "%s/qemu.tunnelmigrate.dest.%s",
- driver->libDir, vm->def->name) < 0) {
- virReportOOMError();
+ if (pipe(dataFD) < 0) {
+ virReportSystemError(errno, "%s",
+ _("cannot create pipe for tunnelled migration"));
goto endjob;
}
- unlink(unixfile);
/* check that this qemu version supports the interactive exec */
if (qemuCapsExtractVersionInfo(vm->def->emulator, vm->def->os.arch,
@@ -304,25 +302,11 @@ qemuMigrationPrepareTunnel(struct qemud_driver *driver,
vm->def->emulator);
goto endjob;
}
- if (qemuCapsGet(qemuCaps, QEMU_CAPS_MIGRATE_QEMU_UNIX))
- internalret = virAsprintf(&migrateFrom, "unix:%s", unixfile);
- else if (qemuCapsGet(qemuCaps, QEMU_CAPS_MIGRATE_QEMU_EXEC))
- internalret = virAsprintf(&migrateFrom, "exec:nc -U -l %s",
unixfile);
- else {
- qemuReportError(VIR_ERR_OPERATION_FAILED,
- "%s", _("Destination qemu is too old to support
tunnelled migration"));
- goto endjob;
- }
- if (internalret < 0) {
- virReportOOMError();
- goto endjob;
- }
/* Start the QEMU daemon, with the same command-line arguments plus
- * -incoming unix:/path/to/file or exec:nc -U /path/to/file
+ * -incoming stdin (which qemu_command might convert to exec:cat or fd:n)
*/
- internalret = qemuProcessStart(dconn, driver, vm, migrateFrom, true,
- -1, NULL, VIR_VM_OP_MIGRATE_IN_START);
- VIR_FREE(migrateFrom);
+ internalret = qemuProcessStart(dconn, driver, vm, "stdin", true,
dataFD[1],
+ NULL, VIR_VM_OP_MIGRATE_IN_START);
if (internalret < 0) {
qemuDomainStartAudit(vm, "migrated", false);
/* Note that we don't set an error here because qemuProcessStart
@@ -335,9 +319,7 @@ qemuMigrationPrepareTunnel(struct qemud_driver *driver,
goto endjob;
}
- if (virFDStreamConnectUNIX(st,
- unixfile,
- false) < 0) {
+ if (virFDStreamOpen(st, dataFD[0]) < 0) {
qemuDomainStartAudit(vm, "migrated", false);
qemuProcessStop(driver, vm, 0);
if (!vm->persistent) {
@@ -345,9 +327,8 @@ qemuMigrationPrepareTunnel(struct qemud_driver *driver,
virDomainRemoveInactive(&driver->domains, vm);
vm = NULL;
}
- virReportSystemError(errno,
- _("cannot open unix socket '%s' for tunnelled
migration"),
- unixfile);
+ virReportSystemError(errno, "%s",
+ _("cannot pass pipe for tunnelled migration"));
goto endjob;
}
@@ -378,9 +359,8 @@ endjob:
cleanup:
qemuCapsFree(qemuCaps);
virDomainDefFree(def);
- if (unixfile)
- unlink(unixfile);
- VIR_FREE(unixfile);
+ VIR_FORCE_CLOSE(dataFD[0]);
+ VIR_FORCE_CLOSE(dataFD[1]);
if (vm)
virDomainObjUnlock(vm);
if (event)
--
1.7.4
5:54 a.m.
New subject: [libvirt] [RFC PATCHv2 1/3] qemu: use lighter-weight fd:n on incoming tunneled migration
On Wed, Mar 02, 2011 at 03:07:04PM -0700, Eric Blake wrote:
Outgoing migration still uses a Unix socket and or exec netcat until
the next patch.
* src/qemu/qemu_migration.c (qemuMigrationPrepareTunnel):
Replace Unix socket with simpler pipe.
Suggested by Paolo Bonzini.
---
v1 was ack'd here but not yet applied:
https://www.redhat.com/archives/libvir-list/2011-January/msg00742.html
changes in v2: rebase to latest upstream
src/qemu/qemu_migration.c | 44 ++++++++++++--------------------------------
1 files changed, 12 insertions(+), 32 deletions(-)
ACK
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
4:06 p.m.
New subject: [libvirt] [RFC PATCHv2 1/3] qemu: use lighter-weight fd:n on incoming tunneled migration
On 03/03/2011 04:54 AM, Daniel P. Berrange wrote:
On Wed, Mar 02, 2011 at 03:07:04PM -0700, Eric Blake wrote:
> Outgoing migration still uses a Unix socket and or exec netcat until
> the next patch.
>
> * src/qemu/qemu_migration.c (qemuMigrationPrepareTunnel):
> Replace Unix socket with simpler pipe.
> Suggested by Paolo Bonzini.
> ---
>
> v1 was ack'd here but not yet applied:
> https://www.redhat.com/archives/libvir-list/2011-January/msg00742.html
> changes in v2: rebase to latest upstream
>
> src/qemu/qemu_migration.c | 44 ++++++++++++--------------------------------
> 1 files changed, 12 insertions(+), 32 deletions(-)
ACK
Self-NACK - I forgot to FD_CLOEXEC the write side of the pipe
(virCommand explicitly closes it anyways, but we might as well be clean
on our intentions). I'll fix that in v3. Patch 2/3 is fine as is, but
no point in pushing it until there's a client. And patch 3/3 will be
heavily reworked in v3 (I'm up to at least 5 separate patches, and
counting).
--
Eric Blake eblake(a)redhat.com +1-801-349-2682
Libvirt virtualization library http://libvirt.org
4:07 p.m.
New subject: [libvirt] [RFC PATCHv2 2/3] qemu: support migration to fd
* src/qemu/qemu_monitor.h (qemuMonitorMigrateToFd): New
prototype.
* src/qemu/qemu_monitor_text.h (qemuMonitorTextMigrateToFd):
Likewise.
* src/qemu/qemu_monitor_json.h (qemuMonitorJSONMigrateToFd):
Likewise.
* src/qemu/qemu_monitor.c (qemuMonitorMigrateToFd): New function.
* src/qemu/qemu_monitor_text.c (qemuMonitorTextMigrateToFd):
Likewise.
* src/qemu/qemu_monitor_json.c (qemuMonitorJSONMigrateToFd):
Likewise.
---
v2: new patch
src/qemu/qemu_monitor.c | 22 ++++++++++++++++++++++
src/qemu/qemu_monitor.h | 4 ++++
src/qemu/qemu_monitor_json.c | 21 ++++++++++++++++++++-
src/qemu/qemu_monitor_json.h | 6 +++++-
src/qemu/qemu_monitor_text.c | 24 ++++++++++++++++++++++--
src/qemu/qemu_monitor_text.h | 6 +++++-
6 files changed, 78 insertions(+), 5 deletions(-)
diff --git a/src/qemu/qemu_monitor.c b/src/qemu/qemu_monitor.c
index dfb1aad..312e797 100644
--- a/src/qemu/qemu_monitor.c
+++ b/src/qemu/qemu_monitor.c
@@ -1384,6 +1384,28 @@ int qemuMonitorGetMigrationStatus(qemuMonitorPtr mon,
}
+int qemuMonitorMigrateToFd(qemuMonitorPtr mon,
+ unsigned int flags,
+ int fd)
+{
+ int ret;
+ VIR_DEBUG("mon=%p fd=%d flags=%u",
+ mon, fd, flags);
+
+ if (!mon) {
+ qemuReportError(VIR_ERR_INVALID_ARG, "%s",
+ _("monitor must not be NULL"));
+ return -1;
+ }
+
+ if (mon->json)
+ ret = qemuMonitorJSONMigrateToFd(mon, flags, fd);
+ else
+ ret = qemuMonitorTextMigrateToFd(mon, flags, fd);
+ return ret;
+}
+
+
int qemuMonitorMigrateToHost(qemuMonitorPtr mon,
unsigned int flags,
const char *hostname,
diff --git a/src/qemu/qemu_monitor.h b/src/qemu/qemu_monitor.h
index 0ea1330..1a64ac0 100644
--- a/src/qemu/qemu_monitor.h
+++ b/src/qemu/qemu_monitor.h
@@ -260,6 +260,10 @@ typedef enum {
QEMU_MONITOR_MIGRATION_FLAGS_LAST
} QEMU_MONITOR_MIGRATE;
+int qemuMonitorMigrateToFd(qemuMonitorPtr mon,
+ unsigned int flags,
+ int fd);
+
int qemuMonitorMigrateToHost(qemuMonitorPtr mon,
unsigned int flags,
const char *hostname,
diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c
index e6903a1..81201ff 100644
--- a/src/qemu/qemu_monitor_json.c
+++ b/src/qemu/qemu_monitor_json.c
@@ -1,7 +1,7 @@
/*
* qemu_monitor_json.c: interaction with QEMU monitor console
*
- * Copyright (C) 2006-2010 Red Hat, Inc.
+ * Copyright (C) 2006-2011 Red Hat, Inc.
* Copyright (C) 2006 Daniel P. Berrange
*
* This library is free software; you can redistribute it and/or
@@ -1697,6 +1697,25 @@ static int qemuMonitorJSONMigrate(qemuMonitorPtr mon,
}
+int qemuMonitorJSONMigrateToFd(qemuMonitorPtr mon,
+ unsigned int flags,
+ int fd)
+{
+ int ret;
+
+ if (qemuMonitorJSONSendFileHandle(mon, "migrate", fd) < 0)
+ return -1;
+
+ ret = qemuMonitorJSONMigrate(mon, flags, "fd:migrate");
+
+ if (ret < 0) {
+ if (qemuMonitorJSONCloseFileHandle(mon, "migrate") < 0)
+ VIR_WARN0("failed to close migration handle");
+ }
+
+ return ret;
+}
+
int qemuMonitorJSONMigrateToHost(qemuMonitorPtr mon,
unsigned int flags,
const char *hostname,
diff --git a/src/qemu/qemu_monitor_json.h b/src/qemu/qemu_monitor_json.h
index 4ae472a..c2b45f3 100644
--- a/src/qemu/qemu_monitor_json.h
+++ b/src/qemu/qemu_monitor_json.h
@@ -1,7 +1,7 @@
/*
* qemu_monitor_json.h: interaction with QEMU monitor console
*
- * Copyright (C) 2006-2009 Red Hat, Inc.
+ * Copyright (C) 2006-2009, 2011 Red Hat, Inc.
* Copyright (C) 2006 Daniel P. Berrange
*
* This library is free software; you can redistribute it and/or
@@ -104,6 +104,10 @@ int qemuMonitorJSONGetMigrationStatus(qemuMonitorPtr mon,
unsigned long long *remaining,
unsigned long long *total);
+int qemuMonitorJSONMigrateToFd(qemuMonitorPtr mon,
+ unsigned int flags,
+ int fd);
+
int qemuMonitorJSONMigrateToHost(qemuMonitorPtr mon,
unsigned int flags,
const char *hostname,
diff --git a/src/qemu/qemu_monitor_text.c b/src/qemu/qemu_monitor_text.c
index 0aed690..76e0ec0 100644
--- a/src/qemu/qemu_monitor_text.c
+++ b/src/qemu/qemu_monitor_text.c
@@ -1,7 +1,7 @@
/*
* qemu_monitor_text.c: interaction with QEMU monitor console
*
- * Copyright (C) 2006-2010 Red Hat, Inc.
+ * Copyright (C) 2006-2011 Red Hat, Inc.
* Copyright (C) 2006 Daniel P. Berrange
*
* This library is free software; you can redistribute it and/or
@@ -1256,7 +1256,8 @@ static int qemuMonitorTextMigrate(qemuMonitorPtr mon,
* unknown command: migrate" */
if (strstr(info, "unknown command:")) {
qemuReportError(VIR_ERR_NO_SUPPORT,
- _("migration to '%s' not supported by this qemu:
%s"), dest, info);
+ _("migration to '%s' not supported by this qemu:
%s"),
+ dest, info);
goto cleanup;
}
@@ -1271,6 +1272,25 @@ cleanup:
return ret;
}
+int qemuMonitorTextMigrateToFd(qemuMonitorPtr mon,
+ unsigned int flags,
+ int fd)
+{
+ int ret;
+
+ if (qemuMonitorTextSendFileHandle(mon, "migrate", fd) < 0)
+ return -1;
+
+ ret = qemuMonitorTextMigrate(mon, flags, "fd:migrate");
+
+ if (ret < 0) {
+ if (qemuMonitorTextCloseFileHandle(mon, "migrate") < 0)
+ VIR_WARN0("failed to close migration handle");
+ }
+
+ return ret;
+}
+
int qemuMonitorTextMigrateToHost(qemuMonitorPtr mon,
unsigned int flags,
const char *hostname,
diff --git a/src/qemu/qemu_monitor_text.h b/src/qemu/qemu_monitor_text.h
index b29dbcc..f1b53fc 100644
--- a/src/qemu/qemu_monitor_text.h
+++ b/src/qemu/qemu_monitor_text.h
@@ -1,7 +1,7 @@
/*
* qemu_monitor_text.h: interaction with QEMU monitor console
*
- * Copyright (C) 2006-2009 Red Hat, Inc.
+ * Copyright (C) 2006-2009, 2011 Red Hat, Inc.
* Copyright (C) 2006 Daniel P. Berrange
*
* This library is free software; you can redistribute it and/or
@@ -102,6 +102,10 @@ int qemuMonitorTextGetMigrationStatus(qemuMonitorPtr mon,
unsigned long long *remaining,
unsigned long long *total);
+int qemuMonitorTextMigrateToFd(qemuMonitorPtr mon,
+ unsigned int flags,
+ int fd);
+
int qemuMonitorTextMigrateToHost(qemuMonitorPtr mon,
unsigned int flags,
const char *hostname,
--
1.7.4
5:57 a.m.
New subject: [libvirt] [RFC PATCHv2 2/3] qemu: support migration to fd
On Wed, Mar 02, 2011 at 03:07:05PM -0700, Eric Blake wrote:
* src/qemu/qemu_monitor.h (qemuMonitorMigrateToFd): New
prototype.
* src/qemu/qemu_monitor_text.h (qemuMonitorTextMigrateToFd):
Likewise.
* src/qemu/qemu_monitor_json.h (qemuMonitorJSONMigrateToFd):
Likewise.
* src/qemu/qemu_monitor.c (qemuMonitorMigrateToFd): New function.
* src/qemu/qemu_monitor_text.c (qemuMonitorTextMigrateToFd):
Likewise.
* src/qemu/qemu_monitor_json.c (qemuMonitorJSONMigrateToFd):
Likewise.
---
v2: new patch
src/qemu/qemu_monitor.c | 22 ++++++++++++++++++++++
src/qemu/qemu_monitor.h | 4 ++++
src/qemu/qemu_monitor_json.c | 21 ++++++++++++++++++++-
src/qemu/qemu_monitor_json.h | 6 +++++-
src/qemu/qemu_monitor_text.c | 24 ++++++++++++++++++++++--
src/qemu/qemu_monitor_text.h | 6 +++++-
6 files changed, 78 insertions(+), 5 deletions(-)
ACK
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
4:07 p.m.
New subject: [libvirt] [RFC PATCHv2 3/3] qemu: allow simple domain save to use fd: protocol
Work in progress - this fails for root-squash NFS, as well as for
older qemu that don't support fd: migration, and doesn't cover
compression coupled with migration. All of those need to be
provided by a final solution.
* src/qemu/qemu_driver.c (qemudDomainSaveFlag): Use new function
when there is no compression.
---
v2: new patch, provided for getting feedback on the approach
src/qemu/qemu_driver.c | 16 +++++++++++++---
1 files changed, 13 insertions(+), 3 deletions(-)
diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c
index c9095bb..54795ac 100644
--- a/src/qemu/qemu_driver.c
+++ b/src/qemu/qemu_driver.c
@@ -1981,10 +1981,20 @@ static int qemudDomainSaveFlag(struct qemud_driver *driver,
virDomainPtr dom,
if (header.compressed == QEMUD_SAVE_FORMAT_RAW) {
const char *args[] = { "cat", NULL };
+ /* XXX gross - why don't we reuse the fd already opened earlier */
+ int fd = open(path, O_WRONLY);
qemuDomainObjEnterMonitorWithDriver(driver, vm);
- rc = qemuMonitorMigrateToFile(priv->mon,
- QEMU_MONITOR_MIGRATE_BACKGROUND,
- args, path, offset);
+ /* XXX needs to depend on QEMU_CAPS_MIGRATE_QEMU_FD */
+ if (fd >= 0 && lseek(fd, offset, SEEK_SET) == offset) {
+ rc = qemuMonitorMigrateToFd(priv->mon,
+ QEMU_MONITOR_MIGRATE_BACKGROUND,
+ fd);
+ } else {
+ rc = qemuMonitorMigrateToFile(priv->mon,
+ QEMU_MONITOR_MIGRATE_BACKGROUND,
+ args, path, offset);
+ }
+ VIR_FORCE_CLOSE(fd);
qemuDomainObjExitMonitorWithDriver(driver, vm);
} else {
const char *prog = qemudSaveCompressionTypeToString(header.compressed);
--
1.7.4
5016
days inactive
5017
days old
6 comments
2 participants
participants (2)
-
Daniel P. Berrange -
Eric Blake