[libvirt] [PATCH v4 0/5] Expose FSFreeze/FSThaw within the guest as API

Hello, This is patchset v4 to add FSFreeze/FSThaw API for custom disk snapshotting. Changes to v3: * fix typp and label spacing * rebased to recent tree (v3: http://www.redhat.com/archives/libvir-list/2014-March/msg01358.html ) === Description === Currently FSFreeze and FSThaw are supported by qemu guest agent and they are used internally in snapshot-create command with --quiesce option. However, when users want to utilize the native snapshot feature of storage devices (such as LVM over iSCSI, enterprise storage appliances, etc.), they need to issue fsfreeze command separately from libvirt-driven snapshots. (OpenStack cinder provides these storages' snapshot feature, but it cannot quiesce the guest filesystems automatically for now.) Although virDomainQemuGuestAgent() API could be used for this purpose, it is only for debugging and is not supported officially. This patchset adds virDomainFSFreeze()/virDomainFSThaw() APIs and virsh domfsfreeze/domfsthaw commands to enable the users to freeze and thaw domain's filesystems cleanly. The APIs have flags option currently unsupported for future extension. --- Tomoki Sekiyama (5): Introduce virDomainFSFreeze() and virDomainFSThaw() public API remote: Implement virDomainFSFreeze and virDomainFSThaw qemu: Track domain quiesced status qemu: Implement virDomainFSFreeze and virDomainFSThaw virsh: Expose new virDomainFSFreeze and virDomainFSThaw API include/libvirt/libvirt.h.in | 6 ++ src/access/viraccessperm.c | 2 - src/access/viraccessperm.h | 6 ++ src/driver.h | 10 +++ src/libvirt.c | 70 ++++++++++++++++++++ src/libvirt_public.syms | 2 + src/qemu/qemu_domain.c | 5 + src/qemu/qemu_domain.h | 2 + src/qemu/qemu_driver.c | 144 ++++++++++++++++++++++++++++++++++++++---- src/remote/remote_driver.c | 2 + src/remote/remote_protocol.x | 25 +++++++ src/remote_protocol-structs | 9 +++ src/rpc/gendispatch.pl | 2 + tools/virsh-domain.c | 92 +++++++++++++++++++++++++++ tools/virsh.pod | 15 ++++ 15 files changed, 376 insertions(+), 16 deletions(-)

These will freeze and thaw filesystems within guest. The APIs take @flags arguments which are currently not used, for future extensions. Signed-off-by: Tomoki Sekiyama <tomoki.sekiyama@hds.com> --- include/libvirt/libvirt.h.in | 6 ++++ src/driver.h | 10 ++++++ src/libvirt.c | 70 ++++++++++++++++++++++++++++++++++++++++++ src/libvirt_public.syms | 2 + 4 files changed, 88 insertions(+) diff --git a/include/libvirt/libvirt.h.in b/include/libvirt/libvirt.h.in index 91e3e3b..afe92c6 100644 --- a/include/libvirt/libvirt.h.in +++ b/include/libvirt/libvirt.h.in @@ -5277,6 +5277,12 @@ int virDomainFSTrim(virDomainPtr dom, unsigned long long minimum, unsigned int flags); +int virDomainFSFreeze(virDomainPtr dom, + unsigned int flags); + +int virDomainFSThaw(virDomainPtr dom, + unsigned int flags); + /** * virSchedParameterType: * diff --git a/src/driver.h b/src/driver.h index e66fc7a..50258e2 100644 --- a/src/driver.h +++ b/src/driver.h @@ -1149,6 +1149,14 @@ typedef int unsigned int flags, int cancelled); +typedef int +(*virDrvDomainFSFreeze)(virDomainPtr dom, + unsigned int flags); + +typedef int +(*virDrvDomainFSThaw)(virDomainPtr dom, + unsigned int flags); + typedef struct _virDriver virDriver; typedef virDriver *virDriverPtr; @@ -1363,6 +1371,8 @@ struct _virDriver { virDrvDomainMigrateFinish3Params domainMigrateFinish3Params; virDrvDomainMigrateConfirm3Params domainMigrateConfirm3Params; virDrvConnectGetCPUModelNames connectGetCPUModelNames; + virDrvDomainFSFreeze domainFSFreeze; + virDrvDomainFSThaw domainFSThaw; }; diff --git a/src/libvirt.c b/src/libvirt.c index 4454829..48b9902 100644 --- a/src/libvirt.c +++ b/src/libvirt.c @@ -20658,3 +20658,73 @@ virDomainFSTrim(virDomainPtr dom, virDispatchError(dom->conn); return -1; } + +/** + * virDomainFSFreeze: + * @dom: a domain object + * @flags: extra flags, not used yet, so callers should always pass 0 + * + * Freeze filesystems within the guest (hence guest agent may be + * required depending on hypervisor used). + * + * Returns 0 on success, -1 otherwise. + */ +int +virDomainFSFreeze(virDomainPtr dom, + unsigned int flags) +{ + VIR_DOMAIN_DEBUG(dom, "flags=%x", flags); + + virResetLastError(); + + virCheckDomainReturn(dom, -1); + virCheckReadOnlyGoto(dom->conn->flags, error); + + if (dom->conn->driver->domainFSFreeze) { + int ret = dom->conn->driver->domainFSFreeze(dom, flags); + if (ret < 0) + goto error; + return ret; + } + + virReportUnsupportedError(); + + error: + virDispatchError(dom->conn); + return -1; +} + +/** + * virDomainFSThaw: + * @dom: a domain object + * @flags: extra flags, not used yet, so callers should always pass 0 + * + * Thaw the frozen filesystems within the guest (hence guest agent + * may be required depending on hypervisor used). + * + * Returns 0 on success, -1 otherwise. + */ +int +virDomainFSThaw(virDomainPtr dom, + unsigned int flags) +{ + VIR_DOMAIN_DEBUG(dom, "flags=%x", flags); + + virResetLastError(); + + virCheckDomainReturn(dom, -1); + virCheckReadOnlyGoto(dom->conn->flags, error); + + if (dom->conn->driver->domainFSThaw) { + int ret = dom->conn->driver->domainFSThaw(dom, flags); + if (ret < 0) + goto error; + return ret; + } + + virReportUnsupportedError(); + + error: + virDispatchError(dom->conn); + return -1; +} diff --git a/src/libvirt_public.syms b/src/libvirt_public.syms index 9ab0c92..42793c9 100644 --- a/src/libvirt_public.syms +++ b/src/libvirt_public.syms @@ -648,6 +648,8 @@ LIBVIRT_1.2.1 { LIBVIRT_1.2.3 { global: virDomainCoreDumpWithFormat; + virDomainFSFreeze; + virDomainFSThaw; } LIBVIRT_1.2.1;

New rules are added in fixup_name in gendispatch.pl to keep the name FSFreeze and FSThaw. This adds a new ACL permission 'fs_freeze', which is also applied to VIR_DOMAIN_SNAPSHOT_CREATE_QUIESCE flag. Signed-off-by: Tomoki Sekiyama <tomoki.sekiyama@hds.com> --- src/access/viraccessperm.c | 2 +- src/access/viraccessperm.h | 6 ++++++ src/remote/remote_driver.c | 2 ++ src/remote/remote_protocol.x | 25 +++++++++++++++++++++++-- src/remote_protocol-structs | 9 +++++++++ src/rpc/gendispatch.pl | 2 ++ 6 files changed, 43 insertions(+), 3 deletions(-) diff --git a/src/access/viraccessperm.c b/src/access/viraccessperm.c index d517c66..462f46c 100644 --- a/src/access/viraccessperm.c +++ b/src/access/viraccessperm.c @@ -39,7 +39,7 @@ VIR_ENUM_IMPL(virAccessPermDomain, "start", "stop", "reset", "save", "delete", "migrate", "snapshot", "suspend", "hibernate", "core_dump", "pm_control", - "init_control", "inject_nmi", "send_input", "send_signal", "fs_trim", + "init_control", "inject_nmi", "send_input", "send_signal", "fs_trim", "fs_freeze", "block_read", "block_write", "mem_read", "open_graphics", "open_device", "screenshot", "open_namespace"); diff --git a/src/access/viraccessperm.h b/src/access/viraccessperm.h index 6d14f05..ac48d70 100644 --- a/src/access/viraccessperm.h +++ b/src/access/viraccessperm.h @@ -242,6 +242,12 @@ typedef enum { */ VIR_ACCESS_PERM_DOMAIN_FS_TRIM, /* Issue TRIM to guest filesystems */ + /** + * @desc: Freeze and thaw domain filesystems + * @message: Freezing and thawing domain filesystems require authorization + */ + VIR_ACCESS_PERM_DOMAIN_FS_FREEZE, /* Freeze and thaw guest filesystems */ + /* Peeking at guest */ /** diff --git a/src/remote/remote_driver.c b/src/remote/remote_driver.c index ed7dde6..f26a2e4 100644 --- a/src/remote/remote_driver.c +++ b/src/remote/remote_driver.c @@ -7800,6 +7800,8 @@ static virDriver remote_driver = { .domainMigrateFinish3Params = remoteDomainMigrateFinish3Params, /* 1.1.0 */ .domainMigrateConfirm3Params = remoteDomainMigrateConfirm3Params, /* 1.1.0 */ .connectGetCPUModelNames = remoteConnectGetCPUModelNames, /* 1.1.3 */ + .domainFSFreeze = remoteDomainFSFreeze, /* 1.2.3 */ + .domainFSThaw = remoteDomainFSThaw, /* 1.2.3 */ }; static virNetworkDriver network_driver = { diff --git a/src/remote/remote_protocol.x b/src/remote/remote_protocol.x index 6c445cc..f8ce568 100644 --- a/src/remote/remote_protocol.x +++ b/src/remote/remote_protocol.x @@ -2959,6 +2959,15 @@ struct remote_network_event_lifecycle_msg { int detail; }; +struct remote_domain_fsfreeze_args { + remote_nonnull_domain dom; + unsigned int flags; +}; + +struct remote_domain_fsthaw_args { + remote_nonnull_domain dom; + unsigned int flags; +}; /*----- Protocol. -----*/ @@ -4289,7 +4298,7 @@ enum remote_procedure { /** * @generate: both * @acl: domain:snapshot - * @acl: domain:write:VIR_DOMAIN_SNAPSHOT_CREATE_QUIESCE + * @acl: domain:fs_freeze:VIR_DOMAIN_SNAPSHOT_CREATE_QUIESCE */ REMOTE_PROC_DOMAIN_SNAPSHOT_CREATE_XML = 185, @@ -5275,5 +5284,17 @@ enum remote_procedure { * @generate: both * @acl: domain:core_dump */ - REMOTE_PROC_DOMAIN_CORE_DUMP_WITH_FORMAT = 334 + REMOTE_PROC_DOMAIN_CORE_DUMP_WITH_FORMAT = 334, + + /** + * @generate: both + * @acl: domain:fs_freeze + */ + REMOTE_PROC_DOMAIN_FSFREEZE = 335, + + /** + * @generate: both + * @acl: domain:fs_freeze + */ + REMOTE_PROC_DOMAIN_FSTHAW = 336 }; diff --git a/src/remote_protocol-structs b/src/remote_protocol-structs index 456d0da..b5b21e3 100644 --- a/src/remote_protocol-structs +++ b/src/remote_protocol-structs @@ -2426,6 +2426,13 @@ struct remote_network_event_lifecycle_msg { remote_nonnull_network net; int event; int detail; +struct remote_domain_fsfreeze_args { + remote_nonnull_domain dom; + u_int flags; +}; +struct remote_domain_fsthaw_args { + remote_nonnull_domain dom; + u_int flags; }; enum remote_procedure { REMOTE_PROC_CONNECT_OPEN = 1, @@ -2762,4 +2769,6 @@ enum remote_procedure { REMOTE_PROC_DOMAIN_EVENT_CALLBACK_PMSUSPEND_DISK = 332, REMOTE_PROC_DOMAIN_EVENT_CALLBACK_DEVICE_REMOVED = 333, REMOTE_PROC_DOMAIN_CORE_DUMP_WITH_FORMAT = 334, + REMOTE_PROC_DOMAIN_FSFREEZE = 335, + REMOTE_PROC_DOMAIN_FSTHAW = 336, }; diff --git a/src/rpc/gendispatch.pl b/src/rpc/gendispatch.pl index b76bbac..9cd620e 100755 --- a/src/rpc/gendispatch.pl +++ b/src/rpc/gendispatch.pl @@ -64,6 +64,8 @@ sub fixup_name { $name =~ s/Nmi$/NMI/; $name =~ s/Pm/PM/; $name =~ s/Fstrim$/FSTrim/; + $name =~ s/Fsfreeze$/FSFreeze/; + $name =~ s/Fsthaw$/FSThaw/; $name =~ s/Scsi/SCSI/; $name =~ s/Wwn$/WWN/;

Adds an quiesced flag into qemuDomainObjPrivate that tracks whether guest filesystems of the domain is quiesced or not. It also modify error code from qemuDomainSnapshotFSFreeze and qemuDomainSnapshotFSThaw, so that a caller can know whether the command is actually sent to the guest agent. If the error is caused before sending a freeze command, a counterpart thaw command shouldn't be sent either, not to thaw the guest unexpectedly by error handling code. Signed-off-by: Tomoki Sekiyama <tomoki.sekiyama@hds.com> --- src/qemu/qemu_domain.c | 5 ++++ src/qemu/qemu_domain.h | 2 + src/qemu/qemu_driver.c | 66 +++++++++++++++++++++++++++++++++++++++--------- 3 files changed, 60 insertions(+), 13 deletions(-) diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 36cb2c6..52aa09d 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -357,6 +357,9 @@ qemuDomainObjPrivateXMLFormat(virBufferPtr buf, void *data) virBufferAddLit(buf, "</devices>\n"); } + if (priv->quiesced) + virBufferAddLit(buf, "<quiesced/>\n"); + return 0; } @@ -518,6 +521,8 @@ qemuDomainObjPrivateXMLParse(xmlXPathContextPtr ctxt, void *data) } VIR_FREE(nodes); + priv->quiesced = virXPathBoolean("boolean(./quiesced)", ctxt) == 1; + return 0; error: diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h index b2830c4..5fb1665 100644 --- a/src/qemu/qemu_domain.h +++ b/src/qemu/qemu_domain.h @@ -176,6 +176,8 @@ struct _qemuDomainObjPrivate { char **qemuDevices; /* NULL-terminated list of devices aliases known to QEMU */ bool hookRun; /* true if there was a hook run over this domain */ + + bool quiesced; /* true if the domain filesystems are quiesced */ }; typedef enum { diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index b032441..7fd8b6d 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -12100,32 +12100,61 @@ qemuDomainPrepareDiskChainElement(virQEMUDriverPtr driver, } +/* Return -1 if request is not sent to agent due to misconfig, -2 if request + * is sent but failed, and number of frozen filesystems on success. */ static int -qemuDomainSnapshotFSFreeze(virDomainObjPtr vm) +qemuDomainSnapshotFSFreeze(virQEMUDriverPtr driver, virDomainObjPtr vm) { qemuDomainObjPrivatePtr priv = vm->privateData; + virQEMUDriverConfigPtr cfg; int freezed; if (!qemuDomainAgentAvailable(priv, true)) return -1; + if (priv->quiesced) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("domain is already quiesced")); + return -1; + } + qemuDomainObjEnterAgent(vm); freezed = qemuAgentFSFreeze(priv->agent); qemuDomainObjExitAgent(vm); - return freezed; + if (freezed >= 0) + priv->quiesced = true; + + cfg = virQEMUDriverGetConfig(driver); + if (virDomainSaveStatus(driver->xmlopt, cfg->stateDir, vm) < 0) { + virObjectUnref(cfg); + return -2; + } + virObjectUnref(cfg); + + return freezed < 0 ? -2 : freezed; } +/* Return -1 if request is not sent to agent due to misconfig, -2 if request + * is send but failed, and number of thawed filesystems on success. */ static int -qemuDomainSnapshotFSThaw(virDomainObjPtr vm, bool report) +qemuDomainSnapshotFSThaw(virQEMUDriverPtr driver, + virDomainObjPtr vm, bool report) { qemuDomainObjPrivatePtr priv = vm->privateData; + virQEMUDriverConfigPtr cfg; int thawed; virErrorPtr err = NULL; if (!qemuDomainAgentAvailable(priv, report)) return -1; + if (!priv->quiesced && report) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("domain is not quiesced")); + return -1; + } + qemuDomainObjEnterAgent(vm); if (!report) err = virSaveLastError(); @@ -12134,8 +12163,18 @@ qemuDomainSnapshotFSThaw(virDomainObjPtr vm, bool report) virSetError(err); qemuDomainObjExitAgent(vm); + if (thawed >= 0) + priv->quiesced = false; + + cfg = virQEMUDriverGetConfig(driver); + if (virDomainSaveStatus(driver->xmlopt, cfg->stateDir, vm) < 0) { + virObjectUnref(cfg); + return -2; + } + virObjectUnref(cfg); + virFreeError(err); - return thawed; + return thawed < 0 ? -2 : thawed; } /* The domain is expected to be locked and inactive. */ @@ -13114,17 +13153,18 @@ qemuDomainSnapshotCreateActiveExternal(virConnectPtr conn, goto cleanup; /* If quiesce was requested, then issue a freeze command, and a - * counterpart thaw command, no matter what. The command will - * fail if the guest is paused or the guest agent is not - * running. */ + * counterpart thaw command when it is actually sent to agent. + * The command will fail if the guest is paused or the guest agent + * is not running, or is already quiesced. */ if (flags & VIR_DOMAIN_SNAPSHOT_CREATE_QUIESCE) { - if (qemuDomainSnapshotFSFreeze(vm) < 0) { - /* helper reported the error */ - thaw = -1; + int freeze = qemuDomainSnapshotFSFreeze(driver, vm); + if (freeze < 0) { + /* the helper reported the error */ + if (freeze == -2) + thaw = -1; /* the command is sent but agent failed */ goto endjob; - } else { - thaw = 1; } + thaw = 1; } /* We need to track what state the guest is in, since taking the @@ -13265,7 +13305,7 @@ qemuDomainSnapshotCreateActiveExternal(virConnectPtr conn, goto cleanup; } if (vm && thaw != 0 && - qemuDomainSnapshotFSThaw(vm, thaw > 0) < 0) { + qemuDomainSnapshotFSThaw(driver, vm, thaw > 0) < 0) { /* helper reported the error, if it was needed */ if (thaw > 0) ret = -1;

Use qemuDomainSnapshotFSFreeze() and qemuDomainSnapshotFSFreeze() which are already implemented for snapshot quiescing. Signed-off-by: Tomoki Sekiyama <tomoki.sekiyama@hds.com> --- src/qemu/qemu_driver.c | 78 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 7fd8b6d..ebd27dd 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -16701,6 +16701,82 @@ qemuConnectGetCPUModelNames(virConnectPtr conn, } +static int +qemuDomainFSFreeze(virDomainPtr dom, + unsigned int flags) +{ + virQEMUDriverPtr driver = dom->conn->privateData; + virDomainObjPtr vm; + int ret = -1; + + virCheckFlags(0, -1); + + if (!(vm = qemuDomObjFromDomain(dom))) + goto cleanup; + + if (virDomainFSFreezeEnsureACL(dom->conn, vm->def) < 0) + goto cleanup; + + if (qemuDomainObjBeginJob(driver, vm, QEMU_JOB_MODIFY) < 0) + goto cleanup; + + if (!virDomainObjIsActive(vm)) { + virReportError(VIR_ERR_OPERATION_INVALID, + "%s", _("domain is not running")); + goto endjob; + } + + ret = qemuDomainSnapshotFSFreeze(driver, vm); + + endjob: + if (!qemuDomainObjEndJob(driver, vm)) + vm = NULL; + + cleanup: + if (vm) + virObjectUnlock(vm); + return ret; +} + + +static int +qemuDomainFSThaw(virDomainPtr dom, + unsigned int flags) +{ + virQEMUDriverPtr driver = dom->conn->privateData; + virDomainObjPtr vm; + int ret = -1; + + virCheckFlags(0, -1); + + if (!(vm = qemuDomObjFromDomain(dom))) + goto cleanup; + + if (virDomainFSThawEnsureACL(dom->conn, vm->def) < 0) + goto cleanup; + + if (qemuDomainObjBeginJob(driver, vm, QEMU_JOB_MODIFY) < 0) + goto cleanup; + + if (!virDomainObjIsActive(vm)) { + virReportError(VIR_ERR_OPERATION_INVALID, + "%s", _("domain is not running")); + goto endjob; + } + + ret = qemuDomainSnapshotFSThaw(driver, vm, true); + + endjob: + if (!qemuDomainObjEndJob(driver, vm)) + vm = NULL; + + cleanup: + if (vm) + virObjectUnlock(vm); + return ret; +} + + static virDriver qemuDriver = { .no = VIR_DRV_QEMU, .name = QEMU_DRIVER_NAME, @@ -16891,6 +16967,8 @@ static virDriver qemuDriver = { .domainMigrateFinish3Params = qemuDomainMigrateFinish3Params, /* 1.1.0 */ .domainMigrateConfirm3Params = qemuDomainMigrateConfirm3Params, /* 1.1.0 */ .connectGetCPUModelNames = qemuConnectGetCPUModelNames, /* 1.1.3 */ + .domainFSFreeze = qemuDomainFSFreeze, /* 1.2.3 */ + .domainFSThaw = qemuDomainFSThaw, /* 1.2.3 */ };

These are exposed under domfsfreeze command and domfsthaw command. Signed-off-by: Tomoki Sekiyama <tomoki.sekiyama@hds.com> --- tools/virsh-domain.c | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++ tools/virsh.pod | 15 ++++++++ 2 files changed, 107 insertions(+) diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c index d9aa4fa..b0cf68d 100644 --- a/tools/virsh-domain.c +++ b/tools/virsh-domain.c @@ -11391,6 +11391,86 @@ cmdDomFSTrim(vshControl *ctl, const vshCmd *cmd) return ret; } +static const vshCmdInfo info_domfsfreeze[] = { + {.name = "help", + .data = N_("Freeze domain's mounted filesystems.") + }, + {.name = "desc", + .data = N_("Freeze domain's mounted filesystems.") + }, + {.name = NULL} +}; + +static const vshCmdOptDef opts_domfsfreeze[] = { + {.name = "domain", + .type = VSH_OT_DATA, + .flags = VSH_OFLAG_REQ, + .help = N_("domain name, id or uuid") + }, + {.name = NULL} +}; +static bool +cmdDomFSFreeze(vshControl *ctl, const vshCmd *cmd) +{ + virDomainPtr dom = NULL; + bool ret = false; + unsigned int flags = 0; + + if (!(dom = vshCommandOptDomain(ctl, cmd, NULL))) + return ret; + + if (virDomainFSFreeze(dom, flags) < 0) { + vshError(ctl, _("Unable to freeze filesystems")); + goto cleanup; + } + + ret = true; + + cleanup: + virDomainFree(dom); + return ret; +} + +static const vshCmdInfo info_domfsthaw[] = { + {.name = "help", + .data = N_("Thaw domain's mounted filesystems.") + }, + {.name = "desc", + .data = N_("Thaw domain's mounted filesystems.") + }, + {.name = NULL} +}; + +static const vshCmdOptDef opts_domfsthaw[] = { + {.name = "domain", + .type = VSH_OT_DATA, + .flags = VSH_OFLAG_REQ, + .help = N_("domain name, id or uuid") + }, + {.name = NULL} +}; +static bool +cmdDomFSThaw(vshControl *ctl, const vshCmd *cmd) +{ + virDomainPtr dom = NULL; + bool ret = false; + unsigned int flags = 0; + + if (!(dom = vshCommandOptDomain(ctl, cmd, NULL))) + return ret; + + if (virDomainFSThaw(dom, flags) < 0) { + vshError(ctl, _("Unable to thaw filesystems")); + goto cleanup; + } + + ret = true; + + cleanup: + virDomainFree(dom); + return ret; +} + const vshCmdDef domManagementCmds[] = { {.name = "attach-device", .handler = cmdAttachDevice, @@ -11538,6 +11618,18 @@ const vshCmdDef domManagementCmds[] = { .info = info_domdisplay, .flags = 0 }, + {.name = "domfsfreeze", + .handler = cmdDomFSFreeze, + .opts = opts_domfsfreeze, + .info = info_domfsfreeze, + .flags = 0 + }, + {.name = "domfsthaw", + .handler = cmdDomFSThaw, + .opts = opts_domfsthaw, + .info = info_domfsthaw, + .flags = 0 + }, {.name = "domfstrim", .handler = cmdDomFSTrim, .opts = opts_domfstrim, diff --git a/tools/virsh.pod b/tools/virsh.pod index 20352cb..3f41f77 100644 --- a/tools/virsh.pod +++ b/tools/virsh.pod @@ -941,6 +941,21 @@ Output a URI which can be used to connect to the graphical display of the domain via VNC, SPICE or RDP. If I<--include-password> is specified, the SPICE channel password will be included in the URI. +=item B<domfsfreeze> I<domain> + +Freeze all mounted filesystems within a running domain to prepare for +consistent snapshots. + +Note that B<snapshot-create> command has a I<--quiesce> option to freeze +and thaw the filesystems automatically to keep snapshots consistent. +B<domfsfreeze> command is only needed when a user wants to utilize the +native snapshot features of storage devices not supported by libvirt yet. + +=item B<domfsthaw> I<domain> + +Thaw all mounted filesystems within a running domain, which are frozen +by domfsfreeze command. + =item B<domfstrim> I<domain> [I<--minimum> B<bytes>] [I<--mountpoint mountPoint>]

* Tomoki Sekiyama <tomoki.sekiyama@hds.com> wrote:
Hello,
This is patchset v4 to add FSFreeze/FSThaw API for custom disk snapshotting.
This patchset works for my tests. Without pausing the guest I am able to quiesce all attached volumes. Does there exists an option to create a snapshot of only one (or some) of the volumes attached to a particular instance? -- Jon

On 03/27/2014 04:21 PM, Jon Bernard wrote:
* Tomoki Sekiyama <tomoki.sekiyama@hds.com> wrote:
Hello,
This is patchset v4 to add FSFreeze/FSThaw API for custom disk snapshotting.
This patchset works for my tests. Without pausing the guest I am able to quiesce all attached volumes. Does there exists an option to create a snapshot of only one (or some) of the volumes attached to a particular instance?
Interesting question. The virDomainSnapshotCreateXML call can quiesce a subset of disks (namely, only the disks that were specified in the snapshot). But as currently designed, the FSFreeze/FSThaw API have no way to specify a subset of disks; it is an all-or-none proposition. Perhaps that means the API is not quite right, and that we need a 'char**disks, int ndisks' parameter pair; if 'NULL, 0', we do all disks, if non-NULL, then we freeze only the listed set of disks (doing the same lookup for disks as for other APIs). I feel a bit bad that this was posted before freeze of 1.2.3, but we don't have it in yet, which makes it harder to commit until the 1.2.4 cycle. On the other hand, if we decide the API needs to be tweaked to add parameters for full usability, then maybe it's a good thing we don't have it in yet. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org

* Eric Blake <eblake@redhat.com> wrote:
On 03/27/2014 04:21 PM, Jon Bernard wrote:
* Tomoki Sekiyama <tomoki.sekiyama@hds.com> wrote:
Hello,
This is patchset v4 to add FSFreeze/FSThaw API for custom disk snapshotting.
This patchset works for my tests. Without pausing the guest I am able to quiesce all attached volumes. Does there exists an option to create a snapshot of only one (or some) of the volumes attached to a particular instance?
Interesting question. The virDomainSnapshotCreateXML call can quiesce a subset of disks (namely, only the disks that were specified in the snapshot). But as currently designed, the FSFreeze/FSThaw API have no way to specify a subset of disks; it is an all-or-none proposition.
Yes, if a user intends to only snapshot a subset of the attached volumes then quiescing all volumes would be unnecessary. I can think of pathological cases where the one one volume you don't need to snapshot is under heavy I/O and that causes the fsfreeze() to timeout and fail the entire operation (or something similar).
Perhaps that means the API is not quite right, and that we need a 'char**disks, int ndisks' parameter pair; if 'NULL, 0', we do all disks, if non-NULL, then we freeze only the listed set of disks (doing the same lookup for disks as for other APIs).
This is the direction I would lean in, IMO. In the case of Nova (openstack), the user would want to create a consistent set of snapshots for one, some, or all of the attached volumes. Having the granularity to only freeze, snapshot, and thaw the volumes in question would mean only the required amount of work is performed. One thing comes to mind: once volumes are quiesced, qemu-img could be used directly to create the snapshot. In the case of iSCSI-attached volumes, a remote snapshot command could be issued to create a snapshot. Once all have completed the guest then thawed. For local volumes, using qemu-img seems cumbersome to me. I feel there should be a libvirt call to handle at least the local volumes. -- Jon

On 03/27/2014 04:48 PM, Jon Bernard wrote:
One thing comes to mind: once volumes are quiesced, qemu-img could be used directly to create the snapshot. In the case of iSCSI-attached volumes, a remote snapshot command could be issued to create a snapshot. Once all have completed the guest then thawed.
For local volumes, using qemu-img seems cumbersome to me. I feel there should be a libvirt call to handle at least the local volumes.
virDomainSnapshotCreateXML already handles a snapshot of a subset of local volumes via qemu. The reason we are adding FSFreeze/FSThaw is to cater to situations where creating the snapshot via qemu is not feasible. But be aware that you generally must NOT use qemu-img on a disk that is in-use by qemu, as qemu may have pending changes to the image metadata that would cause qemu-img to misbehave. If a guest is live (even if paused or quiesced), the only safe things you can do to that guest are to backing files (which are read-only to qemu) or actions supported directly by qemu (and yes, we still need qemu to have a way to pivot to a new disk file if you have a way of doing a fast snapshot while the disk is quiesced but in a way different than how qemu does snapshots with qcow2 files). -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org

On 3/27/14 18:27 , "Eric Blake" <eblake@redhat.com> wrote:
On 03/27/2014 04:21 PM, Jon Bernard wrote:
* Tomoki Sekiyama <tomoki.sekiyama@hds.com> wrote:
Hello,
This is patchset v4 to add FSFreeze/FSThaw API for custom disk snapshotting.
This patchset works for my tests. Without pausing the guest I am able to quiesce all attached volumes. Does there exists an option to create a snapshot of only one (or some) of the volumes attached to a particular instance?
Interesting question. The virDomainSnapshotCreateXML call can quiesce a subset of disks (namely, only the disks that were specified in the snapshot). But as currently designed, the FSFreeze/FSThaw API have no way to specify a subset of disks; it is an all-or-none proposition.
Perhaps that means the API is not quite right, and that we need a 'char**disks, int ndisks' parameter pair; if 'NULL, 0', we do all disks, if non-NULL, then we freeze only the listed set of disks (doing the same lookup for disks as for other APIs).
This sounds reasonable for me to add disks parameters. I will try adding them in next version. Then the api will look like: int virDomainFSFreeze(virDomainPtr dom, char** disks, int ndisks, unsigned int flags); and int virDomainFSThaw(virDomainPtr dom, unsigned int flags); I feel that thaw api shouldn't have disks parameter and thaw every frozen disk in order to simplify fsfreeze exclusion. Is it acceptable? BTW, in current qemu-kvm, disk device names in Linux guests don't seem to always correspond to that is specified by libvirt (e.g. the first virtio storage is named "vda" even if it is specified "vdz" in the libvirt's xml). Until this is resolved, guest agent may ignore the disks parameter and fallback to freezing all mounted filesystems. But still, having disks parameter in the API level would be good for future extension.
I feel a bit bad that this was posted before freeze of 1.2.3, but we don't have it in yet, which makes it harder to commit until the 1.2.4 cycle. On the other hand, if we decide the API needs to be tweaked to add parameters for full usability, then maybe it's a good thing we don't have it in yet.
And I will update API version number for the API to 1.2.4 in next patch. :) Thanks, -- Tomoki Sekiyama

On 03/27/2014 05:54 PM, Tomoki Sekiyama wrote:
This sounds reasonable for me to add disks parameters. I will try adding them in next version. Then the api will look like:
int virDomainFSFreeze(virDomainPtr dom, char** disks, int ndisks, unsigned int flags); and int virDomainFSThaw(virDomainPtr dom, unsigned int flags);
I feel that thaw api shouldn't have disks parameter and thaw every frozen disk in order to simplify fsfreeze exclusion. Is it acceptable?
Maybe, maybe not. It means on a guest with 2 disks, I can't freeze one, then freeze the second - I can only have one atomic freeze at a time, and therefore thaw needs no parameters because it is always thawing the most recent freeze action. But we already know from painful experience that assuming at most one active job at a time doesn't scale well. Maybe it should be: int virDomainFSFreeze(...char **disks...) returns a handle virDomainFSThaw(virDomainPtr dom, int handle, unsigned int flags); where FSThaw then thaws according to the handle returned by a freeze. That would allow me to do: interleaved subsets: freeze A => returns 1 freeze B => returns 2 thaw 1 => thaws A thaw 2 => thaws B nested subsets freeze A => returns 3 freeze B => returns 4 thaw 4 => thaws B thaw 3 => thaws A
BTW, in current qemu-kvm, disk device names in Linux guests don't seem to always correspond to that is specified by libvirt (e.g. the first virtio storage is named "vda" even if it is specified "vdz" in the libvirt's xml).
Correct. Libvirt specifies destination names as a hint that must be unique to libvirt, but which has no bearing on the names used by the guest.
Until this is resolved, guest agent may ignore the disks parameter and fallback to freezing all mounted filesystems. But still, having disks parameter in the API level would be good for future extension.
Hmm, interesting point. Really, that means we want some way to map the strings understood by libvirt API into the disk names understood by the guest agent, and I don't know if we have such a means. But I guess this is an issue even for the existing virDomainSnapshotCreateXML with the quiesce flag, so it's not a new problem. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org

On 3/27/14 20:32 , "Eric Blake" <eblake@redhat.com> wrote:
On 03/27/2014 05:54 PM, Tomoki Sekiyama wrote:
This sounds reasonable for me to add disks parameters. I will try adding them in next version. Then the api will look like:
int virDomainFSFreeze(virDomainPtr dom, char** disks, int ndisks, unsigned int flags); and int virDomainFSThaw(virDomainPtr dom, unsigned int flags);
I feel that thaw api shouldn't have disks parameter and thaw every frozen disk in order to simplify fsfreeze exclusion. Is it acceptable?
Maybe, maybe not. It means on a guest with 2 disks, I can't freeze one, then freeze the second - I can only have one atomic freeze at a time, and therefore thaw needs no parameters because it is always thawing the most recent freeze action.
But we already know from painful experience that assuming at most one active job at a time doesn't scale well. Maybe it should be:
int virDomainFSFreeze(...char **disks...) returns a handle virDomainFSThaw(virDomainPtr dom, int handle, unsigned int flags);
where FSThaw then thaws according to the handle returned by a freeze. That would allow me to do:
interleaved subsets: freeze A => returns 1 freeze B => returns 2 thaw 1 => thaws A thaw 2 => thaws B
nested subsets freeze A => returns 3 freeze B => returns 4 thaw 4 => thaws B thaw 3 => thaws A
What should happen when freezing the same disk is requested? Currently it causes error. Another model can be counting freeze request and defer thawing until thaw is requested the same time: freeze A,B => returns 1 [counter A:1 B:1] freeze A => returns 2 [counter A:2 B:1] thaw 1 => thaw B [counter A:1 B:0] thaw 2 => thaw A [counter A:0 B:0] And in qemu driver, until the issue about mapping disk name of guests and libvirt is resolved, we could fallback to freeze every filesystem while the counter > 0: freeze A,B => returns 1 [counter 1] freeze every fs freeze A => returns 2 [counter 2] thaw 1 => [counter 1] thaw 2 => [counter 0] thaw every fs Another aspect we need to consider is that freezing is done on filesystem level, not on disk level. So if the filesystems lay on multiple disks using LVM and so on, the situation will be more complex. Maybe the fs should be frozen if a part of its disks are contained in the request, although that wouldn't be useful for consistent snapshotting....
BTW, in current qemu-kvm, disk device names in Linux guests don't seem to always correspond to that is specified by libvirt (e.g. the first virtio storage is named "vda" even if it is specified "vdz" in the libvirt's xml).
Correct. Libvirt specifies destination names as a hint that must be unique to libvirt, but which has no bearing on the names used by the guest.
Until this is resolved, guest agent may ignore the disks parameter and fallback to freezing all mounted filesystems. But still, having disks parameter in the API level would be good for future extension.
Hmm, interesting point. Really, that means we want some way to map the strings understood by libvirt API into the disk names understood by the guest agent, and I don't know if we have such a means. But I guess this is an issue even for the existing virDomainSnapshotCreateXML with the quiesce flag, so it's not a new problem.
Was there any discussion about this issue before? Thanks, Tomoki Sekiyama

On 3/28/14 12:08 , "Tomoki Sekiyama" <tomoki.sekiyama@hds.com> wrote:
On 3/27/14 20:32 , "Eric Blake" <eblake@redhat.com> wrote:
On 03/27/2014 05:54 PM, Tomoki Sekiyama wrote:
This sounds reasonable for me to add disks parameters. I will try adding them in next version. Then the api will look like:
int virDomainFSFreeze(virDomainPtr dom, char** disks, int ndisks, unsigned int flags); and int virDomainFSThaw(virDomainPtr dom, unsigned int flags);
I feel that thaw api shouldn't have disks parameter and thaw every frozen disk in order to simplify fsfreeze exclusion. Is it acceptable?
Maybe, maybe not. It means on a guest with 2 disks, I can't freeze one, then freeze the second - I can only have one atomic freeze at a time, and therefore thaw needs no parameters because it is always thawing the most recent freeze action.
But we already know from painful experience that assuming at most one active job at a time doesn't scale well. Maybe it should be:
int virDomainFSFreeze(...char **disks...) returns a handle virDomainFSThaw(virDomainPtr dom, int handle, unsigned int flags);
where FSThaw then thaws according to the handle returned by a freeze. <snip>
While trying this, now I'm feeling that int virDomainFSFreeze(...char **disks...) and int virDomainFSThaw(...char **disks...) are more flexible and simpler for both users and libvirt, because tracking handles makes it much complex because fsfreeze may also called from some APIs like virDomainSnapshotCreate internally. So I'll come up with the patchset v5 with the prototypes above. Thanks, Tomoki Sekiyama
participants (3)
-
Eric Blake
-
Jon Bernard
-
Tomoki Sekiyama