11:11 a.m.
From: Eric Blake <eblake(a)redhat.com>
Prepare for new backup APIs by describing the XML that will represent
a backup. The XML resembles snapshots and checkpoints in being able
to select actions for a set of disks, but has other differences. It
can support both push model (the hypervisor does the backup directly
into the destination file) and pull model (the hypervisor exposes an
access port for a third party to grab what is necessary). Add
testsuite coverage for some minimal uses of the XML.
The <disk> element within <domainbackup> tries to model the same
elements as a <disk> under <domain>, but sharing the RNG grammar
proved to be hairy. That is in part because while <domain> use
<source> to describe a host resource in use by the guest, a backup job
is using a host resource that is not visible to the guest: a push
backup action is instead describing a <target> (which ultimately could
be a remote network resource, but for simplicity the RNG just
validates a local file for now), and a pull backup action is instead
describing a temporary local file <scratch> (which probably should not
be a remote resource). A future refactoring may thus introduce some
way to parameterize RNG to accept <disk type='FOO'>...</disk> so that
the name of the subelement can be <source> for domain, or <target> or
<scratch> as needed for backups. Future patches may improve this area
of code.
Signed-off-by: Eric Blake <eblake(a)redhat.com>
---
docs/docs.html.in | 3 +-
docs/format.html.in | 1 +
docs/formatbackup.html.in | 168 ++++++++++++++
docs/formatcheckpoint.html.in | 12 +-
docs/index.html.in | 3 +-
docs/schemas/domainbackup.rng | 219 +++++++++++++++++++
libvirt.spec.in | 1 +
mingw-libvirt.spec.in | 2 +
tests/Makefile.am | 2 +
tests/domainbackupxml2xmlin/backup-pull.xml | 9 +
tests/domainbackupxml2xmlin/backup-push.xml | 9 +
tests/domainbackupxml2xmlin/empty.xml | 1 +
tests/domainbackupxml2xmlout/backup-pull.xml | 9 +
tests/domainbackupxml2xmlout/backup-push.xml | 9 +
tests/domainbackupxml2xmlout/empty.xml | 7 +
tests/virschematest.c | 2 +
16 files changed, 449 insertions(+), 8 deletions(-)
create mode 100644 docs/formatbackup.html.in
create mode 100644 docs/schemas/domainbackup.rng
create mode 100644 tests/domainbackupxml2xmlin/backup-pull.xml
create mode 100644 tests/domainbackupxml2xmlin/backup-push.xml
create mode 100644 tests/domainbackupxml2xmlin/empty.xml
create mode 100644 tests/domainbackupxml2xmlout/backup-pull.xml
create mode 100644 tests/domainbackupxml2xmlout/backup-push.xml
create mode 100644 tests/domainbackupxml2xmlout/empty.xml
diff --git a/docs/docs.html.in b/docs/docs.html.in
index 6cf09f51bc..6eb85b8d25 100644
--- a/docs/docs.html.in
+++ b/docs/docs.html.in
@@ -82,7 +82,8 @@
<a href="formatnode.html">node devices</a>,
<a href="formatsecret.html">secrets</a>,
<a href="formatsnapshot.html">snapshots</a>,
- <a
href="formatcheckpoint.html">checkpoints</a></dd>
+ <a href="formatcheckpoint.html">checkpoints</a>,
+ <a href="formatbackup.html">backup jobs</a></dd>
<dt><a href="uri.html">URI format</a></dt>
<dd>The URI formats used for connecting to libvirt</dd>
diff --git a/docs/format.html.in b/docs/format.html.in
index 3be2237663..d013528fe0 100644
--- a/docs/format.html.in
+++ b/docs/format.html.in
@@ -27,6 +27,7 @@
<li><a href="formatsecret.html">Secrets</a></li>
<li><a
href="formatsnapshot.html">Snapshots</a></li>
<li><a
href="formatcheckpoint.html">Checkpoints</a></li>
+ <li><a href="formatbackup.html">Backup
jobs</a></li>
</ul>
<h2>Command line validation</h2>
diff --git a/docs/formatbackup.html.in b/docs/formatbackup.html.in
new file mode 100644
index 0000000000..272d9cf6ab
--- /dev/null
+++ b/docs/formatbackup.html.in
@@ -0,0 +1,168 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE html>
+<html xmlns="http://www.w3.org/1999/xhtml">
+ <body>
+ <h1>Backup XML format</h1>
+
+ <ul id="toc"></ul>
+
+ <h2><a id="BackupAttributes">Backup XML</a></h2>
+
+ <p>
+ Creating a backup, whether full or incremental, is done
+ via <code>virDomainBackupBegin()</code>, which takes an XML
+ description of the actions to perform, as well as an optional
+ second XML document <a href="formatcheckpoint.html">describing a
+ checkpoint</a> to create at the same point in time. See
+ also <a href="domainstatecapture.html">a comparison</a>
between
+ the various state capture APIs.
+ </p>
+ <p>
+ There are two general modes for backups: a push mode (where the
+ hypervisor writes out the data to the destination file, which
+ may be local or remote), and a pull mode (where the hypervisor
+ creates an NBD server that a third-party client can then read as
+ needed, and which requires the use of temporary storage,
+ typically local, until the backup is complete).
+ </p>
+ <p>
+ The instructions for beginning a backup job are provided as
+ attributes and elements of the
+ top-level <code>domainbackup</code> element. This element
+ includes an optional attribute <code>mode</code> which can be
+ either "push" or "pull" (default
+ push). <code>virDomainBackupGetXMLDesc()</code> can be used to
+ see the actual values selected for elements omitted during
+ creation (for example, learning which port the NBD server is
+ using in the pull model or what file names libvirt generated
+ when none were supplied). The following child elements and attributes
+ are supported:
+ </p>
+ <dl>
+ <dt>attribute <code>id</code></dt>
+ <dd>Ignored on input, this element is the job id of the backup
+ operation returned on success
+ from <code>virDomainBackupBegin()</code>, and is used for
+ selecting which backup operation to target
+ during <code>virDomainBackupGetXMLDesc()</code>
+ and <code>virDomainBackupEnd()</code>. </dd>
+ <dt><code>incremental</code></dt>
+ <dd>An optional element giving the name of an existing
+ checkpoint of the domain, which will be used to make this
+ backup an incremental one. In the push model, only changes
+ since the named checkpoint are written to the destination. In
+ the pull model, the NBD server uses the
+ NBD_OPT_SET_META_CONTEXT extension to advertise to the client
+ which portions of the export contain changes since the named
+ checkpoint. If omitted, a full backup is performed.
+ </dd>
+ <dt><code>server</code></dt>
+ <dd>Present only for a pull mode backup. Contains the same
+ attributes as
+ the <a
href="formatdomain.html#elementsDisks"><code>protocol</code>
+ element of a disk</a> attached via NBD in the domain (such as
+ transport, socket, name, port, or tls), necessary to set up an
+ NBD server that exposes the content of each disk at the time
+ the backup is started.
+ </dd>
+ <dt><code>disks</code></dt>
+ <dd>An optional listing of instructions for disks participating
+ in the backup (if omitted, all disks participate and libvirt
+ attempts to generate filenames by appending the current
+ timestamp as a suffix). If the entire element was omitted on
+ input, then all disks participate in the backup, otherwise,
+ only the disks explicitly listed which do not also
+ use <code>backup='no'</code> will participate. On output,
this
+ is the state of each of the domain's disk in relation to the
+ backup operation.
+ <dl>
+ <dt><code>disk</code></dt>
+ <dd>This sub-element describes the backup properties of a
+ specific disk, with the following attributes and child
+ elements:
+ <dl>
+ <dt><code>name</code></dt>
+ <dd>A mandatory attribute which must match
+ the <code><target dev='name'/></code>
+ of one of
+ the <a href="formatdomain.html#elementsDisks">disk
+ devices</a> specified for the domain at the time of
+ the checkpoint.</dd>
+ <dt><code>backup</code></dt>
+ <dd>Setting this attribute to <code>yes</code>(default)
specifies
+ that the disk should take part in the backup and using
+ <code>no</code> excludes the disk from the
backup.</dd>
+ <dt><code>type</code></dt>
+ <dd>A mandatory attribute to describe the type of the
+ disk, except when <code>backup='no'</code> is
+ used. Valid values include <code>file</code>,
+ <code>block</code>, or <code>network</code>.
+ Similar to a disk declaration for a domain, the choice of type
+ controls what additional sub-elements are needed to describe
+ the destination (such as <code>protocol</code> for a
+ network destination).</dd>
+ <dt><code>target</code></dt>
+ <dd>Valid only for push mode backups, this is the
+ primary sub-element that describes the file name of
+ the backup destination, similar to
+ the <code>source</code> sub-element of a domain
+ disk. An optional sub-element <code>driver</code> can
+ also be used, with an attribute <code>type</code> to
+ specify a destination format different from
+ qcow2. </dd>
+ <dt><code>scratch</code></dt>
+ <dd>Valid only for pull mode backups, this is the
+ primary sub-element that describes the file name of
+ the local scratch file to be used in facilitating the
+ backup, and is similar to the <code>source</code>
+ sub-element of a domain disk.</dd>
+ </dl>
+ </dd>
+ </dl>
+ </dd>
+ </dl>
+
+ <h2><a id="example">Examples</a></h2>
+
+ <p>Use <code>virDomainBackupBegin()</code> to perform a full
+ backup using push mode. The example lets libvirt pick the
+ destination and format for 'vda', fully specifies that we want a
+ raw backup of 'vdb', and omits 'vdc' from the operation.
+ </p>
+ <pre>
+<domainbackup>
+ <disks/>
+ <disk name='vda' backup='yes'/>
+ <disk name='vdb' type='file'>
+ <target file='/path/to/vdb.backup'/>
+ <driver type='raw'/>
+ </disk>
+ <disk name='vdc' backup='no'/>
+ </disks/>
+</domainbackup>
+ </pre>
+
+ <p>If the previous full backup also passed a parameter describing
+ <a href="formatcheckpoint.html">checkpoint XML</a> that
resulted
+ in a checkpoint named <code>1525889631</code>, we can make
+ another call to <code>virDomainBackupBegin()</code> to perform
+ an incremental backup of just the data changed since that
+ checkpoint, this time using the following XML to start a pull
+ model export of the 'vda' and 'vdb' disks, where a third-party
+ NBD client connecting to '/path/to/server' completes the backup
+ (omitting 'vdc' from the explicit list has the same effect as
+ the backup='no' from the previous example):
+ </p>
+ <pre>
+<domainbackup mode="pull">
+ <incremental>1525889631</incremental>
+ <server transport="unix" socket="/path/to/server"/>
+ <disks/>
+ <disk name='vda' backup='yes' type='file'>
+ <scratch file='/path/to/file1.scratch'/>
+ </disk>
+ </disks/>
+</domainbackup>
+ </pre>
+ </body>
+</html>
diff --git a/docs/formatcheckpoint.html.in b/docs/formatcheckpoint.html.in
index 044bbfe4b0..ee56194523 100644
--- a/docs/formatcheckpoint.html.in
+++ b/docs/formatcheckpoint.html.in
@@ -28,12 +28,12 @@
first checkpoint and the second backup operation), it is
possible to do an offline reconstruction of the state of the
disk at the time of the second backup without having to copy as
- much data as a second full backup would require. Future API
- additions will make it possible to create checkpoints in
- conjunction with a backup
- via <code>virDomainBackupBegin()</code> or with an external
- snapshot via <code>virDomainSnapshotCreateXML2</code>; but for
- now, libvirt exposes enough support to create disk checkpoints
+ much data as a second full backup would require. Most disk
+ checkpoints are created in conjunction with a backup
+ via <code>virDomainBackupBegin()</code>, although a future API
+ addition of <code>virDomainSnapshotCreateXML2()</code> will also
+ make this possible when creating external snapshots; however,
+ libvirt also exposes enough support to create disk checkpoints
independently from a backup operation
via <code>virDomainCheckpointCreateXML()</code> <span
class="since">since
5.6.0</span>. Likewise, the creation of checkpoints when
diff --git a/docs/index.html.in b/docs/index.html.in
index 7d0ab650e3..26e8406917 100644
--- a/docs/index.html.in
+++ b/docs/index.html.in
@@ -59,7 +59,8 @@
<a href="formatnode.html">node devices</a>,
<a href="formatsecret.html">secrets</a>,
<a href="formatsnapshot.html">snapshots</a>,
- <a
href="formatcheckpoint.html">checkpoints</a></dd>
+ <a href="formatcheckpoint.html">checkpoints</a>,
+ <a href="formatbackup.html">backup jobs</a></dd>
<dt><a
href="http://wiki.libvirt.org">Wiki</a></dt>
<dd>Read further community contributed content</dd>
</dl>
diff --git a/docs/schemas/domainbackup.rng b/docs/schemas/domainbackup.rng
new file mode 100644
index 0000000000..92327e7077
--- /dev/null
+++ b/docs/schemas/domainbackup.rng
@@ -0,0 +1,219 @@
+<?xml version="1.0"?>
+<!-- A Relax NG schema for the libvirt domain backup properties XML format -->
+<grammar xmlns="http://relaxng.org/ns/structure/1.0">
+ <start>
+ <ref name='domainbackup'/>
+ </start>
+
+ <include href='domaincommon.rng'/>
+
+ <define name='domainbackup'>
+ <element name='domainbackup'>
+ <optional>
+ <attribute name='id'>
+ <ref name="unsignedInt"/>
+ </attribute>
+ </optional>
+ <interleave>
+ <optional>
+ <element name='incremental'>
+ <text/>
+ </element>
+ </optional>
+ <choice>
+ <group>
+ <optional>
+ <attribute name='mode'>
+ <value>push</value>
+ </attribute>
+ </optional>
+ <ref name='backupDisksPush'/>
+ </group>
+ <group>
+ <attribute name='mode'>
+ <value>pull</value>
+ </attribute>
+ <interleave>
+ <element name='server'>
+ <choice>
+ <group>
+ <optional>
+ <attribute name='transport'>
+ <value>tcp</value>
+ </attribute>
+ </optional>
+ <attribute name='name'>
+ <choice>
+ <ref name='dnsName'/>
+ <ref name='ipAddr'/>
+ </choice>
+ </attribute>
+ <optional>
+ <attribute name='port'>
+ <ref name='unsignedInt'/>
+ </attribute>
+ </optional>
+ <!-- add tls? -->
+ </group>
+ <group>
+ <attribute name='transport'>
+ <value>unix</value>
+ </attribute>
+ <attribute name='socket'>
+ <ref name='absFilePath'/>
+ </attribute>
+ </group>
+ </choice>
+ </element>
+ <ref name='backupDisksPull'/>
+ </interleave>
+ </group>
+ </choice>
+ </interleave>
+ </element>
+ </define>
+
+ <define name='backupPushDriver'>
+ <optional>
+ <element name='driver'>
+ <attribute name='type'>
+ <ref name='storageFormat'/>
+ </attribute>
+ </element>
+ </optional>
+ </define>
+
+ <define name='backupAttr'>
+ <!-- valid values of <disk backup='XXX'> other than 'no'
-->
+ <optional>
+ <attribute name='backup'>
+ <choice>
+ <value>begin</value>
+ <value>inprogress</value>
+ <value>ready</value>
+ </choice>
+ </attribute>
+ </optional>
+ </define>
+
+ <define name='backupDisksPush'>
+ <optional>
+ <element name='disks'>
+ <oneOrMore>
+ <element name='disk'>
+ <attribute name='name'>
+ <choice>
+ <ref name='diskTarget'/>
+ <ref name='absFilePath'/>
+ </choice>
+ </attribute>
+ <optional>
+ <attribute name='shallow'>
+ <value>on</value>
+ </attribute>
+ </optional>
+ <choice>
+ <group>
+ <attribute name='backup'>
+ <value>no</value>
+ </attribute>
+ </group>
+ <!-- FIXME allow push to a network location, perhaps by
+ refactoring 'diskSource' to take element name by a
+ per-grammar ref -->
+ <group>
+ <ref name='backupAttr'/>
+ <optional>
+ <attribute name='type'>
+ <value>file</value>
+ </attribute>
+ </optional>
+ <interleave>
+ <optional>
+ <element name='target'>
+ <attribute name='file'>
+ <ref name='absFilePath'/>
+ </attribute>
+ </element>
+ </optional>
+ <ref name='backupPushDriver'/>
+ </interleave>
+ </group>
+ <group>
+ <ref name='backupAttr'/>
+ <attribute name='type'>
+ <value>block</value>
+ </attribute>
+ <interleave>
+ <optional>
+ <element name='target'>
+ <attribute name='dev'>
+ <ref name='absFilePath'/>
+ </attribute>
+ </element>
+ </optional>
+ <ref name='backupPushDriver'/>
+ </interleave>
+ </group>
+ <!--
+ <group>
+ <ref name='backupAttr'/>
+ <attribute name='type'>
+ <value>network</value>
+ </attribute>
+ ...
+ </group>
+ -->
+ </choice>
+ </element>
+ </oneOrMore>
+ </element>
+ </optional>
+ </define>
+
+ <define name='backupDisksPull'>
+ <optional>
+ <element name='disks'>
+ <oneOrMore>
+ <element name='disk'>
+ <attribute name='name'>
+ <choice>
+ <ref name='diskTarget'/>
+ <ref name='absFilePath'/>
+ </choice>
+ </attribute>
+ <choice>
+ <group>
+ <optional>
+ <attribute name='type'>
+ <value>file</value>
+ </attribute>
+ </optional>
+ <optional>
+ <element name='scratch'>
+ <attribute name='file'>
+ <ref name='absFilePath'/>
+ </attribute>
+ </element>
+ </optional>
+ </group>
+ <group>
+ <attribute name='type'>
+ <value>block</value>
+ </attribute>
+ <optional>
+ <element name='scratch'>
+ <attribute name='dev'>
+ <ref name='absFilePath'/>
+ </attribute>
+ </element>
+ </optional>
+ </group>
+ </choice>
+ </element>
+ </oneOrMore>
+ </element>
+ </optional>
+ </define>
+
+</grammar>
diff --git a/libvirt.spec.in b/libvirt.spec.in
index dcad08cb5f..0565992907 100644
--- a/libvirt.spec.in
+++ b/libvirt.spec.in
@@ -1900,6 +1900,7 @@ exit 0
%{_datadir}/libvirt/schemas/capability.rng
%{_datadir}/libvirt/schemas/cputypes.rng
%{_datadir}/libvirt/schemas/domain.rng
+%{_datadir}/libvirt/schemas/domainbackup.rng
%{_datadir}/libvirt/schemas/domaincaps.rng
%{_datadir}/libvirt/schemas/domaincheckpoint.rng
%{_datadir}/libvirt/schemas/domaincommon.rng
diff --git a/mingw-libvirt.spec.in b/mingw-libvirt.spec.in
index c29f3eeed2..453570fc3c 100644
--- a/mingw-libvirt.spec.in
+++ b/mingw-libvirt.spec.in
@@ -233,6 +233,7 @@ rm -rf $RPM_BUILD_ROOT%{mingw64_libexecdir}/libvirt-guests.sh
%{mingw32_datadir}/libvirt/schemas/capability.rng
%{mingw32_datadir}/libvirt/schemas/cputypes.rng
%{mingw32_datadir}/libvirt/schemas/domain.rng
+%{mingw32_datadir}/libvirt/schemas/domainbackup.rng
%{mingw32_datadir}/libvirt/schemas/domaincaps.rng
%{mingw32_datadir}/libvirt/schemas/domaincheckpoint.rng
%{mingw32_datadir}/libvirt/schemas/domaincommon.rng
@@ -324,6 +325,7 @@ rm -rf $RPM_BUILD_ROOT%{mingw64_libexecdir}/libvirt-guests.sh
%{mingw64_datadir}/libvirt/schemas/capability.rng
%{mingw64_datadir}/libvirt/schemas/cputypes.rng
%{mingw64_datadir}/libvirt/schemas/domain.rng
+%{mingw64_datadir}/libvirt/schemas/domainbackup.rng
%{mingw64_datadir}/libvirt/schemas/domaincaps.rng
%{mingw64_datadir}/libvirt/schemas/domaincheckpoint.rng
%{mingw64_datadir}/libvirt/schemas/domaincommon.rng
diff --git a/tests/Makefile.am b/tests/Makefile.am
index a9acd88670..d7459de9b1 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -89,6 +89,8 @@ EXTRA_DIST = \
capabilityschemadata \
commanddata \
cputestdata \
+ domainbackupxml2xmlin \
+ domainbackupxml2xmlout \
domaincapsschemadata \
domainconfdata \
domainschemadata \
diff --git a/tests/domainbackupxml2xmlin/backup-pull.xml
b/tests/domainbackupxml2xmlin/backup-pull.xml
new file mode 100644
index 0000000000..2ce5cd6711
--- /dev/null
+++ b/tests/domainbackupxml2xmlin/backup-pull.xml
@@ -0,0 +1,9 @@
+<domainbackup mode="pull">
+ <incremental>1525889631</incremental>
+ <server transport='tcp' name='localhost' port='10809'/>
+ <disks>
+ <disk name='vda' type='file'>
+ <scratch file='/path/to/file'/>
+ </disk>
+ </disks>
+</domainbackup>
diff --git a/tests/domainbackupxml2xmlin/backup-push.xml
b/tests/domainbackupxml2xmlin/backup-push.xml
new file mode 100644
index 0000000000..1b7d3061fd
--- /dev/null
+++ b/tests/domainbackupxml2xmlin/backup-push.xml
@@ -0,0 +1,9 @@
+<domainbackup mode="push">
+ <incremental>1525889631</incremental>
+ <disks>
+ <disk name='vda' type='file'>
+ <driver type='raw'/>
+ <target file='/path/to/file'/>
+ </disk>
+ </disks>
+</domainbackup>
diff --git a/tests/domainbackupxml2xmlin/empty.xml
b/tests/domainbackupxml2xmlin/empty.xml
new file mode 100644
index 0000000000..7ed511f97b
--- /dev/null
+++ b/tests/domainbackupxml2xmlin/empty.xml
@@ -0,0 +1 @@
+<domainbackup/>
diff --git a/tests/domainbackupxml2xmlout/backup-pull.xml
b/tests/domainbackupxml2xmlout/backup-pull.xml
new file mode 100644
index 0000000000..2ce5cd6711
--- /dev/null
+++ b/tests/domainbackupxml2xmlout/backup-pull.xml
@@ -0,0 +1,9 @@
+<domainbackup mode="pull">
+ <incremental>1525889631</incremental>
+ <server transport='tcp' name='localhost' port='10809'/>
+ <disks>
+ <disk name='vda' type='file'>
+ <scratch file='/path/to/file'/>
+ </disk>
+ </disks>
+</domainbackup>
diff --git a/tests/domainbackupxml2xmlout/backup-push.xml
b/tests/domainbackupxml2xmlout/backup-push.xml
new file mode 100644
index 0000000000..1b7d3061fd
--- /dev/null
+++ b/tests/domainbackupxml2xmlout/backup-push.xml
@@ -0,0 +1,9 @@
+<domainbackup mode="push">
+ <incremental>1525889631</incremental>
+ <disks>
+ <disk name='vda' type='file'>
+ <driver type='raw'/>
+ <target file='/path/to/file'/>
+ </disk>
+ </disks>
+</domainbackup>
diff --git a/tests/domainbackupxml2xmlout/empty.xml
b/tests/domainbackupxml2xmlout/empty.xml
new file mode 100644
index 0000000000..13600fbb1c
--- /dev/null
+++ b/tests/domainbackupxml2xmlout/empty.xml
@@ -0,0 +1,7 @@
+<domainbackup mode="push">
+ <disks>
+ <disk name="vda" type="file">
+ <target file="/path/to/file1.copy"/>
+ </disk>
+ </disks>
+</domainbackup>
diff --git a/tests/virschematest.c b/tests/virschematest.c
index dabbc02163..330c42b010 100644
--- a/tests/virschematest.c
+++ b/tests/virschematest.c
@@ -221,6 +221,8 @@ mymain(void)
"lxcxml2xmloutdata", "bhyvexml2argvdata",
"genericxml2xmlindata",
"genericxml2xmloutdata", "xlconfigdata",
"libxlxml2domconfigdata",
"qemuhotplugtestdomains");
+ DO_TEST_DIR("domainbackup.rng", "domainbackupxml2xmlin",
+ "domainbackupxml2xmlout");
DO_TEST_DIR("domaincaps.rng", "domaincapsschemadata");
DO_TEST_DIR("domaincheckpoint.rng",
"qemudomaincheckpointxml2xmlin",
"qemudomaincheckpointxml2xmlout");
--
2.21.0