10:59 p.m.
This is the counter-proposal to my earlier RFC for storage migration via
snapshot mirrors[1], resulting from a NACK on the code review for that
earlier proposal[2]. In particular, this proposal fleshes out some of
Paolo's design overview on the qemu wiki[3].
[1] https://www.redhat.com/archives/libvir-list/2012-March/msg00578.html
[2] https://www.redhat.com/archives/libvir-list/2012-March/msg01033.html
[3] http://wiki.qemu.org/Features/SnapshotsMultipleDevices
My plan is to have everything in this RFC coded up in the next couple of
days (hopefully no later than Thursday); this has missed the feature
freeze for 0.9.11, so it should not be applied upstream until after the
weekend release, as one of the first patches for 0.9.12. Backport-wise,
the new flags can be backported as far back as the 0.9.10 .so API, but
the new virDomainBlockCopy() API cannot be exported when doing a
backport without breaking .so versions (although it's implemenation can
be used internally).
Additions
=========
The following new error code will be added:
VIR_ERR_BLOCK_COPY_IN_PROGRESS
The following new API will be added:
int virDomainBlockCopy(virDomainPtr dom,
const char *disk,
const char *base,
const char *dest,
const char *format,
unsigned long bandwidth,
unsigned int flags);
The following new named values will be added:
enum virDomainBlockJobType (used in virDomainBlockJobInfo):
VIR_DOMAIN_BLOCK_JOB_TYPE_COPY = 2
The following new flags will be added:
for virDomainBlockRebase:
VIR_DOMAIN_BLOCK_REBASE_SHALLOW = 1 << 0
VIR_DOMAIN_BLOCK_REBASE_REUSE_EXT = 1 << 1
VIR_DOMAIN_BLOCK_REBASE_COPY = 1 << 2
for virDomainBlockCopy:
VIR_DOMAIN_BLOCK_COPY_SHALLOW = 1 << 0
VIR_DOMAIN_BLOCK_COPY_REUSE_EXT = 1 << 1
for virDomainBlockJobAbort:
VIR_DOMAIN_BLOCK_JOB_ABORT_PIVOT
Add some XML:
Under //domain/drivers/disk, next to <source file='...'/>, add <mirror
file='...'/>
Semantics
=========
virDomainBlockCopy sets up a BLOCK_JOB_TYPE_COPY job. 'disk' names the
disk to be copied (can be 'vda' or '/path/to/source', as with other
block commands) and must not be NULL. 'base' names the path to the
backing file in the chain of the source that will be the new backing
file of the destination; if this parameter is NULL, then the destination
file defaults to a complete block pull, but the COPY_SHALLOW flag
instead requests a pull of just the top file in the source backing
chain. 'dest' names the copy being created and must not be NULL;
normally, this file is created by the hypervisor/libvirt, but the
COPY_REUSE_EXT flag lets an application pass in a pre-created file
(allowing metadata to include a relative instead of absolute backing
file name). 'format' gives the format of the copy, or NULL to either
probe the format of a COPY_REUSE_EXT dest or to reuse the same format as
the source. flags cannot contain COPY_SHALLOW unless 'base' is NULL.
Once a block copy job is started, calls to virDomainGetBlockJobInfo()
for the same 'disk' will report an info with
VIR_DOMAIN_BLOCK_JOB_TYPE_COPY as the type. This job never completes on
its own, but must be stopped by the user (this enables mirroring to
continue until the user informs libvirt that any backing files, perhaps
located at different locations as specified by relative path names using
REUSE_EXT, have been externally copied into place). There are two
phases to a TYPE_COPY job. In the first phase, cur < end when querying
progress, calls to virDomainBlockJobAbort(dom, disk, 0) will cancel the
operation and revert to the source, and calls to
virDomainBlockJobAbort(dom, disk, VIR_DOMAIN_BLOCK_JOB_ABORT_PIVOT) will
fail with VIR_ERR_BLOCK_COPY_IN_PROGRESS. In the second phase, cur ==
end when querying progress, calls to virDomainBlockJobAbort(dom, disk,
0) will break the mirroring and revert to the source, while calls to
virDomainBlockJobAbort(dom, disk, VIR_DOMAIN_BLOCK_JOB_ABORT_PIVOT) will
break the mirroring and pivot to the destination. Use of
VIR_DOMAIN_BLOCK_JOB_ABORT_PIVOT on a non-copy block job will fail with
VIR_ERR_INVALID_ARG.
virDomainBlockRebase(dom, disk, dest, bandwidth,
VIR_DOMAIN_BLOCK_REBASE_COPY | (flags & 3)) is shorthand for
virDomainBlockCopy(dom, disk, NULL, dest, NULL, bandwidth, (flags & 3))
- that is, use of the REBASE_COPY flag treats the BlockRebase 'base'
argument as the BlockCopy 'dest' argument, creates the destination with
the same file format as the source (or probes the backing file format if
REBASE_REUSE_EXT is used), and passes the COPY_SHALLOW and
COPY_REUSE_EXT flags through (note that the similarly named flags were
conveniently chosen to be the same values). Attempts to use
REBASE_SHALLOW or REBASE_REUSE_EXT without also using REBASE_COPY will
fail with VIR_ERR_INVALID_ARG.
While a copy operation is in place, virDomainGetXMLDesc (dumpxml) will
show the <mirror> element for that <disk>.
Initial Implementation
======================
When virDomainBlockCopy is called (perhaps via the virDomainBlockRebase
alias), it first sets up a mirror using the 'drive-mirror' monitor
command and the destination file name. The mirror is opened with the
'existing' mode if _REUSE_EXT is present; otherwise it is opened with
the 'absolute-paths' mode if _SHALLOW is present or 'no-backing-file'
mode if no flags are present. Next, the function calls the
'block_stream' monitor command to start the streaming. The streaming
command uses 'base' as its starting point, except that when _SHALLOW was
specified, libvirt will use the backing file of the source disk, rather
than NULL (this can be obtained by the 'query-block' monitor command;
although someday libvirt should start tracking this information in
<domain> XML rather than relying on qemu). At this point, control
returns to the user, and the stream proceeds in the background;
virDomainBlockJobSetSpeed can tune the speed of the block streaming.
At least in the initial implementation, as long as the block job is
active, libvirt will prevent 'virDomainMigrate', 'virDomainSave',
'virDomainSnapshotCreateXML' in general, and 'virDomainDetachDevice' of
the disk being mirrored, all with the new VIR_ERR_BLOCK_COPY_IN_PROGRESS
error. This is because I don't have an easy way to resume mirroring
when restarting a new qemu process; preventing these actions until the
user first cancels the ongoing mirroring will result in fewer corner
cases that libvirt has to worry about. This also implies that the
initial implementation will fail for persistent domains, and only be
useful for transient domains. Attempts to define a domain with a
<mirror> element are rejected, leaving <mirror> as output-only XML
useful in restoring state when restarting libvirtd. It may be possible
to add persistent support in the future, once we determine how to make
qemu resume a mirrored block device; at that point, it would be possible
to specify <mirror> in domain xml during domain creation or during
device hotplug.
When the block stream finishes, qemu will send an event to libvirt
(libvirt will also have to manually check for completion on a libvirtd
restart, based on whether cur == end in the block job info). I'm not
yet sure whether to expose this event to the user so that they do not
have to poll the block job info, or whether to consume it internally.
At any rate, before this event occurs, the BLOCK_JOB_ABORT_PIVOT flag is
rejected, and virDomainBlockJobAbort without flags uses the
'block_job_cancel' monitor command to stop the streaming early, then the
'drive-reopen' monitor command to break the mirroring back to the
source; it is feasible that there is a race where a 'block_job_cancel'
can be called after the pull is complete but before the completion event
has been processed, so the code must proceed on to the 'drive-reopen'
even if the job cancel fails. After the event occurs,
virDomainBlockJobAbort only needs to use the 'drive-reopen' monitor
command, with either the source or the destination file depending on the
BLOCK_JOB_ABORT_PIVOT flag.
Until 'drive-reopen' is made atomic in qemu (by adding code to support
it inside 'transaction'), the user risks a block job abort rendering the
disk unusable, because the source was closed before the destination was
opened; hopefully this situation is rare, in part because libvirt will
do stat() checks and SELinux labeling on destination files before
starting qemu monitor commands, as a sanity check that qemu will be able
to use the specified files. If qemu ever adds atomic 'drive-reopen'
support, we can add a new flag BLOCK_JOB_ABORT_ATOMIC that fails on
older qemu, and ensures the use of 'transaction' on the newer qemu that
supports an atomic reopen.
If a block mirror is aborted (whether by the user calling
virDomainBlockJobAbort with no flag, or by the qemu process ending due
to things like a guest-initiated shutdown), then the mirror can be
safely discarded, and restarting the domain will be unmirrored where the
virDomainBlockRebase can be called again from scratch.
Examples
========
For some examples, starting with base <- snap1 <- snap2 <- snap3 as the
backing chain for disk 'vda',
virDomainBlockCopy(dom, "vda", NULL, "/path/to/copy", NULL, 0)
would set up a job that results in /path/to/copy being the same file
format as snap3, but containing the entire chain
virDomainBlockCopy(dom, "vda", "/path/to/snap1",
"/path/to/copy",
"qed", 0)
would set up a job that results in base <- snap1 <- copy as the mirrored
backing chain, and ensuring that copy is formatted as qed regardless of
the format of snap3
virDomainBlockCopy(dom, "vda", NULL, "/path/to/copy",
NULL, VIR_DOMAIN_BLOCK_COPY_SHALLOW)
is shorthand for
virDomainBlockCopy(dom, "vda", "/path/to/snap2",
"/path/to/copy",
NULL, 0)
and results in base <- snap1 <- snap2 <- copy creates copy using the
same format as snap3
virDomainBlockCopy(dom, "vda", "/path/to/snap2",
"/path/to/copy",
NULL, VIR_DOMAIN_BLOCK_COPY_REUSE_EXT)
requires /path/to/copy to already exist, probes it for existing format
(which might be different from snap3), and proceeds to mirror everything
so that snap2 is the base of copy (and the user is at fault if the
pre-existing file doesn't call out a backing file that happens to be
identical in content to snap2)
oVirt will probably use the sequence:
- use qemu-img to create an empty qcow2 file with relative backing name
to the destination storage
- call virDomainBlockRebase(dom, disk, "/path/to/copy",
VIR_DOMAIN_BLOCK_REBASE_COPY | VIR_DOMAIN_BLOCK_REBASE_SHALLOW |
VIR_DOMAIN_BLOCK_REBASE_REUSE_EXT)
- copy the base files from source to destination storage (this can be
done in parallel, either before or after the virDomainBlockRebase call)
- wait for the block pull to finish (either by waiting for an event if I
propagate the event, or by polling virDomainBlockJobInfo, or even by
polling virDomainBlockJobAbort(VIR_DOMAIN_BLOCK_JOB_ABORT_PIVOT) and
checking for VIR_ERR_BLOCK_COPY_IN_PROGRESS
- once both the base files are in place and the block pull half of the
copy job is complete (and without regard to whether the block stream or
the external base file copying completed first), call
virDomainBlockJobAbort(,VIR_DOMAIN_BLOCK_JOB_ABORT_PIVOT) to reopen to
the new storage domain chain
Comparison to first RFC
=======================
This proposal exposes only one disk at a time, while the earlier
virDomainSnapshotCreateXML <mirror> approach could atomically set up
mirroring on multiple disks. However, the nature of block jobs being a
background process means that parallel jobs can be run on independent
disks, so the user can do the overall block migration with the time cost
of the slowest disk, rather than having to do things serially with the
time cost of all disks added together.
This proposal avoids having to create an intermediate snapshot, so the
pull is more efficient and the source chain does not get longer, no
matter how many times the process is aborted and restarted.
This proposal can expose the no-backing-file mode, while the snapshot
approach did not.
To survive across libvirtd restarts, the snapshot approach was using
<domainsnapshot> to store the mirroring status in a user-visible
location; this approach has to modify the internal live xml (alongside
other internal data, such as the qemu pid). Or perhaps I can add a
<mirror> subelement to <disk> of a domain and make it user-visible after
all, and treat that as an output-only parameter for now.
Both approaches face the dilemma of how to start a new qemu process with
mirroring intact, and my solution in both patch series will be to
prevent any action that would force libvirt to save domain state until
after the user has first canceled all current mirroring jobs. This
limitation is not permanent - if future qemu provides better ways to
restart mirroring, and as libvirt is taught to store the full backing
chain in <domain> xml instead of probing it on the fly, we can relax
this restriction in the future.
--
Eric Blake eblake(a)redhat.com +1-919-301-3266
Libvirt virtualization library http://libvirt.org