9:21 a.m.
[CC Nikolay]
21.08.2019 1:25, John Snow wrote:
Hi, downstream here at Red Hat I've been fielding some questions
about
the usability and feature readiness of Bitmaps (and related features) in
QEMU.
Here are some questions I answered internally that I am copying to the
list for two reasons:
(1) To make sure my answers are actually correct, and
(2) To share this pseudo-reference with the community at large.
This is long, and mostly for reference. There's a summary at the bottom
with some todo items and observations about the usability of the feature
as it exists in QEMU.
Before too long, I intend to send a more summarized "roadmap" mail which
details all of the current and remaining work to be done in and around
the bitmaps feature in QEMU.
Questions:
> "What format(s) is/are required for this functionality?"
From the QEMU API, any format can be used to create and author
incremental backups. The only known format limitations are:
1. Persistent bitmaps cannot be created on any format except qcow2,
although there are hooks to add support to other formats at a later date
if desired.
DANGER CAVEAT #1: Adding bitmaps to QEMU by default creates transient
bitmaps instead of persistent ones.
Possible TODO: Allow users to 'upgrade' transient bitmaps to persistent
ones in case they made a mistake.
I doubt, as in my opinion real users of Qemu are not people but libvirt, which
should never make such mistake.
2. When using push backups (blockdev-backup, drive-backup), you may use
any format as a target format. >
DANGER CAVEAT #2: without backing file and/or filesystem-less sparse
support, these images will be unusable.
You mean incremental backups of course, as the whole document is about bitmaps.
EXAMPLE: Backing up to a raw file loses allocation information, so we
can no longer distinguish between zeroes and unallocated regions. The
cluster size is also lost. This file will not be usable without
additional metadata recorded elsewhere.*
(* This is complicated, but it is in theory possible to do a push backup
to e.g. an NBD target with custom server code that saves allocation
information to a metadata file, which would allow you to reconstruct
backups. For instance, recording in a .json file which extents were
written out would allow you to -- with a custom binary -- write this
information on top of a base file to reconstruct a backup.)
3. Any format can be used for either shared storage or live storage
migrations. There are TWO distinct mechanisms for migrating bitmaps:
A) The bitmap is flushed to storage and re-opened on the destination.
This is only supported for qcow2 and shared-storage migrations.
cons: flushing/reopening is done during migration downtime, so if you have
a lot of bitmap data (for example, 64k granulared bitmap for 16tb disk is
~30MB, and there may be several bitmaps) downtime will become long.
B) The bitmap is live-migrated to the destination. This is supported for
any format and can be used for either shared storage or live storage
migrations.
DANGER CAVEAT #3: The second bitmap migration technique there is an
optional migration capability that must be enabled explicitly.
Otherwise, some migration combinations may drop bitmaps.
Also, bad thing may happen if we try to migrate persistent bitmap to not qcow2.
Also, with enabled capability, flushing/reopening is automatically disabled.
Matrix:
> migrate = migrate_capability or (persistent and shared_storage)
Enumerated:
live storage + raw : transient + no-capability: Dropped
live-storage + raw : transient + bm-capability: Migrated
live-storage + qcow2 : transient + no-capability: Dropped
live-storage + qcow2 : transient + bm-capability: Migrated
live-storage + qcow2 : persistent + no-capability: Dropped (!)
live-storage + qcow2 : persistent + bm-capability: Migrated
shared-storage + raw : transient - no-capability: Dropped
shared-storage + raw : transient + bm-capability: Migrated
shared-storage + qcow2 : transient + no-capability: Migrated
Dropped you mean
shared-storage + qcow2 : transient + bm-capability: Migrated
shared-storage + qcow2 : persistent + no-capability: Migrated
shared-storage + qcow2 : persistent + bm-capability: Migrated
Enabling the bitmap migration capability will ALWAYS migrate the bitmap.
If it's disabled, we will only migrate the bitmaps for shared storage
migrations where the bitmap is persistent, which is a qcow2-only case.
There is no warning or error if you attempt to migrate in a manner that
loses your bitmaps.
(I might be persuaded to add a case for when you are doing a live
storage migration of qcow2 with persistent bitmaps, which is somewhat a
conflicting case: you've asked for the bitmap to be persistent, but it
seems likely that if this ever happens in practice, it's because you
have neglected to ask for it to be migrated to the new host.)
See iotest 169 for more details on this.
At present, these are the only format limitations I am consciously aware
of. From a management API/GUI perspective, it makes sense to restrict
the feature set to "qcow2 only" to minimize edge cases.
> "Is libvirt aware of these 'gotcha' cases?"
From talks I've had with Eric Blake and Peter Krempa, they certainly are
now.
> "Is it possible to make persistent the default?"
Not quickly.
In QEMU, not without a deprecation period or some other incompatibility.
Default values are not (yet?) introspectable via the schema. We need
(possibly) up to two QAPI extensions:
I) The ability to return deprecation warnings when issuing a command
that will cease to work in the future.
This has been being discussed somewhat on-list recently. It seems like
there is not a big appetite for tackling something perceived as
low-value because it is likely to be ignored.
II) The ability to document default values in the QAPI schema for the
purposes of introspection.
With one or both of these extensions, we could remove the default value
for persistence and promote it to a required argument with a
transitionary period where it will work with a warning. Then, in the
future, users will be forced to specify if they want persistent=true or
persistent=false.
This is not on my personal roadmap to implement.
> "Is it possible to make bitmap migration the default?"
I don't know at present. Migration capabilities are either "on" or
"off"
and the existing negotiation mechanisms for capabilities are extremely
rudimentary.
Changing this might require fiddling with machine compat properties,
adding features to the migration protocol, or more. I don't know what I
don't know, so I will estimate this change as likely invasive.
I've discussed this with David Gilbert and it seems like a complicated
project for the benefit of this sub-project alone, so this isn't on my
personal roadmap to resolve.
The general consensus appears to be that protecting the user is
libvirt's job.
> "Where do we stand with external snapshot support?"
Still broken. In the aftermath of 4.1, it's the most obvious outstanding
broken feature. Vladimir has patches to fix it, but they need some
attention.
You mean "[PATCH v4 00/10] qcow2-bitmaps: rewrite reopening logic"
yes, reviews are welcome.
> "What needs to happen to bitmaps when doing stream or commit?"
Uncertain in QEMU; creating an external snapshot implicitly ends the
timeslice represented by the old bitmap, but an explicit checkpoint is
better.
With recent patches to QAPI we moved to the following way:
Qemu don't care and don't do any magic around bitmaps automatically, so
bitmap is staying at the same node and do its work. So, ofcourse, if we commit top
to base, enabled bitmaps in base will track all writes of this commit job. And same
thing with stream job. So if user wants, it should move, copy, enable/disable bitmaps
as desired by hand.
I think some little ascii charts will help people understand what we're
talking about here, so let's cover some examples.
SCENARIO 1)
Here's a timeline for a single node (one image, no backing files), with
some points in time highlighted:
Time T = 0.........................n
+rec: [--X------Y------Z--------]
-rec: [---------x------y--------]
region: [aabbbbbbbcccccccddddddddd]
X, Y, and Z are points in time where bitmaps 'x', 'y', and 'z'
were
created and began recording. x and y are points in time where bitmaps
'x' and 'y' stopped recording.
This creates a few distinct regions / timeslices.
a: Data written before we began tracking writes.
b: Data written to bitmap 'x'
c: Data written to bitmap 'y'
d: data written to bitmap 'z'
region 'a' is of an unknown size and indeterminate length, because there
is no reference point (checkpoint) prior to it.
regions 'b' and 'c' are of finite size and determinate length, because
they have fixed reference points on either sides of their timeslice.
region 'd' is also of an unknown size and indeterminate length, because
it is actively recording and has no checkpoint to its right. It may be
fixed at any time by disabling bitmap 'z'.
In QEMU, generally what we want to do is to do several things at one
atomic moment to keep these regions adjacent, contiguous, and disjoint.
So from a high-level (using a fictional simplified syntax), we do:
Transaction(
create('y'),
disable('x'),
backup('x')
)
which together performs a backup+checkpoint.
We can do a backup without a checkpoint:
4.1:
Transaction(
create('tmp')
merge('tmp', 'x')
backup('tmp')
)
4.2:
> backup('x', bitmap_sync=never)
Or a checkpoint without a backup:
Transaction(
create('y'),
disable('x')
)
SCENARIO 2)
Now, what happens when we make an external snapshot and do nothing at
all to our bitmaps?
Time T = 0.......................................n
+rec: [--X------Y------Z--------] <-- [-------]
-rec: [---------x------y--------] <-- [-------]
region: [aabbbbbbbcccccccddddddddd] <-- [eeeeeee]
{ base } <-- { top }
We've created a new implicit timeslice, "e" without creating a new
bitmap. Because the bitmap 'z' was still active at the time of the
snapshot, it now has a temporarily-determinate endpoint to its region.
This is kind of like an "implied checkpoint", but it's a very poor one
because it's not really addressable.
DANGER CAVEAT #4: We have no way to create incremental backups anymore,
because the current moment in time has no addressable point.
That's not great; but it is likely a fixable scenario when commit is
both commit and snapshot are to be fixed, or more precisely reopening bitmaps
ro->rw and rw->ro are all broken.
fixed: committing the top layer back down into the base layer will
add
all new writes to the end of the old region; restoring our backup chain:
Time T = 0.........................C.......n
+rec: [--X------Y------Z-------- -------]
-rec: [---------x------y-------- -------]
region: [aabbbbbbbcccccccddddddddd ddddddd]
Here, region 'e' just gets appended to region d, and we can make
incremental backups from any checkpoint X, Y, Z to the current moment again.
SCENARIO 3)
What happens if we make a firm checkpoint at the same time we make the
snapshot?
Transaction(
disable('z'),
snapshot('top'),
create('w')
)
Time T = 0......................... ......n
+rec: [--X------Y------Z-------- ] <-- [W------]
-rec: [---------x------y--------z] <-- [-------]
region: [aabbbbbbbcccccccddddddddd ] <-- [eeeeeee]
{ base } <-- { top }
Now instead of the new region 'e' being implied, it's explicit. We can
make backups between any point and the current moment *across* the gap.
It was my thought that this was the most preferable method that libvirt
should use, but there is some doubt from Peter Krempa. We'll see how it
shakes out.
There are questions about what QEMU should do by default, without
libvirt's help. At the moment, it's "nothing" but there have been
questions about "something".
Keeping in mind that we likely can't change our existing behavior
without some kind of flag, there are still some API/usability questions:
> If we create an external snapshot on top of an image with actively
> recording bitmaps, should we disable them?
We can leave them enabled, but they'll never see any writes. Or we can
explicitly disable them. Explicitly disabling them may make more sense
to prevent modifying bitmaps accidentally on commit.
My guess: No. we should leave them alone; allow checkpoint creation
mechanisms to do the disable+create dance for bitmaps as needed.
Agreed: No. We've exposed APIs for the user to enable/disable/move bitmaps in
any way, so default to do nothing is the safest way.
And making bitmap disabled automatically is a great danger of losing dirty
bits and finally produce inconsistent backup.
Potential problems: The backing image is read-only, and if we change our
mind later, we will need to find a way to re-open the backing image as
read-write for the purposes of toggling the recording bit prior to any
legitimate guest usage of that node. Then, re-open as RO again.
> Should we fork bitmaps (on snapshot)?
If a bitmap named 'z' is recording when we create an external snapshot,
should that bitmap be *copied* into the top layer?
My guess: No.
Agreed. Nothing automatically. If user wants let him copy by hands. Consider
case when user uses UUIDs to name bitmaps and is not prepared to appearing
of two bitmaps with the same name in the system.
This would allow us to create external snapshots *without* creating a
checkpoint, but conceptually that's a nightmare: It would allow for
mutually independent creation of snapshots OR checkpoints. This would be
hard to corral when undoing a snapshot, for instance.
In my opinion, snapshots MUST be checkpoints, and therefore allowing a
snapshot without creating a checkpoint is a no-go.
> (Should we fork bitmaps) if we're not using checkpoints?
If we are using a checkpoint-less paradigm (i.e. the rolling incremental
backup using only one bitmap) we might want to copy the bitmap up to
make the next incremental backup as if nothing ever happened.
However, rolling incremental backups doesn't need any kind of auto-copy
feature. This is possible today:
> create('base', 'A')
> transact(snapshot('top'), create('top', 'B'))
> merge('B', [('base', 'A'), ('top', 'B')])
Exactly. As I noted, we are already on the way of exposing APIs instead of
automatic logic.
i.e., we create a new bitmap on the top layer, then merge in the old
data from the backing file, which remains addressable.
Whether the user wants to copy up or not, there are commands that will
do that already.
> Should we create new bitmaps by default when we can?
If a backing image has bitmaps, should QEMU automatically create a new
bitmap for the top layer? Should it be named something new, something
user-provided, or based on existing active bitmaps?
If a user creates a new external snapshot with no consideration paid to
bitmaps (like "SCENARIO 2" above), they temporarily lose the ability to
do incremental backups. They might be able to commit the image back to
"try again."
That's not great. Here are some options for resolving this:
But I don't see any problem with it. If user forget to create bitmap, it's
not our problem and not the reason to create bitmaps automatically.
Libvirt will not forget. Managing layer will not foreget, or it has bug which
should be fixed. Libvirt will not be able to use any bitmaps with automatic
names, as it don't know "what is it" anyway.
- Automatic names: Might cause collisions out-of-band with management
tooling by accident, tooling has to query to discover what bitmaps were
automatically created.
And "when" they are created.. And create separate naming namespace by
restricting
naming or additional bitmap field..
- Same names: Can create namespace confusion when committing snapshots
later; although each layer of a backing chain can have bitmaps named the
same thing, it causes future problems when committing together that can
be hard to resolve.
And user may be unprepared to appearing of bitmaps with the same name, if bitmap
names directly bound to checkpoints.
- User-provided name: This is workable, and requires an amendment to the
snapshot command to automatically create a new bitmap on the snapshot.
We have transactions to combine commands, so I don't think that combining commands
into one command is a good idea.
My guess: No, we can't automatically create a new bitmap for the user.
We can amend the snapshot commands to accept bitmap names, but at that
point we've just duplicated transactions:
Transact(
snapshot('top'),
create('top', 'new-bitmap')
)
All that said (Mostly a lot "No, let's not do anything"), maybe
there's
room for an "assistive" mode for users, a bitmap-aware snapshot creation
command. It could do the following well-defined magic:
bitmap-snapshot(base, top, bitmap_name):
1. disable any active bitmaps in the base node.
2. create a bitmap named "bitmap_name" in the top node, failing if
a bitmap by that name already exists on either node.
We don't have bitmap-creation option even for backup command which is the
main user of bitmaps.. I don't think we need to implement new commands only
to move in libvirt code from transaction to one command (and libvirt will have
firstly to check existence of newer command and if it exist use it instead of
transaction? It's not a simplification)
What this accomplishes:
- Disables any bitmaps in the base layer ahead of time, in preparation
for an eventual commit operation.
I really doubt that we should disable bitmaps in the base layer on snapshot..
Why? On commit they must be enabled. On separate opening of this image later
they must be enabled. The only reason to store disabled bitmap A in the image
is when we have the bitmap B which starts its history from the point when A
was disabled and B is in the _same_ image.
And automatic disabling of bitmaps is always dangerous. We'll never predict all
possible effects.
- Always creates a new, enabled bitmap on the snapshot mode which is
guaranteed not to conflict with a name on the base node. This bitmap can
be used to create additional copies post-hoc, if desired.
- Formalizes our "best practice" suggestion for mixing bitmaps and
snapshots into a single, documented command.
Is this strictly needed? No, if you have the foresight, you can do this
instead:
Transact(
disable('a'),
disable('b'),
disable('c'),
# plus however many more ...
Why do you think we should disable bitmaps in the base? Am I missing something?
snapshot('top', ...),
create('top', 'd')
)
but a convenience command might still have a role to play in helping
take the guesswork out for non-libvirt users.
Hmm, my problem is that I don't know, do we really have such users? Of course
some vendors have to workaround libvirt sometimes and call QMP directly (for
example, Virtuozzo), but it's not a problem to use transaction for us.. But
do we have real users which are not programmers and not testers but real users
of Qemu and QMP? OK, my view is a bit one-sided, as I mostly think about
Virtuozzo usecases and we don't have such users.
That's the bulk of what was discussed.
Summary:
GOTCHAs:
#1: Bitmaps are created non-persistent by default, and can't be changed.
IMHO, not a problem
#2: Push backup destination formats will happily back up to a format
that isn't semantically useful.
That is interesting one.. We have some checks and warning around cluster
sizes of target and source, but all it is not very reliable..
#3: Migrating non-shared block storage can drop even persistent bitmaps
if you don't pass the bitmap migration capability flag to both QEMU
instances.
Losing a bitmap was never considered as a real problem as it just means
"do full backup".. Also, actually they may be dropped even if we enabled
the capability: bitmaps are migrated through "postcopy" mechanism, so any
error leads to unrecoverable loss of bitmaps.
#4: Creating a snapshot without doing some bitmap manipulation
beforehand can temporarily render your bitmaps unusable. Failing to
disable bitmaps before creating a snapshot might make commits very
tricky later on.
Aha, understand now why you want to disable bitmaps in base: you don't want
them to catch commit writes, for which we most probably have bitmap in top
which we are going to move to the base...
Anyway, I think it is more correct to disable bitmaps in one transaction with
commit command. Ok, we don't have transaction for commit, may bit we want it.
As a workaround user may disable bitmaps after commit start. They will become
a bit more dirty than they should but it's not a problem.
Gotchas 1 and 4 can be at least partially alleviated. gotcha 2 remains a
pain point we cannot intercept at the QEMU layer. gotcha 3 has potential
remedies, but they are complicated.
QEMU todo items:
- Fix bitmap data corruption on commit (Ongoing, by Vladimir@Virtuozzo)
;)
- add a set_persistence method for bitmaps that allows us to change the
storage class of a bitmap after creation. (Helps alleviate gotcha #1.)
Still, I just don't know users who need it.
- Add a command that allows us to merge allocation data into a bitmap.
This helps alleviate gotcha #4: If we create a new image but neglected
to do the proper transaction dance, we can simply copy the allocation
data into a new bitmap. (Note, we'd still need set_persistence to help
us disable the old bitmap before any commit happens.)
Ooops. You are going to use set_persistence to disable persitence to workaround
the fact that you can't disable bitmap without marking it IN_USE in the image?
This is wrong. We must not modify bitmap which is not marked IN_USE in the image,
as it may lead to any kind of data corruption (in case of crash, we'll finish up
with bitmap not marked IN_USE but invalid, and we'll consider it valid on next
vm start).. So, it's a reason to never implement set_persistence (or at least
set_persistence must set IN_USE flag in the image, and will not help in considered
case).
And also, I don't know users who may forget to create bitmap if they need.. It's
not about libvirt of course.
- Add convenience command for easy + safe combination of bitmaps +
snapshots. Helps prevent #4.
Research items:
- How hard is it to reopen a backing image as RW while it's in-use,
disable a bitmap, and then reopen as RO? This is to partially address
gotcha #4; if we forget to disable bitmaps before creating the snapshot.
It's unsafe to have an image with disabled topmost bitmap. So I think the right way
is
to disable it in one transaction with commit, or after commit start. And may be even
create new enabled top bitmap in base to really catch writes, so in case of failure we
can finish up with valid bitmap chain... But I'm afraid I don't have a complete
view
of how it should work.
- How hard is the reverse operation? Can we reopen a backing image RW,
enable a bitmap, and then reopen as RO? This gives us better control
over what happens on commit.
- After we fix the commit bug, what does/should commit actually do with
bitmaps? What about bitmaps that collide? The current behavior is that
any bitmaps don't transfer from top to base. Any bitmaps active in the
base record all the new writes from the top.
I think current behavior (after bug fix) is correct thing for Qemu and all
customizations should be done with help of bitmaps API and transactions.
That's all!
--js
--
Best regards,
Vladimir