Re: [libvirt] Overview of libvirt incremental backup API, part 2 (incremental/differential pull mode)

On 10/9/18 8:29 AM, Nir Soffer wrote: Fleecing. If you want to examine a portion of the disk at a given point in time, then kicking off a pull model backup gives you access to the state of the disk at that time, and your actions are transient. Ending the job when you are done with the fleece cleans up everything needed to perform the fleece operation, and since you did not intend to capture a full (well, a complete) incremental backup, but were rather grabbing just a subset of the disk, you really don't want that point in time to be recorded as a new checkpoint. Also, incremental backups (which are what require checkpoints) are limited to qcow2 disks, but full backups can be performed on any format (including raw disks). If you have a guest that does not use qcow2 disks, you can perform a full backup, but cannot create a checkpoint. But we do have real cases for backup without checkpoint. Nothing prevents you from using a UUID as your name. But this particular choice of XML (<name>) matches what already exists in the snapshot XML. It's about a one-line change to get whichever behavior you find more useful. Right now, I'm leaning towards: <disks> omitted == backup all disks, <disks> present: you MUST have at least one <disk> subelement that explicitly requests a checkpoint (because any omitted <disk> when <disks> is present is skipped). A checkpoint only makes sense as long as there is at least one disk to create a checkpoint with. But I could also go with: <disks> omitted == backup all disks, <disks> present but <disk> subelements missing: the missing elements default to being backed up, and you have to explicitly provide <disk name='foo' checkpoint='no'> to skip a particular disk. Or even: <disks> omitted, or <disks> present but <disk> subelements missing: the missing elements defer to the hypervisor for their default state, and the qemu hypervisor defaults to qcow2 disks being backed up/checkpointed and to non-qcow2 disks being omitted. But this latter one feels like more magic, which is harder to document and liable to go wrong. A stricter version would be <disks> is mandatory, and no <disk> subelement can be missing (or else the API fails because you weren't explicit in your choice). But that's rather strict, especially since existing snapshots XML handling is not that strict. And indeed, since a future enhancement will be figuring out how we can create a checkpoint at the same time as a snapshot (as mentioned elsewhere in the email). A snapshot and checkpoint created at the same atomic point should obviously both be able to happen at a quiescent point in guest I/O. Full backups can happen regardless of underlying format. But incremental backups require checkpoints, and checkpoints require qcow2 persistent bitmaps. As long as you have a qcow2 format on rbd or glusterfs, you should be able to create checkpoints on that image, and therefore perform incremental backups. Storage-wise, during a pull model backup, you would have your qcow2 format on remote glusterfs storage which is where the persistent bitmap is written, and temporarily also have a scratch qcow2 file on the local machine for performing copy-on-write needed to preserve the point in time semantics for as long as the backup operation is running. qemu already promises that a "transaction" succeeds or fails as a group. As to other failures, the full recovery sequence is handled by libvirt, and looks like: Fail on "nbd-server-start": - nothing to roll back Fail on first "blockdev-add": - nbd-server-stop Fail on subsequent "blockdev-add": - blockdev-remove on earlier scratch file additions - nbd-server-stop Fail on any "block-dirty-bitmap-add" or "x-block-dirty-bitmap-merge": - block-dirty-bitmap-remove on any temporary bitmaps that were created - blockdev-remove on all scratch file additions - nbd-server-stop Fail on "transaction": - block-dirty-bitmap-remove on all temporary bitmaps - blockdev-remove on all additions - nbd-server-stop Fail on "nbd-server-add" or "x-nbd-server-add-bitmap": - if a checkpoint was attempted during "transaction": -- perform x-block-dirty-bitmap-enable to re-enable bitmap that was in use prior to transaction -- perform x-block-dirty-bitmap-merge to merge new bitmap into re-enabled bitmap -- perform block-dirty-bitmap-remove on the new bitmap - block-job-cancel - block-dirty-bitmap-remove on all temporary bitmaps - blockdev-remove on all scratch file additions - nbd-server-stop I still need to finish writing that, but part 3 will be a demonstration of the push model (where qemu writes the backup to a given destination, without a scratch file, and without an NBD server, but where you are limited to what qemu knows how to write). -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org