Re: [libvirt] [PATCH 2/8] backup: Document nuances between different state capture APIs

On 06/26/2018 11:36 AM, Nir Soffer wrote: Sure. I actually think it will be more common to create checkpoints via virDomainBackupBegin(), and not virDomainCheckpointCreateXML (the latter exists because it is easy, and may have a use independent from incremental backups, but it is the former that makes chains of incremental backups reliable). That is, your first backup will be a full backup (no checkpoint as its start) but will create a checkpoint at the same time; then your second backup is an incremental backup (use the checkpoint created at the first backup as the start) and also creates a checkpoint in anticipation of a third incremental backup. You do have an interesting step in there - the ability to query how much data is pointed to in the delta between two checkpoints (that is, before I actually create a backup, can I pre-guess how much data it will end up copying). On the other hand, the size of the temporary storage for the backup is not related to the amount of data tracked in the bitmap. Expanding on the examples in my 1/8 reply to you: At T3, we have: S1: |AAAA----| <- S2: |---BBB--| B1: |XXXX----| B2: |---XXX--| guest sees: |AAABBB--| where by T4 we will have: S1: |AAAA----| <- S2: |D--BBDD-| B1: |XXXX----| B2: |---XXX--| B3: |X----XX-| guest sees: |DAABBDD-| Back at T3, using B2 as our dirty bitmap, there are two backup models we can pursue to get at the data tracked by that bitmap. The first is push-model backup (blockdev-backup with "sync":"top" to the actual backup file) - qemu directly writes the |---BBB--| sequence into the destination file (based on the contents of B2), whether or not S2 is modified in the meantime; in this mode, qemu is smart enough to not bother copying clusters to the destination that were not in the bitmap. So the fact that B2 mentions 3 dirty clusters indeed proves to be the right size for the destination file. The second is pull-model backup (blockdev-backup with "sync":"none" to a temporary file, coupled with a read-only NBD server on the temporary file that also exposes bitmap B2 via NBD_CMD_BLOCK_STATUS) - here, if qemu can guarantee that the client would read only dirty clusters, then it only has to write to the temporary file if the guest changes a cluster that was tracked in B2 (so at most the temporary file would contain |-----B--| if the NBD client finishes before T4); but more likely, qemu will play conservative and write to the temporary file for ANY changes whether or not they are to areas covered by B2 (in which case the temporary file could contain |A----B0-| for the three writes done by T4). Or put another way, if qemu can guarantee a nice client, then the size of B2 probably overestimates the size of the temporary file; but if qemu plays conservative by assuming the client will read even portions of the file that weren't dirty, then keeping those reads constant will require the temporary file to be as large as the guest is able to dirty data while the backup continues, which may be far larger than the size of B2. [And maybe this argues that we want a way for an NBD export to force EIO read errors for anything outside of the exported dirty bitmap, thus making the client play nice, so that the temporary file does not have to grow beyond the size of the bitmap - but that's a future feature request] I'm trying to convey the point that with example 1... ...if you are interrupted here, your <domain> XML has changed to point to the snapshot file... ...and it is not until here that your <domain> XML is back to its pre-backup state. If the backup is interrupted for any reason, you have to manually get things back to the pre-backup layout, whether or not you were able to salvage the backup data. Can do. In this example 2, using the new APIs, the <domain> XML is unchanged through the entire operation. If you interrupt things in the middle, you may have to scrap the backup data as not being viable, but you don't have to do any manual cleanup to get your domain back to the pre-backup layout. virDomainBackupBegin() has an optional <disks> XML element - if provided, then YOU can control the files (the destination on push model, ultimately including a remote network destination, such as via NBD, gluster, sheepdog, ...; or the scratch file for pull model, which probably only makes sense locally as the file gets thrown away as soon as the 3rd-party NBD client finishes). Libvirt only generates a filename if you don't provide that level of detail. You're right that the local storage running out of space can be a concern - but also remember that incremental backups are designed to be less invasive than full backups, AND that if one backup fails, you can then kick off another backup using the same checkpoint as starting point as the one that failed (that is, when libvirt is using B1 as its basis for a backup, but also created B2 at the same time, then you can use virDomainCheckpointDelete to remove B2 by merging the B1/B2 bitmaps back into B1, with B1 once again tracking changes from the previous successful backup to the current point in time). I have not exposed one so far, nor do I know if qemu has that easily available. But since it matters to you, we can make it a priority to add that (and the API would need to be added to libvirt.so at the same time as the other new APIs, whether or not I can make it in time for the freeze at the end of this week). Again, note that oVirt will probably NOT call virDomainCreateCheckpointXML() directly, but will instead do: virDomainFSFreeze(); virDomainBackupBegin(dom, "<domainbackup type='pull'/>", "<domaincheckpoint><name>B1</name></domaincheckpoint>", 0); virDomainFSThaw(); third party copy data virDomainBackupEnd(); for the first full backup, then for the next incremental backup, do: virDomainFSFreeze(); virDomainBackupBegin(dom, "<domainbackup type='pull'><incremental>B1</incremental></domainbackup>", "domaincheckpoint><name>B2</name></domaincheckpoint>", 0); virDomainFSThaw(); third party copy data virDomainBackupEnd(); where you are creating bitmap B2 at the time of the first incremental backup (the second backup overall), and that backup consists of the data changed since the creation of bitmap B1 at the time of the earlier full backup. Then, as I mentioned earlier, the minimal XML forces libvirt to generate filenames (which may or may not match what you want), so you can certainly pass in more verbose XML: <domainbackup type='pull'> <incremental>B1</incremental> <server transport='unix' socket='/path/to/server'> <disks> <disk name='vda' type='block'> <scratch dev='/path/to/scratch/dev'> </disk> </disks> </domainbackup> and of course, we'll eventually want TLS thrown in the mix (my initial implementation has completely bypassed that, other than the fact that the <server> element is a great place to stick in the information needed for telling qemu's server to only accept clients that know the right TLS magic). If this example helps, I can flush out the html to give these further insights. And, if wrapping FSFreeze/Thaw is that common, we'll probably want to reach the point where we add VIR_DOMAIN_BACKUP_QUIESCE as a flag argument to automatically do it as part of virDomainBackupBegin(). Well, and it's also been a great resource for me as I continue to hammer out the (LOADS) of code needed to reach a working demo. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org