Re: [libvirt] [RFC v2] external (pull) backup API

On 04/11/2018 12:32 PM, Eric Blake wrote: Oh, I see -- you're using blockdev-backup sync=none to accomplish fleecing snapshots. It's a little confusing here without the sync=none information, as usually blockdev-backup provides... backups, not snapshots. Your cover letter would be a little clearer with that information. For everyone else: Node fleecing is a very dumb name that means "Live copying of arbitrary blocks from a live node." In QEMU, it works like this: (1) Create a new active layer for the node to be fleeced. [node] <-- [snapshot] Note that this snapshot node is backed by the node named "node", not a file named "node". The snapshot is supported by a qcow2 file on local storage that starts empty. (2) Start blockdev-backup sync=none FROM the node TO the snapshot: This means that any time 'node' is written to, the information will get written to 'snapshot' on-demand, preserving snapshot as a point-in-time snapshot of node, while leaving 'node' in-use for any other nodes/devices using it. It's effectively the opposite of an external snapshot, facilitated by a live COW job. (3) an NBD server may be started to export the "snapshot" nbd-server-add device=snapshot (4+) At this point, the NBD client can copy the snapshot data out, and the NBD export can be closed upon completion. Then, the snapshot can be removed/deleted. Unlike traditional external snapshot and commit workflow, this snapshot can be deleted at any time without jeopardizing the data it is a snapshot of. It makes sense to avoid the bitmap name in libvirt, but do these indeed correlate 1:1 with bitmaps? I assume each bitmap will have name=%%UUID%% ? In fleecing workflow, the image can either be created by QEMU or pre-created by libvirt, but in keeping with best practices libvirt should probably create it. It should be an empty qcow2 backed by the current node of interest. No; based on how fleecing is implemented in QEMU. Effectively, "blockdev-backup sync=none" _IS_ the fleecing command for QEMU, but it requires two steps: (1) Create a new empty top layer (2) Sync new writes from the active layer, which is the backing image(!) for the snapshot. The empty top layer can be created whenever, but the backup action needs to happen in a transaction. In my earlier email I mentioned it'd be nice to have a convenience job that wrapped up a few steps into one: A: Create a new top layer B: Start the fleecing job itself C: Tied a specified bitmap belonging to the node to the fleecing job D: Exported the snapshot via NBD and this would become a proper "fleecing job." It would have to be, of course, transactionable. Because the temporary snapshot is backed by the live image, the guest sees an unchanging set of blocks. In essence, we are utilizing a COW mechanism to copy the point-in-time data into the snapshot layer while the backing image remains the live/active layer for the devices utilizing it. So far so good from the QEMU end AFAICT... So you are trying to optimize away write penalties if you have, say, ten bitmaps representing checkpoints so we don't have to record all new writes to all ten. This makes sense, and I would have liked to formalize the concept in QEMU, but response to that idea was very poor at the time. Also my design was bad :) Previous, or next? Say we've got bitmaps (in chronological order from oldest to newest) A B C D E F G H and we want to delete bitmap (or "checkpoint") 'C': A B D E F G H the bitmap representing checkpoint 'D' should now contain the bits that used to be in 'C', right? That way all the checkpoints still represent their appropriate points in time. The only problem comes when you delete a checkpoint on the end and the bits have nowhere to go: A B C A B _ In this case you really do lose a checkpoint -- but depending on how we annotate this, it may or may not be possible to delete the most recent checkpoint. Let's assume that the currently active bitmap that doesn't represent *any* point in time yet (because it's still active and recording new writes) is noted as 'X': A B C X If we delete C now, then, that bitmap can get re-merged into the *active bitmap* X: A B _ X Right. A missing bitmap anywhere in the sequence invalidates the entire sequence. That seems... unfortunate. A record could be kept in libvirt instead, couldn't it? A : Bitmap A, Time 12:34:56, Child of (None), Parent of B B : Bitmap B, Time 23:15:46, Child of A, Parent of (None) I suppose in this case you can't *reconstruct* this information from the bitmap stored in the qcow2, which necessitates your naming scheme... ...Still, if you forego this requirement, deleting bitmaps in the middle becomes fairly easy. If this is a hard requirement, it's certainly *easier* to track the relationship in QEMU ... This is a way, really, of storing extra metadata by using the bitmap name as arbitrary data storage. I'd say either we promote QEMU to understanding checkpoints, or enhance libvirt to track what it needs independent of QEMU -- but having to rename bitmaps smells fishy to me. I think the *snapshots*, as temporary objects, are independent and don't carry a relation to each other. The *checkpoints* here, however, are persistent and interrelated. I see; so we compute a new slice based on previous bitmaps and backup arbitrary from that arbitrary slice. So "T" is a temporary bitmap meant to be discarded at the conclusion of the operation, making it much more like a consumable object. Aha. So a checkpoint creates a reference point, but NOT a backup. You are manually creating checkpoint instances. In this case, though, you haven't disabled the previous checkpoint's bitmap (if any?) atomically with the creation of this one... Depending on the number of checkpoints intended to be kept... we certainly make no real promises on the efficiency of marking so many. It's at *least* a linear increase with each checkpoint... Oh, I see, you handle the "disable old" case here. OK, so in this transaction you add a new temporary bitmap for export, and merge the contents of two bitmaps into it. However, it doesn't look like you created a new checkpoint and managed that handoff here, did you? And then here, once the bitmap and the data is already frozen, it's actually alright if we add the export at a later point in time. Don't know much about this, I stopped paying attention to the BLOCK STATUS patches. Is the NBD spec the best way to find out the current state right now? (Is there a less technical, briefer overview somewhere, perhaps from a commit message or a cover letter?) OK, this just deletes the checkpoint. I guess we delete the node and stop the NBD server too, right? Yeah, A-B-C-D terminology would be nice for the examples. It's fine if the actual implementation uses UUIDs. We already have this, right? It doesn't even need to be transactionable. You need this... And this we had originally but since removed, but can be re-added trivially. Do my comments make sense? Am I understanding you right so far? I'll try to offer a competing writeup to make sure we're on the same page with your proposed design before I waste any time trying to critique it -- in case I'm misunderstanding you. Thank you for leading the charge and proposing new APIs for this feature. It will be very nice to expose the incremental backup functionality we've been working on in QEMU to users of libvirt. --js I'm working on this right now, actually! I'm working on JSON format output for bitmap querying, and simple clear/delete commands. I hope to send this out very soon.