Hi all! Here I want to summarize new interfaces and use cases for backup in QEMU. TODO for me: convert this into good rst documentation in docs/. OK, let's begin. First, note that drive-backup qmp command is deprecated. Next, some terminology: push backup: the whole process is inside QEMU process, also may be called "internal backup" pull backup: QEMU only exports a kind of snapshot (for example by NBD), and third party software reads this export and stores it somehow, also called "external backup" copy-before-write operations: We usually do backup of active disk, guest is running and may write to the disk during the process of backup. When guest wants to rewrite data region which is not backed up yet, we must stop this guest write, and copy original data to somewhere before continuing guest write. That's a copy-before-write operation. image-fleecing: the technique that allows to export a "snapshotted" state of the active disk with help of copy-before-write operations. We create a temporary image - target for copy-before-write operations, and provide an interface to the user to read the "snapshotted" state. And for read, we do read from temporary image the data which is already changed in original active disk, and we read unchanged data directly from active disk. The temporary image itself is also called "reverse delta" or "reversed delta". == Simple push backup == Just use blockdev-backup, nothing new here. I just note some technical details, that are relatively new: 1. First, backup job inserts copy-before-write filter above source disk, to do copy-before-write operation. 2. Created copy-before-write filter shares internal block-copy state with backup job, so they work in collaboration, to not copy same things twice. == Full pull backup == Assume, we are going to do incremental backup in future, so we also need to create a dirty bitmap, to track dirtiness of active disk since full backup. 1. Create empty temporary image for fleecing. It must be of the same size as active disk. It's not necessary to be qcow2, and if it's a qcow2, you shouldn't make the original active disk a backing file for the new temporary qcow2 image (it was necessary in old fleecing scheme). Example: qemu-img create -f qcow2 temp.qcow2 64G 2. Initialize fleecing scheme and create dirty bitmap for future incremental backup. Assume, disk0 is an active disk, attached to qdev-id sda, to be backed up. qmp: transaction [ block-dirty-bitmap-add {node: disk0, name: bitmap0, persistent: true} blockdev-add* {node-name: tmp-protocol, driver: file, filename: temp.qcow2} blockdev-add {node-name: tmp, driver: qcow2, file: tmp-protocol} blockdev-add {node-name: cbw, driver: copy-before-write, file: disk0, target: tmp} blockdev-replace** {parent-type: qdev, qdev-id: sda, new-child: cbw} blockdev-add {node-name: acc, driver: snapshot-access, file: cbw} ] qmp: nbd-server-start {...} qmp: nbd-server-add {device: acc, ...} This way we create the following block-graph: [guest] [NBD export] | | | root | root v file v [copy-before-write]<------[snapshot-access] | | | file | target v v [active-disk] [temp.qcow2] * "[PATCH 0/2] blockdev-add transaction" series needed for this ** "[PATCH v3 00/11] blockdev-replace" series needed for this Note additional useful options for copy-before-write filter: "[PATCH 0/3] block: copy-before-write: on-cbw-error behavior" provides possibility of option on-cbw-error=break-snapshot, which means that on failure of CBW operation we will not break guest write, but instead all further reads by NBD client will fail, which formally means: break the backup process, not guest write. "[PATCH 0/4] block: copy-before-write: cbw-timeout" provides an option cbw-timeout, to set a timeout for CBW operations. That's very useful to avoid guest stuck. 3. Now third party backup tool can read data from NBD export NBD_CMD_TRIM (discard) operation is supported on the export, it has the following effects: 1. discard this data from temp image, if it is stored here 2. avoid further copy-before-write operation (guest is free to rewrite corresponding data with no extra latency) 3. all further read requests from discarded areas by NBD client will fail So, NBD client may discard regions that are already backed up to avoid extra latency for guest writes and to free disk space on the host. Possible TODO here is to implement NBD protocol extension, that allows to READ & DISCARD in command. In this case we avoid extra command in the wire, but lose possibility of retrying the READ operation if it failed. 4. After backup is complete, we should destroy the fleecing scheme: qmp: nbd-server-stop qmp: blockdev-del {node-name: acc} qmp: blockdev-replace {parent-type: qdev, qdev-id: sda, new-child: disk0} qmp: blockdev-del {node-name: cbw} qmp: blockdev-del {node-name: tmp} qmp: blockdev-del {node-name: tmp-protocol} 5. If backup failed, we should remove created dirty bitmap: qmp: block-dirty-bitmap-remove {node: disk0, name: bitmap0} == Incremental pull backup == OK, now we have a bitmap called bitmap0, and want to do incremental backup, accordingly to that bitmap. In short, we want: - create a new bitmap to continue dirty tracking for next incremental backup - export "snapshotted" state of disk0 through NBD - export "frozen" bitmap, so that external tool know what to copy Mostly, all points remains the same, let's go through: 1. Create empty temporary image for fleecing -- same as for full backup, no difference 2. Initialize fleecing scheme and create dirty bitmap for future incremental backup. qmp: transaction [ block-dirty-bitmap-add {node: disk0, name: bitmap1, persistent: true} block-dirty-bitmap-disable {node: disk0, name: bitmap0} blockdev-add {node-name: tmp-protocol, driver: file, filename: temp.qcow2} blockdev-add {node-name: tmp, driver: qcow2, file: tmp-protocol} blockdev-add {node-name: cbw, driver: copy-before-write, file: disk0, target: tmp, bitmap: {node: disk0, name: bitmap0}} blockdev-replace {parent-type: qdev, qdev-id: sda, new-child: cbw} blockdev-add {node-name: acc, driver: snapshot-access, file: cbw} ] qmp: nbd-server-start {...} qmp: block-export-add {type: nbd, node-name: acc, bitmaps: [{node: disk0, name: bitmap0}]} 3. Now third party backup tool can read data from NBD export - Client may negotiate meta contexts, to query exported dirty bitmap by NBD_BLOCK_STATUS commend - If client reads "not-dirty" (by bitmap0) areas, it gets an error. - NBD_CMD_TRIM (discard) works as for full backup, no difference 4. After backup is complete, we should destroy the fleecing scheme: - Same as for full backup 5. Next, we should handle dirty bitmaps: 5.1 Failure path Merge-back bitmap1 to bitmap0 and continue tracking in bitmap0: qmp: transaction [ block-dirty-bitmap-enable {node: disk0, name: bitmap0} block-dirty-bitmap-merge {node: disk0, target: bitmap0, bitmaps: ['bitmap1']} block-dirty-bitmap-remove {node: disk0, name: bitmap1} ] 5.2 Success path We have two possible user scenarios on success: 5.2.1 Continue tracking for next incremental backup in bitmap1 In this case, just remove bitmap0: qmp: block-dirty-bitmap-remove {node: disk0, name: bitmap0} Or you may not delete bitmap0 and keep it disabled, to be reused in future for differential backup (see below). 5.2.2 Continue tracking for next incremental backup in bitmap0 (assume, we always work with one bitmap and don't want any kind of differential backups, and don't associate bitmap name with stored backups) In this case, enable and clear bitmap0, merge bitmap1 to bitmap0 and remove bitmap1: qmp: transaction [ block-dirty-bitmap-enable {node: disk0, name: bitmap0} block-dirty-bitmap-clear {node: disk0, name: bitmap0} block-dirty-bitmap-merge {node: disk0, target: bitmap0, bitmaps: ['bitmap1']} block-dirty-bitmap-remove {node: disk0, name: bitmap1} ] == Push backup with fleecing full/incremental == Reasoning: the main problem of simple push backup is that guest writes may be seriously affected by copy-on-write operations, when backup target is slow. To solve this problem, we'll use the scheme like for pull backup: we create local temporary image, which is a target for copy-before-write operations, and instead of exporting the "snapshot-access" node we start internal backup from it to the target. So, the scheme and commands looks exactly the same as for full and incremental pull backup. The only difference is that we don't need to start nbd export, but instead we should add target node to qemu and start internal backup. And good thing is that it may be done in same transaction with initializing fleecing scheme: qmp: transaction [ ... initialize fleecing scheme for full or incremental backup ... # Add target node. Here is qcow2 added, but it may be nbd node or something else blockdev-add {node-name: target-protocol, driver: file, filename: target.qcow2} blockdev-add {node-name: target, driver: qcow2, file: target-protocol} # Start backup blockdev-backup {device: acc, target: target, ...} ] If it is an incremental backup, pass also bitmap parameter: blockdev-backup {..., bitmap: bitmap0, sync: incremental, bitmap-mode: never} Note bitmap-mode=never: this means that backup will do nothing with bitmap0, so we have same scheme like for pull backups (handle bitmaps by hand after backup). Still, push-backup scheme may be adopted to use other bitmap modes. What we lack here is discarding in 'acc' node after successful copying of the block to the target, to safe disk space and avoid extra copy-before-write operations. It's a TODO, should be implemented like discard-source parameter for blockdev-backup. == Differential backups == I'm not fan of this idea, but I think it should be described. Assume we have already a chain of incremental backups (represented as qcow2 chain on backup storage server, for example). They corresponds to some points in time: T0, T1, T2, T3. Assume T3 is the last backup. If we want to create usual incremental backup, it would be diff between T3 and current time (which becomes T4). Differential backup say: I want to make backup starting from T1 to current time. What's for? Maybe T2 and T3 was removed or somehow damaged.. How to do that in Qemu: on each incremental backup you start a new bitmap, and _keep_ old one as disabled. This way we have bitmap0 (which presents diff between T0 and T1), bitmap1 (diff T1 T2), bitmap2 (diff T2 T3), and bitmap3 which shows diff from T3 up to current time. bitmap3 is the only enabled bitmap and others are disabled. So, to make differential backup, use block-dirty-bitmap-merge command, to merge all bitmaps you need into one, and than use it in any backup scheme. The drawback is that all these disabled bitmaps eat RAM. Possible solution is to not keep them in RAM, it's OK to keep them in qcow2, and load only on demand. That's not realized now and that's a TODO for thous who want differential backups. -- Best regards, Vladimir