Re: [libvirt] [RFC] externall (pull) backup API

Table of contents. I Preface 1. Fleece API 2. Export API 3. Incremental backups 4. Other hypervisors II Links I Preface This is a RFC for external (or pull) backup API in libvirt. There was a series [1] with more limited API scope and functionality for this kind of backup API. Besides other issues the series was abandoned as qemu blockdev-del command has experimental status at that time. There is also a long pending RFC series for internal (or push) backup API [2] which however has not much in comman with this RFC. Also there is RFC with overall agreement to having a backup API in libvirt [3]. The aim of external backup API is to provide means for 3d party application to read/write domain disks as block devices for the purpuse of backup. Disk is read on backup operation and in case of active domain is presented at some point in time (preferable in some guest consistent state). Disk is written on restore operation. As to providing disk state at some point in time one can use existing disks snapshots for this purpose. However this RFC introduces API to leverage image fleecing (blockdev-backup command) instead. Image fleecing is somewhat inverse to snapshots. In case of snapshots writes go to top image thus backing image stays constant, in case of fleecing writes go to same image as before but old data is previously popped out to fleece image which have original image as backing. As a result fleece image became disk snapshot. Another task of this API is to provide disks for read/write operations. One could try to leverage libvirt stream API for this purpose but AFAIK clients want random access to disks data which is not what stream API suitable for. I'm not sure what is costs of adding block API to libvirt, particularly what it costs to make it effective implementation at RPC level thus this RFC add means to export disks data thru existing block interfaces. For qemu it is NBD. 1. Fleece API So the below API is to provide means to start/stop/query disk image fleecing. I use BlockSnaphost name for this operation. Other options are Fleecing, BlockFleecing, TempBlockSnapshot etc. /* Start fleecing */ virDomainBlockSnapshotPtr virDomainBlockSnapshotCreateXML(virDomainPtr domain, const char *xmlDesc, unsigned int flags); /* Stop fleecing */ int virDomainBlockSnapshotDelete(virDomainBlockSnapshotPtr snapshot, unsigned int flags); /* List active fleecings */ virDomainBlockSnapshotList(virDomainPtr domain, virDomainBlockSnapshotPtr **snaps, unsigned int flags); /* Get fleecing description */ char* virDomainBlockSnapshotGetXMLDesc(virDomainBlockSnapshotPtr snapshot, unsigned int flags); /* Get fleecing by name */ virDomainBlockSnapshotPtr virDomainBlockSnapshotLookupByName(virDomainPtr domain, const char *name); Here is a minimal block snapshot xml description to feed creating function: <domainblocksnapshot> <snapshot disk='sda'> <fleece file="/path/to/fleece-image-sda"/> </snapshot> <snapshot disk='sdb'> <fleece file="/path/to/fleece-image-sdb"/> </snapshot> </domainblocksnapshot> Below is an example of what getting description function should provide upon successful block snaphost creation. The difference with the above xml is that name element (it can be specified on creation as well) and aliases are generated. Aliases will be useful later to identify block devices on exporting thru nbd. <domainblocksnapshot> <name>5768a388-c1c4-414c-ac4e-eab216ba7c0c</name> <snapshot disk='sda'> <fleece file="/path/to/fleece-image-sda"/> <alias name="scsi0-0-0-0-backup"/> </snapshot> <snapshot disk='sdb'> <fleece file="/path/to/fleece-image-sdb"/> <alias name="scsi0-0-0-1-backup"/> </snapshot> </domainblocksnapshot> 2. Export API During backup operation we need to provide read access to fleecing image. This is done thru qemu process nbd server. We just need to specify the disks to export. /* start block export */ int virDomainBlockExportStart(virDomainPtr domain, const char *xmlDesc, unsigned int flags); /* stop block export */ int virDomainBlockExportStop(virDomainPtr domain, const char *diskName, unsigned int flags); Here is an example of xml for starting function: <blockexport type="nbd" port="8001"> <listen type="address" address="10.0.2.10"/> <disk name="scsi0-0-0-1-backup"/> </blockexport> qemu nbd server is started upon first disk export start and shutted down upon last disk export stop. Another option is to control ndb server explicitly. One way to do it is to consider ndb server a new device so to start/stop/update ndb server we can use attach/detach/update device functions. Then in block export start we need to refer to this device somehow. This can be a generated name/uuid or type/address pair. Actually this approach to expose ndb server looks more natural to me even it includes more management from client side. I am not suggesting it in the first place mostly due to hesitations on how to refer to ndb server on block export. In any case I'd like to provide export info in active domain config: <devices> <blockexport type="nbd" port="8001"> <listen type="address" address="10.0.2.10"/> <disk name="scsi0-0-0-1-backup"/> <disk name="scsi0-0-0-2-backup"/> </blockexport> </devices> This API is used in restore operation too. Domain is started in paused state, the disks to be restored are exported and backup client fills it with the backup data. 3. Incremental backups Qemu can track what disk parts are changed from from fleecing start. This is what typically called CBT (dirty bitmap in qemu community I guess). There are also experimental ndb support [4] and a bunch of merged/agreed/proposed bitmap operation that help to organize incremental backups. Different hypervisors has different bitmap implementations with different costs thus it is up to hyperivsor whether to start CBT or not upon block snapshot create by default. Qemu implementations has memory and disk costs for every bitmap thus I suggest by default start fleecing without bitmap and add flag VIR_DOMAIN_BLOCK_SNAPSHOT_CREATE_CHECKPOINT to ask to start a bitmap. Disks bitmaps are visible in active domain definition with the name of block snapshot for which bitmap was started. <disk type='file' device='disk'> .. <target dev='sda' bus='scsi'/> <alias name='scsi0-0-0-0'/> <checkpoint name="93a5c045-6457-2c09-e56c-927cdf34e178"> <checkpoint name="5768a388-c1c4-414c-ac4e-eab216ba7c0c"> .. </disk> The bitmap can be specified upon disk export like below (I guess there is no need to provide more then one bitmap per disk). Active domain config section for block export is expanded similarly. <blockexport type="nbd" port="8001"> <listen type="address" address="10.0.2.10"/> <disk name="scsi0-0-0-1-backup" checkpoint="5768a388-c1c4-414c-ac4e-eab216ba7c0c"/> </blockexport> If bitmap was created on backup start but client failed to make a backup for some reason then it makes no sense to keep this checkpoint anymore. As having bitmap takes resources it is convinient to drop bitmap in this case. Also one may want to drop bitmap for pure resource managment issues. So we need API to remove bitmap: virDomainBlockCheckpointRemove(virDomainPtr domain, const char *name, unsigned int flags); 4. Other hypervisors I took a somewhat considerable look only at vmware backup interface at [5] etc. Looks like they don't have fleecing like qemu has so for vmware snapshots one can use usual disks snapshots API. Also there is no nbd interface for snapshots expectedly thus to deal with vmware snapshot disks one eventually will have to add block API to libvirt. So the only point this RFC has to vmware backups is exporting checkpoints in disk xml. The vmware documentation does not say much about bitmap limitations but I guess they still can provide only a number of them which can be exposed as suggested for active domain disks. II Links: [1] https://www.redhat.com/archives/libvir-list/2016-September/msg00192.html [2] https://www.redhat.com/archives/libvir-list/2017-May/msg00379.html [3] https://www.redhat.com/archives/libvir-list/2016-March/msg00937.html [4] https://github.com/NetworkBlockDevice/nbd/commit/cfa8ebfc354b2adbdf73b6e6... [5] https://code.vmware.com/doc/preview?id=4076#/doc/vddkBkupVadp.9.3.html#10... -- libvir-list mailing list libvir-list(a)redhat.com https://www.redhat.com/mailman/listinfo/libvir-list