Re: [libvirt] RFC backup API

On Tue, Mar 22, 2016 at 02:19:10PM +0300, Maxim Nestratov wrote: > 21.03.2016 13:36, Daniel P. Berrange пишет: >> On Mon, Mar 21, 2016 at 01:18:19PM +0300, Maxim Nestratov wrote: >>> Hi all, >>> >>> It's been already quite a long time since qemu implemented QMP >>> "drive-backup" command to create block devices backups. Even more, since >>> qemu 2.4 there is a possibility to create incremental backups. Though it is >>> possible to backup all attached to a domain disk drives by combining them >>> into a single QMP transaction command, this way of creating them, not to >>> mention managing, remains inconvenient for an end user of libvirt. Moreover, >>> creating a single drive backup via QMP interface isn't handy either. That >>> said, it looks reasonable to introduce a *new backup API* based on QMP >>> "drive-backup" facilities. >>> >>> Though we can start from a single simple function, allowing to create a disk >>> backup by means of QMP "drive-backup" command, I'd like to discuss here the >>> level of management libvirt could provide for backup operations. To begin >>> with, here is the preliminary list of possible functions that I think make >>> sense for libvirt API. >>> >>> virDomainCreateBackup - which creates a backup full/incremental of >>> all/selected disks, >>> virListBackups - which lists all backups created for a particular >>> domain/target, >>> virRestoreBackup - which restores all/selected disks from a backup, >>> virDeleteBackup - which deletes all/selected disks from a backup. >>> >>> It looks like backup management functions, except create one, shouldn't be >>> or might not be bound to a particular domain and we could possibly leverage >>> storage pool API with some extension. Specifically, volume definition could >>> be extended with necessary meta data related to backups. >>> >>> The *question* is: if the whole idea about this new API as described above >>> or something similar makes sense? >> Well we certainly don't want applications going directly to the QMP >> monitor command to achieve this. So given demand for this kind of >> feature, some kind of facility will be needed in libvirt. The question >> is just what any API would look like. >> >>> If yes, then let's find out requirements for it (if any) and I will try to >>> prepare a patch set with the first RFC implementation to discuss the API in >>> more details. Looking forward for your opinions on the matter. >> Have you researched what VMWare/Hyper-V/VirtualBox APIs support in this >> area ? If they have any existing models, it'd be desirable to look at >> them to align our APIs where appropriate. > As far as I know VirtualBox doesn't have any backup API, at least there is > no reference to it at [1]. > What I know about VMWare is that they use an approach that involves > temporary snapshot creation, providing an access to quiesced disk data and > then deleting this temporary snapshot [2]. Thus, their approach is close to > what libvirt can have with 3d party managed backups, i.e. setup an NBD > server on qemu side and let 3d party backup software access it from outside. > > [1] http://download.virtualbox.org/virtualbox/SDKRef.pdf > [2] http://pubs.vmware.com/vsphere-60/index.jsp#com.vmware.vddk.pg.doc/vddkBk... > >> Also how does this fit in with recent discussions on QEMU mailing list >> about enabling 3rd party vendor backup software by having QEMU expose an >> NBD server to do efficient sparse backups. That suggested a very different >> kind of API where libvirt would not manage the backups, but just provide >> an API to allow an NBD target to be enabled for the 3rd party to manage. > If you mean this thread [3] then as far as I understand, there is an > agreement that NDB protocol gets extended a bit in such a way that external > clients are able to get dirty block bitmap and this shouldn't be exposed in > libvirt API. > > For such scenarios libvirt can have the following calls: > > virDomainStartBackup, > virDomainFinishBackup > > When an external backup is started, qemu starts NBD server, allocates CBT > (changed block tracking) bitmap and returns allocated NBD port to the > caller. After that 3d party backup software is responcible for reading > appropriate portion of data according to requested CBT bitmap. > > [3] http://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg03075.html Yes, that is the thread I was referring to. > Thus, we have two ways of creating backups. The first is external (3d party > managed), which is backed by QMP "blockdev-backup" command, which doesn't > need from my point of view more than two calls mentioned above. The second > is a another one, which is backed by QMP "drive-backup" command. My first > letter covered only the latter case and I believe that this backup > management API will be very useful. What is unclear still is: if there > should be disk backup facilities only or whole VM backup as well, should > libvirt be able to restore deleted VMs, should it be able to list created > backups and so on. As for me, using storage pools with some extensions of > storage pool API could perfectly solve this task. What do you think? Ok, yes, there is clearly two completely separate modes of dealing with backups, externally managed and internally (to libvirt) managed. I can understand the desire to support both modes of operating backups. I was wondering what the difference is between doing a backup of the VM vs taking a snapshot of the VM. At a high level they feel pretty much the same, just no memory is snapshot for backups. IIUC though at the disk level, they pretty much inverted, a snapshot switches the running qcow2 file for the VM to point to a new qcow2 overlay, while a backup never touchs the VM disk, always using separate qcow2 images. So I can see why we'd want explicit backup support separately from snapshot support.

At an API level it feels like the design of our snapshot APIs would map fairly naturally onto the new backups APIs, so getting consistency between the two is desirable IMHO.

In particular the snapshot API for creating a snapshot allows an XML document to be fed in which describes how each disk is to be snapshot. I think we would need the same kind of flexibility for backups, to avoid having to hardcode the fact that backups always use qcow2. ie if a VM is using RBD, we want mgmt apps to be able to indicate that the backup should use a particular RBD volume too. Of course it should also be possible for an RBD backed guest to be able to save its backups to local raw/qcow2 files. We should also be able to indicate that some disks should be skipped for pupose of backups. So backup creation clearly needs a high level of configurability in general. We don't have to implement the full support matrix but the API design should allow for it in the future. There's a question as to whether should have allow for some default backup location if none is specified. eg perhaps we should always just store backups in /var/spool/backups/libvirt/qemu by default if the user/app didn't provide an explicit list of target volumes to hold the backup. This would allow the backup API to have better ease of use in the simple case

Regards, Daniel