[libvirt] [PATCH 0/6] [RFC] Add disk streaming support to libvirt

( Those on Cc: sorry for any duplicated messages. git was configured to use the wrong smtp relay ) I've been working with Anthony Liguori and Stefan Hajnoczi to enable data streaming to copy-on-read disk images in qemu. This work is working its way through review and I expect it to be upstream soon as part of the support for the new QED disk image format. Disk streaming is extremely useful when provisioning domains from a central repository of template images. Currently the domain must be provisioned by either: 1) copying the template image to local storage before the VM can be started or, 2) creating a qcow2 image that backs to a base image in the remote repository. Option 1 can introduce a significant delay when provisioning large disks. Option 2 introduces a permanent dependency on a remote service, increased network load to satisfy disk reads. Device streaming provides the "instant-on" benefits of option 2 without introducing a permanent dependency to the image repository. Once the VM is started, the contents of the disk can be streamed to the local image in parallel. Once streaming is finished, the domain has a complete and coherent copy of the image and no longer depends on the central image repository. Qemu will support two streaming modes: full device and single sector. Full device streaming is the easiest to use because one command will cause the whole device to be streamed as fast as possible. Single sector mode can be used if one wants to throttle streaming to reduce I/O pressure. In this mode, a management tool issues individual commands to stream single sectors. To enable this support in libvirt, I propose the following API... virDomainStreamDisk() will start or stop a full device stream or stream a single sector of a device. The behavior is controlled by setting virDomainStreamDiskFlags. When either starting or stopping a full device stream, the return value is either 0 or -1 to indicate whether the operation succeeded. For a single sector stream, a device offset is returned (or -1 on failure). This value can be used to continue streaming with a subsequent call to virDomainStreamDisk(). virDomainStreamDiskInfo() returns information about active full device streams (the device alias, current streaming position, and total size). While streaming can be used to enable live block migration, this API addresses the direct streaming use case. If we can agree that the above use case is compelling on its own, I would prefer to work on the block migration workflow immediately after this series. In that case, we just need to be careful not to paint ourselves into a corner with respect to migration, but specifics about that feature can be discussed later. Does this seem reasonable? Adam Litke (6): Add new API virDomainStreamDisk[Info] to header and drivers virDomainStreamDisk: Add public symbols to libvirt API Implement disk streaming in the qemu driver Add disk streaming support to the remote driver Add new disk streaming commands to virsh python: Add python bindings for virDomainStreamDisk[Info] daemon/remote.c | 96 +++++++++++++++++++++ daemon/remote_dispatch_args.h | 2 + daemon/remote_dispatch_prototypes.h | 16 ++++ daemon/remote_dispatch_ret.h | 2 + daemon/remote_dispatch_table.h | 10 ++ include/libvirt/libvirt.h.in | 34 ++++++++ python/generator.py | 1 + python/libvirt-override-api.xml | 5 + python/libvirt-override.c | 46 ++++++++++ src/driver.h | 11 +++ src/esx/esx_driver.c | 2 + src/libvirt.c | 121 +++++++++++++++++++++++++++ src/libvirt_public.syms | 6 ++ src/lxc/lxc_driver.c | 2 + src/opennebula/one_driver.c | 2 + src/openvz/openvz_driver.c | 2 + src/phyp/phyp_driver.c | 2 + src/qemu/qemu_driver.c | 75 +++++++++++++++++ src/qemu/qemu_monitor.c | 41 +++++++++ src/qemu/qemu_monitor.h | 6 ++ src/qemu/qemu_monitor_json.c | 104 +++++++++++++++++++++++ src/qemu/qemu_monitor_json.h | 7 ++ src/qemu/qemu_monitor_text.c | 156 ++++++++++++++++++++++++++++++++++ src/qemu/qemu_monitor_text.h | 8 ++ src/remote/remote_driver.c | 85 +++++++++++++++++++ src/remote/remote_protocol.c | 63 ++++++++++++++ src/remote/remote_protocol.h | 51 +++++++++++ src/remote/remote_protocol.x | 37 ++++++++- src/test/test_driver.c | 2 + src/uml/uml_driver.c | 2 + src/vbox/vbox_tmpl.c | 2 + src/xen/xen_driver.c | 2 + tools/virsh.c | 157 +++++++++++++++++++++++++++++++++++ 33 files changed, 1157 insertions(+), 1 deletions(-) -- 1.7.3.2.164.g6f10c