[libvirt] [PATCH 00/38] Implement sparse streams for libvirt

There were already some attempts to do this in the past, but neither of them was successful. Anyway, let me just show you how good these perform: 7.9G -rw-r--r-- 1 root root 21G Feb 15 14:04 /var/lib/libvirt/images/gentoo.qcow2 libvirt.git # time virsh vol-download --sparse /var/lib/libvirt/images/gentoo.qcow2 /mnt/floppy/gentoo.qcow2 real 0m41.627s user 0m3.880s sys 0m5.720s libvirt.git # time virsh vol-download /var/lib/libvirt/images/gentoo.qcow2 /mnt/floppy/gentoo.qcow2 real 2m22.357s user 1m20.590s sys 0m12.510s All the patches can be found on my github: https://github.com/zippy2/libvirt/tree/sparse_iohelper5 (Yes, weird name, but there were plenty of local attempts to implement this and I'm not throwing them away until the feature is merged) Michal Privoznik (38): fdstreamtest: Rename tempdir fdstreamtest: Print more info on read failure fdstream: s/struct virFDStreamData */virFDStreamDataPtr/ virFDStreamData: Turn into virObjectLockable virfdstream: Drop iohelper in favour of a thread virfdstream: Use messages instead of pipe iohelper: Remove unused mode util: Introduce virFileInData Introduce virStreamRecvFlags Implement virStreamRecvFlags to some drivers Introduce virStreamSkip Introduce virStreamHoleSize Introduce VIR_STREAM_RECV_STOP_AT_HOLE flag Introduce virStreamSparseRecvAll Introduce virStreamSparseSendAll Introduce virStreamInData virNetClientStreamNew: Track origin stream Track if stream is skippable RPC: Introduce virNetStreamSkip Introduce VIR_NET_STREAM_SKIP message type Teach wireshark plugin about VIR_NET_STREAM_SKIP daemon: Introduce virNetServerProgramSendStreamSkip virnetclientstream: Introduce virNetClientStreamSendSkip daemon: Implement VIR_NET_STREAM_SKIP handling virnetclientstream: Introduce virNetClientStreamHandleSkip remote_driver: Implement virStreamSkip virNetClientStreamRecvPacket: Introduce @flags argument Introduce virNetClientStreamHoleSize remote: Implement virStreamHoleSize virNetClientStream: Wire up VIR_NET_STREAM_SKIP remote_driver: Implement VIR_STREAM_RECV_STOP_AT_HOLE daemonStreamHandleRead: Wire up seekable stream daemon: Don't call virStreamInData so often fdstream: Implement sparse stream gendispatch: Introduce @sparseflag for our calls Introduce virStorageVol{Download,Upload}Flags virsh: Implement sparse stream to vol-download virsh: Implement sparse stream to vol-upload daemon/remote.c | 2 +- daemon/stream.c | 147 +++++- daemon/stream.h | 3 +- include/libvirt/libvirt-storage.h | 9 + include/libvirt/libvirt-stream.h | 86 +++- src/driver-stream.h | 23 + src/esx/esx_stream.c | 16 +- src/libvirt-storage.c | 4 +- src/libvirt-stream.c | 453 ++++++++++++++++++ src/libvirt_internal.h | 3 + src/libvirt_private.syms | 2 + src/libvirt_public.syms | 9 + src/libvirt_remote.syms | 3 + src/remote/remote_driver.c | 89 +++- src/remote/remote_protocol.x | 2 + src/rpc/gendispatch.pl | 21 +- src/rpc/virnetclient.c | 1 + src/rpc/virnetclientstream.c | 202 +++++++- src/rpc/virnetclientstream.h | 17 +- src/rpc/virnetprotocol.x | 16 +- src/rpc/virnetserverprogram.c | 33 ++ src/rpc/virnetserverprogram.h | 7 + src/storage/storage_driver.c | 4 +- src/storage/storage_util.c | 10 +- src/util/iohelper.c | 72 +-- src/util/virfdstream.c | 888 ++++++++++++++++++++++++++++------- src/util/virfdstream.h | 2 +- src/util/virfile.c | 81 ++++ src/util/virfile.h | 3 + src/virnetprotocol-structs | 4 + tests/fdstreamtest.c | 12 +- tests/virfiletest.c | 203 ++++++++ tools/virsh-util.c | 49 ++ tools/virsh-util.h | 24 + tools/virsh-volume.c | 49 +- tools/virsh.pod | 6 +- tools/wireshark/src/packet-libvirt.c | 48 ++ tools/wireshark/src/packet-libvirt.h | 2 + 38 files changed, 2284 insertions(+), 321 deletions(-) -- 2.10.2