[libvirt PATCH v7 00/35] Use nbdkit for http/ftp/ssh network drives in libvirt

This is the seventh version of this patch series. See https://bugzilla.redhat.com/show_bug.cgi?id=2016527 for more information. Note that testing this requires selinux policy changes which are not fully done, but there is a new policy in development that has allowed me to run with selinux in enforcing mode for the common cases. See https://bugzilla.redhat.com/show_bug.cgi?id=2182505 for more information. The following scenarios should work now with selinux enabled using the selinux policy from that bug: - http/https disks - ssh disks with password authentication - ssh disks with passwordless keyfile The one major thing that doesn't work and is difficult to get working with selinux enabled is the ssh-agent. This is because there doesn't seem to be any selinux policy for ssh-agent, so by default the ssh-agent socket is labeled unconfined_t. We cannot allow access from the libvirt/qemu to unconfined_t because that would open up access to just about anything on the host. So additional work will likely be necessary for ssh-agent/libvirt interaction in the future. Fortunately ssh-agent is something that never was really supported with the old qemu block driver either, so I think we could potentially merge this patchset either without the ssh-agent patches or with a note that ssh-agent won't work with selinux enabled. Note also that gitlab CI will not work for this series without changes to the ci definitions due to the addition of libnbd dependency. Changes in v7: - rebased to latest master - moved restarting of nbdkit process to per-domain event thread - a few other smaller changes suggested by Peter in v6 Jonathon Jongsma (35): schema: allow 'ssh' as a protocol for network disks qemu: Add functions for determining nbdkit availability qemu: expand nbdkit capabilities util: Allow virFileCache data to be any GObject qemu: implement basic virFileCache for nbdkit caps qemu: implement persistent file cache for nbdkit caps qemu: use file cache for nbdkit caps qemu: Add qemuNbdkitProcess qemu: query nbdkit module dir from binary qemu: add functions to start and stop nbdkit Generalize qemuDomainLogContextNew() qemu: Extract qemuDomainLogContext into a new file qemu: move qemuProcessReadLog() to qemuLogContext qemu: log error output from nbdkit tests: add ability to test various nbdkit capabilities qemu: split qemuDomainSecretStorageSourcePrepare qemu: include nbdkit state in private xml util: secure erase virCommand send buffers qemu: pass sensitive data to nbdkit via pipe qemu: use nbdkit to serve network disks if available util: make virCommandSetSendBuffer testable tests: add tests for nbdkit invocation qemu: add test for authenticating a https network disk qemu: Monitor nbdkit process for exit qemu: Taint domain if nbdkit restart fails qemu: try to connect to nbdkit early to detect errors schema: add password configuration for ssh disk qemu: implement password auth for ssh disks with nbdkit schema: add configuration for host verification of ssh disks qemu: implement knownHosts for ssh disks with nbdkit schema: add keyfile configuration for ssh disks qemu: implement keyfile auth for ssh disks with nbdkit schema: add ssh-agent configuration for ssh disks qemu: implement ssh-agent auth for ssh disks with nbdkit rpm: update spec file for for nbdkit support build-aux/syntax-check.mk | 2 +- docs/formatdomain.rst | 45 +- libvirt.spec.in | 8 + meson.build | 14 + meson_options.txt | 1 + po/POTFILES | 2 + src/conf/domain_conf.c | 38 + src/conf/domain_conf.h | 1 + src/conf/schemas/domaincommon.rng | 55 + src/conf/storage_source_conf.c | 6 + src/conf/storage_source_conf.h | 6 +- src/libvirt_private.syms | 1 + src/qemu/meson.build | 3 + src/qemu/qemu_block.c | 162 ++- src/qemu/qemu_conf.c | 22 + src/qemu/qemu_conf.h | 6 + src/qemu/qemu_domain.c | 436 +++--- src/qemu/qemu_domain.h | 31 +- src/qemu/qemu_driver.c | 21 + src/qemu/qemu_extdevice.c | 62 + src/qemu/qemu_hotplug.c | 7 + src/qemu/qemu_logcontext.c | 329 +++++ src/qemu/qemu_logcontext.h | 41 + src/qemu/qemu_nbdkit.c | 1277 +++++++++++++++++ src/qemu/qemu_nbdkit.h | 119 ++ src/qemu/qemu_nbdkitpriv.h | 31 + src/qemu/qemu_process.c | 122 +- src/qemu/qemu_process.h | 3 + src/util/vircommand.c | 19 +- src/util/vircommand.h | 8 + src/util/vircommandpriv.h | 4 + src/util/virfilecache.c | 14 +- src/util/virfilecache.h | 2 +- tests/meson.build | 1 + tests/qemublocktest.c | 2 +- ...w2-invalid.json => network-ssh-qcow2.json} | 0 ...cow2-invalid.xml => network-ssh-qcow2.xml} | 0 .../disk-cdrom-network.args.disk0 | 6 + .../disk-cdrom-network.args.disk1 | 8 + .../disk-cdrom-network.args.disk1.pipe.778 | 1 + .../disk-cdrom-network.args.disk2 | 8 + .../disk-cdrom-network.args.disk2.pipe.780 | 1 + .../disk-network-http.args.disk0 | 6 + .../disk-network-http.args.disk1 | 5 + .../disk-network-http.args.disk2 | 6 + .../disk-network-http.args.disk2.pipe.778 | 1 + .../disk-network-http.args.disk3 | 7 + .../disk-network-http.args.disk3.pipe.780 | 1 + ...work-source-curl-nbdkit-backing.args.disk0 | 7 + ...ce-curl-nbdkit-backing.args.disk0.pipe.778 | 1 + .../disk-network-source-curl.args.disk0 | 7 + ...sk-network-source-curl.args.disk0.pipe.778 | 1 + .../disk-network-source-curl.args.disk1 | 9 + ...sk-network-source-curl.args.disk1.pipe.780 | 1 + ...sk-network-source-curl.args.disk1.pipe.782 | 1 + .../disk-network-source-curl.args.disk2 | 7 + ...sk-network-source-curl.args.disk2.pipe.782 | 1 + ...sk-network-source-curl.args.disk2.pipe.784 | 1 + .../disk-network-source-curl.args.disk3 | 6 + .../disk-network-source-curl.args.disk4 | 6 + .../disk-network-ssh-key.args.disk0 | 9 + .../disk-network-ssh-key.args.disk1 | 9 + .../disk-network-ssh-password.args.disk0 | 9 + ...k-network-ssh-password.args.disk0.pipe.778 | 1 + .../disk-network-ssh.args.disk0 | 7 + .../disk-network-ssh.args.disk1 | 8 + .../disk-network-ssh.args.disk1.pipe.778 | 1 + .../disk-network-ssh.args.disk2 | 9 + tests/qemunbdkittest.c | 310 ++++ tests/qemustatusxml2xmldata/modern-in.xml | 4 + ...sk-cdrom-network-nbdkit.x86_64-latest.args | 42 + .../disk-cdrom-network-nbdkit.xml | 1 + ...isk-network-http-nbdkit.x86_64-latest.args | 44 + .../disk-network-http-nbdkit.xml | 1 + ...rce-curl-nbdkit-backing.x86_64-latest.args | 37 + ...isk-network-source-curl-nbdkit-backing.xml | 45 + ...work-source-curl-nbdkit.x86_64-latest.args | 49 + .../disk-network-source-curl-nbdkit.xml | 1 + ...isk-network-source-curl.x86_64-latest.args | 53 + .../disk-network-source-curl.xml | 74 + .../qemuxml2argvdata/disk-network-ssh-key.xml | 44 + ...disk-network-ssh-nbdkit.x86_64-latest.args | 35 + .../disk-network-ssh-nbdkit.xml | 1 + ...sk-network-ssh-password.x86_64-latest.args | 35 + .../disk-network-ssh-password.xml | 35 + .../disk-network-ssh.x86_64-latest.args | 35 + tests/qemuxml2argvdata/disk-network-ssh.xml | 32 + tests/qemuxml2argvtest.c | 19 + tests/testutilsqemu.c | 26 + tests/testutilsqemu.h | 4 + 90 files changed, 3507 insertions(+), 472 deletions(-) create mode 100644 src/qemu/qemu_logcontext.c create mode 100644 src/qemu/qemu_logcontext.h create mode 100644 src/qemu/qemu_nbdkit.c create mode 100644 src/qemu/qemu_nbdkit.h create mode 100644 src/qemu/qemu_nbdkitpriv.h rename tests/qemublocktestdata/imagecreate/{network-ssh-qcow2-invalid.json => network-ssh-qcow2.json} (100%) rename tests/qemublocktestdata/imagecreate/{network-ssh-qcow2-invalid.xml => network-ssh-qcow2.xml} (100%) create mode 100644 tests/qemunbdkitdata/disk-cdrom-network.args.disk0 create mode 100644 tests/qemunbdkitdata/disk-cdrom-network.args.disk1 create mode 100644 tests/qemunbdkitdata/disk-cdrom-network.args.disk1.pipe.778 create mode 100644 tests/qemunbdkitdata/disk-cdrom-network.args.disk2 create mode 100644 tests/qemunbdkitdata/disk-cdrom-network.args.disk2.pipe.780 create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk0 create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk1 create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk2 create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk2.pipe.778 create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk3 create mode 100644 tests/qemunbdkitdata/disk-network-http.args.disk3.pipe.780 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl-nbdkit-backing.args.disk0 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl-nbdkit-backing.args.disk0.pipe.778 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk0 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk0.pipe.778 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk1 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk1.pipe.780 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk1.pipe.782 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk2 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk2.pipe.782 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk2.pipe.784 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk3 create mode 100644 tests/qemunbdkitdata/disk-network-source-curl.args.disk4 create mode 100644 tests/qemunbdkitdata/disk-network-ssh-key.args.disk0 create mode 100644 tests/qemunbdkitdata/disk-network-ssh-key.args.disk1 create mode 100644 tests/qemunbdkitdata/disk-network-ssh-password.args.disk0 create mode 100644 tests/qemunbdkitdata/disk-network-ssh-password.args.disk0.pipe.778 create mode 100644 tests/qemunbdkitdata/disk-network-ssh.args.disk0 create mode 100644 tests/qemunbdkitdata/disk-network-ssh.args.disk1 create mode 100644 tests/qemunbdkitdata/disk-network-ssh.args.disk1.pipe.778 create mode 100644 tests/qemunbdkitdata/disk-network-ssh.args.disk2 create mode 100644 tests/qemunbdkittest.c create mode 100644 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.x86_64-latest.args create mode 120000 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.xml create mode 100644 tests/qemuxml2argvdata/disk-network-http-nbdkit.x86_64-latest.args create mode 120000 tests/qemuxml2argvdata/disk-network-http-nbdkit.xml create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit-backing.x86_64-latest.args create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit-backing.xml create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.x86_64-latest.args create mode 120000 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.xml create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.x86_64-latest.args create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.xml create mode 100644 tests/qemuxml2argvdata/disk-network-ssh-key.xml create mode 100644 tests/qemuxml2argvdata/disk-network-ssh-nbdkit.x86_64-latest.args create mode 120000 tests/qemuxml2argvdata/disk-network-ssh-nbdkit.xml create mode 100644 tests/qemuxml2argvdata/disk-network-ssh-password.x86_64-latest.args create mode 100644 tests/qemuxml2argvdata/disk-network-ssh-password.xml create mode 100644 tests/qemuxml2argvdata/disk-network-ssh.x86_64-latest.args create mode 100644 tests/qemuxml2argvdata/disk-network-ssh.xml -- 2.41.0