8:42 a.m.
On 6/22/22 23:26, Jonathon Jongsma wrote:
Hi guys,
I've been working on adding support for nbdkit to libvirt for network
storage sources like http and ssh. See
https://bugzilla.redhat.com/show_bug.cgi?id=2016527 for more
information, but the summary is that RHEL does not want to ship the qemu
storage plugins for curl and ssh. Handling them outside of the qemu
process provides several advantages such as reduced attack surface and
stability.
I have something that works for me, but as I have not dealt with the
storage stuff much before, I have a feeling that I'm missing some
things.
A quick summary of the code:
- at startup I query to see whether nbdkit exists on the host and if
so, I query which plugins/filters are installed. This is stored as
qemuNbdkitCaps on the qemu driver
- When the driver prepares the domain, we go through each disk source
and determine whether the nbdkit capabilities allow us to support
this disk via nbdkit, and if so, we allocate a qemuNbdkitProcess
object and stash it in the private data of the virStorageSource.
- The presence or absence of this qemuNbdkitProcess data then indicates
whether this disk will be served to qemu indirectly via nbdkit or not
- When we launch the qemuProcess, as part of the "external device
start" step, I launch a ndkit process for each disk that is supported
by nbdkit. I also optionally fork a child process to communicate
authentication details and cookies to the nbdkit process via a unix
socket.
- for devices which are served by the ndkit process, I change the qemu
commandline in the following ways:
- I no longer pass auth/cookie secrets to qemu (those are handled by
nbdkit)
- I replace the actual network URL of the remote disk source with the
path to the nbdkit unix socket
Known shortcomings
- I don't yet re-query for nbdkit / nbdkit caps, so need to restart libvirt to
pick up newly-installed nbdkit or additional capabilities
There are couple of ways to solve this:
1) mimic what we do for dnsmasq: basically, when initializing the bridge
driver we query dnsmasq for caps networkStateInitialize() ->
dnsmasqCapsNewFromBinary() and when needing the caps for real (i.e.
spawning dnsmasq when starting a network) we forcefully refresh the
caps: networkStartDhcpDaemon() -> networkDnsmasqCapsRefresh().
2) use full blown virFileCache() machinery which reacts to mtime of
given binary.
An argument against going with 1) is that while spawning an extra
process (dnsmasq) when starting a network is acceptable (a network is
not started that often), spawning an extra process for each disk (or
even an image in backingchain) might be too expensive.
- testing is pretty limited at the moment
- selinux not working yet
- creating disks isn't supported, though Rich has added some support
for that upstream in the nbdkit ssh plugin.
I'd appreciate feedback on what i've got so far.
Jonathon Jongsma (3):
docs: clarify 'readahead' and 'timeout' for disks
schema: Be more flexible for diskSourceNetworkProtocolPropsCommon
WIP: use nbdkit for remote disk sources
docs/formatdomain.rst | 10 +-
include/libvirt/virterror.h | 1 +
po/POTFILES | 1 +
src/conf/schemas/domaincommon.rng | 34 +-
src/qemu/meson.build | 1 +
src/qemu/qemu_block.c | 64 +-
src/qemu/qemu_block.h | 1 +
src/qemu/qemu_command.c | 26 +-
src/qemu/qemu_conf.c | 19 +
src/qemu/qemu_conf.h | 5 +
src/qemu/qemu_domain.c | 110 ++-
src/qemu/qemu_domain.h | 5 +
src/qemu/qemu_driver.c | 4 +-
src/qemu/qemu_extdevice.c | 25 +
src/qemu/qemu_nbdkit.c | 629 ++++++++++++++++++
src/qemu/qemu_nbdkit.h | 89 +++
src/qemu/qemu_validate.c | 22 +-
src/qemu/qemu_validate.h | 4 +-
src/util/virerror.c | 1 +
tests/qemublocktest.c | 8 +-
tests/qemustatusxml2xmldata/modern-in.xml | 1 -
...sk-cdrom-network-nbdkit.x86_64-latest.args | 42 ++
.../disk-cdrom-network-nbdkit.xml | 1 +
...isk-network-http-nbdkit.x86_64-latest.args | 45 ++
.../disk-network-http-nbdkit.xml | 1 +
...work-source-curl-nbdkit.x86_64-latest.args | 49 ++
.../disk-network-source-curl-nbdkit.xml | 1 +
...isk-network-source-curl.x86_64-latest.args | 53 ++
.../disk-network-source-curl.xml | 71 ++
tests/qemuxml2argvtest.c | 12 +
tests/testutilsqemu.c | 16 +
tests/testutilsqemu.h | 4 +
32 files changed, 1302 insertions(+), 53 deletions(-)
create mode 100644 src/qemu/qemu_nbdkit.c
create mode 100644 src/qemu/qemu_nbdkit.h
create mode 100644 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.x86_64-latest.args
create mode 120000 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.xml
create mode 100644 tests/qemuxml2argvdata/disk-network-http-nbdkit.x86_64-latest.args
create mode 120000 tests/qemuxml2argvdata/disk-network-http-nbdkit.xml
create mode 100644
tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.x86_64-latest.args
create mode 120000 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.xml
create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.x86_64-latest.args
create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.xml