[libvirt PATCH 0/3] RFC: using nbdkit for network drives in libvirt

Hi guys, I've been working on adding support for nbdkit to libvirt for network storage sources like http and ssh. See https://bugzilla.redhat.com/show_bug.cgi?id=2016527 for more information, but the summary is that RHEL does not want to ship the qemu storage plugins for curl and ssh. Handling them outside of the qemu process provides several advantages such as reduced attack surface and stability. I have something that works for me, but as I have not dealt with the storage stuff much before, I have a feeling that I'm missing some things. A quick summary of the code: - at startup I query to see whether nbdkit exists on the host and if so, I query which plugins/filters are installed. This is stored as qemuNbdkitCaps on the qemu driver - When the driver prepares the domain, we go through each disk source and determine whether the nbdkit capabilities allow us to support this disk via nbdkit, and if so, we allocate a qemuNbdkitProcess object and stash it in the private data of the virStorageSource. - The presence or absence of this qemuNbdkitProcess data then indicates whether this disk will be served to qemu indirectly via nbdkit or not - When we launch the qemuProcess, as part of the "external device start" step, I launch a ndkit process for each disk that is supported by nbdkit. I also optionally fork a child process to communicate authentication details and cookies to the nbdkit process via a unix socket. - for devices which are served by the ndkit process, I change the qemu commandline in the following ways: - I no longer pass auth/cookie secrets to qemu (those are handled by nbdkit) - I replace the actual network URL of the remote disk source with the path to the nbdkit unix socket Known shortcomings - I don't yet re-query for nbdkit / nbdkit caps, so need to restart libvirt to pick up newly-installed nbdkit or additional capabilities - testing is pretty limited at the moment - selinux not working yet - creating disks isn't supported, though Rich has added some support for that upstream in the nbdkit ssh plugin. I'd appreciate feedback on what i've got so far. Jonathon Jongsma (3): docs: clarify 'readahead' and 'timeout' for disks schema: Be more flexible for diskSourceNetworkProtocolPropsCommon WIP: use nbdkit for remote disk sources docs/formatdomain.rst | 10 +- include/libvirt/virterror.h | 1 + po/POTFILES | 1 + src/conf/schemas/domaincommon.rng | 34 +- src/qemu/meson.build | 1 + src/qemu/qemu_block.c | 64 +- src/qemu/qemu_block.h | 1 + src/qemu/qemu_command.c | 26 +- src/qemu/qemu_conf.c | 19 + src/qemu/qemu_conf.h | 5 + src/qemu/qemu_domain.c | 110 ++- src/qemu/qemu_domain.h | 5 + src/qemu/qemu_driver.c | 4 +- src/qemu/qemu_extdevice.c | 25 + src/qemu/qemu_nbdkit.c | 629 ++++++++++++++++++ src/qemu/qemu_nbdkit.h | 89 +++ src/qemu/qemu_validate.c | 22 +- src/qemu/qemu_validate.h | 4 +- src/util/virerror.c | 1 + tests/qemublocktest.c | 8 +- tests/qemustatusxml2xmldata/modern-in.xml | 1 - ...sk-cdrom-network-nbdkit.x86_64-latest.args | 42 ++ .../disk-cdrom-network-nbdkit.xml | 1 + ...isk-network-http-nbdkit.x86_64-latest.args | 45 ++ .../disk-network-http-nbdkit.xml | 1 + ...work-source-curl-nbdkit.x86_64-latest.args | 49 ++ .../disk-network-source-curl-nbdkit.xml | 1 + ...isk-network-source-curl.x86_64-latest.args | 53 ++ .../disk-network-source-curl.xml | 71 ++ tests/qemuxml2argvtest.c | 12 + tests/testutilsqemu.c | 16 + tests/testutilsqemu.h | 4 + 32 files changed, 1302 insertions(+), 53 deletions(-) create mode 100644 src/qemu/qemu_nbdkit.c create mode 100644 src/qemu/qemu_nbdkit.h create mode 100644 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.x86_64-latest.args create mode 120000 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.xml create mode 100644 tests/qemuxml2argvdata/disk-network-http-nbdkit.x86_64-latest.args create mode 120000 tests/qemuxml2argvdata/disk-network-http-nbdkit.xml create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.x86_64-latest.args create mode 120000 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.xml create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.x86_64-latest.args create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.xml -- 2.35.3

Document the format of the 'readahead' and 'timeout' XML elements more accurately. Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> --- docs/formatdomain.rst | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/formatdomain.rst b/docs/formatdomain.rst index 991d45672a..62a94890f0 100644 --- a/docs/formatdomain.rst +++ b/docs/formatdomain.rst @@ -2878,11 +2878,13 @@ paravirtualized driver is specified via the ``disk`` element. more cookies. The cookie name and value must conform to the HTTP specification. :since:`Since 6.2.0` ``readahead`` - Specifies the size of the readahead buffer for protocols which support it. - (all 'curl' based drivers in qemu). The size is in bytes. Note that '0' is - considered as if the value is not provided. :since:`Since 6.2.0` + The ``readahead`` element has a ``size`` attribute which specifies the + size of the readahead buffer in bytes for protocols which support it. + Note that '0' is considered as if the value is not provided. + :since:`Since 6.2.0` ``timeout`` - Specifies the connection timeout for protocols which support it. Note that + The ``timeout`` element has a ``seconds`` attribute which specifies the + connection timeout in seconds for protocols which support it. Note that '0' is considered as if the value is not provided. :since:`Since 6.2.0` ``identity`` When using an ``nfs`` protocol, this is used to provide information on the -- 2.35.3

On 6/22/22 23:26, Jonathon Jongsma wrote:
Document the format of the 'readahead' and 'timeout' XML elements more accurately.
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> --- docs/formatdomain.rst | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> This patch makes sense regardless of the RFC nature of the series. I say go ahead and push it. Michal

Add <interleave> to allow the subproperties to be specified in any order. Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> --- src/conf/schemas/domaincommon.rng | 34 ++++++++++++++++--------------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/src/conf/schemas/domaincommon.rng b/src/conf/schemas/domaincommon.rng index e2246e6b63..da2fb0d5cb 100644 --- a/src/conf/schemas/domaincommon.rng +++ b/src/conf/schemas/domaincommon.rng @@ -1967,22 +1967,24 @@ </define> <define name="diskSourceNetworkProtocolPropsCommon"> - <optional> - <element name="readahead"> - <attribute name="size"> - <ref name="positiveInteger"/> - </attribute> - <empty/> - </element> - </optional> - <optional> - <element name="timeout"> - <attribute name="seconds"> - <ref name="positiveInteger"/> - </attribute> - <empty/> - </element> - </optional> + <interleave> + <optional> + <element name="readahead"> + <attribute name="size"> + <ref name="positiveInteger"/> + </attribute> + <empty/> + </element> + </optional> + <optional> + <element name="timeout"> + <attribute name="seconds"> + <ref name="positiveInteger"/> + </attribute> + <empty/> + </element> + </optional> + </interleave> </define> <define name="diskSourceNetworkProtocolSSLVerify"> -- 2.35.3

On 6/22/22 23:26, Jonathon Jongsma wrote:
Add <interleave> to allow the subproperties to be specified in any order.
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> --- src/conf/schemas/domaincommon.rng | 34 ++++++++++++++++--------------- 1 file changed, 18 insertions(+), 16 deletions(-)
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> This patch makes sense regardless of the RFC nature of the series. I say go ahead and push it. Michal

--- include/libvirt/virterror.h | 1 + po/POTFILES | 1 + src/qemu/meson.build | 1 + src/qemu/qemu_block.c | 64 +- src/qemu/qemu_block.h | 1 + src/qemu/qemu_command.c | 26 +- src/qemu/qemu_conf.c | 19 + src/qemu/qemu_conf.h | 5 + src/qemu/qemu_domain.c | 110 ++- src/qemu/qemu_domain.h | 5 + src/qemu/qemu_driver.c | 4 +- src/qemu/qemu_extdevice.c | 25 + src/qemu/qemu_nbdkit.c | 629 ++++++++++++++++++ src/qemu/qemu_nbdkit.h | 89 +++ src/qemu/qemu_validate.c | 22 +- src/qemu/qemu_validate.h | 4 +- src/util/virerror.c | 1 + tests/qemublocktest.c | 8 +- tests/qemustatusxml2xmldata/modern-in.xml | 1 - ...sk-cdrom-network-nbdkit.x86_64-latest.args | 42 ++ .../disk-cdrom-network-nbdkit.xml | 1 + ...isk-network-http-nbdkit.x86_64-latest.args | 45 ++ .../disk-network-http-nbdkit.xml | 1 + ...work-source-curl-nbdkit.x86_64-latest.args | 49 ++ .../disk-network-source-curl-nbdkit.xml | 1 + ...isk-network-source-curl.x86_64-latest.args | 53 ++ .../disk-network-source-curl.xml | 71 ++ tests/qemuxml2argvtest.c | 12 + tests/testutilsqemu.c | 16 + tests/testutilsqemu.h | 4 + 30 files changed, 1278 insertions(+), 33 deletions(-) create mode 100644 src/qemu/qemu_nbdkit.c create mode 100644 src/qemu/qemu_nbdkit.h create mode 100644 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.x86_64-latest.args create mode 120000 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.xml create mode 100644 tests/qemuxml2argvdata/disk-network-http-nbdkit.x86_64-latest.args create mode 120000 tests/qemuxml2argvdata/disk-network-http-nbdkit.xml create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.x86_64-latest.args create mode 120000 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.xml create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.x86_64-latest.args create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.xml diff --git a/include/libvirt/virterror.h b/include/libvirt/virterror.h index df13e4f11e..dd198dfd7d 100644 --- a/include/libvirt/virterror.h +++ b/include/libvirt/virterror.h @@ -141,6 +141,7 @@ typedef enum { VIR_FROM_TPM = 70, /* Error from TPM (Since: 5.6.0) */ VIR_FROM_BPF = 71, /* Error from BPF code (Since: 5.10.0) */ VIR_FROM_CH = 72, /* Error from Cloud-Hypervisor driver (Since: 7.5.0) */ + VIR_FROM_NBDKIT = 73, /* Error from Nbdkit code (Since: 8.5.0) */ # ifdef VIR_ENUM_SENTINELS VIR_ERR_DOMAIN_LAST /* (Since: 0.9.13) */ diff --git a/po/POTFILES b/po/POTFILES index faaba53c8f..99284b8173 100644 --- a/po/POTFILES +++ b/po/POTFILES @@ -177,6 +177,7 @@ src/qemu/qemu_monitor.c src/qemu/qemu_monitor_json.c src/qemu/qemu_monitor_text.c src/qemu/qemu_namespace.c +src/qemu/qemu_nbdkit.c src/qemu/qemu_process.c src/qemu/qemu_qapi.c src/qemu/qemu_saveimage.c diff --git a/src/qemu/meson.build b/src/qemu/meson.build index 96952cc52d..101cf3591f 100644 --- a/src/qemu/meson.build +++ b/src/qemu/meson.build @@ -28,6 +28,7 @@ qemu_driver_sources = [ 'qemu_monitor_json.c', 'qemu_monitor_text.c', 'qemu_namespace.c', + 'qemu_nbdkit.c', 'qemu_process.c', 'qemu_qapi.c', 'qemu_saveimage.c', diff --git a/src/qemu/qemu_block.c b/src/qemu/qemu_block.c index 9fe22f18f2..91f17f133f 100644 --- a/src/qemu/qemu_block.c +++ b/src/qemu/qemu_block.c @@ -773,6 +773,35 @@ qemuBlockStorageSourceGetCURLProps(virStorageSource *src, } +static virJSONValue * +qemuBlockStorageSourceGetNbdkitProps(virStorageSource *src) +{ + qemuDomainStorageSourcePrivate *srcPriv = QEMU_DOMAIN_STORAGE_SOURCE_PRIVATE(src); + virJSONValue *ret = NULL; + g_autoptr(virJSONValue) serverprops = NULL; + virStorageNetHostDef host = { NULL }; + + /* srcPriv->nbdkitProcess will already be initialized if we can use nbdkit + * to proxy this storage source */ + if (!(srcPriv && srcPriv->nbdkitProcess)) + return NULL; + + host.transport = VIR_STORAGE_NET_HOST_TRANS_UNIX; + host.socket = srcPriv->nbdkitProcess->socketfile; + serverprops = qemuBlockStorageSourceBuildJSONSocketAddress(&host, + false); + if (!serverprops) + return NULL; + + if (virJSONValueObjectAdd(&ret, + "a:server", &serverprops, + NULL) < 0) + return NULL; + + return ret; +} + + static virJSONValue * qemuBlockStorageSourceGetISCSIProps(virStorageSource *src, bool onlytarget) @@ -1207,6 +1236,14 @@ qemuBlockStorageSourceGetBackendProps(virStorageSource *src, case VIR_STORAGE_NET_PROTOCOL_FTP: case VIR_STORAGE_NET_PROTOCOL_FTPS: case VIR_STORAGE_NET_PROTOCOL_TFTP: + /* first try to use nbdkit for http/ftp sources */ + if ((fileprops = qemuBlockStorageSourceGetNbdkitProps(src))) { + driver = "nbd"; + break; + } + + /* nbdkit is not supported for this host/source, fall back to old + * qemu storage plugins */ driver = virStorageNetProtocolTypeToString(src->protocol); if (!(fileprops = qemuBlockStorageSourceGetCURLProps(src, onlytarget))) return NULL; @@ -1237,6 +1274,14 @@ qemuBlockStorageSourceGetBackendProps(virStorageSource *src, break; case VIR_STORAGE_NET_PROTOCOL_SSH: + /* first try to use nbdkit for ssh sources */ + if ((fileprops = qemuBlockStorageSourceGetNbdkitProps(src))) { + driver = "nbd"; + break; + } + + /* nbdkit is not supported for this host/source. fallback to + * the old qemu storage plugins */ driver = "ssh"; if (!(fileprops = qemuBlockStorageSourceGetSshProps(src))) return NULL; @@ -1671,6 +1716,7 @@ qemuBlockStorageSourceAttachPrepareBlockdev(virStorageSource *src, { g_autoptr(qemuBlockStorageSourceAttachData) data = NULL; unsigned int backendpropsflags = 0; + qemuDomainStorageSourcePrivate *srcpriv = QEMU_DOMAIN_STORAGE_SOURCE_PRIVATE(src); if (autoreadonly) backendpropsflags |= QEMU_BLOCK_STORAGE_SOURCE_BACKEND_PROPS_AUTO_READONLY; @@ -1685,6 +1731,7 @@ qemuBlockStorageSourceAttachPrepareBlockdev(virStorageSource *src, data->storageNodeName = src->nodestorage; data->formatNodeName = src->nodeformat; + data->useNbdkit = srcpriv && srcpriv->nbdkitProcess; if (qemuBlockStorageSourceNeedsStorageSliceLayer(src)) { if (!(data->storageSliceProps = qemuBlockStorageSourceGetBlockdevStorageSliceProps(src))) @@ -1701,6 +1748,10 @@ static int qemuBlockStorageSourceAttachApplyStorageDeps(qemuMonitor *mon, qemuBlockStorageSourceAttachData *data) { + /* when using nbdkit, data is not passed via qemu secrets */ + if (data->useNbdkit) + return 0; + if (data->prmgrProps && qemuMonitorAddObject(mon, &data->prmgrProps, &data->prmgrAlias) < 0) return -1; @@ -2205,6 +2256,8 @@ qemuBlockGetBackingStoreString(virStorageSource *src, virJSONValue *props = NULL; g_autoptr(virURI) uri = NULL; g_autofree char *backingJSON = NULL; + qemuDomainStorageSourcePrivate *srcPriv = QEMU_DOMAIN_STORAGE_SOURCE_PRIVATE(src); + bool useNbdkit = srcPriv && srcPriv->nbdkitProcess; if (!src->sliceStorage) { if (virStorageSourceIsLocalStorage(src)) { @@ -2223,7 +2276,8 @@ qemuBlockGetBackingStoreString(virStorageSource *src, src->ncookies == 0 && src->sslverify == VIR_TRISTATE_BOOL_ABSENT && src->timeout == 0 && - src->readahead == 0) { + src->readahead == 0 && + !useNbdkit) { switch ((virStorageNetProtocol) src->protocol) { case VIR_STORAGE_NET_PROTOCOL_NBD: @@ -2602,6 +2656,7 @@ qemuBlockStorageSourceCreateGetStorageProps(virStorageSource *src, g_autoptr(virJSONValue) location = NULL; const char *driver = NULL; const char *filename = NULL; + qemuDomainStorageSourcePrivate *srcPriv = QEMU_DOMAIN_STORAGE_SOURCE_PRIVATE(src); switch (actualType) { case VIR_STORAGE_TYPE_FILE: @@ -2630,6 +2685,13 @@ qemuBlockStorageSourceCreateGetStorageProps(virStorageSource *src, break; case VIR_STORAGE_NET_PROTOCOL_SSH: + if (srcPriv->nbdkitProcess) { + /* disk creation not yet supported with nbdkit, and even if it + * was supported, it would not be done with blockdev-create + * props */ + return 0; + } + driver = "ssh"; if (!(location = qemuBlockStorageSourceGetSshProps(src))) return -1; diff --git a/src/qemu/qemu_block.h b/src/qemu/qemu_block.h index 8641c8a2d2..a1f98f4452 100644 --- a/src/qemu/qemu_block.h +++ b/src/qemu/qemu_block.h @@ -113,6 +113,7 @@ struct qemuBlockStorageSourceAttachData { char *tlsAlias; virJSONValue *tlsKeySecretProps; char *tlsKeySecretAlias; + bool useNbdkit; }; diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index b307d3139c..dd410362fb 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -1578,6 +1578,7 @@ qemuBuildNetworkDriveStr(virStorageSource *src, qemuDomainSecretInfo *secinfo) { g_auto(virBuffer) buf = VIR_BUFFER_INITIALIZER; + qemuDomainStorageSourcePrivate *priv = QEMU_DOMAIN_STORAGE_SOURCE_PRIVATE(src); size_t i; char *ret = NULL; @@ -1637,6 +1638,13 @@ qemuBuildNetworkDriveStr(virStorageSource *src, case VIR_STORAGE_NET_PROTOCOL_FTP: case VIR_STORAGE_NET_PROTOCOL_FTPS: case VIR_STORAGE_NET_PROTOCOL_TFTP: + if (priv && priv->nbdkitProcess) { + virBufferAsprintf(&buf, "nbd:unix:%s", priv->nbdkitProcess->socketfile); + ret = virBufferContentAndReset(&buf); + } else { + ret = qemuBuildNetworkDriveURI(src); + } + break; case VIR_STORAGE_NET_PROTOCOL_ISCSI: case VIR_STORAGE_NET_PROTOCOL_GLUSTER: ret = qemuBuildNetworkDriveURI(src); @@ -2497,13 +2505,17 @@ qemuBuildBlockStorageSourceAttachDataCommandline(virCommand *cmd, { char *tmp; - if (qemuBuildObjectCommandline(cmd, data->prmgrProps, qemuCaps) < 0 || - qemuBuildObjectCommandline(cmd, data->authsecretProps, qemuCaps) < 0 || - qemuBuildObjectCommandline(cmd, data->encryptsecretProps, qemuCaps) < 0 || - qemuBuildObjectCommandline(cmd, data->httpcookiesecretProps, qemuCaps) < 0 || - qemuBuildObjectCommandline(cmd, data->tlsKeySecretProps, qemuCaps) < 0 || - qemuBuildObjectCommandline(cmd, data->tlsProps, qemuCaps) < 0) - return -1; + /* disks that are backed by nbdkit do not send these secrets to qemu, but + * rather directly to nbdkit */ + if (!data->useNbdkit) { + if (qemuBuildObjectCommandline(cmd, data->prmgrProps, qemuCaps) < 0 || + qemuBuildObjectCommandline(cmd, data->authsecretProps, qemuCaps) < 0 || + qemuBuildObjectCommandline(cmd, data->encryptsecretProps, qemuCaps) < 0 || + qemuBuildObjectCommandline(cmd, data->httpcookiesecretProps, qemuCaps) < 0 || + qemuBuildObjectCommandline(cmd, data->tlsKeySecretProps, qemuCaps) < 0 || + qemuBuildObjectCommandline(cmd, data->tlsProps, qemuCaps) < 0) + return -1; + } if (data->driveCmd) virCommandAddArgList(cmd, "-drive", data->driveCmd, NULL); diff --git a/src/qemu/qemu_conf.c b/src/qemu/qemu_conf.c index 3b75cdeb95..22aeab0c49 100644 --- a/src/qemu/qemu_conf.c +++ b/src/qemu/qemu_conf.c @@ -1571,3 +1571,22 @@ qemuGetMemoryBackingPath(virQEMUDriver *driver, *memPath = g_strdup_printf("%s/%s", domainPath, alias); return 0; } + +/* + * qemuGetNbdkitCaps: + * @driver: the qemu driver + * + * Gets the capabilities for Nbdkit for the specified driver. These can be used + * to determine whether a particular disk source can be served by nbdkit or + * not. + * + * Returns: a reference to qemuNbdkitCaps or NULL + */ +qemuNbdkitCaps* +qemuGetNbdkitCaps(virQEMUDriver *driver) +{ + if (!QEMU_IS_NBDKIT_CAPS(driver->nbdkitCaps)) + return NULL; + + return g_object_ref(driver->nbdkitCaps); +} diff --git a/src/qemu/qemu_conf.h b/src/qemu/qemu_conf.h index c40c452f58..f14f9fc4c1 100644 --- a/src/qemu/qemu_conf.h +++ b/src/qemu/qemu_conf.h @@ -36,6 +36,7 @@ #include "virthreadpool.h" #include "locking/lock_manager.h" #include "qemu_capabilities.h" +#include "qemu_nbdkit.h" #include "virclosecallbacks.h" #include "virhostdev.h" #include "virfile.h" @@ -306,6 +307,8 @@ struct _virQEMUDriver { /* Immutable pointer, self-locking APIs */ virHashAtomic *migrationErrors; + + qemuNbdkitCaps *nbdkitCaps; }; virQEMUDriverConfig *virQEMUDriverConfigNew(bool privileged, @@ -359,3 +362,5 @@ int qemuGetMemoryBackingPath(virQEMUDriver *driver, const virDomainDef *def, const char *alias, char **memPath); + +qemuNbdkitCaps * qemuGetNbdkitCaps(virQEMUDriver *driver); diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 9769e3bb92..faeb779b3f 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -21,6 +21,7 @@ #include <config.h> +#include "qemu_conf.h" #include "qemu_domain.h" #include "qemu_alias.h" #include "qemu_block.h" @@ -818,6 +819,7 @@ qemuDomainStorageSourcePrivateDispose(void *obj) g_clear_pointer(&priv->encinfo, qemuDomainSecretInfoFree); g_clear_pointer(&priv->httpcookie, qemuDomainSecretInfoFree); g_clear_pointer(&priv->tlsKeySecret, qemuDomainSecretInfoFree); + g_clear_pointer(&priv->nbdkitProcess, qemuNbdkitProcessFree); } @@ -1293,10 +1295,7 @@ qemuDomainSecretStorageSourcePrepare(qemuDomainObjPrivate *priv, if (!src->auth && !hasEnc && src->ncookies == 0) return 0; - if (!(src->privateData = qemuDomainStorageSourcePrivateNew())) - return -1; - - srcPriv = QEMU_DOMAIN_STORAGE_SOURCE_PRIVATE(src); + srcPriv = qemuDomainStorageSourcePrivateFetch(src); if (src->auth) { virSecretUsageType usageType = VIR_SECRET_USAGE_TYPE_ISCSI; @@ -1321,7 +1320,9 @@ qemuDomainSecretStorageSourcePrepare(qemuDomainObjPrivate *priv, return -1; } - if (src->ncookies && + /* when using nbdkit for http(s) sources, we don't need to pass cookies as + * qemu secrets */ + if (!srcPriv->nbdkitProcess && src->ncookies && virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_BLOCKDEV) && !(srcPriv->httpcookie = qemuDomainSecretStorageSourcePrepareCookies(priv, src, @@ -1792,6 +1793,31 @@ qemuStorageSourcePrivateDataAssignSecinfo(qemuDomainSecretInfo **secinfo, } +static int +qemuStorageSourcePrivateDataParseNbdkit(xmlNodePtr node, + xmlXPathContextPtr ctxt, + virStorageSource *src) +{ + qemuDomainStorageSourcePrivate *srcpriv = qemuDomainStorageSourcePrivateFetch(src); + g_autofree char *pidfile = NULL; + g_autofree char *socketfile = NULL; + VIR_XPATH_NODE_AUTORESTORE(ctxt); + + ctxt->node = node; + + if (!(pidfile = virXPathString("string(./pidfile)", ctxt))) + return -1; + + if (!(socketfile = virXPathString("string(./socketfile)", ctxt))) + return -1; + + if (!srcpriv->nbdkitProcess) + srcpriv->nbdkitProcess = qemuNbdkitProcessLoad(src, pidfile, socketfile); + + return 0; +} + + static int qemuStorageSourcePrivateDataParse(xmlXPathContextPtr ctxt, virStorageSource *src) @@ -1802,6 +1828,7 @@ qemuStorageSourcePrivateDataParse(xmlXPathContextPtr ctxt, g_autofree char *httpcookiealias = NULL; g_autofree char *tlskeyalias = NULL; g_autofree char *thresholdEventWithIndex = NULL; + xmlNodePtr nbdkitnode = NULL; src->nodestorage = virXPathString("string(./nodenames/nodename[@type='storage']/@name)", ctxt); src->nodeformat = virXPathString("string(./nodenames/nodename[@type='format']/@name)", ctxt); @@ -1845,6 +1872,10 @@ qemuStorageSourcePrivateDataParse(xmlXPathContextPtr ctxt, virTristateBoolTypeFromString(thresholdEventWithIndex) == VIR_TRISTATE_BOOL_YES) src->thresholdEventWithIndex = true; + if ((nbdkitnode = virXPathNode("nbdkit", ctxt))) { + if (qemuStorageSourcePrivateDataParseNbdkit(nbdkitnode, ctxt, src) < 0) + return -1; + } return 0; } @@ -1862,6 +1893,23 @@ qemuStorageSourcePrivateDataFormatSecinfo(virBuffer *buf, } +static void +qemuStorageSourcePrivateDataFormatNbdkit(qemuNbdkitProcess *nbdkit, + virBuffer *buf) +{ + g_auto(virBuffer) childBuf = VIR_BUFFER_INIT_CHILD(buf); + + if (!nbdkit) + return; + + virBufferEscapeString(&childBuf, "<pidfile>%s</pidfile>\n", + nbdkit->pidfile); + virBufferEscapeString(&childBuf, "<socketfile>%s</socketfile>\n", + nbdkit->socketfile); + virXMLFormatElement(buf, "nbdkit", NULL, &childBuf); +} + + static int qemuStorageSourcePrivateDataFormat(virStorageSource *src, virBuffer *buf) @@ -1900,6 +1948,9 @@ qemuStorageSourcePrivateDataFormat(virStorageSource *src, if (src->thresholdEventWithIndex) virBufferAddLit(buf, "<thresholdEvent indexUsed='yes'/>\n"); + if (srcPriv) + qemuStorageSourcePrivateDataFormatNbdkit(srcPriv->nbdkitProcess, buf); + return 0; } @@ -4804,6 +4855,7 @@ qemuDomainValidateActualNetDef(const virDomainNetDef *net, int qemuDomainValidateStorageSource(virStorageSource *src, virQEMUCaps *qemuCaps, + qemuNbdkitCaps *nbdkitCaps, bool maskBlockdev) { virStorageType actualType = virStorageSourceGetActualType(src); @@ -4899,7 +4951,8 @@ qemuDomainValidateStorageSource(virStorageSource *src, return -1; } - if (!src->detected && !blockdev) { + if (!src->detected && !blockdev && + !qemuNbdkitCapsGet(nbdkitCaps, QEMU_NBDKIT_CAPS_PLUGIN_CURL)) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", _("http cookies are not supported by this QEMU binary")); return -1; @@ -4920,7 +4973,8 @@ qemuDomainValidateStorageSource(virStorageSource *src, return -1; } - if (!src->detected && !blockdev) { + if (!src->detected && !blockdev && + !qemuNbdkitCapsGet(nbdkitCaps, QEMU_NBDKIT_CAPS_FILTER_READAHEAD)) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", _("readahead setting is not supported with this QEMU binary")); return -1; @@ -4938,7 +4992,8 @@ qemuDomainValidateStorageSource(virStorageSource *src, return -1; } - if (!src->detected && !blockdev) { + if (!src->detected && !blockdev && + !qemuNbdkitCapsGet(nbdkitCaps, QEMU_NBDKIT_CAPS_PLUGIN_CURL)) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", _("timeout setting is not supported with this QEMU binary")); return -1; @@ -7705,6 +7760,7 @@ qemuDomainDetermineDiskChain(virQEMUDriver *driver, bool isSD = qemuDiskBusIsSD(disk->bus); uid_t uid; gid_t gid; + g_autoptr(qemuNbdkitCaps) nbdkitcaps = qemuGetNbdkitCaps(priv->driver); if (!disksrc) disksrc = disk->src; @@ -7787,7 +7843,8 @@ qemuDomainDetermineDiskChain(virQEMUDriver *driver, n->format = VIR_STORAGE_FILE_RAW; /* mask-out blockdev for 'sd' disks */ - if (qemuDomainValidateStorageSource(n, priv->qemuCaps, isSD) < 0) + if (qemuDomainValidateStorageSource(n, priv->qemuCaps, + nbdkitcaps, isSD) < 0) return -1; qemuDomainPrepareStorageSourceConfig(n, cfg, priv->qemuCaps); @@ -10112,6 +10169,29 @@ qemuDomainPrepareStorageSourceNFS(virStorageSource *src) } +/* qemuPrepareStorageSourceNbdkit: + * @src: source for a disk + * + * If src is an network source that is managed by nbdkit, prepare data so that + * nbdkit can be launched before the domain is started + */ +static void +qemuDomainPrepareStorageSourceNbdkit(virDomainDiskDef *disk, + virQEMUDriverConfig *cfg, + qemuDomainObjPrivate *priv) +{ + g_autoptr(qemuNbdkitCaps) nbdkit = qemuGetNbdkitCaps(priv->driver); + if (!nbdkit) + return; + + if (virStorageSourceGetActualType(disk->src) != VIR_STORAGE_TYPE_NETWORK) + return; + + qemuNbdkitInitStorageSource(nbdkit, disk->src, priv->libDir, + disk->info.alias, cfg->user, cfg->group); +} + + /* qemuProcessPrepareStorageSourceTLS: * @source: source for a disk * @cfg: driver configuration @@ -10828,7 +10908,9 @@ qemuDomainPrepareDiskSourceLegacy(virDomainDiskDef *disk, qemuDomainObjPrivate *priv, virQEMUDriverConfig *cfg) { - if (qemuDomainValidateStorageSource(disk->src, priv->qemuCaps, true) < 0) + g_autoptr(qemuNbdkitCaps) nbdkitcaps = qemuGetNbdkitCaps(priv->driver); + if (qemuDomainValidateStorageSource(disk->src, priv->qemuCaps, + nbdkitcaps, true) < 0) return -1; qemuDomainPrepareStorageSourceConfig(disk->src, cfg, priv->qemuCaps); @@ -10846,6 +10928,8 @@ qemuDomainPrepareDiskSourceLegacy(virDomainDiskDef *disk, priv) < 0) return -1; + qemuDomainPrepareStorageSourceNbdkit(disk, cfg, priv); + return 0; } @@ -10857,6 +10941,7 @@ qemuDomainPrepareStorageSourceBlockdevNodename(virDomainDiskDef *disk, qemuDomainObjPrivate *priv, virQEMUDriverConfig *cfg) { + g_autoptr(qemuNbdkitCaps) nbdkitcaps = qemuGetNbdkitCaps(priv->driver); src->nodestorage = g_strdup_printf("%s-storage", nodenameprefix); src->nodeformat = g_strdup_printf("%s-format", nodenameprefix); @@ -10866,7 +10951,8 @@ qemuDomainPrepareStorageSourceBlockdevNodename(virDomainDiskDef *disk, if (src->encryption && src->encryption->engine == VIR_STORAGE_ENCRYPTION_ENGINE_DEFAULT) src->encryption->engine = VIR_STORAGE_ENCRYPTION_ENGINE_QEMU; - if (qemuDomainValidateStorageSource(src, priv->qemuCaps, false) < 0) + if (qemuDomainValidateStorageSource(src, priv->qemuCaps, + nbdkitcaps, false) < 0) return -1; qemuDomainPrepareStorageSourceConfig(src, cfg, priv->qemuCaps); @@ -10887,6 +10973,8 @@ qemuDomainPrepareStorageSourceBlockdevNodename(virDomainDiskDef *disk, if (qemuDomainPrepareStorageSourceNFS(src) < 0) return -1; + qemuDomainPrepareStorageSourceNbdkit(disk, cfg, priv); + return 0; } diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h index a87dfff1bb..34e1bb6fba 100644 --- a/src/qemu/qemu_domain.h +++ b/src/qemu/qemu_domain.h @@ -33,6 +33,7 @@ #include "qemu_conf.h" #include "qemu_capabilities.h" #include "qemu_migration_params.h" +#include "qemu_nbdkit.h" #include "qemu_slirp.h" #include "qemu_fd.h" #include "virchrdev.h" @@ -290,6 +291,9 @@ struct _qemuDomainStorageSourcePrivate { /* key for decrypting TLS certificate */ qemuDomainSecretInfo *tlsKeySecret; + + /* an nbdkit process for serving network storage sources */ + qemuNbdkitProcess *nbdkitProcess; }; virObject *qemuDomainStorageSourcePrivateNew(void); @@ -995,6 +999,7 @@ qemuDomainPrepareDiskSourceData(virDomainDiskDef *disk, int qemuDomainValidateStorageSource(virStorageSource *src, virQEMUCaps *qemuCaps, + qemuNbdkitCaps *nbdkitCaps, bool maskBlockdev); diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 3b5c3db67c..d1a972eb37 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -831,6 +831,9 @@ qemuStateInitialize(bool privileged, defsecmodel))) goto error; + /* find whether nbdkit is available and query its capabilities */ + qemu_driver->nbdkitCaps = qemuNbdkitCapsQuery(); + /* If hugetlbfs is present, then we need to create a sub-directory within * it, since we can't assume the root mount point has permissions that * will let our spawned QEMU instances use it. */ @@ -14471,7 +14474,6 @@ qemuDomainBlockPivot(virQEMUDriver *driver, if (reuse && shallow && virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_BLOCKDEV_SNAPSHOT_ALLOW_WRITE_ONLY) && virStorageSourceHasBacking(disk->mirror)) { - if (!(chainattachdata = qemuBuildStorageSourceChainAttachPrepareBlockdev(disk->mirror->backingStore))) return -1; diff --git a/src/qemu/qemu_extdevice.c b/src/qemu/qemu_extdevice.c index b8e3c1000a..0f9361b294 100644 --- a/src/qemu/qemu_extdevice.c +++ b/src/qemu/qemu_extdevice.c @@ -218,6 +218,14 @@ qemuExtDevicesStart(virQEMUDriver *driver, return -1; } + for (i = 0; i < def->ndisks; i++) { + virDomainDiskDef *disk = def->disks[i]; + qemuDomainStorageSourcePrivate *priv = QEMU_DOMAIN_STORAGE_SOURCE_PRIVATE(disk->src); + if (priv && priv->nbdkitProcess && + qemuNbdkitProcessStart(priv->nbdkitProcess, vm, driver) < 0) + return -1; + } + return 0; } @@ -262,6 +270,14 @@ qemuExtDevicesStop(virQEMUDriver *driver, fs->fsdriver == VIR_DOMAIN_FS_DRIVER_TYPE_VIRTIOFS) qemuVirtioFSStop(driver, vm, fs); } + + for (i = 0; i < def->ndisks; i++) { + virDomainDiskDef *disk = def->disks[i]; + qemuDomainStorageSourcePrivate *priv = QEMU_DOMAIN_STORAGE_SOURCE_PRIVATE(disk->src); + + if (priv && priv->nbdkitProcess) + qemuNbdkitProcessStop(priv->nbdkitProcess); + } } @@ -324,6 +340,15 @@ qemuExtDevicesSetupCgroup(virQEMUDriver *driver, return -1; } + for (i = 0; i < def->ndisks; i++) { + virDomainDiskDef *disk = def->disks[i]; + qemuDomainStorageSourcePrivate *priv = QEMU_DOMAIN_STORAGE_SOURCE_PRIVATE(disk); + + if (priv && priv->nbdkitProcess && + qemuNbdkitProcessSetupCgroup(priv->nbdkitProcess, cgroup) < 0) + return -1; + } + for (i = 0; i < def->nfss; i++) { virDomainFSDef *fs = def->fss[i]; diff --git a/src/qemu/qemu_nbdkit.c b/src/qemu/qemu_nbdkit.c new file mode 100644 index 0000000000..dd8759689c --- /dev/null +++ b/src/qemu/qemu_nbdkit.c @@ -0,0 +1,629 @@ +/* + * qemu_nbdkit.c: helpers for using nbdkit + * + * Copyright (C) 2021 Red Hat, Inc. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library. If not, see + * <http://www.gnu.org/licenses/>. + * + */ + +#include <config.h> +#include <glib.h> + +#include "vircommand.h" +#include "virerror.h" +#include "virlog.h" +#include "virpidfile.h" +#include "virsecureerase.h" +#include "qemu_block.h" +#include "qemu_conf.h" +#include "qemu_domain.h" +#include "qemu_driver.h" +#include "qemu_extdevice.h" +#include "qemu_nbdkit.h" +#include "qemu_security.h" + +#include <fcntl.h> + +#define VIR_FROM_THIS VIR_FROM_NBDKIT + +VIR_LOG_INIT("qemu.nbdkit"); + +struct _qemuNbdkitCaps { + GObject parent; + + char *path; + char *version; + + virBitmap *flags; +}; +G_DEFINE_TYPE(qemuNbdkitCaps, qemu_nbdkit_caps, G_TYPE_OBJECT); + + +static void +qemuNbdkitCheckCommandCap(qemuNbdkitCaps *nbdkit, + virCommand *cmd, + qemuNbdkitCapsFlags cap) +{ + if (virCommandRun(cmd, NULL) != 0) + return; + + VIR_DEBUG("Setting nbdkit capability %i", cap); + ignore_value(virBitmapSetBit(nbdkit->flags, cap)); +} + + +static void +qemuNbdkitQueryFilter(qemuNbdkitCaps *nbdkit, + const char *filter, + qemuNbdkitCapsFlags cap) +{ + g_autoptr(virCommand) cmd = virCommandNewArgList(nbdkit->path, + "--version", + NULL); + + virCommandAddArgPair(cmd, "--filter", filter); + + /* use null plugin to check for filter */ + virCommandAddArg(cmd, "null"); + + qemuNbdkitCheckCommandCap(nbdkit, cmd, cap); +} + + +static void +qemuNbdkitQueryPlugin(qemuNbdkitCaps *nbdkit, + const char *plugin, + qemuNbdkitCapsFlags cap) +{ + g_autoptr(virCommand) cmd = virCommandNewArgList(nbdkit->path, + plugin, + "--version", + NULL); + + qemuNbdkitCheckCommandCap(nbdkit, cmd, cap); +} + + +static void +qemuNbdkitQueryPlugins(qemuNbdkitCaps *nbdkit) +{ + qemuNbdkitQueryPlugin(nbdkit, "curl", QEMU_NBDKIT_CAPS_PLUGIN_CURL); + qemuNbdkitQueryPlugin(nbdkit, "ssh", QEMU_NBDKIT_CAPS_PLUGIN_SSH); +} + + +static void +qemuNbdkitQueryFilters(qemuNbdkitCaps *nbdkit) +{ + qemuNbdkitQueryFilter(nbdkit, "readahead", + QEMU_NBDKIT_CAPS_FILTER_READAHEAD); +} + + +static int +qemuNbdkitQueryVersion(qemuNbdkitCaps *nbdkit) +{ + g_autoptr(virCommand) cmd = virCommandNewArgList(nbdkit->path, + "--version", + NULL); + + virCommandSetOutputBuffer(cmd, &nbdkit->version); + + if (virCommandRun(cmd, NULL) != 0) + return -1; + + VIR_DEBUG("Got nbdkit version %s", nbdkit->version); + return 0; +} + + +static void qemuNbdkitCapsFinalize(GObject *object) +{ + qemuNbdkitCaps *nbdkit = QEMU_NBDKIT_CAPS(object); + + g_clear_pointer(&nbdkit->path, g_free); + g_clear_pointer(&nbdkit->version, g_free); + g_clear_pointer(&nbdkit->flags, virBitmapFree); + + G_OBJECT_CLASS(qemu_nbdkit_caps_parent_class)->finalize(object); +} + + +void qemu_nbdkit_caps_init(qemuNbdkitCaps *caps) +{ + caps->flags = virBitmapNew(QEMU_NBDKIT_CAPS_LAST); + caps->version = NULL; +} + + +static void +qemu_nbdkit_caps_class_init(qemuNbdkitCapsClass *klass) +{ + GObjectClass *obj = G_OBJECT_CLASS(klass); + + obj->finalize = qemuNbdkitCapsFinalize; +} + + +qemuNbdkitCaps * +qemuNbdkitCapsNew(const char *path) +{ + qemuNbdkitCaps *caps = g_object_new(QEMU_TYPE_NBDKIT_CAPS, NULL); + caps->path = g_strdup(path); + + return caps; +} + + +qemuNbdkitCaps * +qemuNbdkitCapsQuery(void) +{ + qemuNbdkitCaps *caps = NULL; + g_autofree char *path = virFindFileInPath("nbdkit"); + + if (!path) + return NULL; + + // make sure it's executable + if (!virFileIsExecutable(path)) { + virReportError(VIR_ERR_INTERNAL_ERROR, _("nbdkit '%s' is not executable"), + path); + return NULL; + } + + VIR_DEBUG("found nbdkit executable '%s'", path); + caps = qemuNbdkitCapsNew(path); + + qemuNbdkitQueryPlugins(caps); + qemuNbdkitQueryFilters(caps); + qemuNbdkitQueryVersion(caps); + + return caps; +} + + +bool +qemuNbdkitCapsGet(qemuNbdkitCaps *nbdkitCaps, qemuNbdkitCapsFlags flag) +{ + return virBitmapIsBitSet(nbdkitCaps->flags, flag); +} + + +void +qemuNbdkitCapsSet(qemuNbdkitCaps *nbdkitCaps, qemuNbdkitCapsFlags flag) +{ + ignore_value(virBitmapSetBit(nbdkitCaps->flags, flag)); +} + + +static int childProcess(int fd, uint8_t *output, size_t outputlen) +{ + if (safewrite(fd, output, outputlen) < 0) { + virReportSystemError(errno, "%s", + _("failed to write to socket for nbdkit")); + return -errno; + } + return 0; +} + + +/* Forks a process to write a password to a socket to the nbdkit process. + * Returns a file descriptor that can be passed to ndkit */ +static int qemuNbdkitForkHandler(uint8_t *output, size_t outputlen) +{ + enum { + PARENT_SOCKET = 0, + CHILD_SOCKET = 1 + }; + int pair[2] = { -1, -1 }; + pid_t pid; + + if (socketpair(AF_UNIX, SOCK_STREAM, 0, pair) < 0) { + virReportSystemError(errno, "%s", + _("failed to create socket for nbdkit")); + return -errno; + } + + pid = virFork(); + + if (pid < 0) + return -errno; + + /* child process */ + if (pid == 0) { + int ret = childProcess(pair[CHILD_SOCKET], output, outputlen); + _exit(ret); + } + + /* parent process */ + VIR_FORCE_CLOSE(pair[CHILD_SOCKET]); + + return pair[PARENT_SOCKET]; +} + + +/* Forks a process to write cookies to a socket to the nbdkit process. + * Returns a file descriptor that can be passed to ndkit */ +static int qemuNbdkitProcessForkCookieHandler(qemuNbdkitProcess *proc) +{ + g_autofree char *cookies = qemuBlockStorageSourceGetCookieString(proc->source); + + if (!cookies) + return 0; + + if ((proc->cookiefd = qemuNbdkitForkHandler((uint8_t*)cookies, strlen(cookies))) < 0) + return -1; + + return 0; +} + + +/* Forks a process to write a password to a socket to the nbdkit process. + * Returns a file descriptor that can be passed to ndkit */ +static int qemuNbdkitProcessForkPasswordHandler(qemuNbdkitProcess *proc) +{ + g_autofree uint8_t *password = NULL; + size_t passwordlen = 0; + g_autoptr(virConnect) conn = virGetConnectSecret(); + + if (!proc->source->auth->username) + return 0; + + if (virSecretGetSecretString(conn, + &proc->source->auth->seclookupdef, + /* FIXME: for some reason auth->authType is always NONE... */ + VIR_SECRET_USAGE_TYPE_ISCSI, + &password, + &passwordlen) < 0) + { + /* FIXME: message */ + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("failed to get auth secret for storage")); + return -1; + } + + if ((proc->authfd = qemuNbdkitForkHandler(password, passwordlen)) < 0) + return -1; + + return 0; +} + + +qemuNbdkitProcess * +qemuNbdkitProcessLoad(virStorageSource *source, + const char *pidfile, + const char *socketfile) +{ + int rc; + qemuNbdkitProcess *nbdkit = g_new0(qemuNbdkitProcess, 1); + + nbdkit->pidfile = g_strdup(pidfile); + nbdkit->socketfile = g_strdup(socketfile); + nbdkit->source = virObjectRef(source); + nbdkit->user = -1; + nbdkit->group = -1; + + if ((rc = virPidFileReadPath(nbdkit->pidfile, &nbdkit->pid)) < 0) + VIR_WARN("Failed to read pidfile %s", nbdkit->pidfile); + + return nbdkit; +} + + +static qemuNbdkitProcess * +qemuNbdkitProcessNew(qemuNbdkitCaps *caps, + virStorageSource *source, + char *statedir, + const char *alias, + uid_t user, + gid_t group + /*, char *selinux_label*/) +{ + qemuNbdkitProcess *proc = g_new0(qemuNbdkitProcess, 1); + g_autofree char *pidfile = g_strdup_printf("nbdkit-%s.pid", alias); + g_autofree char *socketfile = g_strdup_printf("nbdkit-%s.socket", alias); + + proc->caps = g_object_ref(caps); + /* weak reference -- source owns this object, so it will always outlive us */ + proc->source = source; + proc->user = user; + proc->group = group; + proc->pidfile = g_build_filename(statedir, pidfile, NULL); + proc->socketfile = g_build_filename(statedir, socketfile, NULL); + + return proc; +} + + +void +qemuNbdkitInitStorageSource(qemuNbdkitCaps *caps, + virStorageSource *source, + char *statedir, + const char *alias, + uid_t user, + gid_t group + /*, char *selinux_label*/) +{ + /* FIXME: onlytarget ??? */ + qemuDomainStorageSourcePrivate *srcPriv = qemuDomainStorageSourcePrivateFetch(source); + + if (srcPriv->nbdkitProcess) + return; + + switch (source->protocol) { + case VIR_STORAGE_NET_PROTOCOL_HTTP: + case VIR_STORAGE_NET_PROTOCOL_HTTPS: + case VIR_STORAGE_NET_PROTOCOL_FTP: + case VIR_STORAGE_NET_PROTOCOL_FTPS: + case VIR_STORAGE_NET_PROTOCOL_TFTP: + if (!virBitmapIsBitSet(caps->flags, QEMU_NBDKIT_CAPS_PLUGIN_CURL)) + return; + break; + case VIR_STORAGE_NET_PROTOCOL_SSH: + if (!virBitmapIsBitSet(caps->flags, QEMU_NBDKIT_CAPS_PLUGIN_SSH)) + return; + break; + case VIR_STORAGE_NET_PROTOCOL_NONE: + case VIR_STORAGE_NET_PROTOCOL_NBD: + case VIR_STORAGE_NET_PROTOCOL_RBD: + case VIR_STORAGE_NET_PROTOCOL_SHEEPDOG: + case VIR_STORAGE_NET_PROTOCOL_GLUSTER: + case VIR_STORAGE_NET_PROTOCOL_ISCSI: + case VIR_STORAGE_NET_PROTOCOL_VXHS: + case VIR_STORAGE_NET_PROTOCOL_NFS: + case VIR_STORAGE_NET_PROTOCOL_LAST: + return; + } + srcPriv->nbdkitProcess = qemuNbdkitProcessNew(caps, source, statedir, alias, user, group); +} + + +static void +qemuNbdkitProcessBuildCommandCurl(qemuNbdkitProcess *proc, + virCommand *cmd) +{ + g_autoptr(virURI) uri = qemuBlockStorageSourceGetURI(proc->source); + g_autofree char *uristring = virURIFormat(uri); + + /* nbdkit plugin name */ + virCommandAddArg(cmd, "curl"); + virCommandAddArgPair(cmd, "protocols", + virStorageNetProtocolTypeToString(proc->source->protocol)); + virCommandAddArgPair(cmd, "url", uristring); + + if (proc->source->auth) { + virCommandAddArgPair(cmd, "user", + proc->source->auth->username); + } + + // FIXME: if (srcPriv->secinfo)??? + if (proc->authfd > 0) { + /* nbdkit auth parameter accepts a variation where nbdkit + * will read the cookies from a file descriptor: password=-FD */ + g_autofree char *fdfmt = g_strdup_printf("-%i", proc->authfd); + virCommandAddArgPair(cmd, "password", fdfmt); + virCommandPassFD(cmd, proc->authfd, VIR_COMMAND_PASS_FD_CLOSE_PARENT); + } + + if (proc->cookiefd > 0) { + /* nbdkit cookie parameter accepts a variation where nbdkit + * will read the cookies from a file descriptor: cookie=-FD */ + g_autofree char *fdfmt = g_strdup_printf("-%i", proc->cookiefd); + virCommandAddArgPair(cmd, "cookie", fdfmt); + virCommandPassFD(cmd, proc->cookiefd, VIR_COMMAND_PASS_FD_CLOSE_PARENT); + } + + if (proc->source->sslverify == VIR_TRISTATE_BOOL_NO) { + virCommandAddArgPair(cmd, "sslverify", "false"); + } + + if (proc->source->timeout > 0) { + g_autofree char *timeout = g_strdup_printf("%llu", proc->source->timeout); + virCommandAddArgPair(cmd, "timeout", timeout); + } +} + + +static void +qemuNbdkitProcessBuildCommandSSH(qemuNbdkitProcess *proc, + virCommand *cmd) +{ + char *user = NULL; + virStorageNetHostDef *host = &proc->source->hosts[0]; + + /* nbdkit plugin name */ + virCommandAddArg(cmd, "ssh"); + + virCommandAddArgPair(cmd, "host", g_strdup(host->name)); + virCommandAddArgPair(cmd, "port", g_strdup_printf("%u", + host->port)); + virCommandAddArgPair(cmd, "path", g_strdup(proc->source->path)); + + if (proc->source->auth) { + user = g_strdup(proc->source->auth->username); + } else if (proc->source->ssh_user) { + user = g_strdup(proc->source->ssh_user); + } + + if (user) { + virCommandAddArgPair(cmd, "user", user); + } + + if (proc->source->ssh_host_key_check_disabled) { + virCommandAddArgPair(cmd, "verify-remote-host", + g_strdup("false")); + } +} + + +static virCommand * +qemuNbdkitProcessBuildCommand(qemuNbdkitProcess *proc) +{ + g_autoptr(virCommand) cmd = virCommandNewArgList(proc->caps->path, + "--exit-with-parent", + "--unix", + proc->socketfile, + "--foreground", + "--pidfile", + proc->pidfile, + "--verbose", + //"--selinux-label", + //selinux_label, + NULL); + + if (proc->source->readonly) + virCommandAddArg(cmd, "--readonly"); + + if (qemuNbdkitCapsGet(proc->caps, QEMU_NBDKIT_CAPS_FILTER_READAHEAD) && + proc->source->readahead > 0) + virCommandAddArgPair(cmd, "--filter", "readahead"); + + switch (proc->source->protocol) { + case VIR_STORAGE_NET_PROTOCOL_HTTP: + case VIR_STORAGE_NET_PROTOCOL_HTTPS: + case VIR_STORAGE_NET_PROTOCOL_FTP: + case VIR_STORAGE_NET_PROTOCOL_FTPS: + case VIR_STORAGE_NET_PROTOCOL_TFTP: + qemuNbdkitProcessBuildCommandCurl(proc, cmd); + break; + case VIR_STORAGE_NET_PROTOCOL_SSH: + qemuNbdkitProcessBuildCommandSSH(proc, cmd); + break; + + case VIR_STORAGE_NET_PROTOCOL_NONE: + case VIR_STORAGE_NET_PROTOCOL_NBD: + case VIR_STORAGE_NET_PROTOCOL_RBD: + case VIR_STORAGE_NET_PROTOCOL_SHEEPDOG: + case VIR_STORAGE_NET_PROTOCOL_GLUSTER: + case VIR_STORAGE_NET_PROTOCOL_ISCSI: + case VIR_STORAGE_NET_PROTOCOL_VXHS: + case VIR_STORAGE_NET_PROTOCOL_NFS: + case VIR_STORAGE_NET_PROTOCOL_LAST: + virReportError(VIR_ERR_NO_SUPPORT, + _("protocol '%s' is not supported by nbdkit"), + virStorageNetProtocolTypeToString(proc->source->protocol)); + return NULL; + } + + virCommandDaemonize(cmd); + + return g_steal_pointer(&cmd); +} + + +void +qemuNbdkitProcessFree(qemuNbdkitProcess *proc) +{ + if (virProcessKillPainfully(proc->pid, true) < 0) + VIR_WARN("Unable to kill nbdkit process"); + + unlink(proc->pidfile); + g_clear_pointer(&proc->pidfile, g_free); + unlink(proc->socketfile); + g_clear_pointer(&proc->socketfile, g_free); + VIR_FORCE_CLOSE(proc->cookiefd); + VIR_FORCE_CLOSE(proc->authfd); + g_clear_object(&proc->caps); + g_free(proc); +} + + +int +qemuNbdkitProcessSetupCgroup(qemuNbdkitProcess *proc, + virCgroup *cgroup) +{ + return virCgroupAddProcess(cgroup, proc->pid); +} + + +/* FIXME: selinux permissions errors */ +int +qemuNbdkitProcessStart(qemuNbdkitProcess *proc, + virDomainObj *vm, + virQEMUDriver *driver) +{ + g_autoptr(virCommand) cmd = NULL; + int rc; + int exitstatus = 0; + int cmdret = 0; + unsigned int loops = 0; + int errfd = -1; + + if (qemuNbdkitProcessForkCookieHandler(proc) < 0) + return -1; + + if (qemuNbdkitProcessForkPasswordHandler(proc) < 0) + return -1; + + if (!(cmd = qemuNbdkitProcessBuildCommand(proc))) + return -1; + + virCommandSetErrorFD(cmd, &errfd); + + if (qemuExtDeviceLogCommand(driver, vm, cmd, "nbdkit") < 0) + goto error; + + if (qemuSecurityCommandRun(driver, vm, cmd, proc->user, proc->group, &exitstatus, &cmdret) < 0) + goto error; + + if (cmdret < 0 || exitstatus != 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("Could not start 'nbdkit'. exitstatus: %d"), exitstatus); + goto error; + } + + /* Wait for the pid file to appear. The file doesn't appear until nbdkit is + * ready to accept connections to the socket */ + while (!virFileExists(proc->pidfile)) { + VIR_DEBUG("waiting for nbdkit pidfile %s: %u", proc->pidfile, loops); + /* wait for 100ms before checking again, but don't do it for ever */ + if (errno == ENOENT && loops < 10) { + g_usleep(100 * 1000); + loops++; + } else { + char errbuf[1024] = { 0 }; + if (errfd && saferead(errfd, errbuf, sizeof(errbuf) - 1) > 0) { + virReportError(VIR_ERR_OPERATION_FAILED, + _("nbdkit failed to start: %s"), errbuf); + } else { + virReportSystemError(errno, "%s", + _("Gave up waiting for nbdkit to start")); + } + goto error; + } + } + + if ((rc = virPidFileReadPath(proc->pidfile, &proc->pid)) < 0) { + virReportSystemError(-rc, + _("Failed to read pidfile %s"), + proc->pidfile); + goto error; + } + + return 0; + + error: + if (proc->pid) + virProcessKillPainfully(proc->pid, true); + return -1; +} + + +int +qemuNbdkitProcessStop(qemuNbdkitProcess *proc) +{ + return virProcessKillPainfully(proc->pid, true); +} diff --git a/src/qemu/qemu_nbdkit.h b/src/qemu/qemu_nbdkit.h new file mode 100644 index 0000000000..628b5010cc --- /dev/null +++ b/src/qemu/qemu_nbdkit.h @@ -0,0 +1,89 @@ +/* + * qemu_nbdkit.h: helpers for using nbdkit + * + * Copyright (C) 2021 Red Hat, Inc. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library. If not, see + * <http://www.gnu.org/licenses/>. + * + */ + +#pragma once + +#include "internal.h" +#include "virbitmap.h" +#include "vircgroup.h" +#include "vircommand.h" +#include "virstorageobj.h" +#include "viruri.h" + +typedef struct _qemuNbdkitCaps qemuNbdkitCaps; +typedef struct _qemuNbdkitProcess qemuNbdkitProcess; + +typedef enum { + QEMU_NBDKIT_CAPS_PLUGIN_CURL, + QEMU_NBDKIT_CAPS_PLUGIN_SSH, + QEMU_NBDKIT_CAPS_FILTER_READAHEAD, + QEMU_NBDKIT_CAPS_LAST, +} qemuNbdkitCapsFlags; + +qemuNbdkitCaps* qemuNbdkitCapsQuery(void); + +qemuNbdkitCaps* qemuNbdkitCapsNew(const char *path); + +void qemuNbdkitInitStorageSource(qemuNbdkitCaps *nbdkitCaps, + virStorageSource *source, + char *statedir, + const char *alias, + uid_t user, + gid_t group + /*, char *selinux_label*/); + +bool qemuNbdkitCapsGet(qemuNbdkitCaps *nbdkitCaps, qemuNbdkitCapsFlags flag); + +void qemuNbdkitCapsSet(qemuNbdkitCaps *nbdkitCaps, qemuNbdkitCapsFlags flag); + +#define QEMU_TYPE_NBDKIT_CAPS qemu_nbdkit_caps_get_type() +G_DECLARE_FINAL_TYPE(qemuNbdkitCaps, qemu_nbdkit_caps, QEMU, NBDKIT_CAPS, GObject); + +struct _qemuNbdkitProcess { + qemuNbdkitCaps *caps; + virStorageSource *source; + + char *pidfile; + char *socketfile; + int cookiefd; + int authfd; + uid_t user; + gid_t group; + pid_t pid; +}; + +typedef struct _virQEMUDriver virQEMUDriver; + +int qemuNbdkitProcessStart(qemuNbdkitProcess *proc, + virDomainObj *vm, + virQEMUDriver *driver); + +qemuNbdkitProcess * qemuNbdkitProcessLoad(virStorageSource *source, + const char *pidfile, + const char *socketfile); + +int qemuNbdkitProcessSetupCgroup(qemuNbdkitProcess *proc, virCgroup *cgroup); + +int qemuNbdkitProcessStop(qemuNbdkitProcess *proc); + +void qemuNbdkitProcessFree(qemuNbdkitProcess *proc); + +G_DEFINE_AUTOPTR_CLEANUP_FUNC(qemuNbdkitProcess, qemuNbdkitProcessFree); diff --git a/src/qemu/qemu_validate.c b/src/qemu/qemu_validate.c index 39210ba65b..b6cc75b651 100644 --- a/src/qemu/qemu_validate.c +++ b/src/qemu/qemu_validate.c @@ -602,7 +602,8 @@ qemuValidateDomainDefPM(const virDomainDef *def, static int qemuValidateDomainDefNvram(const virDomainDef *def, - virQEMUCaps *qemuCaps) + virQEMUCaps *qemuCaps, + qemuNbdkitCaps *nbdkitCaps) { virStorageSource *src = def->os.loader->nvram; @@ -655,7 +656,7 @@ qemuValidateDomainDefNvram(const virDomainDef *def, return -1; } - if (qemuDomainValidateStorageSource(src, qemuCaps, false) < 0) + if (qemuDomainValidateStorageSource(src, qemuCaps, nbdkitCaps, false) < 0) return -1; return 0; @@ -664,7 +665,8 @@ qemuValidateDomainDefNvram(const virDomainDef *def, static int qemuValidateDomainDefBoot(const virDomainDef *def, - virQEMUCaps *qemuCaps) + virQEMUCaps *qemuCaps, + qemuNbdkitCaps *nbdkitCaps) { if (def->os.loader) { if (def->os.loader->secure == VIR_TRISTATE_BOOL_YES) { @@ -695,7 +697,7 @@ qemuValidateDomainDefBoot(const virDomainDef *def, } } - if (qemuValidateDomainDefNvram(def, qemuCaps) < 0) + if (qemuValidateDomainDefNvram(def, qemuCaps, nbdkitCaps) < 0) return -1; } @@ -1167,6 +1169,7 @@ qemuValidateDomainDef(const virDomainDef *def, virQEMUDriver *driver = opaque; g_autoptr(virQEMUCaps) qemuCapsLocal = NULL; virQEMUCaps *qemuCaps = parseOpaque; + g_autoptr(qemuNbdkitCaps) nbdkitCaps = qemuGetNbdkitCaps(driver); size_t i; if (!qemuCaps) { @@ -1277,7 +1280,7 @@ qemuValidateDomainDef(const virDomainDef *def, if (qemuValidateDomainDefPM(def, qemuCaps) < 0) return -1; - if (qemuValidateDomainDefBoot(def, qemuCaps) < 0) + if (qemuValidateDomainDefBoot(def, qemuCaps, nbdkitCaps) < 0) return -1; if (qemuValidateDomainVCpuTopology(def, qemuCaps) < 0) @@ -3240,7 +3243,8 @@ qemuValidateDomainDeviceDefDiskTransient(const virDomainDiskDef *disk, int qemuValidateDomainDeviceDefDisk(const virDomainDiskDef *disk, const virDomainDef *def, - virQEMUCaps *qemuCaps) + virQEMUCaps *qemuCaps, + qemuNbdkitCaps *nbdkitCaps) { const char *driverName = virDomainDiskGetDriver(disk); bool isSD = qemuDiskBusIsSD(disk->bus); @@ -3287,7 +3291,7 @@ qemuValidateDomainDeviceDefDisk(const virDomainDiskDef *disk, for (n = disk->src; virStorageSourceIsBacking(n); n = n->backingStore) { /* blockdev support is masked out for 'sd' disks */ - if (qemuDomainValidateStorageSource(n, qemuCaps, isSD) < 0) + if (qemuDomainValidateStorageSource(n, qemuCaps, nbdkitCaps, isSD) < 0) return -1; } @@ -5234,6 +5238,7 @@ qemuValidateDomainDeviceDef(const virDomainDeviceDef *dev, virQEMUDriver *driver = opaque; g_autoptr(virQEMUCaps) qemuCapsLocal = NULL; virQEMUCaps *qemuCaps = parseOpaque; + g_autoptr(qemuNbdkitCaps) nbdkitcaps = qemuGetNbdkitCaps(driver); if (!qemuCaps) { if (!(qemuCapsLocal = virQEMUCapsCacheLookup(driver->qemuCapsCache, @@ -5273,7 +5278,8 @@ qemuValidateDomainDeviceDef(const virDomainDeviceDef *dev, return qemuValidateDomainDeviceDefVideo(dev->data.video, qemuCaps); case VIR_DOMAIN_DEVICE_DISK: - return qemuValidateDomainDeviceDefDisk(dev->data.disk, def, qemuCaps); + return qemuValidateDomainDeviceDefDisk(dev->data.disk, def, qemuCaps, + nbdkitcaps); case VIR_DOMAIN_DEVICE_CONTROLLER: return qemuValidateDomainDeviceDefController(dev->data.controller, def, diff --git a/src/qemu/qemu_validate.h b/src/qemu/qemu_validate.h index e06a43b8e3..1498a978ed 100644 --- a/src/qemu/qemu_validate.h +++ b/src/qemu/qemu_validate.h @@ -21,6 +21,7 @@ #pragma once #include "qemu_capabilities.h" +#include "qemu_nbdkit.h" int qemuValidateDomainDef(const virDomainDef *def, @@ -30,7 +31,8 @@ qemuValidateDomainDef(const virDomainDef *def, int qemuValidateDomainDeviceDefDisk(const virDomainDiskDef *disk, const virDomainDef *def, - virQEMUCaps *qemuCaps); + virQEMUCaps *qemuCaps, + qemuNbdkitCaps *nbdkitCaps); int qemuValidateDomainDeviceDef(const virDomainDeviceDef *dev, diff --git a/src/util/virerror.c b/src/util/virerror.c index d114c0a346..5b665931ca 100644 --- a/src/util/virerror.c +++ b/src/util/virerror.c @@ -145,6 +145,7 @@ VIR_ENUM_IMPL(virErrorDomain, "TPM", /* 70 */ "BPF", "Cloud-Hypervisor Driver", + "Nbdkit", ); diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 57116c930b..18dd1f61fe 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -280,7 +280,8 @@ testQemuDiskXMLToProps(const void *opaque) virDomainDiskInsert(vmdef, disk); - if (qemuValidateDomainDeviceDefDisk(disk, vmdef, data->qemuCaps) < 0) { + /* FIXME: test with nbdkit */ + if (qemuValidateDomainDeviceDefDisk(disk, vmdef, data->qemuCaps, NULL) < 0) { VIR_TEST_VERBOSE("invalid configuration for disk"); return -1; } @@ -294,7 +295,8 @@ testQemuDiskXMLToProps(const void *opaque) if (testQemuDiskXMLToJSONFakeSecrets(n) < 0) return -1; - if (qemuDomainValidateStorageSource(n, data->qemuCaps, false) < 0) + /* FIXME: test with nbdkit */ + if (qemuDomainValidateStorageSource(n, data->qemuCaps, NULL, false) < 0) return -1; qemuDomainPrepareDiskSourceData(disk, n); @@ -519,7 +521,7 @@ testQemuImageCreate(const void *opaque) src->capacity = UINT_MAX * 2ULL; src->physical = UINT_MAX + 1ULL; - if (qemuDomainValidateStorageSource(src, data->qemuCaps, false) < 0) + if (qemuDomainValidateStorageSource(src, data->qemuCaps, NULL, false) < 0) return -1; if (qemuBlockStorageSourceCreateGetStorageProps(src, &protocolprops) < 0) diff --git a/tests/qemustatusxml2xmldata/modern-in.xml b/tests/qemustatusxml2xmldata/modern-in.xml index 7759034f7a..49b005cfc7 100644 --- a/tests/qemustatusxml2xmldata/modern-in.xml +++ b/tests/qemustatusxml2xmldata/modern-in.xml @@ -337,7 +337,6 @@ <objects> <secret type='auth' alias='test-auth-alias'/> <secret type='encryption' alias='test-encryption-alias'/> - <secret type='httpcookie' alias='http-cookie-alias'/> <secret type='tlskey' alias='tls-certificate-key-alias'/> <TLSx509 alias='transport-alias'/> </objects> diff --git a/tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.x86_64-latest.args b/tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.x86_64-latest.args new file mode 100644 index 0000000000..c944a3ead4 --- /dev/null +++ b/tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.x86_64-latest.args @@ -0,0 +1,42 @@ +LC_ALL=C \ +PATH=/bin \ +HOME=/tmp/lib/domain--1-QEMUGuest1 \ +USER=test \ +LOGNAME=test \ +XDG_DATA_HOME=/tmp/lib/domain--1-QEMUGuest1/.local/share \ +XDG_CACHE_HOME=/tmp/lib/domain--1-QEMUGuest1/.cache \ +XDG_CONFIG_HOME=/tmp/lib/domain--1-QEMUGuest1/.config \ +/usr/bin/qemu-system-x86_64 \ +-name guest=QEMUGuest1,debug-threads=on \ +-S \ +-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/tmp/lib/domain--1-QEMUGuest1/master-key.aes"}' \ +-machine pc,usb=off,dump-guest-core=off,memory-backend=pc.ram \ +-accel kvm \ +-cpu qemu64 \ +-m 1024 \ +-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":1073741824}' \ +-overcommit mem-lock=off \ +-smp 1,sockets=1,cores=1,threads=1 \ +-uuid c7a5fdbd-edaf-9455-926a-d65c16db1809 \ +-display none \ +-no-user-config \ +-nodefaults \ +-chardev socket,id=charmonitor,fd=1729,server=on,wait=off \ +-mon chardev=charmonitor,id=monitor,mode=control \ +-rtc base=utc \ +-no-shutdown \ +-boot strict=on \ +-device '{"driver":"piix3-usb-uhci","id":"usb","bus":"pci.0","addr":"0x1.0x2"}' \ +-blockdev '{"driver":"nbd","server":{"type":"unix","path":"/tmp/lib/domain--1-QEMUGuest1/nbdkit-ide0-0-0.socket"},"node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}' \ +-blockdev '{"node-name":"libvirt-3-format","read-only":true,"driver":"raw","file":"libvirt-3-storage"}' \ +-device '{"driver":"ide-cd","bus":"ide.0","unit":0,"drive":"libvirt-3-format","id":"ide0-0-0","bootindex":1}' \ +-blockdev '{"driver":"nbd","server":{"type":"unix","path":"/tmp/lib/domain--1-QEMUGuest1/nbdkit-ide0-0-1.socket"},"node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \ +-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \ +-device '{"driver":"ide-cd","bus":"ide.0","unit":1,"drive":"libvirt-2-format","id":"ide0-0-1"}' \ +-blockdev '{"driver":"nbd","server":{"type":"unix","path":"/tmp/lib/domain--1-QEMUGuest1/nbdkit-ide0-1-0.socket"},"node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \ +-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \ +-device '{"driver":"ide-cd","bus":"ide.1","unit":0,"drive":"libvirt-1-format","id":"ide0-1-0"}' \ +-audiodev '{"id":"audio1","driver":"none"}' \ +-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.0","addr":"0x2"}' \ +-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ +-msg timestamp=on diff --git a/tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.xml b/tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.xml new file mode 120000 index 0000000000..55f677546f --- /dev/null +++ b/tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.xml @@ -0,0 +1 @@ +disk-cdrom-network.xml \ No newline at end of file diff --git a/tests/qemuxml2argvdata/disk-network-http-nbdkit.x86_64-latest.args b/tests/qemuxml2argvdata/disk-network-http-nbdkit.x86_64-latest.args new file mode 100644 index 0000000000..378dcc3e62 --- /dev/null +++ b/tests/qemuxml2argvdata/disk-network-http-nbdkit.x86_64-latest.args @@ -0,0 +1,45 @@ +LC_ALL=C \ +PATH=/bin \ +HOME=/tmp/lib/domain--1-QEMUGuest1 \ +USER=test \ +LOGNAME=test \ +XDG_DATA_HOME=/tmp/lib/domain--1-QEMUGuest1/.local/share \ +XDG_CACHE_HOME=/tmp/lib/domain--1-QEMUGuest1/.cache \ +XDG_CONFIG_HOME=/tmp/lib/domain--1-QEMUGuest1/.config \ +/usr/bin/qemu-system-x86_64 \ +-name guest=QEMUGuest1,debug-threads=on \ +-S \ +-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/tmp/lib/domain--1-QEMUGuest1/master-key.aes"}' \ +-machine pc,usb=off,dump-guest-core=off,memory-backend=pc.ram \ +-accel kvm \ +-cpu qemu64 \ +-m 214 \ +-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":224395264}' \ +-overcommit mem-lock=off \ +-smp 1,sockets=1,cores=1,threads=1 \ +-uuid c7a5fdbd-edaf-9455-926a-d65c16db1809 \ +-display none \ +-no-user-config \ +-nodefaults \ +-chardev socket,id=charmonitor,fd=1729,server=on,wait=off \ +-mon chardev=charmonitor,id=monitor,mode=control \ +-rtc base=utc \ +-no-shutdown \ +-no-acpi \ +-boot strict=on \ +-device '{"driver":"piix3-usb-uhci","id":"usb","bus":"pci.0","addr":"0x1.0x2"}' \ +-blockdev '{"driver":"nbd","server":{"type":"unix","path":"/tmp/lib/domain--1-QEMUGuest1/nbdkit-virtio-disk0.socket"},"node-name":"libvirt-4-storage","auto-read-only":true,"discard":"unmap"}' \ +-blockdev '{"node-name":"libvirt-4-format","read-only":false,"driver":"raw","file":"libvirt-4-storage"}' \ +-device '{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x2","drive":"libvirt-4-format","id":"virtio-disk0","bootindex":1}' \ +-blockdev '{"driver":"nbd","server":{"type":"unix","path":"/tmp/lib/domain--1-QEMUGuest1/nbdkit-virtio-disk1.socket"},"node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}' \ +-blockdev '{"node-name":"libvirt-3-format","read-only":false,"driver":"raw","file":"libvirt-3-storage"}' \ +-device '{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x3","drive":"libvirt-3-format","id":"virtio-disk1"}' \ +-blockdev '{"driver":"nbd","server":{"type":"unix","path":"/tmp/lib/domain--1-QEMUGuest1/nbdkit-virtio-disk2.socket"},"node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \ +-blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"raw","file":"libvirt-2-storage"}' \ +-device '{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x4","drive":"libvirt-2-format","id":"virtio-disk2"}' \ +-blockdev '{"driver":"nbd","server":{"type":"unix","path":"/tmp/lib/domain--1-QEMUGuest1/nbdkit-virtio-disk3.socket"},"node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \ +-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"raw","file":"libvirt-1-storage"}' \ +-device '{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x5","drive":"libvirt-1-format","id":"virtio-disk3"}' \ +-audiodev '{"id":"audio1","driver":"none"}' \ +-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ +-msg timestamp=on diff --git a/tests/qemuxml2argvdata/disk-network-http-nbdkit.xml b/tests/qemuxml2argvdata/disk-network-http-nbdkit.xml new file mode 120000 index 0000000000..6a05204e8a --- /dev/null +++ b/tests/qemuxml2argvdata/disk-network-http-nbdkit.xml @@ -0,0 +1 @@ +disk-network-http.xml \ No newline at end of file diff --git a/tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.x86_64-latest.args b/tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.x86_64-latest.args new file mode 100644 index 0000000000..2410d8dfa5 --- /dev/null +++ b/tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.x86_64-latest.args @@ -0,0 +1,49 @@ +LC_ALL=C \ +PATH=/bin \ +HOME=/tmp/lib/domain--1-QEMUGuest1 \ +USER=test \ +LOGNAME=test \ +XDG_DATA_HOME=/tmp/lib/domain--1-QEMUGuest1/.local/share \ +XDG_CACHE_HOME=/tmp/lib/domain--1-QEMUGuest1/.cache \ +XDG_CONFIG_HOME=/tmp/lib/domain--1-QEMUGuest1/.config \ +/usr/bin/qemu-system-x86_64 \ +-name guest=QEMUGuest1,debug-threads=on \ +-S \ +-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/tmp/lib/domain--1-QEMUGuest1/master-key.aes"}' \ +-machine pc,usb=off,dump-guest-core=off,memory-backend=pc.ram \ +-accel tcg \ +-cpu qemu64 \ +-m 214 \ +-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":224395264}' \ +-overcommit mem-lock=off \ +-smp 1,sockets=1,cores=1,threads=1 \ +-uuid c7a5fdbd-edaf-9455-926a-d65c16db1809 \ +-display none \ +-no-user-config \ +-nodefaults \ +-chardev socket,id=charmonitor,fd=1729,server=on,wait=off \ +-mon chardev=charmonitor,id=monitor,mode=control \ +-rtc base=utc \ +-no-shutdown \ +-no-acpi \ +-boot strict=on \ +-device '{"driver":"piix3-usb-uhci","id":"usb","bus":"pci.0","addr":"0x1.0x2"}' \ +-device '{"driver":"ahci","id":"sata0","bus":"pci.0","addr":"0x2"}' \ +-blockdev '{"driver":"nbd","server":{"type":"unix","path":"/tmp/lib/domain--1-QEMUGuest1/nbdkit-virtio-disk0.socket"},"node-name":"libvirt-5-storage","auto-read-only":true,"discard":"unmap"}' \ +-blockdev '{"node-name":"libvirt-5-format","read-only":true,"driver":"raw","file":"libvirt-5-storage"}' \ +-device '{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x3","drive":"libvirt-5-format","id":"virtio-disk0","bootindex":1}' \ +-blockdev '{"driver":"nbd","server":{"type":"unix","path":"/tmp/lib/domain--1-QEMUGuest1/nbdkit-virtio-disk4.socket"},"node-name":"libvirt-4-storage","auto-read-only":true,"discard":"unmap"}' \ +-blockdev '{"node-name":"libvirt-4-format","read-only":false,"driver":"luks","key-secret":"libvirt-4-format-encryption-secret0","file":"libvirt-4-storage"}' \ +-device '{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x4","drive":"libvirt-4-format","id":"virtio-disk4"}' \ +-blockdev '{"driver":"nbd","server":{"type":"unix","path":"/tmp/lib/domain--1-QEMUGuest1/nbdkit-sata0-0-1.socket"},"node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}' \ +-blockdev '{"node-name":"libvirt-3-format","read-only":true,"driver":"raw","file":"libvirt-3-storage"}' \ +-device '{"driver":"ide-cd","bus":"sata0.1","drive":"libvirt-3-format","id":"sata0-0-1"}' \ +-blockdev '{"driver":"nbd","server":{"type":"unix","path":"/tmp/lib/domain--1-QEMUGuest1/nbdkit-sata0-0-2.socket"},"node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \ +-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \ +-device '{"driver":"ide-cd","bus":"sata0.2","drive":"libvirt-2-format","id":"sata0-0-2"}' \ +-blockdev '{"driver":"nbd","server":{"type":"unix","path":"/tmp/lib/domain--1-QEMUGuest1/nbdkit-sata0-0-3.socket"},"node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \ +-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \ +-device '{"driver":"ide-cd","bus":"sata0.3","drive":"libvirt-1-format","id":"sata0-0-3"}' \ +-audiodev '{"id":"audio1","driver":"none"}' \ +-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ +-msg timestamp=on diff --git a/tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.xml b/tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.xml new file mode 120000 index 0000000000..4a1e40bd70 --- /dev/null +++ b/tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.xml @@ -0,0 +1 @@ +disk-network-source-curl.xml \ No newline at end of file diff --git a/tests/qemuxml2argvdata/disk-network-source-curl.x86_64-latest.args b/tests/qemuxml2argvdata/disk-network-source-curl.x86_64-latest.args new file mode 100644 index 0000000000..ec6dd13f6c --- /dev/null +++ b/tests/qemuxml2argvdata/disk-network-source-curl.x86_64-latest.args @@ -0,0 +1,53 @@ +LC_ALL=C \ +PATH=/bin \ +HOME=/tmp/lib/domain--1-QEMUGuest1 \ +USER=test \ +LOGNAME=test \ +XDG_DATA_HOME=/tmp/lib/domain--1-QEMUGuest1/.local/share \ +XDG_CACHE_HOME=/tmp/lib/domain--1-QEMUGuest1/.cache \ +XDG_CONFIG_HOME=/tmp/lib/domain--1-QEMUGuest1/.config \ +/usr/bin/qemu-system-x86_64 \ +-name guest=QEMUGuest1,debug-threads=on \ +-S \ +-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/tmp/lib/domain--1-QEMUGuest1/master-key.aes"}' \ +-machine pc,usb=off,dump-guest-core=off,memory-backend=pc.ram \ +-accel tcg \ +-cpu qemu64 \ +-m 214 \ +-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":224395264}' \ +-overcommit mem-lock=off \ +-smp 1,sockets=1,cores=1,threads=1 \ +-uuid c7a5fdbd-edaf-9455-926a-d65c16db1809 \ +-display none \ +-no-user-config \ +-nodefaults \ +-chardev socket,id=charmonitor,fd=1729,server=on,wait=off \ +-mon chardev=charmonitor,id=monitor,mode=control \ +-rtc base=utc \ +-no-shutdown \ +-no-acpi \ +-boot strict=on \ +-device '{"driver":"piix3-usb-uhci","id":"usb","bus":"pci.0","addr":"0x1.0x2"}' \ +-device '{"driver":"ahci","id":"sata0","bus":"pci.0","addr":"0x2"}' \ +-object '{"qom-type":"secret","id":"libvirt-5-storage-httpcookie-secret0","data":"BUU0KmnWfonHdjzhYhwVQZ5iTI1KweTJ22q8XWUVoBCVu1z70reDuczPBIabZtC3","keyid":"masterKey0","iv":"AAECAwQFBgcICQoLDA0ODw==","format":"base64"}' \ +-blockdev '{"driver":"https","url":"https://https.example.org:8443/path/to/disk1.iso","cookie-secret":"libvirt-5-storage-httpcookie-secret0","node-name":"libvirt-5-storage","auto-read-only":true,"discard":"unmap"}' \ +-blockdev '{"node-name":"libvirt-5-format","read-only":true,"driver":"raw","file":"libvirt-5-storage"}' \ +-device '{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x3","drive":"libvirt-5-format","id":"virtio-disk0","bootindex":1}' \ +-object '{"qom-type":"secret","id":"libvirt-4-format-encryption-secret0","data":"9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1","keyid":"masterKey0","iv":"AAECAwQFBgcICQoLDA0ODw==","format":"base64"}' \ +-object '{"qom-type":"secret","id":"libvirt-4-storage-httpcookie-secret0","data":"BUU0KmnWfonHdjzhYhwVQZ5iTI1KweTJ22q8XWUVoBCVu1z70reDuczPBIabZtC3","keyid":"masterKey0","iv":"AAECAwQFBgcICQoLDA0ODw==","format":"base64"}' \ +-blockdev '{"driver":"https","url":"https://https.example.org:8443/path/to/disk5.iso?foo=bar","sslverify":false,"cookie-secret":"libvirt-4-storage-httpcookie-secret0","node-name":"libvirt-4-storage","auto-read-only":true,"discard":"unmap"}' \ +-blockdev '{"node-name":"libvirt-4-format","read-only":false,"driver":"luks","key-secret":"libvirt-4-format-encryption-secret0","file":"libvirt-4-storage"}' \ +-device '{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x4","drive":"libvirt-4-format","id":"virtio-disk4"}' \ +-object '{"qom-type":"secret","id":"libvirt-3-storage-httpcookie-secret0","data":"BUU0KmnWfonHdjzhYhwVQZ5iTI1KweTJ22q8XWUVoBBv7TuTgTkyAyOPpC2P5qLbOIypLoHpppjz+u5O+X8oT+jA1m7q/OJQ8dk2EFD5c0A=","keyid":"masterKey0","iv":"AAECAwQFBgcICQoLDA0ODw==","format":"base64"}' \ +-blockdev '{"driver":"http","url":"http://http.example.org:8080/path/to/disk2.iso","cookie-secret":"libvirt-3-storage-httpcookie-secret0","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}' \ +-blockdev '{"node-name":"libvirt-3-format","read-only":true,"driver":"raw","file":"libvirt-3-storage"}' \ +-device '{"driver":"ide-cd","bus":"sata0.1","drive":"libvirt-3-format","id":"sata0-0-1"}' \ +-blockdev '{"driver":"ftp","url":"ftp://ftp.example.org:20/path/to/disk3.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \ +-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \ +-device '{"driver":"ide-cd","bus":"sata0.2","drive":"libvirt-2-format","id":"sata0-0-2"}' \ +-blockdev '{"driver":"ftps","url":"ftps://ftps.example.org:22/path/to/disk4.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \ +-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \ +-device '{"driver":"ide-cd","bus":"sata0.3","drive":"libvirt-1-format","id":"sata0-0-3"}' \ +-audiodev '{"id":"audio1","driver":"none"}' \ +-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ +-msg timestamp=on diff --git a/tests/qemuxml2argvdata/disk-network-source-curl.xml b/tests/qemuxml2argvdata/disk-network-source-curl.xml new file mode 100644 index 0000000000..1e50314abe --- /dev/null +++ b/tests/qemuxml2argvdata/disk-network-source-curl.xml @@ -0,0 +1,71 @@ +<domain type='qemu'> + <name>QEMUGuest1</name> + <uuid>c7a5fdbd-edaf-9455-926a-d65c16db1809</uuid> + <memory unit='KiB'>219136</memory> + <currentMemory unit='KiB'>219136</currentMemory> + <vcpu placement='static'>1</vcpu> + <os> + <type arch='x86_64' machine='pc'>hvm</type> + <boot dev='hd'/> + </os> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <emulator>/usr/bin/qemu-system-x86_64</emulator> + <disk type='network' device='disk'> + <source protocol='https' name='path/to/disk1.iso'> + <host name='https.example.org' port='8443'/> + <cookies> + <cookie name='cookie1'>cookievalue1</cookie> + <cookie name='cookie2'>cookievalue2</cookie> + </cookies> + </source> + <target dev='vda' bus='virtio'/> + <readonly/> + </disk> + <disk type='network' device='cdrom'> + <source protocol='http' name='path/to/disk2.iso'> + <host name='http.example.org' port='8080'/> + <cookies> + <cookie name='cookie1'>cookievalue1</cookie> + <cookie name='cookie2'>cookievalue2</cookie> + <cookie name='cookie3'>cookievalue3</cookie> + </cookies> + </source> + <target dev='hdb' bus='sata'/> + </disk> + <disk type='network' device='cdrom'> + <source protocol='ftp' name='path/to/disk3.iso'> + <host name='ftp.example.org' port='20'/> + </source> + <target dev='hdc' bus='sata'/> + </disk> + <disk type='network' device='cdrom'> + <source protocol='ftps' name='path/to/disk4.iso'> + <host name='ftps.example.org' port='22'/> + </source> + <target dev='hdd' bus='sata'/> + </disk> + <disk type='network' device='disk'> + <source protocol='https' name='path/to/disk5.iso' query='foo=bar'> + <host name='https.example.org' port='8443'/> + <ssl verify='no'/> + <cookies> + <cookie name='cookie1'>cookievalue1</cookie> + <cookie name='cookie2'>cookievalue2</cookie> + </cookies> + <encryption format='luks'> + <secret type='passphrase' uuid='1148b693-0843-4cef-9f97-8feb4e1ae365'/> + </encryption> + </source> + <target dev='vde' bus='virtio'/> + </disk> + <controller type='usb' index='0'/> + <controller type='pci' index='0' model='pci-root'/> + <input type='mouse' bus='ps2'/> + <input type='keyboard' bus='ps2'/> + <memballoon model='none'/> + </devices> +</domain> diff --git a/tests/qemuxml2argvtest.c b/tests/qemuxml2argvtest.c index 48dd20458e..379ea307f8 100644 --- a/tests/qemuxml2argvtest.c +++ b/tests/qemuxml2argvtest.c @@ -705,6 +705,11 @@ testCompareXMLToArgv(const void *data) if (rc < 0) goto cleanup; + if (info->nbdkitCaps) { + driver.nbdkitCaps = info->nbdkitCaps; + g_object_add_weak_pointer(G_OBJECT(info->nbdkitCaps), (void**)&driver.nbdkitCaps); + } + if (info->migrateFrom && !(migrateURI = qemuMigrationDstGetURI(info->migrateFrom, info->migrateFd))) @@ -963,6 +968,9 @@ mymain(void) # define DO_TEST_CAPS_ARCH_VER(name, arch, ver) \ DO_TEST_CAPS_ARCH_VER_FULL(name, arch, ver, ARG_END) +# define DO_TEST_CAPS_LATEST_NBDKIT(name, ...) \ + DO_TEST_CAPS_ARCH_LATEST_FULL(name, "x86_64", ARG_NBDKIT_CAPS, __VA_ARGS__, QEMU_NBDKIT_CAPS_LAST, ARG_END) + # define DO_TEST_CAPS_LATEST(name) \ DO_TEST_CAPS_ARCH_LATEST(name, "x86_64") @@ -1330,6 +1338,7 @@ mymain(void) DO_TEST_CAPS_LATEST("disk-cdrom-bus-other"); DO_TEST_CAPS_VER("disk-cdrom-network", "4.1.0"); DO_TEST_CAPS_LATEST("disk-cdrom-network"); + DO_TEST_CAPS_LATEST_NBDKIT("disk-cdrom-network-nbdkit", QEMU_NBDKIT_CAPS_PLUGIN_CURL); DO_TEST_CAPS_VER("disk-cdrom-tray", "4.1.0"); DO_TEST_CAPS_LATEST("disk-cdrom-tray"); DO_TEST_CAPS_VER("disk-floppy", "4.1.0"); @@ -1390,6 +1399,8 @@ mymain(void) DO_TEST_CAPS_VER("disk-network-sheepdog", "6.0.0"); DO_TEST_CAPS_VER("disk-network-source-auth", "4.1.0"); DO_TEST_CAPS_LATEST("disk-network-source-auth"); + DO_TEST_CAPS_LATEST("disk-network-source-curl"); + DO_TEST_CAPS_LATEST_NBDKIT("disk-network-source-curl-nbdkit", QEMU_NBDKIT_CAPS_PLUGIN_CURL); DO_TEST_CAPS_LATEST("disk-network-nfs"); driver.config->vxhsTLS = 1; driver.config->nbdTLSx509secretUUID = g_strdup("6fd3f62d-9fe7-4a4e-a869-7acd6376d8ea"); @@ -1402,6 +1413,7 @@ mymain(void) DO_TEST_CAPS_LATEST("disk-network-tlsx509-nbd-hostname"); DO_TEST_CAPS_VER("disk-network-tlsx509-vxhs", "5.0.0"); DO_TEST_CAPS_LATEST("disk-network-http"); + DO_TEST_CAPS_LATEST_NBDKIT("disk-network-http-nbdkit", QEMU_NBDKIT_CAPS_PLUGIN_CURL); driver.config->vxhsTLS = 0; VIR_FREE(driver.config->vxhsTLSx509certdir); DO_TEST_CAPS_LATEST("disk-no-boot"); diff --git a/tests/testutilsqemu.c b/tests/testutilsqemu.c index 6dabbaf36a..65b403c991 100644 --- a/tests/testutilsqemu.c +++ b/tests/testutilsqemu.c @@ -395,6 +395,7 @@ void qemuTestDriverFree(virQEMUDriver *driver) virObjectUnref(driver->caps); virObjectUnref(driver->config); virObjectUnref(driver->securityManager); + g_clear_object(&driver->nbdkitCaps); virCPUDefFree(cpuDefault); virCPUDefFree(cpuHaswell); @@ -632,6 +633,8 @@ int qemuTestDriverInit(virQEMUDriver *driver) if (!driver->caps) goto error; + driver->nbdkitCaps = NULL; + /* Using /dev/null for libDir and cacheDir automatically produces errors * upon attempt to use any of them */ driver->qemuCapsCache = virQEMUCapsCacheNew("/dev/null", "/dev/null", 0, 0); @@ -858,6 +861,7 @@ testQemuInfoSetArgs(struct testQemuInfo *info, info->conf = conf; info->args.newargs = true; + info->args.fakeNbdkitCaps = qemuNbdkitCapsNew("/bin/true"); va_start(argptr, conf); while ((argname = va_arg(argptr, testQemuInfoArgName)) != ARG_END) { @@ -869,6 +873,13 @@ testQemuInfoSetArgs(struct testQemuInfo *info, virQEMUCapsSet(info->args.fakeCaps, flag); break; + case ARG_NBDKIT_CAPS: + info->args.fakeNbdkitCapsUsed = true; + + while ((flag = va_arg(argptr, int)) < QEMU_NBDKIT_CAPS_LAST) + qemuNbdkitCapsSet(info->args.fakeNbdkitCaps, flag); + break; + case ARG_GIC: info->args.gic = va_arg(argptr, int); break; @@ -993,6 +1004,9 @@ testQemuInfoInitArgs(struct testQemuInfo *info) info->qemuCaps = g_steal_pointer(&info->args.fakeCaps); } + if (info->args.fakeNbdkitCapsUsed) + info->nbdkitCaps = g_steal_pointer(&info->args.fakeNbdkitCaps); + if (info->args.gic != GIC_NONE && testQemuCapsSetGIC(info->qemuCaps, info->args.gic) < 0) return -1; @@ -1010,6 +1024,8 @@ testQemuInfoClear(struct testQemuInfo *info) VIR_FREE(info->errfile); virObjectUnref(info->qemuCaps); g_clear_pointer(&info->args.fakeCaps, virObjectUnref); + g_clear_object(&info->nbdkitCaps); + g_clear_object(&info->args.fakeNbdkitCaps); } diff --git a/tests/testutilsqemu.h b/tests/testutilsqemu.h index 7ce4c4ad8d..7d407004b4 100644 --- a/tests/testutilsqemu.h +++ b/tests/testutilsqemu.h @@ -49,6 +49,7 @@ typedef enum { ARG_CAPS_VER, ARG_CAPS_HOST_CPU_MODEL, ARG_HOST_OS, + ARG_NBDKIT_CAPS, ARG_END, } testQemuInfoArgName; @@ -79,6 +80,8 @@ struct testQemuArgs { bool newargs; virQEMUCaps *fakeCaps; bool fakeCapsUsed; + qemuNbdkitCaps *fakeNbdkitCaps; + bool fakeNbdkitCapsUsed; char *capsver; char *capsarch; qemuTestCPUDef capsHostCPUModel; @@ -93,6 +96,7 @@ struct testQemuInfo { char *outfile; char *errfile; virQEMUCaps *qemuCaps; + qemuNbdkitCaps *nbdkitCaps; const char *migrateFrom; int migrateFd; unsigned int flags; -- 2.35.3

On Wed, Jun 22, 2022 at 16:26:26 -0500, Jonathon Jongsma wrote:
--- include/libvirt/virterror.h | 1 + po/POTFILES | 1 + src/qemu/meson.build | 1 + src/qemu/qemu_block.c | 64 +- src/qemu/qemu_block.h | 1 + src/qemu/qemu_command.c | 26 +- src/qemu/qemu_conf.c | 19 + src/qemu/qemu_conf.h | 5 + src/qemu/qemu_domain.c | 110 ++- src/qemu/qemu_domain.h | 5 + src/qemu/qemu_driver.c | 4 +- src/qemu/qemu_extdevice.c | 25 + src/qemu/qemu_nbdkit.c | 629 ++++++++++++++++++ src/qemu/qemu_nbdkit.h | 89 +++ src/qemu/qemu_validate.c | 22 +- src/qemu/qemu_validate.h | 4 +- src/util/virerror.c | 1 + tests/qemublocktest.c | 8 +- tests/qemustatusxml2xmldata/modern-in.xml | 1 - ...sk-cdrom-network-nbdkit.x86_64-latest.args | 42 ++ .../disk-cdrom-network-nbdkit.xml | 1 + ...isk-network-http-nbdkit.x86_64-latest.args | 45 ++ .../disk-network-http-nbdkit.xml | 1 + ...work-source-curl-nbdkit.x86_64-latest.args | 49 ++ .../disk-network-source-curl-nbdkit.xml | 1 + ...isk-network-source-curl.x86_64-latest.args | 53 ++ .../disk-network-source-curl.xml | 71 ++ tests/qemuxml2argvtest.c | 12 + tests/testutilsqemu.c | 16 + tests/testutilsqemu.h | 4 + 30 files changed, 1278 insertions(+), 33 deletions(-) create mode 100644 src/qemu/qemu_nbdkit.c create mode 100644 src/qemu/qemu_nbdkit.h create mode 100644 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.x86_64-latest.args create mode 120000 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.xml create mode 100644 tests/qemuxml2argvdata/disk-network-http-nbdkit.x86_64-latest.args create mode 120000 tests/qemuxml2argvdata/disk-network-http-nbdkit.xml create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.x86_64-latest.args create mode 120000 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.xml create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.x86_64-latest.args create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.xml
I presume you plan to split this into commits.
diff --git a/include/libvirt/virterror.h b/include/libvirt/virterror.h index df13e4f11e..dd198dfd7d 100644 --- a/include/libvirt/virterror.h +++ b/include/libvirt/virterror.h @@ -141,6 +141,7 @@ typedef enum { VIR_FROM_TPM = 70, /* Error from TPM (Since: 5.6.0) */ VIR_FROM_BPF = 71, /* Error from BPF code (Since: 5.10.0) */ VIR_FROM_CH = 72, /* Error from Cloud-Hypervisor driver (Since: 7.5.0) */ + VIR_FROM_NBDKIT = 73, /* Error from Nbdkit code (Since: 8.5.0) */
# ifdef VIR_ENUM_SENTINELS VIR_ERR_DOMAIN_LAST /* (Since: 0.9.13) */ diff --git a/po/POTFILES b/po/POTFILES index faaba53c8f..99284b8173 100644 --- a/po/POTFILES +++ b/po/POTFILES @@ -177,6 +177,7 @@ src/qemu/qemu_monitor.c src/qemu/qemu_monitor_json.c src/qemu/qemu_monitor_text.c src/qemu/qemu_namespace.c +src/qemu/qemu_nbdkit.c src/qemu/qemu_process.c src/qemu/qemu_qapi.c src/qemu/qemu_saveimage.c diff --git a/src/qemu/meson.build b/src/qemu/meson.build index 96952cc52d..101cf3591f 100644 --- a/src/qemu/meson.build +++ b/src/qemu/meson.build @@ -28,6 +28,7 @@ qemu_driver_sources = [ 'qemu_monitor_json.c', 'qemu_monitor_text.c', 'qemu_namespace.c', + 'qemu_nbdkit.c', 'qemu_process.c', 'qemu_qapi.c', 'qemu_saveimage.c', diff --git a/src/qemu/qemu_block.c b/src/qemu/qemu_block.c index 9fe22f18f2..91f17f133f 100644 --- a/src/qemu/qemu_block.c +++ b/src/qemu/qemu_block.c @@ -773,6 +773,35 @@ qemuBlockStorageSourceGetCURLProps(virStorageSource *src, }
+static virJSONValue * +qemuBlockStorageSourceGetNbdkitProps(virStorageSource *src) +{ + qemuDomainStorageSourcePrivate *srcPriv = QEMU_DOMAIN_STORAGE_SOURCE_PRIVATE(src); + virJSONValue *ret = NULL; + g_autoptr(virJSONValue) serverprops = NULL; + virStorageNetHostDef host = { NULL }; + + /* srcPriv->nbdkitProcess will already be initialized if we can use nbdkit + * to proxy this storage source */ + if (!(srcPriv && srcPriv->nbdkitProcess)) + return NULL; + + host.transport = VIR_STORAGE_NET_HOST_TRANS_UNIX; + host.socket = srcPriv->nbdkitProcess->socketfile; + serverprops = qemuBlockStorageSourceBuildJSONSocketAddress(&host, + false); + if (!serverprops) + return NULL; + + if (virJSONValueObjectAdd(&ret, + "a:server", &serverprops, + NULL) < 0) + return NULL; + + return ret; +} + + static virJSONValue * qemuBlockStorageSourceGetISCSIProps(virStorageSource *src, bool onlytarget) @@ -1207,6 +1236,14 @@ qemuBlockStorageSourceGetBackendProps(virStorageSource *src, case VIR_STORAGE_NET_PROTOCOL_FTP: case VIR_STORAGE_NET_PROTOCOL_FTPS: case VIR_STORAGE_NET_PROTOCOL_TFTP: + /* first try to use nbdkit for http/ftp sources */ + if ((fileprops = qemuBlockStorageSourceGetNbdkitProps(src))) { + driver = "nbd"; + break; + } + + /* nbdkit is not supported for this host/source, fall back to old + * qemu storage plugins */ driver = virStorageNetProtocolTypeToString(src->protocol); if (!(fileprops = qemuBlockStorageSourceGetCURLProps(src, onlytarget))) return NULL; @@ -1237,6 +1274,14 @@ qemuBlockStorageSourceGetBackendProps(virStorageSource *src, break;
case VIR_STORAGE_NET_PROTOCOL_SSH: + /* first try to use nbdkit for ssh sources */ + if ((fileprops = qemuBlockStorageSourceGetNbdkitProps(src))) { + driver = "nbd"; + break; + } + + /* nbdkit is not supported for this host/source. fallback to + * the old qemu storage plugins */ driver = "ssh"; if (!(fileprops = qemuBlockStorageSourceGetSshProps(src))) return NULL;
I think you can do this as the very first thing. If the nbdkit data is set we want to use it for any protocol it was instantiated rather than have some more extra redundant logic here. [...]
@@ -1701,6 +1748,10 @@ static int qemuBlockStorageSourceAttachApplyStorageDeps(qemuMonitor *mon, qemuBlockStorageSourceAttachData *data) { + /* when using nbdkit, data is not passed via qemu secrets */ + if (data->useNbdkit) + return 0;
This feels like a hack to be honest. If the encryption/authentication objects are not needed they should not be instantiated in the first place, rather than skipping formatting them here.
+ if (data->prmgrProps && qemuMonitorAddObject(mon, &data->prmgrProps, &data->prmgrAlias) < 0) return -1; @@ -2205,6 +2256,8 @@ qemuBlockGetBackingStoreString(virStorageSource *src, virJSONValue *props = NULL; g_autoptr(virURI) uri = NULL; g_autofree char *backingJSON = NULL; + qemuDomainStorageSourcePrivate *srcPriv = QEMU_DOMAIN_STORAGE_SOURCE_PRIVATE(src); + bool useNbdkit = srcPriv && srcPriv->nbdkitProcess;
if (!src->sliceStorage) { if (virStorageSourceIsLocalStorage(src)) { @@ -2223,7 +2276,8 @@ qemuBlockGetBackingStoreString(virStorageSource *src, src->ncookies == 0 && src->sslverify == VIR_TRISTATE_BOOL_ABSENT && src->timeout == 0 && - src->readahead == 0) { + src->readahead == 0 && + !useNbdkit) {
switch ((virStorageNetProtocol) src->protocol) { case VIR_STORAGE_NET_PROTOCOL_NBD:
[...]
@@ -2630,6 +2685,13 @@ qemuBlockStorageSourceCreateGetStorageProps(virStorageSource *src, break;
case VIR_STORAGE_NET_PROTOCOL_SSH: + if (srcPriv->nbdkitProcess) { + /* disk creation not yet supported with nbdkit, and even if it + * was supported, it would not be done with blockdev-create + * props */ + return 0; + }
So this introduces a regression in functionality ...
+ driver = "ssh"; if (!(location = qemuBlockStorageSourceGetSshProps(src))) return -1;
.. as previously we'd be able to create the image.
diff --git a/src/qemu/qemu_block.h b/src/qemu/qemu_block.h index 8641c8a2d2..a1f98f4452 100644 --- a/src/qemu/qemu_block.h +++ b/src/qemu/qemu_block.h @@ -113,6 +113,7 @@ struct qemuBlockStorageSourceAttachData { char *tlsAlias; virJSONValue *tlsKeySecretProps; char *tlsKeySecretAlias; + bool useNbdkit; };
diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index b307d3139c..dd410362fb 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -1578,6 +1578,7 @@ qemuBuildNetworkDriveStr(virStorageSource *src, qemuDomainSecretInfo *secinfo) { g_auto(virBuffer) buf = VIR_BUFFER_INITIALIZER; + qemuDomainStorageSourcePrivate *priv = QEMU_DOMAIN_STORAGE_SOURCE_PRIVATE(src); size_t i; char *ret = NULL;
@@ -1637,6 +1638,13 @@ qemuBuildNetworkDriveStr(virStorageSource *src, case VIR_STORAGE_NET_PROTOCOL_FTP: case VIR_STORAGE_NET_PROTOCOL_FTPS: case VIR_STORAGE_NET_PROTOCOL_TFTP: + if (priv && priv->nbdkitProcess) { + virBufferAsprintf(&buf, "nbd:unix:%s", priv->nbdkitProcess->socketfile); + ret = virBufferContentAndReset(&buf); + } else { + ret = qemuBuildNetworkDriveURI(src); + } + break; case VIR_STORAGE_NET_PROTOCOL_ISCSI: case VIR_STORAGE_NET_PROTOCOL_GLUSTER: ret = qemuBuildNetworkDriveURI(src);
Please don't add the functionality to this pre-blockdev helper. It will be deleted soon anyways. Support for the nbdkit stuff should be only allowed in blockdev mode.
@@ -2497,13 +2505,17 @@ qemuBuildBlockStorageSourceAttachDataCommandline(virCommand *cmd, { char *tmp;
- if (qemuBuildObjectCommandline(cmd, data->prmgrProps, qemuCaps) < 0 || - qemuBuildObjectCommandline(cmd, data->authsecretProps, qemuCaps) < 0 || - qemuBuildObjectCommandline(cmd, data->encryptsecretProps, qemuCaps) < 0 || - qemuBuildObjectCommandline(cmd, data->httpcookiesecretProps, qemuCaps) < 0 || - qemuBuildObjectCommandline(cmd, data->tlsKeySecretProps, qemuCaps) < 0 || - qemuBuildObjectCommandline(cmd, data->tlsProps, qemuCaps) < 0) - return -1; + /* disks that are backed by nbdkit do not send these secrets to qemu, but + * rather directly to nbdkit */ + if (!data->useNbdkit) {
NACK to this hack, modify the code ...
+ if (qemuBuildObjectCommandline(cmd, data->prmgrProps, qemuCaps) < 0 || + qemuBuildObjectCommandline(cmd, data->authsecretProps, qemuCaps) < 0 || + qemuBuildObjectCommandline(cmd, data->encryptsecretProps, qemuCaps) < 0 || + qemuBuildObjectCommandline(cmd, data->httpcookiesecretProps, qemuCaps) < 0 || + qemuBuildObjectCommandline(cmd, data->tlsKeySecretProps, qemuCaps) < 0 || + qemuBuildObjectCommandline(cmd, data->tlsProps, qemuCaps) < 0)
So that these are not instantiated rather than hacking them out.
+ return -1;
[...]
diff --git a/src/qemu/qemu_conf.c b/src/qemu/qemu_conf.c index 3b75cdeb95..22aeab0c49 100644 --- a/src/qemu/qemu_conf.c +++ b/src/qemu/qemu_conf.c @@ -1571,3 +1571,22 @@ qemuGetMemoryBackingPath(virQEMUDriver *driver, *memPath = g_strdup_printf("%s/%s", domainPath, alias); return 0; } + +/* + * qemuGetNbdkitCaps: + * @driver: the qemu driver + * + * Gets the capabilities for Nbdkit for the specified driver. These can be used + * to determine whether a particular disk source can be served by nbdkit or + * not. + * + * Returns: a reference to qemuNbdkitCaps or NULL + */ +qemuNbdkitCaps* +qemuGetNbdkitCaps(virQEMUDriver *driver) +{ + if (!QEMU_IS_NBDKIT_CAPS(driver->nbdkitCaps)) + return NULL; + + return g_object_ref(driver->nbdkitCaps); +}
Since the _virQEMUDriverConfig struct is not confined to qemu_conf.c you don't need to have this helper here.
diff --git a/src/qemu/qemu_conf.h b/src/qemu/qemu_conf.h index c40c452f58..f14f9fc4c1 100644 --- a/src/qemu/qemu_conf.h +++ b/src/qemu/qemu_conf.h @@ -36,6 +36,7 @@ #include "virthreadpool.h" #include "locking/lock_manager.h" #include "qemu_capabilities.h" +#include "qemu_nbdkit.h" #include "virclosecallbacks.h" #include "virhostdev.h" #include "virfile.h" @@ -306,6 +307,8 @@ struct _virQEMUDriver {
/* Immutable pointer, self-locking APIs */ virHashAtomic *migrationErrors; + + qemuNbdkitCaps *nbdkitCaps; };
virQEMUDriverConfig *virQEMUDriverConfigNew(bool privileged, @@ -359,3 +362,5 @@ int qemuGetMemoryBackingPath(virQEMUDriver *driver, const virDomainDef *def, const char *alias, char **memPath); + +qemuNbdkitCaps * qemuGetNbdkitCaps(virQEMUDriver *driver); diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 9769e3bb92..faeb779b3f 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c
[...]
@@ -1293,10 +1295,7 @@ qemuDomainSecretStorageSourcePrepare(qemuDomainObjPrivate *priv, if (!src->auth && !hasEnc && src->ncookies == 0) return 0;
- if (!(src->privateData = qemuDomainStorageSourcePrivateNew())) - return -1; - - srcPriv = QEMU_DOMAIN_STORAGE_SOURCE_PRIVATE(src); + srcPriv = qemuDomainStorageSourcePrivateFetch(src);
Extract this into a separate patch.
if (src->auth) { virSecretUsageType usageType = VIR_SECRET_USAGE_TYPE_ISCSI; @@ -1321,7 +1320,9 @@ qemuDomainSecretStorageSourcePrepare(qemuDomainObjPrivate *priv, return -1; }
- if (src->ncookies && + /* when using nbdkit for http(s) sources, we don't need to pass cookies as + * qemu secrets */ + if (!srcPriv->nbdkitProcess && src->ncookies && virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_BLOCKDEV) && !(srcPriv->httpcookie = qemuDomainSecretStorageSourcePrepareCookies(priv, src,
Skip setup of all the objects which are not needed when nbdkit is un use including auth, so that the code formatting the commandline doesn't need to be special-cased. [...]
@@ -4899,7 +4951,8 @@ qemuDomainValidateStorageSource(virStorageSource *src, return -1; }
- if (!src->detected && !blockdev) { + if (!src->detected && !blockdev && + !qemuNbdkitCapsGet(nbdkitCaps, QEMU_NBDKIT_CAPS_PLUGIN_CURL)) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", _("http cookies are not supported by this QEMU binary")); return -1;
As noted before, don't add support fro nbdkit for pre-blockdev setups; it's pointless.
@@ -4920,7 +4973,8 @@ qemuDomainValidateStorageSource(virStorageSource *src, return -1; }
- if (!src->detected && !blockdev) { + if (!src->detected && !blockdev && + !qemuNbdkitCapsGet(nbdkitCaps, QEMU_NBDKIT_CAPS_FILTER_READAHEAD)) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", _("readahead setting is not supported with this QEMU binary")); return -1;
ditto.
@@ -4938,7 +4992,8 @@ qemuDomainValidateStorageSource(virStorageSource *src, return -1; }
- if (!src->detected && !blockdev) { + if (!src->detected && !blockdev && + !qemuNbdkitCapsGet(nbdkitCaps, QEMU_NBDKIT_CAPS_PLUGIN_CURL)) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", _("timeout setting is not supported with this QEMU binary")); return -1;
This too. [...]
@@ -10112,6 +10169,29 @@ qemuDomainPrepareStorageSourceNFS(virStorageSource *src) }
+/* qemuPrepareStorageSourceNbdkit: + * @src: source for a disk + * + * If src is an network source that is managed by nbdkit, prepare data so that + * nbdkit can be launched before the domain is started + */ +static void +qemuDomainPrepareStorageSourceNbdkit(virDomainDiskDef *disk, + virQEMUDriverConfig *cfg, + qemuDomainObjPrivate *priv) +{ + g_autoptr(qemuNbdkitCaps) nbdkit = qemuGetNbdkitCaps(priv->driver); + if (!nbdkit) + return; + + if (virStorageSourceGetActualType(disk->src) != VIR_STORAGE_TYPE_NETWORK) + return; + + qemuNbdkitInitStorageSource(nbdkit, disk->src, priv->libDir, + disk->info.alias, cfg->user, cfg->group);
You pass disk alias here, but a backing chain of a disk can have multiple network sourced images theoretically so for a single disk you can have multiple nbdkit instances. You should name them based on the node name. So this function should return whether nbdkit is going to be used for a given storage source so that the upper logic skips all the setup of the helper objects, so that we don't need to have hacks in the code avoiding the use of the perpared stuff.
+} + + /* qemuProcessPrepareStorageSourceTLS: * @source: source for a disk * @cfg: driver configuration @@ -10828,7 +10908,9 @@ qemuDomainPrepareDiskSourceLegacy(virDomainDiskDef *disk, qemuDomainObjPrivate *priv, virQEMUDriverConfig *cfg) { - if (qemuDomainValidateStorageSource(disk->src, priv->qemuCaps, true) < 0) + g_autoptr(qemuNbdkitCaps) nbdkitcaps = qemuGetNbdkitCaps(priv->driver); + if (qemuDomainValidateStorageSource(disk->src, priv->qemuCaps, + nbdkitcaps, true) < 0) return -1;
Don't bother with legacy stuff.
qemuDomainPrepareStorageSourceConfig(disk->src, cfg, priv->qemuCaps); @@ -10846,6 +10928,8 @@ qemuDomainPrepareDiskSourceLegacy(virDomainDiskDef *disk, priv) < 0) return -1;
+ qemuDomainPrepareStorageSourceNbdkit(disk, cfg, priv); + return 0; }
@@ -10857,6 +10941,7 @@ qemuDomainPrepareStorageSourceBlockdevNodename(virDomainDiskDef *disk, qemuDomainObjPrivate *priv, virQEMUDriverConfig *cfg) { + g_autoptr(qemuNbdkitCaps) nbdkitcaps = qemuGetNbdkitCaps(priv->driver); src->nodestorage = g_strdup_printf("%s-storage", nodenameprefix); src->nodeformat = g_strdup_printf("%s-format", nodenameprefix);
@@ -10866,7 +10951,8 @@ qemuDomainPrepareStorageSourceBlockdevNodename(virDomainDiskDef *disk, if (src->encryption && src->encryption->engine == VIR_STORAGE_ENCRYPTION_ENGINE_DEFAULT) src->encryption->engine = VIR_STORAGE_ENCRYPTION_ENGINE_QEMU;
- if (qemuDomainValidateStorageSource(src, priv->qemuCaps, false) < 0) + if (qemuDomainValidateStorageSource(src, priv->qemuCaps, + nbdkitcaps, false) < 0) return -1;
qemuDomainPrepareStorageSourceConfig(src, cfg, priv->qemuCaps); @@ -10887,6 +10973,8 @@ qemuDomainPrepareStorageSourceBlockdevNodename(virDomainDiskDef *disk, if (qemuDomainPrepareStorageSourceNFS(src) < 0) return -1;
+ qemuDomainPrepareStorageSourceNbdkit(disk, cfg, priv);
So this should be done right after all image frontend/format setup is done (encryption etc) and all image protocol setup should be skipped, so that there's a central point of setup. You'll need to refactor qemuDomainSecretStorageSourcePrepare to have two distinct steps for encryption which needs to be done also for nbdkit-backed disks and then authentication which needs to be skipped. TLS/cookies/and NFS setup can then be skipped too.
+ return 0; }
[...]
diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 3b5c3db67c..d1a972eb37 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -831,6 +831,9 @@ qemuStateInitialize(bool privileged, defsecmodel))) goto error;
+ /* find whether nbdkit is available and query its capabilities */ + qemu_driver->nbdkitCaps = qemuNbdkitCapsQuery(); + /* If hugetlbfs is present, then we need to create a sub-directory within * it, since we can't assume the root mount point has permissions that * will let our spawned QEMU instances use it. */ @@ -14471,7 +14474,6 @@ qemuDomainBlockPivot(virQEMUDriver *driver, if (reuse && shallow && virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_BLOCKDEV_SNAPSHOT_ALLOW_WRITE_ONLY) && virStorageSourceHasBacking(disk->mirror)) { -
Spurious change;
if (!(chainattachdata = qemuBuildStorageSourceChainAttachPrepareBlockdev(disk->mirror->backingStore))) return -1;
diff --git a/src/qemu/qemu_extdevice.c b/src/qemu/qemu_extdevice.c index b8e3c1000a..0f9361b294 100644 --- a/src/qemu/qemu_extdevice.c +++ b/src/qemu/qemu_extdevice.c @@ -218,6 +218,14 @@ qemuExtDevicesStart(virQEMUDriver *driver, return -1; }
+ for (i = 0; i < def->ndisks; i++) { + virDomainDiskDef *disk = def->disks[i]; + qemuDomainStorageSourcePrivate *priv = QEMU_DOMAIN_STORAGE_SOURCE_PRIVATE(disk->src); + if (priv && priv->nbdkitProcess &&
This is insufficient. If you have multiple images in an explicitly defined chain, any of them can have an 'nbdkit' process but this code checks only the topmost image. You'll need to walk the full bakcing chain and spawn all of the helpers.
+ qemuNbdkitProcessStart(priv->nbdkitProcess, vm, driver) < 0) + return -1; + } + return 0; }
@@ -262,6 +270,14 @@ qemuExtDevicesStop(virQEMUDriver *driver, fs->fsdriver == VIR_DOMAIN_FS_DRIVER_TYPE_VIRTIOFS) qemuVirtioFSStop(driver, vm, fs); } + + for (i = 0; i < def->ndisks; i++) { + virDomainDiskDef *disk = def->disks[i]; + qemuDomainStorageSourcePrivate *priv = QEMU_DOMAIN_STORAGE_SOURCE_PRIVATE(disk->src); + + if (priv && priv->nbdkitProcess) + qemuNbdkitProcessStop(priv->nbdkitProcess);
Same as above.
+ } }
@@ -324,6 +340,15 @@ qemuExtDevicesSetupCgroup(virQEMUDriver *driver, return -1; }
+ for (i = 0; i < def->ndisks; i++) { + virDomainDiskDef *disk = def->disks[i]; + qemuDomainStorageSourcePrivate *priv = QEMU_DOMAIN_STORAGE_SOURCE_PRIVATE(disk); + + if (priv && priv->nbdkitProcess && + qemuNbdkitProcessSetupCgroup(priv->nbdkitProcess, cgroup) < 0) + return -1;
And once more
+ } + for (i = 0; i < def->nfss; i++) { virDomainFSDef *fs = def->fss[i];
diff --git a/src/qemu/qemu_nbdkit.c b/src/qemu/qemu_nbdkit.c new file mode 100644 index 0000000000..dd8759689c --- /dev/null +++ b/src/qemu/qemu_nbdkit.c @@ -0,0 +1,629 @@ +/* + * qemu_nbdkit.c: helpers for using nbdkit + * + * Copyright (C) 2021 Red Hat, Inc. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library. If not, see + * <http://www.gnu.org/licenses/>. + * + */ + +#include <config.h> +#include <glib.h> + +#include "vircommand.h" +#include "virerror.h" +#include "virlog.h" +#include "virpidfile.h" +#include "virsecureerase.h" +#include "qemu_block.h" +#include "qemu_conf.h" +#include "qemu_domain.h" +#include "qemu_driver.h" +#include "qemu_extdevice.h" +#include "qemu_nbdkit.h" +#include "qemu_security.h" + +#include <fcntl.h> + +#define VIR_FROM_THIS VIR_FROM_NBDKIT + +VIR_LOG_INIT("qemu.nbdkit"); + +struct _qemuNbdkitCaps { + GObject parent; + + char *path; + char *version; + + virBitmap *flags; +}; +G_DEFINE_TYPE(qemuNbdkitCaps, qemu_nbdkit_caps, G_TYPE_OBJECT); + + +static void +qemuNbdkitCheckCommandCap(qemuNbdkitCaps *nbdkit, + virCommand *cmd, + qemuNbdkitCapsFlags cap) +{ + if (virCommandRun(cmd, NULL) != 0) + return; + + VIR_DEBUG("Setting nbdkit capability %i", cap); + ignore_value(virBitmapSetBit(nbdkit->flags, cap)); +} + + +static void +qemuNbdkitQueryFilter(qemuNbdkitCaps *nbdkit, + const char *filter, + qemuNbdkitCapsFlags cap) +{ + g_autoptr(virCommand) cmd = virCommandNewArgList(nbdkit->path, + "--version", + NULL); + + virCommandAddArgPair(cmd, "--filter", filter); + + /* use null plugin to check for filter */ + virCommandAddArg(cmd, "null"); + + qemuNbdkitCheckCommandCap(nbdkit, cmd, cap); +} + + +static void +qemuNbdkitQueryPlugin(qemuNbdkitCaps *nbdkit, + const char *plugin, + qemuNbdkitCapsFlags cap) +{ + g_autoptr(virCommand) cmd = virCommandNewArgList(nbdkit->path, + plugin, + "--version", + NULL); + + qemuNbdkitCheckCommandCap(nbdkit, cmd, cap); +} + + +static void +qemuNbdkitQueryPlugins(qemuNbdkitCaps *nbdkit) +{ + qemuNbdkitQueryPlugin(nbdkit, "curl", QEMU_NBDKIT_CAPS_PLUGIN_CURL); + qemuNbdkitQueryPlugin(nbdkit, "ssh", QEMU_NBDKIT_CAPS_PLUGIN_SSH);
Do we really have to run nbdkit multiple times to check these?
+} + + +static void +qemuNbdkitQueryFilters(qemuNbdkitCaps *nbdkit) +{ + qemuNbdkitQueryFilter(nbdkit, "readahead", + QEMU_NBDKIT_CAPS_FILTER_READAHEAD); +} + + +static int +qemuNbdkitQueryVersion(qemuNbdkitCaps *nbdkit) +{ + g_autoptr(virCommand) cmd = virCommandNewArgList(nbdkit->path, + "--version", + NULL); + + virCommandSetOutputBuffer(cmd, &nbdkit->version); + + if (virCommandRun(cmd, NULL) != 0) + return -1; + + VIR_DEBUG("Got nbdkit version %s", nbdkit->version); + return 0; +} + + +static void qemuNbdkitCapsFinalize(GObject *object) +{ + qemuNbdkitCaps *nbdkit = QEMU_NBDKIT_CAPS(object); + + g_clear_pointer(&nbdkit->path, g_free); + g_clear_pointer(&nbdkit->version, g_free); + g_clear_pointer(&nbdkit->flags, virBitmapFree); + + G_OBJECT_CLASS(qemu_nbdkit_caps_parent_class)->finalize(object); +} + + +void qemu_nbdkit_caps_init(qemuNbdkitCaps *caps) +{ + caps->flags = virBitmapNew(QEMU_NBDKIT_CAPS_LAST); + caps->version = NULL; +} + + +static void +qemu_nbdkit_caps_class_init(qemuNbdkitCapsClass *klass) +{ + GObjectClass *obj = G_OBJECT_CLASS(klass); + + obj->finalize = qemuNbdkitCapsFinalize; +} + + +qemuNbdkitCaps * +qemuNbdkitCapsNew(const char *path) +{ + qemuNbdkitCaps *caps = g_object_new(QEMU_TYPE_NBDKIT_CAPS, NULL); + caps->path = g_strdup(path); + + return caps; +} + + +qemuNbdkitCaps * +qemuNbdkitCapsQuery(void) +{ + qemuNbdkitCaps *caps = NULL; + g_autofree char *path = virFindFileInPath("nbdkit"); + + if (!path) + return NULL; + + // make sure it's executable + if (!virFileIsExecutable(path)) { + virReportError(VIR_ERR_INTERNAL_ERROR, _("nbdkit '%s' is not executable"), + path); + return NULL; + } + + VIR_DEBUG("found nbdkit executable '%s'", path); + caps = qemuNbdkitCapsNew(path); + + qemuNbdkitQueryPlugins(caps); + qemuNbdkitQueryFilters(caps); + qemuNbdkitQueryVersion(caps); + + return caps; +} + + +bool +qemuNbdkitCapsGet(qemuNbdkitCaps *nbdkitCaps, qemuNbdkitCapsFlags flag) +{ + return virBitmapIsBitSet(nbdkitCaps->flags, flag); +} + + +void +qemuNbdkitCapsSet(qemuNbdkitCaps *nbdkitCaps, qemuNbdkitCapsFlags flag) +{ + ignore_value(virBitmapSetBit(nbdkitCaps->flags, flag)); +} + + +static int childProcess(int fd, uint8_t *output, size_t outputlen) +{ + if (safewrite(fd, output, outputlen) < 0) { + virReportSystemError(errno, "%s", + _("failed to write to socket for nbdkit")); + return -errno; + } + return 0; +} + + +/* Forks a process to write a password to a socket to the nbdkit process. + * Returns a file descriptor that can be passed to ndkit */ +static int qemuNbdkitForkHandler(uint8_t *output, size_t outputlen) +{ + enum { + PARENT_SOCKET = 0, + CHILD_SOCKET = 1 + }; + int pair[2] = { -1, -1 }; + pid_t pid; + + if (socketpair(AF_UNIX, SOCK_STREAM, 0, pair) < 0) { + virReportSystemError(errno, "%s", + _("failed to create socket for nbdkit")); + return -errno; + } + + pid = virFork(); + + if (pid < 0) + return -errno; + + /* child process */ + if (pid == 0) { + int ret = childProcess(pair[CHILD_SOCKET], output, outputlen); + _exit(ret); + } + + /* parent process */ + VIR_FORCE_CLOSE(pair[CHILD_SOCKET]); + + return pair[PARENT_SOCKET]; +} + + +/* Forks a process to write cookies to a socket to the nbdkit process. + * Returns a file descriptor that can be passed to ndkit */ +static int qemuNbdkitProcessForkCookieHandler(qemuNbdkitProcess *proc) +{ + g_autofree char *cookies = qemuBlockStorageSourceGetCookieString(proc->source); + + if (!cookies) + return 0; + + if ((proc->cookiefd = qemuNbdkitForkHandler((uint8_t*)cookies, strlen(cookies))) < 0) + return -1; + + return 0; +} + + +/* Forks a process to write a password to a socket to the nbdkit process. + * Returns a file descriptor that can be passed to ndkit */ +static int qemuNbdkitProcessForkPasswordHandler(qemuNbdkitProcess *proc) +{ + g_autofree uint8_t *password = NULL; + size_t passwordlen = 0; + g_autoptr(virConnect) conn = virGetConnectSecret(); + + if (!proc->source->auth->username) + return 0; + + if (virSecretGetSecretString(conn, + &proc->source->auth->seclookupdef, + /* FIXME: for some reason auth->authType is always NONE... */ + VIR_SECRET_USAGE_TYPE_ISCSI, + &password, + &passwordlen) < 0) + { + /* FIXME: message */ + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("failed to get auth secret for storage")); + return -1; + } + + if ((proc->authfd = qemuNbdkitForkHandler(password, passwordlen)) < 0) + return -1; + + return 0; +} + + +qemuNbdkitProcess * +qemuNbdkitProcessLoad(virStorageSource *source, + const char *pidfile, + const char *socketfile) +{ + int rc; + qemuNbdkitProcess *nbdkit = g_new0(qemuNbdkitProcess, 1); + + nbdkit->pidfile = g_strdup(pidfile); + nbdkit->socketfile = g_strdup(socketfile); + nbdkit->source = virObjectRef(source); + nbdkit->user = -1; + nbdkit->group = -1; + + if ((rc = virPidFileReadPath(nbdkit->pidfile, &nbdkit->pid)) < 0) + VIR_WARN("Failed to read pidfile %s", nbdkit->pidfile); + + return nbdkit; +} + +
[...]
+ + +static void +qemuNbdkitProcessBuildCommandSSH(qemuNbdkitProcess *proc, + virCommand *cmd) +{ + char *user = NULL; + virStorageNetHostDef *host = &proc->source->hosts[0]; + + /* nbdkit plugin name */ + virCommandAddArg(cmd, "ssh"); + + virCommandAddArgPair(cmd, "host", g_strdup(host->name));
virCommandAddArgPair calls: virCommandAddArgFormat(cmd, "%s=%s", name, val); Which in turn calls g_strdup_vprintf with '%s=%s' as argument and 'name, val' as vargs. Thus the copied memory from the second argument is leaked as the function doesn't take ownership.
+ virCommandAddArgPair(cmd, "port", g_strdup_printf("%u", + host->port)); + virCommandAddArgPair(cmd, "path", g_strdup(proc->source->path));
same here.
+ + if (proc->source->auth) { + user = g_strdup(proc->source->auth->username); + } else if (proc->source->ssh_user) { + user = g_strdup(proc->source->ssh_user); + } + + if (user) { + virCommandAddArgPair(cmd, "user", user);
'user' variable is leaked.
+ } + + if (proc->source->ssh_host_key_check_disabled) { + virCommandAddArgPair(cmd, "verify-remote-host", + g_strdup("false"));
The copied string will be leaked.
+ } +} + +
[...]
+int +qemuNbdkitProcessSetupCgroup(qemuNbdkitProcess *proc, + virCgroup *cgroup) +{ + return virCgroupAddProcess(cgroup, proc->pid); +} + + +/* FIXME: selinux permissions errors */ +int +qemuNbdkitProcessStart(qemuNbdkitProcess *proc, + virDomainObj *vm, + virQEMUDriver *driver) +{ + g_autoptr(virCommand) cmd = NULL; + int rc; + int exitstatus = 0; + int cmdret = 0; + unsigned int loops = 0; + int errfd = -1; + + if (qemuNbdkitProcessForkCookieHandler(proc) < 0) + return -1; + + if (qemuNbdkitProcessForkPasswordHandler(proc) < 0) + return -1; + + if (!(cmd = qemuNbdkitProcessBuildCommand(proc))) + return -1; + + virCommandSetErrorFD(cmd, &errfd); + + if (qemuExtDeviceLogCommand(driver, vm, cmd, "nbdkit") < 0) + goto error; + + if (qemuSecurityCommandRun(driver, vm, cmd, proc->user, proc->group, &exitstatus, &cmdret) < 0) + goto error; + + if (cmdret < 0 || exitstatus != 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("Could not start 'nbdkit'. exitstatus: %d"), exitstatus); + goto error; + } + + /* Wait for the pid file to appear. The file doesn't appear until nbdkit is + * ready to accept connections to the socket */ + while (!virFileExists(proc->pidfile)) {
I don't think we want to go this way. Isn't there a possibility to pass the socket or an FD so that we are notified?
+ VIR_DEBUG("waiting for nbdkit pidfile %s: %u", proc->pidfile, loops); + /* wait for 100ms before checking again, but don't do it for ever */ + if (errno == ENOENT && loops < 10) { + g_usleep(100 * 1000);
Because here you'll get a 100ms penalty per process of startup delay if it doesn't come up quickly enough.
+ loops++; + } else { + char errbuf[1024] = { 0 }; + if (errfd && saferead(errfd, errbuf, sizeof(errbuf) - 1) > 0) { + virReportError(VIR_ERR_OPERATION_FAILED, + _("nbdkit failed to start: %s"), errbuf); + } else { + virReportSystemError(errno, "%s", + _("Gave up waiting for nbdkit to start")); + } + goto error; + } + } + + if ((rc = virPidFileReadPath(proc->pidfile, &proc->pid)) < 0) { + virReportSystemError(-rc, + _("Failed to read pidfile %s"), + proc->pidfile); + goto error; + } + + return 0;
[...]
diff --git a/src/qemu/qemu_validate.c b/src/qemu/qemu_validate.c index 39210ba65b..b6cc75b651 100644 --- a/src/qemu/qemu_validate.c +++ b/src/qemu/qemu_validate.c @@ -602,7 +602,8 @@ qemuValidateDomainDefPM(const virDomainDef *def,
static int qemuValidateDomainDefNvram(const virDomainDef *def, - virQEMUCaps *qemuCaps) + virQEMUCaps *qemuCaps, + qemuNbdkitCaps *nbdkitCaps)
If you intend to instantiate nbdkit even for the nvram backing you'll need to make sure to start the process too.
{ virStorageSource *src = def->os.loader->nvram;
[...]
diff --git a/tests/qemustatusxml2xmldata/modern-in.xml b/tests/qemustatusxml2xmldata/modern-in.xml index 7759034f7a..49b005cfc7 100644 --- a/tests/qemustatusxml2xmldata/modern-in.xml +++ b/tests/qemustatusxml2xmldata/modern-in.xml @@ -337,7 +337,6 @@ <objects> <secret type='auth' alias='test-auth-alias'/> <secret type='encryption' alias='test-encryption-alias'/> - <secret type='httpcookie' alias='http-cookie-alias'/>
Spurious change?
<secret type='tlskey' alias='tls-certificate-key-alias'/> <TLSx509 alias='transport-alias'/> </objects>

On 6/22/22 23:26, Jonathon Jongsma wrote:
---
When you send this for review, please split it into more patches. There's too much going on in this single patch.
include/libvirt/virterror.h | 1 + po/POTFILES | 1 + src/qemu/meson.build | 1 + src/qemu/qemu_block.c | 64 +- src/qemu/qemu_block.h | 1 + src/qemu/qemu_command.c | 26 +- src/qemu/qemu_conf.c | 19 + src/qemu/qemu_conf.h | 5 + src/qemu/qemu_domain.c | 110 ++- src/qemu/qemu_domain.h | 5 + src/qemu/qemu_driver.c | 4 +- src/qemu/qemu_extdevice.c | 25 + src/qemu/qemu_nbdkit.c | 629 ++++++++++++++++++ src/qemu/qemu_nbdkit.h | 89 +++ src/qemu/qemu_validate.c | 22 +- src/qemu/qemu_validate.h | 4 +- src/util/virerror.c | 1 + tests/qemublocktest.c | 8 +- tests/qemustatusxml2xmldata/modern-in.xml | 1 - ...sk-cdrom-network-nbdkit.x86_64-latest.args | 42 ++ .../disk-cdrom-network-nbdkit.xml | 1 + ...isk-network-http-nbdkit.x86_64-latest.args | 45 ++ .../disk-network-http-nbdkit.xml | 1 + ...work-source-curl-nbdkit.x86_64-latest.args | 49 ++ .../disk-network-source-curl-nbdkit.xml | 1 + ...isk-network-source-curl.x86_64-latest.args | 53 ++ .../disk-network-source-curl.xml | 71 ++ tests/qemuxml2argvtest.c | 12 + tests/testutilsqemu.c | 16 + tests/testutilsqemu.h | 4 + 30 files changed, 1278 insertions(+), 33 deletions(-) create mode 100644 src/qemu/qemu_nbdkit.c create mode 100644 src/qemu/qemu_nbdkit.h create mode 100644 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.x86_64-latest.args create mode 120000 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.xml create mode 100644 tests/qemuxml2argvdata/disk-network-http-nbdkit.x86_64-latest.args create mode 120000 tests/qemuxml2argvdata/disk-network-http-nbdkit.xml create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.x86_64-latest.args create mode 120000 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.xml create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.x86_64-latest.args create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.xml
+ + +int +qemuNbdkitProcessSetupCgroup(qemuNbdkitProcess *proc, + virCgroup *cgroup) +{ + return virCgroupAddProcess(cgroup, proc->pid); +} + + +/* FIXME: selinux permissions errors */ +int +qemuNbdkitProcessStart(qemuNbdkitProcess *proc, + virDomainObj *vm, + virQEMUDriver *driver) +{ + g_autoptr(virCommand) cmd = NULL; + int rc; + int exitstatus = 0; + int cmdret = 0; + unsigned int loops = 0; + int errfd = -1; + + if (qemuNbdkitProcessForkCookieHandler(proc) < 0) + return -1; +
Creating an extra process so that a buffer can be written into an FD looks like an overkill.
+ if (qemuNbdkitProcessForkPasswordHandler(proc) < 0) + return -1;
Same here. What happens if you'd just write into these sockets (btw. can it be just a pipe?) after virCommandRun()?
+ + if (!(cmd = qemuNbdkitProcessBuildCommand(proc))) + return -1; +> + virCommandSetErrorFD(cmd, &errfd); + + if (qemuExtDeviceLogCommand(driver, vm, cmd, "nbdkit") < 0) + goto error; + + if (qemuSecurityCommandRun(driver, vm, cmd, proc->user, proc->group, &exitstatus, &cmdret) < 0) + goto error; + + if (cmdret < 0 || exitstatus != 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("Could not start 'nbdkit'. exitstatus: %d"), exitstatus); + goto error; + } +
This or .. [1]
+ /* Wait for the pid file to appear. The file doesn't appear until nbdkit is + * ready to accept connections to the socket */ + while (!virFileExists(proc->pidfile)) {
This relies on executed binary to create pidfile. I find it more robust when the pidfile is created by virCommand*() - also because it's going to be locked and we can detect later on whether it's stale or not. But more importantly - the pidfile is written before exec(). You can take inspiration from qemuDBusStart().
+ VIR_DEBUG("waiting for nbdkit pidfile %s: %u", proc->pidfile, loops); + /* wait for 100ms before checking again, but don't do it for ever */ + if (errno == ENOENT && loops < 10) { + g_usleep(100 * 1000); + loops++; + } else { + char errbuf[1024] = { 0 }; + if (errfd && saferead(errfd, errbuf, sizeof(errbuf) - 1) > 0) { + virReportError(VIR_ERR_OPERATION_FAILED, + _("nbdkit failed to start: %s"), errbuf); + } else { + virReportSystemError(errno, "%s", + _("Gave up waiting for nbdkit to start")); + } + goto error; + } + } + + if ((rc = virPidFileReadPath(proc->pidfile, &proc->pid)) < 0) { + virReportSystemError(-rc, + _("Failed to read pidfile %s"), + proc->pidfile); + goto error; + } +
1: ... this looks like a good place to write those secrets. Moreover, if we'd get error writing into the pipe we know the process died. I don't feel confident reviewing the storage part of the feature, sorry. Michal

On 6/22/22 23:26, Jonathon Jongsma wrote:
Hi guys,
I've been working on adding support for nbdkit to libvirt for network storage sources like http and ssh. See https://bugzilla.redhat.com/show_bug.cgi?id=2016527 for more information, but the summary is that RHEL does not want to ship the qemu storage plugins for curl and ssh. Handling them outside of the qemu process provides several advantages such as reduced attack surface and stability.
I have something that works for me, but as I have not dealt with the storage stuff much before, I have a feeling that I'm missing some things.
A quick summary of the code: - at startup I query to see whether nbdkit exists on the host and if so, I query which plugins/filters are installed. This is stored as qemuNbdkitCaps on the qemu driver - When the driver prepares the domain, we go through each disk source and determine whether the nbdkit capabilities allow us to support this disk via nbdkit, and if so, we allocate a qemuNbdkitProcess object and stash it in the private data of the virStorageSource. - The presence or absence of this qemuNbdkitProcess data then indicates whether this disk will be served to qemu indirectly via nbdkit or not - When we launch the qemuProcess, as part of the "external device start" step, I launch a ndkit process for each disk that is supported by nbdkit. I also optionally fork a child process to communicate authentication details and cookies to the nbdkit process via a unix socket. - for devices which are served by the ndkit process, I change the qemu commandline in the following ways: - I no longer pass auth/cookie secrets to qemu (those are handled by nbdkit) - I replace the actual network URL of the remote disk source with the path to the nbdkit unix socket
Known shortcomings - I don't yet re-query for nbdkit / nbdkit caps, so need to restart libvirt to pick up newly-installed nbdkit or additional capabilities
There are couple of ways to solve this: 1) mimic what we do for dnsmasq: basically, when initializing the bridge driver we query dnsmasq for caps networkStateInitialize() -> dnsmasqCapsNewFromBinary() and when needing the caps for real (i.e. spawning dnsmasq when starting a network) we forcefully refresh the caps: networkStartDhcpDaemon() -> networkDnsmasqCapsRefresh(). 2) use full blown virFileCache() machinery which reacts to mtime of given binary. An argument against going with 1) is that while spawning an extra process (dnsmasq) when starting a network is acceptable (a network is not started that often), spawning an extra process for each disk (or even an image in backingchain) might be too expensive.
- testing is pretty limited at the moment - selinux not working yet - creating disks isn't supported, though Rich has added some support for that upstream in the nbdkit ssh plugin.
I'd appreciate feedback on what i've got so far.
Jonathon Jongsma (3): docs: clarify 'readahead' and 'timeout' for disks schema: Be more flexible for diskSourceNetworkProtocolPropsCommon WIP: use nbdkit for remote disk sources
docs/formatdomain.rst | 10 +- include/libvirt/virterror.h | 1 + po/POTFILES | 1 + src/conf/schemas/domaincommon.rng | 34 +- src/qemu/meson.build | 1 + src/qemu/qemu_block.c | 64 +- src/qemu/qemu_block.h | 1 + src/qemu/qemu_command.c | 26 +- src/qemu/qemu_conf.c | 19 + src/qemu/qemu_conf.h | 5 + src/qemu/qemu_domain.c | 110 ++- src/qemu/qemu_domain.h | 5 + src/qemu/qemu_driver.c | 4 +- src/qemu/qemu_extdevice.c | 25 + src/qemu/qemu_nbdkit.c | 629 ++++++++++++++++++ src/qemu/qemu_nbdkit.h | 89 +++ src/qemu/qemu_validate.c | 22 +- src/qemu/qemu_validate.h | 4 +- src/util/virerror.c | 1 + tests/qemublocktest.c | 8 +- tests/qemustatusxml2xmldata/modern-in.xml | 1 - ...sk-cdrom-network-nbdkit.x86_64-latest.args | 42 ++ .../disk-cdrom-network-nbdkit.xml | 1 + ...isk-network-http-nbdkit.x86_64-latest.args | 45 ++ .../disk-network-http-nbdkit.xml | 1 + ...work-source-curl-nbdkit.x86_64-latest.args | 49 ++ .../disk-network-source-curl-nbdkit.xml | 1 + ...isk-network-source-curl.x86_64-latest.args | 53 ++ .../disk-network-source-curl.xml | 71 ++ tests/qemuxml2argvtest.c | 12 + tests/testutilsqemu.c | 16 + tests/testutilsqemu.h | 4 + 32 files changed, 1302 insertions(+), 53 deletions(-) create mode 100644 src/qemu/qemu_nbdkit.c create mode 100644 src/qemu/qemu_nbdkit.h create mode 100644 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.x86_64-latest.args create mode 120000 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.xml create mode 100644 tests/qemuxml2argvdata/disk-network-http-nbdkit.x86_64-latest.args create mode 120000 tests/qemuxml2argvdata/disk-network-http-nbdkit.xml create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.x86_64-latest.args create mode 120000 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.xml create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.x86_64-latest.args create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.xml
participants (3)
-
Jonathon Jongsma
-
Michal Prívozník
-
Peter Krempa