[libvirt] [PATCH 0/7] introduce support for an embedded driver mode
This is a followup to: https://www.redhat.com/archives/libvir-list/2019-May/msg00467.html This series implements support for an embedded driver mode for libvirt, with initial support in the QEMU and secrets drivers. In this mode of operation, the driver stores all its config and state under a private directory tree. See the individual patches for the illustrated directory hierarchy used. The intent of this embedded mode is to suit cases where the application is using virtualization as a building block for some functionality, as opposed to running traditional "full OS" builds. The long time posterchild example would be libguestfs, while a more recent example could be Kata containers. The general principal in enabling this embedded mode is that the functionality available should be identical to that seen when the driver is running inside libvirtd. This is achieved by loading the exact same driver .so module as libvirtd would load, and simply configuring it with a different directory layout. The result of this is that when running in embedded mode, the driver can still talk to other secondary drivers running inside libvirtd if desired. This is useful, for example, to connect a VM to the default virtual network. The secondary drivers can be made to operate in embedded mode as well, however, this will require some careful consideration for each driver to ensure they don't clash with each other. Thus in this series only the secret driver is enabled for embedded mode. This is required to enable use of VMs with encrypted disks, or authenticated network block storage. In this series we introduce a new command line tool 'virt-qemu-run' which is a really simple tool for launching a VM in embedded mode. I'm not entirely sure whether we should provide this as an official supported tool in this way, or merely put it into the 'examples' directory as demo-ware. With testing of the virt-qemu-run tool we can immediately see what the next important thing to tackle is: performance. We have not really cared too much about the startup performance of libvirtd as this is a one time cost when the mgmt application connects. We did none the less cache capabilities because probing caps for 30 QEMU binaries takes a long time. Even with this caching it takes an unacceptably long time to start a VM in embedded mode. About 100 ms to open the embedded QEMU driver, assuming pre-cached capabilies - ~2 seconds if not cached and all 30 QEMU targets are present. Then about 300 ms to actually start the QEMU guest. IOW, about 400 ms to get QEMU running. NB this is measuring time from launching the virt-run-qemu program, to the point at which the API call 'virDomainCreate' returns control. This has both libvirt & QEMU overhead in & I don't have clear figures to distinguish, but I can see a 40 ms delay between issuing the 'qmp_capabilities' call and getting a reply, which is QEMU startup overead. This is a i440fx based QEMU with a general purpose virtio-pci config (disk, net, etc) tyupical for running a full OS. I've not tried any kind of optimized QEMU config with microvm. I've already started on measuring & optimizing & identified several key areas that can be addressed, but it is all ultimately about not doing work before we need the answers from that work (which often means we will never do the work at all). For example, we shouldn't probe all 30 QEMU's upfront. If the app is only going to create an x86_64 KVM guest we should only care about that 1 QEMU. This is painful because parsing any guest XML requires a virCapsPtr which in turn causes probing of every QEMU binary. I've got in progress patches to eliminate virCapsPtr almost entirely and work directly with the virQEMUCapsPtr instead. It is possible we'll want to use a different file format for storing the cached QEMU capabilities, and the CPU feature/model info. Parsing this XML is a non-negligible time sink. A binary format is likely way quicker, especially if its designed to be just mmap'able for direct read. To be investigated... We shouldn't probe for whether host PM suspend is possible unless someone wants that info, or tries to issue that API call. After starting QEMU we spend 150-200 ms issuing a massive number of qom-get calls to check whether QEMU enabled each individual CPU feature flag. We only need this info if someone asks for the live XML or we intend to live migrate etc. So we shouldn't issue these qom-get calls in the "hot path" of QEMU startup. It can be done later in a non-time critical point. Also the QEMU API for this is horribly inefficient to require so many qom-get calls. There's more but I won't talk about it now. Suffice to say that I think we can get libvirt overhead down to less than 100 ms fairly easily and probably even down to less than 50 ms without much effort. The exact figure will depend on what libvirt features you want enabled, and how much work we want/need to put into optimization. We'll want to fix the really gross mistakes & slow downs, but we'll want guidance from likely users as to their VM startup targets to decide how much work needs investing. This optimization will ultimately help non-embedded QEMU mode too, making it faster to respond & start. Changed in v2: - Use a simplified directory layout for embedded mode. Previously we just put a dir prefix onto the normal paths. This has the downside that the embedded drivers paths are needlessly different for privileged vs unprivileged user. It also results in very long paths which can be a problem for the UNIX socket name length limits. - Also ported the secret driver to support embedded mode - Check to validate that the event loop is registered. - Add virt-qemu-run tool for embedded usage. - Added docs for the qemu & secret driver explaining embedded mode Daniel P. Berrangé (7): access: report an error if no access manager is present libvirt: pass a directory path into drivers for embedded usage event: add API for requiring an event loop impl to be registered libvirt: support an "embed" URI path selector for opening drivers qemu: add support for running QEMU driver in embedded mode secrets: add support for running secret driver in embedded mode qemu: introduce a new "virt-qemu-run" program build-aux/syntax-check.mk | 2 +- docs/drivers.html.in | 1 + docs/drvqemu.html.in | 84 +++++++ docs/drvsecret.html.in | 82 +++++++ libvirt.spec.in | 2 + po/POTFILES.in | 1 + src/Makefile.am | 9 + src/access/viraccessmanager.c | 5 + src/driver-state.h | 2 + src/driver.h | 2 + src/interface/interface_backend_netcf.c | 7 + src/interface/interface_backend_udev.c | 7 + src/libvirt.c | 93 ++++++- src/libvirt_internal.h | 4 +- src/libxl/libxl_driver.c | 7 + src/lxc/lxc_driver.c | 8 + src/network/bridge_driver.c | 7 + src/node_device/node_device_hal.c | 7 + src/node_device/node_device_udev.c | 7 + src/nwfilter/nwfilter_driver.c | 7 + src/qemu/Makefile.inc.am | 26 ++ src/qemu/qemu_conf.c | 38 ++- src/qemu/qemu_conf.h | 6 +- src/qemu/qemu_driver.c | 21 +- src/qemu/qemu_process.c | 15 +- src/qemu/qemu_shim.c | 313 ++++++++++++++++++++++++ src/qemu/qemu_shim.pod | 94 +++++++ src/remote/remote_daemon.c | 1 + src/remote/remote_driver.c | 1 + src/secret/secret_driver.c | 41 +++- src/storage/storage_driver.c | 7 + src/util/virevent.c | 25 ++ src/util/virevent.h | 2 + src/vz/vz_driver.c | 7 + tests/domaincapstest.c | 2 +- tests/testutilsqemu.c | 2 +- 36 files changed, 920 insertions(+), 25 deletions(-) create mode 100644 docs/drvsecret.html.in create mode 100644 src/qemu/qemu_shim.c create mode 100644 src/qemu/qemu_shim.pod -- 2.23.0
The code calling this method expects it to have reported an error on failure. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/access/viraccessmanager.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/src/access/viraccessmanager.c b/src/access/viraccessmanager.c index 31e1787919..f4120c6139 100644 --- a/src/access/viraccessmanager.c +++ b/src/access/viraccessmanager.c @@ -65,6 +65,11 @@ VIR_ONCE_GLOBAL_INIT(virAccessManager); virAccessManagerPtr virAccessManagerGetDefault(void) { + if (virAccessManagerDefault == NULL) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("No access manager registered")); + return NULL; + } return virObjectRef(virAccessManagerDefault); } -- 2.23.0
On 12/2/19 10:03 AM, Daniel P. Berrangé wrote:
The code calling this method expects it to have reported an error on failure.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/access/viraccessmanager.c | 5 +++++ 1 file changed, 5 insertions(+)
diff --git a/src/access/viraccessmanager.c b/src/access/viraccessmanager.c index 31e1787919..f4120c6139 100644 --- a/src/access/viraccessmanager.c +++ b/src/access/viraccessmanager.c @@ -65,6 +65,11 @@ VIR_ONCE_GLOBAL_INIT(virAccessManager);
virAccessManagerPtr virAccessManagerGetDefault(void) { + if (virAccessManagerDefault == NULL) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("No access manager registered")); + return NULL; + }
I would expect to see a newline here
return virObjectRef(virAccessManagerDefault); }
Reviewed-by: Cole Robinson <crobinso@redhat.com> - Cole
The intent here is to allow the virt drivers to be run directly embedded in an arbitrary process without interfering with libvirtd. To achieve this they need to store all their configuration & state in a separate directory tree from the main system or session libvirtd instances. This can be useful for doing testing of the virt drivers in "make check" without interfering with the user's own libvirtd instances. It can also be used for applications using KVM/QEMU as a piece of infrastructure to build an service, rather than for general purpose OS hosting. A long standing example is libguestfs, which would prefer if its temporary VMs did show up in the main libvirtd VM list, because this confuses apps such as OpenStack Nova. A more recent example would be Kata which is using KVM as a technology to build containers. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/driver-state.h | 1 + src/interface/interface_backend_netcf.c | 7 +++++++ src/interface/interface_backend_udev.c | 7 +++++++ src/libvirt.c | 21 +++++++++++++++++++++ src/libvirt_internal.h | 4 +++- src/libxl/libxl_driver.c | 7 +++++++ src/lxc/lxc_driver.c | 8 ++++++++ src/network/bridge_driver.c | 7 +++++++ src/node_device/node_device_hal.c | 7 +++++++ src/node_device/node_device_udev.c | 7 +++++++ src/nwfilter/nwfilter_driver.c | 7 +++++++ src/qemu/qemu_driver.c | 7 +++++++ src/remote/remote_daemon.c | 1 + src/remote/remote_driver.c | 1 + src/secret/secret_driver.c | 7 +++++++ src/storage/storage_driver.c | 7 +++++++ src/vz/vz_driver.c | 7 +++++++ 17 files changed, 112 insertions(+), 1 deletion(-) diff --git a/src/driver-state.h b/src/driver-state.h index 69e2678dfc..1e2f6ed247 100644 --- a/src/driver-state.h +++ b/src/driver-state.h @@ -32,6 +32,7 @@ typedef enum { typedef virDrvStateInitResult (*virDrvStateInitialize)(bool privileged, + const char *root, virStateInhibitCallback callback, void *opaque); diff --git a/src/interface/interface_backend_netcf.c b/src/interface/interface_backend_netcf.c index 4f46717cf3..951a7e6c23 100644 --- a/src/interface/interface_backend_netcf.c +++ b/src/interface/interface_backend_netcf.c @@ -89,9 +89,16 @@ virNetcfDriverStateDispose(void *obj) static int netcfStateInitialize(bool privileged, + const char *root, virStateInhibitCallback callback G_GNUC_UNUSED, void *opaque G_GNUC_UNUSED) { + if (root != NULL) { + virReportError(VIR_ERR_INVALID_ARG, "%s", + _("Driver does not support embedded mode")); + return -1; + } + if (virNetcfDriverStateInitialize() < 0) return VIR_DRV_STATE_INIT_ERROR; diff --git a/src/interface/interface_backend_udev.c b/src/interface/interface_backend_udev.c index b7b06ed67a..a4f5deb26d 100644 --- a/src/interface/interface_backend_udev.c +++ b/src/interface/interface_backend_udev.c @@ -1150,11 +1150,18 @@ udevStateCleanup(void); static int udevStateInitialize(bool privileged, + const char *root, virStateInhibitCallback callback G_GNUC_UNUSED, void *opaque G_GNUC_UNUSED) { int ret = VIR_DRV_STATE_INIT_ERROR; + if (root != NULL) { + virReportError(VIR_ERR_INVALID_ARG, "%s", + _("Driver does not support embedded mode")); + return -1; + } + if (VIR_ALLOC(driver) < 0) goto cleanup; diff --git a/src/libvirt.c b/src/libvirt.c index 9d783761e6..bd2952d036 100644 --- a/src/libvirt.c +++ b/src/libvirt.c @@ -637,16 +637,36 @@ virRegisterStateDriver(virStateDriverPtr driver) * virStateInitialize: * @privileged: set to true if running with root privilege, false otherwise * @mandatory: set to true if all drivers must report success, not skipped + * @root: directory to use for embedded mode * @callback: callback to invoke to inhibit shutdown of the daemon * @opaque: data to pass to @callback * * Initialize all virtualization drivers. * + * Passing a non-NULL @root instructs the driver to run in embedded mode. + * Instead of using the compile time $prefix as the basis for directory + * paths, @root should be used instead. In addition any '/libvirt' + * component of the paths should be stripped. + * + * eg consider a build with prefix=/usr/local. A driver might use the + * locations + * + * /usr/local/etc/libvirt/$DRIVER/ + * /usr/local/var/lib/libvirt/$DRIVER/ + * /usr/local/run/libvirt/$DRIVER/ + * + * When run with @root, the locations should instead be + * + * @root/etc/$DRIVER/ + * @root/var/lib/$DRIVER/ + * @root/run/$DRIVER/ + * * Returns 0 if all succeed, -1 upon any failure. */ int virStateInitialize(bool privileged, bool mandatory, + const char *root, virStateInhibitCallback callback, void *opaque) { @@ -661,6 +681,7 @@ virStateInitialize(bool privileged, VIR_DEBUG("Running global init for %s state driver", virStateDriverTab[i]->name); ret = virStateDriverTab[i]->stateInitialize(privileged, + root, callback, opaque); VIR_DEBUG("State init result %d (mandatory=%d)", ret, mandatory); diff --git a/src/libvirt_internal.h b/src/libvirt_internal.h index 4a74dbc2af..00ef7aaf25 100644 --- a/src/libvirt_internal.h +++ b/src/libvirt_internal.h @@ -31,8 +31,10 @@ typedef void (*virStateInhibitCallback)(bool inhibit, int virStateInitialize(bool privileged, bool mandatory, + const char *root, virStateInhibitCallback inhibit, - void *opaque); + void *opaque) + ATTRIBUTE_NONNULL(2); int virStateCleanup(void); int virStateReload(void); int virStateStop(void); diff --git a/src/libxl/libxl_driver.c b/src/libxl/libxl_driver.c index 44a74e8779..23ab29c4e8 100644 --- a/src/libxl/libxl_driver.c +++ b/src/libxl/libxl_driver.c @@ -647,6 +647,7 @@ libxlAddDom0(libxlDriverPrivatePtr driver) static int libxlStateInitialize(bool privileged, + const char *root, virStateInhibitCallback callback G_GNUC_UNUSED, void *opaque G_GNUC_UNUSED) { @@ -655,6 +656,12 @@ libxlStateInitialize(bool privileged, char ebuf[1024]; bool autostart = true; + if (root != NULL) { + virReportError(VIR_ERR_INVALID_ARG, "%s", + _("Driver does not support embedded mode")); + return -1; + } + if (!libxlDriverShouldLoad(privileged)) return VIR_DRV_STATE_INIT_SKIPPED; diff --git a/src/lxc/lxc_driver.c b/src/lxc/lxc_driver.c index 826bf074e3..7c2c4798e5 100644 --- a/src/lxc/lxc_driver.c +++ b/src/lxc/lxc_driver.c @@ -85,6 +85,7 @@ VIR_LOG_INIT("lxc.lxc_driver"); static int lxcStateInitialize(bool privileged, + const char *root, virStateInhibitCallback callback, void *opaque); static int lxcStateCleanup(void); @@ -1526,6 +1527,7 @@ lxcSecurityInit(virLXCDriverConfigPtr cfg) static int lxcStateInitialize(bool privileged, + const char *root, virStateInhibitCallback callback G_GNUC_UNUSED, void *opaque G_GNUC_UNUSED) { @@ -1533,6 +1535,12 @@ static int lxcStateInitialize(bool privileged, virLXCDriverConfigPtr cfg = NULL; bool autostart = true; + if (root != NULL) { + virReportError(VIR_ERR_INVALID_ARG, "%s", + _("Driver does not support embedded mode")); + return -1; + } + /* Check that the user is root, silently disable if not */ if (!privileged) { VIR_INFO("Not running privileged, disabling driver"); diff --git a/src/network/bridge_driver.c b/src/network/bridge_driver.c index e360645969..90f894bd78 100644 --- a/src/network/bridge_driver.c +++ b/src/network/bridge_driver.c @@ -702,6 +702,7 @@ firewalld_dbus_filter_bridge(DBusConnection *connection G_GNUC_UNUSED, */ static int networkStateInitialize(bool privileged, + const char *root, virStateInhibitCallback callback G_GNUC_UNUSED, void *opaque G_GNUC_UNUSED) { @@ -713,6 +714,12 @@ networkStateInitialize(bool privileged, DBusConnection *sysbus = NULL; #endif + if (root != NULL) { + virReportError(VIR_ERR_INVALID_ARG, "%s", + _("Driver does not support embedded mode")); + return -1; + } + if (VIR_ALLOC(network_driver) < 0) goto error; diff --git a/src/node_device/node_device_hal.c b/src/node_device/node_device_hal.c index b40f93df46..1cc484a4ce 100644 --- a/src/node_device/node_device_hal.c +++ b/src/node_device/node_device_hal.c @@ -588,6 +588,7 @@ device_prop_modified(LibHalContext *ctx G_GNUC_UNUSED, static int nodeStateInitialize(bool privileged G_GNUC_UNUSED, + const char *root, virStateInhibitCallback callback G_GNUC_UNUSED, void *opaque G_GNUC_UNUSED) { @@ -599,6 +600,12 @@ nodeStateInitialize(bool privileged G_GNUC_UNUSED, DBusConnection *sysbus; DBusError err; + if (root != NULL) { + virReportError(VIR_ERR_INVALID_ARG, "%s", + _("Driver does not support embedded mode")); + return -1; + } + /* Ensure caps_tbl is sorted by capability name */ qsort(caps_tbl, G_N_ELEMENTS(caps_tbl), sizeof(caps_tbl[0]), cmpstringp); diff --git a/src/node_device/node_device_udev.c b/src/node_device/node_device_udev.c index fabd2ec454..97d9982ac5 100644 --- a/src/node_device/node_device_udev.c +++ b/src/node_device/node_device_udev.c @@ -1782,6 +1782,7 @@ udevPCITranslateInit(bool privileged G_GNUC_UNUSED) static int nodeStateInitialize(bool privileged, + const char *root, virStateInhibitCallback callback G_GNUC_UNUSED, void *opaque G_GNUC_UNUSED) { @@ -1789,6 +1790,12 @@ nodeStateInitialize(bool privileged, struct udev *udev = NULL; virThread enumThread; + if (root != NULL) { + virReportError(VIR_ERR_INVALID_ARG, "%s", + _("Driver does not support embedded mode")); + return -1; + } + if (VIR_ALLOC(driver) < 0) return VIR_DRV_STATE_INIT_ERROR; diff --git a/src/nwfilter/nwfilter_driver.c b/src/nwfilter/nwfilter_driver.c index cc3ce98cc5..1c407727db 100644 --- a/src/nwfilter/nwfilter_driver.c +++ b/src/nwfilter/nwfilter_driver.c @@ -177,11 +177,18 @@ virNWFilterTriggerRebuildImpl(void *opaque) */ static int nwfilterStateInitialize(bool privileged, + const char *root, virStateInhibitCallback callback G_GNUC_UNUSED, void *opaque G_GNUC_UNUSED) { DBusConnection *sysbus = NULL; + if (root != NULL) { + virReportError(VIR_ERR_INVALID_ARG, "%s", + _("Driver does not support embedded mode")); + return -1; + } + if (virDBusHasSystemBus() && !(sysbus = virDBusGetSystemBus())) return VIR_DRV_STATE_INIT_ERROR; diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index b5300241a8..31fe921ee3 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -646,6 +646,7 @@ qemuDomainFindMaxID(virDomainObjPtr vm, */ static int qemuStateInitialize(bool privileged, + const char *root, virStateInhibitCallback callback, void *opaque) { @@ -657,6 +658,12 @@ qemuStateInitialize(bool privileged, bool autostart = true; size_t i; + if (root != NULL) { + virReportError(VIR_ERR_INVALID_ARG, "%s", + _("Driver does not support embedded mode")); + return -1; + } + if (VIR_ALLOC(qemu_driver) < 0) return VIR_DRV_STATE_INIT_ERROR; diff --git a/src/remote/remote_daemon.c b/src/remote/remote_daemon.c index b400b1dd10..f0a15279e0 100644 --- a/src/remote/remote_daemon.c +++ b/src/remote/remote_daemon.c @@ -834,6 +834,7 @@ static void daemonRunStateInit(void *opaque) * seriously delay OS bootup process */ if (virStateInitialize(virNetDaemonIsPrivileged(dmn), mandatory, + NULL, daemonInhibitCallback, dmn) < 0) { VIR_ERROR(_("Driver state initialization failed")); diff --git a/src/remote/remote_driver.c b/src/remote/remote_driver.c index a1384fc655..d8eb6e7833 100644 --- a/src/remote/remote_driver.c +++ b/src/remote/remote_driver.c @@ -236,6 +236,7 @@ static int remoteSplitURIScheme(virURIPtr uri, static int remoteStateInitialize(bool privileged G_GNUC_UNUSED, + const char *root G_GNUC_UNUSED, virStateInhibitCallback callback G_GNUC_UNUSED, void *opaque G_GNUC_UNUSED) { diff --git a/src/secret/secret_driver.c b/src/secret/secret_driver.c index 93b4256450..d248121327 100644 --- a/src/secret/secret_driver.c +++ b/src/secret/secret_driver.c @@ -452,9 +452,16 @@ secretStateCleanup(void) static int secretStateInitialize(bool privileged, + const char *root, virStateInhibitCallback callback G_GNUC_UNUSED, void *opaque G_GNUC_UNUSED) { + if (root != NULL) { + virReportError(VIR_ERR_INVALID_ARG, "%s", + _("Driver does not support embedded mode")); + return -1; + } + if (VIR_ALLOC(driver) < 0) return VIR_DRV_STATE_INIT_ERROR; diff --git a/src/storage/storage_driver.c b/src/storage/storage_driver.c index 580a5e6f15..6b3a36821b 100644 --- a/src/storage/storage_driver.c +++ b/src/storage/storage_driver.c @@ -251,6 +251,7 @@ storageDriverAutostart(void) */ static int storageStateInitialize(bool privileged, + const char *root, virStateInhibitCallback callback G_GNUC_UNUSED, void *opaque G_GNUC_UNUSED) { @@ -258,6 +259,12 @@ storageStateInitialize(bool privileged, g_autofree char *rundir = NULL; bool autostart = true; + if (root != NULL) { + virReportError(VIR_ERR_INVALID_ARG, "%s", + _("Driver does not support embedded mode")); + return -1; + } + if (VIR_ALLOC(driver) < 0) return VIR_DRV_STATE_INIT_ERROR; diff --git a/src/vz/vz_driver.c b/src/vz/vz_driver.c index 6b925653d5..67ea4f8786 100644 --- a/src/vz/vz_driver.c +++ b/src/vz/vz_driver.c @@ -4089,12 +4089,19 @@ vzStateCleanup(void) static int vzStateInitialize(bool privileged, + const char *root, virStateInhibitCallback callback G_GNUC_UNUSED, void *opaque G_GNUC_UNUSED) { if (!privileged) return VIR_DRV_STATE_INIT_SKIPPED; + if (root != NULL) { + virReportError(VIR_ERR_INVALID_ARG, "%s", + _("Driver does not support embedded mode")); + return -1; + } + vz_driver_privileged = privileged; if (virFileMakePathWithMode(VZ_STATEDIR, S_IRWXU) < 0) { -- 2.23.0
On 12/2/19 10:03 AM, Daniel P. Berrangé wrote:
The intent here is to allow the virt drivers to be run directly embedded in an arbitrary process without interfering with libvirtd. To achieve this they need to store all their configuration & state in a separate directory tree from the main system or session libvirtd instances.
This can be useful for doing testing of the virt drivers in "make check" without interfering with the user's own libvirtd instances.
It can also be used for applications using KVM/QEMU as a piece of infrastructure to build an service, rather than for general purpose OS hosting. A long standing example is libguestfs, which would prefer if its temporary VMs did show up in the main libvirtd VM list, because this confuses apps such as OpenStack Nova. A more recent example would be Kata which is using KVM as a technology to build containers.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Cole Robinson <crobinso@redhat.com> - Cole
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- po/POTFILES.in | 1 + src/util/virevent.c | 25 +++++++++++++++++++++++++ src/util/virevent.h | 2 ++ 3 files changed, 28 insertions(+) diff --git a/po/POTFILES.in b/po/POTFILES.in index debb51cd70..b396797ff2 100644 --- a/po/POTFILES.in +++ b/po/POTFILES.in @@ -238,6 +238,7 @@ @SRCDIR@/src/util/virdnsmasq.c @SRCDIR@/src/util/virerror.c @SRCDIR@/src/util/virerror.h +@SRCDIR@/src/util/virevent.c @SRCDIR@/src/util/vireventpoll.c @SRCDIR@/src/util/virfcp.c @SRCDIR@/src/util/virfdstream.c diff --git a/src/util/virevent.c b/src/util/virevent.c index 3cac9f9472..a86acf64c0 100644 --- a/src/util/virevent.c +++ b/src/util/virevent.c @@ -29,6 +29,9 @@ VIR_LOG_INIT("util.event"); + +#define VIR_FROM_THIS VIR_FROM_EVENT + static virEventAddHandleFunc addHandleImpl; static virEventUpdateHandleFunc updateHandleImpl; static virEventRemoveHandleFunc removeHandleImpl; @@ -251,6 +254,26 @@ void virEventRegisterImpl(virEventAddHandleFunc addHandle, removeTimeoutImpl = removeTimeout; } + +/** + * virEventRequireImpl: + * + * Require that there is an event loop implementation + * registered. + * + * Returns: -1 if no event loop is registered, 0 otherwise + */ +int virEventRequireImpl(void) +{ + if (!addHandleImpl || !addTimeoutImpl) { + virReportError(VIR_ERR_NO_SUPPORT, "%s", + _("An event loop implementation must be registered")); + return -1; + } + + return 0; +} + /** * virEventRegisterDefaultImpl: * @@ -276,6 +299,8 @@ int virEventRegisterDefaultImpl(void) { VIR_DEBUG("registering default event implementation"); + virInitialize(); + virResetLastError(); if (virEventPollInit() < 0) { diff --git a/src/util/virevent.h b/src/util/virevent.h index 0b4f7d2b1b..6c8fdcbac4 100644 --- a/src/util/virevent.h +++ b/src/util/virevent.h @@ -21,3 +21,5 @@ #pragma once #include "internal.h" + +int virEventRequireImpl(void); -- 2.23.0
On 12/2/19 10:03 AM, Daniel P. Berrangé wrote:
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- po/POTFILES.in | 1 + src/util/virevent.c | 25 +++++++++++++++++++++++++ src/util/virevent.h | 2 ++ 3 files changed, 28 insertions(+)
diff --git a/po/POTFILES.in b/po/POTFILES.in index debb51cd70..b396797ff2 100644 --- a/po/POTFILES.in +++ b/po/POTFILES.in @@ -238,6 +238,7 @@ @SRCDIR@/src/util/virdnsmasq.c @SRCDIR@/src/util/virerror.c @SRCDIR@/src/util/virerror.h +@SRCDIR@/src/util/virevent.c @SRCDIR@/src/util/vireventpoll.c @SRCDIR@/src/util/virfcp.c @SRCDIR@/src/util/virfdstream.c diff --git a/src/util/virevent.c b/src/util/virevent.c index 3cac9f9472..a86acf64c0 100644 --- a/src/util/virevent.c +++ b/src/util/virevent.c @@ -29,6 +29,9 @@
VIR_LOG_INIT("util.event");
+ +#define VIR_FROM_THIS VIR_FROM_EVENT +
Other files I looked at, this comes before VIR_LOG_INIT, but I don't think that has any effect
static virEventAddHandleFunc addHandleImpl; static virEventUpdateHandleFunc updateHandleImpl; static virEventRemoveHandleFunc removeHandleImpl; @@ -251,6 +254,26 @@ void virEventRegisterImpl(virEventAddHandleFunc addHandle, removeTimeoutImpl = removeTimeout; }
+ +/** + * virEventRequireImpl: + * + * Require that there is an event loop implementation + * registered. + * + * Returns: -1 if no event loop is registered, 0 otherwise + */ +int virEventRequireImpl(void) +{ + if (!addHandleImpl || !addTimeoutImpl) { + virReportError(VIR_ERR_NO_SUPPORT, "%s", + _("An event loop implementation must be registered")); + return -1; + } + + return 0; +} + /**
Spacing is inconsistent here, there's two lines before the function and one after Reviewed-by: Cole Robinson <crobinso@redhat.com> - Cole
On 12/2/19 10:03 AM, Daniel P. Berrangé wrote:
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- po/POTFILES.in | 1 + src/util/virevent.c | 25 +++++++++++++++++++++++++ src/util/virevent.h | 2 ++ 3 files changed, 28 insertions(+)
diff --git a/po/POTFILES.in b/po/POTFILES.in index debb51cd70..b396797ff2 100644 --- a/po/POTFILES.in +++ b/po/POTFILES.in @@ -238,6 +238,7 @@ @SRCDIR@/src/util/virdnsmasq.c @SRCDIR@/src/util/virerror.c @SRCDIR@/src/util/virerror.h +@SRCDIR@/src/util/virevent.c @SRCDIR@/src/util/vireventpoll.c @SRCDIR@/src/util/virfcp.c @SRCDIR@/src/util/virfdstream.c diff --git a/src/util/virevent.c b/src/util/virevent.c index 3cac9f9472..a86acf64c0 100644 --- a/src/util/virevent.c +++ b/src/util/virevent.c @@ -29,6 +29,9 @@
VIR_LOG_INIT("util.event");
+ +#define VIR_FROM_THIS VIR_FROM_EVENT + static virEventAddHandleFunc addHandleImpl; static virEventUpdateHandleFunc updateHandleImpl; static virEventRemoveHandleFunc removeHandleImpl; @@ -251,6 +254,26 @@ void virEventRegisterImpl(virEventAddHandleFunc addHandle, removeTimeoutImpl = removeTimeout; }
+ +/** + * virEventRequireImpl: + * + * Require that there is an event loop implementation + * registered. + * + * Returns: -1 if no event loop is registered, 0 otherwise + */ +int virEventRequireImpl(void) +{ + if (!addHandleImpl || !addTimeoutImpl) { + virReportError(VIR_ERR_NO_SUPPORT, "%s", + _("An event loop implementation must be registered")); + return -1; + } + + return 0; +} +
I just noticed, maybe VIR_ERR_NO_SUPPORT isn't the best thing here, since this is more a client config issue if it didn't register an event loop impl - Cole
The driver URI scheme: "$drivername:///embed?root=/some/path" enables a new way to use the drivers by embedding them directly in the calling process. To use this the process must have a thread running the libvirt event loop. This URI will then cause libvirt to dynamically load the driver module and call its global initialization function. This syntax is applicable to any driver, but only those will have been modified to support a custom root directory and embed URI path will successfully open. The application can now make normal libvirt API calls which are all serviced in-process with no RPC layer involved. It is required to specify an explicit root directory, and locks will be acquired on this directory to avoid conflicting with another app that might accidentally pick the same directory. Use of '/' is not explicitly forbidden, but note that the file layout used underneath the embedded driver root does not match the file layout used by system/session mode drivers. So this cannot be used as a backdoor to interact with, or fake, the system/session mode drivers. Libvirt will create arbitrary files underneath this root directory. The root directory can be kept untouched across connection open attempts if the application needs persistence. The application is responsible for purging everything underneath this root directory when finally no longer required. Even when a virt driver is used in embedded mode, it is still possible for it to in turn use functionality that calls out to other secondary drivers in libvirtd. For example an embedded instance of QEMU can open the network, secret or storage drivers in the system libvirtd. That said, the application would typically want to at least open an embedded secret driver ("secret:///embed?root=/some/path"). Note that multiple different embedded drivers can use the same root prefix and co-operate just as they would inside a normal libvirtd daemon. A key thing to note is that for this to work, the application that links to libvirt *MUST* be built with -Wl,--export-dynamic to ensure that symbols from libvirt.so are exported & thus available to the dynamically loaded driver module. If libvirt.so itself was dynamically loaded then RTLD_GLOBAL must be passed to dlopen(). Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/driver-state.h | 1 + src/driver.h | 2 ++ src/libvirt.c | 72 ++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 73 insertions(+), 2 deletions(-) diff --git a/src/driver-state.h b/src/driver-state.h index 1e2f6ed247..6b3f501e05 100644 --- a/src/driver-state.h +++ b/src/driver-state.h @@ -50,6 +50,7 @@ typedef virStateDriver *virStateDriverPtr; struct _virStateDriver { const char *name; + bool initialized; virDrvStateInitialize stateInitialize; virDrvStateCleanup stateCleanup; virDrvStateReload stateReload; diff --git a/src/driver.h b/src/driver.h index ca82ac974b..6278aa05b3 100644 --- a/src/driver.h +++ b/src/driver.h @@ -82,6 +82,8 @@ struct _virConnectDriver { bool localOnly; /* Whether driver needs a server in the URI */ bool remoteOnly; + /* Whether driver can be used in embedded mode */ + bool embeddable; /* * NULL terminated list of supported URI schemes. * - Single element { NULL } list indicates no supported schemes diff --git a/src/libvirt.c b/src/libvirt.c index bd2952d036..17b6506faa 100644 --- a/src/libvirt.c +++ b/src/libvirt.c @@ -52,6 +52,7 @@ # include "rpc/virnettlscontext.h" #endif #include "vircommand.h" +#include "virevent.h" #include "virfile.h" #include "virrandom.h" #include "viruri.h" @@ -84,6 +85,7 @@ #ifdef WITH_BHYVE # include "bhyve/bhyve_driver.h" #endif +#include "access/viraccessmanager.h" #define VIR_FROM_THIS VIR_FROM_NONE @@ -676,10 +678,12 @@ virStateInitialize(bool privileged, return -1; for (i = 0; i < virStateDriverTabCount; i++) { - if (virStateDriverTab[i]->stateInitialize) { + if (virStateDriverTab[i]->stateInitialize && + !virStateDriverTab[i]->initialized) { virDrvStateInitResult ret; VIR_DEBUG("Running global init for %s state driver", virStateDriverTab[i]->name); + virStateDriverTab[i]->initialized = true; ret = virStateDriverTab[i]->stateInitialize(privileged, root, callback, @@ -872,6 +876,7 @@ virConnectOpenInternal(const char *name, virConnectPtr ret; g_autoptr(virConf) conf = NULL; char *uristr = NULL; + bool embed = false; ret = virGetConnect(); if (ret == NULL) @@ -962,6 +967,52 @@ virConnectOpenInternal(const char *name, ret->uri) < 0) { goto failed; } + + if (STREQ(ret->uri->path, "/embed")) { + const char *root = NULL; + g_autofree char *regMethod = NULL; + VIR_DEBUG("URI path requests %s driver embedded mode", + ret->uri->scheme); + if (strspn(ret->uri->scheme, "abcdefghijklmnopqrstuvwxyz") != + strlen(ret->uri->scheme)) { + virReportError(VIR_ERR_NO_CONNECT, + _("URI scheme '%s' for embedded driver is not valid"), + ret->uri->scheme); + goto failed; + } + + for (i = 0; i < ret->uri->paramsCount; i++) { + virURIParamPtr var = &ret->uri->params[i]; + if (STREQ(var->name, "root")) + root = var->value; + } + + if (!root) { + virReportError(VIR_ERR_INVALID_ARG, "%s", + _("root parameter required for embedded driver")); + goto failed; + } + + if (virEventRequireImpl() < 0) + goto failed; + + regMethod = g_strdup_printf("%sRegister", ret->uri->scheme); + + if (virDriverLoadModule(ret->uri->scheme, regMethod, false) < 0) + goto failed; + + if (virAccessManagerGetDefault() == NULL) { + virAccessManagerPtr acl = virAccessManagerNew("none"); + if (!acl) + goto failed; + virAccessManagerSetDefault(acl); + } + + if (virStateInitialize(geteuid() == 0, true, root, NULL, NULL) < 0) + goto failed; + + embed = true; + } } else { VIR_DEBUG("no name, allowing driver auto-select"); } @@ -1018,6 +1069,12 @@ virConnectOpenInternal(const char *name, VIR_DEBUG("No URI, skipping driver with URI whitelist"); continue; } + if (embed && !virConnectDriverTab[i]->embeddable) { + VIR_DEBUG("Ignoring non-embeddable driver %s", + virConnectDriverTab[i]->hypervisorDriver->name); + continue; + } + VIR_DEBUG("Checking for supported URI schemes"); for (s = 0; virConnectDriverTab[i]->uriSchemes[s] != NULL; s++) { if (STREQ(ret->uri->scheme, virConnectDriverTab[i]->uriSchemes[s])) { @@ -1031,9 +1088,20 @@ virConnectOpenInternal(const char *name, continue; } } else { - VIR_DEBUG("Matching any URI scheme for '%s'", ret->uri ? ret->uri->scheme : ""); + if (embed) { + VIR_DEBUG("Skipping wildcard for embedded URI"); + continue; + } else { + VIR_DEBUG("Matching any URI scheme for '%s'", ret->uri ? ret->uri->scheme : ""); + } } + if (embed && !virConnectDriverTab[i]->embeddable) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, + _("Driver %s cannot be used in embedded mode"), + virConnectDriverTab[i]->hypervisorDriver->name); + goto failed; + } /* before starting the new connection, check if the driver only works * with a server, and so return an error if the server is missing */ if (virConnectDriverTab[i]->remoteOnly && ret->uri && !ret->uri->server) { -- 2.23.0
On 12/2/19 10:03 AM, Daniel P. Berrangé wrote:
The driver URI scheme:
"$drivername:///embed?root=/some/path"
enables a new way to use the drivers by embedding them directly in the calling process. To use this the process must have a thread running the libvirt event loop. This URI will then cause libvirt to dynamically load the driver module and call its global initialization function. This syntax is applicable to any driver, but only those will have been modified to support a custom root directory and embed URI path will successfully open.
The application can now make normal libvirt API calls which are all serviced in-process with no RPC layer involved.
It is required to specify an explicit root directory, and locks will be acquired on this directory to avoid conflicting with another app that might accidentally pick the same directory.
Use of '/' is not explicitly forbidden, but note that the file layout used underneath the embedded driver root does not match the file layout used by system/session mode drivers. So this cannot be used as a backdoor to interact with, or fake, the system/session mode drivers.
Libvirt will create arbitrary files underneath this root directory. The root directory can be kept untouched across connection open attempts if the application needs persistence. The application is responsible for purging everything underneath this root directory when finally no longer required.
Even when a virt driver is used in embedded mode, it is still possible for it to in turn use functionality that calls out to other secondary drivers in libvirtd. For example an embedded instance of QEMU can open the network, secret or storage drivers in the system libvirtd.
That said, the application would typically want to at least open an embedded secret driver ("secret:///embed?root=/some/path"). Note that multiple different embedded drivers can use the same root prefix and co-operate just as they would inside a normal libvirtd daemon.
A key thing to note is that for this to work, the application that links to libvirt *MUST* be built with -Wl,--export-dynamic to ensure that symbols from libvirt.so are exported & thus available to the dynamically loaded driver module. If libvirt.so itself was dynamically loaded then RTLD_GLOBAL must be passed to dlopen().
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/driver-state.h | 1 + src/driver.h | 2 ++ src/libvirt.c | 72 ++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 73 insertions(+), 2 deletions(-)
diff --git a/src/driver-state.h b/src/driver-state.h index 1e2f6ed247..6b3f501e05 100644 --- a/src/driver-state.h +++ b/src/driver-state.h @@ -50,6 +50,7 @@ typedef virStateDriver *virStateDriverPtr;
struct _virStateDriver { const char *name; + bool initialized; virDrvStateInitialize stateInitialize; virDrvStateCleanup stateCleanup; virDrvStateReload stateReload; diff --git a/src/driver.h b/src/driver.h index ca82ac974b..6278aa05b3 100644 --- a/src/driver.h +++ b/src/driver.h @@ -82,6 +82,8 @@ struct _virConnectDriver { bool localOnly; /* Whether driver needs a server in the URI */ bool remoteOnly; + /* Whether driver can be used in embedded mode */ + bool embeddable; /* * NULL terminated list of supported URI schemes. * - Single element { NULL } list indicates no supported schemes diff --git a/src/libvirt.c b/src/libvirt.c index bd2952d036..17b6506faa 100644 --- a/src/libvirt.c +++ b/src/libvirt.c @@ -52,6 +52,7 @@ # include "rpc/virnettlscontext.h" #endif #include "vircommand.h" +#include "virevent.h" #include "virfile.h" #include "virrandom.h" #include "viruri.h" @@ -84,6 +85,7 @@ #ifdef WITH_BHYVE # include "bhyve/bhyve_driver.h" #endif +#include "access/viraccessmanager.h"
#define VIR_FROM_THIS VIR_FROM_NONE
@@ -676,10 +678,12 @@ virStateInitialize(bool privileged, return -1;
for (i = 0; i < virStateDriverTabCount; i++) { - if (virStateDriverTab[i]->stateInitialize) { + if (virStateDriverTab[i]->stateInitialize && + !virStateDriverTab[i]->initialized) { virDrvStateInitResult ret; VIR_DEBUG("Running global init for %s state driver", virStateDriverTab[i]->name); + virStateDriverTab[i]->initialized = true; ret = virStateDriverTab[i]->stateInitialize(privileged, root, callback, @@ -872,6 +876,7 @@ virConnectOpenInternal(const char *name, virConnectPtr ret; g_autoptr(virConf) conf = NULL; char *uristr = NULL; + bool embed = false;
ret = virGetConnect(); if (ret == NULL) @@ -962,6 +967,52 @@ virConnectOpenInternal(const char *name, ret->uri) < 0) { goto failed; } + + if (STREQ(ret->uri->path, "/embed")) { + const char *root = NULL; + g_autofree char *regMethod = NULL; + VIR_DEBUG("URI path requests %s driver embedded mode", + ret->uri->scheme); + if (strspn(ret->uri->scheme, "abcdefghijklmnopqrstuvwxyz") != + strlen(ret->uri->scheme)) { + virReportError(VIR_ERR_NO_CONNECT, + _("URI scheme '%s' for embedded driver is not valid"), + ret->uri->scheme); + goto failed; + } + + for (i = 0; i < ret->uri->paramsCount; i++) { + virURIParamPtr var = &ret->uri->params[i]; + if (STREQ(var->name, "root")) + root = var->value; + } + + if (!root) { + virReportError(VIR_ERR_INVALID_ARG, "%s", + _("root parameter required for embedded driver")); + goto failed; + } + + if (virEventRequireImpl() < 0) + goto failed; + + regMethod = g_strdup_printf("%sRegister", ret->uri->scheme); + + if (virDriverLoadModule(ret->uri->scheme, regMethod, false) < 0) + goto failed; + + if (virAccessManagerGetDefault() == NULL) { + virAccessManagerPtr acl = virAccessManagerNew("none"); + if (!acl) + goto failed; + virAccessManagerSetDefault(acl); + } + + if (virStateInitialize(geteuid() == 0, true, root, NULL, NULL) < 0) + goto failed; + + embed = true; + }
It would be nice if this logic was moved to a separate function I've hit a couple issues in testing, not sure if/where the fixes will live, so I'll just mention them here. Also the reviewed patches are pushable IMO I tried this code: from gi.repository import LibvirtGLib import libvirt LibvirtGLib.init(None) LibvirtGLib.event_register() conn1 = libvirt.open("qemu:///embed?root=/tmp/foo") conn2 = libvirt.open("qemu:///embed?root=/tmp/bar") print(conn1.listAllDomains()) print(conn2.listAllDomains()) With /tmp/foo populated with a VM: both connections see the same values. So this should be rejected. Even trying to close conn1 fully and open a new embed root will only see the old root. So maybe this needs knowledge in the driver lookup. The second issue: testing with virt-manager, everything locks up with OpenGraphicsFD: #0 0x00007ffff7a7f07a in pthread_cond_timedwait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0 #1 0x00007fffe94be113 in virCondWaitUntil (c=c@entry=0x7fffc8071f98, m=m@entry=0x7fffc8071ed0, whenms=whenms@entry=1576602286073) at /home/crobinso/src/libvirt/src/util/virthread.c:159 #2 0x00007fffe44fc549 in qemuDomainObjBeginJobInternal (driver=driver@entry=0x7fffc8004f30, obj=0x7fffc8071ec0, job=job@entry=QEMU_JOB_MODIFY, agentJob=agentJob@entry=QEMU_AGENT_JOB_NONE, asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_NONE, nowait=nowait@entry=false) at /home/crobinso/src/libvirt/src/qemu/qemu_domain.c:9357 #3 0x00007fffe4500aa1 in qemuDomainObjBeginJob (driver=driver@entry=0x7fffc8004f30, obj=<optimized out>, job=job@entry=QEMU_JOB_MODIFY) at /home/crobinso/src/libvirt/src/qemu/qemu_domain.c:9521 #4 0x00007fffe4582572 in qemuDomainOpenGraphicsFD (dom=<optimized out>, idx=<optimized out>, flags=0) at /home/crobinso/src/libvirt/src/qemu/qemu_driver.c:18990 #5 0x00007fffe968699c in virDomainOpenGraphicsFD (dom=0x7fffd0005830, idx=0, flags=0) at /home/crobinso/src/libvirt/src/libvirt-domain.c:10664 #6 0x00007fffe98cc6b1 in libvirt_virDomainOpenGraphicsFD () at /usr/lib64/python3.7/site-packages/libvirtmod.cpython-37m-x86_64-linux-gnu.so I didn't dig into it any more than that. Otherwise in some mild testing I'm surprised how much things seemed to 'just work' :) - Cole
On 12/17/19 6:41 PM, Cole Robinson wrote:
On 12/2/19 10:03 AM, Daniel P. Berrangé wrote:
The driver URI scheme:
"$drivername:///embed?root=/some/path"
enables a new way to use the drivers by embedding them directly in the calling process. To use this the process must have a thread running the libvirt event loop. This URI will then cause libvirt to dynamically load the driver module and call its global initialization function. This syntax is applicable to any driver, but only those will have been modified to support a custom root directory and embed URI path will successfully open.
The application can now make normal libvirt API calls which are all serviced in-process with no RPC layer involved.
It is required to specify an explicit root directory, and locks will be acquired on this directory to avoid conflicting with another app that might accidentally pick the same directory.
Use of '/' is not explicitly forbidden, but note that the file layout used underneath the embedded driver root does not match the file layout used by system/session mode drivers. So this cannot be used as a backdoor to interact with, or fake, the system/session mode drivers.
Libvirt will create arbitrary files underneath this root directory. The root directory can be kept untouched across connection open attempts if the application needs persistence. The application is responsible for purging everything underneath this root directory when finally no longer required.
Even when a virt driver is used in embedded mode, it is still possible for it to in turn use functionality that calls out to other secondary drivers in libvirtd. For example an embedded instance of QEMU can open the network, secret or storage drivers in the system libvirtd.
That said, the application would typically want to at least open an embedded secret driver ("secret:///embed?root=/some/path"). Note that multiple different embedded drivers can use the same root prefix and co-operate just as they would inside a normal libvirtd daemon.
A key thing to note is that for this to work, the application that links to libvirt *MUST* be built with -Wl,--export-dynamic to ensure that symbols from libvirt.so are exported & thus available to the dynamically loaded driver module. If libvirt.so itself was dynamically loaded then RTLD_GLOBAL must be passed to dlopen().
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/driver-state.h | 1 + src/driver.h | 2 ++ src/libvirt.c | 72 ++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 73 insertions(+), 2 deletions(-)
diff --git a/src/driver-state.h b/src/driver-state.h index 1e2f6ed247..6b3f501e05 100644 --- a/src/driver-state.h +++ b/src/driver-state.h @@ -50,6 +50,7 @@ typedef virStateDriver *virStateDriverPtr;
struct _virStateDriver { const char *name; + bool initialized; virDrvStateInitialize stateInitialize; virDrvStateCleanup stateCleanup; virDrvStateReload stateReload; diff --git a/src/driver.h b/src/driver.h index ca82ac974b..6278aa05b3 100644 --- a/src/driver.h +++ b/src/driver.h @@ -82,6 +82,8 @@ struct _virConnectDriver { bool localOnly; /* Whether driver needs a server in the URI */ bool remoteOnly; + /* Whether driver can be used in embedded mode */ + bool embeddable; /* * NULL terminated list of supported URI schemes. * - Single element { NULL } list indicates no supported schemes diff --git a/src/libvirt.c b/src/libvirt.c index bd2952d036..17b6506faa 100644 --- a/src/libvirt.c +++ b/src/libvirt.c @@ -52,6 +52,7 @@ # include "rpc/virnettlscontext.h" #endif #include "vircommand.h" +#include "virevent.h" #include "virfile.h" #include "virrandom.h" #include "viruri.h" @@ -84,6 +85,7 @@ #ifdef WITH_BHYVE # include "bhyve/bhyve_driver.h" #endif +#include "access/viraccessmanager.h"
#define VIR_FROM_THIS VIR_FROM_NONE
@@ -676,10 +678,12 @@ virStateInitialize(bool privileged, return -1;
for (i = 0; i < virStateDriverTabCount; i++) { - if (virStateDriverTab[i]->stateInitialize) { + if (virStateDriverTab[i]->stateInitialize && + !virStateDriverTab[i]->initialized) { virDrvStateInitResult ret; VIR_DEBUG("Running global init for %s state driver", virStateDriverTab[i]->name); + virStateDriverTab[i]->initialized = true; ret = virStateDriverTab[i]->stateInitialize(privileged, root, callback, @@ -872,6 +876,7 @@ virConnectOpenInternal(const char *name, virConnectPtr ret; g_autoptr(virConf) conf = NULL; char *uristr = NULL; + bool embed = false;
ret = virGetConnect(); if (ret == NULL) @@ -962,6 +967,52 @@ virConnectOpenInternal(const char *name, ret->uri) < 0) { goto failed; } + + if (STREQ(ret->uri->path, "/embed")) { + const char *root = NULL; + g_autofree char *regMethod = NULL; + VIR_DEBUG("URI path requests %s driver embedded mode", + ret->uri->scheme); + if (strspn(ret->uri->scheme, "abcdefghijklmnopqrstuvwxyz") != + strlen(ret->uri->scheme)) { + virReportError(VIR_ERR_NO_CONNECT, + _("URI scheme '%s' for embedded driver is not valid"), + ret->uri->scheme); + goto failed; + } + + for (i = 0; i < ret->uri->paramsCount; i++) { + virURIParamPtr var = &ret->uri->params[i]; + if (STREQ(var->name, "root")) + root = var->value; + } + + if (!root) { + virReportError(VIR_ERR_INVALID_ARG, "%s", + _("root parameter required for embedded driver")); + goto failed; + } + + if (virEventRequireImpl() < 0) + goto failed; + + regMethod = g_strdup_printf("%sRegister", ret->uri->scheme); + + if (virDriverLoadModule(ret->uri->scheme, regMethod, false) < 0) + goto failed; + + if (virAccessManagerGetDefault() == NULL) { + virAccessManagerPtr acl = virAccessManagerNew("none"); + if (!acl) + goto failed; + virAccessManagerSetDefault(acl); + } + + if (virStateInitialize(geteuid() == 0, true, root, NULL, NULL) < 0) + goto failed; + + embed = true; + }
It would be nice if this logic was moved to a separate function
I've hit a couple issues in testing, not sure if/where the fixes will live, so I'll just mention them here. Also the reviewed patches are pushable IMO
I tried this code:
from gi.repository import LibvirtGLib
import libvirt
LibvirtGLib.init(None)
LibvirtGLib.event_register()
conn1 = libvirt.open("qemu:///embed?root=/tmp/foo")
conn2 = libvirt.open("qemu:///embed?root=/tmp/bar")
print(conn1.listAllDomains())
print(conn2.listAllDomains())
I didn't get this far, when I wanted to run virt-install all I can see is the following error: [Tue, 17 Dec 2019 20:22:39 virt-install 129850] ERROR (cli:259) this function is not supported by the connection driver: An event loop implementation must be registered which I tracked to here: <no frame> <no attribute num on current thread> $ r /home/zippy/work/virt-manager.git/virt-install --connect "qemu:///embed?root=/tmp/a" --name f31-uefi --ram 2048 --disk none --boot uefi --import --debug Starting program: /usr/bin/python /home/zippy/work/virt-manager.git/virt-install --connect "qemu:///embed?root=/tmp/a" --name f31-uefi --ram 2048 --disk none --boot uefi --import --debug process 129850 is executing new program: /usr/bin/python3.6m [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [Tue, 17 Dec 2019 20:21:34 virt-install 129850] DEBUG (cli:203) Launched with command line: /home/zippy/work/virt-manager.git/virt-install --connect qemu:///embed?root=/tmp/a --name f31-uefi --ram 2048 --disk none --boot uefi --import --debug [Tue, 17 Dec 2019 20:21:34 virt-install 129850] DEBUG (virt-install:208) Distilled --network options: ['default'] [Tue, 17 Dec 2019 20:21:34 virt-install 129850] DEBUG (virt-install:140) Distilled --disk options: ['none'] [Tue, 17 Dec 2019 20:21:34 virt-install 129850] DEBUG (cli:219) Requesting libvirt URI qemu:///embed?root=/tmp/a Breakpoint 1, virEventRequireImpl () at ../../src/util/virevent.c:268 268 if (!addHandleImpl || !addTimeoutImpl) { virEventRequireImpl 1 $ bt #0 0x00007ffff693ca53 in virEventRequireImpl () at ../../src/util/virevent.c:268 #1 0x00007ffff6bd2c33 in virConnectOpenInternal (name=0x7ffff710bdd0 "qemu:///embed?root=/tmp/a", auth=0x7fffffffc8a0, flags=0) at ../../src/libvirt.c:996 #2 0x00007ffff6bd38ab in virConnectOpenAuth (name=0x7ffff710bdd0 "qemu:///embed?root=/tmp/a", auth=0x7fffffffc8a0, flags=0) at ../../src/libvirt.c:1272 #3 0x00007ffff6e17b55 in libvirt_virConnectOpenAuth () at /usr/lib64/python3.6/site-packages/libvirtmod.cpython-36m-x86_64-linux-gnu.so But this looks weird, isn't virt-install registerin an event loop? How else does it get events? Michal
On 12/17/19 2:28 PM, Michal Prívozník wrote:
But this looks weird, isn't virt-install registerin an event loop? How else does it get events?
virt-install doesn't register an event loop, it polls for the single VM state change we care about. But if registering an event loop is necessary for qemu:///embed usage then it's worth adding the libvirtglib init to virt-install, if only for testing purposes. - Cole
On Tue, Dec 17, 2019 at 12:41:27PM -0500, Cole Robinson wrote:
On 12/2/19 10:03 AM, Daniel P. Berrangé wrote:
The driver URI scheme:
"$drivername:///embed?root=/some/path"
enables a new way to use the drivers by embedding them directly in the calling process. To use this the process must have a thread running the libvirt event loop. This URI will then cause libvirt to dynamically load the driver module and call its global initialization function. This syntax is applicable to any driver, but only those will have been modified to support a custom root directory and embed URI path will successfully open.
The application can now make normal libvirt API calls which are all serviced in-process with no RPC layer involved.
It is required to specify an explicit root directory, and locks will be acquired on this directory to avoid conflicting with another app that might accidentally pick the same directory.
Use of '/' is not explicitly forbidden, but note that the file layout used underneath the embedded driver root does not match the file layout used by system/session mode drivers. So this cannot be used as a backdoor to interact with, or fake, the system/session mode drivers.
Libvirt will create arbitrary files underneath this root directory. The root directory can be kept untouched across connection open attempts if the application needs persistence. The application is responsible for purging everything underneath this root directory when finally no longer required.
Even when a virt driver is used in embedded mode, it is still possible for it to in turn use functionality that calls out to other secondary drivers in libvirtd. For example an embedded instance of QEMU can open the network, secret or storage drivers in the system libvirtd.
That said, the application would typically want to at least open an embedded secret driver ("secret:///embed?root=/some/path"). Note that multiple different embedded drivers can use the same root prefix and co-operate just as they would inside a normal libvirtd daemon.
A key thing to note is that for this to work, the application that links to libvirt *MUST* be built with -Wl,--export-dynamic to ensure that symbols from libvirt.so are exported & thus available to the dynamically loaded driver module. If libvirt.so itself was dynamically loaded then RTLD_GLOBAL must be passed to dlopen().
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/driver-state.h | 1 + src/driver.h | 2 ++ src/libvirt.c | 72 ++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 73 insertions(+), 2 deletions(-)
It would be nice if this logic was moved to a separate function
I've hit a couple issues in testing, not sure if/where the fixes will live, so I'll just mention them here. Also the reviewed patches are pushable IMO
I tried this code:
from gi.repository import LibvirtGLib
import libvirt
LibvirtGLib.init(None)
LibvirtGLib.event_register()
conn1 = libvirt.open("qemu:///embed?root=/tmp/foo")
conn2 = libvirt.open("qemu:///embed?root=/tmp/bar")
print(conn1.listAllDomains())
print(conn2.listAllDomains())
With /tmp/foo populated with a VM: both connections see the same values. So this should be rejected. Even trying to close conn1 fully and open a new embed root will only see the old root. So maybe this needs knowledge in the driver lookup.
Right, so because of our design with the single global driver struct, we can only have 1 instance of the driver active per process. So yeah, we need to enforce this in some place or other. It should be valid to open multiple connections to the same embedded driver instance directory though.
The second issue: testing with virt-manager, everything locks up with OpenGraphicsFD:
#0 0x00007ffff7a7f07a in pthread_cond_timedwait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0 #1 0x00007fffe94be113 in virCondWaitUntil (c=c@entry=0x7fffc8071f98, m=m@entry=0x7fffc8071ed0, whenms=whenms@entry=1576602286073) at /home/crobinso/src/libvirt/src/util/virthread.c:159 #2 0x00007fffe44fc549 in qemuDomainObjBeginJobInternal (driver=driver@entry=0x7fffc8004f30, obj=0x7fffc8071ec0, job=job@entry=QEMU_JOB_MODIFY, agentJob=agentJob@entry=QEMU_AGENT_JOB_NONE, asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_NONE, nowait=nowait@entry=false) at /home/crobinso/src/libvirt/src/qemu/qemu_domain.c:9357 #3 0x00007fffe4500aa1 in qemuDomainObjBeginJob (driver=driver@entry=0x7fffc8004f30, obj=<optimized out>, job=job@entry=QEMU_JOB_MODIFY) at /home/crobinso/src/libvirt/src/qemu/qemu_domain.c:9521 #4 0x00007fffe4582572 in qemuDomainOpenGraphicsFD (dom=<optimized out>, idx=<optimized out>, flags=0) at /home/crobinso/src/libvirt/src/qemu/qemu_driver.c:18990 #5 0x00007fffe968699c in virDomainOpenGraphicsFD (dom=0x7fffd0005830, idx=0, flags=0) at /home/crobinso/src/libvirt/src/libvirt-domain.c:10664 #6 0x00007fffe98cc6b1 in libvirt_virDomainOpenGraphicsFD () at /usr/lib64/python3.7/site-packages/libvirtmod.cpython-37m-x86_64-linux-gnu.so
I didn't dig into it any more than that. Otherwise in some mild testing I'm surprised how much things seemed to 'just work' :)
Hard to tell but my guess is that there's probably some event loop interaction causing us problems. Obviously this stack trace is waiting for the job lock. I'm guessing that the job lock is held by another API call made from the event loop thread. Will investigate it some more. In libvirtd no libvirt APIs calls are ever made from the event loop thread, they can only be made from worker threads. We'll need to make it clear that apps using the embedded driver must likewise ensure that libvirt APIs calls are *NEVER* made from the event loop thread. This is good practice even when using traditional libvirt connections, because the RPC calls to libvirtd can take an arbitrary amount of time to complete & so make the event loop response very poor. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Dec 19, 2019 at 11:15:50AM +0000, Daniel P. Berrangé wrote:
On Tue, Dec 17, 2019 at 12:41:27PM -0500, Cole Robinson wrote:
The second issue: testing with virt-manager, everything locks up with OpenGraphicsFD:
#0 0x00007ffff7a7f07a in pthread_cond_timedwait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0 #1 0x00007fffe94be113 in virCondWaitUntil (c=c@entry=0x7fffc8071f98, m=m@entry=0x7fffc8071ed0, whenms=whenms@entry=1576602286073) at /home/crobinso/src/libvirt/src/util/virthread.c:159 #2 0x00007fffe44fc549 in qemuDomainObjBeginJobInternal (driver=driver@entry=0x7fffc8004f30, obj=0x7fffc8071ec0, job=job@entry=QEMU_JOB_MODIFY, agentJob=agentJob@entry=QEMU_AGENT_JOB_NONE, asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_NONE, nowait=nowait@entry=false) at /home/crobinso/src/libvirt/src/qemu/qemu_domain.c:9357 #3 0x00007fffe4500aa1 in qemuDomainObjBeginJob (driver=driver@entry=0x7fffc8004f30, obj=<optimized out>, job=job@entry=QEMU_JOB_MODIFY) at /home/crobinso/src/libvirt/src/qemu/qemu_domain.c:9521 #4 0x00007fffe4582572 in qemuDomainOpenGraphicsFD (dom=<optimized out>, idx=<optimized out>, flags=0) at /home/crobinso/src/libvirt/src/qemu/qemu_driver.c:18990 #5 0x00007fffe968699c in virDomainOpenGraphicsFD (dom=0x7fffd0005830, idx=0, flags=0) at /home/crobinso/src/libvirt/src/libvirt-domain.c:10664 #6 0x00007fffe98cc6b1 in libvirt_virDomainOpenGraphicsFD () at /usr/lib64/python3.7/site-packages/libvirtmod.cpython-37m-x86_64-linux-gnu.so
I didn't dig into it any more than that. Otherwise in some mild testing I'm surprised how much things seemed to 'just work' :)
Hard to tell but my guess is that there's probably some event loop interaction causing us problems. Obviously this stack trace is waiting for the job lock. I'm guessing that the job lock is held by another API call made from the event loop thread. Will investigate it some more.
In libvirtd no libvirt APIs calls are ever made from the event loop thread, they can only be made from worker threads. We'll need to make it clear that apps using the embedded driver must likewise ensure that libvirt APIs calls are *NEVER* made from the event loop thread. This is good practice even when using traditional libvirt connections, because the RPC calls to libvirtd can take an arbitrary amount of time to complete & so make the event loop response very poor.
I reproduced this issue and can confirm that it is indeed caused by calling virDomainOpenGraphicsFD from the main event loop. THis call sends a QMP monitor command and I/O for the monitor is performed by the event loop. So we're maining for QMP send that will enver happen since virtman is blocking the event loop. This is quite tedious to fix with the way virt-manager is written using the libvirt python bindings. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
This enables support for running QEMU embedded to the calling application process using a URI: qemu:///embed?root=/some/path Note that it is important to keep the path reasonably short to avoid risk of hitting the limit on UNIX socket path names which is 108 characters. When using the embedded mode with a root=/var/tmp/embed, the driver will use the following paths: logDir: /var/tmp/embed/log/qemu swtpmLogDir: /var/tmp/embed/log/swtpm configBaseDir: /var/tmp/embed/etc/qemu stateDir: /var/tmp/embed/run/qemu swtpmStateDir: /var/tmp/embed/run/swtpm cacheDir: /var/tmp/embed/cache/qemu libDir: /var/tmp/embed/lib/qemu swtpmStorageDir: /var/tmp/embed/lib/swtpm defaultTLSx509certdir: /var/tmp/embed/etc/pki/qemu These are identical whether the embedded driver is privileged or unprivileged. This compares with the system instance which uses logDir: /var/log/libvirt/qemu swtpmLogDir: /var/log/swtpm/libvirt/qemu configBaseDir: /etc/libvirt/qemu stateDir: /run/libvirt/qemu swtpmStateDir: /run/libvirt/qemu/swtpm cacheDir: /var/cache/libvirt/qemu libDir: /var/lib/libvirt/qemu swtpmStorageDir: /var/lib/libvirt/swtpm defaultTLSx509certdir: /etc/pki/qemu At this time all features present in the QEMU driver are available when running in embedded mode, availability matching whether the embedded driver is privileged or unprivileged. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- docs/drvqemu.html.in | 84 +++++++++++++++++++++++++++++++++++++++++ src/qemu/qemu_conf.c | 38 +++++++++++++++++-- src/qemu/qemu_conf.h | 6 ++- src/qemu/qemu_driver.c | 26 +++++++------ src/qemu/qemu_process.c | 15 ++++++-- tests/domaincapstest.c | 2 +- tests/testutilsqemu.c | 2 +- 7 files changed, 151 insertions(+), 22 deletions(-) diff --git a/docs/drvqemu.html.in b/docs/drvqemu.html.in index 294117ee1f..4bcf5ac002 100644 --- a/docs/drvqemu.html.in +++ b/docs/drvqemu.html.in @@ -63,6 +63,90 @@ qemu+tcp://example.com/system (remote access, SASl/Kerberos) qemu+ssh://root@example.com/system (remote access, SSH tunnelled) </pre> + <h3><a id="uriembedded">Embedded driver</a></h3> + + <p> + Since 6.0.0 the QEMU driver has experimental support for operating + in an embedded mode. In this scenario, rather than connecting to + the libvirtd daemon, the QEMU driver runs in the client application + process directly. To use this the client application must have + registered & be running an instance of the event loop. To open + the driver in embedded mode the app use the new URI path and specify + a virtual root directory under which the driver will create content. + </p> + + <pre> + qemu:///embed?root=/some/dir + </pre> + + <p> + Under the specified root directory the following locations will + be used + </p> + + <pre> +/some/dir + | + +- log + | | + | +- qemu + | +- swtpm + | + +- etc + | | + | +- qemu + | +- pki + | | + | +- qemu + | + +- run + | | + | +- qemu + | +- swtpm + | + +- cache + | | + | +- qemu + | + +- lib + | + +- qemu + +- swtpm + </pre> + + <p> + Note that UNIX domain sockets used for QEMU virtual machines had + a maximum filename length of 108 characters. Bear this in mind + when picking a root directory to avoid risk of exhausting the + filename space. The application is responsible for recursively + purging the contents of this directory tree once they no longer + require a connection, though it can also be left intact for reuse + when opening a future connection. + </p> + + <p> + Broadly speaking the range of functionality is intended to be + on a par with that seen when using the traditional system or + session libvirt connections to QEMU. The features will of course + differ depending on whether the application using the embedded + driver is running privileged or unprivileged. For example PCI + device assignment or TAP based networking are only available + when running privileged. While the embedded mode is still classed + as experimental some features may change their default settings + between releases. + </p> + + <p> + By default if the application uses any APIs associated with + secondary drivers, these will result in a connection being + opened to the corresponding driver in libvirtd. For example, + this allows a virtual machine from the embedded QEMU to connect + its NIC to a virtual network or connect its disk to a storage + volume. Some of the secondary drivers will also be able to support + running in embedded mode. Currently this is supported by the + secrets driver, to allow for use of VMs with encrypted disks + </p> + <h2><a id="security">Driver security architecture</a></h2> <p> diff --git a/src/qemu/qemu_conf.c b/src/qemu/qemu_conf.c index 30637b21ac..e2b4c26b9b 100644 --- a/src/qemu/qemu_conf.c +++ b/src/qemu/qemu_conf.c @@ -104,7 +104,8 @@ qemuDriverUnlock(virQEMUDriverPtr driver) #endif -virQEMUDriverConfigPtr virQEMUDriverConfigNew(bool privileged) +virQEMUDriverConfigPtr virQEMUDriverConfigNew(bool privileged, + const char *root) { g_autoptr(virQEMUDriverConfig) cfg = NULL; @@ -130,7 +131,24 @@ virQEMUDriverConfigPtr virQEMUDriverConfigNew(bool privileged) cfg->cgroupControllers = -1; /* -1 == auto-detect */ - if (privileged) { + if (root != NULL) { + cfg->logDir = g_strdup_printf("%s/log/qemu", root); + cfg->swtpmLogDir = g_strdup_printf("%s/log/swtpm", root); + cfg->configBaseDir = g_strdup_printf("%s/etc", root); + cfg->stateDir = g_strdup_printf("%s/run/qemu", root); + cfg->swtpmStateDir = g_strdup_printf("%s/run/swtpm", root); + cfg->cacheDir = g_strdup_printf("%s/cache/qemu", root); + cfg->libDir = g_strdup_printf("%s/lib/qemu", root); + cfg->swtpmStorageDir = g_strdup_printf("%s/lib/swtpm", root); + + cfg->saveDir = g_strdup_printf("%s/save", cfg->libDir); + cfg->snapshotDir = g_strdup_printf("%s/snapshot", cfg->libDir); + cfg->checkpointDir = g_strdup_printf("%s/checkpoint", cfg->libDir); + cfg->autoDumpPath = g_strdup_printf("%s/dump", cfg->libDir); + cfg->channelTargetDir = g_strdup_printf("%s/channel/target", cfg->libDir); + cfg->nvramDir = g_strdup_printf("%s/nvram", cfg->libDir); + cfg->memoryBackingDir = g_strdup_printf("%s/ram", cfg->libDir); + } else if (privileged) { cfg->logDir = g_strdup_printf("%s/log/libvirt/qemu", LOCALSTATEDIR); cfg->swtpmLogDir = g_strdup_printf("%s/log/swtpm/libvirt/qemu", @@ -194,6 +212,16 @@ virQEMUDriverConfigPtr virQEMUDriverConfigNew(bool privileged) cfg->memoryBackingDir = g_strdup_printf("%s/qemu/ram", cfg->configBaseDir); cfg->swtpmStorageDir = g_strdup_printf("%s/qemu/swtpm", cfg->configBaseDir); + } + + if (privileged) { + if (!virDoesUserExist("tss") || + virGetUserID("tss", &cfg->swtpm_user) < 0) + cfg->swtpm_user = 0; /* fall back to root */ + if (!virDoesGroupExist("tss") || + virGetGroupID("tss", &cfg->swtpm_group) < 0) + cfg->swtpm_group = 0; /* fall back to root */ + } else { cfg->swtpm_user = (uid_t)-1; cfg->swtpm_group = (gid_t)-1; } @@ -206,7 +234,11 @@ virQEMUDriverConfigPtr virQEMUDriverConfigNew(bool privileged) * This will then be used as a fallback if the service specific * directory doesn't exist (although we don't check if this exists). */ - cfg->defaultTLSx509certdir = g_strdup(SYSCONFDIR "/pki/qemu"); + if (root == NULL) { + cfg->defaultTLSx509certdir = g_strdup(SYSCONFDIR "pki/qemu"); + } else { + cfg->defaultTLSx509certdir = g_strdup_printf("%s/etc/pki/qemu", root); + } cfg->vncListen = g_strdup(VIR_LOOPBACK_IPV4_ADDR); cfg->spiceListen = g_strdup(VIR_LOOPBACK_IPV4_ADDR); diff --git a/src/qemu/qemu_conf.h b/src/qemu/qemu_conf.h index 95b33a1093..5de9d2f913 100644 --- a/src/qemu/qemu_conf.h +++ b/src/qemu/qemu_conf.h @@ -241,8 +241,9 @@ struct _virQEMUDriver { /* Atomic inc/dec only */ unsigned int nactive; - /* Immutable value */ + /* Immutable values */ bool privileged; + bool embedded; /* Immutable pointers. Caller must provide locking */ virStateInhibitCallback inhibitCallback; @@ -301,7 +302,8 @@ struct _virQEMUDriver { virHashAtomicPtr migrationErrors; }; -virQEMUDriverConfigPtr virQEMUDriverConfigNew(bool privileged); +virQEMUDriverConfigPtr virQEMUDriverConfigNew(bool privileged, + const char *root); int virQEMUDriverConfigLoadFile(virQEMUDriverConfigPtr cfg, const char *filename, diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 31fe921ee3..e51086ea5e 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -658,12 +658,6 @@ qemuStateInitialize(bool privileged, bool autostart = true; size_t i; - if (root != NULL) { - virReportError(VIR_ERR_INVALID_ARG, "%s", - _("Driver does not support embedded mode")); - return -1; - } - if (VIR_ALLOC(qemu_driver) < 0) return VIR_DRV_STATE_INIT_ERROR; @@ -680,6 +674,7 @@ qemuStateInitialize(bool privileged, qemu_driver->inhibitOpaque = opaque; qemu_driver->privileged = privileged; + qemu_driver->embedded = root != NULL; if (!(qemu_driver->domains = virDomainObjListNew())) goto error; @@ -693,7 +688,7 @@ qemuStateInitialize(bool privileged, if (privileged) qemu_driver->hostsysinfo = virSysinfoRead(); - if (!(qemu_driver->config = cfg = virQEMUDriverConfigNew(privileged))) + if (!(qemu_driver->config = cfg = virQEMUDriverConfigNew(privileged, root))) goto error; if (!(driverConf = g_strdup_printf("%s/qemu.conf", cfg->configBaseDir))) @@ -1208,10 +1203,18 @@ static virDrvOpenStatus qemuConnectOpen(virConnectPtr conn, return VIR_DRV_OPEN_ERROR; } - if (!virConnectValidateURIPath(conn->uri->path, - "qemu", - virQEMUDriverIsPrivileged(qemu_driver))) - return VIR_DRV_OPEN_ERROR; + if (qemu_driver->embedded) { + if (STRNEQ(conn->uri->path, "/embed")) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("URI must be qemu:///embed")); + return VIR_DRV_OPEN_ERROR; + } + } else { + if (!virConnectValidateURIPath(conn->uri->path, + "qemu", + virQEMUDriverIsPrivileged(qemu_driver))) + return VIR_DRV_OPEN_ERROR; + } if (virConnectOpenEnsureACL(conn) < 0) return VIR_DRV_OPEN_ERROR; @@ -22950,6 +22953,7 @@ static virHypervisorDriver qemuHypervisorDriver = { static virConnectDriver qemuConnectDriver = { .localOnly = true, .uriSchemes = (const char *[]){ "qemu", NULL }, + .embeddable = true, .hypervisorDriver = &qemuHypervisorDriver, }; diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index a588ee25f8..c812fa680c 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -6717,10 +6717,17 @@ qemuProcessLaunch(virConnectPtr conn, cfg = virQEMUDriverGetConfig(driver); - if ((flags & VIR_QEMU_PROCESS_START_AUTODESTROY) && !conn) { - virReportError(VIR_ERR_INTERNAL_ERROR, "%s", - _("Domain autodestroy requires a connection handle")); - return -1; + if (flags & VIR_QEMU_PROCESS_START_AUTODESTROY) { + if (!conn) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("Domain autodestroy requires a connection handle")); + return -1; + } + if (driver->embedded) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("Domain autodestroy not supported for embedded drivers yet")); + return -1; + } } hookData.vm = vm; diff --git a/tests/domaincapstest.c b/tests/domaincapstest.c index c6700f7e78..16a37b37fe 100644 --- a/tests/domaincapstest.c +++ b/tests/domaincapstest.c @@ -373,7 +373,7 @@ mymain(void) #endif #if WITH_QEMU - virQEMUDriverConfigPtr cfg = virQEMUDriverConfigNew(false); + virQEMUDriverConfigPtr cfg = virQEMUDriverConfigNew(false, ""); if (!cfg) return EXIT_FAILURE; diff --git a/tests/testutilsqemu.c b/tests/testutilsqemu.c index 7b9e427463..3a21dfac71 100644 --- a/tests/testutilsqemu.c +++ b/tests/testutilsqemu.c @@ -694,7 +694,7 @@ int qemuTestDriverInit(virQEMUDriver *driver) if (virMutexInit(&driver->lock) < 0) return -1; - driver->config = virQEMUDriverConfigNew(false); + driver->config = virQEMUDriverConfigNew(false, ""); if (!driver->config) goto error; -- 2.23.0
This enables support for running the secret driver embedded to the calling application process using a URI: secret:///embed?root=/some/path When using the embedded mode with a root=/var/tmp/embed, the driver will use the following paths: configDir: /var/tmp/embed/etc/secrets stateDir: /var/tmp/embed/run/secrets These are identical whether the embedded driver is privileged or unprivileged. This compares with the system instance which uses configDir: /etc/libvirt/secrets stateDir: /var/lib/libvirt/secrets When an embedded instance of the secret driver is open, any other embedded drivers will automatically use the embedded secret driver. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- docs/drivers.html.in | 1 + docs/drvsecret.html.in | 82 ++++++++++++++++++++++++++++++++++++++ src/secret/secret_driver.c | 46 ++++++++++++++++----- 3 files changed, 118 insertions(+), 11 deletions(-) create mode 100644 docs/drvsecret.html.in diff --git a/docs/drivers.html.in b/docs/drivers.html.in index 4539eedbcd..cb1ddc9212 100644 --- a/docs/drivers.html.in +++ b/docs/drivers.html.in @@ -8,6 +8,7 @@ <li><a href="#hypervisor">Hypervisor drivers</a></li> <li><a href="storage.html">Storage drivers</a></li> <li><a href="drvnodedev.html">Node device driver</a></li> + <li><a href="drvsecret.html">Secret driver</a></li> </ul> <p> diff --git a/docs/drvsecret.html.in b/docs/drvsecret.html.in new file mode 100644 index 0000000000..9a05fe1f09 --- /dev/null +++ b/docs/drvsecret.html.in @@ -0,0 +1,82 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE html> +<html xmlns="http://www.w3.org/1999/xhtml"> + <body> + <h1>Secret information management</h1> + + <p> + The secrets driver in libvirt provides a simple interface for + storing and retrieving secret information. + </p> + + <h2><a id="uris">Connections to SECRET driver</a></h2> + + <p> + The libvirt SECRET driver is a multi-instance driver, providing a single + system wide privileged driver (the "system" instance), and per-user + unprivileged drivers (the "session" instance). A connection to the secret + driver is automatically available when opening a connection to one of the + stateful primary hypervisor drivers. It is none the less also possible to + explicitly open just the secret driver, using the URI protocol "secret" + Some example connection URIs for the driver are: + </p> + +<pre> +secret:///session (local access to per-user instance) +secret+unix:///session (local access to per-user instance) + +secret:///system (local access to system instance) +secret+unix:///system (local access to system instance) +secret://example.com/system (remote access, TLS/x509) +secret+tcp://example.com/system (remote access, SASl/Kerberos) +secret+ssh://root@example.com/system (remote access, SSH tunnelled) +</pre> + + <h3><a id="uriembedded">Embedded driver</a></h3> + + <p> + Since 6.0.0 the secret driver has experimental support for operating + in an embedded mode. In this scenario, rather than connecting to + the libvirtd daemon, the secret driver runs in the client application + process directly. To open the driver in embedded mode the app use the + new URI path and specify a virtual root directory under which the + driver will create content. + </p> + + <pre> + secret:///embed?root=/some/dir + </pre> + + <p> + Under the specified root directory the following locations will + be used + </p> + + <pre> +/some/dir + | + +- etc + | | + | +- secrets + | + +- run + | + +- secrets + </pre> + + <p> + The application is responsible for recursively purging the contents + of this directory tree once they no longer require a connection, + though it can also be left intact for reuse when opening a future + connection. + </p> + + <p> + The range of functionality is intended to be on a par with that + seen when using the traditional system or session libvirt connections + to QEMU. Normal practice would be to open the secret driver in embedded + mode any time one of the other drivers is opened in embedded mode so + that the two drivers can interact in-process. + </p> + </body> +</html> diff --git a/src/secret/secret_driver.c b/src/secret/secret_driver.c index d248121327..c791bde2f9 100644 --- a/src/secret/secret_driver.c +++ b/src/secret/secret_driver.c @@ -55,6 +55,8 @@ typedef virSecretDriverState *virSecretDriverStatePtr; struct _virSecretDriverState { virMutex lock; bool privileged; /* readonly */ + bool embedded; /* readonly */ + int embeddedRefs; virSecretObjListPtr secrets; char *stateDir; char *configDir; @@ -456,12 +458,6 @@ secretStateInitialize(bool privileged, virStateInhibitCallback callback G_GNUC_UNUSED, void *opaque G_GNUC_UNUSED) { - if (root != NULL) { - virReportError(VIR_ERR_INVALID_ARG, "%s", - _("Driver does not support embedded mode")); - return -1; - } - if (VIR_ALLOC(driver) < 0) return VIR_DRV_STATE_INIT_ERROR; @@ -474,8 +470,12 @@ secretStateInitialize(bool privileged, driver->secretEventState = virObjectEventStateNew(); driver->privileged = privileged; + driver->embedded = root != NULL; - if (privileged) { + if (root) { + driver->configDir = g_strdup_printf("%s/etc/secrets", root); + driver->stateDir = g_strdup_printf("%s/run/secrets", root); + } else if (privileged) { driver->configDir = g_strdup_printf("%s/libvirt/secrets", SYSCONFDIR); driver->stateDir = g_strdup_printf("%s/libvirt/secrets", RUNSTATEDIR); } else { @@ -552,19 +552,42 @@ secretConnectOpen(virConnectPtr conn, return VIR_DRV_OPEN_ERROR; } - if (!virConnectValidateURIPath(conn->uri->path, - "secret", - driver->privileged)) - return VIR_DRV_OPEN_ERROR; + if (driver->embedded) { + if (STRNEQ(conn->uri->path, "/embed")) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("URI must be secret:///embed")); + return VIR_DRV_OPEN_ERROR; + } + } else { + if (!virConnectValidateURIPath(conn->uri->path, + "secret", + driver->privileged)) + return VIR_DRV_OPEN_ERROR; + } if (virConnectOpenEnsureACL(conn) < 0) return VIR_DRV_OPEN_ERROR; + if (driver->embedded) { + secretDriverLock(); + if (driver->embeddedRefs == 0) + virSetConnectSecret(conn); + driver->embeddedRefs++; + secretDriverUnlock(); + } + return VIR_DRV_OPEN_SUCCESS; } static int secretConnectClose(virConnectPtr conn G_GNUC_UNUSED) { + if (driver->embedded) { + secretDriverLock(); + driver->embeddedRefs--; + if (driver->embeddedRefs == 0) + virSetConnectSecret(NULL); + secretDriverUnlock(); + } return 0; } @@ -657,6 +680,7 @@ static virHypervisorDriver secretHypervisorDriver = { static virConnectDriver secretConnectDriver = { .localOnly = true, .uriSchemes = (const char *[]){ "secret", NULL }, + .embeddable = true, .hypervisorDriver = &secretHypervisorDriver, .secretDriver = &secretDriver, }; -- 2.23.0
The previous "QEMU shim" proof of concept was taking an approach of only caring about initial spawning of the QEMU process. It was then registered with the libvirtd daemon who took over management of it. The intent was that later libvirtd would be refactored so that the shim retained control over the QEMU monitor and libvirt just forwarded APIs to each shim as needed. This forwarding of APIs would require quite alot of significant refactoring of libvirtd to achieve. This impl thus takes a quite different approach, explicitly deciding to keep the VMs completely separate from those seen & managed by libvirtd. Instead it uses the new "qemu:///embed" URI scheme to embed the entire QEMU driver in the shim, running with a custom root directory. Once the driver is initialization, the shim starts a VM and then waits to shutdown automatically when QEMU shuts down, or should kill QEMU if it is terminated itself. This ought to use the AUTO_DESTROY feature but that is not yet available in embedded mode, so we rely on installing a few signal handlers to gracefully kill QEMU. This isn't reliable if we crash of course, but you can restart with the same root dir. Note this program does not expose any way to manage the QEMU process, since there's no RPC interface enabled. It merely starts the VM and cleans up when the guest shuts down at the end. This program is installed to /usr/bin/virt-qemu-run enabling direct use by end users. Most use cases will probably want to integrate the concept directly into their respective application codebases. This standalone binary serves as a nice demo though, and also provides a way to measure performance of the startup process quite simply. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- build-aux/syntax-check.mk | 2 +- libvirt.spec.in | 2 + src/Makefile.am | 9 ++ src/qemu/Makefile.inc.am | 26 ++++ src/qemu/qemu_shim.c | 313 ++++++++++++++++++++++++++++++++++++++ src/qemu/qemu_shim.pod | 94 ++++++++++++ 6 files changed, 445 insertions(+), 1 deletion(-) create mode 100644 src/qemu/qemu_shim.c create mode 100644 src/qemu/qemu_shim.pod diff --git a/build-aux/syntax-check.mk b/build-aux/syntax-check.mk index 94e4945323..627734fc55 100644 --- a/build-aux/syntax-check.mk +++ b/build-aux/syntax-check.mk @@ -2340,7 +2340,7 @@ exclude_file_name_regexp--sc_prohibit_devname = \ ^(tools/virsh.pod|build-aux/syntax-check\.mk|docs/.*)$$ exclude_file_name_regexp--sc_prohibit_virXXXFree = \ - ^(docs/|tests/|examples/|tools/|build-aux/syntax-check\.mk|src/test/test_driver.c|src/libvirt_public.syms|include/libvirt/libvirt-(domain|network|nodedev|storage|stream|secret|nwfilter|interface|domain-snapshot).h|src/libvirt-(domain|qemu|network|nodedev|storage|stream|secret|nwfilter|interface|domain-snapshot).c$$) + ^(docs/|tests/|examples/|tools/|build-aux/syntax-check\.mk|src/test/test_driver.c|src/libvirt_public.syms|include/libvirt/libvirt-(domain|network|nodedev|storage|stream|secret|nwfilter|interface|domain-snapshot).h|src/libvirt-(domain|qemu|network|nodedev|storage|stream|secret|nwfilter|interface|domain-snapshot).c|src/qemu/qemu_shim.c$$) exclude_file_name_regexp--sc_prohibit_sysconf_pagesize = \ ^(build-aux/syntax-check\.mk|src/util/virutil\.c)$$ diff --git a/libvirt.spec.in b/libvirt.spec.in index c7fac104d9..886c7b73d5 100644 --- a/libvirt.spec.in +++ b/libvirt.spec.in @@ -1757,6 +1757,8 @@ exit 0 %{_libdir}/%{name}/connection-driver/libvirt_driver_qemu.so %dir %attr(0711, root, root) %{_localstatedir}/lib/libvirt/swtpm/ %dir %attr(0711, root, root) %{_localstatedir}/log/swtpm/libvirt/qemu/ +%{_bindir}/virt-qemu-run +%{_mandir}/man1/virt-qemu-run.1* %endif %if %{with_lxc} diff --git a/src/Makefile.am b/src/Makefile.am index 696e64c52e..f86ba3fe3d 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -90,6 +90,8 @@ SYSTEMD_UNIT_FILES = SYSTEMD_UNIT_FILES_IN = SYSCONF_FILES = sbin_PROGRAMS = +bin_PROGRAMS = +man1_MANS = man8_MANS = DRIVER_SOURCES = man7_MANS = @@ -659,6 +661,13 @@ endif WITH_LIBVIRTD < $< > $@-t && \ mv $@-t $@ +%.1: %.1.in $(top_srcdir)/configure.ac + $(AM_V_GEN)sed \ + -e 's|[@]sysconfdir[@]|$(sysconfdir)|g' \ + -e 's|[@]runstatedir[@]|$(runstatedir)|g' \ + < $< > $@-t && \ + mv $@-t $@ + CLEANFILES += \ $(man8_MANS) \ $(MANINFILES) diff --git a/src/qemu/Makefile.inc.am b/src/qemu/Makefile.inc.am index bf30f8a3c5..d41853ac9e 100644 --- a/src/qemu/Makefile.inc.am +++ b/src/qemu/Makefile.inc.am @@ -231,3 +231,29 @@ EXTRA_DIST += \ qemu/THREADS.txt \ libvirt_qemu_probes.d \ $(NULL) + +QEMU_SHIM_SOURCES = qemu/qemu_shim.c +QEMU_SHIM_POD = qemu/qemu_shim.pod + +EXTRA_DIST += $(QEMU_SHIM_SOURCES) $(QEMU_SHIM_POD) + +if WITH_QEMU +bin_PROGRAMS += virt-qemu-run + +man1_MANS += virt-qemu-run.1 + +virt_qemu_run_SOURCES = $(QEMU_SHIM_SOURCES) + +virt_qemu_run_LDADD = libvirt.la +virt_qemu_run_LDFLAGS = -Wl,--export-dynamic +endif WITH_QEMU + +virt-qemu-run.1.in: qemu/qemu_shim.pod + $(AM_V_GEN)$(POD2MAN) $< $@-t1 && \ + if grep 'POD ERROR' $@-t1; then rm $@-t1; exit 1; fi && \ + sed \ + -e 's|SYSCONFDIR|\@sysconfdir\@|g' \ + -e 's|LOCALSTATEDIR|\@localstatedir\@|g' \ + < $@-t1 > $@-t2 && \ + rm -f $@-t1 && \ + mv $@-t2 $@ diff --git a/src/qemu/qemu_shim.c b/src/qemu/qemu_shim.c new file mode 100644 index 0000000000..10f9682143 --- /dev/null +++ b/src/qemu/qemu_shim.c @@ -0,0 +1,313 @@ +/* + * qemu_shim.c: standalone binary for running QEMU instances + * + * Copyright (C) 2019 Red Hat, Inc. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library. If not, see + * <http://www.gnu.org/licenses/>. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdbool.h> + +#include "virfile.h" +#include "virstring.h" +#include "virgettext.h" + +#define VIR_FROM_THIS VIR_FROM_QEMU + +static bool eventQuitFlag; +static int eventQuitFD = -1; +static virDomainPtr dom; + +static void * +qemuShimEventLoop(void *opaque G_GNUC_UNUSED) +{ + while (!eventQuitFlag) + virEventRunDefaultImpl(); + + return NULL; +} + +/* Runs in event loop thread context */ +static void +qemuShimEventLoopStop(int watch G_GNUC_UNUSED, + int fd G_GNUC_UNUSED, + int event G_GNUC_UNUSED, + void *opaque G_GNUC_UNUSED) +{ + char c; + ignore_value(read(fd, &c, 1)); + eventQuitFlag = true; +} + +/* Runs in event loop thread context */ +static int +qemuShimDomShutdown(virConnectPtr econn G_GNUC_UNUSED, + virDomainPtr edom G_GNUC_UNUSED, + int event, + int detail G_GNUC_UNUSED, + void *opaque G_GNUC_UNUSED) +{ + if (event == VIR_DOMAIN_EVENT_STOPPED) + eventQuitFlag = true; + + return 0; +} + +/* Runs in unknown thread context */ +static void +qemuShimSigShutdown(int sig G_GNUC_UNUSED) +{ + if (dom) + virDomainDestroy(dom); + ignore_value(safewrite(eventQuitFD, "c", 1)); +} + +static void +qemuShimQuench(void *userData G_GNUC_UNUSED, + virErrorPtr error G_GNUC_UNUSED) +{ +} + +int main(int argc, char **argv) +{ + GThread *eventLoopThread = NULL; + virConnectPtr conn = NULL; + virConnectPtr sconn = NULL; + g_autofree char *xml = NULL; + g_autofree char *uri = NULL; + g_autofree char *suri = NULL; + char *root = NULL; + bool tmproot = false; + int ret = 1; + g_autoptr(GError) error = NULL; + g_auto(GStrv) secrets = NULL; + bool verbose = false; + GStrv tmpsecrets; + GOptionContext *ctx; + GOptionEntry entries[] = { + { "secret", 's', 0, G_OPTION_ARG_STRING_ARRAY, &secrets, "Load secret file", "SECRET-XML-FILE,SECRET-VALUE-FILE" }, + { "root", 'r', 0, G_OPTION_ARG_STRING, &root, "Root directory", "DIR"}, + { "verbose", 'v', 0, G_OPTION_ARG_NONE, &verbose, "Verbose output", NULL }, + { NULL } + }; + int quitfd[2] = {-1, -1}; + long long start = g_get_monotonic_time(); + +#define deltams() ((long long)g_get_monotonic_time() - start) + + ctx = g_option_context_new("- run a standalone QEMU process"); + g_option_context_add_main_entries(ctx, entries, PACKAGE); + if (!g_option_context_parse(ctx, &argc, &argv, &error)) { + g_printerr("%s: option parsing failed: %s\n", + argv[0], error->message); + return 1; + } + if (argc != 2) { + g_autofree char *help = g_option_context_get_help(ctx, TRUE, NULL); + g_printerr("%s", help); + return 1; + } + + if (verbose) + g_printerr("%s: %lld: initializing libvirt\n", + argv[0], deltams()); + + if (virInitialize() < 0) { + g_printerr("%s: cannot initialize libvirt\n", argv[0]); + return 1; + } + if (virGettextInitialize() < 0) { + g_printerr("%s: cannot initialize libvirt translations\n", argv[0]); + return 1; + } + + virSetErrorFunc(NULL, qemuShimQuench); + + if (verbose) + g_printerr("%s: %lld: initializing signal handlers\n", + argv[0], deltams()); + + signal(SIGTERM, qemuShimSigShutdown); + signal(SIGINT, qemuShimSigShutdown); + signal(SIGQUIT, qemuShimSigShutdown); + signal(SIGHUP, qemuShimSigShutdown); + + if (root == NULL) { + if (!(root = g_dir_make_tmp("libvirt-qemu-shim-XXXXXX", &error))) { + g_printerr("%s: cannot create temporary dir: %s\n", + argv[0], error->message); + return 1; + } + tmproot = true; + } + + virFileActivateDirOverrideForProg(argv[0]); + + if (verbose) + g_printerr("%s: %lld: preparing event loop thread\n", + argv[0], deltams()); + virEventRegisterDefaultImpl(); + + if (pipe(quitfd) < 0) { + g_printerr("%s: cannot create event loop pipe: %s", + argv[0], g_strerror(errno)); + goto cleanup; + } + + if (virEventAddHandle(quitfd[0], VIR_EVENT_HANDLE_READABLE, qemuShimEventLoopStop, NULL, NULL) < 0) { + VIR_FORCE_CLOSE(quitfd[0]); + VIR_FORCE_CLOSE(quitfd[1]); + quitfd[0] = quitfd[1] = -1; + g_printerr("%s: cannot register event loop handle: %s", + argv[0], virGetLastErrorMessage()); + goto cleanup; + } + eventQuitFD = quitfd[1]; + + eventLoopThread = g_thread_new("event-loop", qemuShimEventLoop, NULL); + + if (secrets && *secrets) { + suri = g_strdup_printf("secret:///embed?root=%s", root); + + if (verbose) + g_printerr("%s: %lld: opening %s\n", + argv[0], deltams(), suri); + + sconn = virConnectOpen(suri); + if (!sconn) { + g_printerr("%s: cannot open %s: %s\n", + argv[0], suri, virGetLastErrorMessage()); + goto cleanup; + } + + tmpsecrets = secrets; + while (tmpsecrets && *tmpsecrets) { + g_auto(GStrv) bits = g_strsplit(*tmpsecrets, ",", 2); + g_autofree char *sxml = NULL; + g_autofree char *value = NULL; + virSecretPtr sec; + size_t nvalue; + + if (!bits || bits[0] == NULL || bits[1] == NULL) { + g_printerr("%s: expected a pair of filenames for --secret argument\n", + argv[0]); + goto cleanup; + } + + if (verbose) + g_printerr("%s: %lld: loading secret %s and %s\n", + argv[0], deltams(), bits[0], bits[1]); + + if (!g_file_get_contents(bits[0], &sxml, NULL, &error)) { + g_printerr("%s: cannot read secret XML %s: %s\n", + argv[0], bits[0], error->message); + goto cleanup; + } + + if (!g_file_get_contents(bits[1], &value, &nvalue, &error)) { + g_printerr("%s: cannot read secret value %s: %s\n", + argv[0], bits[1], error->message); + goto cleanup; + } + + if (!(sec = virSecretDefineXML(sconn, sxml, 0))) { + g_printerr("%s: cannot define secret %s: %s\n", + argv[0], bits[0], virGetLastErrorMessage()); + goto cleanup; + } + + if (virSecretSetValue(sec, (unsigned char *)value, nvalue, 0) < 0) { + virSecretFree(sec); + g_printerr("%s: cannot set value for secret %s: %s\n", + argv[0], bits[0], virGetLastErrorMessage()); + goto cleanup; + } + virSecretFree(sec); + + tmpsecrets++; + } + } + + uri = g_strdup_printf("qemu:///embed?root=%s", root); + + if (verbose) + g_printerr("%s: %lld: opening %s\n", + argv[0], deltams(), uri); + + conn = virConnectOpen(uri); + if (!conn) { + g_printerr("%s: cannot open %s: %s\n", + argv[0], uri, virGetLastErrorMessage()); + goto cleanup; + } + + if (virConnectDomainEventRegisterAny( + conn, dom, VIR_DOMAIN_EVENT_ID_LIFECYCLE, + VIR_DOMAIN_EVENT_CALLBACK(qemuShimDomShutdown), + NULL, NULL) < 0) { + g_printerr("%s: cannot regiser for lifecycle events: %s\n", + argv[0], virGetLastErrorMessage()); + goto cleanup; + } + + if (verbose) + g_printerr("%s: %lld: starting guest %s\n", + argv[0], deltams(), argv[1]); + + if (!g_file_get_contents(argv[1], &xml, NULL, &error)) { + g_printerr("%s: cannot read %s: %s\n", + argv[0], argv[1], error->message); + goto cleanup; + } + + dom = virDomainCreateXML(conn, xml, 0); + if (!dom) { + g_printerr("%s: cannot start VM: %s\n", + argv[0], virGetLastErrorMessage()); + goto cleanup; + } + if (verbose) + g_printerr("%s: %lld: guest running, Ctrl-C to stop\n", + argv[0], deltams()); + + ret = 0; + + cleanup: + if (ret != 0 && eventQuitFD != -1) + ignore_value(safewrite(eventQuitFD, "c", 1)); + + if (eventLoopThread != NULL && (ret == 0 || eventQuitFD != -1)) + g_thread_join(eventLoopThread); + + VIR_FORCE_CLOSE(quitfd[0]); + VIR_FORCE_CLOSE(quitfd[1]); + + if (dom != NULL) + virDomainFree(dom); + if (sconn != NULL) + virConnectClose(sconn); + if (conn != NULL) + virConnectClose(conn); + if (tmproot) + virFileDeleteTree(root); + + if (verbose) + g_printerr("%s: %lld: cleaned up, exiting\n", + argv[0], deltams()); + return ret; +} diff --git a/src/qemu/qemu_shim.pod b/src/qemu/qemu_shim.pod new file mode 100644 index 0000000000..eb17fa0ac7 --- /dev/null +++ b/src/qemu/qemu_shim.pod @@ -0,0 +1,94 @@ +=encoding utf8 + +=head1 NAME + +virt-qemu-run - run a standalone QEMU guest + +=head1 SYNOPSIS + +B<virt-qemu-run> [I<OPTIONS>...] [I<GUEST-XML>] + +=head1 DESCRIPTION + +This tool provides a way to run a standalone QEMU guest such that it +is completely independant of libvirtd. It makes use of the embedded +QEMU driver support to run the VM placing files under an isolated +directory tree. When the guest is run with this tool it is invisible +to libvirtd and thus also invisible to other libvirt tools such as +virsh. + +The virt-qemu-run program will run the QEMU virtual machine, and then +block until the guest OS shuts down, at which point it will exit. + +If the virt-qemu-run program is interrupted (eg Ctrl-C) it will +immediately terminate the virtual machine without giving the guest +OS any opportunity to gracefully shutdown. + +=head1 OPTIONS + +=over 4 + +=item C<GUEST-XML> + +The full path to the XML file describing the guest virtual machine +to be booted. + +=item C<-h>, C<--help> + +Display the command line help + +=item C<-v>, C<--verbose> + +Display verbose information about startup + +=item C<-r DIR>, C<--root=DIR> + +Specify the root directory to use for storing state associated with +the virtual machine. The caller is responsible for deleting this +directory when it is no longer required. + +If this parameter is omitted, then a random temporary directory +will be created, and its contents be automaticlaly deleted at +VM shutdown. + +=item C<-s XML-FILE,VALUE-FILE> C<--secret=XML-FILE,VALUE-FILE> + +Specify a secret to be loaded into the secret driver. The C<XML-FILE> +is a path to the XML description of the secret, whose UUID should +match a secret referenced in the guest domain XML. The C<VALUE-FILE> +is a path containing the raw value of the secret. + +=back + +=head1 EXIT STATUS + +Upon successful shutdown, an exit status of 0 will be set. Upon +failure a non-zero status will be set. + +=head1 AUTHOR + +Daniel P. Berrangé + +=head1 BUGS + +Report any bugs discovered to the libvirt community via the +mailing list L<https://libvirt.org/contact.html> or bug tracker +L<https://libvirt.org/bugs.html>. +Alternatively report bugs to your software distributor / vendor. + +=head1 COPYRIGHT + +Copyright (C) 2019 by Red Hat, Inc. + +=head1 LICENSE + +virt-run-qemu is distributed under the terms of the GNU LGPL v2+. +This is free software; see the source for copying conditions. There +is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR +PURPOSE + +=head1 SEE ALSO + +L<virsh(1)> + +=cut -- 2.23.0
participants (3)
-
Cole Robinson -
Daniel P. Berrangé -
Michal Prívozník