[libvirt] [PATCH v3 5/5] qemu: introduce a new "virt-qemu-run" program

The previous "QEMU shim" proof of concept was taking an approach of only caring about initial spawning of the QEMU process. It was then registered with the libvirtd daemon who took over management of it. The intent was that later libvirtd would be refactored so that the shim retained control over the QEMU monitor and libvirt just forwarded APIs to each shim as needed. This forwarding of APIs would require quite alot of significant refactoring of libvirtd to achieve. This impl thus takes a quite different approach, explicitly deciding to keep the VMs completely separate from those seen & managed by libvirtd. Instead it uses the new "qemu:///embed" URI scheme to embed the entire QEMU driver in the shim, running with a custom root directory. Once the driver is initialization, the shim starts a VM and then waits to shutdown automatically when QEMU shuts down, or should kill QEMU if it is terminated itself. This ought to use the AUTO_DESTROY feature but that is not yet available in embedded mode, so we rely on installing a few signal handlers to gracefully kill QEMU. This isn't reliable if we crash of course, but you can restart with the same root dir. Note this program does not expose any way to manage the QEMU process, since there's no RPC interface enabled. It merely starts the VM and cleans up when the guest shuts down at the end. This program is installed to /usr/bin/virt-qemu-run enabling direct use by end users. Most use cases will probably want to integrate the concept directly into their respective application codebases. This standalone binary serves as a nice demo though, and also provides a way to measure performance of the startup process quite simply. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com&gt; --- build-aux/syntax-check.mk | 2 +- docs/Makefile.am | 5 + docs/manpages/index.rst | 1 + docs/manpages/virt-qemu-run.rst | 114 +++++++++++ libvirt.spec.in | 2 + src/Makefile.am | 1 + src/qemu/Makefile.inc.am | 13 ++ src/qemu/qemu_shim.c | 322 ++++++++++++++++++++++++++++++++ 8 files changed, 459 insertions(+), 1 deletion(-) create mode 100644 docs/manpages/virt-qemu-run.rst create mode 100644 src/qemu/qemu_shim.c diff --git a/build-aux/syntax-check.mk b/build-aux/syntax-check.mk index 8abf1520ed..630b3f8033 100644 --- a/build-aux/syntax-check.mk +++ b/build-aux/syntax-check.mk @@ -2334,7 +2334,7 @@ exclude_file_name_regexp--sc_prohibit_devname = \ ^(tools/virsh.pod|build-aux/syntax-check\.mk|docs/.*)$$ exclude_file_name_regexp--sc_prohibit_virXXXFree = \ - ^(docs/|tests/|examples/|tools/|build-aux/syntax-check\.mk|src/test/test_driver.c|src/libvirt_public.syms|include/libvirt/libvirt-(domain|network|nodedev|storage|stream|secret|nwfilter|interface|domain-snapshot).h|src/libvirt-(domain|qemu|network|nodedev|storage|stream|secret|nwfilter|interface|domain-snapshot).c$$) + ^(docs/|tests/|examples/|tools/|build-aux/syntax-check\.mk|src/test/test_driver.c|src/libvirt_public.syms|include/libvirt/libvirt-(domain|network|nodedev|storage|stream|secret|nwfilter|interface|domain-snapshot).h|src/libvirt-(domain|qemu|network|nodedev|storage|stream|secret|nwfilter|interface|domain-snapshot).c|src/qemu/qemu_shim.c$$) exclude_file_name_regexp--sc_prohibit_sysconf_pagesize = \ ^(build-aux/syntax-check\.mk|src/util/virutil\.c)$$ diff --git a/docs/Makefile.am b/docs/Makefile.am index 66f164a1ac..6d593dfb62 100644 --- a/docs/Makefile.am +++ b/docs/Makefile.am @@ -248,6 +248,11 @@ if WITH_SANLOCK else ! WITH_SANLOCK manpages_rst += manpages/virt-sanlock-cleanup.rst endif ! WITH_SANLOCK +if WITH_QEMU + manpages1_rst += manpages/virt-qemu-run.rst +else ! WITH_QEMU + manpages_rst += manpages/virt-qemu-run.rst +endif ! WITH_QEMU manpages_rst_html_in = \ $(manpages_rst:%.rst=%.html.in) manpages_html = \ diff --git a/docs/manpages/index.rst b/docs/manpages/index.rst index 4945ad59e2..2e71f81962 100644 --- a/docs/manpages/index.rst +++ b/docs/manpages/index.rst @@ -19,6 +19,7 @@ Tools * `virt-login-shell(1) <virt-login-shell.html>`__ - tool to execute a shell within a container * `virt-admin(1) <virt-admin.html>`__ - daemon administration interface * `virsh(1) <virsh.html>`__ - management user interface +* `virt-qemu-run(1) <virt-qemu-run.html)`__ - run standalone QEMU instances Key codes ========= diff --git a/docs/manpages/virt-qemu-run.rst b/docs/manpages/virt-qemu-run.rst new file mode 100644 index 0000000000..7f66f7c3ab --- /dev/null +++ b/docs/manpages/virt-qemu-run.rst @@ -0,0 +1,114 @@ +============= +virt-qemu-run +============= + +--------------------------- +Run a standalone QEMU guest +--------------------------- + +:Manual section: 1 +:Manual group: Virtualization Support + +.. contents:: + +SYNOPSIS +======== + +``virt-qemu-run [OPTIONS...] [IGUEST-XML] + +DESCRIPTION +=========== + +This tool provides a way to run a standalone QEMU guest such that it +is completely independant of libvirtd. It makes use of the embedded +QEMU driver support to run the VM placing files under an isolated +directory tree. When the guest is run with this tool it is invisible +to libvirtd and thus also invisible to other libvirt tools such as +virsh. + +The virt-qemu-run program will run the QEMU virtual machine, and then +block until the guest OS shuts down, at which point it will exit. + +If the virt-qemu-run program is interrupted (eg Ctrl-C) it will +immediately terminate the virtual machine without giving the guest +OS any opportunity to gracefully shutdown. + +OPTIONS +======= + +``GUEST-XML`` + +The full path to the XML file describing the guest virtual machine +to be booted. + +``-h``, ``--help`` + +Display the command line help + +``-v``, ``--verbose`` + +Display verbose information about startup + +``-r DIR``, ``--root=DIR`` + +Specify the root directory to use for storing state associated with +the virtual machine. The caller is responsible for deleting this +directory when it is no longer required. + +If this parameter is omitted, then a random temporary directory +will be created, and its contents be automaticlaly deleted at +VM shutdown. + +``-s XML-FILE,VALUE-FILE``, ``--secret=XML-FILE,VALUE-FILE`` + +Specify a secret to be loaded into the secret driver. The ``XML-FILE`` +is a path to the XML description of the secret, whose UUID should +match a secret referenced in the guest domain XML. The ``VALUE-FILE`` +is a path containing the raw value of the secret. + +EXIT STATUS +=========== + +Upon successful shutdown, an exit status of 0 will be set. Upon +failure a non-zero status will be set. + +AUTHOR +====== + +Daniel P. Berrangé + + +BUGS +==== + +Please report all bugs you discover. This should be done via either: + +#. the mailing list + + `https://libvirt.org/contact.html <https://libvirt.org/contact.html>`_ + +#. the bug tracker + + `https://libvirt.org/bugs.html <https://libvirt.org/bugs.html>`_ + +Alternatively, you may report bugs to your software distributor / vendor. + + +COPYRIGHT +========= + +Copyright (C) 2019 by Red Hat, Inc. + + +LICENSE +======= + +``virt-run-qemu`` is distributed under the terms of the GNU LGPL v2+. +This is free software; see the source for copying conditions. There +is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR +PURPOSE + +SEE ALSO +======== + +virsh(1), `https://libvirt.org/ <https://libvirt.org/>`_ diff --git a/libvirt.spec.in b/libvirt.spec.in index 249a4b5425..1c0fc76f77 100644 --- a/libvirt.spec.in +++ b/libvirt.spec.in @@ -1758,6 +1758,8 @@ exit 0 %{_libdir}/%{name}/connection-driver/libvirt_driver_qemu.so %dir %attr(0711, root, root) %{_localstatedir}/lib/libvirt/swtpm/ %dir %attr(0711, root, root) %{_localstatedir}/log/swtpm/libvirt/qemu/ +%{_bindir}/virt-qemu-run +%{_mandir}/man1/virt-qemu-run.1* %endif %if %{with_lxc} diff --git a/src/Makefile.am b/src/Makefile.am index cd01796d67..a5a757c148 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -87,6 +87,7 @@ OPENRC_INIT_FILES_IN = OPENRC_CONF_FILES = SYSCONF_FILES = sbin_PROGRAMS = +bin_PROGRAMS = DRIVER_SOURCES = COMMON_UNIT_VARS = \ diff --git a/src/qemu/Makefile.inc.am b/src/qemu/Makefile.inc.am index 967f6e75a2..c6b04c3217 100644 --- a/src/qemu/Makefile.inc.am +++ b/src/qemu/Makefile.inc.am @@ -243,3 +243,16 @@ EXTRA_DIST += \ qemu/THREADS.txt \ libvirt_qemu_probes.d \ $(NULL) + +QEMU_SHIM_SOURCES = qemu/qemu_shim.c + +EXTRA_DIST += $(QEMU_SHIM_SOURCES) + +if WITH_QEMU +bin_PROGRAMS += virt-qemu-run + +virt_qemu_run_SOURCES = $(QEMU_SHIM_SOURCES) + +virt_qemu_run_LDADD = libvirt.la +virt_qemu_run_LDFLAGS = -Wl,--export-dynamic +endif WITH_QEMU diff --git a/src/qemu/qemu_shim.c b/src/qemu/qemu_shim.c new file mode 100644 index 0000000000..b10d95c996 --- /dev/null +++ b/src/qemu/qemu_shim.c @@ -0,0 +1,322 @@ +/* + * qemu_shim.c: standalone binary for running QEMU instances + * + * Copyright (C) 2019 Red Hat, Inc. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library. If not, see + * <http://www.gnu.org/licenses/>;. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdbool.h> + +#include "virfile.h" +#include "virstring.h" +#include "virgettext.h" + +#define VIR_FROM_THIS VIR_FROM_QEMU + +static bool eventQuitFlag; +static int eventQuitFD = -1; +static virDomainPtr dom; + +static void * +qemuShimEventLoop(void *opaque G_GNUC_UNUSED) +{ + while (!eventQuitFlag) + virEventRunDefaultImpl(); + + return NULL; +} + +/* Runs in event loop thread context */ +static void +qemuShimEventLoopStop(int watch G_GNUC_UNUSED, + int fd G_GNUC_UNUSED, + int event G_GNUC_UNUSED, + void *opaque G_GNUC_UNUSED) +{ + char c; + ignore_value(read(fd, &c, 1)); + eventQuitFlag = true; +} + +/* Runs in event loop thread context */ +static int +qemuShimDomShutdown(virConnectPtr econn G_GNUC_UNUSED, + virDomainPtr edom G_GNUC_UNUSED, + int event, + int detail G_GNUC_UNUSED, + void *opaque G_GNUC_UNUSED) +{ + if (event == VIR_DOMAIN_EVENT_STOPPED) + eventQuitFlag = true; + + return 0; +} + +/* Runs in unknown thread context */ +static void +qemuShimSigShutdown(int sig G_GNUC_UNUSED) +{ + if (dom) + virDomainDestroy(dom); + ignore_value(safewrite(eventQuitFD, "c", 1)); +} + +static void +qemuShimQuench(void *userData G_GNUC_UNUSED, + virErrorPtr error G_GNUC_UNUSED) +{ +} + +int main(int argc, char **argv) +{ + GThread *eventLoopThread = NULL; + virConnectPtr conn = NULL; + virConnectPtr sconn = NULL; + g_autofree char *xml = NULL; + g_autofree char *uri = NULL; + g_autofree char *suri = NULL; + char *root = NULL; + bool tmproot = false; + int ret = 1; + g_autoptr(GError) error = NULL; + g_auto(GStrv) secrets = NULL; + gboolean verbose = false; + gboolean debug = false; + GStrv tmpsecrets; + GOptionContext *ctx; + GOptionEntry entries[] = { + { "secret", 's', 0, G_OPTION_ARG_STRING_ARRAY, &secrets, "Load secret file", "SECRET-XML-FILE,SECRET-VALUE-FILE" }, + { "root", 'r', 0, G_OPTION_ARG_STRING, &root, "Root directory", "DIR"}, + { "debug", 'd', 0, G_OPTION_ARG_NONE, &debug, "Debug output", NULL }, + { "verbose", 'v', 0, G_OPTION_ARG_NONE, &verbose, "Verbose output", NULL }, + { NULL } + }; + int quitfd[2] = {-1, -1}; + long long start = g_get_monotonic_time(); + +#define deltams() ((long long)g_get_monotonic_time() - start) + + ctx = g_option_context_new("- run a standalone QEMU process"); + g_option_context_add_main_entries(ctx, entries, PACKAGE); + if (!g_option_context_parse(ctx, &argc, &argv, &error)) { + g_printerr("%s: option parsing failed: %s\n", + argv[0], error->message); + return 1; + } + + if (argc != 2) { + g_autofree char *help = g_option_context_get_help(ctx, TRUE, NULL); + g_printerr("%s", help); + return 1; + } + + if (verbose) + g_printerr("%s: %lld: initializing libvirt\n", + argv[0], deltams()); + + if (virInitialize() < 0) { + g_printerr("%s: cannot initialize libvirt\n", argv[0]); + return 1; + } + if (virGettextInitialize() < 0) { + g_printerr("%s: cannot initialize libvirt translations\n", argv[0]); + return 1; + } + + virSetErrorFunc(NULL, qemuShimQuench); + + if (verbose) + g_printerr("%s: %lld: initializing signal handlers\n", + argv[0], deltams()); + + signal(SIGTERM, qemuShimSigShutdown); + signal(SIGINT, qemuShimSigShutdown); + signal(SIGQUIT, qemuShimSigShutdown); + signal(SIGHUP, qemuShimSigShutdown); + + if (root == NULL) { + if (!(root = g_dir_make_tmp("libvirt-qemu-shim-XXXXXX", &error))) { + g_printerr("%s: cannot create temporary dir: %s\n", + argv[0], error->message); + return 1; + } + tmproot = true; + } + + virFileActivateDirOverrideForProg(argv[0]); + + if (verbose) + g_printerr("%s: %lld: preparing event loop thread\n", + argv[0], deltams()); + virEventRegisterDefaultImpl(); + + if (pipe(quitfd) < 0) { + g_printerr("%s: cannot create event loop pipe: %s", + argv[0], g_strerror(errno)); + goto cleanup; + } + + if (virEventAddHandle(quitfd[0], VIR_EVENT_HANDLE_READABLE, qemuShimEventLoopStop, NULL, NULL) < 0) { + VIR_FORCE_CLOSE(quitfd[0]); + VIR_FORCE_CLOSE(quitfd[1]); + quitfd[0] = quitfd[1] = -1; + g_printerr("%s: cannot register event loop handle: %s", + argv[0], virGetLastErrorMessage()); + goto cleanup; + } + eventQuitFD = quitfd[1]; + + eventLoopThread = g_thread_new("event-loop", qemuShimEventLoop, NULL); + + if (secrets && *secrets) { + suri = g_strdup_printf("secret:///embed?root=%s", root); + + if (verbose) + g_printerr("%s: %lld: opening %s\n", + argv[0], deltams(), suri); + + sconn = virConnectOpen(suri); + if (!sconn) { + g_printerr("%s: cannot open %s: %s\n", + argv[0], suri, virGetLastErrorMessage()); + goto cleanup; + } + + tmpsecrets = secrets; + while (tmpsecrets && *tmpsecrets) { + g_auto(GStrv) bits = g_strsplit(*tmpsecrets, ",", 2); + g_autofree char *sxml = NULL; + g_autofree char *value = NULL; + virSecretPtr sec; + size_t nvalue; + + if (!bits || bits[0] == NULL || bits[1] == NULL) { + g_printerr("%s: expected a pair of filenames for --secret argument\n", + argv[0]); + goto cleanup; + } + + if (verbose) + g_printerr("%s: %lld: loading secret %s and %s\n", + argv[0], deltams(), bits[0], bits[1]); + + if (!g_file_get_contents(bits[0], &sxml, NULL, &error)) { + g_printerr("%s: cannot read secret XML %s: %s\n", + argv[0], bits[0], error->message); + goto cleanup; + } + + if (!g_file_get_contents(bits[1], &value, &nvalue, &error)) { + g_printerr("%s: cannot read secret value %s: %s\n", + argv[0], bits[1], error->message); + goto cleanup; + } + + if (!(sec = virSecretDefineXML(sconn, sxml, 0))) { + g_printerr("%s: cannot define secret %s: %s\n", + argv[0], bits[0], virGetLastErrorMessage()); + goto cleanup; + } + + if (virSecretSetValue(sec, (unsigned char *)value, nvalue, 0) < 0) { + virSecretFree(sec); + g_printerr("%s: cannot set value for secret %s: %s\n", + argv[0], bits[0], virGetLastErrorMessage()); + goto cleanup; + } + virSecretFree(sec); + + tmpsecrets++; + } + } + + uri = g_strdup_printf("qemu:///embed?root=%s", root); + + if (verbose) + g_printerr("%s: %lld: opening %s\n", + argv[0], deltams(), uri); + + conn = virConnectOpen(uri); + if (!conn) { + g_printerr("%s: cannot open %s: %s\n", + argv[0], uri, virGetLastErrorMessage()); + goto cleanup; + } + + if (virConnectDomainEventRegisterAny( + conn, dom, VIR_DOMAIN_EVENT_ID_LIFECYCLE, + VIR_DOMAIN_EVENT_CALLBACK(qemuShimDomShutdown), + NULL, NULL) < 0) { + g_printerr("%s: cannot regiser for lifecycle events: %s\n", + argv[0], virGetLastErrorMessage()); + goto cleanup; + } + + if (verbose) + g_printerr("%s: %lld: starting guest %s\n", + argv[0], deltams(), argv[1]); + + if (!g_file_get_contents(argv[1], &xml, NULL, &error)) { + g_printerr("%s: cannot read %s: %s\n", + argv[0], argv[1], error->message); + goto cleanup; + } + + dom = virDomainCreateXML(conn, xml, 0); + if (!dom) { + g_printerr("%s: cannot start VM: %s\n", + argv[0], virGetLastErrorMessage()); + goto cleanup; + } + if (verbose) + g_printerr("%s: %lld: guest running, Ctrl-C to stop nowbbbb\n", + argv[0], deltams()); + + if (debug) { + g_autofree char *newxml = NULL; + newxml = virDomainGetXMLDesc(dom, 0); + g_printerr("%s: XML: %s\n", argv[0], newxml); + } + + ret = 0; + + cleanup: + if (ret != 0 && eventQuitFD != -1) + ignore_value(safewrite(eventQuitFD, "c", 1)); + + if (eventLoopThread != NULL && (ret == 0 || eventQuitFD != -1)) + g_thread_join(eventLoopThread); + + VIR_FORCE_CLOSE(quitfd[0]); + VIR_FORCE_CLOSE(quitfd[1]); + + if (dom != NULL) + virDomainFree(dom); + if (sconn != NULL) + virConnectClose(sconn); + if (conn != NULL) + virConnectClose(conn); + if (tmproot) + virFileDeleteTree(root); + + if (verbose) + g_printerr("%s: %lld: cleaned up, exiting\n", + argv[0], deltams()); + return ret; +} -- 2.23.0