[libvirt] [PATCH 7/7] qemu: introduce a new "virt-qemu-run" program

The previous "QEMU shim" proof of concept was taking an approach of only caring about initial spawning of the QEMU process. It was then registered with the libvirtd daemon who took over management of it. The intent was that later libvirtd would be refactored so that the shim retained control over the QEMU monitor and libvirt just forwarded APIs to each shim as needed. This forwarding of APIs would require quite alot of significant refactoring of libvirtd to achieve. This impl thus takes a quite different approach, explicitly deciding to keep the VMs completely separate from those seen & managed by libvirtd. Instead it uses the new "qemu:///embed" URI scheme to embed the entire QEMU driver in the shim, running with a custom root directory. Once the driver is initialization, the shim starts a VM and then waits to shutdown automatically when QEMU shuts down, or should kill QEMU if it is terminated itself. This ought to use the AUTO_DESTROY feature but that is not yet available in embedded mode, so we rely on installing a few signal handlers to gracefully kill QEMU. This isn't reliable if we crash of course, but you can restart with the same root dir. Note this program does not expose any way to manage the QEMU process, since there's no RPC interface enabled. It merely starts the VM and cleans up when the guest shuts down at the end. This program is installed to /usr/bin/virt-qemu-run enabling direct use by end users. Most use cases will probably want to integrate the concept directly into their respective application codebases. This standalone binary serves as a nice demo though, and also provides a way to measure performance of the startup process quite simply. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com&gt; --- build-aux/syntax-check.mk | 2 +- libvirt.spec.in | 2 + src/Makefile.am | 9 ++ src/qemu/Makefile.inc.am | 26 ++++ src/qemu/qemu_shim.c | 313 ++++++++++++++++++++++++++++++++++++++ src/qemu/qemu_shim.pod | 94 ++++++++++++ 6 files changed, 445 insertions(+), 1 deletion(-) create mode 100644 src/qemu/qemu_shim.c create mode 100644 src/qemu/qemu_shim.pod diff --git a/build-aux/syntax-check.mk b/build-aux/syntax-check.mk index 94e4945323..627734fc55 100644 --- a/build-aux/syntax-check.mk +++ b/build-aux/syntax-check.mk @@ -2340,7 +2340,7 @@ exclude_file_name_regexp--sc_prohibit_devname = \ ^(tools/virsh.pod|build-aux/syntax-check\.mk|docs/.*)$$ exclude_file_name_regexp--sc_prohibit_virXXXFree = \ - ^(docs/|tests/|examples/|tools/|build-aux/syntax-check\.mk|src/test/test_driver.c|src/libvirt_public.syms|include/libvirt/libvirt-(domain|network|nodedev|storage|stream|secret|nwfilter|interface|domain-snapshot).h|src/libvirt-(domain|qemu|network|nodedev|storage|stream|secret|nwfilter|interface|domain-snapshot).c$$) + ^(docs/|tests/|examples/|tools/|build-aux/syntax-check\.mk|src/test/test_driver.c|src/libvirt_public.syms|include/libvirt/libvirt-(domain|network|nodedev|storage|stream|secret|nwfilter|interface|domain-snapshot).h|src/libvirt-(domain|qemu|network|nodedev|storage|stream|secret|nwfilter|interface|domain-snapshot).c|src/qemu/qemu_shim.c$$) exclude_file_name_regexp--sc_prohibit_sysconf_pagesize = \ ^(build-aux/syntax-check\.mk|src/util/virutil\.c)$$ diff --git a/libvirt.spec.in b/libvirt.spec.in index c7fac104d9..886c7b73d5 100644 --- a/libvirt.spec.in +++ b/libvirt.spec.in @@ -1757,6 +1757,8 @@ exit 0 %{_libdir}/%{name}/connection-driver/libvirt_driver_qemu.so %dir %attr(0711, root, root) %{_localstatedir}/lib/libvirt/swtpm/ %dir %attr(0711, root, root) %{_localstatedir}/log/swtpm/libvirt/qemu/ +%{_bindir}/virt-qemu-run +%{_mandir}/man1/virt-qemu-run.1* %endif %if %{with_lxc} diff --git a/src/Makefile.am b/src/Makefile.am index 696e64c52e..f86ba3fe3d 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -90,6 +90,8 @@ SYSTEMD_UNIT_FILES = SYSTEMD_UNIT_FILES_IN = SYSCONF_FILES = sbin_PROGRAMS = +bin_PROGRAMS = +man1_MANS = man8_MANS = DRIVER_SOURCES = man7_MANS = @@ -659,6 +661,13 @@ endif WITH_LIBVIRTD < $< > $@-t && \ mv $@-t $@ +%.1: %.1.in $(top_srcdir)/configure.ac + $(AM_V_GEN)sed \ + -e 's|[@]sysconfdir[@]|$(sysconfdir)|g' \ + -e 's|[@]runstatedir[@]|$(runstatedir)|g' \ + < $< > $@-t && \ + mv $@-t $@ + CLEANFILES += \ $(man8_MANS) \ $(MANINFILES) diff --git a/src/qemu/Makefile.inc.am b/src/qemu/Makefile.inc.am index bf30f8a3c5..d41853ac9e 100644 --- a/src/qemu/Makefile.inc.am +++ b/src/qemu/Makefile.inc.am @@ -231,3 +231,29 @@ EXTRA_DIST += \ qemu/THREADS.txt \ libvirt_qemu_probes.d \ $(NULL) + +QEMU_SHIM_SOURCES = qemu/qemu_shim.c +QEMU_SHIM_POD = qemu/qemu_shim.pod + +EXTRA_DIST += $(QEMU_SHIM_SOURCES) $(QEMU_SHIM_POD) + +if WITH_QEMU +bin_PROGRAMS += virt-qemu-run + +man1_MANS += virt-qemu-run.1 + +virt_qemu_run_SOURCES = $(QEMU_SHIM_SOURCES) + +virt_qemu_run_LDADD = libvirt.la +virt_qemu_run_LDFLAGS = -Wl,--export-dynamic +endif WITH_QEMU + +virt-qemu-run.1.in: qemu/qemu_shim.pod + $(AM_V_GEN)$(POD2MAN) $< $@-t1 && \ + if grep 'POD ERROR' $@-t1; then rm $@-t1; exit 1; fi && \ + sed \ + -e 's|SYSCONFDIR|\@sysconfdir\@|g' \ + -e 's|LOCALSTATEDIR|\@localstatedir\@|g' \ + < $@-t1 > $@-t2 && \ + rm -f $@-t1 && \ + mv $@-t2 $@ diff --git a/src/qemu/qemu_shim.c b/src/qemu/qemu_shim.c new file mode 100644 index 0000000000..10f9682143 --- /dev/null +++ b/src/qemu/qemu_shim.c @@ -0,0 +1,313 @@ +/* + * qemu_shim.c: standalone binary for running QEMU instances + * + * Copyright (C) 2019 Red Hat, Inc. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library. If not, see + * <http://www.gnu.org/licenses/>;. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdbool.h> + +#include "virfile.h" +#include "virstring.h" +#include "virgettext.h" + +#define VIR_FROM_THIS VIR_FROM_QEMU + +static bool eventQuitFlag; +static int eventQuitFD = -1; +static virDomainPtr dom; + +static void * +qemuShimEventLoop(void *opaque G_GNUC_UNUSED) +{ + while (!eventQuitFlag) + virEventRunDefaultImpl(); + + return NULL; +} + +/* Runs in event loop thread context */ +static void +qemuShimEventLoopStop(int watch G_GNUC_UNUSED, + int fd G_GNUC_UNUSED, + int event G_GNUC_UNUSED, + void *opaque G_GNUC_UNUSED) +{ + char c; + ignore_value(read(fd, &c, 1)); + eventQuitFlag = true; +} + +/* Runs in event loop thread context */ +static int +qemuShimDomShutdown(virConnectPtr econn G_GNUC_UNUSED, + virDomainPtr edom G_GNUC_UNUSED, + int event, + int detail G_GNUC_UNUSED, + void *opaque G_GNUC_UNUSED) +{ + if (event == VIR_DOMAIN_EVENT_STOPPED) + eventQuitFlag = true; + + return 0; +} + +/* Runs in unknown thread context */ +static void +qemuShimSigShutdown(int sig G_GNUC_UNUSED) +{ + if (dom) + virDomainDestroy(dom); + ignore_value(safewrite(eventQuitFD, "c", 1)); +} + +static void +qemuShimQuench(void *userData G_GNUC_UNUSED, + virErrorPtr error G_GNUC_UNUSED) +{ +} + +int main(int argc, char **argv) +{ + GThread *eventLoopThread = NULL; + virConnectPtr conn = NULL; + virConnectPtr sconn = NULL; + g_autofree char *xml = NULL; + g_autofree char *uri = NULL; + g_autofree char *suri = NULL; + char *root = NULL; + bool tmproot = false; + int ret = 1; + g_autoptr(GError) error = NULL; + g_auto(GStrv) secrets = NULL; + bool verbose = false; + GStrv tmpsecrets; + GOptionContext *ctx; + GOptionEntry entries[] = { + { "secret", 's', 0, G_OPTION_ARG_STRING_ARRAY, &secrets, "Load secret file", "SECRET-XML-FILE,SECRET-VALUE-FILE" }, + { "root", 'r', 0, G_OPTION_ARG_STRING, &root, "Root directory", "DIR"}, + { "verbose", 'v', 0, G_OPTION_ARG_NONE, &verbose, "Verbose output", NULL }, + { NULL } + }; + int quitfd[2] = {-1, -1}; + long long start = g_get_monotonic_time(); + +#define deltams() ((long long)g_get_monotonic_time() - start) + + ctx = g_option_context_new("- run a standalone QEMU process"); + g_option_context_add_main_entries(ctx, entries, PACKAGE); + if (!g_option_context_parse(ctx, &argc, &argv, &error)) { + g_printerr("%s: option parsing failed: %s\n", + argv[0], error->message); + return 1; + } + if (argc != 2) { + g_autofree char *help = g_option_context_get_help(ctx, TRUE, NULL); + g_printerr("%s", help); + return 1; + } + + if (verbose) + g_printerr("%s: %lld: initializing libvirt\n", + argv[0], deltams()); + + if (virInitialize() < 0) { + g_printerr("%s: cannot initialize libvirt\n", argv[0]); + return 1; + } + if (virGettextInitialize() < 0) { + g_printerr("%s: cannot initialize libvirt translations\n", argv[0]); + return 1; + } + + virSetErrorFunc(NULL, qemuShimQuench); + + if (verbose) + g_printerr("%s: %lld: initializing signal handlers\n", + argv[0], deltams()); + + signal(SIGTERM, qemuShimSigShutdown); + signal(SIGINT, qemuShimSigShutdown); + signal(SIGQUIT, qemuShimSigShutdown); + signal(SIGHUP, qemuShimSigShutdown); + + if (root == NULL) { + if (!(root = g_dir_make_tmp("libvirt-qemu-shim-XXXXXX", &error))) { + g_printerr("%s: cannot create temporary dir: %s\n", + argv[0], error->message); + return 1; + } + tmproot = true; + } + + virFileActivateDirOverrideForProg(argv[0]); + + if (verbose) + g_printerr("%s: %lld: preparing event loop thread\n", + argv[0], deltams()); + virEventRegisterDefaultImpl(); + + if (pipe(quitfd) < 0) { + g_printerr("%s: cannot create event loop pipe: %s", + argv[0], g_strerror(errno)); + goto cleanup; + } + + if (virEventAddHandle(quitfd[0], VIR_EVENT_HANDLE_READABLE, qemuShimEventLoopStop, NULL, NULL) < 0) { + VIR_FORCE_CLOSE(quitfd[0]); + VIR_FORCE_CLOSE(quitfd[1]); + quitfd[0] = quitfd[1] = -1; + g_printerr("%s: cannot register event loop handle: %s", + argv[0], virGetLastErrorMessage()); + goto cleanup; + } + eventQuitFD = quitfd[1]; + + eventLoopThread = g_thread_new("event-loop", qemuShimEventLoop, NULL); + + if (secrets && *secrets) { + suri = g_strdup_printf("secret:///embed?root=%s", root); + + if (verbose) + g_printerr("%s: %lld: opening %s\n", + argv[0], deltams(), suri); + + sconn = virConnectOpen(suri); + if (!sconn) { + g_printerr("%s: cannot open %s: %s\n", + argv[0], suri, virGetLastErrorMessage()); + goto cleanup; + } + + tmpsecrets = secrets; + while (tmpsecrets && *tmpsecrets) { + g_auto(GStrv) bits = g_strsplit(*tmpsecrets, ",", 2); + g_autofree char *sxml = NULL; + g_autofree char *value = NULL; + virSecretPtr sec; + size_t nvalue; + + if (!bits || bits[0] == NULL || bits[1] == NULL) { + g_printerr("%s: expected a pair of filenames for --secret argument\n", + argv[0]); + goto cleanup; + } + + if (verbose) + g_printerr("%s: %lld: loading secret %s and %s\n", + argv[0], deltams(), bits[0], bits[1]); + + if (!g_file_get_contents(bits[0], &sxml, NULL, &error)) { + g_printerr("%s: cannot read secret XML %s: %s\n", + argv[0], bits[0], error->message); + goto cleanup; + } + + if (!g_file_get_contents(bits[1], &value, &nvalue, &error)) { + g_printerr("%s: cannot read secret value %s: %s\n", + argv[0], bits[1], error->message); + goto cleanup; + } + + if (!(sec = virSecretDefineXML(sconn, sxml, 0))) { + g_printerr("%s: cannot define secret %s: %s\n", + argv[0], bits[0], virGetLastErrorMessage()); + goto cleanup; + } + + if (virSecretSetValue(sec, (unsigned char *)value, nvalue, 0) < 0) { + virSecretFree(sec); + g_printerr("%s: cannot set value for secret %s: %s\n", + argv[0], bits[0], virGetLastErrorMessage()); + goto cleanup; + } + virSecretFree(sec); + + tmpsecrets++; + } + } + + uri = g_strdup_printf("qemu:///embed?root=%s", root); + + if (verbose) + g_printerr("%s: %lld: opening %s\n", + argv[0], deltams(), uri); + + conn = virConnectOpen(uri); + if (!conn) { + g_printerr("%s: cannot open %s: %s\n", + argv[0], uri, virGetLastErrorMessage()); + goto cleanup; + } + + if (virConnectDomainEventRegisterAny( + conn, dom, VIR_DOMAIN_EVENT_ID_LIFECYCLE, + VIR_DOMAIN_EVENT_CALLBACK(qemuShimDomShutdown), + NULL, NULL) < 0) { + g_printerr("%s: cannot regiser for lifecycle events: %s\n", + argv[0], virGetLastErrorMessage()); + goto cleanup; + } + + if (verbose) + g_printerr("%s: %lld: starting guest %s\n", + argv[0], deltams(), argv[1]); + + if (!g_file_get_contents(argv[1], &xml, NULL, &error)) { + g_printerr("%s: cannot read %s: %s\n", + argv[0], argv[1], error->message); + goto cleanup; + } + + dom = virDomainCreateXML(conn, xml, 0); + if (!dom) { + g_printerr("%s: cannot start VM: %s\n", + argv[0], virGetLastErrorMessage()); + goto cleanup; + } + if (verbose) + g_printerr("%s: %lld: guest running, Ctrl-C to stop\n", + argv[0], deltams()); + + ret = 0; + + cleanup: + if (ret != 0 && eventQuitFD != -1) + ignore_value(safewrite(eventQuitFD, "c", 1)); + + if (eventLoopThread != NULL && (ret == 0 || eventQuitFD != -1)) + g_thread_join(eventLoopThread); + + VIR_FORCE_CLOSE(quitfd[0]); + VIR_FORCE_CLOSE(quitfd[1]); + + if (dom != NULL) + virDomainFree(dom); + if (sconn != NULL) + virConnectClose(sconn); + if (conn != NULL) + virConnectClose(conn); + if (tmproot) + virFileDeleteTree(root); + + if (verbose) + g_printerr("%s: %lld: cleaned up, exiting\n", + argv[0], deltams()); + return ret; +} diff --git a/src/qemu/qemu_shim.pod b/src/qemu/qemu_shim.pod new file mode 100644 index 0000000000..eb17fa0ac7 --- /dev/null +++ b/src/qemu/qemu_shim.pod @@ -0,0 +1,94 @@ +=encoding utf8 + +=head1 NAME + +virt-qemu-run - run a standalone QEMU guest + +=head1 SYNOPSIS + +B<virt-qemu-run> [I<OPTIONS>...] [I<GUEST-XML>] + +=head1 DESCRIPTION + +This tool provides a way to run a standalone QEMU guest such that it +is completely independant of libvirtd. It makes use of the embedded +QEMU driver support to run the VM placing files under an isolated +directory tree. When the guest is run with this tool it is invisible +to libvirtd and thus also invisible to other libvirt tools such as +virsh. + +The virt-qemu-run program will run the QEMU virtual machine, and then +block until the guest OS shuts down, at which point it will exit. + +If the virt-qemu-run program is interrupted (eg Ctrl-C) it will +immediately terminate the virtual machine without giving the guest +OS any opportunity to gracefully shutdown. + +=head1 OPTIONS + +=over 4 + +=item C<GUEST-XML> + +The full path to the XML file describing the guest virtual machine +to be booted. + +=item C<-h>, C<--help> + +Display the command line help + +=item C<-v>, C<--verbose> + +Display verbose information about startup + +=item C<-r DIR>, C<--root=DIR> + +Specify the root directory to use for storing state associated with +the virtual machine. The caller is responsible for deleting this +directory when it is no longer required. + +If this parameter is omitted, then a random temporary directory +will be created, and its contents be automaticlaly deleted at +VM shutdown. + +=item C<-s XML-FILE,VALUE-FILE> C<--secret=XML-FILE,VALUE-FILE> + +Specify a secret to be loaded into the secret driver. The C<XML-FILE> +is a path to the XML description of the secret, whose UUID should +match a secret referenced in the guest domain XML. The C<VALUE-FILE> +is a path containing the raw value of the secret. + +=back + +=head1 EXIT STATUS + +Upon successful shutdown, an exit status of 0 will be set. Upon +failure a non-zero status will be set. + +=head1 AUTHOR + +Daniel P. Berrangé + +=head1 BUGS + +Report any bugs discovered to the libvirt community via the +mailing list L<https://libvirt.org/contact.html> or bug tracker +L<https://libvirt.org/bugs.html>. +Alternatively report bugs to your software distributor / vendor. + +=head1 COPYRIGHT + +Copyright (C) 2019 by Red Hat, Inc. + +=head1 LICENSE + +virt-run-qemu is distributed under the terms of the GNU LGPL v2+. +This is free software; see the source for copying conditions. There +is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR +PURPOSE + +=head1 SEE ALSO + +L<virsh(1)> + +=cut -- 2.23.0