- Devel - libvirt List Archives

[libvirt] [PATCH] ci: Fix VPATH builds
by Andrea Bolognani 12 Dec '19

12 Dec '19

The CI build machinery is intentionally not handled by autotools, so for VPATH builds - which are mandatory now - we need to make sure we're looking into $(srcdir). Signed-off-by: Andrea Bolognani <abologna(a)redhat.com> --- Makefile.am | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile.am b/Makefile.am index 15df6ed68e..4edb3fdd43 100644 --- a/Makefile.am +++ b/Makefile.am @@ -148,4 +148,4 @@ gen-AUTHORS: fi ci-%: - $(MAKE) -C ci/ $@ + $(MAKE) -C $(srcdir)/ci/ $@ -- 2.23.0

2 1

[libvirt] [dockerfiles PATCH] Add CentOS 8 Dockerfiles
by Andrea Bolognani 12 Dec '19

12 Dec '19

The corresponding libvirt-jenkins-ci commit is b7cd4b07a064. Signed-off-by: Andrea Bolognani <abologna(a)redhat.com> --- Pushed under the Dockerfile refresh rule. Text diff follows. buildenv-libosinfo-centos-8.zip | Bin 0 -> 578 bytes buildenv-libvirt-centos-8.zip | Bin 0 -> 781 bytes 2 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 buildenv-libosinfo-centos-8.zip create mode 100644 buildenv-libvirt-centos-8.zip diff --git a/buildenv-libosinfo-centos-8.zip b/buildenv-libosinfo-centos-8.zip new file mode 100644 index 0000000..e739059 --- /dev/null +++ b/buildenv-libosinfo-centos-8.zip @@ -0,0 +1,55 @@ +FROM centos:centos8 + +RUN dnf update -y && \ + dnf install 'dnf-command(config-manager)' -y && \ + dnf config-manager --set-enabled PowerTools -y && \ + dnf install -y epel-release && \ + dnf install -y \ + autoconf \ + automake \ + bash \ + bash-completion \ + ca-certificates \ + check-devel \ + chrony \ + gcc \ + gdb \ + gettext \ + gettext-devel \ + git \ + glib2-devel \ + glibc-devel \ + gobject-introspection-devel \ + gtk-doc \ + hwdata \ + intltool \ + json-glib-devel \ + libarchive-devel \ + libsoup-devel \ + libtool \ + libxml2 \ + libxml2-devel \ + libxslt-devel \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconfig \ + python3 \ + python3-lxml \ + python3-pytest \ + python3-requests \ + python3-setuptools \ + rpm-build \ + screen \ + strace \ + sudo \ + vala \ + vim && \ + dnf autoremove -y && \ + dnf clean all -y + +ENV LANG "en_US.UTF-8" diff --git a/buildenv-libvirt-centos-8.zip b/buildenv-libvirt-centos-8.zip new file mode 100644 index 0000000..5410201 --- /dev/null +++ b/buildenv-libvirt-centos-8.zip @@ -0,0 +1,93 @@ +FROM centos:centos8 + +RUN dnf update -y && \ + dnf install 'dnf-command(config-manager)' -y && \ + dnf config-manager --set-enabled PowerTools -y && \ + dnf install -y epel-release && \ + dnf install -y \ + audit-libs-devel \ + augeas \ + autoconf \ + automake \ + avahi-devel \ + bash \ + bash-completion \ + ca-certificates \ + chrony \ + cyrus-sasl-devel \ + dbus-devel \ + device-mapper-devel \ + dnsmasq \ + ebtables \ + fuse-devel \ + gcc \ + gdb \ + gettext \ + gettext-devel \ + git \ + glib2-devel \ + glibc-devel \ + glusterfs-api-devel \ + gnutls-devel \ + iproute \ + iproute-tc \ + iscsi-initiator-utils \ + kmod \ + libacl-devel \ + libattr-devel \ + libblkid-devel \ + libcap-ng-devel \ + libcurl-devel \ + libiscsi-devel \ + libnl3-devel \ + libpcap-devel \ + libpciaccess-devel \ + librbd-devel \ + libselinux-devel \ + libssh-devel \ + libtirpc-devel \ + libtool \ + libudev-devel \ + libwsman-devel \ + libxml2 \ + libxml2-devel \ + libxslt \ + lsof \ + lvm2 \ + make \ + meson \ + ncurses-devel \ + net-tools \ + netcf-devel \ + nfs-utils \ + ninja-build \ + numactl-devel \ + numad \ + parted \ + parted-devel \ + patch \ + perl \ + pkgconfig \ + polkit \ + python3 \ + python3-docutils \ + python3-flake8 \ + python3-setuptools \ + qemu-img \ + radvd \ + readline-devel \ + rpcgen \ + rpm-build \ + sanlock-devel \ + screen \ + scrub \ + strace \ + sudo \ + systemtap-sdt-devel \ + vim \ + xfsprogs-devel \ + yajl-devel && \ + dnf autoremove -y && \ + dnf clean all -y + +ENV LANG "en_US.UTF-8" -- 2.23.0

1 0

[libvirt] [dockerfiles PATCH] Refresh Dockerfiles after dropping Python 2 support
by Andrea Bolognani 12 Dec '19

12 Dec '19

The corresponding libvirt-jenkins-ci commit is 8c30b5e1782b. Signed-off-by: Andrea Bolognani <abologna(a)redhat.com> --- Pushed under the Dockerfile refresh rule. Text diff follows. buildenv-libosinfo-centos-7.zip | Bin 575 -> 572 bytes buildenv-libvirt-centos-7.zip | Bin 765 -> 761 bytes 2 files changed, 0 insertions(+), 0 deletions(-) diff --git a/buildenv-libosinfo-centos-7.zip b/buildenv-libosinfo-centos-7.zip index c9dd791..b91f3aa 100644 --- a/buildenv-libosinfo-centos-7.zip +++ b/buildenv-libosinfo-centos-7.zip @@ -36,7 +36,6 @@ RUN yum update -y && \ patch \ perl \ pkgconfig \ - python2-setuptools \ python3 \ python3-pip \ python3-setuptools \ diff --git a/buildenv-libvirt-centos-7.zip b/buildenv-libvirt-centos-7.zip index 5fc4f71..4e9c3db 100644 --- a/buildenv-libvirt-centos-7.zip +++ b/buildenv-libvirt-centos-7.zip @@ -67,7 +67,6 @@ RUN yum update -y && \ perl \ pkgconfig \ polkit \ - python2-setuptools \ python3 \ python3-pip \ python3-setuptools \ -- 2.23.0

1 0

[libvirt] [jenkins-ci PATCH] Run the virt-manager test suite on FreeBSD
by Andrea Bolognani 12 Dec '19

12 Dec '19

The bug affecting python3-libxml2 has been fixed in FreeBSD, and virt-manager itself has recently tweaked its test suite so that it works correctly on that platform as well. Signed-off-by: Andrea Bolognani <abologna(a)redhat.com> --- guests/playbooks/build/projects/virt-manager.yml | 6 +++--- jenkins/projects/virt-manager.yaml | 5 ++--- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/guests/playbooks/build/projects/virt-manager.yml b/guests/playbooks/build/projects/virt-manager.yml index 9184a74..9286931 100644 --- a/guests/playbooks/build/projects/virt-manager.yml +++ b/guests/playbooks/build/projects/virt-manager.yml @@ -23,9 +23,6 @@ $PYTHON ./setup.py configure --prefix=$VIRT_PREFIX - include: '{{ playbook_base }}/jobs/python-distutils-check-job.yml' vars: - # libxml2's Python 3 bindings don't work properly on FreeBSD, - # so skip the test suite there for the time being. See - # https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=224902 machines: - libvirt-debian-9 - libvirt-debian-10 @@ -33,6 +30,9 @@ - libvirt-fedora-30 - libvirt-fedora-31 - libvirt-fedora-rawhide + - libvirt-freebsd-11 + - libvirt-freebsd-12 + - libvirt-freebsd-current - libvirt-ubuntu-1804 - include: '{{ playbook_base }}/jobs/python-distutils-rpm-job.yml' vars: diff --git a/jenkins/projects/virt-manager.yaml b/jenkins/projects/virt-manager.yaml index 4524831..b027f21 100644 --- a/jenkins/projects/virt-manager.yaml +++ b/jenkins/projects/virt-manager.yaml @@ -22,15 +22,14 @@ $PYTHON ./setup.py configure --prefix=$VIRT_PREFIX - python-distutils-check-job: parent_jobs: 'virt-manager-build' - # libxml2's Python 3 bindings don't work properly on FreeBSD, - # so skip the test suite there for the time being. See - # https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=224902 machines: - libvirt-debian-9 - libvirt-debian-10 - libvirt-fedora-30 - libvirt-fedora-31 - libvirt-fedora-rawhide + - libvirt-freebsd-11 + - libvirt-freebsd-12 - python-distutils-rpm-job: parent_jobs: 'virt-manager-check' machines: -- 2.23.0

2 1

[libvirt] [PATCH] docs: ensure outputfile is deleted if rst2html/rst2man fail
by Daniel P. Berrangé 11 Dec '19

11 Dec '19

This avoids leaving a zero length or partially generated output file on errors. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- docs/Makefile.am | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/Makefile.am b/docs/Makefile.am index eb8de80b9c..9a1f7a6117 100644 --- a/docs/Makefile.am +++ b/docs/Makefile.am @@ -259,21 +259,21 @@ man8_MANS = $(manpages8_rst:%.rst=%.8) grep -v '^\.\. contents::' < $< | \ sed -e 's|SYSCONFDIR|$(sysconfdir)|g' \ -e 's|RUNSTATEDIR|$(runstatedir)|g' | \ - $(RST2MAN) > $@ + $(RST2MAN) > $@ || { rm $@ && exit 1; } %.7: %.rst $(AM_V_GEN)$(MKDIR_P) `dirname $@` && \ grep -v '^\.\. contents::' < $< | \ sed -e 's|SYSCONFDIR|$(sysconfdir)|g' \ -e 's|RUNSTATEDIR|$(runstatedir)|g' | \ - $(RST2MAN) > $@ + $(RST2MAN) > $@ || { rm $@ && exit 1; } %.8: %.rst $(AM_V_GEN)$(MKDIR_P) `dirname $@` && \ grep -v '^\.\. contents::' < $< | \ sed -e 's|SYSCONFDIR|$(sysconfdir)|g' \ -e 's|RUNSTATEDIR|$(runstatedir)|g' | \ - $(RST2MAN) > $@ + $(RST2MAN) > $@ || { rm $@ && exit 1; } manpages/virkeycode-%.rst: $(top_srcdir)/src/keycodemapdb/data/keymaps.csv \ $(top_srcdir)/src/keycodemapdb/tools/keymap-gen Makefile.am @@ -414,11 +414,11 @@ manpages/%.html.in: manpages/%.rst grep -v '^:Manual ' < $< | \ sed -e 's|SYSCONFDIR|$(sysconfdir)|g' \ -e 's|RUNSTATEDIR|$(runstatedir)|g' | \ - $(RST2HTML) > $@ + $(RST2HTML) > $@ || { rm $@ && exit 1; } %.html.in: %.rst $(AM_V_GEN)$(MKDIR_P) `dirname $@` && \ - $(RST2HTML) $< > $@ + $(RST2HTML) $< > $@ || { rm $@ && exit 1; } %.html.tmp: %.html.in site.xsl subsite.xsl page.xsl \ $(acl_generated) -- 2.23.0

2 3

[libvirt] [PATCH] doc: vtpm only support secrets by UUID at this point
by marcandre.lureau＠redhat.com 11 Dec '19

11 Dec '19

From: Marc-André Lureau <marcandre.lureau(a)redhat.com> Support by usage name can be considered separately (with a 'usage' attribute?). Cc: Stefan Berger <stefanb(a)linux.ibm.com> Signed-off-by: Marc-André Lureau <marcandre.lureau(a)redhat.com> --- docs/formatsecret.html.in | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/formatsecret.html.in b/docs/formatsecret.html.in index 8d0630a7c3..8f5383cf64 100644 --- a/docs/formatsecret.html.in +++ b/docs/formatsecret.html.in @@ -334,8 +334,8 @@ Secret value set of the vTPM. The <code><usage type='vtpm'></code> element must contain a single <code>name</code> element that specifies a usage name - for the secret. The vTPM secret can then be used by UUID or by - this usage name via the <code><encryption></code> element of + for the secret. The vTPM secret can then be used by UUID + via the <code><encryption></code> element of a <a href="formatdomain.html#elementsTpm">tpm</a> when using an emulator. <span class="since">Since 5.6.0</span>. The following is an example -- 2.24.0.308.g228f53135a

4 3

[libvirt] [PATCH] travis: Update name for Ubuntu 18.04 image
by Andrea Bolognani 11 Dec '19

11 Dec '19

The corresponding libvirt-jenkins-ci commit is f289e64a5fd9. Signed-off-by: Andrea Bolognani <abologna(a)redhat.com> --- Pushed as trivial. .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index b47f54553e..44ff5e9898 100644 --- a/.travis.yml +++ b/.travis.yml @@ -21,7 +21,7 @@ matrix: - services: - docker env: - - IMAGE="ubuntu-18" + - IMAGE="ubuntu-1804" - MAKE_ARGS="syntax-check distcheck" script: - make -C ci/ ci-build@$IMAGE CI_MAKE_ARGS="$MAKE_ARGS" -- 2.23.0

1 0

[libvirt] [PATCH] qemu: keep capabilities when running QEMU as root
by Daniel P. Berrangé 11 Dec '19

11 Dec '19

When QEMU uid/gid is set to non-root this is pointless as if we just used a regular setuid/setgid call, the process will have all its capabilities cleared anyway by the kernel. When QEMU uid/gid is set to root, this is almost (always?) never what people actually want. People make QEMU run as root in order to access some privileged resource that libvirt doesn't support yet and this often requires capabilities. As a result they have to go find the qemu.conf param to turn this off. This is not viable for libguestfs - they want to control everything via the XML security label to request running as root regardless of the qemu.conf settings for user/group. Clearing capabilities was implemented originally because there was a proposal in Fedora to change permissions such that root, with no capabilities would not be able to compromise the system. ie a locked down root account. This never went anywhere though, and as a result clearing capabilities when running as root does not really get us any security benefit AFAICT. The root user can easily do something like create a cronjob, which will then faithfully be run with full capabilities, trivially bypassing the restriction we place. IOW, our clearing of capabilities is both useless from a security POV, and breaks valid use cases when people need to run as root. This removes the clear_emulator_capabilities configuration option from qemu.conf, and always runs QEMU with capabilities when root. The behaviour when non-root is unchanged. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- docs/drvqemu.html.in | 46 +++++++++++------------------- src/qemu/libvirtd_qemu.aug | 8 +++++- src/qemu/qemu.conf | 11 ------- src/qemu/qemu_conf.c | 4 --- src/qemu/qemu_conf.h | 1 - src/qemu/qemu_domain.c | 3 +- src/qemu/qemu_process.c | 5 ---- src/qemu/test_libvirtd_qemu.aug.in | 1 - 8 files changed, 25 insertions(+), 54 deletions(-) diff --git a/docs/drvqemu.html.in b/docs/drvqemu.html.in index 294117ee1f..8beb28655c 100644 --- a/docs/drvqemu.html.in +++ b/docs/drvqemu.html.in @@ -187,41 +187,29 @@ chmod o+x /path/to/directory </li> </ul> - <h3><a id="securitycap">Linux process capabilities</a></h3> - <p> - The libvirt QEMU driver has a build time option allowing it to use - the <a href="http://people.redhat.com/sgrubb/libcap-ng/index.html">libcap-ng</a> - library to manage process capabilities. If this build option is - enabled, then the QEMU driver will use this to ensure that all - process capabilities are dropped before executing a QEMU virtual - machine. Process capabilities are what gives the 'root' account - its high power, in particular the CAP_DAC_OVERRIDE capability - is what allows a process running as 'root' to access files owned - by any user. + The libvirt maintainers <strong>strongly recommend against</strong> + running QEMU as the root user/group. This should not be required + in most supported usage scenarios, as libvirt will generally do the + right thing to grant QEMU access to files it is permitted to + use when it is running non-root. </p> + <h3><a id="securitycap">Linux process capabilities</a></h3> + <p> - If the QEMU driver is configured to run virtual machines as non-root, - then they will already lose all their process capabilities at time - of startup. The Linux capability feature is thus aimed primarily at - the scenario where the QEMU processes are running as root. In this - case, before launching a QEMU virtual machine, libvirtd will use - libcap-ng APIs to drop all process capabilities. It is important - for administrators to note that this implies the QEMU process will - <strong>only</strong> be able to access files owned by root, and - not files owned by any other user. + In versions of libvirt prior to 6.0.0, even if QEMU was configured + to run as the root user / group, libvirt would strip all process + capabilities. This meant that QEMU could only read/write files + owned by root, or with open permissions. In reality, stripping + capabilities did not have any security benefit, as it was trivial + to get commands to run in another context with full capabilities, + for example, by creating a cronjob. </p> - <p> - Thus, if a vendor / distributor has configured their libvirt package - to run as 'qemu' by default, a number of changes will be required - before an administrator can change a host to run guests as root. - In particular it will be necessary to change ownership on the - directories <code>/var/run/libvirt/qemu/</code>, - <code>/var/lib/libvirt/qemu/</code> and - <code>/var/cache/libvirt/qemu/</code> back to root, in addition - to changing the <code>/etc/libvirt/qemu.conf</code> settings. + Thus since 6.0.0, if QEMU is running as root, it will keep all + process capabilities. Behaviour when QEMU is running non-root + is unchanged, it still has no capabilities. </p> <h3><a id="securityselinux">SELinux basic confinement</a></h3> diff --git a/src/qemu/libvirtd_qemu.aug b/src/qemu/libvirtd_qemu.aug index 7d7844dc09..86bea2a32a 100644 --- a/src/qemu/libvirtd_qemu.aug +++ b/src/qemu/libvirtd_qemu.aug @@ -86,7 +86,6 @@ module Libvirtd_qemu = | bool_entry "auto_start_bypass_cache" let process_entry = str_entry "hugetlbfs_mount" - | bool_entry "clear_emulator_capabilities" | str_entry "bridge_helper" | str_entry "pr_helper" | str_entry "slirp_helper" @@ -129,6 +128,12 @@ module Libvirtd_qemu = let swtpm_entry = str_entry "swtpm_user" | str_entry "swtpm_group" + (* Entries that used to exist in the config which are now + * deleted. We keep on parsing them so we don't break + * ability to parse old configs after upgrade + *) + let obsolete_entry = bool_entry "clear_emulator_capabilities" + let capability_filters_entry = str_array_entry "capability_filters" (* Each entry in the config is one of the following ... *) @@ -153,6 +158,7 @@ module Libvirtd_qemu = | nbd_entry | swtpm_entry | capability_filters_entry + | obsolete_entry let comment = [ label "#comment" . del /#[ \t]*/ "# " . store /([^ \t\n][^\n]*)?/ . del /\n/ "\n" ] let empty = [ label "#empty" . eol ] diff --git a/src/qemu/qemu.conf b/src/qemu/qemu.conf index 7a056b037e..b6805ffc41 100644 --- a/src/qemu/qemu.conf +++ b/src/qemu/qemu.conf @@ -583,17 +583,6 @@ #bridge_helper = "/usr/libexec/qemu-bridge-helper" - -# If clear_emulator_capabilities is enabled, libvirt will drop all -# privileged capabilities of the QEMU/KVM emulator. This is enabled by -# default. -# -# Warning: Disabling this option means that a compromised guest can -# exploit the privileges and possibly do damage to the host. -# -#clear_emulator_capabilities = 1 - - # If enabled, libvirt will have QEMU set its process name to # "qemu:VM_NAME", where VM_NAME is the name of the VM. The QEMU # process will appear as "qemu:VM_NAME" in process listings and diff --git a/src/qemu/qemu_conf.c b/src/qemu/qemu_conf.c index 30637b21ac..0e14d60bc9 100644 --- a/src/qemu/qemu_conf.c +++ b/src/qemu/qemu_conf.c @@ -234,8 +234,6 @@ virQEMUDriverConfigPtr virQEMUDriverConfigNew(bool privileged) cfg->prHelperName = g_strdup(QEMU_PR_HELPER); cfg->slirpHelperName = g_strdup(QEMU_SLIRP_HELPER); - cfg->clearEmulatorCapabilities = true; - cfg->securityDefaultConfined = true; cfg->securityRequireConfined = false; @@ -602,8 +600,6 @@ virQEMUDriverConfigLoadProcessEntry(virQEMUDriverConfigPtr cfg, } } - if (virConfGetValueBool(conf, "clear_emulator_capabilities", &cfg->clearEmulatorCapabilities) < 0) - return -1; if (virConfGetValueString(conf, "bridge_helper", &cfg->bridgeHelperName) < 0) return -1; diff --git a/src/qemu/qemu_conf.h b/src/qemu/qemu_conf.h index 95b33a1093..a641bbd983 100644 --- a/src/qemu/qemu_conf.h +++ b/src/qemu/qemu_conf.h @@ -161,7 +161,6 @@ struct _virQEMUDriverConfig { bool relaxedACS; bool vncAllowHostAudio; bool nogfxAllowHostAudio; - bool clearEmulatorCapabilities; bool setProcessName; unsigned int maxProcesses; diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 470d342afc..0dadca4894 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -9345,8 +9345,7 @@ void qemuDomainObjCheckTaint(virQEMUDriverPtr driver, bool custom_hypervisor_feat = false; if (virQEMUDriverIsPrivileged(driver) && - (!cfg->clearEmulatorCapabilities || - cfg->user == 0 || + (cfg->user == 0 || cfg->group == 0)) qemuDomainObjTaint(driver, obj, VIR_DOMAIN_TAINT_HIGH_PRIVILEGES, logCtxt); diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 75ee3893c6..979ea36f0f 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -6811,11 +6811,6 @@ qemuProcessLaunch(virConnectPtr conn, if (qemuDomainCreateNamespace(driver, vm) < 0) goto cleanup; - VIR_DEBUG("Clear emulator capabilities: %d", - cfg->clearEmulatorCapabilities); - if (cfg->clearEmulatorCapabilities) - virCommandClearCaps(cmd); - VIR_DEBUG("Setting up raw IO"); if (qemuProcessSetupRawIO(driver, vm, cmd) < 0) goto cleanup; diff --git a/src/qemu/test_libvirtd_qemu.aug.in b/src/qemu/test_libvirtd_qemu.aug.in index b9cf8d6688..dd90edf687 100644 --- a/src/qemu/test_libvirtd_qemu.aug.in +++ b/src/qemu/test_libvirtd_qemu.aug.in @@ -72,7 +72,6 @@ module Test_libvirtd_qemu = { "auto_start_bypass_cache" = "0" } { "hugetlbfs_mount" = "/dev/hugepages" } { "bridge_helper" = "/usr/libexec/qemu-bridge-helper" } -{ "clear_emulator_capabilities" = "1" } { "set_process_name" = "1" } { "max_processes" = "0" } { "max_files" = "0" } -- 2.21.0

2 3

[libvirt] [jenkins-ci PATCH v2 0/3] Enable building of the VZ driver on CentOS 7
by Daniel P. Berrangé 11 Dec '19

11 Dec '19

The VZ driver in libvirt periodically gets broken by refactoring in libvirt. This is not noticed either before or after merge because none of our CI tests nor common developer build hosts includ the deps needed for the VZ driver. The OpenVZ project, however, does provide builds of the required packages for RHEL-7. We can use these packages in our CentOS 7 CI VMs to enable build testing of the VZ driver. This closes the only hole we have in driver build coverage for CI. Changed in v2: - Enable GPG verification - Use older repo avoid temporary deps problem - Setup repos in dockerfile too. Daniel P. Berrangé (3): guests: add openvz repository on CentOS 7 guests: define mapping for the libprlsdk package guests: add libprlsdk package to libvirt project guests/lcitool | 22 ++++++++++++++++ guests/playbooks/update/tasks/base.yml | 25 +++++++++++++++++++ guests/playbooks/update/templates/openvz.key | 20 +++++++++++++++ .../playbooks/update/templates/openvz.repo.j2 | 9 +++++++ guests/vars/mappings.yml | 4 +++ guests/vars/projects/libvirt.yml | 1 + 6 files changed, 81 insertions(+) create mode 100644 guests/playbooks/update/templates/openvz.key create mode 100644 guests/playbooks/update/templates/openvz.repo.j2 -- 2.23.0

2 7

[libvirt] [dockerfiles PATCH] Rename Ubuntu Dockerfiles
by Andrea Bolognani 11 Dec '19

11 Dec '19

The corresponding libvirt-jenkins-ci commit is f289e64a5fd9. Signed-off-by: Andrea Bolognani <abologna(a)redhat.com> --- Pushed as trivial. ...ntu-16.zip => buildenv-libosinfo-ubuntu-1604.zip | Bin ...ntu-18.zip => buildenv-libosinfo-ubuntu-1804.zip | Bin ...buntu-16.zip => buildenv-libvirt-ubuntu-1604.zip | Bin ...buntu-18.zip => buildenv-libvirt-ubuntu-1804.zip | Bin 4 files changed, 0 insertions(+), 0 deletions(-) rename buildenv-libosinfo-ubuntu-16.zip => buildenv-libosinfo-ubuntu-1604.zip (100%) rename buildenv-libosinfo-ubuntu-18.zip => buildenv-libosinfo-ubuntu-1804.zip (100%) rename buildenv-libvirt-ubuntu-16.zip => buildenv-libvirt-ubuntu-1604.zip (100%) rename buildenv-libvirt-ubuntu-18.zip => buildenv-libvirt-ubuntu-1804.zip (100%) diff --git a/buildenv-libosinfo-ubuntu-16.zip b/buildenv-libosinfo-ubuntu-1604.zip similarity index 100% rename from buildenv-libosinfo-ubuntu-16.zip rename to buildenv-libosinfo-ubuntu-1604.zip diff --git a/buildenv-libosinfo-ubuntu-18.zip b/buildenv-libosinfo-ubuntu-1804.zip similarity index 100% rename from buildenv-libosinfo-ubuntu-18.zip rename to buildenv-libosinfo-ubuntu-1804.zip diff --git a/buildenv-libvirt-ubuntu-16.zip b/buildenv-libvirt-ubuntu-1604.zip similarity index 100% rename from buildenv-libvirt-ubuntu-16.zip rename to buildenv-libvirt-ubuntu-1604.zip diff --git a/buildenv-libvirt-ubuntu-18.zip b/buildenv-libvirt-ubuntu-1804.zip similarity index 100% rename from buildenv-libvirt-ubuntu-18.zip rename to buildenv-libvirt-ubuntu-1804.zip -- 2.23.0

1 0

[libvirt] [PATCH] spec: fix indentation fix
by Ján Tomko 11 Dec '19

11 Dec '19

The RPM tags must not be indented. Fixes: 6b8ab20f9b9b1a9383bd2cb9a075f57beb196c1c Signed-off-by: Ján Tomko <jtomko(a)redhat.com> --- libvirt.spec.in | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) Pushed as a build breaker fix. diff --git a/libvirt.spec.in b/libvirt.spec.in index 4d75fbecaf..bd08222527 100644 --- a/libvirt.spec.in +++ b/libvirt.spec.in @@ -336,11 +336,11 @@ BuildRequires: device-mapper-devel BuildRequires: xfsprogs-devel %if %{with_storage_rbd} %if 0%{?fedora} || 0%{?rhel} > 7 - BuildRequires: librados-devel - BuildRequires: librbd-devel +BuildRequires: librados-devel +BuildRequires: librbd-devel %else - BuildRequires: librados2-devel - BuildRequires: librbd1-devel +BuildRequires: librados2-devel +BuildRequires: librbd1-devel %endif %endif %if %{with_storage_gluster} -- 2.21.0

1 0

[libvirt] [PATCH] spec: fix indentation
by Ján Tomko 11 Dec '19

11 Dec '19

The recent specfile addition broke syntax-check: cppi: ../libvirt.spec.in: line 338: not properly indented cppi: ../libvirt.spec.in: line 341: not properly indented cppi: ../libvirt.spec.in: line 344: not properly indented Fixes: ac063cb2e76d64a907f96bf0b6a29da4eb484ebc Signed-off-by: Ján Tomko <jtomko(a)redhat.com> --- Push under the 'are we still doing syntax-check?' rule libvirt.spec.in | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/libvirt.spec.in b/libvirt.spec.in index 01f28e62d6..4d75fbecaf 100644 --- a/libvirt.spec.in +++ b/libvirt.spec.in @@ -335,13 +335,13 @@ BuildRequires: device-mapper-devel # For XFS reflink clone support BuildRequires: xfsprogs-devel %if %{with_storage_rbd} -%if 0%{?fedora} || 0%{?rhel} > 7 -BuildRequires: librados-devel -BuildRequires: librbd-devel -%else -BuildRequires: librados2-devel -BuildRequires: librbd1-devel -%endif + %if 0%{?fedora} || 0%{?rhel} > 7 + BuildRequires: librados-devel + BuildRequires: librbd-devel + %else + BuildRequires: librados2-devel + BuildRequires: librbd1-devel + %endif %endif %if %{with_storage_gluster} BuildRequires: glusterfs-api-devel >= 3.4.1 -- 2.19.2

2 1

[libvirt] [jenkins-ci PATCH 0/1] jenkins: Start building on Ubuntu
by Andrea Bolognani 11 Dec '19

11 Dec '19

This patch is intended to start a slightly larger discussion about our plans for the CentOS CI environment going forward. At the moment, we have active builders for CentOS 7 Debian 9 Debian 10 Fedora 30 Fedora 31 Fedora Rawhide FreeBSD 11 FreeBSD 12 but we don't have builder for Debian sid FreeBSD -CURRENT Ubuntu 16.04 Ubuntu 18.04 despite them being fully supported in the libvirt-jenkins-ci repository. This makes sense for sid and -CURRENT, since the former covers the same "freshest Linux packages" angle that Rawhide already takes care of and the latter is often broken and not trivial to keep updated; both Ubuntu targets, however, should IMHO be part of the CentOS CI environment. Hence this series :) Moreover, we're in the process of adding CentOS 8 openSUSE Leap 15.1 openSUSE Tumbleweed as targets, of which the first two should also IMHO be added as they would provide useful additional coverage. The only reason why I'm even questioning whether this should be done is capacity for the hypervisor host: the machine we're running all builders on has CPUs: 8 Memory: 32 GiB Storage: 450 GiB and each of the guests is configured to use CPUs: 2 Memory: 2 GiB Storage: 20 GiB So while we're good, and actually have plenty of room to grow, on the memory and storage front, we're already overcommitting our CPUs pretty significantly, which I guess is at least part of the reason why builds take so long. Can we afford to add 50% more load on the machine without making it unusable? I don't know. But I think it would be worthwhile to at least try and see how it handles an additional 25%, which is exactly what this series does. In my opinion, as long as the machine can keep up with demand and not end up in a situation where it starts accumulating backlog jobs, it's fine if builds take longer: developers who want to run a relatively quick smoke test before posting patches can use Travis CI or 'make ci-check@...' locally for the purpose, whereas the role of CentOS CI in my eyes is to try and catch as many issues as possible after merge so that they don't end up in a release. But I realize others might see it differently, hence this lenghty cover letter :) Andrea Bolognani (1): jenkins: Start building on Ubuntu jenkins/jobs/defaults.yaml | 2 ++ jenkins/projects/libvirt-dbus.yaml | 1 + jenkins/projects/libvirt-sandbox.yaml | 2 ++ jenkins/projects/libvirt-tck.yaml | 2 ++ jenkins/projects/libvirt.yaml | 2 ++ jenkins/projects/virt-manager.yaml | 2 ++ 6 files changed, 11 insertions(+) -- 2.23.0

2 4

[libvirt] [jenkins-ci PATCH v2 0/8] Add CentOS 8 to libvirt-jenkins-ci
by Fabiano Fidêncio 11 Dec '19

11 Dec '19

In order to add CentOS 8 to jenkins-ci, we had to: - Adjust lcitool to generate unattended scripts based on templates instead of using the templates themselves, as a new parameter is expected on kickstart files for RHEL / CentOS 8 (or newer); - Enable the EPEL and PowerTools repos; Then, the following projects have been added: - libvirt - Some adjustments had to be done as some dependencies got removed from CentOS 8; - libvirt-glib - libvirt-dbus - Some adjustments had to be done as flake8 doesn't pull all the needed dependencies when installed on CentOS 8; - libvirt-go - libvirt-go-xml - libvirt-ocaml - libvirt-perl - libvirt-python - virt-viewer - Some adjustments had to be done as libgovirt-devel and gtk-vnc2-devel are not present anymore on CentOS 8; - osinfo-db-tools - oisnfo-db - libosinfo - virt-manager The projects which were not added due to missing dependencies are: - libvirt-cim - libvirt-sandbox - libvirt-tck Fabiano Fidêncio (8): lcitool: Generate the unattended script file lcitool: Use install_url in the unattended install files guests: Add CentOS 8 mappings: Adjustments for CentOS 8 guests: Install EPEL repo on all CentOS guests guests: Enable PowerTools repo on CentOS 8 guests guests: Add projects to CentOS 8 guests: Build projects on CentOS 8 guests/configs/kickstart.cfg | 6 +++++ guests/host_vars/libvirt-centos-8/docker.yml | 2 ++ guests/host_vars/libvirt-centos-8/install.yml | 2 ++ guests/host_vars/libvirt-centos-8/main.yml | 22 ++++++++++++++++ guests/inventory | 1 + guests/lcitool | 26 ++++++++++++++++++- guests/playbooks/build/jobs/defaults.yml | 2 ++ guests/playbooks/build/projects/libosinfo.yml | 1 + .../playbooks/build/projects/libvirt-dbus.yml | 2 ++ guests/playbooks/build/projects/libvirt.yml | 1 + .../build/projects/osinfo-db-tools.yml | 1 + .../playbooks/build/projects/virt-manager.yml | 3 +++ guests/playbooks/update/tasks/base.yml | 9 ++++++- guests/vars/mappings.yml | 4 +++ jenkins/jobs/defaults.yaml | 2 ++ jenkins/projects/libosinfo.yaml | 1 + jenkins/projects/libvirt-dbus.yaml | 2 ++ jenkins/projects/libvirt.yaml | 1 + jenkins/projects/osinfo-db-tools.yaml | 1 + jenkins/projects/virt-manager.yaml | 3 +++ 20 files changed, 90 insertions(+), 2 deletions(-) create mode 100644 guests/host_vars/libvirt-centos-8/docker.yml create mode 100644 guests/host_vars/libvirt-centos-8/install.yml create mode 100644 guests/host_vars/libvirt-centos-8/main.yml -- 2.23.0

2 20

[libvirt] [PATCH] docs: prefer to use rst2html5 instead of rst2html
by Daniel P. Berrangé 11 Dec '19

11 Dec '19

Our website is written assuming HTML5 standard & doctype: commit b1c81567c7172bc9dcd701cf46ea3f87725d62c7 Author: Daniel P. Berrangé <berrange(a)redhat.com> Date: Wed Jul 26 18:01:25 2017 +0100 docs: switch to using HTML5 doctype declaration so we want the RST conversion to also use HTML5. Ubuntu 16.04 still only has the HTML4 generating tools though, so we have that as a fallback. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- m4/virt-external-programs.m4 | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/m4/virt-external-programs.m4 b/m4/virt-external-programs.m4 index ed634a4c73..aad21c81bf 100644 --- a/m4/virt-external-programs.m4 +++ b/m4/virt-external-programs.m4 @@ -33,10 +33,13 @@ AC_DEFUN([LIBVIRT_CHECK_EXTERNAL_PROGRAMS], [ then AC_MSG_ERROR("xsltproc is required to build libvirt") fi - AC_PATH_PROGS([RST2HTML], [rst2html rst2html.py rst2html-3], []) + + dnl Drop the rst2html (aka HTML4) variants once we + dnl stop supporting Ubuntu 16.04 (Xenial) + AC_PATH_PROGS([RST2HTML], [rst2html5 rst2html5.py rst2html5-3 rst2html rst2html.py rst2html-3], []) if test -z "$RST2HTML" then - AC_MSG_ERROR("rst2html is required to build libvirt") + AC_MSG_ERROR("rst2html5/rst2html is required to build libvirt") fi AC_PATH_PROG([AUGPARSE], [augparse], [/usr/bin/augparse]) AC_PROG_MKDIR_P -- 2.23.0

2 1

[libvirt] [PATCH] scripts: ignore remote protocol checks if pdwtags crashes
by Daniel P. Berrangé 11 Dec '19

11 Dec '19

On Debian 10, pdwtags reliably segfaults when parsing the libvirt remote protocol files. This crash was previously ignored by 'make check' because of the way we piped the pdwtags output to the perl post-processing scripts. When this was converted to use python it mistakenly started being a fatal error. We need to explicitly ignore pdwtags output if it exited with non-zero return code. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- scripts/check-remote-protocol.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/scripts/check-remote-protocol.py b/scripts/check-remote-protocol.py index 201166fd9e..25caa19563 100644 --- a/scripts/check-remote-protocol.py +++ b/scripts/check-remote-protocol.py @@ -86,8 +86,12 @@ out, err = pdwtagsproc.communicate() out = out.decode("utf-8") err = err.decode("utf-8") -if out == "" and err != "": - print("WARNING: no output, pdwtags appears broken:", file=sys.stderr) +if out == "" or pdwtagsproc.returncode != 0: + if out == "": + print("WARNING: no output, pdwtags appears broken:", file=sys.stderr) + else: + print("WARNING: exit code %d, pdwtags appears broken:" % + pdwtagsproc.returncode, file=sys.stderr) for l in err.strip().split("\n"): print("WARNING: %s" % l, file=sys.stderr) print("WARNING: skipping the remote protocol test", file=sys.stderr) -- 2.23.0

2 3

[libvirt] [PATCH] docs: switch to use rst2html5 instead of rst2html
by Daniel P. Berrangé 11 Dec '19

11 Dec '19

Now that we don't have to support CentOS 7's python2-docutils, we can use the rst2html5 tool instead, which generates HTML5 output instead of HTML4 output. This matches what we used for the original HTML docs since commit b1c81567c7172bc9dcd701cf46ea3f87725d62c7 Author: Daniel P. Berrangé <berrange(a)redhat.com> Date: Wed Jul 26 18:01:25 2017 +0100 docs: switch to using HTML5 doctype declaration Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- m4/virt-external-programs.m4 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/m4/virt-external-programs.m4 b/m4/virt-external-programs.m4 index ed634a4c73..a23c69d54b 100644 --- a/m4/virt-external-programs.m4 +++ b/m4/virt-external-programs.m4 @@ -33,7 +33,7 @@ AC_DEFUN([LIBVIRT_CHECK_EXTERNAL_PROGRAMS], [ then AC_MSG_ERROR("xsltproc is required to build libvirt") fi - AC_PATH_PROGS([RST2HTML], [rst2html rst2html.py rst2html-3], []) + AC_PATH_PROGS([RST2HTML], [rst2html5 rst2html5.py rst2html5-3], []) if test -z "$RST2HTML" then AC_MSG_ERROR("rst2html is required to build libvirt") -- 2.23.0

2 2

[libvirt] [PATCH 00/17] docs: remove all use of POD for man pages in favour of RST
by Daniel P. Berrangé 11 Dec '19

11 Dec '19

As part of the goal to eliminate Perl from libvirt, this gets rid of the use of POD format for man pages. There's nothing especially bad about POD as a markup language compared to other lightweight markup languages like RST or Markdown. It hasn't found widespread usage outside of the Perl world though, and so switching from POD to RST brings a language which likely has more familiarity to contributors. This also nicely aligns with our use of RST of web pages, and indeed in this series things are setup so that all the man pages get published on the main libvirt.org website. Over time this will hopefuly draw search engines traffic to libvirt.org instead of random 3rd party websites hosting various out of date copies of the man pages. Reviewing the individual RST files is likely a very unpleasant task, especially the enourmous virsh man page. Most of the conversion work was automated with pod2rst, followed by lots of editting to cleanup its output. virsh had some further automated processing done to create headers for each command. It is probably more useful to review the rendered man page output and/or websites to see that it looks sane when read. Daniel P. Berrangé (17): src: update keycodemapdb submodule rpm: move pod2man & rst2html deps outside the autotools conditional rpm: use python3-docutils as the direct dep docs: introduce rst2man as a mandatory tool for building docs docs: describe the basic RST structure for a man page docs: convert libvirtd man page from pod to rst docs: convert virtlockd man page from pod to rst docs: convert virtlogd man page from pod to rst docs: convert virt-host-validate man page from pod to rst docs: convert virt-pki-validate man page from pod to rst docs: convert virt-xml-validate man page from pod to rst docs: convert virt-sanlock-cleanup man page from pod to rst docs: convert virt-login-shell man page from pod to rst docs: convert virt-admin man page from pod to rst docs: convert virsh man page from pod to rst docs: convert virkeycode*/virkeyname* man pages from pod to rst docs: remove build recipes related to pod2man usage docs/Makefile.am | 118 + docs/docs.html.in | 3 + docs/manpages/index.rst | 37 + docs/manpages/libvirtd.rst | 259 + docs/manpages/virsh.rst | 7096 +++++++++++++++++ docs/manpages/virt-admin.rst | 610 ++ docs/manpages/virt-host-validate.rst | 95 + docs/manpages/virt-login-shell.rst | 144 + docs/manpages/virt-pki-validate.rst | 89 + docs/manpages/virt-sanlock-cleanup.rst | 79 + .../manpages/virt-xml-validate.rst | 107 +- docs/manpages/virtlockd.rst | 177 + docs/manpages/virtlogd.rst | 179 + docs/styleguide.rst | 86 + libvirt.spec.in | 7 +- m4/virt-external-programs.m4 | 5 + src/Makefile.am | 18 - src/keycodemapdb | 2 +- src/locking/Makefile.inc.am | 18 - src/locking/virtlockd.pod | 165 - src/logging/Makefile.inc.am | 13 - src/logging/virtlogd.pod | 165 - src/remote/Makefile.inc.am | 15 - src/remote/libvirtd.pod | 235 - src/util/Makefile.inc.am | 47 +- tools/Makefile.am | 68 - tools/virsh.pod | 5526 ------------- tools/virt-admin.pod | 497 -- tools/virt-host-validate.pod | 68 - tools/virt-login-shell.pod | 112 - tools/virt-pki-validate.pod | 61 - tools/virt-sanlock-cleanup.pod | 49 - 32 files changed, 9052 insertions(+), 7098 deletions(-) create mode 100644 docs/manpages/index.rst create mode 100644 docs/manpages/libvirtd.rst create mode 100644 docs/manpages/virsh.rst create mode 100644 docs/manpages/virt-admin.rst create mode 100644 docs/manpages/virt-host-validate.rst create mode 100644 docs/manpages/virt-login-shell.rst create mode 100644 docs/manpages/virt-pki-validate.rst create mode 100644 docs/manpages/virt-sanlock-cleanup.rst rename tools/virt-xml-validate.pod => docs/manpages/virt-xml-validate.rst (53%) create mode 100644 docs/manpages/virtlockd.rst create mode 100644 docs/manpages/virtlogd.rst delete mode 100644 src/locking/virtlockd.pod delete mode 100644 src/logging/virtlogd.pod delete mode 100644 src/remote/libvirtd.pod delete mode 100644 tools/virsh.pod delete mode 100644 tools/virt-admin.pod delete mode 100644 tools/virt-host-validate.pod delete mode 100644 tools/virt-login-shell.pod delete mode 100644 tools/virt-pki-validate.pod delete mode 100644 tools/virt-sanlock-cleanup.pod -- 2.23.0

2 19

[libvirt] [PATCH 0/2] spec: Update Fedora minimum version & librbd / librados development package names
by Fabiano Fidêncio 11 Dec '19

11 Dec '19

Fedora 29 has reached its EOL on November 26th 2019; librbd1-devel has been replaced by librbd-devel; librados2-devel has been replaced by librados; Fabiano Fidêncio (2): spec: Update Fedora minimum supported version spec: Adjust librbd / librados dependency names libvirt.spec.in | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) -- 2.23.0

2 6

[libvirt] [jenkins-ci PATCH] Update comment for virt-manager
by Andrea Bolognani 11 Dec '19

11 Dec '19

We have Python 3 on CentOS 7 now, so the existing comment is no longer accurate. Signed-off-by: Andrea Bolognani <abologna(a)redhat.com> --- Pushed as trivial. guests/playbooks/build/projects/virt-manager.yml | 4 +--- jenkins/projects/virt-manager.yaml | 4 +--- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/guests/playbooks/build/projects/virt-manager.yml b/guests/playbooks/build/projects/virt-manager.yml index c0d4294..9184a74 100644 --- a/guests/playbooks/build/projects/virt-manager.yml +++ b/guests/playbooks/build/projects/virt-manager.yml @@ -1,9 +1,7 @@ --- - set_fact: name: virt-manager - # virt-manager is Python 3 only, so it can't be built on CentOS 7; - # Ubuntu 16.04 has Python 3 but not the libxml2 bindings, so it can't - # build the project either + # CentOS 7 and Ubuntu 16.04 have Python 3 but not the libxml2 bindings machines: - libvirt-debian-9 - libvirt-debian-10 diff --git a/jenkins/projects/virt-manager.yaml b/jenkins/projects/virt-manager.yaml index 3dc8e2e..4524831 100644 --- a/jenkins/projects/virt-manager.yaml +++ b/jenkins/projects/virt-manager.yaml @@ -1,9 +1,7 @@ --- - project: name: virt-manager - # virt-manager is Python 3 only, so it can't be built on CentOS 7; - # Ubuntu 16.04 has Python 3 but not the libxml2 bindings, so it can't - # build the project either + # CentOS 7 and Ubuntu 16.04 have Python 3 but not the libxml2 bindings machines: - libvirt-debian-9 - libvirt-debian-10 -- 2.23.0

1 0

[libvirt] [PATCH v4 0/4] qemu: block: implement optional removal of committed snapshot images
by Pavel Mores 11 Dec '19

11 Dec '19

This version incorporates Peter's feedback to v3. Pavel Mores (4): qemu: block: propagate the delete flag to where it can actually be used qemu: block: use the delete flag to delete snapshot images if requested qemu: block: store the delete flag in libvirtd's status XML qemu: block: enable the snapshot image deletion feature src/qemu/qemu_blockjob.c | 50 ++++++++++++++++++- src/qemu/qemu_blockjob.h | 4 +- src/qemu/qemu_domain.c | 4 ++ src/qemu/qemu_driver.c | 12 ++++- .../blockjob-blockdev-in.xml | 1 + 5 files changed, 67 insertions(+), 4 deletions(-) -- 2.21.0

2 9

[libvirt] [PATCH] docs: fix duplication variable name for rst files
by Daniel P. Berrangé 10 Dec '19

10 Dec '19

Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- docs/Makefile.am | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/Makefile.am b/docs/Makefile.am index 5f5dce28eb..9462f77458 100644 --- a/docs/Makefile.am +++ b/docs/Makefile.am @@ -173,10 +173,10 @@ gif = \ internals_html_in = \ $(patsubst $(srcdir)/%,%,$(wildcard $(srcdir)/internals/*.html.in)) -kbase_rst = \ - $(patsubst $(srcdir)/%,%,$(wildcard $(srcdir)/kbase/*.rst)) -kbase_rst_html_in = \ - $(kbase_rst:%.rst=%.html.in) +internals_rst = \ + $(patsubst $(srcdir)/%,%,$(wildcard $(srcdir)/internals/*.rst)) +internals_rst_html_in = \ + $(internals_rst:%.rst=%.html.in) internals_html = \ $(internals_html_in:%.html.in=%.html) \ $(internals_rst_html_in:%.html.in=%.html) -- 2.23.0

2 1

[libvirt] [PATCH] Revert "qemu: directly create virResctrlInfo ignoring capabilities"
by Daniel P. Berrangé 10 Dec '19

10 Dec '19

This reverts commit 7be5fe66cd024b9ffba7c52cdbf5effedeac2c0d. This commit broke resctrl, because it missed the fact that the virResctrlInfoGetCache() has side-effects causing it to actually change the virResctrlInfo parameter, not merely get data from it. This code will need some refactoring before we can try separating it from virCapabilities again. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- src/qemu/qemu_process.c | 24 ++++++++++++++++-------- src/util/virresctrl.h | 2 -- 2 files changed, 16 insertions(+), 10 deletions(-) diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 10259027c9..23cc528817 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -2724,24 +2724,29 @@ qemuProcessSetupEmulator(virDomainObjPtr vm) static int -qemuProcessResctrlCreate(virDomainObjPtr vm) +qemuProcessResctrlCreate(virQEMUDriverPtr driver, + virDomainObjPtr vm) { + int ret = -1; size_t i = 0; + virCapsPtr caps = NULL; qemuDomainObjPrivatePtr priv = vm->privateData; - g_autoptr(virResctrlInfo) resctrl = NULL; if (!vm->def->nresctrls) return 0; - if (!(resctrl = virResctrlInfoNew())) + /* Force capability refresh since resctrl info can change + * XXX: move cache info into virresctrl so caps are not needed */ + caps = virQEMUDriverGetCapabilities(driver, true); + if (!caps) return -1; for (i = 0; i < vm->def->nresctrls; i++) { size_t j = 0; - if (virResctrlAllocCreate(resctrl, + if (virResctrlAllocCreate(caps->host.resctrl, vm->def->resctrls[i]->alloc, priv->machineName) < 0) - return -1; + goto cleanup; for (j = 0; j < vm->def->resctrls[i]->nmonitors; j++) { virDomainResctrlMonDefPtr mon = NULL; @@ -2749,11 +2754,14 @@ qemuProcessResctrlCreate(virDomainObjPtr vm) mon = vm->def->resctrls[i]->monitors[j]; if (virResctrlMonitorCreate(mon->instance, priv->machineName) < 0) - return -1; + goto cleanup; } } - return 0; + ret = 0; + cleanup: + virObjectUnref(caps); + return ret; } @@ -6881,7 +6889,7 @@ qemuProcessLaunch(virConnectPtr conn, goto cleanup; VIR_DEBUG("Setting up resctrl"); - if (qemuProcessResctrlCreate(vm) < 0) + if (qemuProcessResctrlCreate(driver, vm) < 0) goto cleanup; VIR_DEBUG("Setting up managed PR daemon"); diff --git a/src/util/virresctrl.h b/src/util/virresctrl.h index 759320d0fd..3dd7c96348 100644 --- a/src/util/virresctrl.h +++ b/src/util/virresctrl.h @@ -100,8 +100,6 @@ typedef virResctrlInfo *virResctrlInfoPtr; virResctrlInfoPtr virResctrlInfoNew(void); -G_DEFINE_AUTOPTR_CLEANUP_FUNC(virResctrlInfo, virObjectUnref); - int virResctrlInfoGetCache(virResctrlInfoPtr resctrl, unsigned int level, -- 2.23.0

2 1

[libvirt] [PATCH 00/30] conf: eliminate use of virCapsPtr for parsing/formatting XML
by Daniel P. Berrangé 10 Dec '19

10 Dec '19

The virCapsPtr object is a serious source of performance problems for the QEMU driver, because populating it means probing every QEMU system emulator binary installed on the host. This is is crazy because parsing any single XML document only ever needs information about a single QEMU binary. We've already done a great job of untangling stuff with the post-parse and validation functions, but the job was never quite finished. This series tackles the last pieces so that parsing & formatting XML will never use the virCapsPtr object directly. Some of the virt drivers will still use virCapsPtr in their private callbacks, but the QEMU driver will instead use the QEMU emulator capabilities cache directly. The first ten patches are the key bits of work to remove use of the virCapsPtr parameter from domain XML APIs. The next thirteen patches remove the (now unused) virCapsPtr parameter from the domain XML APIs & their callers. The final six patches help various QEMU driver APIs stop relying on the virCapsPtr, so that it is no longer even created at startup. This has a nice benefit on libvirtd startup tie when all 30 QEMU emulators are installed, reducing it from 4 seconds down to a fraction of a second. More work is still needed to follow on from this... A couple of places in the QEMU driver still use virCapsPtr - The virConnectGetVersion API. - The virNodeGetSecurityModel API. - virDomainGetResctrlMonData API. In all cases we just need to move to store the relevant data directly in virQEMUDriverPtr, and not use the virCaps. next, apps can use virDomainGetCapabilities to fetch info about a specific QEMU binary they care about to avoid the cost of the virConnectGetCapabilities API. There are still times when apps need info in virConnectGetCapabilities and this will annoyingly probe all QEMU binaries, even though the app doesn't want the guest info. virConnectGetCapabilities is one of our very early APIs which now flags parameter, so this problem is unfixable in that API. I'm thus included to introduce a new virNodeGetCapabilities that is specifically only returning information about the host hypervisor. Essentially the <capabilities> schema, but only the <host> sub-tree. I do, however, wonder if even this is overkill & we should aim for a much more fine grained way for reporting host capabilities. eg if an app wants the host CPU, do they really want to pay the penality to probe host cache nformation, or NUMA topology and vica-verca. I'm not sure what the right kind of tradeoff here is, whether we should try individual APIs, or have one API and a set of flags to control what info is filled in. Daniel P. Berrangé (30): qemu: cache host arch separately from virCapsPtr qemu: add machines & arch to fake test capabilities cache conf: sanitize virDomainSaveStatus & virDomainSaveConfig APIs conf: sanitize virDomainObjFormat & virDomainDefFormat* APIs conf: pass in default architecture via domain XML options conf: move virt type / os type / arch validation to post-parse conf: don't use passed in caps in post parse method conf: pass netprefix in the domain parser config struct conf: move seclabel validation into post-parse phase conf: pass default sec model in parser config conf: drop virCapsPtr param from assign addresses callback conf: drop virCapsPtr param from basic post parse callback conf: drop virCapsPtr param from post parse data alloc callback conf: drop virCapsPtr param from post parse callback conf: drop virCapsPtr param from device post parse callback conf: drop virCapsPtr param from domain validate callback conf: drop virCapsPtr param from domain formatting APIs conf: drop virCapsPtr param from domain post parse & validate APIs conf: drop virCapsPtr param from domain parse APIs conf: drop virCapsPtr param from domain APIs for copying config conf: drop virCapsPtr param from APIs for saving domains conf: drop virCapsPtr param from domain obj list APIs conf: drop virCapsPtr param from snapshot & checkpoint APIs qemu: drop virCapsPtr param & vars from many APIs conf: move NUMA capabilities into self contained object qemu: use NUMA capabilities object directly conf: make virCPUDef into a ref counted struct qemu: use host CPU object directly qemu: directly create virResctrlInfo ignoring capabilities qemu: stop creating capabilities at driver startup docs/formatdomain.html.in | 6 +- src/bhyve/bhyve_domain.c | 16 +- src/bhyve/bhyve_driver.c | 46 +- src/bhyve/bhyve_process.c | 22 +- src/conf/capabilities.c | 217 +++++--- src/conf/capabilities.h | 34 +- src/conf/checkpoint_conf.c | 21 +- src/conf/checkpoint_conf.h | 2 - src/conf/cpu_conf.c | 29 +- src/conf/cpu_conf.h | 4 + src/conf/domain_conf.c | 404 ++++++-------- src/conf/domain_conf.h | 84 ++- src/conf/snapshot_conf.c | 28 +- src/conf/snapshot_conf.h | 3 - src/conf/virconftypes.h | 3 + src/conf/virdomainobjlist.c | 9 +- src/conf/virdomainobjlist.h | 1 - src/cpu/cpu.c | 3 +- src/cpu/cpu_arm.c | 5 +- src/cpu/cpu_ppc64.c | 3 +- src/cpu/cpu_x86.c | 6 +- src/esx/esx_driver.c | 14 +- src/hyperv/hyperv_driver.c | 1 + src/libvirt_private.syms | 18 +- src/libxl/libxl_capabilities.c | 23 +- src/libxl/libxl_conf.c | 3 +- src/libxl/libxl_conf.h | 2 +- src/libxl/libxl_domain.c | 37 +- src/libxl/libxl_driver.c | 83 ++- src/libxl/libxl_migration.c | 20 +- src/libxl/xen_xl.c | 13 +- src/libxl/xen_xm.c | 2 +- src/lxc/lxc_conf.c | 9 +- src/lxc/lxc_conf.h | 2 +- src/lxc/lxc_controller.c | 44 +- src/lxc/lxc_domain.c | 13 +- src/lxc/lxc_driver.c | 98 ++-- src/lxc/lxc_native.c | 4 +- src/lxc/lxc_process.c | 20 +- src/openvz/openvz_conf.c | 62 ++- src/openvz/openvz_conf.h | 1 + src/openvz/openvz_driver.c | 64 +-- src/phyp/phyp_driver.c | 26 +- src/qemu/qemu_capabilities.c | 176 ++++--- src/qemu/qemu_capabilities.h | 12 +- src/qemu/qemu_capspriv.h | 10 + src/qemu/qemu_checkpoint.c | 32 +- src/qemu/qemu_checkpoint.h | 1 - src/qemu/qemu_command.c | 6 +- src/qemu/qemu_conf.c | 57 +- src/qemu/qemu_conf.h | 18 +- src/qemu/qemu_domain.c | 108 ++-- src/qemu/qemu_domain.h | 1 - src/qemu/qemu_driver.c | 293 ++++------- src/qemu/qemu_hotplug.c | 8 +- src/qemu/qemu_migration.c | 36 +- src/qemu/qemu_migration_cookie.c | 8 +- src/qemu/qemu_process.c | 196 +++---- src/security/virt-aa-helper.c | 2 +- src/test/test_driver.c | 45 +- src/util/virresctrl.h | 2 + src/vbox/vbox_common.c | 16 +- src/vmware/vmware_conf.c | 2 +- src/vmware/vmware_driver.c | 25 +- src/vmx/vmx.c | 23 +- src/vmx/vmx.h | 2 +- src/vz/vz_driver.c | 29 +- tests/Makefile.am | 17 +- tests/bhyveargv2xmltest.c | 2 +- tests/bhyvexml2argvtest.c | 2 +- tests/cputest.c | 11 +- tests/domaincapstest.c | 14 +- tests/domainconftest.c | 2 +- tests/libxlxml2domconfigtest.c | 22 +- tests/lxcconf2xmltest.c | 17 +- tests/lxcxml2xmltest.c | 13 +- tests/openvzutilstest.c | 8 +- tests/qemuagenttest.c | 2 +- tests/qemublocktest.c | 2 +- tests/qemucapabilitiestest.c | 9 +- tests/qemudomaincheckpointxml2xmltest.c | 4 +- tests/qemudomainsnapshotxml2xmltest.c | 4 +- tests/qemuhotplugtest.c | 11 +- tests/qemumemlocktest.c | 6 +- tests/qemusecuritytest.c | 1 - tests/qemuxml2argvtest.c | 35 +- tests/qemuxml2xmltest.c | 4 +- tests/securityselinuxlabeltest.c | 2 +- tests/testutils.c | 39 +- tests/testutils.h | 5 +- tests/testutilslxc.c | 34 +- tests/testutilslxc.h | 4 + tests/testutilsqemu.c | 670 +++++++----------------- tests/testutilsqemu.h | 9 +- tests/testutilsxen.c | 34 +- tests/testutilsxen.h | 9 +- tests/vircaps2xmltest.c | 6 +- tests/vircapstest.c | 11 +- tests/vmx2xmltest.c | 4 +- tests/xlconfigtest.c | 27 +- tests/xmconfigtest.c | 23 +- tests/xml2vmxtest.c | 4 +- 102 files changed, 1687 insertions(+), 1993 deletions(-) -- 2.23.0

3 36

[libvirt] [PATCH] qemu: honour parseOpaque instead of refetching caps
by Daniel P. Berrangé 10 Dec '19

10 Dec '19

The use of the parseOpaque parameter was mistakenly removed in commit 4a4132b4625778cf80acb9c92d06351b44468ac3 Author: Daniel P. Berrangé <berrange(a)redhat.com> Date: Tue Dec 3 10:49:49 2019 +0000 conf: don't use passed in caps in post parse method causing the method to re-fetch qemuCaps that were already just fetched and put into parseOpaque. This is inefficient when parsing incoming XML, but for live XML this is more serious as it means we use the capabilities for the current QEMU binary on disk, rather than the running QEMU. That commit, however, did have a useful side effect of fixing a crasher bug in the qemu post parse callback introduced by commit 5e939cea896fb3373a6f68f86e325c657429ed3d Author: Jiri Denemark <jdenemar(a)redhat.com> Date: Thu Sep 26 18:42:02 2019 +0200 qemu: Store default CPU in domain XML The qemuDomainDefSetDefaultCPU() method in that patch did not allow for the possibility that qemuCaps would be NULL and thus resulted in a SEGV. This shows a risk in letting each check in the post parse callback look for qemuCaps == NULL. The safer option is to check once upfront and immediately stop (postpone) further validation. Honouring the cached caps for the live status XML, highlights a second flaw, which is that it shouldn't check the virt type or arch for running guests. The info needed to check this is not in the cache caps, only the list of flags are populated. Thus some of the post parse logic is made to only run for inactive guests. This showed a bug in one test data file which lacked an ID attribute for the live guest. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- src/qemu/qemu_domain.c | 52 +++++++++++-------- .../qemustatusxml2xmldata/vcpus-multi-in.xml | 2 +- 2 files changed, 31 insertions(+), 23 deletions(-) diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 6f53e17b6a..2abe93c829 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -4703,16 +4703,17 @@ static int qemuDomainDefPostParse(virDomainDefPtr def, unsigned int parseFlags, void *opaque, - void *parseOpaque G_GNUC_UNUSED) + void *parseOpaque) { virQEMUDriverPtr driver = opaque; g_autoptr(virQEMUDriverConfig) cfg = virQEMUDriverGetConfig(driver); - g_autoptr(virQEMUCaps) qemuCaps = NULL; + /* Note that qemuCaps may be NULL when this function is called. This + * function shall not fail in that case. It will be re-run on VM startup + * with the capabilities populated. */ + virQEMUCapsPtr qemuCaps = parseOpaque; - if (!(qemuCaps = virQEMUCapsCacheLookup(driver->qemuCapsCache, - def->emulator))) { + if (!qemuCaps) return 1; - } if (def->os.type != VIR_DOMAIN_OSTYPE_HVM) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, @@ -4721,18 +4722,31 @@ qemuDomainDefPostParse(virDomainDefPtr def, return -1; } - if (!virQEMUCapsIsArchSupported(qemuCaps, def->os.arch)) { - virReportError(VIR_ERR_CONFIG_UNSUPPORTED, - _("Emulator '%s' does not support arch '%s'"), - def->emulator, virArchToString(def->os.arch)); - return -1; - } + if (def->id == -1) { + if (!virQEMUCapsIsArchSupported(qemuCaps, def->os.arch)) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, + _("Emulator '%s' does not support arch '%s'"), + def->emulator, virArchToString(def->os.arch)); + return -1; + } - if (!virQEMUCapsIsVirtTypeSupported(qemuCaps, def->virtType)) { - virReportError(VIR_ERR_CONFIG_UNSUPPORTED, - _("Emulator '%s' does not support virt type '%s'"), - def->emulator, virDomainVirtTypeToString(def->virtType)); - return -1; + if (!virQEMUCapsIsVirtTypeSupported(qemuCaps, def->virtType)) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, + _("Emulator '%s' does not support virt type '%s'"), + def->emulator, virDomainVirtTypeToString(def->virtType)); + return -1; + } + if (!def->os.machine) { + const char *machine = virQEMUCapsGetPreferredMachine(qemuCaps, + def->virtType); + def->os.machine = g_strdup(machine); + } + } else { + if (!def->os.machine) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("missing machine type")); + return -1; + } } if (def->os.bootloader || def->os.bootloaderArgs) { @@ -4741,12 +4755,6 @@ qemuDomainDefPostParse(virDomainDefPtr def, return -1; } - if (!def->os.machine) { - const char *machine = virQEMUCapsGetPreferredMachine(qemuCaps, - def->virtType); - def->os.machine = g_strdup(machine); - } - qemuDomainNVRAMPathGenerate(cfg, def); if (qemuDomainDefAddDefaultDevices(def, qemuCaps) < 0) diff --git a/tests/qemustatusxml2xmldata/vcpus-multi-in.xml b/tests/qemustatusxml2xmldata/vcpus-multi-in.xml index 11ec74ecf8..be48c55fe0 100644 --- a/tests/qemustatusxml2xmldata/vcpus-multi-in.xml +++ b/tests/qemustatusxml2xmldata/vcpus-multi-in.xml @@ -310,7 +310,7 @@ <allowReboot value='yes'/> <blockjobs active='no'/> <agentTimeout>-2</agentTimeout> - <domain type='kvm'> + <domain type='kvm' id='1729'> <name>QEMUGuest1</name> <uuid>c7a5fdbd-edaf-9455-926a-d65c16db1809</uuid> <memory unit='KiB'>219100</memory> -- 2.23.0

2 1