[libvirt PATCH 0/4] [RFC] Functional CI - GitLab enablement
by Erik Skultety
I'd like to propose a PoC of a GitLab-driven functional CI based on GitLab's
custom executor runner feature.
** TL;DR **
- there's some RH owned HW that can be utilized for upstream functional CI
running workloads on Red Hat-sponsored distros (see the pipeline)
- the PoC utilizes GitLab's custom executor feature [1] which allows literally
any infrastructure to be plugged into the CI machinery
- the PoC solution (as proposed) is relying on nested virt capability and
backed by plain libvirt + QEMU/KVM
- the test environment comprises of the libvirt-TCK suite [2] and Avocado test
framework [3] as the harness engine (as a wrapper and future replacement of
the existing Perl implementation)
- example pipeline (simplified to only a couple of distros) can be found here [4]
- libvirt debug logs are exposed as artifacts and can be downloaded on job
failure
** LONG VERSION **
This RFC is driven by GitLab's custom executor feature which can be integrated
with any VMM and hence libvirt+QEMU/KVM as well. Example pipeline can be found
here [4].
Architecture
============
Platform
~~~~~~~~
It doesn't matter what platform is used with this PoC, but FWIW the one tested
with this PoC is a baremetal KVM virtualization host provided and owned by
Red Hat, utilizing libvirt, QEMU, a bit of lcitool [5], and libguestfs.
Custom executor
~~~~~~~~~~~~~~
If you're wondering why this PoC revolves around the custom executor GitLab
runner type [6], some time ago GitLab decided that instead of adding and
supporting different types of virtualization runners, they'd create some kind
of gateway suitable basically for any solution out there and named it 'custom
executor' - it's nothing more than a bunch of shell scripts categorized to
various stages of a pipeline job (which will run either on the host or the
guest depending how you set it up).
Provisioner
~~~~~~~~~~~
Wait, why is this section needed, just take the example scripts from [1]...
If you look at the example Bash scripts in GitLab's docs on custom executor [1]
you'll see that the example scripts are very straight forward and concise but
there are few things that are suboptimal
- creating a new disk image for every single VM instance of the same distro out
of the virt-builder template
- injecting a bunch of commands directly with virt-builder instead of using
something more robust like Ansible when there is more than a single VM to be
created
- static hostname for every single VM instance
- waiting for the either the VM getting an IP or for the SSH connection to be
available for a quite limited fixed period of time
Because I wanted to address all of the ^above pitfalls, I created a Python
based provisioner relying on libvirt-NSS, transient machines,
backing chain overlay images, cloud-init, etc. instead. You can find the code
base for the provisioner here [7].
Child pipelines & Multiproject CI pipelines
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
child pipeline = a "downstream" (or subpipeline) of the main pipeline useful
when there are multiple stages which should be part of a
single stage of the main pipeline
multiproject pipeline = a pipeline that is kicked off by the main pipeline of
your project and runs in context of a different project
which your main pipeline can then depend on
Technically speaking none of the above 2 GitLab features is needed to really
integrate this PoC to upstream libvirt CI, but it was a nice experiment from 3
perspectives:
1) seeing that GitLab's own public SaaS deployment has features from all
tiers available
2) it made some job descriptions shorter because e.g. we didn't have to
rebuild libvirt in the VM when it was done in the previous step in a
container; we don't need to build the Perl bindings (which we need for
libvirt-TCK) in the VM either, since we already have a project and hence
a pipeline for that already, we just need to harness the results which
combined makes the actual VM's purpose more obvious
3) installing libvirt and libvirt-perl using the RPM artifacts built in
previous pipeline stages is closer to a real life deployment/environment
than re-building everything in the VM followed by doing
'sudo make install'
Artifacts
~~~~~~~~~
Apart from the RPMs being published as artifacts from an earlier stage of the
pipeline as mentioned in the previous section, in order for this to be any
useful to the developers (I know, NOTHING beats a local setup, just be patient
with us/me for another while...), when there's a genuine test failure (no
infrastructure is failproof), the job description as proposed in patch 4/5
exposes libvirt debug logs as artifacts which can be downloaded and inspected.
Test framework
==============
For the SW testing part a combination of libvirt-TCK [2] and Avocado [3] were
used. The reasoning for this combined test environment was discussed in this
thread [8], but essentially we'd like to port most of the Perl-native TCK tests
to Python, as more people are familiar with the language, making maintenance of
them easier as well as making it easier to contribute new tests to the somewhat
ancient test suite. Avocado can definitely help here as it does provide a good
foundation and building stones to achieve both.
Work in progress
================
One of the things that we already know is problematic with the current hybrid
TCK+Avocado setup is that if a test fails, the summary that Avocado produces
doesn't quickly tell us which tests have failed and one has to either scroll
through the GitLab job output or wade through Avocado job logs.
-> this issue is tracked in upstream Avocado and should be addressed soon:
https://github.com/avocado-framework/avocado/issues/5200
Another annoying thing in terms of test failure analysis is that TCK in its
current shape cannot save libvirt debug logs on a per-test basis and so a
massive dump is produced for the whole test run. This is very suboptimal, but
this can only be solved with libvirt admin API which can enable/modify/redirect
logs at runtime, however, the perl bindings are currently missings
-> I'm already working on adding the bindings to libvirt-perl and then we
need to teach TCK to use that feature
Known issues
============
There are already a couple of known test failures which can be seen in the
example pipeline:
1) modular daemons occasionally hang in some test scenarios
(doesn't happen with monolithic)
https://bugzilla.redhat.com/show_bug.cgi?id=2044379
2) QEMU tray status detection when a CD media is changed
-> this one is also intermittent, but no good reproducer data to attach
to an official QEMU BZ has been harnessed so far
[1] https://docs.gitlab.com/runner/executors/custom.html
[2] https://gitlab.com/libvirt/libvirt-tck/
[3] https://avocado-framework.readthedocs.io/
[4] https://gitlab.com/eskultety/libvirt/-/pipelines/457836923
[5] https://gitlab.com/libvirt/libvirt-ci/
[6] https://docs.gitlab.com/runner/executors/
[7] https://gitlab.com/eskultety/libvirt-gitlab-executor
[8] https://listman.redhat.com/archives/libvir-list/2021-June/msg00836.html
Erik Skultety (4):
ci: manifest: Allow RPM builds on CentOS Stream 8
ci: containers: Add CentOS Stream 9 target
ci: manifest: Publish RPMs as artifacts on CentOS Stream and Fedoras
gitlab-ci: Introduce a new test 'integration' pipeline stage
.gitlab-ci-integration.yml | 116 +++++++++++++++++++++++
.gitlab-ci.yml | 17 +++-
ci/containers/centos-stream-9.Dockerfile | 87 +++++++++++++++++
ci/gitlab.yml | 33 ++++++-
ci/manifest.yml | 26 ++++-
5 files changed, 274 insertions(+), 5 deletions(-)
create mode 100644 .gitlab-ci-integration.yml
create mode 100644 ci/containers/centos-stream-9.Dockerfile
--
2.34.1
3 years, 1 month
[PATCH] libxl: Turn on user aliases
by Michal Privoznik
When I implemented user aliases I've invented this
virDomainDefFeatures flag so that individual drivers can signal
support for user provided aliases. The reasoning was that a
device alias might be part of guest ABI, or used in a different
way then in QEMU. Well, neither applies to the libxl driver, so
it's safe to allow user aliases there.
Resolves: https://gitlab.com/libvirt/libvirt/-/issues/231
Signed-off-by: Michal Privoznik <mprivozn(a)redhat.com>
---
src/libxl/libxl_domain.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/libxl/libxl_domain.c b/src/libxl/libxl_domain.c
index c91e531a9a..0816c5baa4 100644
--- a/src/libxl/libxl_domain.c
+++ b/src/libxl/libxl_domain.c
@@ -452,7 +452,8 @@ virDomainDefParserConfig libxlDomainDefParserConfig = {
.domainPostParseCallback = libxlDomainDefPostParse,
.domainValidateCallback = libxlDomainDefValidate,
- .features = VIR_DOMAIN_DEF_FEATURE_FW_AUTOSELECT |
+ .features = VIR_DOMAIN_DEF_FEATURE_USER_ALIAS |
+ VIR_DOMAIN_DEF_FEATURE_FW_AUTOSELECT |
VIR_DOMAIN_DEF_FEATURE_NET_MODEL_STRING,
};
--
2.34.1
3 years, 1 month
[PATCH v5 0/8] Implement detach device related APIs for test driver
by Luke Yue
diff to v4:
- Rebase to current master
diff to v3:
- Add virDomainDeviceDefOperationsCallbacks to xmlopt for de-duplicating purpose
- Add virDomainDeviceTypeFlags for de-duplicating purpose
- Remove the memballoon helper function
- Squash test_driver commits
- Move test device xmls to generichotplugdata
- Reimplement tests with internal APIs
link to v4: https://listman.redhat.com/archives/libvir-list/2021-December/msg00108.html
link to v3: https://listman.redhat.com/archives/libvir-list/2021-November/msg00288.html
link to CI: https://gitlab.com/lukedyue/libvirt/-/pipelines/464756840
Luke Yue (8):
conf: Introduce virDomainInputDefRemove and fix memory leak
conf: Introduce virDomainDeviceDefOperationsCallbacks to xmlopt
conf: Add virDomainDeviceTypeFlags and use it in various drivers
conf: Add tpm helpers for future use
domain_driver: extract DetachXXXDeviceConfig related functions and use
them
test_driver: Implement virDomainDetachDeviceFlags
examples: xml: test: add xml for testing devices related APIs
tests: Add generichotplugtest
examples/xml/test/meson.build | 1 +
examples/xml/test/testdomfc5.xml | 54 ++
examples/xml/test/testnodeinline.xml | 54 ++
src/bhyve/bhyve_domain.c | 2 +-
src/ch/ch_conf.c | 2 +-
src/conf/domain_conf.c | 572 +++++++++++++++++-
src/conf/domain_conf.h | 135 ++++-
src/conf/virconftypes.h | 2 +
src/hyperv/hyperv_driver.c | 3 +-
src/libvirt_private.syms | 21 +
src/libxl/libxl_conf.c | 3 +-
src/libxl/libxl_domain.c | 8 +
src/libxl/libxl_domain.h | 1 +
src/libxl/libxl_driver.c | 73 +--
src/lxc/lxc_conf.c | 3 +-
src/lxc/lxc_domain.c | 7 +
src/lxc/lxc_domain.h | 1 +
src/lxc/lxc_driver.c | 62 +-
src/openvz/openvz_conf.c | 2 +-
src/qemu/qemu_conf.c | 3 +-
src/qemu/qemu_domain.c | 39 ++
src/qemu/qemu_domain.h | 6 +
src/qemu/qemu_driver.c | 206 +------
src/qemu/qemu_process.c | 2 +-
src/security/virt-aa-helper.c | 2 +-
src/test/test_driver.c | 197 +++++-
src/vbox/vbox_common.c | 2 +-
src/vmware/vmware_driver.c | 2 +-
src/vmx/vmx.c | 2 +-
src/vz/vz_driver.c | 2 +-
tests/bhyveargv2xmltest.c | 2 +-
.../generichotplug-controller.xml | 1 +
.../generichotplug-disk-cdrom.xml | 5 +
.../generichotplug-filesystem.xml | 6 +
.../generichotplug-hostdev.xml | 5 +
.../generichotplug-input.xml | 1 +
.../generichotplug-interface.xml | 6 +
.../generichotplug-lease.xml | 5 +
.../generichotplug-memballoon.xml | 3 +
.../generichotplug-memory.xml | 6 +
.../generichotplugdata/generichotplug-rng.xml | 4 +
.../generichotplug-shmem.xml | 4 +
.../generichotplug-sound.xml | 3 +
.../generichotplugdata/generichotplug-tpm.xml | 5 +
.../generichotplug-vsock.xml | 3 +
.../generichotplug-watchdog.xml | 1 +
tests/generichotplugtest.c | 178 ++++++
tests/meson.build | 1 +
tests/testutils.c | 2 +-
49 files changed, 1378 insertions(+), 332 deletions(-)
create mode 100644 tests/generichotplugdata/generichotplug-controller.xml
create mode 100644 tests/generichotplugdata/generichotplug-disk-cdrom.xml
create mode 100644 tests/generichotplugdata/generichotplug-filesystem.xml
create mode 100644 tests/generichotplugdata/generichotplug-hostdev.xml
create mode 100644 tests/generichotplugdata/generichotplug-input.xml
create mode 100644 tests/generichotplugdata/generichotplug-interface.xml
create mode 100644 tests/generichotplugdata/generichotplug-lease.xml
create mode 100644 tests/generichotplugdata/generichotplug-memballoon.xml
create mode 100644 tests/generichotplugdata/generichotplug-memory.xml
create mode 100644 tests/generichotplugdata/generichotplug-rng.xml
create mode 100644 tests/generichotplugdata/generichotplug-shmem.xml
create mode 100644 tests/generichotplugdata/generichotplug-sound.xml
create mode 100644 tests/generichotplugdata/generichotplug-tpm.xml
create mode 100644 tests/generichotplugdata/generichotplug-vsock.xml
create mode 100644 tests/generichotplugdata/generichotplug-watchdog.xml
create mode 100644 tests/generichotplugtest.c
--
2.35.1
3 years, 1 month
[PATCH] libxl: remove redundant variable from libxlDomainJobObj
by Kristina Hanicova
It makes no sense to have 'started' variable in the
libxlDomainJobObj as the same one is already in virDomainJobData,
but never used.
Signed-off-by: Kristina Hanicova <khanicov(a)redhat.com>
---
src/libxl/libxl_domain.c | 10 +++++-----
src/libxl/libxl_domain.h | 1 -
2 files changed, 5 insertions(+), 6 deletions(-)
diff --git a/src/libxl/libxl_domain.c b/src/libxl/libxl_domain.c
index ee031267ca..205049f98a 100644
--- a/src/libxl/libxl_domain.c
+++ b/src/libxl/libxl_domain.c
@@ -118,7 +118,7 @@ libxlDomainObjBeginJob(libxlDriverPrivate *driver G_GNUC_UNUSED,
VIR_DEBUG("Starting job: %s", libxlDomainJobTypeToString(job));
priv->job.active = job;
priv->job.owner = virThreadSelfID();
- priv->job.started = now;
+ priv->job.current->started = now;
priv->job.current->jobType = VIR_DOMAIN_JOB_UNBOUNDED;
return 0;
@@ -171,18 +171,18 @@ libxlDomainJobUpdateTime(struct libxlDomainJobObj *job)
virDomainJobData *jobData = job->current;
unsigned long long now;
- if (!job->started)
+ if (!jobData->started)
return 0;
if (virTimeMillisNow(&now) < 0)
return -1;
- if (now < job->started) {
- job->started = 0;
+ if (now < jobData->started) {
+ jobData->started = 0;
return 0;
}
- jobData->timeElapsed = now - job->started;
+ jobData->timeElapsed = now - jobData->started;
return 0;
}
diff --git a/src/libxl/libxl_domain.h b/src/libxl/libxl_domain.h
index 475e4a6933..157f480b93 100644
--- a/src/libxl/libxl_domain.h
+++ b/src/libxl/libxl_domain.h
@@ -46,7 +46,6 @@ struct libxlDomainJobObj {
virCond cond; /* Use to coordinate jobs */
enum libxlDomainJob active; /* Currently running job */
int owner; /* Thread which set current job */
- unsigned long long started; /* When the job started */
virDomainJobData *current; /* Statistics for the current job */
};
--
2.35.1
3 years, 1 month
[libvirt PATCH 0/5] nwfilter: fix deadlock with binding create/delete
by Daniel P. Berrangé
Historically the nwfilter driver didn't keep track of any
instances of filters. When it needed to rebuild filters it
called back to the virt drivers to ask them to re-create
the filters for their VMs. This lead to some complex locking
requirements where the virt driver needed to acquire locks
on the nwfilter driver before creating/deleting filter
instances.
Somewhat recently we introduced the virNWFilterBinding
concept which allowed the nwfilter driver to keep track of
all filter instances aka bindings. Thus it could rebuild
them without talking to the virt drivers. We never updated
the locking model when doing this though, so we were still
reliant on the virt drivers acquiring locks before creating
or deleting virNWFilterBinding objects.
When we started using the modular daemons though, the locks
acquired by the virt drivers no longer protected the nwfilter
driver as they were separate processes. Thus the race condition
fixed back in
commit 6e5c79a1b5a8b3a23e7df7ffe58fb272aa17fbfb
Author: Daniel P. Berrangé <berrange(a)redhat.com>
Date: Wed Jan 22 17:28:29 2014 +0000
Push nwfilter update locking up to top level
has now re-appeared when running modular daemons. In fact
it is possible to trigger it even in libvirtd if you ignore
the virt drivers and directly call the virNWFilterBinding
APIs, though that won't hit users in practice.
The solution is surprisingly simple. Since the nwfilter
driver keeps track of virNWFilterBinding objects, we
can simply put locking calls around the Create/Delete
methods for those objects, and remove the locking from
the virt drivers.
The only slight difference is that previous the locking
serialized the entire VM startup/shutdown sequence, while
the new locking only serializes individual NICs. This
should not matter in practice, since there's no shared
state between filters for NICs on the same VM.
This is a minimal fix suitable for the freeze. There is
scope for some significant improvements in locking which
I'm still working on. In particular we currently have a
terrible writer starvation problem with the RWLock that
protects the filter create/delete/redefine operations
that has existed forever. Repeatedly creating/deleting
virNWFilterBinding objects can prevent a virNWFilterDefineXML
call from ever running.
Daniel P. Berrangé (5):
nwfilter: stop using recursive mutex for IP learning
nwfilter: hold filter update lock when creating/deleting bindings
qemu,lxc: remove use to nwfilter update lock
nwfilter: remove decl of virNWFilterCreateVarHashmap
nwfilter: make some gentech driver methods static
src/lxc/lxc_driver.c | 6 ------
src/nwfilter/nwfilter_driver.c | 5 +++++
src/nwfilter/nwfilter_gentech_driver.c | 4 ++--
src/nwfilter/nwfilter_gentech_driver.h | 8 --------
src/nwfilter/nwfilter_learnipaddr.c | 2 +-
src/qemu/qemu_driver.c | 18 ------------------
src/qemu/qemu_migration.c | 3 ---
src/qemu/qemu_process.c | 4 ----
8 files changed, 8 insertions(+), 42 deletions(-)
--
2.35.1
3 years, 1 month
Libvirt Rust bindings could use some work
by Wim de With
Hello,
I was reading the Rust bindings for libvirt and I noticed that they are
not very idiomatic. A couple of examples:
1. It is conventional to have a *-sys crate containing only the C
interface and the linking configuration. The virt crate would then
depend on this libvirt-sys crate to implement a safe and ergonomic
wrapper.
2. The API heavily uses integers to indicate flags. Sometimes those
integers are type aliased and sometimes they are not. In Rust, this can
be much nicer, using enums to represent only valid flags. The same
applies to some return values.
3. Since libvirt is thread-safe, the Rust types could be Send and Sync,
so that they can also be used in a multithreaded context. Special care
needs to be taken with translating virGetLastError to Rust Results, but
it shouldn't be a problem.
For a nice example of a Rust wrapper around a C library, see:
https://github.com/rust-lang/git2-rs
I am willing to send some patches to fix some of these problems, but
this will almost certainly lead to breaking API changes, and I am not
sure what your opinion is on that.
Finally, while I am not a legal expert, I believe that licensing a Rust
library as LGPL does not make much sense. Rust does not have a stable
ABI so everything is statically linked. This means that adding virt to a
Cargo.toml file of a non-GPL application is already in violation of the
license.
Thanks,
Wim
3 years, 1 month
[PATCH v3 00/11] blockdev-replace
by Vladimir Sementsov-Ogievskiy
Hi all!
Finally, that's a proposal for new interface for filter insertion, which
provides generic way for inserting between different block graph nodes,
like BDS nodes, block exports and block devices.
v3: - add transaction support
- add test, that shows transactional filter insertion in different
cases
- drop RFC mark. I think it's now close to be a good solution. And
anyway, no comments on "RFC v2" version :) Still, I want to keep
x- prefix for now, just because there were too many different
ideas on this topic.
Vladimir Sementsov-Ogievskiy (11):
block-backend: blk_root(): drop const specifier on return type
block/export: add blk_by_export_id()
block: make bdrv_find_child() function public
block: bdrv_replace_child_bs(): move to external transaction
qapi: add x-blockdev-replace command
qapi: add x-blockdev-replace transaction action
block: bdrv_get_xdbg_block_graph(): report export ids
iotests.py: qemu_img_create: use imgfmt by default
iotests.py: introduce VM.assert_edges_list() method
iotests.py: add VM.qmp_check() helper
iotests: add filter-insertion
qapi/block-core.json | 62 +++++
qapi/transaction.json | 14 +-
include/block/block.h | 2 +-
include/block/block_int.h | 1 +
include/block/export.h | 1 +
include/sysemu/block-backend.h | 3 +-
block.c | 59 ++--
block/block-backend.c | 10 +-
block/export/export.c | 31 +++
blockdev.c | 113 +++++++-
stubs/blk-by-qdev-id.c | 9 +
stubs/blk-exp-find-by-blk.c | 9 +
stubs/meson.build | 2 +
tests/qemu-iotests/iotests.py | 23 ++
tests/qemu-iotests/tests/filter-insertion | 253 ++++++++++++++++++
tests/qemu-iotests/tests/filter-insertion.out | 5 +
16 files changed, 563 insertions(+), 34 deletions(-)
create mode 100644 stubs/blk-by-qdev-id.c
create mode 100644 stubs/blk-exp-find-by-blk.c
create mode 100755 tests/qemu-iotests/tests/filter-insertion
create mode 100644 tests/qemu-iotests/tests/filter-insertion.out
--
2.31.1
3 years, 1 month
GSoC 2022 CI project idea proposal
by Erik Skultety
So, we're offloading as much CI configuration/workflow stuff to lcitool
as possible. We can generate config files, install/update machines
(local or remote), dump Dockerfiles...we can't build and run those containers
locally. Instead, we have a CI helper script in libvirt repo which essentially
just wraps a Makefile which pulls a gitlab container for you and:
- gives you shell, or
- runs the build and tests
I'm not sure how many people actually know we have that helper script let alone
use it. I've been playing with the idea that we could integrate what's done in
the Makefile to lcitool utilizing either the podman library [1] or the docker
library [2]. Apart from consolidating all CI services-related efforts to
lcitool the other benefit would be that we could gain the ability to run and
debug in a project-specific container also in other libvirt projects not just
main libvirt.
So, I though this could be a nice project for GSoC. Ideas?
Unless there are arguments against this idea then I'd eventually add the idea
to our gitlab GSoC issue tracker.
[1] https://github.com/containers/podman-py
[2] https://docker-py.readthedocs.io/en/stable/
Erik
3 years, 1 month
Re: REST service for libvirt to simplify SEV(ES) launch measurement
by Dewey, Larry
[AMD Official Use Only]
I definitely agree that adding functionality to simplify, and hopefully unify, the attestation workflow for confidential containers would be extremely helpful, and would be very interested in being an integral part of the process. I have my own opinions about using a REST service instead of others due to potential maintenance overhead and lack of contract rule enforcement, but there are definitely pros and cons to all of the current protocols. I am excited to see where this goes.
3 years, 1 month
Entering freeze for libvirt-8.1.0
by Jiri Denemark
I have just tagged v8.1.0-rc1 in the repository and pushed signed
tarballs and source RPMs to https://libvirt.org/sources/
Please give the release candidate some testing and in case you find a
serious issue which should have a fix in the upcoming release, feel
free to reply to this thread to make sure the issue is more visible.
If you have not done so yet, please update NEWS.rst to document any
significant change you made since the last release.
Thanks,
Jirka
3 years, 1 month
