[libvirt PATCH 0/4] [RFC] Functional CI - GitLab enablement
I'd like to propose a PoC of a GitLab-driven functional CI based on GitLab's custom executor runner feature. ** TL;DR ** - there's some RH owned HW that can be utilized for upstream functional CI running workloads on Red Hat-sponsored distros (see the pipeline) - the PoC utilizes GitLab's custom executor feature [1] which allows literally any infrastructure to be plugged into the CI machinery - the PoC solution (as proposed) is relying on nested virt capability and backed by plain libvirt + QEMU/KVM - the test environment comprises of the libvirt-TCK suite [2] and Avocado test framework [3] as the harness engine (as a wrapper and future replacement of the existing Perl implementation) - example pipeline (simplified to only a couple of distros) can be found here [4] - libvirt debug logs are exposed as artifacts and can be downloaded on job failure ** LONG VERSION ** This RFC is driven by GitLab's custom executor feature which can be integrated with any VMM and hence libvirt+QEMU/KVM as well. Example pipeline can be found here [4]. Architecture ============ Platform ~~~~~~~~ It doesn't matter what platform is used with this PoC, but FWIW the one tested with this PoC is a baremetal KVM virtualization host provided and owned by Red Hat, utilizing libvirt, QEMU, a bit of lcitool [5], and libguestfs. Custom executor ~~~~~~~~~~~~~~ If you're wondering why this PoC revolves around the custom executor GitLab runner type [6], some time ago GitLab decided that instead of adding and supporting different types of virtualization runners, they'd create some kind of gateway suitable basically for any solution out there and named it 'custom executor' - it's nothing more than a bunch of shell scripts categorized to various stages of a pipeline job (which will run either on the host or the guest depending how you set it up). Provisioner ~~~~~~~~~~~ Wait, why is this section needed, just take the example scripts from [1]... If you look at the example Bash scripts in GitLab's docs on custom executor [1] you'll see that the example scripts are very straight forward and concise but there are few things that are suboptimal - creating a new disk image for every single VM instance of the same distro out of the virt-builder template - injecting a bunch of commands directly with virt-builder instead of using something more robust like Ansible when there is more than a single VM to be created - static hostname for every single VM instance - waiting for the either the VM getting an IP or for the SSH connection to be available for a quite limited fixed period of time Because I wanted to address all of the ^above pitfalls, I created a Python based provisioner relying on libvirt-NSS, transient machines, backing chain overlay images, cloud-init, etc. instead. You can find the code base for the provisioner here [7]. Child pipelines & Multiproject CI pipelines ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ child pipeline = a "downstream" (or subpipeline) of the main pipeline useful when there are multiple stages which should be part of a single stage of the main pipeline multiproject pipeline = a pipeline that is kicked off by the main pipeline of your project and runs in context of a different project which your main pipeline can then depend on Technically speaking none of the above 2 GitLab features is needed to really integrate this PoC to upstream libvirt CI, but it was a nice experiment from 3 perspectives: 1) seeing that GitLab's own public SaaS deployment has features from all tiers available 2) it made some job descriptions shorter because e.g. we didn't have to rebuild libvirt in the VM when it was done in the previous step in a container; we don't need to build the Perl bindings (which we need for libvirt-TCK) in the VM either, since we already have a project and hence a pipeline for that already, we just need to harness the results which combined makes the actual VM's purpose more obvious 3) installing libvirt and libvirt-perl using the RPM artifacts built in previous pipeline stages is closer to a real life deployment/environment than re-building everything in the VM followed by doing 'sudo make install' Artifacts ~~~~~~~~~ Apart from the RPMs being published as artifacts from an earlier stage of the pipeline as mentioned in the previous section, in order for this to be any useful to the developers (I know, NOTHING beats a local setup, just be patient with us/me for another while...), when there's a genuine test failure (no infrastructure is failproof), the job description as proposed in patch 4/5 exposes libvirt debug logs as artifacts which can be downloaded and inspected. Test framework ============== For the SW testing part a combination of libvirt-TCK [2] and Avocado [3] were used. The reasoning for this combined test environment was discussed in this thread [8], but essentially we'd like to port most of the Perl-native TCK tests to Python, as more people are familiar with the language, making maintenance of them easier as well as making it easier to contribute new tests to the somewhat ancient test suite. Avocado can definitely help here as it does provide a good foundation and building stones to achieve both. Work in progress ================ One of the things that we already know is problematic with the current hybrid TCK+Avocado setup is that if a test fails, the summary that Avocado produces doesn't quickly tell us which tests have failed and one has to either scroll through the GitLab job output or wade through Avocado job logs. -> this issue is tracked in upstream Avocado and should be addressed soon: https://github.com/avocado-framework/avocado/issues/5200 Another annoying thing in terms of test failure analysis is that TCK in its current shape cannot save libvirt debug logs on a per-test basis and so a massive dump is produced for the whole test run. This is very suboptimal, but this can only be solved with libvirt admin API which can enable/modify/redirect logs at runtime, however, the perl bindings are currently missings -> I'm already working on adding the bindings to libvirt-perl and then we need to teach TCK to use that feature Known issues ============ There are already a couple of known test failures which can be seen in the example pipeline: 1) modular daemons occasionally hang in some test scenarios (doesn't happen with monolithic) https://bugzilla.redhat.com/show_bug.cgi?id=2044379 2) QEMU tray status detection when a CD media is changed -> this one is also intermittent, but no good reproducer data to attach to an official QEMU BZ has been harnessed so far [1] https://docs.gitlab.com/runner/executors/custom.html [2] https://gitlab.com/libvirt/libvirt-tck/ [3] https://avocado-framework.readthedocs.io/ [4] https://gitlab.com/eskultety/libvirt/-/pipelines/457836923 [5] https://gitlab.com/libvirt/libvirt-ci/ [6] https://docs.gitlab.com/runner/executors/ [7] https://gitlab.com/eskultety/libvirt-gitlab-executor [8] https://listman.redhat.com/archives/libvir-list/2021-June/msg00836.html Erik Skultety (4): ci: manifest: Allow RPM builds on CentOS Stream 8 ci: containers: Add CentOS Stream 9 target ci: manifest: Publish RPMs as artifacts on CentOS Stream and Fedoras gitlab-ci: Introduce a new test 'integration' pipeline stage .gitlab-ci-integration.yml | 116 +++++++++++++++++++++++ .gitlab-ci.yml | 17 +++- ci/containers/centos-stream-9.Dockerfile | 87 +++++++++++++++++ ci/gitlab.yml | 33 ++++++- ci/manifest.yml | 26 ++++- 5 files changed, 274 insertions(+), 5 deletions(-) create mode 100644 .gitlab-ci-integration.yml create mode 100644 ci/containers/centos-stream-9.Dockerfile -- 2.34.1
The meson version provided by the package managing system satisfies our minimum requirement. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/gitlab.yml | 1 - ci/manifest.yml | 2 -- 2 files changed, 3 deletions(-) diff --git a/ci/gitlab.yml b/ci/gitlab.yml index 120ece8fb7..8e27bca812 100644 --- a/ci/gitlab.yml +++ b/ci/gitlab.yml @@ -400,7 +400,6 @@ x86_64-centos-stream-8: allow_failure: false variables: NAME: centos-stream-8 - RPM: skip x86_64-debian-10: diff --git a/ci/manifest.yml b/ci/manifest.yml index 2b64effef8..cb7c5c4035 100644 --- a/ci/manifest.yml +++ b/ci/manifest.yml @@ -21,8 +21,6 @@ targets: centos-stream-8: jobs: - arch: x86_64 - variables: - RPM: skip debian-10: jobs: -- 2.34.1
On Mon, Jan 31, 2022 at 07:00:58PM +0100, Erik Skultety wrote:
The meson version provided by the package managing system satisfies our minimum requirement.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/gitlab.yml | 1 - ci/manifest.yml | 2 -- 2 files changed, 3 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/containers/centos-stream-9.Dockerfile | 87 ++++++++++++++++++++++++ ci/gitlab.yml | 14 ++++ ci/manifest.yml | 3 + 3 files changed, 104 insertions(+) create mode 100644 ci/containers/centos-stream-9.Dockerfile diff --git a/ci/containers/centos-stream-9.Dockerfile b/ci/containers/centos-stream-9.Dockerfile new file mode 100644 index 0000000000..4e2e10b078 --- /dev/null +++ b/ci/containers/centos-stream-9.Dockerfile @@ -0,0 +1,87 @@ +# THIS FILE WAS AUTO-GENERATED +# +# $ lcitool manifest ci/manifest.yml +# +# https://gitlab.com/libvirt/libvirt-ci + +FROM quay.io/centos/centos:stream9 + +RUN dnf update -y && \ + dnf install 'dnf-command(config-manager)' -y && \ + dnf config-manager --set-enabled -y crb && \ + dnf install -y \ + audit-libs-devel \ + augeas \ + bash-completion \ + ca-certificates \ + clang \ + cpp \ + cyrus-sasl-devel \ + device-mapper-devel \ + diffutils \ + dnsmasq \ + dwarves \ + ebtables \ + firewalld-filesystem \ + fuse-devel \ + gcc \ + gettext \ + git \ + glib2-devel \ + glibc-devel \ + glibc-langpack-en \ + gnutls-devel \ + grep \ + iproute \ + iproute-tc \ + iptables \ + iscsi-initiator-utils \ + kmod \ + libacl-devel \ + libattr-devel \ + libblkid-devel \ + libcap-ng-devel \ + libcurl-devel \ + libnl3-devel \ + libpcap-devel \ + libpciaccess-devel \ + librbd-devel \ + libselinux-devel \ + libssh-devel \ + libtirpc-devel \ + libwsman-devel \ + libxml2 \ + libxml2-devel \ + libxslt \ + lvm2 \ + make \ + meson \ + nfs-utils \ + ninja-build \ + numactl-devel \ + numad \ + parted-devel \ + perl-base \ + pkgconfig \ + polkit \ + python3 \ + python3-docutils \ + qemu-img \ + readline-devel \ + rpcgen \ + rpm-build \ + sanlock-devel \ + scrub \ + sed \ + systemd-devel \ + systemtap-sdt-devel \ + wireshark-devel \ + yajl-devel && \ + dnf autoremove -y && \ + dnf clean all -y && \ + rpm -qa | sort > /packages.txt + +ENV LANG "en_US.UTF-8" +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" diff --git a/ci/gitlab.yml b/ci/gitlab.yml index 8e27bca812..03dee70480 100644 --- a/ci/gitlab.yml +++ b/ci/gitlab.yml @@ -94,6 +94,13 @@ x86_64-centos-stream-8-container: NAME: centos-stream-8 +x86_64-centos-stream-9-container: + extends: .container_job + allow_failure: false + variables: + NAME: centos-stream-9 + + x86_64-debian-10-container: extends: .container_job allow_failure: false @@ -401,6 +408,13 @@ x86_64-centos-stream-8: variables: NAME: centos-stream-8 +x86_64-centos-stream-9: + extends: .native_build_job + needs: + - x86_64-centos-stream-9-container + allow_failure: false + variables: + NAME: centos-stream-9 x86_64-debian-10: extends: .native_build_job diff --git a/ci/manifest.yml b/ci/manifest.yml index cb7c5c4035..1cc589955c 100644 --- a/ci/manifest.yml +++ b/ci/manifest.yml @@ -22,6 +22,9 @@ targets: jobs: - arch: x86_64 + centos-stream-9: + jobs: + - arch: x86_64 debian-10: jobs: - arch: x86_64 -- 2.34.1
On Mon, Jan 31, 2022 at 07:00:59PM +0100, Erik Skultety wrote:
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/containers/centos-stream-9.Dockerfile | 87 ++++++++++++++++++++++++ ci/gitlab.yml | 14 ++++ ci/manifest.yml | 3 + 3 files changed, 104 insertions(+) create mode 100644 ci/containers/centos-stream-9.Dockerfile
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
diff --git a/ci/gitlab.yml b/ci/gitlab.yml index 8e27bca812..03dee70480 100644 --- a/ci/gitlab.yml +++ b/ci/gitlab.yml @@ -94,6 +94,13 @@ x86_64-centos-stream-8-container: NAME: centos-stream-8
+x86_64-centos-stream-9-container: + extends: .container_job + allow_failure: false + variables: + NAME: centos-stream-9 + + x86_64-debian-10-container: extends: .container_job allow_failure: false @@ -401,6 +408,13 @@ x86_64-centos-stream-8: variables: NAME: centos-stream-8
+x86_64-centos-stream-9: + extends: .native_build_job + needs: + - x86_64-centos-stream-9-container + allow_failure: false + variables: + NAME: centos-stream-9
Bearing in mind that c9s is an unreleased distro, we might find ourselves needing allow_failure: true - hopefully not, but lets keep an eye on how reliable this is once merged. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
...
+x86_64-centos-stream-9: + extends: .native_build_job + needs: + - x86_64-centos-stream-9-container + allow_failure: false + variables: + NAME: centos-stream-9
Bearing in mind that c9s is an unreleased distro, we might find ourselves needing allow_failure: true - hopefully not, but lets keep an eye on how reliable this is once merged.
What do you mean by unreleased? Erik
On Tue, Feb 01, 2022 at 02:22:16PM +0100, Erik Skultety wrote:
...
+x86_64-centos-stream-9: + extends: .native_build_job + needs: + - x86_64-centos-stream-9-container + allow_failure: false + variables: + NAME: centos-stream-9
Bearing in mind that c9s is an unreleased distro, we might find ourselves needing allow_failure: true - hopefully not, but lets keep an eye on how reliable this is once merged.
What do you mean by unreleased?
CentOS 9 stream is a rolling release distro. Stuff in c9s has had automated QA done, but it is by no means as reliable as the RHEL-9.0 GA release should eventually be, as its just a snapshot in time of unreleased distro content. I know openstack upstream CI integration has been hit by regressions appearing in c9s. One example was hotplug completely breaking recently due to the QEMU -device JSON bug. Of course this very CI for libvirt would identify that particular bug, but we're vulnerable to bugs in the rest of the distro still while its under development. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Feb 01, 2022 at 01:48:54PM +0000, Daniel P. Berrangé wrote:
On Tue, Feb 01, 2022 at 02:22:16PM +0100, Erik Skultety wrote:
...
+x86_64-centos-stream-9: + extends: .native_build_job + needs: + - x86_64-centos-stream-9-container + allow_failure: false + variables: + NAME: centos-stream-9
Bearing in mind that c9s is an unreleased distro, we might find ourselves needing allow_failure: true - hopefully not, but lets keep an eye on how reliable this is once merged.
What do you mean by unreleased?
CentOS 9 stream is a rolling release distro.
Stuff in c9s has had automated QA done, but it is by no means as reliable as the RHEL-9.0 GA release should eventually be, as its just a snapshot in time of unreleased distro content.
I know openstack upstream CI integration has been hit by regressions appearing in c9s. One example was hotplug completely breaking recently due to the QEMU -device JSON bug. Of course this very CI for libvirt would identify that particular bug, but we're vulnerable to bugs in the rest of the distro still while its under development.
I see what you mean now. I hope it'll behave more like regular Fedoras and not like Rawhide. I'll push the first 2 patches then. Thanks, Erik
We're already building libvirt in the containers already, if we publish the build in form of, say, RPMs, later stages of the pipeline can consume the RPMs instead of re-building libvirt from scratch. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- .gitlab-ci.yml | 3 ++- ci/gitlab.yml | 18 ++++++++++++++++++ ci/manifest.yml | 21 ++++++++++++++++++++- 3 files changed, 40 insertions(+), 2 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 6ba11a0431..4bcaf22ce2 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -30,7 +30,8 @@ include: '/ci/gitlab.yml' - meson dist -C build --no-tests - if test -x /usr/bin/rpmbuild && test "$RPM" != "skip"; then - rpmbuild --nodeps -ta build/meson-dist/libvirt-*.tar.xz; + rpmbuild --clean --nodeps --define "_topdir $PWD/rpmbuild" -ta build/meson-dist/libvirt-*.tar.xz; + mv rpmbuild/RPMS/x86_64 libvirt-rpms ; else meson compile -C build; meson test -C build --no-suite syntax-check --print-errorlogs; diff --git a/ci/gitlab.yml b/ci/gitlab.yml index 03dee70480..3fbe4b3f8c 100644 --- a/ci/gitlab.yml +++ b/ci/gitlab.yml @@ -407,6 +407,11 @@ x86_64-centos-stream-8: allow_failure: false variables: NAME: centos-stream-8 + artifacts: + expire_in: 1 day + paths: + - libvirt-rpms + x86_64-centos-stream-9: extends: .native_build_job @@ -415,6 +420,11 @@ x86_64-centos-stream-9: allow_failure: false variables: NAME: centos-stream-9 + artifacts: + expire_in: 1 day + paths: + - libvirt-rpms + x86_64-debian-10: extends: .native_build_job @@ -459,6 +469,10 @@ x86_64-fedora-34: allow_failure: false variables: NAME: fedora-34 + artifacts: + expire_in: 1 day + paths: + - libvirt-rpms x86_64-fedora-35: @@ -468,6 +482,10 @@ x86_64-fedora-35: allow_failure: false variables: NAME: fedora-35 + artifacts: + expire_in: 1 day + paths: + - libvirt-rpms x86_64-fedora-rawhide: diff --git a/ci/manifest.yml b/ci/manifest.yml index 1cc589955c..2aba242948 100644 --- a/ci/manifest.yml +++ b/ci/manifest.yml @@ -21,10 +21,19 @@ targets: centos-stream-8: jobs: - arch: x86_64 + artifacts: + expire_in: 1 day + paths: + - libvirt-rpms centos-stream-9: jobs: - arch: x86_64 + artifacts: + expire_in: 1 day + paths: + - libvirt-rpms + debian-10: jobs: - arch: x86_64 @@ -126,11 +135,21 @@ targets: - arch: s390x allow-failure: true - fedora-34: x86_64 + fedora-34: + jobs: + - arch: x86_64 + artifacts: + expire_in: 1 day + paths: + - libvirt-rpms fedora-35: jobs: - arch: x86_64 + artifacts: + expire_in: 1 day + paths: + - libvirt-rpms - arch: mingw32 allow-failure: true -- 2.34.1
On Mon, Jan 31, 2022 at 07:01:00PM +0100, Erik Skultety wrote:
We're already building libvirt in the containers already, if we publish the build in form of, say, RPMs, later stages of the pipeline can consume the RPMs instead of re-building libvirt from scratch.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- .gitlab-ci.yml | 3 ++- ci/gitlab.yml | 18 ++++++++++++++++++ ci/manifest.yml | 21 ++++++++++++++++++++- 3 files changed, 40 insertions(+), 2 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, Jan 31, 2022 at 07:01:00PM +0100, Erik Skultety wrote:
We're already building libvirt in the containers already,
You already used "already" in this sentence already ;)
if we publish the build in form of, say, RPMs,
No need to be vague - we're publishing them *exactly* as RPMs.
+++ b/.gitlab-ci.yml @@ -30,7 +30,8 @@ include: '/ci/gitlab.yml' - meson dist -C build --no-tests - if test -x /usr/bin/rpmbuild && test "$RPM" != "skip"; then - rpmbuild --nodeps -ta build/meson-dist/libvirt-*.tar.xz; + rpmbuild --clean --nodeps --define "_topdir $PWD/rpmbuild" -ta build/meson-dist/libvirt-*.tar.xz; + mv rpmbuild/RPMS/x86_64 libvirt-rpms ;
No whitespace before ";" please. I'd also like to have "/" at the end of each directory name to make it obvious that we're moving around directories rather than files.
+++ b/ci/gitlab.yml @@ -407,6 +407,11 @@ x86_64-centos-stream-8: allow_failure: false variables: NAME: centos-stream-8 + artifacts: + expire_in: 1 day + paths: + - libvirt-rpms +
x86_64-centos-stream-9: extends: .native_build_job
This looks like it's adding additional empty lines, and if you regenerate the file using 'lcitool manifest' you'll find that they get removed. But they are actually supposed to be there, and the fact that they aren't is a bug in lcitool. https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/226 -- Andrea Bolognani / Red Hat / Virtualization
Create an integration child pipeline in this stage which will trigger a multi-project CI build of Perl bindings which are required by the TCK test suite. In general, this stage will install all the necessary build artifacts and configure logging on the worker node prior to executing the actual test suite. In case of a failure, libvirt and Avocado logs are saved and published as artifacts. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- .gitlab-ci-integration.yml | 116 +++++++++++++++++++++++++++++++++++++ .gitlab-ci.yml | 14 +++++ 2 files changed, 130 insertions(+) create mode 100644 .gitlab-ci-integration.yml diff --git a/.gitlab-ci-integration.yml b/.gitlab-ci-integration.yml new file mode 100644 index 0000000000..cabefc5166 --- /dev/null +++ b/.gitlab-ci-integration.yml @@ -0,0 +1,116 @@ +stages: + - bindings + - integration + +.tests: + stage: integration + before_script: + - mkdir "$SCRATCH_DIR" + - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/* + - sudo pip3 install --prefix=/usr avocado-framework + - source /etc/os-release # in order to query the vendor-provided variables + - if test "$ID" == "centos" && test "$VERSION_ID" -lt 9 || + test "$ID" == "fedora" && test "$VERSION_ID" -lt 35; + then + DAEMONS="libvirtd virtlogd virtlockd"; + else + DAEMONS="virtproxyd virtqemud virtinterfaced virtsecretd virtstoraged virtnwfilterd virtnodedevd virtlogd virtlockd"; + fi + - for daemon in $DAEMONS; + do + sudo sed -Ei "s/^(#)?(log_outputs=).*/\2'1:file:\/var\/log\/libvirt\/${daemon}.log'/" /etc/libvirt/${daemon}.conf; + sudo sed -Ei "s/^(#)?(log_filters=).*/\2'4:*object* 4:*json* 4:*event* 4:*rpc* 4:daemon.remote 4:util.threadjob 4:*access* 1:*'/" /etc/libvirt/${daemon}.conf; + sudo systemctl --quiet stop ${daemon}.service; + sudo systemctl restart ${daemon}.socket; + done + - sudo virsh net-start default &>/dev/null || true; + + script: + - mkdir logs + - cd "$SCRATCH_DIR" + - git clone --depth 1 https://gitlab.com/libvirt/libvirt-tck.git + - cd libvirt-tck + - sudo avocado --config avocado.config run --job-results-dir "$SCRATCH_DIR"/avocado + after_script: + - if test -e "$SCRATCH_DIR"/avocado; + then + sudo mv "$SCRATCH_DIR"/avocado/latest/test-results logs/avocado; + fi + - sudo mv /var/log/libvirt logs/libvirt + - sudo chown -R $(whoami):$(whoami) logs + variables: + SCRATCH_DIR: "/tmp/scratch" + artifacts: + name: logs + paths: + - logs + when: on_failure + + +libvirt-perl-bindings: + stage: bindings + trigger: + project: eskultety/libvirt-perl + branch: multi-project-ci + strategy: depend + + +centos-stream-8-tests: + extends: .tests + needs: + - libvirt-perl-bindings + - pipeline: $PARENT_PIPELINE_ID + job: x86_64-centos-stream-8 + - project: eskultety/libvirt-perl + job: x86_64-centos-stream-8 + ref: multi-project-ci + artifacts: true + variables: + DISTRO: centos-stream-8 + tags: + - centos-stream-vm + +centos-stream-9-tests: + extends: .tests + needs: + - libvirt-perl-bindings + - pipeline: $PARENT_PIPELINE_ID + job: x86_64-centos-stream-9 + - project: eskultety/libvirt-perl + job: x86_64-centos-stream-9 + ref: multi-project-ci + artifacts: true + variables: + DISTRO: centos-stream-9 + tags: + - centos-stream-vm + +fedora-34-tests: + extends: .tests + needs: + - libvirt-perl-bindings + - pipeline: $PARENT_PIPELINE_ID + job: x86_64-fedora-34 + - project: eskultety/libvirt-perl + job: x86_64-fedora-34 + ref: multi-project-ci + artifacts: true + variables: + DISTRO: fedora-34 + tags: + - fedora-vm + +fedora-35-tests: + extends: .tests + needs: + - libvirt-perl-bindings + - pipeline: $PARENT_PIPELINE_ID + job: x86_64-fedora-35 + - project: eskultety/libvirt-perl + job: x86_64-fedora-35 + ref: multi-project-ci + artifacts: true + variables: + DISTRO: fedora-35 + tags: + - fedora-vm diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 4bcaf22ce2..453472c8be 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -4,6 +4,7 @@ variables: stages: - containers - builds + - test - sanity_checks .script_variables: &script_variables | @@ -128,3 +129,16 @@ coverity: - curl https://scan.coverity.com/builds?project=$COVERITY_SCAN_PROJECT_NAME --form token=$COVERITY_SCAN_TOKEN --form email=$GITLAB_USER_EMAIL --form file=@cov-int.tar.gz --form version="$(git describe --tags)" --form description="$(git describe --tags) / $CI_COMMIT_TITLE / $CI_COMMIT_REF_NAME:$CI_PIPELINE_ID" rules: - if: "$CI_PIPELINE_SOURCE == 'schedule' && $COVERITY_SCAN_PROJECT_NAME && $COVERITY_SCAN_TOKEN" + +integration: + stage: test + needs: + - x86_64-centos-stream-8 + - x86_64-centos-stream-9 + - x86_64-fedora-34 + - x86_64-fedora-35 + trigger: + include: .gitlab-ci-integration.yml + strategy: depend + variables: + PARENT_PIPELINE_ID: $CI_PIPELINE_ID -- 2.34.1
On Mon, Jan 31, 2022 at 07:01:01PM +0100, Erik Skultety wrote:
Create an integration child pipeline in this stage which will trigger a multi-project CI build of Perl bindings which are required by the TCK test suite. In general, this stage will install all the necessary build artifacts and configure logging on the worker node prior to executing the actual test suite. In case of a failure, libvirt and Avocado logs are saved and published as artifacts.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- .gitlab-ci-integration.yml | 116 +++++++++++++++++++++++++++++++++++++ .gitlab-ci.yml | 14 +++++ 2 files changed, 130 insertions(+) create mode 100644 .gitlab-ci-integration.yml
diff --git a/.gitlab-ci-integration.yml b/.gitlab-ci-integration.yml new file mode 100644 index 0000000000..cabefc5166 --- /dev/null +++ b/.gitlab-ci-integration.yml @@ -0,0 +1,116 @@ +stages: + - bindings + - integration + +.tests: + stage: integration + before_script: + - mkdir "$SCRATCH_DIR" + - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/* + - sudo pip3 install --prefix=/usr avocado-framework + - source /etc/os-release # in order to query the vendor-provided variables + - if test "$ID" == "centos" && test "$VERSION_ID" -lt 9 || + test "$ID" == "fedora" && test "$VERSION_ID" -lt 35; + then + DAEMONS="libvirtd virtlogd virtlockd"; + else + DAEMONS="virtproxyd virtqemud virtinterfaced virtsecretd virtstoraged virtnwfilterd virtnodedevd virtlogd virtlockd"; + fi + - for daemon in $DAEMONS; + do + sudo sed -Ei "s/^(#)?(log_outputs=).*/\2'1:file:\/var\/log\/libvirt\/${daemon}.log'/" /etc/libvirt/${daemon}.conf; + sudo sed -Ei "s/^(#)?(log_filters=).*/\2'4:*object* 4:*json* 4:*event* 4:*rpc* 4:daemon.remote 4:util.threadjob 4:*access* 1:*'/" /etc/libvirt/${daemon}.conf; + sudo systemctl --quiet stop ${daemon}.service; + sudo systemctl restart ${daemon}.socket; + done + - sudo virsh net-start default &>/dev/null || true;
What context is all this being run in ? There's no docker image listed in the container, and I see the 'tags' referring to VMs. Is this really installing stuff as root in the VM runnerrs ? Basically I'm not seeing where any of this work is cleaned up to give a pristine environment for the next pipeline, or what happens if two pipelines run concurrently ?
+ + script: + - mkdir logs + - cd "$SCRATCH_DIR" + - git clone --depth 1 https://gitlab.com/libvirt/libvirt-tck.git + - cd libvirt-tck + - sudo avocado --config avocado.config run --job-results-dir "$SCRATCH_DIR"/avocado + after_script: + - if test -e "$SCRATCH_DIR"/avocado; + then + sudo mv "$SCRATCH_DIR"/avocado/latest/test-results logs/avocado; + fi + - sudo mv /var/log/libvirt logs/libvirt + - sudo chown -R $(whoami):$(whoami) logs + variables: + SCRATCH_DIR: "/tmp/scratch" + artifacts: + name: logs + paths: + - logs + when: on_failure + + +libvirt-perl-bindings: + stage: bindings + trigger: + project: eskultety/libvirt-perl
I assume pointing to this personal repo is a temp hack for some fix that's not merged into main libvirt-perl ?
+ branch: multi-project-ci + strategy: depend + + +centos-stream-8-tests: + extends: .tests + needs: + - libvirt-perl-bindings + - pipeline: $PARENT_PIPELINE_ID + job: x86_64-centos-stream-8 + - project: eskultety/libvirt-perl + job: x86_64-centos-stream-8 + ref: multi-project-ci + artifacts: true + variables: + DISTRO: centos-stream-8 + tags: + - centos-stream-vm + +centos-stream-9-tests: + extends: .tests + needs: + - libvirt-perl-bindings + - pipeline: $PARENT_PIPELINE_ID + job: x86_64-centos-stream-9 + - project: eskultety/libvirt-perl + job: x86_64-centos-stream-9 + ref: multi-project-ci + artifacts: true + variables: + DISTRO: centos-stream-9 + tags: + - centos-stream-vm + +fedora-34-tests: + extends: .tests + needs: + - libvirt-perl-bindings + - pipeline: $PARENT_PIPELINE_ID + job: x86_64-fedora-34 + - project: eskultety/libvirt-perl + job: x86_64-fedora-34 + ref: multi-project-ci + artifacts: true + variables: + DISTRO: fedora-34 + tags: + - fedora-vm + +fedora-35-tests: + extends: .tests + needs: + - libvirt-perl-bindings + - pipeline: $PARENT_PIPELINE_ID + job: x86_64-fedora-35 + - project: eskultety/libvirt-perl + job: x86_64-fedora-35 + ref: multi-project-ci + artifacts: true + variables: + DISTRO: fedora-35 + tags: + - fedora-vm diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 4bcaf22ce2..453472c8be 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -4,6 +4,7 @@ variables: stages: - containers - builds + - test - sanity_checks
.script_variables: &script_variables | @@ -128,3 +129,16 @@ coverity: - curl https://scan.coverity.com/builds?project=$COVERITY_SCAN_PROJECT_NAME --form token=$COVERITY_SCAN_TOKEN --form email=$GITLAB_USER_EMAIL --form file=@cov-int.tar.gz --form version="$(git describe --tags)" --form description="$(git describe --tags) / $CI_COMMIT_TITLE / $CI_COMMIT_REF_NAME:$CI_PIPELINE_ID" rules: - if: "$CI_PIPELINE_SOURCE == 'schedule' && $COVERITY_SCAN_PROJECT_NAME && $COVERITY_SCAN_TOKEN" + +integration: + stage: test + needs: + - x86_64-centos-stream-8 + - x86_64-centos-stream-9 + - x86_64-fedora-34 + - x86_64-fedora-35 + trigger: + include: .gitlab-ci-integration.yml + strategy: depend + variables: + PARENT_PIPELINE_ID: $CI_PIPELINE_ID
I've not really thought about the implications, so I'm curious what's the rationale for using a separate pipeline in this way, as opposed to have x86_64-fedora-35-integration, x86_64-centos-stream-9-integration, etc jobs in the existing pipeline ? Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Feb 01, 2022 at 12:36:11PM +0000, Daniel P. Berrangé wrote:
On Mon, Jan 31, 2022 at 07:01:01PM +0100, Erik Skultety wrote:
Create an integration child pipeline in this stage which will trigger a multi-project CI build of Perl bindings which are required by the TCK test suite. In general, this stage will install all the necessary build artifacts and configure logging on the worker node prior to executing the actual test suite. In case of a failure, libvirt and Avocado logs are saved and published as artifacts.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- .gitlab-ci-integration.yml | 116 +++++++++++++++++++++++++++++++++++++ .gitlab-ci.yml | 14 +++++ 2 files changed, 130 insertions(+) create mode 100644 .gitlab-ci-integration.yml
diff --git a/.gitlab-ci-integration.yml b/.gitlab-ci-integration.yml new file mode 100644 index 0000000000..cabefc5166 --- /dev/null +++ b/.gitlab-ci-integration.yml @@ -0,0 +1,116 @@ +stages: + - bindings + - integration + +.tests: + stage: integration + before_script: + - mkdir "$SCRATCH_DIR" + - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/* + - sudo pip3 install --prefix=/usr avocado-framework + - source /etc/os-release # in order to query the vendor-provided variables + - if test "$ID" == "centos" && test "$VERSION_ID" -lt 9 || + test "$ID" == "fedora" && test "$VERSION_ID" -lt 35; + then + DAEMONS="libvirtd virtlogd virtlockd"; + else + DAEMONS="virtproxyd virtqemud virtinterfaced virtsecretd virtstoraged virtnwfilterd virtnodedevd virtlogd virtlockd"; + fi + - for daemon in $DAEMONS; + do + sudo sed -Ei "s/^(#)?(log_outputs=).*/\2'1:file:\/var\/log\/libvirt\/${daemon}.log'/" /etc/libvirt/${daemon}.conf; + sudo sed -Ei "s/^(#)?(log_filters=).*/\2'4:*object* 4:*json* 4:*event* 4:*rpc* 4:daemon.remote 4:util.threadjob 4:*access* 1:*'/" /etc/libvirt/${daemon}.conf; + sudo systemctl --quiet stop ${daemon}.service; + sudo systemctl restart ${daemon}.socket; + done + - sudo virsh net-start default &>/dev/null || true;
What context is all this being run in ?
There's no docker image listed in the container, and I see the 'tags' referring to VMs. Is this really installing stuff as root in the VM runnerrs ?
Basically I'm not seeing where any of this work is cleaned up to give a pristine environment for the next pipeline, or what happens if two pipelines run concurrently ?
- all of ^this runs in the VM runner and yes, the gitlab-runner user has passwordless sudo ONLY inside the VM => note that the same is not true for the host where the gitlab-runner agent runs and dispatches jobs to the VMs with no permissions and can essentially just SSH into the VM; that was IIRC the only way how to make it all work -> I also remember having read in the docs that the gitlab-runner user must have passwordless sudo in the VM with custom executor, although I can't find that piece in the docs now => if you're concerned about breaking from the isolation, well, that has nothing to do with CI, but libvirt+QEMU/KVM in general; also, even if a malicious process took over the VM itself, GitLab has a watchdog, so the VM would get destroyed anyway after the job timed out - the work doesn't need to be cleaned up, because the VM (which is transient) gets destroyed automatically at the end of the job along with its storage overlay which is re-created for every VM, kinda like OpenStack does it - there's a hard limit on number of jobs a gitlab-runner-controlled host can take which we're in charge of, IOW depending on the baremetal capabilities, we select the appropriate number of jobs to avoid resource overcommit => in any case, if you want to see how a machine is created, you can find it in https://gitlab.com/eskultety/libvirt-gitlab-executor/-/blob/master/src/provi...
+ + script: + - mkdir logs + - cd "$SCRATCH_DIR" + - git clone --depth 1 https://gitlab.com/libvirt/libvirt-tck.git + - cd libvirt-tck + - sudo avocado --config avocado.config run --job-results-dir "$SCRATCH_DIR"/avocado + after_script: + - if test -e "$SCRATCH_DIR"/avocado; + then + sudo mv "$SCRATCH_DIR"/avocado/latest/test-results logs/avocado; + fi + - sudo mv /var/log/libvirt logs/libvirt + - sudo chown -R $(whoami):$(whoami) logs + variables: + SCRATCH_DIR: "/tmp/scratch" + artifacts: + name: logs + paths: + - logs + when: on_failure + + +libvirt-perl-bindings: + stage: bindings + trigger: + project: eskultety/libvirt-perl
I assume pointing to this personal repo is a temp hack for some fix that's not merged into main libvirt-perl ?
Yeah, sorry about that. I mean this is RFC, so I don't expect this to be merged as is and because in main libvirt there are more changes than in libvirt-perl, I didn't bother fixing it just for the sake of this RFC :). After all, this way, you can see it works in my fork. ...
.script_variables: &script_variables | @@ -128,3 +129,16 @@ coverity: - curl https://scan.coverity.com/builds?project=$COVERITY_SCAN_PROJECT_NAME --form token=$COVERITY_SCAN_TOKEN --form email=$GITLAB_USER_EMAIL --form file=@cov-int.tar.gz --form version="$(git describe --tags)" --form description="$(git describe --tags) / $CI_COMMIT_TITLE / $CI_COMMIT_REF_NAME:$CI_PIPELINE_ID" rules: - if: "$CI_PIPELINE_SOURCE == 'schedule' && $COVERITY_SCAN_PROJECT_NAME && $COVERITY_SCAN_TOKEN" + +integration: + stage: test + needs: + - x86_64-centos-stream-8 + - x86_64-centos-stream-9 + - x86_64-fedora-34 + - x86_64-fedora-35 + trigger: + include: .gitlab-ci-integration.yml + strategy: depend + variables: + PARENT_PIPELINE_ID: $CI_PIPELINE_ID
I've not really thought about the implications, so I'm curious what's the rationale for using a separate pipeline in this way, as opposed to have x86_64-fedora-35-integration, x86_64-centos-stream-9-integration, etc jobs in the existing pipeline ?
Because you still need the perl bindings. My reasoning is that despite being possibly less efficient it is desirable if we build the bindings in the respective repo (libvirt-perl) which already has the pipeline. We only need to kick off that one only once, not per each integration platform. If the consensus is going to be that we want to re-build the bindings in the VM, then we don't need to spawn a multi-project pipeline and indeed e.g. x86_64-centos-stream-9-integration could be a standalone stage placed right after the builds. Secondly, it was exciting trying out the functionality in GitLab. It also leaves the stages "self-contained" meaning that the VM only does what it's supposed to, because the rest can be handled by something else which is already in place and working. Erik
On Tue, Feb 01, 2022 at 02:45:15PM +0100, Erik Skultety wrote:
On Tue, Feb 01, 2022 at 12:36:11PM +0000, Daniel P. Berrangé wrote:
On Mon, Jan 31, 2022 at 07:01:01PM +0100, Erik Skultety wrote:
Create an integration child pipeline in this stage which will trigger a multi-project CI build of Perl bindings which are required by the TCK test suite. In general, this stage will install all the necessary build artifacts and configure logging on the worker node prior to executing the actual test suite. In case of a failure, libvirt and Avocado logs are saved and published as artifacts.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- .gitlab-ci-integration.yml | 116 +++++++++++++++++++++++++++++++++++++ .gitlab-ci.yml | 14 +++++ 2 files changed, 130 insertions(+) create mode 100644 .gitlab-ci-integration.yml
diff --git a/.gitlab-ci-integration.yml b/.gitlab-ci-integration.yml new file mode 100644 index 0000000000..cabefc5166 --- /dev/null +++ b/.gitlab-ci-integration.yml @@ -0,0 +1,116 @@ +stages: + - bindings + - integration + +.tests: + stage: integration + before_script: + - mkdir "$SCRATCH_DIR" + - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/* + - sudo pip3 install --prefix=/usr avocado-framework + - source /etc/os-release # in order to query the vendor-provided variables + - if test "$ID" == "centos" && test "$VERSION_ID" -lt 9 || + test "$ID" == "fedora" && test "$VERSION_ID" -lt 35; + then + DAEMONS="libvirtd virtlogd virtlockd"; + else + DAEMONS="virtproxyd virtqemud virtinterfaced virtsecretd virtstoraged virtnwfilterd virtnodedevd virtlogd virtlockd"; + fi + - for daemon in $DAEMONS; + do + sudo sed -Ei "s/^(#)?(log_outputs=).*/\2'1:file:\/var\/log\/libvirt\/${daemon}.log'/" /etc/libvirt/${daemon}.conf; + sudo sed -Ei "s/^(#)?(log_filters=).*/\2'4:*object* 4:*json* 4:*event* 4:*rpc* 4:daemon.remote 4:util.threadjob 4:*access* 1:*'/" /etc/libvirt/${daemon}.conf; + sudo systemctl --quiet stop ${daemon}.service; + sudo systemctl restart ${daemon}.socket; + done + - sudo virsh net-start default &>/dev/null || true;
What context is all this being run in ?
There's no docker image listed in the container, and I see the 'tags' referring to VMs. Is this really installing stuff as root in the VM runnerrs ?
Basically I'm not seeing where any of this work is cleaned up to give a pristine environment for the next pipeline, or what happens if two pipelines run concurrently ?
- all of ^this runs in the VM runner and yes, the gitlab-runner user has passwordless sudo ONLY inside the VM => note that the same is not true for the host where the gitlab-runner agent runs and dispatches jobs to the VMs with no permissions and can essentially just SSH into the VM; that was IIRC the only way how to make it all work -> I also remember having read in the docs that the gitlab-runner user must have passwordless sudo in the VM with custom executor, although I can't find that piece in the docs now
=> if you're concerned about breaking from the isolation, well, that has nothing to do with CI, but libvirt+QEMU/KVM in general; also, even if a malicious process took over the VM itself, GitLab has a watchdog, so the VM would get destroyed anyway after the job timed out
- the work doesn't need to be cleaned up, because the VM (which is transient) gets destroyed automatically at the end of the job along with its storage overlay which is re-created for every VM, kinda like OpenStack does it
- there's a hard limit on number of jobs a gitlab-runner-controlled host can take which we're in charge of, IOW depending on the baremetal capabilities, we select the appropriate number of jobs to avoid resource overcommit
=> in any case, if you want to see how a machine is created, you can find it in https://gitlab.com/eskultety/libvirt-gitlab-executor/-/blob/master/src/provi...
Thanks, this all makes alot of sense, and exactly what I wanted to see being done. So for the libvirt project we'll just need to get the bare metal host registered as the custom runer, and then it'll spawn the real VMs per job from pre-built template base images.
.script_variables: &script_variables | @@ -128,3 +129,16 @@ coverity: - curl https://scan.coverity.com/builds?project=$COVERITY_SCAN_PROJECT_NAME --form token=$COVERITY_SCAN_TOKEN --form email=$GITLAB_USER_EMAIL --form file=@cov-int.tar.gz --form version="$(git describe --tags)" --form description="$(git describe --tags) / $CI_COMMIT_TITLE / $CI_COMMIT_REF_NAME:$CI_PIPELINE_ID" rules: - if: "$CI_PIPELINE_SOURCE == 'schedule' && $COVERITY_SCAN_PROJECT_NAME && $COVERITY_SCAN_TOKEN" + +integration: + stage: test + needs: + - x86_64-centos-stream-8 + - x86_64-centos-stream-9 + - x86_64-fedora-34 + - x86_64-fedora-35 + trigger: + include: .gitlab-ci-integration.yml + strategy: depend + variables: + PARENT_PIPELINE_ID: $CI_PIPELINE_ID
I've not really thought about the implications, so I'm curious what's the rationale for using a separate pipeline in this way, as opposed to have x86_64-fedora-35-integration, x86_64-centos-stream-9-integration, etc jobs in the existing pipeline ?
Because you still need the perl bindings. My reasoning is that despite being possibly less efficient it is desirable if we build the bindings in the respective repo (libvirt-perl) which already has the pipeline. We only need to kick off that one only once, not per each integration platform. If the consensus is going to be that we want to re-build the bindings in the VM, then we don't need to spawn a multi-project pipeline and indeed e.g. x86_64-centos-stream-9-integration could be a standalone stage placed right after the builds.
Yes that makes sense
Secondly, it was exciting trying out the functionality in GitLab. It also leaves the stages "self-contained" meaning that the VM only does what it's supposed to, because the rest can be handled by something else which is already in place and working.
Indeed, it is a quite interesting & clever capability. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, Jan 31, 2022 at 07:01:01PM +0100, Erik Skultety wrote:
Create an integration child pipeline in this stage which will trigger a multi-project CI build of Perl bindings which are required by the TCK test suite. In general, this stage will install all the necessary build artifacts and configure logging on the worker node prior to executing the actual test suite. In case of a failure, libvirt and Avocado logs are saved and published as artifacts.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- .gitlab-ci-integration.yml | 116 +++++++++++++++++++++++++++++++++++++
Sine we have a free choice of naming in this case, how about we put it in ci/integration.yml
diff --git a/.gitlab-ci-integration.yml b/.gitlab-ci-integration.yml new file mode 100644 index 0000000000..cabefc5166 --- /dev/null +++ b/.gitlab-ci-integration.yml @@ -0,0 +1,116 @@ +stages: + - bindings + - integration + +.tests: + stage: integration + before_script: + - mkdir "$SCRATCH_DIR" + - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/* + - sudo pip3 install --prefix=/usr avocado-framework
I'd prefer it if we can just 'dnf install avocado' on Fedora at least, so that we validate that if someone is running avocado locally we're compatible with what's packaged.
+ - source /etc/os-release # in order to query the vendor-provided variables + - if test "$ID" == "centos" && test "$VERSION_ID" -lt 9 || + test "$ID" == "fedora" && test "$VERSION_ID" -lt 35; + then + DAEMONS="libvirtd virtlogd virtlockd"; + else + DAEMONS="virtproxyd virtqemud virtinterfaced virtsecretd virtstoraged virtnwfilterd virtnodedevd virtlogd virtlockd"; + fi + - for daemon in $DAEMONS; + do + sudo sed -Ei "s/^(#)?(log_outputs=).*/\2'1:file:\/var\/log\/libvirt\/${daemon}.log'/" /etc/libvirt/${daemon}.conf; + sudo sed -Ei "s/^(#)?(log_filters=).*/\2'4:*object* 4:*json* 4:*event* 4:*rpc* 4:daemon.remote 4:util.threadjob 4:*access* 1:*'/" /etc/libvirt/${daemon}.conf; + sudo systemctl --quiet stop ${daemon}.service; + sudo systemctl restart ${daemon}.socket; + done + - sudo virsh net-start default &>/dev/null || true; + + script: + - mkdir logs + - cd "$SCRATCH_DIR" + - git clone --depth 1 https://gitlab.com/libvirt/libvirt-tck.git + - cd libvirt-tck + - sudo avocado --config avocado.config run --job-results-dir "$SCRATCH_DIR"/avocado + after_script: + - if test -e "$SCRATCH_DIR"/avocado; + then + sudo mv "$SCRATCH_DIR"/avocado/latest/test-results logs/avocado; + fi + - sudo mv /var/log/libvirt logs/libvirt + - sudo chown -R $(whoami):$(whoami) logs + variables: + SCRATCH_DIR: "/tmp/scratch" + artifacts: + name: logs + paths: + - logs + when: on_failure + + +libvirt-perl-bindings: + stage: bindings + trigger: + project: eskultety/libvirt-perl + branch: multi-project-ci + strategy: depend
IIUC, what this does is to spawn a pipeline in the 'libvirt-perl' project on the 'multi-project-ci' branch. Normally this is asyncronous, but because of the 'strategy: depend' that causes us to block until this async pipeline is complete.
+centos-stream-8-tests: + extends: .tests + needs: + - libvirt-perl-bindings
So this triggers the perl bindings pipeline build
+ - pipeline: $PARENT_PIPELINE_ID + job: x86_64-centos-stream-8
This is making us wait for the centos-tream-8 job in the normal CI pipeline. Should we need 'artifacts: true' here too ? IIRC, artifacts true was the default, but it feels sane to make it explicit, especially since you were explicit for the libvirt-perl job below
+ - project: eskultety/libvirt-perl + job: x86_64-centos-stream-8 + ref: multi-project-ci + artifacts: true
And this making us wait for the cento-stream-8 job in te lbivirt-perl pipeline that we spawned in the 'libvirt-perl-bindings'. IIUC there should be no waiting needed since 'libvirt-perl-bindings' was blocking on completion of the trigger pipeline, so this effectivectly justpulls in the artifacts from the already finishd job. I presume your eskultety/libvirt-perl repo multi-project-ci branch has a cyhange that causes the perl-Sys-Virt RPMs to be publish as artifacts, similar to your change in the previous patch in this series.
+ variables: + DISTRO: centos-stream-8 + tags: + - centos-stream-vm
This means none of these jobs will run by default, unless you're registered a custom runner with this tag. So no forks will get these jobs, it'll be post-merge only. That's fine for now. When we switch to merge requests we will gain ability to trigger this even for forks without runners as merge request jobs will attach to the primary project runners, which is nice.
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 4bcaf22ce2..453472c8be 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -4,6 +4,7 @@ variables: stages: - containers - builds + - test - sanity_checks
Perhaps just put this in the 'sanity_checks' stage ? Since all the real jobs are in the child pipeline, I don't think we need an extra stage here.
.script_variables: &script_variables | @@ -128,3 +129,16 @@ coverity: - curl https://scan.coverity.com/builds?project=$COVERITY_SCAN_PROJECT_NAME --form token=$COVERITY_SCAN_TOKEN --form email=$GITLAB_USER_EMAIL --form file=@cov-int.tar.gz --form version="$(git describe --tags)" --form description="$(git describe --tags) / $CI_COMMIT_TITLE / $CI_COMMIT_REF_NAME:$CI_PIPELINE_ID" rules: - if: "$CI_PIPELINE_SOURCE == 'schedule' && $COVERITY_SCAN_PROJECT_NAME && $COVERITY_SCAN_TOKEN" + +integration: + stage: test + needs: + - x86_64-centos-stream-8 + - x86_64-centos-stream-9 + - x86_64-fedora-34 + - x86_64-fedora-35
Ok, so any job in the child pipeline with a dependancy on a job in this pipeline needs to have the dependancy repeated here.
+ trigger: + include: .gitlab-ci-integration.yml + strategy: depend + variables: + PARENT_PIPELINE_ID: $CI_PIPELINE_ID
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Feb 10, 2022 at 09:40:45AM +0000, Daniel P. Berrangé wrote:
On Mon, Jan 31, 2022 at 07:01:01PM +0100, Erik Skultety wrote:
Create an integration child pipeline in this stage which will trigger a multi-project CI build of Perl bindings which are required by the TCK test suite. In general, this stage will install all the necessary build artifacts and configure logging on the worker node prior to executing the actual test suite. In case of a failure, libvirt and Avocado logs are saved and published as artifacts.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- .gitlab-ci-integration.yml | 116 +++++++++++++++++++++++++++++++++++++
...
diff --git a/.gitlab-ci-integration.yml b/.gitlab-ci-integration.yml new file mode 100644 index 0000000000..cabefc5166 --- /dev/null +++ b/.gitlab-ci-integration.yml @@ -0,0 +1,116 @@ +stages: + - bindings + - integration + +.tests: + stage: integration + before_script: + - mkdir "$SCRATCH_DIR" + - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/* + - sudo pip3 install --prefix=/usr avocado-framework
I'd prefer it if we can just 'dnf install avocado' on Fedora at least, so that we validate that if someone is running avocado locally we're compatible with what's packaged.
That's my desire as well, but that is currently not possible because: a) the Avocado shipped over the standard package manager channel is too old for ^this, to be precise it would mark every skipped job as failed due to a a missing TAP parser fix (we need avocado-91+) b) the 3rd party RPM repo currently returns 404 on my Fedora-34, nevertheless last time I checked, even this repo shipped avocado-90 only Cleber, how realistic is it for the Avocado project to build RPMs with every release? ...
+ + +libvirt-perl-bindings: + stage: bindings + trigger: + project: eskultety/libvirt-perl + branch: multi-project-ci + strategy: depend
IIUC, what this does is to spawn a pipeline in the 'libvirt-perl' project on the 'multi-project-ci' branch. Normally this is asyncronous, but because of the 'strategy: depend' that causes us to block until this async pipeline is complete.
Yes, that's correct.
+centos-stream-8-tests: + extends: .tests + needs: + - libvirt-perl-bindings
So this triggers the perl bindings pipeline build
No, the trigger is the job 'libvirt-perl-bindings' above, but we need to wait for the trigger as well, otherwise the integration job would not wait until the new RPMs are available and instead it would just download the currently latest ones (which will be available even if expired!) I don't know whether this is a bug, but I originally didn't do it and I noticed the integration jobs never waited for the bindings to be built and instead downloaded the latest available copy.
+ - pipeline: $PARENT_PIPELINE_ID + job: x86_64-centos-stream-8
This is making us wait for the centos-tream-8 job in the normal CI pipeline.
Should we need 'artifacts: true' here too ?
I guess we can be explicit, but as long as the dependencies come from the same pipeline, artifacts are automatically available (that is by design)
IIRC, artifacts true was the default, but it feels sane to make it explicit, especially since you were explicit for the libvirt-perl job below
I can try whether gitlab is happy with that too, but conceptually, sure, why not.
+ - project: eskultety/libvirt-perl + job: x86_64-centos-stream-8 + ref: multi-project-ci + artifacts: true
And this making us wait for the cento-stream-8 job in te lbivirt-perl pipeline that we spawned in the 'libvirt-perl-bindings'. IIUC there should be no waiting needed since 'libvirt-perl-bindings' was blocking on completion of the trigger pipeline, so this effectivectly justpulls in the artifacts from the already finishd job.
Yes, exactly, this would only download the artifacts, but the syntax hints that a job is spawned :/ . However, as I wrote above, without waiting for the trigger job, ^this hunk would just pull the latest available artifacts instead of waiting for the ones currently being built
I presume your eskultety/libvirt-perl repo multi-project-ci branch has a cyhange that causes the perl-Sys-Virt RPMs to be publish as artifacts, similar to your change in the previous patch in this series.
Yes, in fact, I just submitted the following MRs: https://gitlab.com/libvirt/libvirt-perl/-/merge_requests/54 https://gitlab.com/libvirt/libvirt-perl/-/merge_requests/55 with the latter enabling what we need here.
+ variables: + DISTRO: centos-stream-8 + tags: + - centos-stream-vm
This means none of these jobs will run by default, unless you're registered a custom runner with this tag. So no forks will get these jobs, it'll be post-merge only.
That's fine for now. When we switch to merge requests we will gain ability to trigger this even for forks without runners as merge request jobs will attach to the primary project runners, which is nice.
Yes. Custom runners are always private to the project, so forks never have access to those unless we manually run the MR in context of the libvirt project.
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 4bcaf22ce2..453472c8be 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -4,6 +4,7 @@ variables: stages: - containers - builds + - test - sanity_checks
Perhaps just put this in the 'sanity_checks' stage ? Since all the real jobs are in the child pipeline, I don't think we need an extra stage here.
Not my preference, but I can have it either way, a new stage can be added at any time if needed.
.script_variables: &script_variables | @@ -128,3 +129,16 @@ coverity: - curl https://scan.coverity.com/builds?project=$COVERITY_SCAN_PROJECT_NAME --form token=$COVERITY_SCAN_TOKEN --form email=$GITLAB_USER_EMAIL --form file=@cov-int.tar.gz --form version="$(git describe --tags)" --form description="$(git describe --tags) / $CI_COMMIT_TITLE / $CI_COMMIT_REF_NAME:$CI_PIPELINE_ID" rules: - if: "$CI_PIPELINE_SOURCE == 'schedule' && $COVERITY_SCAN_PROJECT_NAME && $COVERITY_SCAN_TOKEN" + +integration: + stage: test + needs: + - x86_64-centos-stream-8 + - x86_64-centos-stream-9 + - x86_64-fedora-34 + - x86_64-fedora-35
Ok, so any job in the child pipeline with a dependancy on a job in this pipeline needs to have the dependancy repeated here.
I don't think you need it really, but we have so many container build running that waiting for ALL of them seemed pointless to me, so I added explicitly the ones we need so that the integration stage can start ASAP. Thanks, Erik
On Mon, Jan 31, 2022 at 07:01:01PM +0100, Erik Skultety wrote:
+++ b/.gitlab-ci-integration.yml @@ -0,0 +1,116 @@ +.tests: + stage: integration + before_script: + - mkdir "$SCRATCH_DIR" + - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/* + - sudo pip3 install --prefix=/usr avocado-framework + - source /etc/os-release # in order to query the vendor-provided variables + - if test "$ID" == "centos" && test "$VERSION_ID" -lt 9 || + test "$ID" == "fedora" && test "$VERSION_ID" -lt 35;
Using == with test is a bashism, please stick to the portable version even though it's very likely that the script will ultimately run under bash.
+ - for daemon in $DAEMONS; + do + sudo sed -Ei "s/^(#)?(log_outputs=).*/\2'1:file:\/var\/log\/libvirt\/${daemon}.log'/" /etc/libvirt/${daemon}.conf; + sudo sed -Ei "s/^(#)?(log_filters=).*/\2'4:*object* 4:*json* 4:*event* 4:*rpc* 4:daemon.remote 4:util.threadjob 4:*access* 1:*'/" /etc/libvirt/${daemon}.conf; + sudo systemctl --quiet stop ${daemon}.service; + sudo systemctl restart ${daemon}.socket; + done
I suggest changing this to something along the lines of - for daemon in $DAEMONS; do log_outputs="file:/var/log/libvirt/${daemon}.log" log_filters="3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 1:*" sed -Ei -e "s;^#*\\s*log_outputs\\s*=.*$;log_outputs=\"$log_outputs\";g" \ -e "s;^#*\\s*log_filters\\s*=.*$;log_filters=\"$log_filters\";g" \ "src/remote/${daemon}.conf.in" # ... done Advantages: * saves one call to sed; * doesn't need the awkward quoting for slashes in paths; * produces key="value" which is consistent with the existing contents of the configuration files (even though the parser will accept single quotes too); * handles slight, semantically-irrelevant changes to the contents of the configuration files such as whitespace being added; * doesn't unnecessarily use captures; * avoids excessively long lines. Note that I have adjusted the value of log_filters to match what is recommended in https://libvirt.org/kbase/debuglogs.html#less-verbose-logging-for-qemu-vms but maybe there's a reason you had picked a different set of filters. Also note that I haven't actually tested that the above works correctly O:-)
+ - sudo virsh net-start default &>/dev/null || true; + + script:
Unnecessary empty line here.
+ after_script: + - if test -e "$SCRATCH_DIR"/avocado; + then + sudo mv "$SCRATCH_DIR"/avocado/latest/test-results logs/avocado; + fi + - sudo mv /var/log/libvirt logs/libvirt + - sudo chown -R $(whoami):$(whoami) logs
Same as in the previous patch, I'd like for names of directories to end with a slash.
+libvirt-perl-bindings: + stage: bindings + trigger: + project: eskultety/libvirt-perl + branch: multi-project-ci + strategy: depend + + +centos-stream-8-tests: + extends: .tests + needs: + - libvirt-perl-bindings + - pipeline: $PARENT_PIPELINE_ID + job: x86_64-centos-stream-8 + - project: eskultety/libvirt-perl + job: x86_64-centos-stream-8 + ref: multi-project-ci + artifacts: true
IIUC from the documentation and from reading around, using strategy:depend will cause the local job to reflect the status of the triggered pipeline. So far so good. What I am unclear about is, is there any guarantee that using artifact:true with a job from an external project's pipeline will expose the artifacts from the job that was executed as part of the specific pipeline that we've just triggered ourselves as opposed to some other pipeline that might have already been completed in the past of might have completed in the meantime? Taking a step back, why exactly are we triggering a rebuild of libvirt-perl in the first place? Once we change that project's pipeline so that RPMs are published as artifacts, can't we just grab the ones from the latest successful pipeline? Maybe you've already explained why you did things this way and I just missed it!
+ variables: + DISTRO: centos-stream-8
This variable doesn't seem to be used anywhere, so I assume it's a leftover from development. Maybe you tried to implement the .test template so that using it didn't require as much repetition and unfortunately it didn't work out?
+ tags: + - centos-stream-vm
IIUC this is used both by the GitLab scheduler to pick suitable nodes on which to execute the job (our own hardware in this case) and also by the runner to decide which template to use for the VM. So shouldn't this be more specific? I would expect something like tags: - centos-stream-8-vm or tags: - centos-stream-8 - vm
+++ b/.gitlab-ci.yml @@ -4,6 +4,7 @@ variables: stages: - containers - builds + - test - sanity_checks
Unlike Dan, I actually think it makes sense to have these as a separate stage. I'd call it integration_tests and list it last though. But admittedly this is entirely in bikeshedding territory and I'm fine with the current solution or the one Dan proposed too :) -- Andrea Bolognani / Red Hat / Virtualization
...
+ - for daemon in $DAEMONS; + do + sudo sed -Ei "s/^(#)?(log_outputs=).*/\2'1:file:\/var\/log\/libvirt\/${daemon}.log'/" /etc/libvirt/${daemon}.conf; + sudo sed -Ei "s/^(#)?(log_filters=).*/\2'4:*object* 4:*json* 4:*event* 4:*rpc* 4:daemon.remote 4:util.threadjob 4:*access* 1:*'/" /etc/libvirt/${daemon}.conf; + sudo systemctl --quiet stop ${daemon}.service; + sudo systemctl restart ${daemon}.socket; + done
I suggest changing this to something along the lines of
- for daemon in $DAEMONS; do log_outputs="file:/var/log/libvirt/${daemon}.log" log_filters="3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 1:*" sed -Ei -e "s;^#*\\s*log_outputs\\s*=.*$;log_outputs=\"$log_outputs\";g" \ -e "s;^#*\\s*log_filters\\s*=.*$;log_filters=\"$log_filters\";g" \ "src/remote/${daemon}.conf.in" # ... done
Advantages:
* saves one call to sed; * doesn't need the awkward quoting for slashes in paths; * produces key="value" which is consistent with the existing contents of the configuration files (even though the parser will accept single quotes too); * handles slight, semantically-irrelevant changes to the contents of the configuration files such as whitespace being added; * doesn't unnecessarily use captures; * avoids excessively long lines.
Yes, ^this looks better.
Note that I have adjusted the value of log_filters to match what is recommended in
https://libvirt.org/kbase/debuglogs.html#less-verbose-logging-for-qemu-vms
but maybe there's a reason you had picked a different set of filters.
I didn't even know we document this, so I always use the filters I empirically settled with. Given the general feeling about warnings usefulness in libvirt logs I either use 1 for debug logs or 4 for errors. If you feel strong I should use what we recommend on that page, I'll go with that, but I'll add '4:util.threadjob' as well as threadjobs are also verbose and don't add any value. ...
Also note that I haven't actually tested that the above works correctly O:-)
It's a good starting point, I'll handle the rest. ...
+libvirt-perl-bindings: + stage: bindings + trigger: + project: eskultety/libvirt-perl + branch: multi-project-ci + strategy: depend + + +centos-stream-8-tests: + extends: .tests + needs: + - libvirt-perl-bindings + - pipeline: $PARENT_PIPELINE_ID + job: x86_64-centos-stream-8 + - project: eskultety/libvirt-perl + job: x86_64-centos-stream-8 + ref: multi-project-ci + artifacts: true
IIUC from the documentation and from reading around, using strategy:depend will cause the local job to reflect the status of the triggered pipeline. So far so good.
What I am unclear about is, is there any guarantee that using artifact:true with a job from an external project's pipeline will expose the artifacts from the job that was executed as part of the specific pipeline that we've just triggered ourselves as opposed to some other pipeline that might have already been completed in the past of might have completed in the meantime?
Not just by using artifact:true or strategy:depend. The important bit is having 'libvirt-perl-bindings' in the job's needs list. Let me explain, if you don't put the bindings trigger job to the requirements list of another job (centos-stream-8-tests in this case) what will happen is that the trigger job will be waiting for the external pipeline to finish, but centos-stream-8-tests job would execute basically as soon as the container project builds are finished because artifacts:true would download the latest RPM artifacts from an earlier build...
Taking a step back, why exactly are we triggering a rebuild of libvirt-perl in the first place? Once we change that project's pipeline so that RPMs are published as artifacts, can't we just grab the ones from the latest successful pipeline? Maybe you've already explained why you did things this way and I just missed it!
...which brings us here. Well, I adopted the mantra that all libvirt-friends projects depend on libvirt and given that we need libvirt-perl bindings to test upstream, I'd like to always have the latest bindings available to test with the current upstream build. The other reason why I did the way you commented on is that during development of the proposal many times I had to make changes to both libvirt and libvirt-perl in lockstep and it was tremendously frustrating to wait for the pipeline to get to the integration stage only to realize that the integration job didn't wait for the latest bindings and instead picked up the previous latest artifacts which I knew were either faulty or didn't contain the necessary changes yet.
+ variables: + DISTRO: centos-stream-8
This variable doesn't seem to be used anywhere, so I assume it's a leftover from development. Maybe you tried to implement the .test template so that using it didn't require as much repetition and unfortunately it didn't work out?
Oh but it is. This is how the gitlab provisioner script knows which distro to provision, it's equivalent to lcitool's target.
+ tags: + - centos-stream-vm
IIUC this is used both by the GitLab scheduler to pick suitable nodes on which to execute the job (our own hardware in this case) and also by the runner to decide which template to use for the VM.
So shouldn't this be more specific? I would expect something like
tags: - centos-stream-8-vm
What's the point, we'd have to constantly refresh the tags if the platforms come and go given our support, whereas fedora-vm and centos-stream-vm cover all currently supported versions - always! Other than that, I'm not sure that tags are passed on to the gitlab job itself, I may have missed it, but unless the tags are exposed as env variables, the provisioner script wouldn't know which template to provision. Also, the tag is supposed to annotate the baremetal host in this case, so in that context having '-vm' in the tag name makes sense, but doesn't for the provisioner script which relies on/tries to be compatible with lcitool as much as possible.
or
tags: - centos-stream-8 - vm
+++ b/.gitlab-ci.yml @@ -4,6 +4,7 @@ variables: stages: - containers - builds + - test - sanity_checks
Unlike Dan, I actually think it makes sense to have these as a separate stage. I'd call it integration_tests and list it last though. But admittedly this is entirely in bikeshedding territory and I'm fine with the current solution or the one Dan proposed too :)
I feel like having test/intengration_tests is more bulletproof for the future of our CI :). Erik
On Tue, Feb 22, 2022 at 03:19:58PM +0100, Erik Skultety wrote:
Note that I have adjusted the value of log_filters to match what is recommended in
https://libvirt.org/kbase/debuglogs.html#less-verbose-logging-for-qemu-vms
but maybe there's a reason you had picked a different set of filters.
I didn't even know we document this, so I always use the filters I empirically settled with. Given the general feeling about warnings usefulness in libvirt logs I either use 1 for debug logs or 4 for errors.
If you feel strong I should use what we recommend on that page, I'll go with that, but I'll add '4:util.threadjob' as well as threadjobs are also verbose and don't add any value.
Maybe change the page so 4:util.threadjob is included in the recommended configuration? Other than that, if you feel that your set of filters will work best for the situation at hand you don't need to change them.
+libvirt-perl-bindings: + stage: bindings + trigger: + project: eskultety/libvirt-perl + branch: multi-project-ci + strategy: depend + + +centos-stream-8-tests: + extends: .tests + needs: + - libvirt-perl-bindings + - pipeline: $PARENT_PIPELINE_ID + job: x86_64-centos-stream-8 + - project: eskultety/libvirt-perl + job: x86_64-centos-stream-8 + ref: multi-project-ci + artifacts: true
IIUC from the documentation and from reading around, using strategy:depend will cause the local job to reflect the status of the triggered pipeline. So far so good.
What I am unclear about is, is there any guarantee that using artifact:true with a job from an external project's pipeline will expose the artifacts from the job that was executed as part of the specific pipeline that we've just triggered ourselves as opposed to some other pipeline that might have already been completed in the past of might have completed in the meantime?
Not just by using artifact:true or strategy:depend. The important bit is having 'libvirt-perl-bindings' in the job's needs list. Let me explain, if you don't put the bindings trigger job to the requirements list of another job (centos-stream-8-tests in this case) what will happen is that the trigger job will be waiting for the external pipeline to finish, but centos-stream-8-tests job would execute basically as soon as the container project builds are finished because artifacts:true would download the latest RPM artifacts from an earlier build...
No, I got that part. My question was whether other-project-pipeline: trigger: project: other-project strategy: depend our-job: needs: - other-project-pipeline - project: other-project job: other-project-job artifacts: true actually guarantees that the instance of other-project-job whose artifacts are available to our-job is the same one that was started as part of the pipeline triggered by the other-project-pipeline job. Looking at the YAML I wouldn't bet on this being the case, but perhaps I've missed this guarantee being documented somewhere?
Taking a step back, why exactly are we triggering a rebuild of libvirt-perl in the first place? Once we change that project's pipeline so that RPMs are published as artifacts, can't we just grab the ones from the latest successful pipeline? Maybe you've already explained why you did things this way and I just missed it!
...which brings us here. Well, I adopted the mantra that all libvirt-friends projects depend on libvirt and given that we need libvirt-perl bindings to test upstream, I'd like to always have the latest bindings available to test with the current upstream build. The other reason why I did the way you commented on is that during development of the proposal many times I had to make changes to both libvirt and libvirt-perl in lockstep and it was tremendously frustrating to wait for the pipeline to get to the integration stage only to realize that the integration job didn't wait for the latest bindings and instead picked up the previous latest artifacts which I knew were either faulty or didn't contain the necessary changes yet.
Of course that would be annoying when you're making changes to both projects at the same time, but is that a scenario that we can expect to be common once the integration tests are in place? To be clear, I'm not necessarily against the way you're doing things right now, it's just that it feels like using the artifacts from the latest successful libvirt-perl pipeline would lower complexity, avoid burning additional resources and reduce wait times. If the only only downside is having a worse experience when making changes to the pipeline, and we can expect that to be infrequent enough, perhaps that's a reasonable tradeoff.
+ variables: + DISTRO: centos-stream-8
This variable doesn't seem to be used anywhere, so I assume it's a leftover from development. Maybe you tried to implement the .test template so that using it didn't require as much repetition and unfortunately it didn't work out?
Oh but it is. This is how the gitlab provisioner script knows which distro to provision, it's equivalent to lcitool's target.
I've looked at https://gitlab.com/eskultety/libvirt-gitlab-executor and understand how this value is used now, but without the additional context it's basically impossible to figure out its purpose. Please make sure you document it somehow.
+ tags: + - centos-stream-vm
IIUC this is used both by the GitLab scheduler to pick suitable nodes on which to execute the job (our own hardware in this case) and also by the runner to decide which template to use for the VM.
So shouldn't this be more specific? I would expect something like
tags: - centos-stream-8-vm
What's the point, we'd have to constantly refresh the tags if the platforms come and go given our support, whereas fedora-vm and centos-stream-vm cover all currently supported versions - always! Other than that, I'm not sure that tags are passed on to the gitlab job itself, I may have missed it, but unless the tags are exposed as env variables, the provisioner script wouldn't know which template to provision. Also, the tag is supposed to annotate the baremetal host in this case, so in that context having '-vm' in the tag name makes sense, but doesn't for the provisioner script which relies on/tries to be compatible with lcitool as much as possible.
Okay, my misunderstanding was caused by not figuring out the purpose of DISTRO. I agree that more specific tags are not necessary. Should we make them *less* specific instead? As in, is there any reason for having different tags for Fedora and CentOS jobs as opposed to using a generic "this needs to run in a VM" tag for both? -- Andrea Bolognani / Red Hat / Virtualization
...
+libvirt-perl-bindings: + stage: bindings + trigger: + project: eskultety/libvirt-perl + branch: multi-project-ci + strategy: depend + + +centos-stream-8-tests: + extends: .tests + needs: + - libvirt-perl-bindings + - pipeline: $PARENT_PIPELINE_ID + job: x86_64-centos-stream-8 + - project: eskultety/libvirt-perl + job: x86_64-centos-stream-8 + ref: multi-project-ci + artifacts: true
IIUC from the documentation and from reading around, using strategy:depend will cause the local job to reflect the status of the triggered pipeline. So far so good.
What I am unclear about is, is there any guarantee that using artifact:true with a job from an external project's pipeline will expose the artifacts from the job that was executed as part of the specific pipeline that we've just triggered ourselves as opposed to some other pipeline that might have already been completed in the past of might have completed in the meantime?
Not just by using artifact:true or strategy:depend. The important bit is having 'libvirt-perl-bindings' in the job's needs list. Let me explain, if you don't put the bindings trigger job to the requirements list of another job (centos-stream-8-tests in this case) what will happen is that the trigger job will be waiting for the external pipeline to finish, but centos-stream-8-tests job would execute basically as soon as the container project builds are finished because artifacts:true would download the latest RPM artifacts from an earlier build...
No, I got that part. My question was whether
other-project-pipeline: trigger: project: other-project strategy: depend
our-job: needs: - other-project-pipeline - project: other-project job: other-project-job artifacts: true
actually guarantees that the instance of other-project-job whose artifacts are available to our-job is the same one that was started as part of the pipeline triggered by the other-project-pipeline job.
Sorry for a delayed response. I don't think so. We can basically only rely on a fact that the jobs would actually be queued in order they arrive which means that jobs submitted earlier should finish earlier, but that of course is only a premise not a guarantee. On the other hand I never intended to run the integration CI on every single push to the master branch, instead, I wanted to make this a scheduled pipeline which would effectively alleviate the problem, because with scheduled pipelines there would very likely not be a concurrent pipeline running in libvirt-perl which would make us download artifacts from a pipeline we didn't trigger ourselves.
Looking at the YAML I wouldn't bet on this being the case, but perhaps I've missed this guarantee being documented somewhere?
Taking a step back, why exactly are we triggering a rebuild of libvirt-perl in the first place? Once we change that project's pipeline so that RPMs are published as artifacts, can't we just grab the ones from the latest successful pipeline? Maybe you've already explained why you did things this way and I just missed it!
...which brings us here. Well, I adopted the mantra that all libvirt-friends projects depend on libvirt and given that we need libvirt-perl bindings to test upstream, I'd like to always have the latest bindings available to test with the current upstream build. The other reason why I did the way you commented on is that during development of the proposal many times I had to make changes to both libvirt and libvirt-perl in lockstep and it was tremendously frustrating to wait for the pipeline to get to the integration stage only to realize that the integration job didn't wait for the latest bindings and instead picked up the previous latest artifacts which I knew were either faulty or didn't contain the necessary changes yet.
Of course that would be annoying when you're making changes to both projects at the same time, but is that a scenario that we can expect to be common once the integration tests are in place?
To be clear, I'm not necessarily against the way you're doing things right now, it's just that it feels like using the artifacts from the latest successful libvirt-perl pipeline would lower complexity, avoid burning additional resources and reduce wait times.
If the only only downside is having a worse experience when making changes to the pipeline, and we can expect that to be infrequent enough, perhaps that's a reasonable tradeoff.
I gave this more thought. What you suggest is viable, but the following is worth considering if we go with your proposal: - libvirt-perl jobs build upstream libvirt first in order to build the bindings -> generally it takes until right before the release that APIs/constants are added to the respective bindings (Perl/Python) -> if we rely on the latest libvirt-perl artifacts without actually triggering the pipeline, yes, the artifacts would be stable, but fairly old (unless we schedule a recurrent pipeline in the project to refresh them), thus not giving us feedback from the integration stage that bindings need to be added first, because the API coverage would likely fail, thus failing the whole libvirt-perl pipeline and thus invalidating the integration test stage in the libvirt project => now, I admit this would get pretty annoying because it would force contributors (or the maintainer) who add new APIs to add respective bindings as well in a timely manner, but then again ultimately we'd like our contributors to also introduce an integration test along with their feature... - as for resource consumption, given that this would either execute on a merge request basis or as a scheduled pipeline, it won't be that much of a resource waste, especially if we stop building containers in libvirt-<project> unless there was a change to the underlying Dockerfile, that is IMO the biggest resource waste, so we can always cut down on resources and still keep using fresh builds from both projects.
+ variables: + DISTRO: centos-stream-8
This variable doesn't seem to be used anywhere, so I assume it's a leftover from development. Maybe you tried to implement the .test template so that using it didn't require as much repetition and unfortunately it didn't work out?
Oh but it is. This is how the gitlab provisioner script knows which distro to provision, it's equivalent to lcitool's target.
I've looked at
https://gitlab.com/eskultety/libvirt-gitlab-executor
and understand how this value is used now, but without the additional context it's basically impossible to figure out its purpose. Please make sure you document it somehow.
I'll add a commentary documenting the variable.
+ tags: + - centos-stream-vm
IIUC this is used both by the GitLab scheduler to pick suitable nodes on which to execute the job (our own hardware in this case) and also by the runner to decide which template to use for the VM.
So shouldn't this be more specific? I would expect something like
tags: - centos-stream-8-vm
What's the point, we'd have to constantly refresh the tags if the platforms come and go given our support, whereas fedora-vm and centos-stream-vm cover all currently supported versions - always! Other than that, I'm not sure that tags are passed on to the gitlab job itself, I may have missed it, but unless the tags are exposed as env variables, the provisioner script wouldn't know which template to provision. Also, the tag is supposed to annotate the baremetal host in this case, so in that context having '-vm' in the tag name makes sense, but doesn't for the provisioner script which relies on/tries to be compatible with lcitool as much as possible.
Okay, my misunderstanding was caused by not figuring out the purpose of DISTRO. I agree that more specific tags are not necessary.
Should we make them *less* specific instead? As in, is there any reason for having different tags for Fedora and CentOS jobs as opposed to using a generic "this needs to run in a VM" tag for both?
Well, I would not be against, but I feel this is more of a political issue: this HW was provided by Red Hat with the intention to be dedicated for Red Hat workloads. If another interested 3rd party comes (and I do hope they will) and provides HW, we should utilize the resources fairly in a way respectful to the donor's/owner's intentions, IOW if party A provides a single machine to run CI workloads using Debian VMs, we should not schedule Fedora/CentOS workloads in there effectively saturating it. So if the tags are to be adjusted, then I'd be in favour of recording the owner of the runner in the tag. Erik
On Wed, Mar 02, 2022 at 08:42:30AM +0100, Erik Skultety wrote:
...
+libvirt-perl-bindings: + stage: bindings + trigger: + project: eskultety/libvirt-perl + branch: multi-project-ci + strategy: depend + + +centos-stream-8-tests: + extends: .tests + needs: + - libvirt-perl-bindings + - pipeline: $PARENT_PIPELINE_ID + job: x86_64-centos-stream-8 + - project: eskultety/libvirt-perl + job: x86_64-centos-stream-8 + ref: multi-project-ci + artifacts: true
IIUC from the documentation and from reading around, using strategy:depend will cause the local job to reflect the status of the triggered pipeline. So far so good.
What I am unclear about is, is there any guarantee that using artifact:true with a job from an external project's pipeline will expose the artifacts from the job that was executed as part of the specific pipeline that we've just triggered ourselves as opposed to some other pipeline that might have already been completed in the past of might have completed in the meantime?
Not just by using artifact:true or strategy:depend. The important bit is having 'libvirt-perl-bindings' in the job's needs list. Let me explain, if you don't put the bindings trigger job to the requirements list of another job (centos-stream-8-tests in this case) what will happen is that the trigger job will be waiting for the external pipeline to finish, but centos-stream-8-tests job would execute basically as soon as the container project builds are finished because artifacts:true would download the latest RPM artifacts from an earlier build...
No, I got that part. My question was whether
other-project-pipeline: trigger: project: other-project strategy: depend
our-job: needs: - other-project-pipeline - project: other-project job: other-project-job artifacts: true
actually guarantees that the instance of other-project-job whose artifacts are available to our-job is the same one that was started as part of the pipeline triggered by the other-project-pipeline job.
Sorry for a delayed response.
I don't think so. We can basically only rely on a fact that the jobs would actually be queued in order they arrive which means that jobs submitted earlier should finish earlier, but that of course is only a premise not a guarantee.
On the other hand I never intended to run the integration CI on every single push to the master branch, instead, I wanted to make this a scheduled pipeline which would effectively alleviate the problem, because with scheduled pipelines there would very likely not be a concurrent pipeline running in libvirt-perl which would make us download artifacts from a pipeline we didn't trigger ourselves.
Ultimately when we switch to using merge requests, the integration tests should be run as a gating job, triggered from the merge train when the code gets applied to git, so that we prevent regressions actually making it into git master at all. Post-merge integration testing always exhibits the problem that people will consider it somebody else's problem to fix the regression. So effectively whoever creates the integration testing system ends up burdened with the job of investigating failures and finding someone to poke to fix it. With it run pre-merge then whoever wants to get their code merged needs to investigate the problems. Now sometimes the problems with of course be with the integration test system itself, not the submitters code, but this is OK because it leads to situation where the job of maintaining the integration tests are more equitably spread across all involved and builds a mindset that functional / integration testing is a critical part of delivering code, which is something we've lacked for too long in libvirt.
Taking a step back, why exactly are we triggering a rebuild of libvirt-perl in the first place? Once we change that project's pipeline so that RPMs are published as artifacts, can't we just grab the ones from the latest successful pipeline? Maybe you've already explained why you did things this way and I just missed it!
...which brings us here. Well, I adopted the mantra that all libvirt-friends projects depend on libvirt and given that we need libvirt-perl bindings to test upstream, I'd like to always have the latest bindings available to test with the current upstream build. The other reason why I did the way you commented on is that during development of the proposal many times I had to make changes to both libvirt and libvirt-perl in lockstep and it was tremendously frustrating to wait for the pipeline to get to the integration stage only to realize that the integration job didn't wait for the latest bindings and instead picked up the previous latest artifacts which I knew were either faulty or didn't contain the necessary changes yet.
Of course that would be annoying when you're making changes to both projects at the same time, but is that a scenario that we can expect to be common once the integration tests are in place?
To be clear, I'm not necessarily against the way you're doing things right now, it's just that it feels like using the artifacts from the latest successful libvirt-perl pipeline would lower complexity, avoid burning additional resources and reduce wait times.
If the only only downside is having a worse experience when making changes to the pipeline, and we can expect that to be infrequent enough, perhaps that's a reasonable tradeoff.
I gave this more thought. What you suggest is viable, but the following is worth considering if we go with your proposal:
- libvirt-perl jobs build upstream libvirt first in order to build the bindings -> generally it takes until right before the release that APIs/constants are added to the respective bindings (Perl/Python) -> if we rely on the latest libvirt-perl artifacts without actually triggering the pipeline, yes, the artifacts would be stable, but fairly old (unless we schedule a recurrent pipeline in the project to refresh them), thus not giving us feedback from the integration stage that bindings need to be added first, because the API coverage would likely fail, thus failing the whole libvirt-perl pipeline and thus invalidating the integration test stage in the libvirt project => now, I admit this would get pretty annoying because it would force contributors (or the maintainer) who add new APIs to add respective bindings as well in a timely manner, but then again ultimately we'd like our contributors to also introduce an integration test along with their feature...
Note right now the perl API coverage tests are configured to only be gating when run on nightly scheduled jobs. I stopped them being gating on contributions because if someone if fixing a bug in the bindings it is silly to force their merge request to also add new API bindings. I'm thinking about whether we should even making the API coverage tests be non-gating even for scheduled jobs. I miss the fact that we when we see a notification of a failed pipeline we don't see at a glance whether it is a genuine build failure or merely a new API missing. The python bindings have a little different of a situation. Sometimes the code generator can do the job on its own, but other times the code generator trips over its cane and breaks a leg. In the latter cases, we're always going to get a hard CI failure we can't ignore, unless we teach the code generator to make it a soft failure and just skip the API with a warning when it is something it can't cope with. I think if we're going to use the python bindings from automated tests we'll have no choice but to make the code generator treat it as a soft failure, if we want to use the tests as a gating check, otherwise you'll end up with a chicken & egg problem between merging new APIs to C lib and Python.
What's the point, we'd have to constantly refresh the tags if the platforms come and go given our support, whereas fedora-vm and centos-stream-vm cover all currently supported versions - always! Other than that, I'm not sure that tags are passed on to the gitlab job itself, I may have missed it, but unless the tags are exposed as env variables, the provisioner script wouldn't know which template to provision. Also, the tag is supposed to annotate the baremetal host in this case, so in that context having '-vm' in the tag name makes sense, but doesn't for the provisioner script which relies on/tries to be compatible with lcitool as much as possible.
Okay, my misunderstanding was caused by not figuring out the purpose of DISTRO. I agree that more specific tags are not necessary.
Should we make them *less* specific instead? As in, is there any reason for having different tags for Fedora and CentOS jobs as opposed to using a generic "this needs to run in a VM" tag for both?
Well, I would not be against, but I feel this is more of a political issue: this HW was provided by Red Hat with the intention to be dedicated for Red Hat workloads. If another interested 3rd party comes (and I do hope they will) and provides HW, we should utilize the resources fairly in a way respectful to the donor's/owner's intentions, IOW if party A provides a single machine to run CI workloads using Debian VMs, we should not schedule Fedora/CentOS workloads in there effectively saturating it. So if the tags are to be adjusted, then I'd be in favour of recording the owner of the runner in the tag.
If we have hardware available, we should use to the best of its ability. Nothing is gained by leaving it idle if it has spare capacity to run jobs. Until we start using it though, we will not have a clear idea of how many distro combinations we can cope with for integration testing. We'll also want to see how stable the jobs prove to be as we start using it for real. With that in mind it makes sense to start off with a limit number of distro jobs and monitor the situation. If it is reliable and the machine shows it has capacity to run more then we can add more, picking distros that give the maximum benefit in terms of identifying bugs. IOW, I would much rather run 1x CentOS Stream + 1x Fedora + 1x Debian + 1x Suse, than 2 x CentOS and 2x Fedora, because the former will give much broader ability to find bugs. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, Mar 02, 2022 at 09:43:24AM +0000, Daniel P. Berrangé wrote: ...
No, I got that part. My question was whether
other-project-pipeline: trigger: project: other-project strategy: depend
our-job: needs: - other-project-pipeline - project: other-project job: other-project-job artifacts: true
actually guarantees that the instance of other-project-job whose artifacts are available to our-job is the same one that was started as part of the pipeline triggered by the other-project-pipeline job.
Sorry for a delayed response.
I don't think so. We can basically only rely on a fact that the jobs would actually be queued in order they arrive which means that jobs submitted earlier should finish earlier, but that of course is only a premise not a guarantee.
On the other hand I never intended to run the integration CI on every single push to the master branch, instead, I wanted to make this a scheduled pipeline which would effectively alleviate the problem, because with scheduled pipelines there would very likely not be a concurrent pipeline running in libvirt-perl which would make us download artifacts from a pipeline we didn't trigger ourselves.
Ultimately when we switch to using merge requests, the integration tests should be run as a gating job, triggered from the merge train when the code gets applied to git, so that we prevent regressions actually making it into git master at all.
Post-merge integration testing always exhibits the problem that people will consider it somebody else's problem to fix the regression. So effectively whoever creates the integration testing system ends up burdened with the job of investigating failures and finding someone to poke to fix it. With it run pre-merge then whoever wants to get their code merged needs to investigate the problems. Now sometimes the problems with of course be with the integration test system itself, not the submitters code, but this is OK because it leads to situation where the job of maintaining the integration tests are more equitably spread across all involved and builds a mindset that functional / integration testing is a critical part of delivering code, which is something we've lacked for too long in libvirt.
Agreed.
Taking a step back, why exactly are we triggering a rebuild of libvirt-perl in the first place? Once we change that project's pipeline so that RPMs are published as artifacts, can't we just grab the ones from the latest successful pipeline? Maybe you've already explained why you did things this way and I just missed it!
...which brings us here. Well, I adopted the mantra that all libvirt-friends projects depend on libvirt and given that we need libvirt-perl bindings to test upstream, I'd like to always have the latest bindings available to test with the current upstream build. The other reason why I did the way you commented on is that during development of the proposal many times I had to make changes to both libvirt and libvirt-perl in lockstep and it was tremendously frustrating to wait for the pipeline to get to the integration stage only to realize that the integration job didn't wait for the latest bindings and instead picked up the previous latest artifacts which I knew were either faulty or didn't contain the necessary changes yet.
Of course that would be annoying when you're making changes to both projects at the same time, but is that a scenario that we can expect to be common once the integration tests are in place?
To be clear, I'm not necessarily against the way you're doing things right now, it's just that it feels like using the artifacts from the latest successful libvirt-perl pipeline would lower complexity, avoid burning additional resources and reduce wait times.
If the only only downside is having a worse experience when making changes to the pipeline, and we can expect that to be infrequent enough, perhaps that's a reasonable tradeoff.
I gave this more thought. What you suggest is viable, but the following is worth considering if we go with your proposal:
- libvirt-perl jobs build upstream libvirt first in order to build the bindings -> generally it takes until right before the release that APIs/constants are added to the respective bindings (Perl/Python) -> if we rely on the latest libvirt-perl artifacts without actually triggering the pipeline, yes, the artifacts would be stable, but fairly old (unless we schedule a recurrent pipeline in the project to refresh them), thus not giving us feedback from the integration stage that bindings need to be added first, because the API coverage would likely fail, thus failing the whole libvirt-perl pipeline and thus invalidating the integration test stage in the libvirt project => now, I admit this would get pretty annoying because it would force contributors (or the maintainer) who add new APIs to add respective bindings as well in a timely manner, but then again ultimately we'd like our contributors to also introduce an integration test along with their feature...
Note right now the perl API coverage tests are configured to only be gating when run on nightly scheduled jobs. I stopped them being gating on contributions because if someone if fixing a bug in the bindings it is silly to force their merge request to also add new API bindings.
I'm thinking about whether we should even making the API coverage tests be non-gating even for scheduled jobs. I miss the fact that we when we see a notification of a failed pipeline we don't see at a glance whether it is a genuine build failure or merely a new API missing.
T think ^this could be done with an external dashboard monitoring gitlab CI pipelines, providing all relevant information: artifacts, job that failed the pipeline (as long as coverage is a separate job). Anyhow, I believe that's a topic for another day :).
The python bindings have a little different of a situation. Sometimes the code generator can do the job on its own, but other times the code generator trips over its cane and breaks a leg. In the latter cases, we're always going to get a hard CI failure we can't ignore, unless we teach the code generator to make it a soft failure and just skip the API with a warning when it is something it can't cope with. I think if we're going to use the python bindings from automated tests we'll have no choice but to make the code generator treat it as a soft failure, if we want to use the tests as a gating check, otherwise you'll end up with a chicken & egg problem between merging new APIs to C lib and Python.
^This one's going to be "fun" though. Once we merge this integration CI prototype I can work on improving the situation with Python bindings, we'll need them anyway.
What's the point, we'd have to constantly refresh the tags if the platforms come and go given our support, whereas fedora-vm and centos-stream-vm cover all currently supported versions - always! Other than that, I'm not sure that tags are passed on to the gitlab job itself, I may have missed it, but unless the tags are exposed as env variables, the provisioner script wouldn't know which template to provision. Also, the tag is supposed to annotate the baremetal host in this case, so in that context having '-vm' in the tag name makes sense, but doesn't for the provisioner script which relies on/tries to be compatible with lcitool as much as possible.
Okay, my misunderstanding was caused by not figuring out the purpose of DISTRO. I agree that more specific tags are not necessary.
Should we make them *less* specific instead? As in, is there any reason for having different tags for Fedora and CentOS jobs as opposed to using a generic "this needs to run in a VM" tag for both?
Well, I would not be against, but I feel this is more of a political issue: this HW was provided by Red Hat with the intention to be dedicated for Red Hat workloads. If another interested 3rd party comes (and I do hope they will) and provides HW, we should utilize the resources fairly in a way respectful to the donor's/owner's intentions, IOW if party A provides a single machine to run CI workloads using Debian VMs, we should not schedule Fedora/CentOS workloads in there effectively saturating it. So if the tags are to be adjusted, then I'd be in favour of recording the owner of the runner in the tag.
If we have hardware available, we should use to the best of its ability. Nothing is gained by leaving it idle if it has spare capacity to run jobs.
Well, logically there's absolutely no disagreement with you here. Personally, I would go about it the same. The problem is that the HW we're talking about wasn't an official donation, Red Hat still owns and controls the HW, so the company can very much disagree with running other workloads on it long term. I'm not saying we shouldn't test the limits, reliability and bandwidth to its fullest potential. What I'm trying to say is that the major issue here is that contributing to open source projects is a collaborative effort of all interested parties (duh, should go without saying) and so we cannot expect a single party which just happens to have the biggest stake in the project to run workloads for everybody else. I mean the situation would have been different if the HW were a proper donation, but unfortunately it is not. If we pick and run workloads on various distros for the sake of getting coverage (which makes total sense btw), it would later be harder to communicate back to the community why the number of distros (or their variety) would need to be decreased once the HW's capabilities are saturated with demanding workloads, e.g. migration testing or device assignment, etc. Whether I do or do not personally feel comfortable being involved in ^this political situation and decision making, as a contributor using Red Hat's email domain I do respect the company's intentions with regards to the offered HW. I think that the final setup we agree on eventually is up for an internal debate and doesn't have a direct impact on this proposal per-se.
Until we start using it though, we will not have a clear idea of how many distro combinations we can cope with for integration testing. We'll also want to see how stable the jobs prove to be as we start using it for real. With that in mind it makes sense to start off with a limit number of distro jobs and monitor the situation. If it is reliable and the machine shows it has capacity to run more then we can add more, picking distros that give
Yes, the number of machines will rise once we progress with the test suite and enable migration tests which is something I haven't polished and tested yet in libvirt-tck.
the maximum benefit in terms of identifying bugs. IOW, I would much rather run 1x CentOS Stream + 1x Fedora + 1x Debian + 1x Suse, than 2 x CentOS and 2x Fedora, because the former will give much broader ability to find bugs.
Regards, Erik
On Wed, Mar 02, 2022 at 01:11:04PM +0100, Erik Skultety wrote:
I gave this more thought. What you suggest is viable, but the following is worth considering if we go with your proposal:
- libvirt-perl jobs build upstream libvirt first in order to build the bindings -> generally it takes until right before the release that APIs/constants are added to the respective bindings (Perl/Python)
This is not entirely accurate. While it's true that bindings generally lag behind the C library, they're usually updated within days. Changes only getting in at the very end of a development cycle is an exception, not the norm.
-> if we rely on the latest libvirt-perl artifacts without actually triggering the pipeline, yes, the artifacts would be stable, but fairly old (unless we schedule a recurrent pipeline in the project to refresh them), thus not giving us feedback from the integration stage that bindings need to be added first, because the API coverage would likely fail, thus failing the whole libvirt-perl pipeline and thus invalidating the integration test stage in the libvirt project => now, I admit this would get pretty annoying because it would force contributors (or the maintainer) who add new APIs to add respective bindings as well in a timely manner, but then again ultimately we'd like our contributors to also introduce an integration test along with their feature...
Note right now the perl API coverage tests are configured to only be gating when run on nightly scheduled jobs. I stopped them being gating on contributions because if someone if fixing a bug in the bindings it is silly to force their merge request to also add new API bindings.
I don't think we can expect integration tests to be merged at the same time as a feature when new APIs are involved. If tests are written in Python, then the Python bindings need to introduce support for the new API before the test can exist, and that can't happen until the feature has been merged. If the feature or bug fix doesn't require new APIs to be introduced this is of course not an issue. Most changes should fall into this bucket. So overall I still think using existing artifacts would be the better approach, at least initially. We can always change things later if we find that we've outgrown it.
Should we make them *less* specific instead? As in, is there any reason for having different tags for Fedora and CentOS jobs as opposed to using a generic "this needs to run in a VM" tag for both?
Well, I would not be against, but I feel this is more of a political issue: this HW was provided by Red Hat with the intention to be dedicated for Red Hat workloads. If another interested 3rd party comes (and I do hope they will) and provides HW, we should utilize the resources fairly in a way respectful to the donor's/owner's intentions, IOW if party A provides a single machine to run CI workloads using Debian VMs, we should not schedule Fedora/CentOS workloads in there effectively saturating it. So if the tags are to be adjusted, then I'd be in favour of recording the owner of the runner in the tag.
If we have hardware available, we should use to the best of its ability. Nothing is gained by leaving it idle if it has spare capacity to run jobs.
Well, logically there's absolutely no disagreement with you here. Personally, I would go about it the same. The problem is that the HW we're talking about wasn't an official donation, Red Hat still owns and controls the HW, so the company can very much disagree with running other workloads on it long term. I'm not saying we shouldn't test the limits, reliability and bandwidth to its fullest potential. What I'm trying to say is that the major issue here is that contributing to open source projects is a collaborative effort of all interested parties (duh, should go without saying) and so we cannot expect a single party which just happens to have the biggest stake in the project to run workloads for everybody else. I mean the situation would have been different if the HW were a proper donation, but unfortunately it is not. If we pick and run workloads on various distros for the sake of getting coverage (which makes total sense btw), it would later be harder to communicate back to the community why the number of distros (or their variety) would need to be decreased once the HW's capabilities are saturated with demanding workloads, e.g. migration testing or device assignment, etc.
Whether I do or do not personally feel comfortable being involved in ^this political situation and decision making, as a contributor using Red Hat's email domain I do respect the company's intentions with regards to the offered HW. I think that the final setup we agree on eventually is up for an internal debate and doesn't have a direct impact on this proposal per-se.
I think it's not unreasonable that when Red Hat, or any other entity, provides hardware access to the project there will be some strings attached. This is already the case for GitLab and Cirrus CI, for example. I could easily see the instance of libvirt-gitlab-executor running on hardware owned and operated by Red Hat returning a failure if a job submitted to it comes with DISTRO=debian-11. For the time being, I think it'd be okay to use a tag like redhat-vm-host or something like that for these jobs. Once we have had jobs running for a bit, we can figure out whether there is spare capacity available and try to convince Red Hat to let more jobs run on the machines. -- Andrea Bolognani / Red Hat / Virtualization
On Wed, Mar 02, 2022 at 06:27:04AM -0800, Andrea Bolognani wrote:
On Wed, Mar 02, 2022 at 01:11:04PM +0100, Erik Skultety wrote:
I gave this more thought. What you suggest is viable, but the following is worth considering if we go with your proposal:
- libvirt-perl jobs build upstream libvirt first in order to build the bindings -> generally it takes until right before the release that APIs/constants are added to the respective bindings (Perl/Python)
This is not entirely accurate. While it's true that bindings generally lag behind the C library, they're usually updated within days. Changes only getting in at the very end of a development cycle is an exception, not the norm.
-> if we rely on the latest libvirt-perl artifacts without actually triggering the pipeline, yes, the artifacts would be stable, but fairly old (unless we schedule a recurrent pipeline in the project to refresh them), thus not giving us feedback from the integration stage that bindings need to be added first, because the API coverage would likely fail, thus failing the whole libvirt-perl pipeline and thus invalidating the integration test stage in the libvirt project => now, I admit this would get pretty annoying because it would force contributors (or the maintainer) who add new APIs to add respective bindings as well in a timely manner, but then again ultimately we'd like our contributors to also introduce an integration test along with their feature...
Note right now the perl API coverage tests are configured to only be gating when run on nightly scheduled jobs. I stopped them being gating on contributions because if someone if fixing a bug in the bindings it is silly to force their merge request to also add new API bindings.
I don't think we can expect integration tests to be merged at the same time as a feature when new APIs are involved. If tests are written in Python, then the Python bindings need to introduce support for the new API before the test can exist, and that can't happen until the feature has been merged.
Again, that is a logical conclusion which brings us to an unrelated process question: How do we change the contribution process so that the contribution of a feature doesn't end with it being merged to the C library, IOW we'd ideally want to have a test introduced with every feature,but given that we'd need the bindings first to actually do that, but we can't have a binding unless the C counterpart is already merged, how do we keep the contributors motivated enough? (Note that it's food for thought, it's only tangential to this effort)
If the feature or bug fix doesn't require new APIs to be introduced this is of course not an issue. Most changes should fall into this bucket.
So overall I still think using existing artifacts would be the better approach, at least initially. We can always change things later if we find that we've outgrown it.
So, given that https://gitlab.com/libvirt/libvirt-perl/-/merge_requests/55 was already merged we should not get to a situation where no artifacts would be available because gitlab documents that even if artifacts expired they won't be deleted until new artifacts become available, I think we can depend on the latest available artifacts without building them. I'll refresh the patch accordingly and test.
Should we make them *less* specific instead? As in, is there any reason for having different tags for Fedora and CentOS jobs as opposed to using a generic "this needs to run in a VM" tag for both?
Well, I would not be against, but I feel this is more of a political issue: this HW was provided by Red Hat with the intention to be dedicated for Red Hat workloads. If another interested 3rd party comes (and I do hope they will) and provides HW, we should utilize the resources fairly in a way respectful to the donor's/owner's intentions, IOW if party A provides a single machine to run CI workloads using Debian VMs, we should not schedule Fedora/CentOS workloads in there effectively saturating it. So if the tags are to be adjusted, then I'd be in favour of recording the owner of the runner in the tag.
If we have hardware available, we should use to the best of its ability. Nothing is gained by leaving it idle if it has spare capacity to run jobs.
Well, logically there's absolutely no disagreement with you here. Personally, I would go about it the same. The problem is that the HW we're talking about wasn't an official donation, Red Hat still owns and controls the HW, so the company can very much disagree with running other workloads on it long term. I'm not saying we shouldn't test the limits, reliability and bandwidth to its fullest potential. What I'm trying to say is that the major issue here is that contributing to open source projects is a collaborative effort of all interested parties (duh, should go without saying) and so we cannot expect a single party which just happens to have the biggest stake in the project to run workloads for everybody else. I mean the situation would have been different if the HW were a proper donation, but unfortunately it is not. If we pick and run workloads on various distros for the sake of getting coverage (which makes total sense btw), it would later be harder to communicate back to the community why the number of distros (or their variety) would need to be decreased once the HW's capabilities are saturated with demanding workloads, e.g. migration testing or device assignment, etc.
Whether I do or do not personally feel comfortable being involved in ^this political situation and decision making, as a contributor using Red Hat's email domain I do respect the company's intentions with regards to the offered HW. I think that the final setup we agree on eventually is up for an internal debate and doesn't have a direct impact on this proposal per-se.
I think it's not unreasonable that when Red Hat, or any other entity, provides hardware access to the project there will be some strings attached. This is already the case for GitLab and Cirrus CI, for example.
I could easily see the instance of libvirt-gitlab-executor running on hardware owned and operated by Red Hat returning a failure if a job submitted to it comes with DISTRO=debian-11.
libvirt-gitlab-executor is supposed to be system owner agnostic, I'd even like to make the project part of the libvirt gitlab namespace umbrella so that anyone can use it to prepare their machine to be plugged into and used for libvirt upstream integration testing. Therefore, I don't think the project is the right place for such checks, those should IMO live solely in the gitlab-ci.yml configuration. Erik
For the time being, I think it'd be okay to use a tag like redhat-vm-host or something like that for these jobs. Once we have had jobs running for a bit, we can figure out whether there is spare capacity available and try to convince Red Hat to let more jobs run on the machines.
-- Andrea Bolognani / Red Hat / Virtualization
On Wed, Mar 02, 2022 at 03:52:49PM +0100, Erik Skultety wrote:
On Wed, Mar 02, 2022 at 06:27:04AM -0800, Andrea Bolognani wrote:
I don't think we can expect integration tests to be merged at the same time as a feature when new APIs are involved. If tests are written in Python, then the Python bindings need to introduce support for the new API before the test can exist, and that can't happen until the feature has been merged.
Again, that is a logical conclusion which brings us to an unrelated process question: How do we change the contribution process so that the contribution of a feature doesn't end with it being merged to the C library, IOW we'd ideally want to have a test introduced with every feature,but given that we'd need the bindings first to actually do that, but we can't have a binding unless the C counterpart is already merged, how do we keep the contributors motivated enough? (Note that it's food for thought, it's only tangential to this effort)
Yeah, let's save this massive topic for another time :)
If the feature or bug fix doesn't require new APIs to be introduced this is of course not an issue. Most changes should fall into this bucket.
So overall I still think using existing artifacts would be the better approach, at least initially. We can always change things later if we find that we've outgrown it.
So, given that https://gitlab.com/libvirt/libvirt-perl/-/merge_requests/55 was already merged we should not get to a situation where no artifacts would be available because gitlab documents that even if artifacts expired they won't be deleted until new artifacts become available, I think we can depend on the latest available artifacts without building them. I'll refresh the patch accordingly and test.
Thanks!
I think it's not unreasonable that when Red Hat, or any other entity, provides hardware access to the project there will be some strings attached. This is already the case for GitLab and Cirrus CI, for example.
I could easily see the instance of libvirt-gitlab-executor running on hardware owned and operated by Red Hat returning a failure if a job submitted to it comes with DISTRO=debian-11.
libvirt-gitlab-executor is supposed to be system owner agnostic, I'd even like to make the project part of the libvirt gitlab namespace umbrella so that anyone can use it to prepare their machine to be plugged into and used for libvirt upstream integration testing.
I absolutely agree and it was always my understanding that the project living under your personal account was only a temporary measure.
Therefore, I don't think the project is the right place for such checks, those should IMO live solely in the gitlab-ci.yml configuration.
To be clear, I didn't mean that the software itself should contain hardcoded limits on what jobs it will process, but rather that it would make sense for it to offer a configuration setting along the lines of accept_distros: - "alpine-*" - "debian-*" that can be used by the local sysadmin to define such a policy. I guess this is already sort of already implicitly implemented due to the fact that a job will fail if the corresponding VM template does not exist... -- Andrea Bolognani / Red Hat / Virtualization
...
I could easily see the instance of libvirt-gitlab-executor running on hardware owned and operated by Red Hat returning a failure if a job submitted to it comes with DISTRO=debian-11.
libvirt-gitlab-executor is supposed to be system owner agnostic, I'd even like to make the project part of the libvirt gitlab namespace umbrella so that anyone can use it to prepare their machine to be plugged into and used for libvirt upstream integration testing.
I absolutely agree and it was always my understanding that the project living under your personal account was only a temporary measure.
Therefore, I don't think the project is the right place for such checks, those should IMO live solely in the gitlab-ci.yml configuration.
To be clear, I didn't mean that the software itself should contain hardcoded limits on what jobs it will process, but rather that it would make sense for it to offer a configuration setting along the lines of
accept_distros: - "alpine-*" - "debian-*"
It's surely up for a discussion, I haven't made any hard decisions wrt where should the project be headed, right now it's very simple, let's see :).
that can be used by the local sysadmin to define such a policy.
true, but I suppose from upstream's perspective it can already be handled using gitlab tags only, so it feels redundant to handle the same on multiple places.
I guess this is already sort of already implicitly implemented due to the fact that a job will fail if the corresponding VM template does not exist...
Yes, it'll throw an error, possibly an ugly exception (I hope not) when this happens. Erik
On Wed, Mar 02, 2022 at 05:22:07PM +0100, Erik Skultety wrote:
that can be used by the local sysadmin to define such a policy.
true, but I suppose from upstream's perspective it can already be handled using gitlab tags only, so it feels redundant to handle the same on multiple places.
Right, but you have to keep in mind that the person working on the project might not be the same one managing the machines, and that the latter shouldn't have to rely on the former getting the configuration just right to ensure that their desired usage policy is enforced. Configuration mistakes on the project's side should result in failed jobs, not usage policy violations.
I guess this is already sort of already implicitly implemented due to the fact that a job will fail if the corresponding VM template does not exist...
Yes, it'll throw an error, possibly an ugly exception (I hope not) when this happens.
Perhaps all we really need to do is to make sure that a nice, explanatory error message is produced in this scenario then :) -- Andrea Bolognani / Red Hat / Virtualization
On Wed, Mar 02, 2022 at 09:47:13AM -0800, Andrea Bolognani wrote:
On Wed, Mar 02, 2022 at 05:22:07PM +0100, Erik Skultety wrote:
that can be used by the local sysadmin to define such a policy.
true, but I suppose from upstream's perspective it can already be handled using gitlab tags only, so it feels redundant to handle the same on multiple places.
Right, but you have to keep in mind that the person working on the project might not be the same one managing the machines, and that the latter shouldn't have to rely on the former getting the configuration just right to ensure that their desired usage policy is enforced. Configuration mistakes on the project's side should result in failed jobs, not usage policy violations.
Fair enough, I can introduce some configuration options to take care of this. Erik
On Wed, Mar 02, 2022 at 09:43:24 +0000, Daniel P. Berrangé wrote:
On Wed, Mar 02, 2022 at 08:42:30AM +0100, Erik Skultety wrote:
... Ultimately when we switch to using merge requests, the integration tests should be run as a gating job, triggered from the merge train when the code gets applied to git, so that we prevent regressions actually making it into git master at all.
Post-merge integration testing always exhibits the problem that people will consider it somebody else's problem to fix the regression. So effectively whoever creates the integration testing system ends up burdened with the job of investigating failures and finding someone to poke to fix it. With it run pre-merge then whoever wants to get their code merged needs to investigate the problems. Now sometimes the problems with of course be with the integration test system itself, not the submitters code, but this is OK because it leads to situation where the job of maintaining the integration tests are more equitably spread across all involved and builds a mindset that functional / integration testing is a critical part of delivering code, which is something we've lacked for too long in libvirt.
This is all fine as long as there's a way to explicitly merge something even if CI fails. I don't like waiting for a fix in something completely unrelated. For example, a MR with translations or an important bug fix should not be blocked by a bug in CI or an infrastructure issue. So having a way (restricted, of course) to merge even if pipeline failed would solve this issue. Jirka
On Wed, Mar 02, 2022 at 04:42:21PM +0100, Jiri Denemark wrote:
On Wed, Mar 02, 2022 at 09:43:24 +0000, Daniel P. Berrangé wrote:
On Wed, Mar 02, 2022 at 08:42:30AM +0100, Erik Skultety wrote:
... Ultimately when we switch to using merge requests, the integration tests should be run as a gating job, triggered from the merge train when the code gets applied to git, so that we prevent regressions actually making it into git master at all.
Post-merge integration testing always exhibits the problem that people will consider it somebody else's problem to fix the regression. So effectively whoever creates the integration testing system ends up burdened with the job of investigating failures and finding someone to poke to fix it. With it run pre-merge then whoever wants to get their code merged needs to investigate the problems. Now sometimes the problems with of course be with the integration test system itself, not the submitters code, but this is OK because it leads to situation where the job of maintaining the integration tests are more equitably spread across all involved and builds a mindset that functional / integration testing is a critical part of delivering code, which is something we've lacked for too long in libvirt.
This is all fine as long as there's a way to explicitly merge something even if CI fails. I don't like waiting for a fix in something completely unrelated. For example, a MR with translations or an important bug fix
I haven't studied GitLab enough to rule it out straight away, but I'd like to have some more intelligent hook (or whatnot) in place that could detect what type of change is proposed so that documentation changes and translations would not go through the whole pipeline and instead would be merged almost immediately (after passing e.g. the docs build). An important bug fix however must go through the whole pipeline IMO whether we like it or not, that's the point, we need to ensure that we're not introducing a regression to the best of our capabilities. As for CI infrastructure issues, well, unfortunately that is an inevitable part of doing any automated tasks. By introducing it upstream it should therefore become a collective responsibility, so that if a CI issue occurs we need to work together to resolve it ASAP rather than going around it and pushing a fix just because it takes long to execute. Erik
should not be blocked by a bug in CI or an infrastructure issue. So having a way (restricted, of course) to merge even if pipeline failed would solve this issue.
Jirka
On Wed, Mar 02, 2022 at 05:18:26PM +0100, Erik Skultety wrote:
On Wed, Mar 02, 2022 at 04:42:21PM +0100, Jiri Denemark wrote:
On Wed, Mar 02, 2022 at 09:43:24 +0000, Daniel P. Berrangé wrote:
On Wed, Mar 02, 2022 at 08:42:30AM +0100, Erik Skultety wrote:
... Ultimately when we switch to using merge requests, the integration tests should be run as a gating job, triggered from the merge train when the code gets applied to git, so that we prevent regressions actually making it into git master at all.
Post-merge integration testing always exhibits the problem that people will consider it somebody else's problem to fix the regression. So effectively whoever creates the integration testing system ends up burdened with the job of investigating failures and finding someone to poke to fix it. With it run pre-merge then whoever wants to get their code merged needs to investigate the problems. Now sometimes the problems with of course be with the integration test system itself, not the submitters code, but this is OK because it leads to situation where the job of maintaining the integration tests are more equitably spread across all involved and builds a mindset that functional / integration testing is a critical part of delivering code, which is something we've lacked for too long in libvirt.
This is all fine as long as there's a way to explicitly merge something even if CI fails. I don't like waiting for a fix in something completely unrelated. For example, a MR with translations or an important bug fix
I haven't studied GitLab enough to rule it out straight away, but I'd like to have some more intelligent hook (or whatnot) in place that could detect what type of change is proposed so that documentation changes and translations would not go through the whole pipeline and instead would be merged almost immediately (after passing e.g. the docs build).
Yep, for translations all we really need todo is trigger the msgmerge commands to validate that printf format specifiers aren't messed up, so that one is quite easy. In theory we could do something similar for docs only changes, but not sure how amenable our meson rules are to a our meson rules dont For docs our 'website' gitlab job only builds the HTML docs skipping the code. So we could potentially filter that too
An important bug fix however must go through the whole pipeline IMO whether we like it or not, that's the point, we need to ensure that we're not introducing a regression to the best of our capabilities. As for CI infrastructure issues, well, unfortunately that is an inevitable part of doing any automated tasks. By introducing it upstream it should therefore become a collective responsibility, so that if a CI issue occurs we need to work together to resolve it ASAP rather than going around it and pushing a fix just because it takes long to execute.
I think the idea of skipping an unreliable pipeline is the wrong way to approach it. For something to be gating the expectation has to be that it is reliable. If something is reliable and it fails with a genuine problem, then it is collective responsibility to drop everything and focus on fixing that problem, not skip it. Sometimes this may cause a delay in merging stuff but that is a good thing, as we've certainly had enough cases over the years where obvious trivial fixes were pushed but were in fact broken. If something breaks and the fix is going to be difficult or out of our control, then we might have no choice but to disable the test jobs completely. For this it should be a case of setting an env var in the gitlab config to skip certain job names, which we can unset once the infra is fixed. An example of this would be where the Cirrus CI API changed recently and completely broke cirrus-run and took a week to fix, and I unset the CIRRUS CI token in the repo. In the case that some jobs are not consistently not reliable then there are a few relevant actions to consider. For example we have ultimately decided to make the all the bleeding edge distros be non-gating because Rawhide/Sid/Tumbleweed regularly broke due to their maintainers pushing broken packages. If we have a test of our own that is periodically failing in a non-determinstic way, then we should immediately disable that test in the test suite, rather than skipping gating CI. The test should only be re-enabled once its reliability is fixed. Tests which are gating must be reliable without false positives. In the absolute worst case, someone with admin permissions can just toggle the 'pipelines must succeed' option in the gitlab repo admin UI, but if it gets to that point we've really failed badly. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Feb 17, 2022 at 09:54:12AM -0800, Andrea Bolognani wrote:
On Mon, Jan 31, 2022 at 07:01:01PM +0100, Erik Skultety wrote:
+++ b/.gitlab-ci-integration.yml @@ -0,0 +1,116 @@ +.tests: + stage: integration + before_script: + - mkdir "$SCRATCH_DIR" + - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/* + - sudo pip3 install --prefix=/usr avocado-framework + - source /etc/os-release # in order to query the vendor-provided variables + - if test "$ID" == "centos" && test "$VERSION_ID" -lt 9 || + test "$ID" == "fedora" && test "$VERSION_ID" -lt 35;
Using == with test is a bashism, please stick to the portable version even though it's very likely that the script will ultimately run under bash.
+ - for daemon in $DAEMONS; + do + sudo sed -Ei "s/^(#)?(log_outputs=).*/\2'1:file:\/var\/log\/libvirt\/${daemon}.log'/" /etc/libvirt/${daemon}.conf; + sudo sed -Ei "s/^(#)?(log_filters=).*/\2'4:*object* 4:*json* 4:*event* 4:*rpc* 4:daemon.remote 4:util.threadjob 4:*access* 1:*'/" /etc/libvirt/${daemon}.conf; + sudo systemctl --quiet stop ${daemon}.service; + sudo systemctl restart ${daemon}.socket; + done
I suggest changing this to something along the lines of
- for daemon in $DAEMONS; do log_outputs="file:/var/log/libvirt/${daemon}.log" log_filters="3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 1:*" sed -Ei -e "s;^#*\\s*log_outputs\\s*=.*$;log_outputs=\"$log_outputs\";g" \ -e "s;^#*\\s*log_filters\\s*=.*$;log_filters=\"$log_filters\";g" \ "src/remote/${daemon}.conf.in" # ... done
I'd suggest simply not using sed at all - for daemon in $DAEMONS; do log_outputs="file:/var/log/libvirt/${daemon}.log" log_filters="3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 1:*" augtool set /files/etc/libvirt/${daemon}.conf/log_filters "$log_filters" augtool set /files/etc/libvirt/${daemon}.conf/log_outputs "$log_outputs" done Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, Jan 31, 2022 at 07:00:57PM +0100, Erik Skultety wrote:
I'd like to propose a PoC of a GitLab-driven functional CI based on GitLab's custom executor runner feature.
ping? I'd like to get more comments on whether this proposal is heading the right direction. Also, patches 1-2/4 were already pushed as those were independent.
On Mon, Feb 07, 2022 at 03:01:18PM +0100, Erik Skultety wrote:
On Mon, Jan 31, 2022 at 07:00:57PM +0100, Erik Skultety wrote:
I'd like to propose a PoC of a GitLab-driven functional CI based on GitLab's custom executor runner feature.
ping?
I'd like to get more comments on whether this proposal is heading the right direction.
It all looks pretty good to me, and I think we should go ahead with it. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 2/10/22 10:52, Daniel P. Berrangé wrote:
On Mon, Feb 07, 2022 at 03:01:18PM +0100, Erik Skultety wrote:
On Mon, Jan 31, 2022 at 07:00:57PM +0100, Erik Skultety wrote:
I'd like to propose a PoC of a GitLab-driven functional CI based on GitLab's custom executor runner feature.
ping?
I'd like to get more comments on whether this proposal is heading the right direction.
It all looks pretty good to me, and I think we should go ahead with it.
Agreed, I like that we have at least some testing to begin with. Michal
On Thu, Feb 10, 2022 at 12:22:36PM +0100, Michal Prívozník wrote:
On 2/10/22 10:52, Daniel P. Berrangé wrote:
On Mon, Feb 07, 2022 at 03:01:18PM +0100, Erik Skultety wrote:
On Mon, Jan 31, 2022 at 07:00:57PM +0100, Erik Skultety wrote:
I'd like to propose a PoC of a GitLab-driven functional CI based on GitLab's custom executor runner feature.
ping?
I'd like to get more comments on whether this proposal is heading the right direction.
It all looks pretty good to me, and I think we should go ahead with it.
Agreed, I like that we have at least some testing to begin with.
Michal
Thanks for the comments, I'll proceed with applying the comments and submit a new revision. I'll have to wait until the corresponding libvirt-perl changes are merged first. Erik
On Mon, Jan 31, 2022 at 07:00:57PM +0100, Erik Skultety wrote:
Another annoying thing in terms of test failure analysis is that TCK in its current shape cannot save libvirt debug logs on a per-test basis and so a massive dump is produced for the whole test run. This is very suboptimal, but this can only be solved with libvirt admin API which can enable/modify/redirect logs at runtime, however, the perl bindings are currently missings -> I'm already working on adding the bindings to libvirt-perl and then we need to teach TCK to use that feature
IIUC you're suggesting that at the start of each test we re-configure the libvirtd.log to say "libvirt-this-test.log", so we can capture the logs per test ? If so, an easy way to achieve this would be to do 'truncate libvirtd.log' at the start of each test. I don't mind which way though. If we ever want ability to run tests concurrently though, we'll not be able to separate the logs at all. I guess we can worry about that another time though. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (5)
-
Andrea Bolognani -
Daniel P. Berrangé -
Erik Skultety -
Jiri Denemark -
Michal Prívozník