7:52 a.m.
Even if users set the LIBVIRT_CI_INTEGRATION variable to 1 to enable the
integration test suite with their own private runner, the pipeline will be stuck
because we expect the runner to have the 'redhat-vm-host' tag. We need to define
the job tag dynamically instead, so that the users can set the variable for
their libvirt fork to match one of their runner's tags.
A recent crash in libvirt showed us that we also need to collect stack traces as
well, so this series also takes care of setting up systemd to be able to collect
them on failed jobs.
Both fixes are demoed with pipeline [1]. Note that for the pipeline I reverted
17fe6a090ba389cdd228c3e176788d9b2d45525f just to demonstrate the stack trace
collection on a crash.
[1] https://gitlab.com/eskultety/libvirt/-/pipelines/497404762
Erik Skultety (2):
ci: Define the integration job tag dynamically via a variable
ci: integration: Collect stack traces with coredumpctl
ci/integration.yml | 20 +++++++++++++++-----
1 file changed, 15 insertions(+), 5 deletions(-)
--
2.34.1
7:52 a.m.
New subject: [libvirt PATCH 1/2] ci: Define the integration job tag dynamically via a variable
Custom runners are private to a project, so naturally forks cannot run
any workloads on these. The integration test suite which requires
access to our custom runner is naturally disabled on forks and can be
enabled by setting LIBVIRT_CI_INTEGRATION=1.
The problem is that the current integration jobs definitions have tags
statically defined as 'redhat-vm-host'. If users are going to supply
their own private runners for their forks, they can define whatever
tags they want with it and so unless they add 'redhat-vm-host' to their
own runner's tags, the pipeline won't run.
To solve this, define the integration job tag using a variable. The
repo config will use the value defined in the job for the variable
while users can override the value easily on a project/pipeline level
thanks to GitLab's CI variable precedence [1].
[1] https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence
Signed-off-by: Erik Skultety <eskultet(a)redhat.com>
---
ci/integration.yml | 17 ++++++++++++-----
1 file changed, 12 insertions(+), 5 deletions(-)
diff --git a/ci/integration.yml b/ci/integration.yml
index 8551ce8776..9726f00bff 100644
--- a/ci/integration.yml
+++ b/ci/integration.yml
@@ -45,7 +45,6 @@
when: on_success
- when: never
-
centos-stream-8-tests:
extends: .integration_tests
needs:
@@ -57,8 +56,10 @@ centos-stream-8-tests:
variables:
# needed by libvirt-gitlab-executor
DISTRO: centos-stream-8
+ # can be overriden in forks to set a different runner tag
+ LIBVIRT_CI_INTEGRATION_RUNNER_TAG: redhat-vm-host
tags:
- - redhat-vm-host
+ - $LIBVIRT_CI_INTEGRATION_RUNNER_TAG
centos-stream-9-tests:
extends: .integration_tests
@@ -71,8 +72,10 @@ centos-stream-9-tests:
variables:
# needed by libvirt-gitlab-executor
DISTRO: centos-stream-9
+ # can be overriden in forks to set a different runner tag
+ LIBVIRT_CI_INTEGRATION_RUNNER_TAG: redhat-vm-host
tags:
- - redhat-vm-host
+ - $LIBVIRT_CI_INTEGRATION_RUNNER_TAG
fedora-34-tests:
extends: .integration_tests
@@ -85,8 +88,10 @@ fedora-34-tests:
variables:
# needed by libvirt-gitlab-executor
DISTRO: fedora-34
+ # can be overriden in forks to set a different runner tag
+ LIBVIRT_CI_INTEGRATION_RUNNER_TAG: redhat-vm-host
tags:
- - redhat-vm-host
+ - $LIBVIRT_CI_INTEGRATION_RUNNER_TAG
fedora-35-tests:
extends: .integration_tests
@@ -99,5 +104,7 @@ fedora-35-tests:
variables:
# needed by libvirt-gitlab-executor
DISTRO: fedora-35
+ # can be overriden in forks to set a different runner tag
+ LIBVIRT_CI_INTEGRATION_RUNNER_TAG: redhat-vm-host
tags:
- - redhat-vm-host
+ - $LIBVIRT_CI_INTEGRATION_RUNNER_TAG
--
2.34.1
7:52 a.m.
New subject: [libvirt PATCH 2/2] ci: integration: Collect stack traces with coredumpctl
Some Red Hat-like distros have cores limited with a soft limit of 0
which means that neither a stack trace nor a core file will be
available. Since we want the stack trace we need to set the core limit
with systemd globally to unlimited/infinity.
Signed-off-by: Erik Skultety <eskultet(a)redhat.com>
---
ci/integration.yml | 3 +++
1 file changed, 3 insertions(+)
diff --git a/ci/integration.yml b/ci/integration.yml
index 9726f00bff..b2932f2f39 100644
--- a/ci/integration.yml
+++ b/ci/integration.yml
@@ -2,6 +2,8 @@
stage: integration_tests
before_script:
- mkdir "$SCRATCH_DIR"
+ - sudo sh -c "echo DefaultLimitCORE=infinity >>
/etc/systemd/system.conf" # Explicitly allow storing cores globally
+ - sudo systemctl daemon-reexec # need to reexec systemd after changing config
- sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/*
- sudo pip3 install --prefix=/usr avocado-framework
- source /etc/os-release # in order to query the vendor-provided variables
@@ -31,6 +33,7 @@
after_script:
- test "$CI_JOB_STATUS" = "success" && exit 0;
- test -e "$SCRATCH_DIR"/avocado && sudo mv
"$SCRATCH_DIR"/avocado/latest/test-results logs/avocado;
+ - sudo coredumpctl info --no-pager > logs/coredumpctl.info
- sudo mv /var/log/libvirt logs/libvirt
- sudo chown -R $(whoami):$(whoami) logs
variables:
--
2.34.1
11:53 a.m.
New subject: [libvirt PATCH 2/2] ci: integration: Collect stack traces with coredumpctl
On Tue, Mar 22, 2022 at 07:52:21AM +0100, Erik Skultety wrote:
Some Red Hat-like distros have cores limited with a soft limit of 0
which means that neither a stack trace nor a core file will be
available. Since we want the stack trace we need to set the core limit
with systemd globally to unlimited/infinity.
Signed-off-by: Erik Skultety <eskultet(a)redhat.com>
---
ci/integration.yml | 3 +++
1 file changed, 3 insertions(+)
diff --git a/ci/integration.yml b/ci/integration.yml
index 9726f00bff..b2932f2f39 100644
--- a/ci/integration.yml
+++ b/ci/integration.yml
@@ -2,6 +2,8 @@
stage: integration_tests
before_script:
- mkdir "$SCRATCH_DIR"
+ - sudo sh -c "echo DefaultLimitCORE=infinity >>
/etc/systemd/system.conf" # Explicitly allow storing cores globally
+ - sudo systemctl daemon-reexec # need to reexec systemd after changing config
- sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/*
- sudo pip3 install --prefix=/usr avocado-framework
- source /etc/os-release # in order to query the vendor-provided variables
@@ -31,6 +33,7 @@
after_script:
- test "$CI_JOB_STATUS" = "success" && exit 0;
- test -e "$SCRATCH_DIR"/avocado && sudo mv
"$SCRATCH_DIR"/avocado/latest/test-results logs/avocado;
+ - sudo coredumpctl info --no-pager > logs/coredumpctl.info
I'd have a little preference for .txt as a file extension, because IIUC
the gitlab artifact browser will serve this as a mime-type that forces
the browser to download, rather than viewing inline as with .txt.
- sudo mv /var/log/libvirt logs/libvirt
- sudo chown -R $(whoami):$(whoami) logs
variables:
--
2.34.1
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
12:47 p.m.
New subject: [libvirt PATCH 2/2] ci: integration: Collect stack traces with coredumpctl
On Tue, Mar 22, 2022 at 10:53:27AM +0000, Daniel P. Berrangé wrote:
On Tue, Mar 22, 2022 at 07:52:21AM +0100, Erik Skultety wrote:
> Some Red Hat-like distros have cores limited with a soft limit of 0
> which means that neither a stack trace nor a core file will be
> available. Since we want the stack trace we need to set the core limit
> with systemd globally to unlimited/infinity.
>
> Signed-off-by: Erik Skultety <eskultet(a)redhat.com>
> ---
> ci/integration.yml | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/ci/integration.yml b/ci/integration.yml
> index 9726f00bff..b2932f2f39 100644
> --- a/ci/integration.yml
> +++ b/ci/integration.yml
> @@ -2,6 +2,8 @@
> stage: integration_tests
> before_script:
> - mkdir "$SCRATCH_DIR"
> + - sudo sh -c "echo DefaultLimitCORE=infinity >>
/etc/systemd/system.conf" # Explicitly allow storing cores globally
> + - sudo systemctl daemon-reexec # need to reexec systemd after changing config
> - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/*
> - sudo pip3 install --prefix=/usr avocado-framework
> - source /etc/os-release # in order to query the vendor-provided variables
> @@ -31,6 +33,7 @@
> after_script:
> - test "$CI_JOB_STATUS" = "success" && exit 0;
> - test -e "$SCRATCH_DIR"/avocado && sudo mv
"$SCRATCH_DIR"/avocado/latest/test-results logs/avocado;
> + - sudo coredumpctl info --no-pager > logs/coredumpctl.info
I'd have a little preference for .txt as a file extension, because IIUC
the gitlab artifact browser will serve this as a mime-type that forces
the browser to download, rather than viewing inline as with .txt.
Fixed and pushed.
Thanks,
Erik
8:51 a.m.
On 3/22/22 07:52, Erik Skultety wrote:
Even if users set the LIBVIRT_CI_INTEGRATION variable to 1 to enable
the
integration test suite with their own private runner, the pipeline will be stuck
because we expect the runner to have the 'redhat-vm-host' tag. We need to define
the job tag dynamically instead, so that the users can set the variable for
their libvirt fork to match one of their runner's tags.
A recent crash in libvirt showed us that we also need to collect stack traces as
well, so this series also takes care of setting up systemd to be able to collect
them on failed jobs.
Both fixes are demoed with pipeline [1]. Note that for the pipeline I reverted
17fe6a090ba389cdd228c3e176788d9b2d45525f just to demonstrate the stack trace
collection on a crash.
[1] https://gitlab.com/eskultety/libvirt/-/pipelines/497404762
Erik Skultety (2):
ci: Define the integration job tag dynamically via a variable
ci: integration: Collect stack traces with coredumpctl
ci/integration.yml | 20 +++++++++++++++-----
1 file changed, 15 insertions(+), 5 deletions(-)
Reviewed-by: Michal Privoznik <mprivozn(a)redhat.com>
Michal
913
days inactive
913
days old
5 comments
3 participants
participants (3)
-
Daniel P. Berrangé -
Erik Skultety -
Michal Prívozník