[libvirt PATCH 0/2] [CI] Fix running integration pipeline in forks + collect daemon stack trace
Even if users set the LIBVIRT_CI_INTEGRATION variable to 1 to enable the integration test suite with their own private runner, the pipeline will be stuck because we expect the runner to have the 'redhat-vm-host' tag. We need to define the job tag dynamically instead, so that the users can set the variable for their libvirt fork to match one of their runner's tags. A recent crash in libvirt showed us that we also need to collect stack traces as well, so this series also takes care of setting up systemd to be able to collect them on failed jobs. Both fixes are demoed with pipeline [1]. Note that for the pipeline I reverted 17fe6a090ba389cdd228c3e176788d9b2d45525f just to demonstrate the stack trace collection on a crash. [1] https://gitlab.com/eskultety/libvirt/-/pipelines/497404762 Erik Skultety (2): ci: Define the integration job tag dynamically via a variable ci: integration: Collect stack traces with coredumpctl ci/integration.yml | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) -- 2.34.1
Custom runners are private to a project, so naturally forks cannot run any workloads on these. The integration test suite which requires access to our custom runner is naturally disabled on forks and can be enabled by setting LIBVIRT_CI_INTEGRATION=1. The problem is that the current integration jobs definitions have tags statically defined as 'redhat-vm-host'. If users are going to supply their own private runners for their forks, they can define whatever tags they want with it and so unless they add 'redhat-vm-host' to their own runner's tags, the pipeline won't run. To solve this, define the integration job tag using a variable. The repo config will use the value defined in the job for the variable while users can override the value easily on a project/pipeline level thanks to GitLab's CI variable precedence [1]. [1] https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/integration.yml | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/ci/integration.yml b/ci/integration.yml index 8551ce8776..9726f00bff 100644 --- a/ci/integration.yml +++ b/ci/integration.yml @@ -45,7 +45,6 @@ when: on_success - when: never - centos-stream-8-tests: extends: .integration_tests needs: @@ -57,8 +56,10 @@ centos-stream-8-tests: variables: # needed by libvirt-gitlab-executor DISTRO: centos-stream-8 + # can be overriden in forks to set a different runner tag + LIBVIRT_CI_INTEGRATION_RUNNER_TAG: redhat-vm-host tags: - - redhat-vm-host + - $LIBVIRT_CI_INTEGRATION_RUNNER_TAG centos-stream-9-tests: extends: .integration_tests @@ -71,8 +72,10 @@ centos-stream-9-tests: variables: # needed by libvirt-gitlab-executor DISTRO: centos-stream-9 + # can be overriden in forks to set a different runner tag + LIBVIRT_CI_INTEGRATION_RUNNER_TAG: redhat-vm-host tags: - - redhat-vm-host + - $LIBVIRT_CI_INTEGRATION_RUNNER_TAG fedora-34-tests: extends: .integration_tests @@ -85,8 +88,10 @@ fedora-34-tests: variables: # needed by libvirt-gitlab-executor DISTRO: fedora-34 + # can be overriden in forks to set a different runner tag + LIBVIRT_CI_INTEGRATION_RUNNER_TAG: redhat-vm-host tags: - - redhat-vm-host + - $LIBVIRT_CI_INTEGRATION_RUNNER_TAG fedora-35-tests: extends: .integration_tests @@ -99,5 +104,7 @@ fedora-35-tests: variables: # needed by libvirt-gitlab-executor DISTRO: fedora-35 + # can be overriden in forks to set a different runner tag + LIBVIRT_CI_INTEGRATION_RUNNER_TAG: redhat-vm-host tags: - - redhat-vm-host + - $LIBVIRT_CI_INTEGRATION_RUNNER_TAG -- 2.34.1
Some Red Hat-like distros have cores limited with a soft limit of 0 which means that neither a stack trace nor a core file will be available. Since we want the stack trace we need to set the core limit with systemd globally to unlimited/infinity. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/integration.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/ci/integration.yml b/ci/integration.yml index 9726f00bff..b2932f2f39 100644 --- a/ci/integration.yml +++ b/ci/integration.yml @@ -2,6 +2,8 @@ stage: integration_tests before_script: - mkdir "$SCRATCH_DIR" + - sudo sh -c "echo DefaultLimitCORE=infinity >> /etc/systemd/system.conf" # Explicitly allow storing cores globally + - sudo systemctl daemon-reexec # need to reexec systemd after changing config - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/* - sudo pip3 install --prefix=/usr avocado-framework - source /etc/os-release # in order to query the vendor-provided variables @@ -31,6 +33,7 @@ after_script: - test "$CI_JOB_STATUS" = "success" && exit 0; - test -e "$SCRATCH_DIR"/avocado && sudo mv "$SCRATCH_DIR"/avocado/latest/test-results logs/avocado; + - sudo coredumpctl info --no-pager > logs/coredumpctl.info - sudo mv /var/log/libvirt logs/libvirt - sudo chown -R $(whoami):$(whoami) logs variables: -- 2.34.1
On Tue, Mar 22, 2022 at 07:52:21AM +0100, Erik Skultety wrote:
Some Red Hat-like distros have cores limited with a soft limit of 0 which means that neither a stack trace nor a core file will be available. Since we want the stack trace we need to set the core limit with systemd globally to unlimited/infinity.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/integration.yml | 3 +++ 1 file changed, 3 insertions(+)
diff --git a/ci/integration.yml b/ci/integration.yml index 9726f00bff..b2932f2f39 100644 --- a/ci/integration.yml +++ b/ci/integration.yml @@ -2,6 +2,8 @@ stage: integration_tests before_script: - mkdir "$SCRATCH_DIR" + - sudo sh -c "echo DefaultLimitCORE=infinity >> /etc/systemd/system.conf" # Explicitly allow storing cores globally + - sudo systemctl daemon-reexec # need to reexec systemd after changing config - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/* - sudo pip3 install --prefix=/usr avocado-framework - source /etc/os-release # in order to query the vendor-provided variables @@ -31,6 +33,7 @@ after_script: - test "$CI_JOB_STATUS" = "success" && exit 0; - test -e "$SCRATCH_DIR"/avocado && sudo mv "$SCRATCH_DIR"/avocado/latest/test-results logs/avocado; + - sudo coredumpctl info --no-pager > logs/coredumpctl.info
I'd have a little preference for .txt as a file extension, because IIUC the gitlab artifact browser will serve this as a mime-type that forces the browser to download, rather than viewing inline as with .txt.
- sudo mv /var/log/libvirt logs/libvirt - sudo chown -R $(whoami):$(whoami) logs variables: -- 2.34.1
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Mar 22, 2022 at 10:53:27AM +0000, Daniel P. Berrangé wrote:
On Tue, Mar 22, 2022 at 07:52:21AM +0100, Erik Skultety wrote:
Some Red Hat-like distros have cores limited with a soft limit of 0 which means that neither a stack trace nor a core file will be available. Since we want the stack trace we need to set the core limit with systemd globally to unlimited/infinity.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/integration.yml | 3 +++ 1 file changed, 3 insertions(+)
diff --git a/ci/integration.yml b/ci/integration.yml index 9726f00bff..b2932f2f39 100644 --- a/ci/integration.yml +++ b/ci/integration.yml @@ -2,6 +2,8 @@ stage: integration_tests before_script: - mkdir "$SCRATCH_DIR" + - sudo sh -c "echo DefaultLimitCORE=infinity >> /etc/systemd/system.conf" # Explicitly allow storing cores globally + - sudo systemctl daemon-reexec # need to reexec systemd after changing config - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/* - sudo pip3 install --prefix=/usr avocado-framework - source /etc/os-release # in order to query the vendor-provided variables @@ -31,6 +33,7 @@ after_script: - test "$CI_JOB_STATUS" = "success" && exit 0; - test -e "$SCRATCH_DIR"/avocado && sudo mv "$SCRATCH_DIR"/avocado/latest/test-results logs/avocado; + - sudo coredumpctl info --no-pager > logs/coredumpctl.info
I'd have a little preference for .txt as a file extension, because IIUC the gitlab artifact browser will serve this as a mime-type that forces the browser to download, rather than viewing inline as with .txt.
Fixed and pushed. Thanks, Erik
On 3/22/22 07:52, Erik Skultety wrote:
Even if users set the LIBVIRT_CI_INTEGRATION variable to 1 to enable the integration test suite with their own private runner, the pipeline will be stuck because we expect the runner to have the 'redhat-vm-host' tag. We need to define the job tag dynamically instead, so that the users can set the variable for their libvirt fork to match one of their runner's tags.
A recent crash in libvirt showed us that we also need to collect stack traces as well, so this series also takes care of setting up systemd to be able to collect them on failed jobs.
Both fixes are demoed with pipeline [1]. Note that for the pipeline I reverted 17fe6a090ba389cdd228c3e176788d9b2d45525f just to demonstrate the stack trace collection on a crash.
[1] https://gitlab.com/eskultety/libvirt/-/pipelines/497404762
Erik Skultety (2): ci: Define the integration job tag dynamically via a variable ci: integration: Collect stack traces with coredumpctl
ci/integration.yml | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Michal
participants (3)
-
Daniel P. Berrangé -
Erik Skultety -
Michal Prívozník