[libvirt PATCH 0/2] Propose a simple mechanism on how to disable pipeline jobs
From time to time it happens that some of the distros for which we run our pipelines break (or the images we pull break). Because we don't have dedicated maintainers for the jobs/runners, we can only employ the best effort approach wrt to making the pipeline green again when we're positive the problem doesn't lie in libvirt. For jobs that are broken for quite some time we should opt to disable them unconditionally unless a volunteer dedicates the time to either fix it or the matter gets fixed on its own in time (e.g. updated container images).
GitLab only provides the "only/except" keywords to conditionally run or skip jobs. However, the problem with this is that since the job is not scheduled, = it won't appear in the pipeline output, so it's by no means clear to an outsider looking at the pipeline that something "is missing" in the UI. There have been a few open feature requests to incorporate exit status reporting for jobs and their interpretation in the pipeline output [1], but the community has been waiting for this for over 4 years already. As for patch 2, I spent some time this morning to figure out why the FreeBSD jobs were failing, not knowing the qemu-utils package is not outdated and surpassed by the qemu package in port. As a result I intended to disable the FreeBSD jobs until the maintainers flip the qemu-utils package from BROKEN to available again. Now that Dan solved the problem in libvirt-ci, I only left it there for trivial demonstration purposes. [1] https://gitlab.com/gitlab-org/gitlab-foss/-/issues/25738 Erik Skultety (2): gitlab-ci.yaml: Introduce a variable to disable long broken jobs DO NOT MERGE: Showcase a disabled job .gitlab-ci.yml | 14 ++++++++++++++ 1 file changed, 14 insertions(+) --=20 2.29.2
Unfortunately GitLab doesn't have a different mechanism on how to mark jobs as skipped or disabled apart from the 'only'/'except' pragmas. Use the 'except' pragma for the job templates to skip jobs which define the DISABLE_JOB variable. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- .gitlab-ci.yml | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index e2d5545f0f..aa55b396a1 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -31,6 +31,9 @@ stages: - docker push "$TAG" after_script: - docker logout + except: + variables: + - $DISABLE_JOB # We build many containers which can be useful to debug problems but are not # needed for the pipeline itself to complete: those sometimes fail, and when @@ -62,6 +65,9 @@ stages: then rpmbuild --nodeps -ta build/meson-dist/libvirt-*.tar.xz; fi + except: + variables: + - $DISABLE_JOB # Jobs that we delegate to Cirrus CI because they require an operating # system other than Linux. These jobs will only run if the required @@ -102,6 +108,9 @@ stages: variables: - $CIRRUS_GITHUB_REPO - $CIRRUS_API_TOKEN + except: + variables: + - $DISABLE_JOB .cross_build_default_job_template: &cross_build_job_definition stage: builds @@ -116,6 +125,9 @@ stages: - meson build --werror $MESON_OPTS || (cat build/meson-logs/meson-log.txt && exit 1) - ninja -C build - if test "$CROSS" = "i686" ; then ninja -C build test ; fi + except: + variables: + - $DISABLE_JOB # Native container build jobs -- 2.29.2
On Mon, Jan 11, 2021 at 02:42:54PM +0100, Erik Skultety wrote:
Unfortunately GitLab doesn't have a different mechanism on how to mark jobs as skipped or disabled apart from the 'only'/'except' pragmas. Use the 'except' pragma for the job templates to skip jobs which define the DISABLE_JOB variable.
What's wrong with using when: never (or manual to let devs force it from the UI) or allow_failure: true for any jobs we have problem with. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, Jan 11, 2021 at 01:53:49PM +0000, Daniel P. Berrangé wrote:
On Mon, Jan 11, 2021 at 02:42:54PM +0100, Erik Skultety wrote:
Unfortunately GitLab doesn't have a different mechanism on how to mark jobs as skipped or disabled apart from the 'only'/'except' pragmas. Use the 'except' pragma for the job templates to skip jobs which define the DISABLE_JOB variable.
What's wrong with using
when: never (or manual to let devs force it from the UI)
'when: never' can only be used in conjunction with the 'rules' keyword [1],[2] which in turn is mutually exclusive with 'only/except' [3] which we currently utilize. 'when: manual' doesn't really feel like it's carrying the meaning properly.
or
allow_failure: true
I could live with ^this one accompanied by a commentary explaining clearly how we intend to use it with regular jobs, but still it could cause some confusion with other jobs where we have perfectly valid use case for using allow_failure all the time. With that in mind, I think what I'm proposing is much more verbose in what it's trying to achieve and in itself it also kind of resembles the try-except construct (only-except). [1] https://docs.gitlab.com/ee/ci/yaml/README.html#when [2] https://gitlab.com/eskultety/libvirt/-/pipelines/240008843 [3] https://docs.gitlab.com/ee/ci/yaml/README.html#rules Erik
for any jobs we have problem with.
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, Jan 11, 2021 at 03:56:03PM +0100, Erik Skultety wrote:
On Mon, Jan 11, 2021 at 01:53:49PM +0000, Daniel P. Berrangé wrote:
On Mon, Jan 11, 2021 at 02:42:54PM +0100, Erik Skultety wrote:
Unfortunately GitLab doesn't have a different mechanism on how to mark jobs as skipped or disabled apart from the 'only'/'except' pragmas. Use the 'except' pragma for the job templates to skip jobs which define the DISABLE_JOB variable.
What's wrong with using
when: never (or manual to let devs force it from the UI)
'when: never' can only be used in conjunction with the 'rules' keyword [1],[2] which in turn is mutually exclusive with 'only/except' [3] which we currently utilize.
Gitlab recommends use of "rules" over "only/except", so we should probably change that. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, Jan 11, 2021 at 03:03:17PM +0000, Daniel P. Berrangé wrote:
On Mon, Jan 11, 2021 at 03:56:03PM +0100, Erik Skultety wrote:
On Mon, Jan 11, 2021 at 01:53:49PM +0000, Daniel P. Berrangé wrote:
On Mon, Jan 11, 2021 at 02:42:54PM +0100, Erik Skultety wrote:
Unfortunately GitLab doesn't have a different mechanism on how to mark jobs as skipped or disabled apart from the 'only'/'except' pragmas. Use the 'except' pragma for the job templates to skip jobs which define the DISABLE_JOB variable.
What's wrong with using
when: never (or manual to let devs force it from the UI)
'when: never' can only be used in conjunction with the 'rules' keyword [1],[2] which in turn is mutually exclusive with 'only/except' [3] which we currently utilize.
Gitlab recommends use of "rules" over "only/except", so we should probably change that.
I noticed, but it either wasn't clear or I simply missed in the docs how you can restrict job execution by a particular branch with rules. Erik
On Mon, Jan 11, 2021 at 04:05:48PM +0100, Erik Skultety wrote:
On Mon, Jan 11, 2021 at 03:03:17PM +0000, Daniel P. Berrangé wrote:
On Mon, Jan 11, 2021 at 03:56:03PM +0100, Erik Skultety wrote:
On Mon, Jan 11, 2021 at 01:53:49PM +0000, Daniel P. Berrangé wrote:
On Mon, Jan 11, 2021 at 02:42:54PM +0100, Erik Skultety wrote:
Unfortunately GitLab doesn't have a different mechanism on how to mark jobs as skipped or disabled apart from the 'only'/'except' pragmas. Use the 'except' pragma for the job templates to skip jobs which define the DISABLE_JOB variable.
What's wrong with using
when: never (or manual to let devs force it from the UI)
'when: never' can only be used in conjunction with the 'rules' keyword [1],[2] which in turn is mutually exclusive with 'only/except' [3] which we currently utilize.
Gitlab recommends use of "rules" over "only/except", so we should probably change that.
I noticed, but it either wasn't clear or I simply missed in the docs how you can restrict job execution by a particular branch with rules.
Do a match against the env variable - if: '$CI_COMMIT_REF_NAME == "master"' Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, 2021-01-11 at 13:53 +0000, Daniel P. Berrangé wrote:
On Mon, Jan 11, 2021 at 02:42:54PM +0100, Erik Skultety wrote:
Unfortunately GitLab doesn't have a different mechanism on how to mark jobs as skipped or disabled apart from the 'only'/'except' pragmas. Use the 'except' pragma for the job templates to skip jobs which define the DISABLE_JOB variable.
What's wrong with using
when: never (or manual to let devs force it from the UI)
or
allow_failure: true
for any jobs we have problem with.
Personally, I like the idea of using something extremely explicit and specific, let's say TEMPORARILY_ALLOW_FAILURE: true because then it becomes possible to tell at a glance, without having to dig through the git history, whether a certain job is known to be affected by some temporary issue, possibly one that's beyond our control. I also believe we should not skip jobs in these cases, but rather start them and allow them to fail, so that we will be reminded about the temporary issue by the soft-failure icon (it's orange IIRC) on the pipeline status page. -- Andrea Bolognani / Red Hat / Virtualization
This showcases how a we'd mark a job as disabled. --- .gitlab-ci.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index aa55b396a1..1057710ead 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -407,6 +407,7 @@ x64-freebsd-11-build: CIRRUS_VM_IMAGE_SELECTOR: image_family CIRRUS_VM_IMAGE_NAME: freebsd-11-4 INSTALL_COMMAND: pkg install -y + DISABLE_JOB: "yes" x64-freebsd-12-build: <<: *cirrus_build_job_definition @@ -416,6 +417,7 @@ x64-freebsd-12-build: CIRRUS_VM_IMAGE_SELECTOR: image_family CIRRUS_VM_IMAGE_NAME: freebsd-12-1 INSTALL_COMMAND: pkg install -y + DISABLE_JOB: "yes" x64-macos-1015-build: <<: *cirrus_build_job_definition -- 2.29.2
participants (3)
-
Andrea Bolognani -
Daniel P. Berrangé -
Erik Skultety