7:42 a.m.
From time to time it happens that some of the distros for which we run
our
pipelines break (or the images we pull break). Because we don't have
dedicated
maintainers for the jobs/runners, we can only employ the best effort approach
wrt to making the pipeline green again when we're positive the problem doesn't
lie in libvirt.
For jobs that are broken for quite some time we should opt to disable them
unconditionally unless a volunteer dedicates the time to either fix it or the
matter gets fixed on its own in time (e.g. updated container images).
GitLab only provides the "only/except" keywords to conditionally run or skip
jobs. However, the problem with this is that since the job is not scheduled, =
it
won't appear in the pipeline output, so it's by no means clear to an outsider
looking at the pipeline that something "is missing" in the UI. There have been
a few open feature requests to incorporate exit status reporting for jobs and
their interpretation in the pipeline output [1], but the community has been
waiting for this for over 4 years already.
As for patch 2, I spent some time this morning to figure out why the FreeBSD
jobs were failing, not knowing the qemu-utils package is not outdated and
surpassed by the qemu package in port. As a result I intended to disable the
FreeBSD jobs until the maintainers flip the qemu-utils package from BROKEN to
available again. Now that Dan solved the problem in libvirt-ci, I only left it
there for trivial demonstration purposes.
[1] https://gitlab.com/gitlab-org/gitlab-foss/-/issues/25738
Erik Skultety (2):
gitlab-ci.yaml: Introduce a variable to disable long broken jobs
DO NOT MERGE: Showcase a disabled job
.gitlab-ci.yml | 14 ++++++++++++++
1 file changed, 14 insertions(+)
--=20
2.29.2
7:42 a.m.
New subject: [libvirt PATCH 1/2] gitlab-ci.yaml: Introduce a variable to disable long broken jobs
Unfortunately GitLab doesn't have a different mechanism on how to mark
jobs as skipped or disabled apart from the 'only'/'except' pragmas.
Use the 'except' pragma for the job templates to skip jobs which define
the DISABLE_JOB variable.
Signed-off-by: Erik Skultety <eskultet(a)redhat.com>
---
.gitlab-ci.yml | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index e2d5545f0f..aa55b396a1 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -31,6 +31,9 @@ stages:
- docker push "$TAG"
after_script:
- docker logout
+ except:
+ variables:
+ - $DISABLE_JOB
# We build many containers which can be useful to debug problems but are not
# needed for the pipeline itself to complete: those sometimes fail, and when
@@ -62,6 +65,9 @@ stages:
then
rpmbuild --nodeps -ta build/meson-dist/libvirt-*.tar.xz;
fi
+ except:
+ variables:
+ - $DISABLE_JOB
# Jobs that we delegate to Cirrus CI because they require an operating
# system other than Linux. These jobs will only run if the required
@@ -102,6 +108,9 @@ stages:
variables:
- $CIRRUS_GITHUB_REPO
- $CIRRUS_API_TOKEN
+ except:
+ variables:
+ - $DISABLE_JOB
.cross_build_default_job_template: &cross_build_job_definition
stage: builds
@@ -116,6 +125,9 @@ stages:
- meson build --werror $MESON_OPTS || (cat build/meson-logs/meson-log.txt &&
exit 1)
- ninja -C build
- if test "$CROSS" = "i686" ; then ninja -C build test ; fi
+ except:
+ variables:
+ - $DISABLE_JOB
# Native container build jobs
--
2.29.2
7:53 a.m.
New subject: [libvirt PATCH 1/2] gitlab-ci.yaml: Introduce a variable to disable long broken jobs
On Mon, Jan 11, 2021 at 02:42:54PM +0100, Erik Skultety wrote:
Unfortunately GitLab doesn't have a different mechanism on how to
mark
jobs as skipped or disabled apart from the 'only'/'except' pragmas.
Use the 'except' pragma for the job templates to skip jobs which define
the DISABLE_JOB variable.
What's wrong with using
when: never (or manual to let devs force it from the UI)
or
allow_failure: true
for any jobs we have problem with.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
8:56 a.m.
New subject: [libvirt PATCH 1/2] gitlab-ci.yaml: Introduce a variable to disable long broken jobs
On Mon, Jan 11, 2021 at 01:53:49PM +0000, Daniel P. Berrangé wrote:
On Mon, Jan 11, 2021 at 02:42:54PM +0100, Erik Skultety wrote:
> Unfortunately GitLab doesn't have a different mechanism on how to mark
> jobs as skipped or disabled apart from the 'only'/'except' pragmas.
> Use the 'except' pragma for the job templates to skip jobs which define
> the DISABLE_JOB variable.
What's wrong with using
when: never (or manual to let devs force it from the UI)
'when: never' can only be used in conjunction with the 'rules' keyword
[1],[2]
which in turn is mutually exclusive with 'only/except' [3] which we currently
utilize.
'when: manual' doesn't really feel like it's carrying the meaning
properly.
or
allow_failure: true
I could live with ^this one accompanied by a commentary explaining clearly how
we intend to use it with regular jobs, but still it could cause some confusion
with other jobs where we have perfectly valid use case for using allow_failure
all the time.
With that in mind, I think what I'm proposing is much more verbose in what it's
trying to achieve and in itself it also kind of resembles the try-except
construct (only-except).
[1] https://docs.gitlab.com/ee/ci/yaml/README.html#when
[2] https://gitlab.com/eskultety/libvirt/-/pipelines/240008843
[3] https://docs.gitlab.com/ee/ci/yaml/README.html#rules
Erik
for any jobs we have problem with.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
9:03 a.m.
New subject: [libvirt PATCH 1/2] gitlab-ci.yaml: Introduce a variable to disable long broken jobs
On Mon, Jan 11, 2021 at 03:56:03PM +0100, Erik Skultety wrote:
On Mon, Jan 11, 2021 at 01:53:49PM +0000, Daniel P. Berrangé wrote:
> On Mon, Jan 11, 2021 at 02:42:54PM +0100, Erik Skultety wrote:
> > Unfortunately GitLab doesn't have a different mechanism on how to mark
> > jobs as skipped or disabled apart from the 'only'/'except'
pragmas.
> > Use the 'except' pragma for the job templates to skip jobs which
define
> > the DISABLE_JOB variable.
>
> What's wrong with using
>
> when: never (or manual to let devs force it from the UI)
>
'when: never' can only be used in conjunction with the 'rules' keyword
[1],[2]
which in turn is mutually exclusive with 'only/except' [3] which we currently
utilize.
Gitlab recommends use of "rules" over "only/except", so we should
probably change that.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
9:05 a.m.
New subject: [libvirt PATCH 1/2] gitlab-ci.yaml: Introduce a variable to disable long broken jobs
On Mon, Jan 11, 2021 at 03:03:17PM +0000, Daniel P. Berrangé wrote:
On Mon, Jan 11, 2021 at 03:56:03PM +0100, Erik Skultety wrote:
> On Mon, Jan 11, 2021 at 01:53:49PM +0000, Daniel P. Berrangé wrote:
> > On Mon, Jan 11, 2021 at 02:42:54PM +0100, Erik Skultety wrote:
> > > Unfortunately GitLab doesn't have a different mechanism on how to
mark
> > > jobs as skipped or disabled apart from the 'only'/'except'
pragmas.
> > > Use the 'except' pragma for the job templates to skip jobs which
define
> > > the DISABLE_JOB variable.
> >
> > What's wrong with using
> >
> > when: never (or manual to let devs force it from the UI)
> >
>
> 'when: never' can only be used in conjunction with the 'rules'
keyword [1],[2]
> which in turn is mutually exclusive with 'only/except' [3] which we
currently
> utilize.
Gitlab recommends use of "rules" over "only/except", so we should
probably change that.
I noticed, but it either wasn't clear or I simply missed in the docs how you
can restrict job execution by a particular branch with rules.
Erik
9:12 a.m.
New subject: [libvirt PATCH 1/2] gitlab-ci.yaml: Introduce a variable to disable long broken jobs
On Mon, Jan 11, 2021 at 04:05:48PM +0100, Erik Skultety wrote:
On Mon, Jan 11, 2021 at 03:03:17PM +0000, Daniel P. Berrangé wrote:
> On Mon, Jan 11, 2021 at 03:56:03PM +0100, Erik Skultety wrote:
> > On Mon, Jan 11, 2021 at 01:53:49PM +0000, Daniel P. Berrangé wrote:
> > > On Mon, Jan 11, 2021 at 02:42:54PM +0100, Erik Skultety wrote:
> > > > Unfortunately GitLab doesn't have a different mechanism on how to
mark
> > > > jobs as skipped or disabled apart from the
'only'/'except' pragmas.
> > > > Use the 'except' pragma for the job templates to skip jobs
which define
> > > > the DISABLE_JOB variable.
> > >
> > > What's wrong with using
> > >
> > > when: never (or manual to let devs force it from the UI)
> > >
> >
> > 'when: never' can only be used in conjunction with the 'rules'
keyword [1],[2]
> > which in turn is mutually exclusive with 'only/except' [3] which we
currently
> > utilize.
>
> Gitlab recommends use of "rules" over "only/except", so we
should
> probably change that.
I noticed, but it either wasn't clear or I simply missed in the docs how you
can restrict job execution by a particular branch with rules.
Do a match against the env variable
- if: '$CI_COMMIT_REF_NAME == "master"'
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
9:22 a.m.
New subject: [libvirt PATCH 1/2] gitlab-ci.yaml: Introduce a variable to disable long broken jobs
On Mon, 2021-01-11 at 13:53 +0000, Daniel P. Berrangé wrote:
On Mon, Jan 11, 2021 at 02:42:54PM +0100, Erik Skultety wrote:
> Unfortunately GitLab doesn't have a different mechanism on how to mark
> jobs as skipped or disabled apart from the 'only'/'except' pragmas.
> Use the 'except' pragma for the job templates to skip jobs which define
> the DISABLE_JOB variable.
What's wrong with using
when: never (or manual to let devs force it from the UI)
or
allow_failure: true
for any jobs we have problem with.
Personally, I like the idea of using something extremely explicit
and specific, let's say
TEMPORARILY_ALLOW_FAILURE: true
because then it becomes possible to tell at a glance, without having
to dig through the git history, whether a certain job is known to be
affected by some temporary issue, possibly one that's beyond our
control.
I also believe we should not skip jobs in these cases, but rather
start them and allow them to fail, so that we will be reminded about
the temporary issue by the soft-failure icon (it's orange IIRC) on
the pipeline status page.
--
Andrea Bolognani / Red Hat / Virtualization
7:42 a.m.
New subject: [libvirt PATCH 2/2] DO NOT MERGE: Showcase a disabled job
This showcases how a we'd mark a job as disabled.
---
.gitlab-ci.yml | 2 ++
1 file changed, 2 insertions(+)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index aa55b396a1..1057710ead 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -407,6 +407,7 @@ x64-freebsd-11-build:
CIRRUS_VM_IMAGE_SELECTOR: image_family
CIRRUS_VM_IMAGE_NAME: freebsd-11-4
INSTALL_COMMAND: pkg install -y
+ DISABLE_JOB: "yes"
x64-freebsd-12-build:
<<: *cirrus_build_job_definition
@@ -416,6 +417,7 @@ x64-freebsd-12-build:
CIRRUS_VM_IMAGE_SELECTOR: image_family
CIRRUS_VM_IMAGE_NAME: freebsd-12-1
INSTALL_COMMAND: pkg install -y
+ DISABLE_JOB: "yes"
x64-macos-1015-build:
<<: *cirrus_build_job_definition
--
2.29.2
1411
days inactive
1411
days old
8 comments
3 participants
participants (3)
-
Andrea Bolognani -
Daniel P. Berrangé -
Erik Skultety