[libvirt PATCH v3 0/3] Propose a simple mechanism on how to disable pipeline jobs
From time to time it happens that some of the distros for which we run our pipelines break (or the images we pull break). Because we don't have dedicated maintainers for the jobs/runners, we can only employ the best effort approach wrt to making the pipeline green again when we're positive the problem doesn't lie in libvirt. For jobs that are broken for quite some time we should opt to disable them unconditionally unless a volunteer dedicates the time to either fix it or the matter gets fixed on its own in time (e.g. updated container images).
v1: https://www.redhat.com/archives/libvir-list/2021-January/msg00573.html v2: https://www.redhat.com/archives/libvir-list/2021-January/msg00672.html Since v2: - renamed all the template jobs according to the review comments - added 'rules' for container jobs as well - fixed the env variable naming inconsistency that somehow made it to the v1 patches Erik Skultety (3): gitlab-ci.yml: Replace template anchors with extends gitlab-ci.yml: Convert only/except to the rules syntax gitlab-ci.yml: Add an explicit env variable to mark a job as broken .gitlab-ci.yml | 164 ++++++++++++++++++++++++++----------------------- 1 file changed, 86 insertions(+), 78 deletions(-) -- 2.29.2
'extends' is slightly more readable and definitely more flexible in terms of allowing includes of templates. The main reason for this patch though is that the next patch converts the 'only/except' syntax to the new (preferable) 'rules' syntax. Variable anchors are still kept intact because the use case there is different from regular template anchors. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- .gitlab-ci.yml | 128 ++++++++++++++++++++++++------------------------- 1 file changed, 64 insertions(+), 64 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 2171905f8e..eefcef526b 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -14,7 +14,7 @@ stages: # Common templates -.container_job_template: &container_job_definition +.container_job: image: docker:stable stage: containers needs: [] @@ -36,11 +36,11 @@ stages: # needed for the pipeline itself to complete: those sometimes fail, and when # that happens it's mostly because of temporary issues with Debian sid. We # don't want those failures to affect the overall pipeline status -.container_optional_job_template: &container_optional_job_definition - <<: *container_job_definition +.container_optional_job: + extends: .container_job allow_failure: true -.native_build_job_template: &native_build_job_definition +.native_build_job: stage: builds image: $CI_REGISTRY_IMAGE/ci-$NAME:latest cache: @@ -75,7 +75,7 @@ stages: # Note that the $PATH environment variable has to be treated with # special care, because we can't just override it at the GitLab CI job # definition level or we risk breaking it completely. -.cirrus_build_job_template: &cirrus_build_job_definition +.cirrus_build_job: stage: builds image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master needs: [] @@ -103,7 +103,7 @@ stages: - $CIRRUS_GITHUB_REPO - $CIRRUS_API_TOKEN -.cross_build_default_job_template: &cross_build_job_definition +.cross_build_job: stage: builds image: $CI_REGISTRY_IMAGE/ci-$NAME-cross-$CROSS:latest cache: @@ -121,57 +121,57 @@ stages: # Native container build jobs x64-centos-7-container: - <<: *container_job_definition + extends: .container_job variables: NAME: centos-7 x64-centos-8-container: - <<: *container_job_definition + extends: .container_job variables: NAME: centos-8 x64-centos-stream-container: - <<: *container_job_definition + extends: .container_job variables: NAME: centos-stream x64-debian-10-container: - <<: *container_job_definition + extends: .container_job variables: NAME: debian-10 x64-debian-sid-container: - <<: *container_job_definition + extends: .container_job variables: NAME: debian-sid x64-fedora-32-container: - <<: *container_job_definition + extends: .container_job variables: NAME: fedora-32 x64-fedora-33-container: - <<: *container_job_definition + extends: .container_job variables: NAME: fedora-33 x64-fedora-rawhide-container: - <<: *container_job_definition + extends: .container_job variables: NAME: fedora-rawhide x64-opensuse-151-container: - <<: *container_job_definition + extends: .container_job variables: NAME: opensuse-151 x64-ubuntu-1804-container: - <<: *container_job_definition + extends: .container_job variables: NAME: ubuntu-1804 x64-ubuntu-2004-container: - <<: *container_job_definition + extends: .container_job variables: NAME: ubuntu-2004 @@ -179,97 +179,97 @@ x64-ubuntu-2004-container: # Cross-build containers build jobs aarch64-debian-10-container: - <<: *container_job_definition + extends: .container_job variables: NAME: debian-10-cross-aarch64 armv6l-debian-10-container: - <<: *container_job_definition + extends: .container_job variables: NAME: debian-10-cross-armv6l armv7l-debian-10-container: - <<: *container_optional_job_definition + extends: .container_optional_job variables: NAME: debian-10-cross-armv7l i686-debian-10-container: - <<: *container_optional_job_definition + extends: .container_optional_job variables: NAME: debian-10-cross-i686 mips-debian-10-container: - <<: *container_job_definition + extends: .container_job variables: NAME: debian-10-cross-mips mips64el-debian-10-container: - <<: *container_job_definition + extends: .container_job variables: NAME: debian-10-cross-mips64el mipsel-debian-10-container: - <<: *container_job_definition + extends: .container_job variables: NAME: debian-10-cross-mipsel ppc64le-debian-10-container: - <<: *container_optional_job_definition + extends: .container_optional_job variables: NAME: debian-10-cross-ppc64le s390x-debian-10-container: - <<: *container_optional_job_definition + extends: .container_optional_job variables: NAME: debian-10-cross-s390x aarch64-debian-sid-container: - <<: *container_optional_job_definition + extends: .container_optional_job variables: NAME: debian-sid-cross-aarch64 armv6l-debian-sid-container: - <<: *container_optional_job_definition + extends: .container_optional_job variables: NAME: debian-sid-cross-armv6l armv7l-debian-sid-container: - <<: *container_job_definition + extends: .container_job variables: NAME: debian-sid-cross-armv7l i686-debian-sid-container: - <<: *container_job_definition + extends: .container_job variables: NAME: debian-sid-cross-i686 mips64el-debian-sid-container: - <<: *container_optional_job_definition + extends: .container_optional_job variables: NAME: debian-sid-cross-mips64el mipsel-debian-sid-container: - <<: *container_optional_job_definition + extends: .container_optional_job variables: NAME: debian-sid-cross-mipsel ppc64le-debian-sid-container: - <<: *container_job_definition + extends: .container_job variables: NAME: debian-sid-cross-ppc64le s390x-debian-sid-container: - <<: *container_optional_job_definition + extends: .container_optional_job variables: NAME: debian-sid-cross-s390x mingw32-fedora-rawhide-container: - <<: *container_job_definition + extends: .container_job variables: NAME: fedora-rawhide-cross-mingw32 mingw64-fedora-rawhide-container: - <<: *container_job_definition + extends: .container_job variables: NAME: fedora-rawhide-cross-mingw64 @@ -277,14 +277,14 @@ mingw64-fedora-rawhide-container: # Native architecture build + test jobs x64-debian-10: - <<: *native_build_job_definition + extends: .native_build_job needs: - x64-debian-10-container variables: NAME: debian-10 x64-debian-10-clang: - <<: *native_build_job_definition + extends: .native_build_job needs: - x64-debian-10-container variables: @@ -292,14 +292,14 @@ x64-debian-10-clang: CC: clang x64-debian-sid: - <<: *native_build_job_definition + extends: .native_build_job needs: - x64-debian-sid-container variables: NAME: debian-sid x64-centos-7: - <<: *native_build_job_definition + extends: .native_build_job needs: - x64-centos-7-container variables: @@ -310,7 +310,7 @@ x64-centos-7: RPM: skip x64-centos-8: - <<: *native_build_job_definition + extends: .native_build_job needs: - x64-centos-8-container variables: @@ -318,7 +318,7 @@ x64-centos-8: RPM: skip x64-centos-8-clang: - <<: *native_build_job_definition + extends: .native_build_job needs: - x64-centos-8-container variables: @@ -327,7 +327,7 @@ x64-centos-8-clang: RPM: skip x64-centos-stream: - <<: *native_build_job_definition + extends: .native_build_job needs: - x64-centos-stream-container variables: @@ -335,7 +335,7 @@ x64-centos-stream: RPM: skip x64-fedora-32: - <<: *native_build_job_definition + extends: .native_build_job needs: - x64-fedora-32-container variables: @@ -343,21 +343,21 @@ x64-fedora-32: RPM: skip x64-fedora-33: - <<: *native_build_job_definition + extends: .native_build_job needs: - x64-fedora-33-container variables: NAME: fedora-33 x64-fedora-rawhide: - <<: *native_build_job_definition + extends: .native_build_job needs: - x64-fedora-rawhide-container variables: NAME: fedora-rawhide x64-fedora-rawhide-clang: - <<: *native_build_job_definition + extends: .native_build_job needs: - x64-fedora-rawhide-container variables: @@ -366,7 +366,7 @@ x64-fedora-rawhide-clang: RPM: skip x64-opensuse-151: - <<: *native_build_job_definition + extends: .native_build_job needs: - x64-opensuse-151-container variables: @@ -374,21 +374,21 @@ x64-opensuse-151: RPM: skip x64-ubuntu-1804: - <<: *native_build_job_definition + extends: .native_build_job needs: - x64-ubuntu-1804-container variables: NAME: ubuntu-1804 x64-ubuntu-2004: - <<: *native_build_job_definition + extends: .native_build_job needs: - x64-ubuntu-2004-container variables: NAME: ubuntu-2004 x64-freebsd-11-build: - <<: *cirrus_build_job_definition + extends: .cirrus_build_job variables: NAME: freebsd-11 CIRRUS_VM_INSTANCE_TYPE: freebsd_instance @@ -397,7 +397,7 @@ x64-freebsd-11-build: INSTALL_COMMAND: pkg install -y x64-freebsd-12-build: - <<: *cirrus_build_job_definition + extends: .cirrus_build_job variables: NAME: freebsd-12 CIRRUS_VM_INSTANCE_TYPE: freebsd_instance @@ -406,7 +406,7 @@ x64-freebsd-12-build: INSTALL_COMMAND: pkg install -y x64-macos-1015-build: - <<: *cirrus_build_job_definition + extends: .cirrus_build_job variables: NAME: macos-1015 CIRRUS_VM_INSTANCE_TYPE: osx_instance @@ -420,7 +420,7 @@ x64-macos-1015-build: # Cross compiled build jobs armv6l-debian-10: - <<: *cross_build_job_definition + extends: .cross_build_job needs: - armv6l-debian-10-container variables: @@ -428,7 +428,7 @@ armv6l-debian-10: CROSS: armv6l armv7l-debian-10: - <<: *cross_build_job_definition + extends: .cross_build_job needs: - armv7l-debian-10-container variables: @@ -436,7 +436,7 @@ armv7l-debian-10: CROSS: armv7l mips64el-debian-10: - <<: *cross_build_job_definition + extends: .cross_build_job needs: - mips64el-debian-10-container variables: @@ -444,7 +444,7 @@ mips64el-debian-10: CROSS: mips64el mips-debian-10: - <<: *cross_build_job_definition + extends: .cross_build_job needs: - mips-debian-10-container variables: @@ -452,7 +452,7 @@ mips-debian-10: CROSS: mips aarch64-debian-10: - <<: *cross_build_job_definition + extends: .cross_build_job needs: - aarch64-debian-10-container variables: @@ -460,7 +460,7 @@ aarch64-debian-10: CROSS: aarch64 mipsel-debian-10: - <<: *cross_build_job_definition + extends: .cross_build_job needs: - mipsel-debian-10-container variables: @@ -468,7 +468,7 @@ mipsel-debian-10: CROSS: mipsel s390x-debian-sid: - <<: *cross_build_job_definition + extends: .cross_build_job needs: - s390x-debian-10-container variables: @@ -476,7 +476,7 @@ s390x-debian-sid: CROSS: s390x i686-debian-sid: - <<: *cross_build_job_definition + extends: .cross_build_job needs: - i686-debian-sid-container variables: @@ -484,7 +484,7 @@ i686-debian-sid: CROSS: i686 ppc64le-debian-sid: - <<: *cross_build_job_definition + extends: .cross_build_job needs: - ppc64le-debian-10-container variables: @@ -492,7 +492,7 @@ ppc64le-debian-sid: CROSS: ppc64le mingw32-fedora-rawhide: - <<: *cross_build_job_definition + extends: .cross_build_job needs: - mingw32-fedora-rawhide-container variables: @@ -500,7 +500,7 @@ mingw32-fedora-rawhide: CROSS: mingw32 mingw64-fedora-rawhide: - <<: *cross_build_job_definition + extends: .cross_build_job needs: - mingw64-fedora-rawhide-container variables: -- 2.29.2
'rules' syntax replaces the only/except syntax with which it is mutually exclusive. In some cases the 'rules' syntax is more readable than the 'only/except' equivalent, in some cases it is not. The idea behind this conversion is to introduce an explicit env variable controlling the 'allow_failure' attribute which would then be attached to a broken build job which would in turn result in a soft failure. Such behaviour is not possible to achieve with the older 'only/except' syntax. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- .gitlab-ci.yml | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index eefcef526b..7636544ee1 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -20,6 +20,8 @@ stages: needs: [] services: - docker:dind + rules: + - when: on_success before_script: - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest" - export COMMON_TAG="$CI_REGISTRY/libvirt/libvirt/ci-$NAME:latest" @@ -43,6 +45,8 @@ stages: .native_build_job: stage: builds image: $CI_REGISTRY_IMAGE/ci-$NAME:latest + rules: + - when: on_success cache: paths: - ccache/ @@ -98,10 +102,8 @@ stages: <ci/cirrus/build.yml >ci/cirrus/$NAME.yml - cat ci/cirrus/$NAME.yml - cirrus-run -v --show-build-log always ci/cirrus/$NAME.yml - only: - variables: - - $CIRRUS_GITHUB_REPO - - $CIRRUS_API_TOKEN + rules: + - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN" .cross_build_job: stage: builds @@ -110,6 +112,8 @@ stages: paths: - ccache/ key: "$CI_JOB_NAME" + rules: + - when: on_success before_script: - *script_variables script: @@ -552,8 +556,8 @@ potfile: image: $CI_REGISTRY_IMAGE/ci-centos-8:latest needs: - x64-centos-8-container - only: - - master + rules: + - if: "$CI_COMMIT_BRANCH == 'master'" before_script: - *script_variables script: @@ -580,9 +584,8 @@ check-dco: image: registry.gitlab.com/libvirt/libvirt-ci/check-dco:master script: - /check-dco - except: - variables: - - $CI_PROJECT_NAMESPACE == 'libvirt' + rules: + - if: "$CI_PROJECT_NAMESPACE != 'libvirt'" variables: GIT_DEPTH: 1000 @@ -600,8 +603,5 @@ coverity: - cov-analysis-linux64-*/bin/cov-build --dir cov-int ninja -C build - tar cfz cov-int.tar.gz cov-int - curl https://scan.coverity.com/builds?project=$COVERITY_SCAN_PROJECT_NAME --form token=$COVERITY_SCAN_TOKEN --form email=$GITLAB_USER_EMAIL --form file=@cov-int.tar.gz --form version="$(git describe --tags)" --form description="$(git describe --tags) / $CI_COMMIT_TITLE / $CI_COMMIT_REF_NAME:$CI_PIPELINE_ID" - only: - refs: - - schedules - variables: - - $COVERITY_SCAN_PROJECT_NAME && $COVERITY_SCAN_TOKEN + rules: + - if: "$CI_PIPELINE_SOURCE == 'schedule' && $COVERITY_SCAN_PROJECT_NAME && $COVERITY_SCAN_TOKEN" -- 2.29.2
Thanks to the 'rules' syntax, this will translate to 'allow_failure:true' and let the job fail but will not affect the rest of the pipeline. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- .gitlab-ci.yml | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 7636544ee1..673327fb9e 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -21,6 +21,8 @@ stages: services: - docker:dind rules: + - if: "$TEMPORARILY_DISABLED" + allow_failure: true - when: on_success before_script: - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest" @@ -46,6 +48,8 @@ stages: stage: builds image: $CI_REGISTRY_IMAGE/ci-$NAME:latest rules: + - if: "$TEMPORARILY_DISABLED" + allow_failure: true - when: on_success cache: paths: @@ -103,6 +107,8 @@ stages: - cat ci/cirrus/$NAME.yml - cirrus-run -v --show-build-log always ci/cirrus/$NAME.yml rules: + - if: "$TEMPORARILY_DISABLED" + allow_failure: true - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN" .cross_build_job: @@ -113,6 +119,8 @@ stages: - ccache/ key: "$CI_JOB_NAME" rules: + - if: "$TEMPORARILY_DISABLED" + allow_failure: true - when: on_success before_script: - *script_variables -- 2.29.2
On Thu, 2021-01-14 at 16:15 +0100, Erik Skultety wrote:
From time to time it happens that some of the distros for which we run our pipelines break (or the images we pull break). Because we don't have dedicated maintainers for the jobs/runners, we can only employ the best effort approach wrt to making the pipeline green again when we're positive the problem doesn't lie in libvirt. For jobs that are broken for quite some time we should opt to disable them unconditionally unless a volunteer dedicates the time to either fix it or the matter gets fixed on its own in time (e.g. updated container images).
v1: https://www.redhat.com/archives/libvir-list/2021-January/msg00573.html v2: https://www.redhat.com/archives/libvir-list/2021-January/msg00672.html
Since v2: - renamed all the template jobs according to the review comments - added 'rules' for container jobs as well - fixed the env variable naming inconsistency that somehow made it to the v1 patches
Erik Skultety (3): gitlab-ci.yml: Replace template anchors with extends gitlab-ci.yml: Convert only/except to the rules syntax gitlab-ci.yml: Add an explicit env variable to mark a job as broken
.gitlab-ci.yml | 164 ++++++++++++++++++++++++++----------------------- 1 file changed, 86 insertions(+), 78 deletions(-)
Reviewed-by: Andrea Bolognani <abologna@redhat.com> -- Andrea Bolognani / Red Hat / Virtualization
participants (2)
-
Andrea Bolognani -
Erik Skultety