[libvirt PATCH v2 0/1] cirrus: Add templates and refresh script
This makes the Cirrus CI configurations as maintainable as the Dockerfiles, by shifting the responsability of keeping the list of dependencies and other details up to date to lcitool. Test pipeline: https://gitlab.com/abologna/libvirt/-/pipelines/161289292 For the refresh script to work, your copy of lcitool needs to include these patches: https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/30 Changes from [v1]: * reduce duplication by only storing variables in the git repository and instantiating the template at runtime. [v1] https://www.redhat.com/archives/libvir-list/2020-June/msg01305.html Andrea Bolognani (1): cirrus: Generate jobs dynamically .gitlab-ci.yml | 36 ++++++++++++++- ci/cirrus/build.yml | 26 +++++++++++ ci/cirrus/freebsd-12.yml.j2 | 73 ------------------------------- ci/cirrus/libvirt-freebsd-12.vars | 7 +++ ci/cirrus/libvirt-macos-1015.vars | 7 +++ ci/cirrus/macos-1015.yml.j2 | 38 ---------------- ci/cirrus/refresh | 22 ++++++++++ 7 files changed, 97 insertions(+), 112 deletions(-) create mode 100644 ci/cirrus/build.yml delete mode 100644 ci/cirrus/freebsd-12.yml.j2 create mode 100644 ci/cirrus/libvirt-freebsd-12.vars create mode 100644 ci/cirrus/libvirt-macos-1015.vars delete mode 100644 ci/cirrus/macos-1015.yml.j2 create mode 100755 ci/cirrus/refresh -- 2.25.4
Instead of having static job definitions for FreeBSD and macOS, use a generic template for both and fill in the details that are actually different, such as the list of packages to install, in the GitLab CI job, right before calling cirrus-run. The target-specific information are provided by lcitool, so that keeping them up to date is just a matter of running the refresh script when necessary. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- .gitlab-ci.yml | 36 ++++++++++++++- ci/cirrus/build.yml | 26 +++++++++++ ci/cirrus/freebsd-12.yml.j2 | 73 ------------------------------- ci/cirrus/libvirt-freebsd-12.vars | 7 +++ ci/cirrus/libvirt-macos-1015.vars | 7 +++ ci/cirrus/macos-1015.yml.j2 | 38 ---------------- ci/cirrus/refresh | 22 ++++++++++ 7 files changed, 97 insertions(+), 112 deletions(-) create mode 100644 ci/cirrus/build.yml delete mode 100644 ci/cirrus/freebsd-12.yml.j2 create mode 100644 ci/cirrus/libvirt-freebsd-12.vars create mode 100644 ci/cirrus/libvirt-macos-1015.vars delete mode 100644 ci/cirrus/macos-1015.yml.j2 create mode 100755 ci/cirrus/refresh diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index e6eb2f9905..5565750b7e 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -58,11 +58,35 @@ stages: # Jobs that we delegate to Cirrus CI because they require an operating # system other than Linux. These jobs will only run if the required # setup has been performed on the GitLab account (see ci/README.rst). +# +# The Cirrus CI configuration is generated by replacing target-specific +# variables in a generic template: some of these variables are provided +# when the GitLab CI job is defined, others are taken from a shell +# snippet generated using lcitool. +# +# Note that the $PATH environment variable has to be treated with +# special care, because we can't just override it at the GitLab CI job +# definition level or we risk breaking it completely. .cirrus_build_job_template: &cirrus_build_job_definition stage: builds image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master script: - - cirrus-run ci/cirrus/$NAME.yml.j2 + - source ci/cirrus/libvirt-$NAME.vars + - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g" + -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g" + -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g" + -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g" + -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g" + -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g" + -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g" + -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g" + -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g" + -e "s|[@]PKGS@|$PKGS|g" + -e "s|[@]MAKE@|$MAKE|g" + -e "s|[@]PYTHON@|$PYTHON|g" + <ci/cirrus/build.yml >ci/cirrus/$NAME.yml + - cat ci/cirrus/$NAME.yml + - cirrus-run ci/cirrus/$NAME.yml only: variables: - $CIRRUS_GITHUB_REPO @@ -351,11 +375,21 @@ x64-freebsd-12-build: <<: *cirrus_build_job_definition variables: NAME: freebsd-12 + CIRRUS_VM_INSTANCE_TYPE: freebsd_instance + CIRRUS_VM_IMAGE_SELECTOR: image_family + CIRRUS_VM_IMAGE_NAME: freebsd-12-1 + INSTALL_COMMAND: pkg install -y x64-macos-1015-build: <<: *cirrus_build_job_definition variables: NAME: macos-1015 + CIRRUS_VM_INSTANCE_TYPE: osx_instance + CIRRUS_VM_IMAGE_SELECTOR: image + CIRRUS_VM_IMAGE_NAME: catalina-base + INSTALL_COMMAND: brew install + PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin:/usr/local/opt/libpcap/bin:/usr/local/opt/libxslt/bin:/usr/local/opt/rpcgen/bin + PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/libpcap/lib/pkgconfig:/usr/local/opt/libxml2/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig # Cross compiled build jobs diff --git a/ci/cirrus/build.yml b/ci/cirrus/build.yml new file mode 100644 index 0000000000..893e13d724 --- /dev/null +++ b/ci/cirrus/build.yml @@ -0,0 +1,26 @@ +@CIRRUS_VM_INSTANCE_TYPE@: + @CIRRUS_VM_IMAGE_SELECTOR@: @CIRRUS_VM_IMAGE_NAME@ + +env: + CI_REPOSITORY_URL: "@CI_REPOSITORY_URL@" + CI_COMMIT_REF_NAME: "@CI_COMMIT_REF_NAME@" + CI_COMMIT_SHA: "@CI_COMMIT_SHA@" + PATH: "@PATH@" + PKG_CONFIG_PATH: "@PKG_CONFIG_PATH@" + PYTHON: "@PYTHON@" + MAKE: "@MAKE@" + +build_task: + install_script: + - @INSTALL_COMMAND@ @PKGS@ + clone_script: + - git clone --depth 100 "$CI_REPOSITORY_URL" . + - git fetch origin "$CI_COMMIT_REF_NAME" + - git reset --hard "$CI_COMMIT_SHA" + build_script: + - mkdir build + - cd build + - ../autogen.sh --prefix=$(pwd)/install-root + - $MAKE -j3 + - $MAKE -j3 install + - $MAKE -j3 dist diff --git a/ci/cirrus/freebsd-12.yml.j2 b/ci/cirrus/freebsd-12.yml.j2 deleted file mode 100644 index a653996850..0000000000 --- a/ci/cirrus/freebsd-12.yml.j2 +++ /dev/null @@ -1,73 +0,0 @@ -freebsd_instance: - image_family: freebsd-12-1 - -env: - CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }} - CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }} - CI_COMMIT_SHA: {{ CI_COMMIT_SHA }} - -freebsd_12_task: - install_script: - - pkg install -y - augeas - autoconf - automake - avahi - bash - bash-completion - ca_root_nss - ccache - chrony - cppi - curl - cyrus-sasl - dbus - diskscrub - dnsmasq - fusefs-libs - gdb - gettext - gettext-tools - git - glib - gmake - gnutls - hal - libpcap - libpciaccess - libssh - libssh2 - libtool - libxml2 - libxslt - lsof - meson - ncurses - ninja - patch - perl5 - pkgconf - polkit - py37-docutils - py37-flake8 - py37-setuptools - py37-wheel - python3 - qemu-utils - radvd - readline - screen - sudo - vim - yajl - clone_script: - - git clone --depth 100 "$CI_REPOSITORY_URL" . - - git fetch origin "$CI_COMMIT_REF_NAME" - - git reset --hard "$CI_COMMIT_SHA" - build_script: - - mkdir build - - cd build - - ../autogen.sh --prefix=$(pwd)/install-root - - gmake -j3 - - gmake -j3 install - - gmake -j3 dist diff --git a/ci/cirrus/libvirt-freebsd-12.vars b/ci/cirrus/libvirt-freebsd-12.vars new file mode 100644 index 0000000000..c7d4fbab0f --- /dev/null +++ b/ci/cirrus/libvirt-freebsd-12.vars @@ -0,0 +1,7 @@ +PACKAGING_COMMAND='pkg' +CC='/usr/bin/clang' +CCACHE='/usr/local/bin/ccache' +MAKE='/usr/local/bin/gmake' +NINJA='/usr/local/bin/ninja' +PYTHON='/usr/local/bin/python3' +PKGS='augeas autoconf automake avahi bash bash-completion ca_root_nss ccache chrony cppi curl cyrus-sasl dbus diskscrub dnsmasq fusefs-libs gdb gettext gettext-tools git glib gmake gnutls hal libpcap libpciaccess libssh libssh2 libtool libxml2 libxslt lsof meson ncurses ninja p5-App-cpanminus patch perl5 pkgconf polkit py37-docutils py37-flake8 py37-pip py37-setuptools py37-wheel python3 qemu-utils radvd readline screen sudo vim yajl' diff --git a/ci/cirrus/libvirt-macos-1015.vars b/ci/cirrus/libvirt-macos-1015.vars new file mode 100644 index 0000000000..a14c41a277 --- /dev/null +++ b/ci/cirrus/libvirt-macos-1015.vars @@ -0,0 +1,7 @@ +PACKAGING_COMMAND='brew' +CC='/usr/bin/clang' +CCACHE='/usr/local/bin/ccache' +MAKE='/usr/local/bin/gmake' +NINJA='/usr/local/bin/ninja' +PYTHON='/usr/local/bin/python3' +PKGS='augeas autoconf automake bash bash-completion ccache cpanminus cppi curl dbus dnsmasq docutils flake8 gdb gettext git glib gnutls gpatch libiscsi libpcap libssh libssh2 libtool libxml2 libxslt lsof make meson ncurses ninja perl pkg-config python3 qemu readline rpcgen screen scrub vim xz yajl' diff --git a/ci/cirrus/macos-1015.yml.j2 b/ci/cirrus/macos-1015.yml.j2 deleted file mode 100644 index d8aa9715d6..0000000000 --- a/ci/cirrus/macos-1015.yml.j2 +++ /dev/null @@ -1,38 +0,0 @@ -osx_instance: - image: catalina-base - -env: - CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }} - CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }} - CI_COMMIT_SHA: {{ CI_COMMIT_SHA }} - PATH: /usr/local/opt/gettext/bin:/usr/local/opt/ccache/libexec:/usr/local/opt/rpcgen/bin:$PATH - PKG_CONFIG_PATH: /usr/local/opt/libxml2/lib/pkgconfig - -macos_1015_task: - install_script: - - brew install - autoconf - automake - ccache - docutils - glib - gnutls - libtool - libxml2 - make - pkg-config - python - rpcgen - xz - yajl - clone_script: - - git clone --depth 100 "$CI_REPOSITORY_URL" . - - git fetch origin "$CI_COMMIT_REF_NAME" - - git reset --hard "$CI_COMMIT_SHA" - build_script: - - mkdir build - - cd build - - ../autogen.sh --prefix=$(pwd)/install-root - - gmake -j3 - - gmake -j3 install - - gmake -j3 dist diff --git a/ci/cirrus/refresh b/ci/cirrus/refresh new file mode 100755 index 0000000000..b84910a645 --- /dev/null +++ b/ci/cirrus/refresh @@ -0,0 +1,22 @@ +#!/bin/sh + +if test -z "$1" +then + echo "syntax: $0 PATH-TO-LCITOOL" + exit 1 +fi + +LCITOOL=$1 + +if ! test -x "$LCITOOL" +then + echo "$LCITOOL is not executable" + exit 1 +fi + +HOSTS=$($LCITOOL hosts | grep -E 'freebsd-12|macos') + +for host in $HOSTS +do + $LCITOOL dockerfile "$host" libvirt --variables >"$host.vars" +done -- 2.25.4
On Mon, Jun 29, 2020 at 08:51:51PM +0200, Andrea Bolognani wrote:
Instead of having static job definitions for FreeBSD and macOS, use a generic template for both and fill in the details that are actually different, such as the list of packages to install, in the GitLab CI job, right before calling cirrus-run.
The target-specific information are provided by lcitool, so that keeping them up to date is just a matter of running the refresh script when necessary.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- .gitlab-ci.yml | 36 ++++++++++++++- ci/cirrus/build.yml | 26 +++++++++++ ci/cirrus/freebsd-12.yml.j2 | 73 ------------------------------- ci/cirrus/libvirt-freebsd-12.vars | 7 +++ ci/cirrus/libvirt-macos-1015.vars | 7 +++ ci/cirrus/macos-1015.yml.j2 | 38 ---------------- ci/cirrus/refresh | 22 ++++++++++ 7 files changed, 97 insertions(+), 112 deletions(-) create mode 100644 ci/cirrus/build.yml delete mode 100644 ci/cirrus/freebsd-12.yml.j2 create mode 100644 ci/cirrus/libvirt-freebsd-12.vars create mode 100644 ci/cirrus/libvirt-macos-1015.vars delete mode 100644 ci/cirrus/macos-1015.yml.j2 create mode 100755 ci/cirrus/refresh
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index e6eb2f9905..5565750b7e 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -58,11 +58,35 @@ stages: # Jobs that we delegate to Cirrus CI because they require an operating # system other than Linux. These jobs will only run if the required # setup has been performed on the GitLab account (see ci/README.rst). +# +# The Cirrus CI configuration is generated by replacing target-specific +# variables in a generic template: some of these variables are provided +# when the GitLab CI job is defined, others are taken from a shell +# snippet generated using lcitool. +# +# Note that the $PATH environment variable has to be treated with +# special care, because we can't just override it at the GitLab CI job +# definition level or we risk breaking it completely.
Informative, thanks :). I just wish I didn't have to google what semantics the various types of parameter expansion in Bash had (obviously not your fault).
.cirrus_build_job_template: &cirrus_build_job_definition stage: builds image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master script: - - cirrus-run ci/cirrus/$NAME.yml.j2 + - source ci/cirrus/libvirt-$NAME.vars + - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g" + -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g" + -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g" + -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g" + -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g" + -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g" + -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g" + -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g" + -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g" + -e "s|[@]PKGS@|$PKGS|g" + -e "s|[@]MAKE@|$MAKE|g" + -e "s|[@]PYTHON@|$PYTHON|g" + <ci/cirrus/build.yml >ci/cirrus/$NAME.yml + - cat ci/cirrus/$NAME.yml
was ^this 'cat' just for debugging purposes or was that intended to be part of the job output in production? ...
diff --git a/ci/cirrus/refresh b/ci/cirrus/refresh new file mode 100755 index 0000000000..b84910a645 --- /dev/null +++ b/ci/cirrus/refresh @@ -0,0 +1,22 @@ +#!/bin/sh + +if test -z "$1" +then + echo "syntax: $0 PATH-TO-LCITOOL" + exit 1 +fi + +LCITOOL=$1 + +if ! test -x "$LCITOOL" +then + echo "$LCITOOL is not executable" + exit 1 +fi + +HOSTS=$($LCITOOL hosts | grep -E 'freebsd-12|macos') + +for host in $HOSTS +do + $LCITOOL dockerfile "$host" libvirt --variables >"$host.vars" +done
Overall, I prefer this to v1 because of the further level of abstraction this introduces. However, the final format we're producing here is a YAML template rather than a Dockerfile, so introducing just an option for the "dockerfile" subcommand rather than a dedicated "cirrus-ci-file" (or similar) subcommand doesn't feel completely right, especially from the long run perspective. At some point we'd like to generate the whole gitlab-ci.yml file, so that one would very likely get a dedicated subcommand as well - it should, as it's just another "plugin" generator. If we expand beyond libvirt, say QEMU, we may need to add a different CI provider plugin as well. Regards, Erik
On Tue, 2020-06-30 at 16:01 +0200, Erik Skultety wrote:
On Mon, Jun 29, 2020 at 08:51:51PM +0200, Andrea Bolognani wrote:
+# Note that the $PATH environment variable has to be treated with +# special care, because we can't just override it at the GitLab CI job +# definition level or we risk breaking it completely.
Informative, thanks :). I just wish I didn't have to google what semantics the various types of parameter expansion in Bash had (obviously not your fault).
I can assure you that I had to Google that while writing the code as well ;)
.cirrus_build_job_template: &cirrus_build_job_definition stage: builds image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master script: - - cirrus-run ci/cirrus/$NAME.yml.j2 + - source ci/cirrus/libvirt-$NAME.vars + - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g" + -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g" + -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g" + -e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g" + -e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g" + -e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g" + -e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g" + -e "s|[@]PATH@|$PATH_EXTRA${PATH_EXTRA:+:}\$PATH|g" + -e "s|[@]PKG_CONFIG_PATH@|$PKG_CONFIG_PATH|g" + -e "s|[@]PKGS@|$PKGS|g" + -e "s|[@]MAKE@|$MAKE|g" + -e "s|[@]PYTHON@|$PYTHON|g" + <ci/cirrus/build.yml >ci/cirrus/$NAME.yml + - cat ci/cirrus/$NAME.yml
was ^this 'cat' just for debugging purposes or was that intended to be part of the job output in production?
Initially it was for debugging purposes, but I didn't remove it before posting because I felt it could be useful to keep it there: unlike the other GitLab CI jobs, in this case there's an additional level of indirection and thus it's a bit harder to figure out what's going on by just looking at the bits that are in the repository. I could drop it, though.
+for host in $HOSTS +do + $LCITOOL dockerfile "$host" libvirt --variables >"$host.vars" +done
Overall, I prefer this to v1 because of the further level of abstraction this introduces. However, the final format we're producing here is a YAML template rather than a Dockerfile, so introducing just an option for the "dockerfile" subcommand rather than a dedicated "cirrus-ci-file" (or similar) subcommand doesn't feel completely right, especially from the long run perspective. At some point we'd like to generate the whole gitlab-ci.yml file, so that one would very likely get a dedicated subcommand as well - it should, as it's just another "plugin" generator. If we expand beyond libvirt, say QEMU, we may need to add a different CI provider plugin as well.
Right. The name was something that Dan also argued against in the libvirt-ci MR[1], and I'm not opposed to changing it. In that case, however, I would go for a more generic "variables" or something like that instead, because there's really nothing specific to Cirrus CI or any one CI service in the variables: they only describe the build environment the CI job will eventually run in. So even if we start introducing sub-commands specific to various CI services, a generic way to get just the variables themselves can be useful and, among other things, can serve as a way to integrate with CI services or local environments that we don't have dedicated support for. [1] https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/30 -- Andrea Bolognani / Red Hat / Virtualization
participants (2)
-
Andrea Bolognani -
Erik Skultety