[libvirt PATCH 0/3] cirrus: Add templates and refresh script
This makes the Cirrus CI configurations as maintainable as the Dockerfiles, by shifting the responsability of keeping the list of dependencies and other details up to date to lcitool. Test pipeline: https://gitlab.com/abologna/libvirt/-/pipelines/161179474 The important job is the macOS one. For the refresh script to work, your copy of lcitool needs to include these patches: https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/30 https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/31 Andrea Bolognani (3): cirrus: Tweak configurations cirrus: Add templates and refresh script cirrus: Refresh configurations ci/cirrus/freebsd-12.yml.j2 | 62 ++++-------------------------- ci/cirrus/macos-1015.yml.j2 | 29 +++++--------- ci/cirrus/refresh | 32 +++++++++++++++ ci/cirrus/templates/freebsd-12.yml | 25 ++++++++++++ ci/cirrus/templates/macos-1015.yml | 27 +++++++++++++ 5 files changed, 100 insertions(+), 75 deletions(-) create mode 100755 ci/cirrus/refresh create mode 100644 ci/cirrus/templates/freebsd-12.yml create mode 100644 ci/cirrus/templates/macos-1015.yml -- 2.25.4
Store the list of packages to install in an environment variable. This will make it easier to review the changes that are coming next. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- ci/cirrus/freebsd-12.yml.j2 | 54 ++----------------------------------- ci/cirrus/macos-1015.yml.j2 | 17 ++---------- 2 files changed, 4 insertions(+), 67 deletions(-) diff --git a/ci/cirrus/freebsd-12.yml.j2 b/ci/cirrus/freebsd-12.yml.j2 index a653996850..a93f94ee2a 100644 --- a/ci/cirrus/freebsd-12.yml.j2 +++ b/ci/cirrus/freebsd-12.yml.j2 @@ -5,61 +5,11 @@ env: CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }} CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }} CI_COMMIT_SHA: {{ CI_COMMIT_SHA }} + PKGS: augeas autoconf automake avahi bash bash-completion ca_root_nss ccache chrony cppi curl cyrus-sasl dbus diskscrub dnsmasq fusefs-libs gdb gettext gettext-tools git glib gmake gnutls hal libpcap libpciaccess libssh libssh2 libtool libxml2 libxslt lsof meson ncurses ninja patch perl5 pkgconf polkit py37-docutils py37-flake8 py37-setuptools py37-wheel python3 qemu-utils radvd readline screen sudo vim yajl freebsd_12_task: install_script: - - pkg install -y - augeas - autoconf - automake - avahi - bash - bash-completion - ca_root_nss - ccache - chrony - cppi - curl - cyrus-sasl - dbus - diskscrub - dnsmasq - fusefs-libs - gdb - gettext - gettext-tools - git - glib - gmake - gnutls - hal - libpcap - libpciaccess - libssh - libssh2 - libtool - libxml2 - libxslt - lsof - meson - ncurses - ninja - patch - perl5 - pkgconf - polkit - py37-docutils - py37-flake8 - py37-setuptools - py37-wheel - python3 - qemu-utils - radvd - readline - screen - sudo - vim - yajl + - pkg install -y $PKGS clone_script: - git clone --depth 100 "$CI_REPOSITORY_URL" . - git fetch origin "$CI_COMMIT_REF_NAME" diff --git a/ci/cirrus/macos-1015.yml.j2 b/ci/cirrus/macos-1015.yml.j2 index d8aa9715d6..0d3d54a97a 100644 --- a/ci/cirrus/macos-1015.yml.j2 +++ b/ci/cirrus/macos-1015.yml.j2 @@ -7,24 +7,11 @@ env: CI_COMMIT_SHA: {{ CI_COMMIT_SHA }} PATH: /usr/local/opt/gettext/bin:/usr/local/opt/ccache/libexec:/usr/local/opt/rpcgen/bin:$PATH PKG_CONFIG_PATH: /usr/local/opt/libxml2/lib/pkgconfig + PKGS: autoconf automake ccache docutils glib gnutls libtool libxml2 make pkg-config python rpcgen xz yajl macos_1015_task: install_script: - - brew install - autoconf - automake - ccache - docutils - glib - gnutls - libtool - libxml2 - make - pkg-config - python - rpcgen - xz - yajl + - brew install $PKGS clone_script: - git clone --depth 100 "$CI_REPOSITORY_URL" . - git fetch origin "$CI_COMMIT_REF_NAME" -- 2.25.4
This is similar to what we already use for Dockerfiles, with one key difference: while we still rely on lcitool taking care of the complicated work for us, in this case we're only provided with a bunch of variables and we have to do the last bit of work (that is replacing them inside an existing template) ourselves. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- ci/cirrus/refresh | 32 ++++++++++++++++++++++++++++++ ci/cirrus/templates/freebsd-12.yml | 25 +++++++++++++++++++++++ ci/cirrus/templates/macos-1015.yml | 27 +++++++++++++++++++++++++ 3 files changed, 84 insertions(+) create mode 100755 ci/cirrus/refresh create mode 100644 ci/cirrus/templates/freebsd-12.yml create mode 100644 ci/cirrus/templates/macos-1015.yml diff --git a/ci/cirrus/refresh b/ci/cirrus/refresh new file mode 100755 index 0000000000..51deca94ac --- /dev/null +++ b/ci/cirrus/refresh @@ -0,0 +1,32 @@ +#!/bin/sh + +if test -z "$1" +then + echo "syntax: $0 PATH-TO-LCITOOL" + exit 1 +fi + +LCITOOL=$1 + +if ! test -x "$LCITOOL" +then + echo "$LCITOOL is not executable" + exit 1 +fi + +for infile in templates/* +do + outfile="${infile##*/}.j2" + host="${outfile%%.*}" + + eval $("$LCITOOL" dockerfile "libvirt-$host" libvirt --variables) + + sed -e "s|[@]PKGS@|$PKGS|g" \ + -e "s|[@]CROSS_PKGS@|$CROSS_PKGS|g" \ + -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g" \ + -e "s|[@]CPAN_PKGS@|$CPAN_PKGS|g" \ + -e "s|[@]MAKE@|$MAKE|g" \ + -e "s|[@]NINJA@|$NINJA|g" \ + -e "s|[@]PYTHON@|$PYTHON|g" \ + <"$infile" >"$outfile" +done diff --git a/ci/cirrus/templates/freebsd-12.yml b/ci/cirrus/templates/freebsd-12.yml new file mode 100644 index 0000000000..228ea67827 --- /dev/null +++ b/ci/cirrus/templates/freebsd-12.yml @@ -0,0 +1,25 @@ +freebsd_instance: + image_family: freebsd-12-1 + +env: + CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }} + CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }} + CI_COMMIT_SHA: {{ CI_COMMIT_SHA }} + PKGS: @PKGS@ + MAKE: @MAKE@ + PYTHON: @PYTHON@ + +freebsd_12_task: + install_script: + - pkg install -y $PKGS + clone_script: + - git clone --depth 100 "$CI_REPOSITORY_URL" . + - git fetch origin "$CI_COMMIT_REF_NAME" + - git reset --hard "$CI_COMMIT_SHA" + build_script: + - mkdir build + - cd build + - ../autogen.sh --prefix=$(pwd)/install-root + - $MAKE -j3 + - $MAKE -j3 install + - $MAKE -j3 dist diff --git a/ci/cirrus/templates/macos-1015.yml b/ci/cirrus/templates/macos-1015.yml new file mode 100644 index 0000000000..5fba3c635b --- /dev/null +++ b/ci/cirrus/templates/macos-1015.yml @@ -0,0 +1,27 @@ +osx_instance: + image: catalina-base + +env: + CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }} + CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }} + CI_COMMIT_SHA: {{ CI_COMMIT_SHA }} + PATH: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin:/usr/local/opt/libpcap/bin:/usr/local/opt/libxslt/bin:/usr/local/opt/rpcgen/bin:$PATH + PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/libpcap/lib/pkgconfig:/usr/local/opt/libxml2/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig + PKGS: @PKGS@ + MAKE: @MAKE@ + PYTHON: @PYTHON@ + +macos_1015_task: + install_script: + - brew install $PKGS + clone_script: + - git clone --depth 100 "$CI_REPOSITORY_URL" . + - git fetch origin "$CI_COMMIT_REF_NAME" + - git reset --hard "$CI_COMMIT_SHA" + build_script: + - mkdir build + - cd build + - ../autogen.sh --prefix=$(pwd)/install-root + - $MAKE -j3 + - $MAKE -j3 install + - $MAKE -j3 dist -- 2.25.4
On Mon, Jun 29, 2020 at 03:58:43PM +0200, Andrea Bolognani wrote:
This is similar to what we already use for Dockerfiles, with one key difference: while we still rely on lcitool taking care of the complicated work for us, in this case we're only provided with a bunch of variables and we have to do the last bit of work (that is replacing them inside an existing template) ourselves.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- ci/cirrus/refresh | 32 ++++++++++++++++++++++++++++++ ci/cirrus/templates/freebsd-12.yml | 25 +++++++++++++++++++++++ ci/cirrus/templates/macos-1015.yml | 27 +++++++++++++++++++++++++
These files are largely the identical content to the same name files in the dir above which feels like it is easy to optimize away.
3 files changed, 84 insertions(+) create mode 100755 ci/cirrus/refresh create mode 100644 ci/cirrus/templates/freebsd-12.yml create mode 100644 ci/cirrus/templates/macos-1015.yml
diff --git a/ci/cirrus/refresh b/ci/cirrus/refresh new file mode 100755 index 0000000000..51deca94ac --- /dev/null +++ b/ci/cirrus/refresh @@ -0,0 +1,32 @@ +#!/bin/sh + +if test -z "$1" +then + echo "syntax: $0 PATH-TO-LCITOOL" + exit 1 +fi + +LCITOOL=$1 + +if ! test -x "$LCITOOL" +then + echo "$LCITOOL is not executable" + exit 1 +fi + +for infile in templates/* +do + outfile="${infile##*/}.j2" + host="${outfile%%.*}" + + eval $("$LCITOOL" dockerfile "libvirt-$host" libvirt --variables) + + sed -e "s|[@]PKGS@|$PKGS|g" \ + -e "s|[@]CROSS_PKGS@|$CROSS_PKGS|g" \ + -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g" \ + -e "s|[@]CPAN_PKGS@|$CPAN_PKGS|g" \ + -e "s|[@]MAKE@|$MAKE|g" \ + -e "s|[@]NINJA@|$NINJA|g" \ + -e "s|[@]PYTHON@|$PYTHON|g" \ + <"$infile" >"$outfile" +done
I feel like this should really be reduced to just: $LCITOOL cirrusci libvirt-$host libvirt > $outfile where the 'cirrusci' command generates this content....
diff --git a/ci/cirrus/templates/freebsd-12.yml b/ci/cirrus/templates/freebsd-12.yml new file mode 100644 index 0000000000..228ea67827 --- /dev/null +++ b/ci/cirrus/templates/freebsd-12.yml @@ -0,0 +1,25 @@ +freebsd_instance: + image_family: freebsd-12-1 + +env: + CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }} + CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }} + CI_COMMIT_SHA: {{ CI_COMMIT_SHA }} + PKGS: @PKGS@ + MAKE: @MAKE@ + PYTHON: @PYTHON@ + +freebsd_12_task: + install_script: + - pkg install -y $PKGS + clone_script: + - git clone --depth 100 "$CI_REPOSITORY_URL" . + - git fetch origin "$CI_COMMIT_REF_NAME" + - git reset --hard "$CI_COMMIT_SHA"
...down to this point.
+ build_script: + - mkdir build + - cd build + - ../autogen.sh --prefix=$(pwd)/install-root + - $MAKE -j3 + - $MAKE -j3 install + - $MAKE -j3 dist
This part can then be stored in ci/cirrus/build.yml since it is common to freebsd & macos. So now in gitlab-ci.yml we can just concatenate the two into a temp file: cat ci/cirrus/$NAME.yml ci/cirrus/build.yml > ci/cirrus/$NAME.yml.j2 cirrus-run ci/cirrus/$NAME.yml.j2 This avoids needing to store the same information in git twice. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, Jun 29, 2020 at 03:32:22PM +0100, Daniel P. Berrangé wrote:
On Mon, Jun 29, 2020 at 03:58:43PM +0200, Andrea Bolognani wrote:
This is similar to what we already use for Dockerfiles, with one key difference: while we still rely on lcitool taking care of the complicated work for us, in this case we're only provided with a bunch of variables and we have to do the last bit of work (that is replacing them inside an existing template) ourselves.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- ci/cirrus/refresh | 32 ++++++++++++++++++++++++++++++ ci/cirrus/templates/freebsd-12.yml | 25 +++++++++++++++++++++++ ci/cirrus/templates/macos-1015.yml | 27 +++++++++++++++++++++++++
These files are largely the identical content to the same name files in the dir above which feels like it is easy to optimize away.
3 files changed, 84 insertions(+) create mode 100755 ci/cirrus/refresh create mode 100644 ci/cirrus/templates/freebsd-12.yml create mode 100644 ci/cirrus/templates/macos-1015.yml
diff --git a/ci/cirrus/refresh b/ci/cirrus/refresh new file mode 100755 index 0000000000..51deca94ac --- /dev/null +++ b/ci/cirrus/refresh @@ -0,0 +1,32 @@ +#!/bin/sh + +if test -z "$1" +then + echo "syntax: $0 PATH-TO-LCITOOL" + exit 1 +fi + +LCITOOL=$1 + +if ! test -x "$LCITOOL" +then + echo "$LCITOOL is not executable" + exit 1 +fi + +for infile in templates/* +do + outfile="${infile##*/}.j2" + host="${outfile%%.*}" + + eval $("$LCITOOL" dockerfile "libvirt-$host" libvirt --variables) + + sed -e "s|[@]PKGS@|$PKGS|g" \ + -e "s|[@]CROSS_PKGS@|$CROSS_PKGS|g" \ + -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g" \ + -e "s|[@]CPAN_PKGS@|$CPAN_PKGS|g" \ + -e "s|[@]MAKE@|$MAKE|g" \ + -e "s|[@]NINJA@|$NINJA|g" \ + -e "s|[@]PYTHON@|$PYTHON|g" \ + <"$infile" >"$outfile" +done
I feel like this should really be reduced to just:
$LCITOOL cirrusci libvirt-$host libvirt > $outfile
where the 'cirrusci' command generates this content....
I agree with the idea of having a new command generating the content below, I tend to look at this as different generator plugins/backends that we support, so having a dedicated command makes much more sense to me.
diff --git a/ci/cirrus/templates/freebsd-12.yml b/ci/cirrus/templates/freebsd-12.yml new file mode 100644 index 0000000000..228ea67827 --- /dev/null +++ b/ci/cirrus/templates/freebsd-12.yml @@ -0,0 +1,25 @@ +freebsd_instance: + image_family: freebsd-12-1 + +env: + CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }} + CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }} + CI_COMMIT_SHA: {{ CI_COMMIT_SHA }} + PKGS: @PKGS@ + MAKE: @MAKE@ + PYTHON: @PYTHON@ + +freebsd_12_task: + install_script: + - pkg install -y $PKGS + clone_script: + - git clone --depth 100 "$CI_REPOSITORY_URL" . + - git fetch origin "$CI_COMMIT_REF_NAME" + - git reset --hard "$CI_COMMIT_SHA"
...down to this point.
+ build_script: + - mkdir build + - cd build + - ../autogen.sh --prefix=$(pwd)/install-root + - $MAKE -j3 + - $MAKE -j3 install + - $MAKE -j3 dist
This part can then be stored in ci/cirrus/build.yml since it is common to freebsd & macos.
So now in gitlab-ci.yml we can just concatenate the two into a temp file:
cat ci/cirrus/$NAME.yml ci/cirrus/build.yml > ci/cirrus/$NAME.yml.j2 cirrus-run ci/cirrus/$NAME.yml.j2
How about instead of storing the same build.yml in every secondary project's ci directory we store the build.yml in libvirt-ci repo and generate the whole thing with lcitool? I think the way the build YAML would look like for every single subproject, we could achieve the level of similarity we have for lcitool native builds - IOW the build job abstraction we have across projects under guests/playbooks/build/projects could also be achieved here with cirrus, just an idea. Erik
On Mon, Jun 29, 2020 at 06:08:27PM +0200, Erik Skultety wrote:
On Mon, Jun 29, 2020 at 03:32:22PM +0100, Daniel P. Berrangé wrote:
On Mon, Jun 29, 2020 at 03:58:43PM +0200, Andrea Bolognani wrote:
This is similar to what we already use for Dockerfiles, with one key difference: while we still rely on lcitool taking care of the complicated work for us, in this case we're only provided with a bunch of variables and we have to do the last bit of work (that is replacing them inside an existing template) ourselves.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- ci/cirrus/refresh | 32 ++++++++++++++++++++++++++++++ ci/cirrus/templates/freebsd-12.yml | 25 +++++++++++++++++++++++ ci/cirrus/templates/macos-1015.yml | 27 +++++++++++++++++++++++++
diff --git a/ci/cirrus/templates/freebsd-12.yml b/ci/cirrus/templates/freebsd-12.yml new file mode 100644 index 0000000000..228ea67827 --- /dev/null +++ b/ci/cirrus/templates/freebsd-12.yml @@ -0,0 +1,25 @@ +freebsd_instance: + image_family: freebsd-12-1 + +env: + CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }} + CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }} + CI_COMMIT_SHA: {{ CI_COMMIT_SHA }} + PKGS: @PKGS@ + MAKE: @MAKE@ + PYTHON: @PYTHON@ + +freebsd_12_task: + install_script: + - pkg install -y $PKGS + clone_script: + - git clone --depth 100 "$CI_REPOSITORY_URL" . + - git fetch origin "$CI_COMMIT_REF_NAME" + - git reset --hard "$CI_COMMIT_SHA"
...down to this point.
+ build_script: + - mkdir build + - cd build + - ../autogen.sh --prefix=$(pwd)/install-root + - $MAKE -j3 + - $MAKE -j3 install + - $MAKE -j3 dist
This part can then be stored in ci/cirrus/build.yml since it is common to freebsd & macos.
So now in gitlab-ci.yml we can just concatenate the two into a temp file:
cat ci/cirrus/$NAME.yml ci/cirrus/build.yml > ci/cirrus/$NAME.yml.j2 cirrus-run ci/cirrus/$NAME.yml.j2
How about instead of storing the same build.yml in every secondary project's ci directory we store the build.yml in libvirt-ci repo and generate the whole thing with lcitool? I think the way the build YAML would look like for every single subproject, we could achieve the level of similarity we have for lcitool native builds - IOW the build job abstraction we have across projects under guests/playbooks/build/projects could also be achieved here with cirrus, just an idea.
Yep, I hesitated to suggest that, as I thought it might need boiling the ocean in the short, whereas the other pieces are 100% static data we can trivially emit with next to no effort. Long term if we can generate the build commands too, that could be nice though. I'd even think about the possibility of generating the main gitlab-ci.yml file to some degree, though at some point you spend more time in the generator than on the original file, so I'm not sure where the line is. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, 2020-06-29 at 17:35 +0100, Daniel P. Berrangé wrote:
On Mon, Jun 29, 2020 at 06:08:27PM +0200, Erik Skultety wrote:
On Mon, Jun 29, 2020 at 03:32:22PM +0100, Daniel P. Berrangé wrote:
This part can then be stored in ci/cirrus/build.yml since it is common to freebsd & macos.
So now in gitlab-ci.yml we can just concatenate the two into a temp file:
cat ci/cirrus/$NAME.yml ci/cirrus/build.yml > ci/cirrus/$NAME.yml.j2 cirrus-run ci/cirrus/$NAME.yml.j2
How about instead of storing the same build.yml in every secondary project's ci directory we store the build.yml in libvirt-ci repo and generate the whole thing with lcitool? I think the way the build YAML would look like for every single subproject, we could achieve the level of similarity we have for lcitool native builds - IOW the build job abstraction we have across projects under guests/playbooks/build/projects could also be achieved here with cirrus, just an idea.
Yep, I hesitated to suggest that, as I thought it might need boiling the ocean in the short, whereas the other pieces are 100% static data we can trivially emit with next to no effort. Long term if we can generate the build commands too, that could be nice though. I'd even think about the possibility of generating the main gitlab-ci.yml file to some degree, though at some point you spend more time in the generator than on the original file, so I'm not sure where the line is.
I agree that it would be incredibly neat if we could take the same information used by 'lcitool build' and generate at least part of the various .gitlab-ci.yml files. However, I think we should *not* attempt something like that until later, when we have enabled full GitLab CI support (including, where applicable, FreeBSD and macOS) for all projects and we can hopefully see the proper path forward more clearly thanks to the insights gained in the process. -- Andrea Bolognani / Red Hat / Virtualization
On Mon, 2020-06-29 at 15:32 +0100, Daniel P. Berrangé wrote:
On Mon, Jun 29, 2020 at 03:58:43PM +0200, Andrea Bolognani wrote:
+for infile in templates/* +do + outfile="${infile##*/}.j2" + host="${outfile%%.*}" + + eval $("$LCITOOL" dockerfile "libvirt-$host" libvirt --variables) + + sed -e "s|[@]PKGS@|$PKGS|g" \ + -e "s|[@]CROSS_PKGS@|$CROSS_PKGS|g" \ + -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g" \ + -e "s|[@]CPAN_PKGS@|$CPAN_PKGS|g" \ + -e "s|[@]MAKE@|$MAKE|g" \ + -e "s|[@]NINJA@|$NINJA|g" \ + -e "s|[@]PYTHON@|$PYTHON|g" \ + <"$infile" >"$outfile" +done
I feel like this should really be reduced to just:
$LCITOOL cirrusci libvirt-$host libvirt > $outfile
+ build_script: + - mkdir build + - cd build + - ../autogen.sh --prefix=$(pwd)/install-root + - $MAKE -j3 + - $MAKE -j3 install + - $MAKE -j3 dist
This part can then be stored in ci/cirrus/build.yml since it is common to freebsd & macos.
So now in gitlab-ci.yml we can just concatenate the two into a temp file:
cat ci/cirrus/$NAME.yml ci/cirrus/build.yml > ci/cirrus/$NAME.yml.j2 cirrus-run ci/cirrus/$NAME.yml.j2
This avoids needing to store the same information in git twice.
Yeah, I don't like having to store the information twice in git, but I'm also not a fan of having an lcitool subcommand that is tightly coupled with the details of a specific CI service rather than dealing with the more general problem of managing build environments. I think I have an approach that can sit somewhere in the middle. Let me give it a try. -- Andrea Bolognani / Red Hat / Virtualization
On Mon, Jun 29, 2020 at 06:27:40PM +0200, Andrea Bolognani wrote:
On Mon, 2020-06-29 at 15:32 +0100, Daniel P. Berrangé wrote:
On Mon, Jun 29, 2020 at 03:58:43PM +0200, Andrea Bolognani wrote:
+for infile in templates/* +do + outfile="${infile##*/}.j2" + host="${outfile%%.*}" + + eval $("$LCITOOL" dockerfile "libvirt-$host" libvirt --variables) + + sed -e "s|[@]PKGS@|$PKGS|g" \ + -e "s|[@]CROSS_PKGS@|$CROSS_PKGS|g" \ + -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g" \ + -e "s|[@]CPAN_PKGS@|$CPAN_PKGS|g" \ + -e "s|[@]MAKE@|$MAKE|g" \ + -e "s|[@]NINJA@|$NINJA|g" \ + -e "s|[@]PYTHON@|$PYTHON|g" \ + <"$infile" >"$outfile" +done
I feel like this should really be reduced to just:
$LCITOOL cirrusci libvirt-$host libvirt > $outfile
+ build_script: + - mkdir build + - cd build + - ../autogen.sh --prefix=$(pwd)/install-root + - $MAKE -j3 + - $MAKE -j3 install + - $MAKE -j3 dist
This part can then be stored in ci/cirrus/build.yml since it is common to freebsd & macos.
So now in gitlab-ci.yml we can just concatenate the two into a temp file:
cat ci/cirrus/$NAME.yml ci/cirrus/build.yml > ci/cirrus/$NAME.yml.j2 cirrus-run ci/cirrus/$NAME.yml.j2
This avoids needing to store the same information in git twice.
Yeah, I don't like having to store the information twice in git, but I'm also not a fan of having an lcitool subcommand that is tightly coupled with the details of a specific CI service rather than dealing with the more general problem of managing build environments.
Hmm, I kind of feel that's exactly the job of lcitool. Generate whatever files/resources we need for integrating with whatever CI infra we're choosing to use. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
The information is coming from lcitool now, so it's more complete: this is immediately apparent when looking at the number of additional packages that end up being installed on macOS. This commit is best viewed with 'git show --word-diff'. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- ci/cirrus/freebsd-12.yml.j2 | 10 ++++++---- ci/cirrus/macos-1015.yml.j2 | 14 ++++++++------ 2 files changed, 14 insertions(+), 10 deletions(-) diff --git a/ci/cirrus/freebsd-12.yml.j2 b/ci/cirrus/freebsd-12.yml.j2 index a93f94ee2a..80083c65d7 100644 --- a/ci/cirrus/freebsd-12.yml.j2 +++ b/ci/cirrus/freebsd-12.yml.j2 @@ -5,7 +5,9 @@ env: CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }} CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }} CI_COMMIT_SHA: {{ CI_COMMIT_SHA }} - PKGS: augeas autoconf automake avahi bash bash-completion ca_root_nss ccache chrony cppi curl cyrus-sasl dbus diskscrub dnsmasq fusefs-libs gdb gettext gettext-tools git glib gmake gnutls hal libpcap libpciaccess libssh libssh2 libtool libxml2 libxslt lsof meson ncurses ninja patch perl5 pkgconf polkit py37-docutils py37-flake8 py37-setuptools py37-wheel python3 qemu-utils radvd readline screen sudo vim yajl + PKGS: augeas autoconf automake avahi bash bash-completion ca_root_nss ccache chrony cppi curl cyrus-sasl dbus diskscrub dnsmasq fusefs-libs gdb gettext gettext-tools git glib gmake gnutls hal libpcap libpciaccess libssh libssh2 libtool libxml2 libxslt lsof meson ncurses ninja p5-App-cpanminus patch perl5 pkgconf polkit py37-docutils py37-flake8 py37-pip py37-setuptools py37-wheel python3 qemu-utils radvd readline screen sudo vim yajl + MAKE: /usr/local/bin/gmake + PYTHON: /usr/local/bin/python3 freebsd_12_task: install_script: @@ -18,6 +20,6 @@ freebsd_12_task: - mkdir build - cd build - ../autogen.sh --prefix=$(pwd)/install-root - - gmake -j3 - - gmake -j3 install - - gmake -j3 dist + - $MAKE -j3 + - $MAKE -j3 install + - $MAKE -j3 dist diff --git a/ci/cirrus/macos-1015.yml.j2 b/ci/cirrus/macos-1015.yml.j2 index 0d3d54a97a..1a4e942613 100644 --- a/ci/cirrus/macos-1015.yml.j2 +++ b/ci/cirrus/macos-1015.yml.j2 @@ -5,9 +5,11 @@ env: CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }} CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }} CI_COMMIT_SHA: {{ CI_COMMIT_SHA }} - PATH: /usr/local/opt/gettext/bin:/usr/local/opt/ccache/libexec:/usr/local/opt/rpcgen/bin:$PATH - PKG_CONFIG_PATH: /usr/local/opt/libxml2/lib/pkgconfig - PKGS: autoconf automake ccache docutils glib gnutls libtool libxml2 make pkg-config python rpcgen xz yajl + PATH: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin:/usr/local/opt/libpcap/bin:/usr/local/opt/libxslt/bin:/usr/local/opt/rpcgen/bin:$PATH + PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/libpcap/lib/pkgconfig:/usr/local/opt/libxml2/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig + PKGS: augeas autoconf automake bash bash-completion ccache cpanminus cppi curl dbus dnsmasq docutils flake8 gdb gettext git glib gnutls gpatch libiscsi libpcap libssh libssh2 libtool libxml2 libxslt lsof make meson ncurses ninja perl pkg-config python3 qemu readline rpcgen screen scrub vim xz yajl + MAKE: /usr/local/bin/gmake + PYTHON: /usr/local/bin/python3 macos_1015_task: install_script: @@ -20,6 +22,6 @@ macos_1015_task: - mkdir build - cd build - ../autogen.sh --prefix=$(pwd)/install-root - - gmake -j3 - - gmake -j3 install - - gmake -j3 dist + - $MAKE -j3 + - $MAKE -j3 install + - $MAKE -j3 dist -- 2.25.4
participants (3)
-
Andrea Bolognani -
Daniel P. Berrangé -
Erik Skultety