8:58 a.m.
This makes the Cirrus CI configurations as maintainable as the
Dockerfiles, by shifting the responsability of keeping the list
of dependencies and other details up to date to lcitool.
Test pipeline:
https://gitlab.com/abologna/libvirt/-/pipelines/161179474
The important job is the macOS one.
For the refresh script to work, your copy of lcitool needs to
include these patches:
https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/30
https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/31
Andrea Bolognani (3):
cirrus: Tweak configurations
cirrus: Add templates and refresh script
cirrus: Refresh configurations
ci/cirrus/freebsd-12.yml.j2 | 62 ++++--------------------------
ci/cirrus/macos-1015.yml.j2 | 29 +++++---------
ci/cirrus/refresh | 32 +++++++++++++++
ci/cirrus/templates/freebsd-12.yml | 25 ++++++++++++
ci/cirrus/templates/macos-1015.yml | 27 +++++++++++++
5 files changed, 100 insertions(+), 75 deletions(-)
create mode 100755 ci/cirrus/refresh
create mode 100644 ci/cirrus/templates/freebsd-12.yml
create mode 100644 ci/cirrus/templates/macos-1015.yml
--
2.25.4
8:58 a.m.
New subject: [libvirt PATCH 1/3] cirrus: Tweak configurations
Store the list of packages to install in an environment variable.
This will make it easier to review the changes that are coming
next.
Signed-off-by: Andrea Bolognani <abologna(a)redhat.com>
---
ci/cirrus/freebsd-12.yml.j2 | 54 ++-----------------------------------
ci/cirrus/macos-1015.yml.j2 | 17 ++----------
2 files changed, 4 insertions(+), 67 deletions(-)
diff --git a/ci/cirrus/freebsd-12.yml.j2 b/ci/cirrus/freebsd-12.yml.j2
index a653996850..a93f94ee2a 100644
--- a/ci/cirrus/freebsd-12.yml.j2
+++ b/ci/cirrus/freebsd-12.yml.j2
@@ -5,61 +5,11 @@ env:
CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }}
CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }}
CI_COMMIT_SHA: {{ CI_COMMIT_SHA }}
+ PKGS: augeas autoconf automake avahi bash bash-completion ca_root_nss ccache chrony
cppi curl cyrus-sasl dbus diskscrub dnsmasq fusefs-libs gdb gettext gettext-tools git glib
gmake gnutls hal libpcap libpciaccess libssh libssh2 libtool libxml2 libxslt lsof meson
ncurses ninja patch perl5 pkgconf polkit py37-docutils py37-flake8 py37-setuptools
py37-wheel python3 qemu-utils radvd readline screen sudo vim yajl
freebsd_12_task:
install_script:
- - pkg install -y
- augeas
- autoconf
- automake
- avahi
- bash
- bash-completion
- ca_root_nss
- ccache
- chrony
- cppi
- curl
- cyrus-sasl
- dbus
- diskscrub
- dnsmasq
- fusefs-libs
- gdb
- gettext
- gettext-tools
- git
- glib
- gmake
- gnutls
- hal
- libpcap
- libpciaccess
- libssh
- libssh2
- libtool
- libxml2
- libxslt
- lsof
- meson
- ncurses
- ninja
- patch
- perl5
- pkgconf
- polkit
- py37-docutils
- py37-flake8
- py37-setuptools
- py37-wheel
- python3
- qemu-utils
- radvd
- readline
- screen
- sudo
- vim
- yajl
+ - pkg install -y $PKGS
clone_script:
- git clone --depth 100 "$CI_REPOSITORY_URL" .
- git fetch origin "$CI_COMMIT_REF_NAME"
diff --git a/ci/cirrus/macos-1015.yml.j2 b/ci/cirrus/macos-1015.yml.j2
index d8aa9715d6..0d3d54a97a 100644
--- a/ci/cirrus/macos-1015.yml.j2
+++ b/ci/cirrus/macos-1015.yml.j2
@@ -7,24 +7,11 @@ env:
CI_COMMIT_SHA: {{ CI_COMMIT_SHA }}
PATH:
/usr/local/opt/gettext/bin:/usr/local/opt/ccache/libexec:/usr/local/opt/rpcgen/bin:$PATH
PKG_CONFIG_PATH: /usr/local/opt/libxml2/lib/pkgconfig
+ PKGS: autoconf automake ccache docutils glib gnutls libtool libxml2 make pkg-config
python rpcgen xz yajl
macos_1015_task:
install_script:
- - brew install
- autoconf
- automake
- ccache
- docutils
- glib
- gnutls
- libtool
- libxml2
- make
- pkg-config
- python
- rpcgen
- xz
- yajl
+ - brew install $PKGS
clone_script:
- git clone --depth 100 "$CI_REPOSITORY_URL" .
- git fetch origin "$CI_COMMIT_REF_NAME"
--
2.25.4
8:58 a.m.
New subject: [libvirt PATCH 2/3] cirrus: Add templates and refresh script
This is similar to what we already use for Dockerfiles, with one
key difference: while we still rely on lcitool taking care of the
complicated work for us, in this case we're only provided with a
bunch of variables and we have to do the last bit of work (that
is replacing them inside an existing template) ourselves.
Signed-off-by: Andrea Bolognani <abologna(a)redhat.com>
---
ci/cirrus/refresh | 32 ++++++++++++++++++++++++++++++
ci/cirrus/templates/freebsd-12.yml | 25 +++++++++++++++++++++++
ci/cirrus/templates/macos-1015.yml | 27 +++++++++++++++++++++++++
3 files changed, 84 insertions(+)
create mode 100755 ci/cirrus/refresh
create mode 100644 ci/cirrus/templates/freebsd-12.yml
create mode 100644 ci/cirrus/templates/macos-1015.yml
diff --git a/ci/cirrus/refresh b/ci/cirrus/refresh
new file mode 100755
index 0000000000..51deca94ac
--- /dev/null
+++ b/ci/cirrus/refresh
@@ -0,0 +1,32 @@
+#!/bin/sh
+
+if test -z "$1"
+then
+ echo "syntax: $0 PATH-TO-LCITOOL"
+ exit 1
+fi
+
+LCITOOL=$1
+
+if ! test -x "$LCITOOL"
+then
+ echo "$LCITOOL is not executable"
+ exit 1
+fi
+
+for infile in templates/*
+do
+ outfile="${infile##*/}.j2"
+ host="${outfile%%.*}"
+
+ eval $("$LCITOOL" dockerfile "libvirt-$host" libvirt
--variables)
+
+ sed -e "s|[@]PKGS@|$PKGS|g" \
+ -e "s|[@]CROSS_PKGS@|$CROSS_PKGS|g" \
+ -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g" \
+ -e "s|[@]CPAN_PKGS@|$CPAN_PKGS|g" \
+ -e "s|[@]MAKE@|$MAKE|g" \
+ -e "s|[@]NINJA@|$NINJA|g" \
+ -e "s|[@]PYTHON@|$PYTHON|g" \
+ <"$infile" >"$outfile"
+done
diff --git a/ci/cirrus/templates/freebsd-12.yml b/ci/cirrus/templates/freebsd-12.yml
new file mode 100644
index 0000000000..228ea67827
--- /dev/null
+++ b/ci/cirrus/templates/freebsd-12.yml
@@ -0,0 +1,25 @@
+freebsd_instance:
+ image_family: freebsd-12-1
+
+env:
+ CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }}
+ CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }}
+ CI_COMMIT_SHA: {{ CI_COMMIT_SHA }}
+ PKGS: @PKGS@
+ MAKE: @MAKE@
+ PYTHON: @PYTHON@
+
+freebsd_12_task:
+ install_script:
+ - pkg install -y $PKGS
+ clone_script:
+ - git clone --depth 100 "$CI_REPOSITORY_URL" .
+ - git fetch origin "$CI_COMMIT_REF_NAME"
+ - git reset --hard "$CI_COMMIT_SHA"
+ build_script:
+ - mkdir build
+ - cd build
+ - ../autogen.sh --prefix=$(pwd)/install-root
+ - $MAKE -j3
+ - $MAKE -j3 install
+ - $MAKE -j3 dist
diff --git a/ci/cirrus/templates/macos-1015.yml b/ci/cirrus/templates/macos-1015.yml
new file mode 100644
index 0000000000..5fba3c635b
--- /dev/null
+++ b/ci/cirrus/templates/macos-1015.yml
@@ -0,0 +1,27 @@
+osx_instance:
+ image: catalina-base
+
+env:
+ CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }}
+ CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }}
+ CI_COMMIT_SHA: {{ CI_COMMIT_SHA }}
+ PATH:
/usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin:/usr/local/opt/libpcap/bin:/usr/local/opt/libxslt/bin:/usr/local/opt/rpcgen/bin:$PATH
+ PKG_CONFIG_PATH:
/usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/libpcap/lib/pkgconfig:/usr/local/opt/libxml2/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
+ PKGS: @PKGS@
+ MAKE: @MAKE@
+ PYTHON: @PYTHON@
+
+macos_1015_task:
+ install_script:
+ - brew install $PKGS
+ clone_script:
+ - git clone --depth 100 "$CI_REPOSITORY_URL" .
+ - git fetch origin "$CI_COMMIT_REF_NAME"
+ - git reset --hard "$CI_COMMIT_SHA"
+ build_script:
+ - mkdir build
+ - cd build
+ - ../autogen.sh --prefix=$(pwd)/install-root
+ - $MAKE -j3
+ - $MAKE -j3 install
+ - $MAKE -j3 dist
--
2.25.4
9:32 a.m.
New subject: [libvirt PATCH 2/3] cirrus: Add templates and refresh script
On Mon, Jun 29, 2020 at 03:58:43PM +0200, Andrea Bolognani wrote:
This is similar to what we already use for Dockerfiles, with one
key difference: while we still rely on lcitool taking care of the
complicated work for us, in this case we're only provided with a
bunch of variables and we have to do the last bit of work (that
is replacing them inside an existing template) ourselves.
Signed-off-by: Andrea Bolognani <abologna(a)redhat.com>
---
ci/cirrus/refresh | 32 ++++++++++++++++++++++++++++++
ci/cirrus/templates/freebsd-12.yml | 25 +++++++++++++++++++++++
ci/cirrus/templates/macos-1015.yml | 27 +++++++++++++++++++++++++
These files are largely the identical content to the same name
files in the dir above which feels like it is easy to optimize
away.
3 files changed, 84 insertions(+)
create mode 100755 ci/cirrus/refresh
create mode 100644 ci/cirrus/templates/freebsd-12.yml
create mode 100644 ci/cirrus/templates/macos-1015.yml
diff --git a/ci/cirrus/refresh b/ci/cirrus/refresh
new file mode 100755
index 0000000000..51deca94ac
--- /dev/null
+++ b/ci/cirrus/refresh
@@ -0,0 +1,32 @@
+#!/bin/sh
+
+if test -z "$1"
+then
+ echo "syntax: $0 PATH-TO-LCITOOL"
+ exit 1
+fi
+
+LCITOOL=$1
+
+if ! test -x "$LCITOOL"
+then
+ echo "$LCITOOL is not executable"
+ exit 1
+fi
+
+for infile in templates/*
+do
+ outfile="${infile##*/}.j2"
+ host="${outfile%%.*}"
+
+ eval $("$LCITOOL" dockerfile "libvirt-$host" libvirt
--variables)
+
+ sed -e "s|[@]PKGS@|$PKGS|g" \
+ -e "s|[@]CROSS_PKGS@|$CROSS_PKGS|g" \
+ -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g" \
+ -e "s|[@]CPAN_PKGS@|$CPAN_PKGS|g" \
+ -e "s|[@]MAKE@|$MAKE|g" \
+ -e "s|[@]NINJA@|$NINJA|g" \
+ -e "s|[@]PYTHON@|$PYTHON|g" \
+ <"$infile" >"$outfile"
+done
I feel like this should really be reduced to just:
$LCITOOL cirrusci libvirt-$host libvirt > $outfile
where the 'cirrusci' command generates this content....
diff --git a/ci/cirrus/templates/freebsd-12.yml
b/ci/cirrus/templates/freebsd-12.yml
new file mode 100644
index 0000000000..228ea67827
--- /dev/null
+++ b/ci/cirrus/templates/freebsd-12.yml
@@ -0,0 +1,25 @@
+freebsd_instance:
+ image_family: freebsd-12-1
+
+env:
+ CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }}
+ CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }}
+ CI_COMMIT_SHA: {{ CI_COMMIT_SHA }}
+ PKGS: @PKGS@
+ MAKE: @MAKE@
+ PYTHON: @PYTHON@
+
+freebsd_12_task:
+ install_script:
+ - pkg install -y $PKGS
+ clone_script:
+ - git clone --depth 100 "$CI_REPOSITORY_URL" .
+ - git fetch origin "$CI_COMMIT_REF_NAME"
+ - git reset --hard "$CI_COMMIT_SHA"
...down to this point.
+ build_script:
+ - mkdir build
+ - cd build
+ - ../autogen.sh --prefix=$(pwd)/install-root
+ - $MAKE -j3
+ - $MAKE -j3 install
+ - $MAKE -j3 dist
This part can then be stored in ci/cirrus/build.yml since it is
common to freebsd & macos.
So now in gitlab-ci.yml we can just concatenate the two into a
temp file:
cat ci/cirrus/$NAME.yml ci/cirrus/build.yml > ci/cirrus/$NAME.yml.j2
cirrus-run ci/cirrus/$NAME.yml.j2
This avoids needing to store the same information in git twice.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
11:08 a.m.
New subject: [libvirt PATCH 2/3] cirrus: Add templates and refresh script
On Mon, Jun 29, 2020 at 03:32:22PM +0100, Daniel P. Berrangé wrote:
On Mon, Jun 29, 2020 at 03:58:43PM +0200, Andrea Bolognani wrote:
> This is similar to what we already use for Dockerfiles, with one
> key difference: while we still rely on lcitool taking care of the
> complicated work for us, in this case we're only provided with a
> bunch of variables and we have to do the last bit of work (that
> is replacing them inside an existing template) ourselves.
>
> Signed-off-by: Andrea Bolognani <abologna(a)redhat.com>
> ---
> ci/cirrus/refresh | 32 ++++++++++++++++++++++++++++++
> ci/cirrus/templates/freebsd-12.yml | 25 +++++++++++++++++++++++
> ci/cirrus/templates/macos-1015.yml | 27 +++++++++++++++++++++++++
These files are largely the identical content to the same name
files in the dir above which feels like it is easy to optimize
away.
> 3 files changed, 84 insertions(+)
> create mode 100755 ci/cirrus/refresh
> create mode 100644 ci/cirrus/templates/freebsd-12.yml
> create mode 100644 ci/cirrus/templates/macos-1015.yml
>
> diff --git a/ci/cirrus/refresh b/ci/cirrus/refresh
> new file mode 100755
> index 0000000000..51deca94ac
> --- /dev/null
> +++ b/ci/cirrus/refresh
> @@ -0,0 +1,32 @@
> +#!/bin/sh
> +
> +if test -z "$1"
> +then
> + echo "syntax: $0 PATH-TO-LCITOOL"
> + exit 1
> +fi
> +
> +LCITOOL=$1
> +
> +if ! test -x "$LCITOOL"
> +then
> + echo "$LCITOOL is not executable"
> + exit 1
> +fi
> +
> +for infile in templates/*
> +do
> + outfile="${infile##*/}.j2"
> + host="${outfile%%.*}"
> +
> + eval $("$LCITOOL" dockerfile "libvirt-$host" libvirt
--variables)
> +
> + sed -e "s|[@]PKGS@|$PKGS|g" \
> + -e "s|[@]CROSS_PKGS@|$CROSS_PKGS|g" \
> + -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g" \
> + -e "s|[@]CPAN_PKGS@|$CPAN_PKGS|g" \
> + -e "s|[@]MAKE@|$MAKE|g" \
> + -e "s|[@]NINJA@|$NINJA|g" \
> + -e "s|[@]PYTHON@|$PYTHON|g" \
> + <"$infile" >"$outfile"
> +done
I feel like this should really be reduced to just:
$LCITOOL cirrusci libvirt-$host libvirt > $outfile
where the 'cirrusci' command generates this content....
I agree with the idea of having a new command generating the content below, I
tend to look at this as different generator plugins/backends that we support,
so having a dedicated command makes much more sense to me.
> diff --git a/ci/cirrus/templates/freebsd-12.yml
b/ci/cirrus/templates/freebsd-12.yml
> new file mode 100644
> index 0000000000..228ea67827
> --- /dev/null
> +++ b/ci/cirrus/templates/freebsd-12.yml
> @@ -0,0 +1,25 @@
> +freebsd_instance:
> + image_family: freebsd-12-1
> +
> +env:
> + CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }}
> + CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }}
> + CI_COMMIT_SHA: {{ CI_COMMIT_SHA }}
> + PKGS: @PKGS@
> + MAKE: @MAKE@
> + PYTHON: @PYTHON@
> +
> +freebsd_12_task:
> + install_script:
> + - pkg install -y $PKGS
> + clone_script:
> + - git clone --depth 100 "$CI_REPOSITORY_URL" .
> + - git fetch origin "$CI_COMMIT_REF_NAME"
> + - git reset --hard "$CI_COMMIT_SHA"
...down to this point.
> + build_script:
> + - mkdir build
> + - cd build
> + - ../autogen.sh --prefix=$(pwd)/install-root
> + - $MAKE -j3
> + - $MAKE -j3 install
> + - $MAKE -j3 dist
This part can then be stored in ci/cirrus/build.yml since it is
common to freebsd & macos.
So now in gitlab-ci.yml we can just concatenate the two into a
temp file:
cat ci/cirrus/$NAME.yml ci/cirrus/build.yml > ci/cirrus/$NAME.yml.j2
cirrus-run ci/cirrus/$NAME.yml.j2
How about instead of storing the same build.yml in every secondary project's ci
directory we store the build.yml in libvirt-ci repo and generate the whole
thing with lcitool? I think the way the build YAML would look like for every
single subproject, we could achieve the level of similarity we have for lcitool
native builds - IOW the build job abstraction we have across projects under
guests/playbooks/build/projects could also be achieved here with cirrus, just
an idea.
Erik
11:35 a.m.
New subject: [libvirt PATCH 2/3] cirrus: Add templates and refresh script
On Mon, Jun 29, 2020 at 06:08:27PM +0200, Erik Skultety wrote:
On Mon, Jun 29, 2020 at 03:32:22PM +0100, Daniel P. Berrangé wrote:
> On Mon, Jun 29, 2020 at 03:58:43PM +0200, Andrea Bolognani wrote:
> > This is similar to what we already use for Dockerfiles, with one
> > key difference: while we still rely on lcitool taking care of the
> > complicated work for us, in this case we're only provided with a
> > bunch of variables and we have to do the last bit of work (that
> > is replacing them inside an existing template) ourselves.
> >
> > Signed-off-by: Andrea Bolognani <abologna(a)redhat.com>
> > ---
> > ci/cirrus/refresh | 32 ++++++++++++++++++++++++++++++
> > ci/cirrus/templates/freebsd-12.yml | 25 +++++++++++++++++++++++
> > ci/cirrus/templates/macos-1015.yml | 27 +++++++++++++++++++++++++
>
> > diff --git a/ci/cirrus/templates/freebsd-12.yml
b/ci/cirrus/templates/freebsd-12.yml
> > new file mode 100644
> > index 0000000000..228ea67827
> > --- /dev/null
> > +++ b/ci/cirrus/templates/freebsd-12.yml
> > @@ -0,0 +1,25 @@
> > +freebsd_instance:
> > + image_family: freebsd-12-1
> > +
> > +env:
> > + CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }}
> > + CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }}
> > + CI_COMMIT_SHA: {{ CI_COMMIT_SHA }}
> > + PKGS: @PKGS@
> > + MAKE: @MAKE@
> > + PYTHON: @PYTHON@
> > +
> > +freebsd_12_task:
> > + install_script:
> > + - pkg install -y $PKGS
> > + clone_script:
> > + - git clone --depth 100 "$CI_REPOSITORY_URL" .
> > + - git fetch origin "$CI_COMMIT_REF_NAME"
> > + - git reset --hard "$CI_COMMIT_SHA"
>
> ...down to this point.
>
>
> > + build_script:
> > + - mkdir build
> > + - cd build
> > + - ../autogen.sh --prefix=$(pwd)/install-root
> > + - $MAKE -j3
> > + - $MAKE -j3 install
> > + - $MAKE -j3 dist
>
> This part can then be stored in ci/cirrus/build.yml since it is
> common to freebsd & macos.
>
> So now in gitlab-ci.yml we can just concatenate the two into a
> temp file:
>
> cat ci/cirrus/$NAME.yml ci/cirrus/build.yml > ci/cirrus/$NAME.yml.j2
> cirrus-run ci/cirrus/$NAME.yml.j2
How about instead of storing the same build.yml in every secondary project's ci
directory we store the build.yml in libvirt-ci repo and generate the whole
thing with lcitool? I think the way the build YAML would look like for every
single subproject, we could achieve the level of similarity we have for lcitool
native builds - IOW the build job abstraction we have across projects under
guests/playbooks/build/projects could also be achieved here with cirrus, just
an idea.
Yep, I hesitated to suggest that, as I thought it might need boiling the
ocean in the short, whereas the other pieces are 100% static data we
can trivially emit with next to no effort. Long term if we can generate
the build commands too, that could be nice though. I'd even think about
the possibility of generating the main gitlab-ci.yml file to some degree,
though at some point you spend more time in the generator than on the
original file, so I'm not sure where the line is.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
1:30 p.m.
New subject: [libvirt PATCH 2/3] cirrus: Add templates and refresh script
On Mon, 2020-06-29 at 17:35 +0100, Daniel P. Berrangé wrote:
On Mon, Jun 29, 2020 at 06:08:27PM +0200, Erik Skultety wrote:
> On Mon, Jun 29, 2020 at 03:32:22PM +0100, Daniel P. Berrangé wrote:
> > This part can then be stored in ci/cirrus/build.yml since it is
> > common to freebsd & macos.
> >
> > So now in gitlab-ci.yml we can just concatenate the two into a
> > temp file:
> >
> > cat ci/cirrus/$NAME.yml ci/cirrus/build.yml > ci/cirrus/$NAME.yml.j2
> > cirrus-run ci/cirrus/$NAME.yml.j2
>
> How about instead of storing the same build.yml in every secondary project's ci
> directory we store the build.yml in libvirt-ci repo and generate the whole
> thing with lcitool? I think the way the build YAML would look like for every
> single subproject, we could achieve the level of similarity we have for lcitool
> native builds - IOW the build job abstraction we have across projects under
> guests/playbooks/build/projects could also be achieved here with cirrus, just
> an idea.
Yep, I hesitated to suggest that, as I thought it might need boiling the
ocean in the short, whereas the other pieces are 100% static data we
can trivially emit with next to no effort. Long term if we can generate
the build commands too, that could be nice though. I'd even think about
the possibility of generating the main gitlab-ci.yml file to some degree,
though at some point you spend more time in the generator than on the
original file, so I'm not sure where the line is.
I agree that it would be incredibly neat if we could take the same
information used by 'lcitool build' and generate at least part of the
various .gitlab-ci.yml files.
However, I think we should *not* attempt something like that until
later, when we have enabled full GitLab CI support (including, where
applicable, FreeBSD and macOS) for all projects and we can hopefully
see the proper path forward more clearly thanks to the insights
gained in the process.
--
Andrea Bolognani / Red Hat / Virtualization
11:27 a.m.
New subject: [libvirt PATCH 2/3] cirrus: Add templates and refresh script
On Mon, 2020-06-29 at 15:32 +0100, Daniel P. Berrangé wrote:
On Mon, Jun 29, 2020 at 03:58:43PM +0200, Andrea Bolognani wrote:
> +for infile in templates/*
> +do
> + outfile="${infile##*/}.j2"
> + host="${outfile%%.*}"
> +
> + eval $("$LCITOOL" dockerfile "libvirt-$host" libvirt
--variables)
> +
> + sed -e "s|[@]PKGS@|$PKGS|g" \
> + -e "s|[@]CROSS_PKGS@|$CROSS_PKGS|g" \
> + -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g" \
> + -e "s|[@]CPAN_PKGS@|$CPAN_PKGS|g" \
> + -e "s|[@]MAKE@|$MAKE|g" \
> + -e "s|[@]NINJA@|$NINJA|g" \
> + -e "s|[@]PYTHON@|$PYTHON|g" \
> + <"$infile" >"$outfile"
> +done
I feel like this should really be reduced to just:
$LCITOOL cirrusci libvirt-$host libvirt > $outfile
> + build_script:
> + - mkdir build
> + - cd build
> + - ../autogen.sh --prefix=$(pwd)/install-root
> + - $MAKE -j3
> + - $MAKE -j3 install
> + - $MAKE -j3 dist
This part can then be stored in ci/cirrus/build.yml since it is
common to freebsd & macos.
So now in gitlab-ci.yml we can just concatenate the two into a
temp file:
cat ci/cirrus/$NAME.yml ci/cirrus/build.yml > ci/cirrus/$NAME.yml.j2
cirrus-run ci/cirrus/$NAME.yml.j2
This avoids needing to store the same information in git twice.
Yeah, I don't like having to store the information twice in git, but
I'm also not a fan of having an lcitool subcommand that is tightly
coupled with the details of a specific CI service rather than dealing
with the more general problem of managing build environments.
I think I have an approach that can sit somewhere in the middle. Let
me give it a try.
--
Andrea Bolognani / Red Hat / Virtualization
11:31 a.m.
New subject: [libvirt PATCH 2/3] cirrus: Add templates and refresh script
On Mon, Jun 29, 2020 at 06:27:40PM +0200, Andrea Bolognani wrote:
On Mon, 2020-06-29 at 15:32 +0100, Daniel P. Berrangé wrote:
> On Mon, Jun 29, 2020 at 03:58:43PM +0200, Andrea Bolognani wrote:
> > +for infile in templates/*
> > +do
> > + outfile="${infile##*/}.j2"
> > + host="${outfile%%.*}"
> > +
> > + eval $("$LCITOOL" dockerfile "libvirt-$host" libvirt
--variables)
> > +
> > + sed -e "s|[@]PKGS@|$PKGS|g" \
> > + -e "s|[@]CROSS_PKGS@|$CROSS_PKGS|g" \
> > + -e "s|[@]PYPI_PKGS@|$PYPI_PKGS|g" \
> > + -e "s|[@]CPAN_PKGS@|$CPAN_PKGS|g" \
> > + -e "s|[@]MAKE@|$MAKE|g" \
> > + -e "s|[@]NINJA@|$NINJA|g" \
> > + -e "s|[@]PYTHON@|$PYTHON|g" \
> > + <"$infile" >"$outfile"
> > +done
>
> I feel like this should really be reduced to just:
>
> $LCITOOL cirrusci libvirt-$host libvirt > $outfile
>
> > + build_script:
> > + - mkdir build
> > + - cd build
> > + - ../autogen.sh --prefix=$(pwd)/install-root
> > + - $MAKE -j3
> > + - $MAKE -j3 install
> > + - $MAKE -j3 dist
>
> This part can then be stored in ci/cirrus/build.yml since it is
> common to freebsd & macos.
>
> So now in gitlab-ci.yml we can just concatenate the two into a
> temp file:
>
> cat ci/cirrus/$NAME.yml ci/cirrus/build.yml > ci/cirrus/$NAME.yml.j2
> cirrus-run ci/cirrus/$NAME.yml.j2
>
> This avoids needing to store the same information in git twice.
Yeah, I don't like having to store the information twice in git, but
I'm also not a fan of having an lcitool subcommand that is tightly
coupled with the details of a specific CI service rather than dealing
with the more general problem of managing build environments.
Hmm, I kind of feel that's exactly the job of lcitool. Generate whatever
files/resources we need for integrating with whatever CI infra we're
choosing to use.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
8:58 a.m.
New subject: [libvirt PATCH 3/3] cirrus: Refresh configurations
The information is coming from lcitool now, so it's more complete:
this is immediately apparent when looking at the number of
additional packages that end up being installed on macOS.
This commit is best viewed with 'git show --word-diff'.
Signed-off-by: Andrea Bolognani <abologna(a)redhat.com>
---
ci/cirrus/freebsd-12.yml.j2 | 10 ++++++----
ci/cirrus/macos-1015.yml.j2 | 14 ++++++++------
2 files changed, 14 insertions(+), 10 deletions(-)
diff --git a/ci/cirrus/freebsd-12.yml.j2 b/ci/cirrus/freebsd-12.yml.j2
index a93f94ee2a..80083c65d7 100644
--- a/ci/cirrus/freebsd-12.yml.j2
+++ b/ci/cirrus/freebsd-12.yml.j2
@@ -5,7 +5,9 @@ env:
CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }}
CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }}
CI_COMMIT_SHA: {{ CI_COMMIT_SHA }}
- PKGS: augeas autoconf automake avahi bash bash-completion ca_root_nss ccache chrony
cppi curl cyrus-sasl dbus diskscrub dnsmasq fusefs-libs gdb gettext gettext-tools git glib
gmake gnutls hal libpcap libpciaccess libssh libssh2 libtool libxml2 libxslt lsof meson
ncurses ninja patch perl5 pkgconf polkit py37-docutils py37-flake8 py37-setuptools
py37-wheel python3 qemu-utils radvd readline screen sudo vim yajl
+ PKGS: augeas autoconf automake avahi bash bash-completion ca_root_nss ccache chrony
cppi curl cyrus-sasl dbus diskscrub dnsmasq fusefs-libs gdb gettext gettext-tools git glib
gmake gnutls hal libpcap libpciaccess libssh libssh2 libtool libxml2 libxslt lsof meson
ncurses ninja p5-App-cpanminus patch perl5 pkgconf polkit py37-docutils py37-flake8
py37-pip py37-setuptools py37-wheel python3 qemu-utils radvd readline screen sudo vim
yajl
+ MAKE: /usr/local/bin/gmake
+ PYTHON: /usr/local/bin/python3
freebsd_12_task:
install_script:
@@ -18,6 +20,6 @@ freebsd_12_task:
- mkdir build
- cd build
- ../autogen.sh --prefix=$(pwd)/install-root
- - gmake -j3
- - gmake -j3 install
- - gmake -j3 dist
+ - $MAKE -j3
+ - $MAKE -j3 install
+ - $MAKE -j3 dist
diff --git a/ci/cirrus/macos-1015.yml.j2 b/ci/cirrus/macos-1015.yml.j2
index 0d3d54a97a..1a4e942613 100644
--- a/ci/cirrus/macos-1015.yml.j2
+++ b/ci/cirrus/macos-1015.yml.j2
@@ -5,9 +5,11 @@ env:
CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }}
CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }}
CI_COMMIT_SHA: {{ CI_COMMIT_SHA }}
- PATH:
/usr/local/opt/gettext/bin:/usr/local/opt/ccache/libexec:/usr/local/opt/rpcgen/bin:$PATH
- PKG_CONFIG_PATH: /usr/local/opt/libxml2/lib/pkgconfig
- PKGS: autoconf automake ccache docutils glib gnutls libtool libxml2 make pkg-config
python rpcgen xz yajl
+ PATH:
/usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin:/usr/local/opt/libpcap/bin:/usr/local/opt/libxslt/bin:/usr/local/opt/rpcgen/bin:$PATH
+ PKG_CONFIG_PATH:
/usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/libpcap/lib/pkgconfig:/usr/local/opt/libxml2/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
+ PKGS: augeas autoconf automake bash bash-completion ccache cpanminus cppi curl dbus
dnsmasq docutils flake8 gdb gettext git glib gnutls gpatch libiscsi libpcap libssh libssh2
libtool libxml2 libxslt lsof make meson ncurses ninja perl pkg-config python3 qemu
readline rpcgen screen scrub vim xz yajl
+ MAKE: /usr/local/bin/gmake
+ PYTHON: /usr/local/bin/python3
macos_1015_task:
install_script:
@@ -20,6 +22,6 @@ macos_1015_task:
- mkdir build
- cd build
- ../autogen.sh --prefix=$(pwd)/install-root
- - gmake -j3
- - gmake -j3 install
- - gmake -j3 dist
+ - $MAKE -j3
+ - $MAKE -j3 install
+ - $MAKE -j3 dist
--
2.25.4
1637
days inactive
1637
days old
9 comments
3 participants
participants (3)
-
Andrea Bolognani -
Daniel P. Berrangé -
Erik Skultety