[libvirt PATCH 0/2] ci: Enable Cirrus CI integration
This is an alternative approach to the one proposed by Erik[1]. Advantages of this approach: * we don't have to build and maintain our own FreeBSD runners; * we make it possible, after some straightforward one-time setup, for any contributor to have their branches build on FreeBSD and macOS, again without having to build and maintain their own runners; * we reduce the number of services external of GitLab that we depend on by having FreeBSD and macOS build happen on the same platform instead of separate ones. Drawbacks of this approach: * requires signing up for yet another online service. To see this in action, check out the https://gitlab.com/abologna/libvirt/-/pipelines/153963978 pipeline, and in particular the https://gitlab.com/abologna/libvirt/-/jobs/585678408 https://gitlab.com/abologna/libvirt/-/jobs/585678413 jobs. libvirt-ci merge request: https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/16 [1] https://www.redhat.com/archives/libvir-list/2020-June/msg00188.html Andrea Bolognani (2): ci: Enable Cirrus CI integration ci: Remove Travis CI .gitlab-ci.yml | 36 ++++++++++++++++++ .travis.yml | 58 ----------------------------- ci/README.rst | 59 ++++++++++++++++++++++++++++++ ci/cirrus/freebsd-11.yml.j2 | 73 +++++++++++++++++++++++++++++++++++++ ci/cirrus/freebsd-12.yml.j2 | 73 +++++++++++++++++++++++++++++++++++++ ci/cirrus/macos-1015.yml.j2 | 38 +++++++++++++++++++ 6 files changed, 279 insertions(+), 58 deletions(-) delete mode 100644 .travis.yml create mode 100644 ci/README.rst create mode 100644 ci/cirrus/freebsd-11.yml.j2 create mode 100644 ci/cirrus/freebsd-12.yml.j2 create mode 100644 ci/cirrus/macos-1015.yml.j2 -- 2.25.4
We use cirrus-run to trigger Cirrus CI jobs from GitLab CI jobs, making it possible to extend our platform coverage to include FreeBSD without having to maintain our own runners; additionally, we'll be able to ditch Travis CI and, since results for Cirrus CI jobs are reflected back to the GitLab CI jobs that triggered them, we will be able to get all information from a single dashboard. The FreeBSD and macOS job definitions can be improved further: for example, we will want to enable caching to speed up builds, and ultimately we should figure out a way to generate at least part of them, notably the list of packages to be installed, using lcitool. All of that will happen in later patches: for now, this is good enough to start using Cirrus CI. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- .gitlab-ci.yml | 36 ++++++++++++++++++ ci/README.rst | 59 ++++++++++++++++++++++++++++++ ci/cirrus/freebsd-11.yml.j2 | 73 +++++++++++++++++++++++++++++++++++++ ci/cirrus/freebsd-12.yml.j2 | 73 +++++++++++++++++++++++++++++++++++++ ci/cirrus/macos-1015.yml.j2 | 38 +++++++++++++++++++ 5 files changed, 279 insertions(+) create mode 100644 ci/README.rst create mode 100644 ci/cirrus/freebsd-11.yml.j2 create mode 100644 ci/cirrus/freebsd-12.yml.j2 create mode 100644 ci/cirrus/macos-1015.yml.j2 diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 7113a1283c..bded69224b 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -38,6 +38,27 @@ stages: - master - /^ci-full-.*$/ +# Jobs that we delegate to Cirrus CI because they require an operating +# system other than Linux +.cirrus_build_default_job_template: &cirrus_build_default_job_definition + stage: native_build + image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master + script: + - | + if test -z "$CIRRUS_GITHUB_REPO" || test -z "$CIRRUS_API_TOKEN"; then + echo "WARN: Cirrus CI integration not set up correctly for your account" + echo "WARN: This job will be skipped" + echo "WARN: See ci/README.rst for more information" + exit 0 + fi + - cirrus-run ci/cirrus/$NAME.yml.j2 + +.cirrus_build_extra_job_template: &cirrus_build_extra_job_definition + <<: *cirrus_build_default_job_definition + only: + - master + - /^ci-full-.*$/ + # Default cross build jobs that are always run .cross_build_default_job_template: &cross_build_default_job_definition @@ -109,6 +130,21 @@ x64-ubuntu-2004: <<: *native_build_default_job_definition image: quay.io/libvirt/buildenv-libvirt-ubuntu-2004:latest +x64-freebsd-11-build: + <<: *cirrus_build_extra_job_definition + variables: + NAME: freebsd-11 + +x64-freebsd-12-build: + <<: *cirrus_build_default_job_definition + variables: + NAME: freebsd-12 + +x64-macos-1015-build: + <<: *cirrus_build_default_job_definition + variables: + NAME: macos-1015 + # Cross compiled build jobs diff --git a/ci/README.rst b/ci/README.rst new file mode 100644 index 0000000000..033db1c847 --- /dev/null +++ b/ci/README.rst @@ -0,0 +1,59 @@ +============== +CI for libvirt +============== + +This document provides some information related to the CI capabilities for the +libvirt project. + + +Cirrus CI integration +===================== + +libvirt currently supports three non-Linux operating systems: Windows, FreeBSD +and macOS. Windows cross-builds can be prepared on Linux by using `MinGW`_, but +for both FreeBSD and macOS we need to use the actual operating system, and +unfortunately GitLab shared runners are currently not available for either. + +To work around this limitation, we take advantage of `Cirrus CI`_'s free +offering: more specifically, we use the `cirrus-run`_ script to trigger Cirrus +CI jobs from GitLab CI jobs so that the workaround is almost entirely +transparent to users and there's no need to constantly check two separate CI +dashboards. + +There is, however, some one-time setup required. If you want FreeBSD and macOS +builds to happen when you push to your GitLab repository, you need to + +* set up a GitHub repository for the project, eg. ``yourusername/libvirt``. + This repository needs to exist for cirrus-run to work, but it doesn't need to + be kept up to date, so you can create it and then forget about it; + +* enable the `Cirrus CI GitHub app`_ for your GitHub account; + +* sign up for Cirrus CI. It's enough to log into the website using your GitHub + account; + +* grab an API token from the `Cirrus CI settings`_ page; + +* in the *CI/CD / Variables* section of the settings page for your GitLab + repository, create two new variables: + + * ``CIRRUS_GITHUB_REPO``, containing the name of the GitHub repository + created earlier, eg. ``yourusername/libvirt``; + + * ``CIRRUS_API_TOKEN``, containing the Cirrus CI API token generated earlier. + This variable **must** be marked as *Masked*, because anyone with knowledge + of it can impersonate you as far as Cirrus CI is concerned. + + Neither of these variables should be marked as *Protected*, because in + general you'll want to be able to trigger Cirrus CI builds from non-protected + branches. + +Once this one-time setup is complete, you can just keep pushing to your GitLab +repository as usual and you'll automatically get the additional CI coverage. + + +.. _Cirrus CI GitHub app: https://github.com/marketplace/cirrus-ci +.. _Cirrus CI settings: https://cirrus-ci.com/settings/profile/ +.. _Cirrus CI: https://cirrus-ci.com/ +.. _MinGW: http://mingw.org/ +.. _cirrus-run: https://github.com/sio/cirrus-run/ diff --git a/ci/cirrus/freebsd-11.yml.j2 b/ci/cirrus/freebsd-11.yml.j2 new file mode 100644 index 0000000000..e8df443d21 --- /dev/null +++ b/ci/cirrus/freebsd-11.yml.j2 @@ -0,0 +1,73 @@ +freebsd_instance: + image_family: freebsd-11-3 + +env: + CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }} + CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }} + CI_COMMIT_SHA: {{ CI_COMMIT_SHA }} + +freebsd_11_task: + install_script: + - pkg install -y + augeas + autoconf + automake + avahi + bash + bash-completion + ca_root_nss + ccache + chrony + cppi + curl + cyrus-sasl + dbus + diskscrub + dnsmasq + fusefs-libs + gdb + gettext + gettext-tools + git + glib + gmake + gnutls + hal + libpcap + libpciaccess + libssh + libssh2 + libtool + libxml2 + libxslt + lsof + meson + ncurses + ninja + patch + perl5 + pkgconf + polkit + py37-docutils + py37-flake8 + py37-setuptools + py37-wheel + python3 + qemu-utils + radvd + readline + screen + sudo + vim + yajl + clone_script: + - git clone --depth 100 "$CI_REPOSITORY_URL" . + - git fetch origin "$CI_COMMIT_REF_NAME" + - git reset --hard "$CI_COMMIT_SHA" + build_script: + - mkdir build + - cd build + - ../autogen.sh --prefix=$(pwd)/install-root + - gmake -j3 + - gmake -j3 install + - gmake -j3 dist diff --git a/ci/cirrus/freebsd-12.yml.j2 b/ci/cirrus/freebsd-12.yml.j2 new file mode 100644 index 0000000000..a653996850 --- /dev/null +++ b/ci/cirrus/freebsd-12.yml.j2 @@ -0,0 +1,73 @@ +freebsd_instance: + image_family: freebsd-12-1 + +env: + CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }} + CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }} + CI_COMMIT_SHA: {{ CI_COMMIT_SHA }} + +freebsd_12_task: + install_script: + - pkg install -y + augeas + autoconf + automake + avahi + bash + bash-completion + ca_root_nss + ccache + chrony + cppi + curl + cyrus-sasl + dbus + diskscrub + dnsmasq + fusefs-libs + gdb + gettext + gettext-tools + git + glib + gmake + gnutls + hal + libpcap + libpciaccess + libssh + libssh2 + libtool + libxml2 + libxslt + lsof + meson + ncurses + ninja + patch + perl5 + pkgconf + polkit + py37-docutils + py37-flake8 + py37-setuptools + py37-wheel + python3 + qemu-utils + radvd + readline + screen + sudo + vim + yajl + clone_script: + - git clone --depth 100 "$CI_REPOSITORY_URL" . + - git fetch origin "$CI_COMMIT_REF_NAME" + - git reset --hard "$CI_COMMIT_SHA" + build_script: + - mkdir build + - cd build + - ../autogen.sh --prefix=$(pwd)/install-root + - gmake -j3 + - gmake -j3 install + - gmake -j3 dist diff --git a/ci/cirrus/macos-1015.yml.j2 b/ci/cirrus/macos-1015.yml.j2 new file mode 100644 index 0000000000..d8aa9715d6 --- /dev/null +++ b/ci/cirrus/macos-1015.yml.j2 @@ -0,0 +1,38 @@ +osx_instance: + image: catalina-base + +env: + CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }} + CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }} + CI_COMMIT_SHA: {{ CI_COMMIT_SHA }} + PATH: /usr/local/opt/gettext/bin:/usr/local/opt/ccache/libexec:/usr/local/opt/rpcgen/bin:$PATH + PKG_CONFIG_PATH: /usr/local/opt/libxml2/lib/pkgconfig + +macos_1015_task: + install_script: + - brew install + autoconf + automake + ccache + docutils + glib + gnutls + libtool + libxml2 + make + pkg-config + python + rpcgen + xz + yajl + clone_script: + - git clone --depth 100 "$CI_REPOSITORY_URL" . + - git fetch origin "$CI_COMMIT_REF_NAME" + - git reset --hard "$CI_COMMIT_SHA" + build_script: + - mkdir build + - cd build + - ../autogen.sh --prefix=$(pwd)/install-root + - gmake -j3 + - gmake -j3 install + - gmake -j3 dist -- 2.25.4
On Mon, Jun 08, 2020 at 05:42:32PM +0200, Andrea Bolognani wrote:
We use cirrus-run to trigger Cirrus CI jobs from GitLab CI jobs, making it possible to extend our platform coverage to include FreeBSD without having to maintain our own runners; additionally, we'll be able to ditch Travis CI and, since results for Cirrus CI jobs are reflected back to the GitLab CI jobs that triggered them, we will be able to get all information from a single dashboard.
The FreeBSD and macOS job definitions can be improved further: for example, we will want to enable caching to speed up builds, and ultimately we should figure out a way to generate at least part of them, notably the list of packages to be installed, using lcitool. All of that will happen in later patches: for now, this is good enough to start using Cirrus CI.
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 7113a1283c..bded69224b 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -38,6 +38,27 @@ stages: - master - /^ci-full-.*$/
+# Jobs that we delegate to Cirrus CI because they require an operating +# system other than Linux +.cirrus_build_default_job_template: &cirrus_build_default_job_definition + stage: native_build + image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master + script: + - | + if test -z "$CIRRUS_GITHUB_REPO" || test -z "$CIRRUS_API_TOKEN"; then + echo "WARN: Cirrus CI integration not set up correctly for your account" + echo "WARN: This job will be skipped" + echo "WARN: See ci/README.rst for more information" + exit 0 + fi
This will give a misleading impression that the macOS / FreeBSD job has been run for a user's merge request. Instead we shoudl do
+ - cirrus-run ci/cirrus/$NAME.yml.j2
only: variables: - $CIRRUS_GITHUB_REPO - $CIRRUS_API_TOKEN this ensures the job is not even created if the contributor has not configured things. Thus we can see from the list of jobs whether the merge request has CI coverage of macOS/FreeBSD, instead of having to read the job log to spot the warning.
+ +.cirrus_build_extra_job_template: &cirrus_build_extra_job_definition + <<: *cirrus_build_default_job_definition + only: + - master + - /^ci-full-.*$/ +
diff --git a/ci/cirrus/freebsd-11.yml.j2 b/ci/cirrus/freebsd-11.yml.j2 new file mode 100644 index 0000000000..e8df443d21 --- /dev/null +++ b/ci/cirrus/freebsd-11.yml.j2 @@ -0,0 +1,73 @@ +freebsd_instance: + image_family: freebsd-11-3 + +env: + CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }} + CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }} + CI_COMMIT_SHA: {{ CI_COMMIT_SHA }} + +freebsd_11_task: + install_script: + - pkg install -y + augeas + autoconf + automake + avahi + bash + bash-completion + ca_root_nss + ccache + chrony + cppi + curl + cyrus-sasl + dbus + diskscrub + dnsmasq + fusefs-libs + gdb + gettext + gettext-tools + git + glib + gmake + gnutls + hal + libpcap + libpciaccess + libssh + libssh2 + libtool + libxml2 + libxslt + lsof + meson + ncurses + ninja + patch + perl5 + pkgconf + polkit + py37-docutils + py37-flake8 + py37-setuptools + py37-wheel + python3 + qemu-utils + radvd + readline + screen + sudo + vim + yajl + clone_script: + - git clone --depth 100 "$CI_REPOSITORY_URL" . + - git fetch origin "$CI_COMMIT_REF_NAME" + - git reset --hard "$CI_COMMIT_SHA" + build_script: + - mkdir build + - cd build + - ../autogen.sh --prefix=$(pwd)/install-root + - gmake -j3 + - gmake -j3 install + - gmake -j3 dist
For the dockerfiles, we're auto-generating using lcitool. IIUC, we should have sufficient info avialable in lcitol that we can wrote a command to generate this entire file, including the build_script commands. In fact eventually we could try to get to a point where we auto-generate the .gitlab-ci.yml from lcitool.
diff --git a/ci/cirrus/macos-1015.yml.j2 b/ci/cirrus/macos-1015.yml.j2 new file mode 100644 index 0000000000..d8aa9715d6 --- /dev/null +++ b/ci/cirrus/macos-1015.yml.j2 @@ -0,0 +1,38 @@ +osx_instance: + image: catalina-base + +env: + CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }} + CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }} + CI_COMMIT_SHA: {{ CI_COMMIT_SHA }} + PATH: /usr/local/opt/gettext/bin:/usr/local/opt/ccache/libexec:/usr/local/opt/rpcgen/bin:$PATH + PKG_CONFIG_PATH: /usr/local/opt/libxml2/lib/pkgconfig + +macos_1015_task: + install_script: + - brew install + autoconf + automake + ccache + docutils + glib + gnutls + libtool + libxml2 + make + pkg-config + python + rpcgen + xz + yajl
Obviously we'd need to define a mapping for macOS in lcitool for this. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, 2020-06-08 at 17:41 +0100, Daniel P. Berrangé wrote:
On Mon, Jun 08, 2020 at 05:42:32PM +0200, Andrea Bolognani wrote:
+.cirrus_build_default_job_template: &cirrus_build_default_job_definition + stage: native_build + image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master + script: + - | + if test -z "$CIRRUS_GITHUB_REPO" || test -z "$CIRRUS_API_TOKEN"; then + echo "WARN: Cirrus CI integration not set up correctly for your account" + echo "WARN: This job will be skipped" + echo "WARN: See ci/README.rst for more information" + exit 0 + fi
This will give a misleading impression that the macOS / FreeBSD job has been run for a user's merge request. Instead we shoudl do
+ - cirrus-run ci/cirrus/$NAME.yml.j2
only: variables: - $CIRRUS_GITHUB_REPO - $CIRRUS_API_TOKEN
this ensures the job is not even created if the contributor has not configured things. Thus we can see from the list of jobs whether the merge request has CI coverage of macOS/FreeBSD, instead of having to read the job log to spot the warning.
That's a great approach! I wanted to find a way to signal that more clearly, and I thought I could mark them as orange by exiting with a special return code, but apparently that's something that GitLab has not implemented despite being requested multiple times... Your solution gives us almost the same result with tools that we can actually use today :)
+freebsd_11_task: + install_script: + - pkg install -y + augeas + autoconf + automake [...]
For the dockerfiles, we're auto-generating using lcitool.
IIUC, we should have sufficient info avialable in lcitol that we can wrote a command to generate this entire file, including the build_script commands.
In fact eventually we could try to get to a point where we auto-generate the .gitlab-ci.yml from lcitool.
I mention this in the commit message. The goal is absolutely to generate these files using lcitool, I just haven't invested time into doing that yet because I wanted to make sure the overall approach was considered acceptable first.
+macos_1015_task: + install_script: + - brew install + autoconf + automake + ccache [...]
Obviously we'd need to define a mapping for macOS in lcitool for this.
Obviously :) -- Andrea Bolognani / Red Hat / Virtualization
On Mon, Jun 08, 2020 at 07:14:20PM +0200, Andrea Bolognani wrote:
+freebsd_11_task: + install_script: + - pkg install -y + augeas + autoconf + automake [...]
For the dockerfiles, we're auto-generating using lcitool.
IIUC, we should have sufficient info avialable in lcitol that we can wrote a command to generate this entire file, including the build_script commands.
In fact eventually we could try to get to a point where we auto-generate the .gitlab-ci.yml from lcitool.
I mention this in the commit message. The goal is absolutely to generate these files using lcitool, I just haven't invested time into doing that yet because I wanted to make sure the overall approach was considered acceptable first.
If we only consider macOS for a minute, then I think this change is a no-brainer. It is clearly better than Travis because it presents all the CI results in the same dashboard, and doesn't require contributors to manually push to github themselves. There is no viable alternative for macOS, as we're never going to provide any VMs for macOS ourselves due to license issues around hardware you have to run on. The versions of macOS/XCode in Cirrus are different from Travis, but that doesn't bother me, given the clear benefits of the workflow. So the question becomes... Given that we should use Cirrus for macOS no matter what, is there a strong enough reason to not use this for FreeBSD too. The main downside I see of using Cirrus CI is that both Cirrus & GitHub are a closed source service. This is the same situation as Travis. I'm willing to ignore that for macOS as there is no way in which we can ever provide a macOS CI runner ourselves. ie the choice for macOS is "no CI at all" vs "CI using closed source" and the latter clearly wins. For FreeBSD things are a different though, as with Erik's work we do have the ability to provide FreeBSD runners and avoid any dep on either GitHub or Cirrus. We'll need Erik's work regardless for runners to do integration testing with, though admittedly our focus there will likely be on Linux runners for integration testing, not FreeBSD. The real benefit of Cirrus CI for FreeBSD is that it lets contributors get FreeBSD CI without having to bring their own VM to the party. They'll be using Cirrus CI for macOS anyway, so it seems silly to tell them they should setup Cirrus for macOS, but then not let them use it for FreeBSD and go through hoops for custom runners. I think this means we simply have to allow FreeBSD on Cirrus for forks, once we have this for macOS. The only variation I can see here is to write the gitlab-ci.yml such that forks do their pre-merge CI against Cirrus, but the master repo does its post-merge CI again our custom runner. I do strongly prefer open source infra wherever possible, but I'm not convinced the FreeBSD CI is a case where its worth the hassle of maintaining FreeBSD runners.... at least not unless we get to a point where we need runners for integration testing on FreeBSD which is not any time soon. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, Jun 08, 2020 at 06:49:10PM +0100, Daniel P. Berrangé wrote:
On Mon, Jun 08, 2020 at 07:14:20PM +0200, Andrea Bolognani wrote:
+freebsd_11_task: + install_script: + - pkg install -y + augeas + autoconf + automake [...]
For the dockerfiles, we're auto-generating using lcitool.
IIUC, we should have sufficient info avialable in lcitol that we can wrote a command to generate this entire file, including the build_script commands.
In fact eventually we could try to get to a point where we auto-generate the .gitlab-ci.yml from lcitool.
I mention this in the commit message. The goal is absolutely to generate these files using lcitool, I just haven't invested time into doing that yet because I wanted to make sure the overall approach was considered acceptable first.
If we only consider macOS for a minute, then I think this change is a no-brainer. It is clearly better than Travis because it presents all the CI results in the same dashboard, and doesn't require contributors to manually push to github themselves.
There is no viable alternative for macOS, as we're never going to provide any VMs for macOS ourselves due to license issues around hardware you have to run on.
The versions of macOS/XCode in Cirrus are different from Travis, but that doesn't bother me, given the clear benefits of the workflow.
So the question becomes...
Given that we should use Cirrus for macOS no matter what, is there a strong enough reason to not use this for FreeBSD too.
The main downside I see of using Cirrus CI is that both Cirrus & GitHub are a closed source service. This is the same situation as Travis. I'm willing to ignore that for macOS as there is no way in which we can ever provide a macOS CI runner ourselves. ie the choice for macOS is "no CI at all" vs "CI using closed source" and the latter clearly wins.
For FreeBSD things are a different though, as with Erik's work we do have the ability to provide FreeBSD runners and avoid any dep on either GitHub or Cirrus. We'll need Erik's work regardless for runners to do integration testing with, though admittedly our focus there will likely be on Linux runners for integration testing, not FreeBSD.
The real benefit of Cirrus CI for FreeBSD is that it lets contributors get FreeBSD CI without having to bring their own VM to the party. They'll be using Cirrus CI for macOS anyway, so it seems silly to tell them they should setup Cirrus for macOS, but then not let them use it for FreeBSD and go through hoops for custom runners. I think this means we simply have to allow FreeBSD on Cirrus for forks, once we have this for macOS.
Maintaining the FreeBSD runners ourselves in our own post-merge CI environment is quite painful to be honest, because the installation can't be automated - partitioning needs to be done in a certain way, then there's the problem with linux kernel not being able to mount UFS as rw, thus not being able to do a proper sysprep in an automated way and then uploading the sparsified+compressed images to OpenStack takes forever. All in all, I'm glad that Andrea came up with this patch series, because it passes the burden on someone else. On the other hand, having FreeBSD runners in the OpenStack environment was just a cherry on top, since lcitool is now capable of generating cloud-ready images and anyone can register their private runners to their gitlab private repo clone, so this series combined with lcitool gives IMO a very solid base to our contributors to test in different environments in an automated fashion (depending on the specific setup of course).
The only variation I can see here is to write the gitlab-ci.yml such that forks do their pre-merge CI against Cirrus, but the master repo does its post-merge CI again our custom runner. I do strongly prefer open source infra wherever possible, but I'm not convinced the FreeBSD CI is a case where its worth the hassle of maintaining FreeBSD runners.... at least not unless we get to a point where we need runners for integration testing on FreeBSD which is not any time soon.
That indeed won't be any time soon, we're getting close to enabling this on CentOS-Stream and Fedoras with TCK executed through the current Avocado WIP. Erik
Since we now use Cirrus CI for macOS jobs, we no longer need to keep Travis CI around. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- .travis.yml | 58 ----------------------------------------------------- 1 file changed, 58 deletions(-) delete mode 100644 .travis.yml diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index 7c8a114054..0000000000 --- a/.travis.yml +++ /dev/null @@ -1,58 +0,0 @@ -language: c -compiler: clang -os: osx - -branches: - except: - - /^.*-maint$/ - -addons: - homebrew: - update: true - packages: - - ccache - - rpcgen - - xz - - yajl - - glib - - docutils - - gnutls - -matrix: - include: - - osx_image: xcode11.3 - - osx_image: xcode10.3 - -env: - global: - - PATH="/usr/local/opt/gettext/bin:/usr/local/opt/ccache/libexec:/usr/local/opt/rpcgen/bin:$PATH" - - PKG_CONFIG_PATH="/usr/local/opt/libxml2/lib/pkgconfig" - -before_script: - # Hack to blow away py2 - - brew link --overwrite python - -script: - # We can't run 'distcheck' or 'syntax-check' because they fail on - # macOS, but doing 'install' and 'dist' gives us some useful coverage - - mkdir build && cd build - - ../autogen.sh --prefix=$(pwd)/install-root && make -j3 && make -j3 install && make -j3 dist - -git: - submodules: true - -notifications: - irc: - # The channel name "irc.oftc.net#virt" is encrypted against libvirt/libvirt - # to prevent IRC notifications from github forks. This was created using: - # $ travis encrypt -r "libvirt/libvirt" "irc.oftc.net#virt" - channels: - - secure: "hUPdkLxX7nh75+clpnk4U0XLExLfV9DFKSvQSAUtf5JtDNMslj7AeOCf2wcbkNsEhkiF557odTAnov1s5m1w/yaa56zbjFAh5agzqRKya3QjqsrvlBKw/WuN+l82iMNLLeebTgCPAXrbAbGWH8YmYssp/7+eMsnKaVh84EQQNbMCHlLg6ovE26Fs18mZ6J5RC3OPa1vbv+xkdCHvGg/Oyp4K8bpU7RYyimA56jdxI/OfdTH9HxntHYSzykR7hDbyzZhdIlAUyRKReQVjcV5+R8fdDL/1imyGA/88KTztMeKXpZ5Rf+Ss3vYLZb6qsLLegCZ4AU/q0vvbWxjpZGJZoeyrVpfBTZdYGIzmLTMl9GYXXa/gDwFlbvRDiPDG4TIy6GlMUROinj7KRKEHu1fWRYu012ife5OjidxcwrTnz21vYaCv3AKWPpMPxwIzQPkY1hex9uLLX6z+TrAxxDLF+7UzRT9w2RLFBkLYlj2aDVrLAVb/ynRsxDz5CGzC61FSQVft2e308SkGjdn8YxvguCuXv+N70Fu1cvFyh5XYeHb4fbBRo0Ctzaec78leHlQvRGWKJxXDXRkE2lvvBc7YbBNSAYh7Fs8Y+zY7l7rMxvXdrt3nuaNQhe74V3yhxPDAld66qmAn9TYMmaZW2f5/KKKILLbCa0t2MxiAc6L2OI8=" - on_success: change - on_failure: always - email: - # The list name 'libvirt-ci@redhat.com" is encrypted against libvirt/libvirt - # to prevent IRC notifications from github forks. This was created using: - # $ travis encrypt -r "libvirt/libvirt" "libvirt-ci@redhat.com" - recipients: - - secure: "QcU9eP96P0RlDNzVRZl/4sxyydPStGzECrpgJhr2IPB/7pHk23yaBrmUsq9S830tB+jwLGma1IscNB8uf7Sf7WY+cYIpfR8v030OffWnaipo/Gcs0dpnlfURWHjOFQI3RJzGEihsqvbwUFOwsM+3IDyO3qdWaiT6cN2Tj9ROlwYCySSX5YWzLyX7arBZ4lp8ESs7ohQaEwp2cegnMP2oGPJJe4SebvlCDjHZbjkU5aEradwUWnRQDJZWTKknpNLArVFxN2/ixp6f/MGY4DmkHoDweio6mHIPN5zTs5Jt32aiX6wDBa+bBa4v8TCRqzhYkQ63ZZhNV8bY5Uf9ufTdyvt96yIANyakd85b1QpMdAX76IyJi1l0/Uub6DTQZAcq3vK7iPjGeTVSpyoXrqTfGy4JxMjqDoocpWvv8ALX1wrYI/HfN2R2Aepw9jModTimOsebYhJ1yMhSt8qnh5AQNftGKL2JBKoA1LWdU2YJ5fO1bGjKNiVEkGFQTPYFWrYCUY5JcT+s5WCzNeMNm8s9na8liYhGl3WtS3rPr5M8bof+BMsBhG2hQ0loduc94x2GkvyhQZUgRbqrwNR+y4hn+rWFC3hBzzyiAULs43vY/PJ+eBdKEf3VAc0MkhQ8GgXGSA61fR6aXYonroI/WnBVItwDmUnnMfSziZXxk09GLl4=" -- 2.25.4
On Mon, Jun 08, 2020 at 05:42:31PM +0200, Andrea Bolognani wrote:
To see this in action, check out the
https://gitlab.com/abologna/libvirt/-/pipelines/153963978
pipeline, and in particular the
https://gitlab.com/abologna/libvirt/-/jobs/585678408 https://gitlab.com/abologna/libvirt/-/jobs/585678413
Ideally the actual build output from Cirrus CI would be pulled into the build output of the GitLab CI job. That way email notifications from failed builds will include the actual compile failure details. This also avoids having to cut+paste non-clickable links to the Cirrus CI job. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, 2020-06-08 at 17:34 +0100, Daniel P. Berrangé wrote:
On Mon, Jun 08, 2020 at 05:42:31PM +0200, Andrea Bolognani wrote:
To see this in action, check out the
https://gitlab.com/abologna/libvirt/-/pipelines/153963978
pipeline, and in particular the
https://gitlab.com/abologna/libvirt/-/jobs/585678408 https://gitlab.com/abologna/libvirt/-/jobs/585678413
Ideally the actual build output from Cirrus CI would be pulled into the build output of the GitLab CI job. That way email notifications from failed builds will include the actual compile failure details. This also avoids having to cut+paste non-clickable links to the Cirrus CI job.
I agree that this would be desirable. Earlier today, while reviewing the code for cirrus-run, I found a reference to a CIRRUS_LOG_FILE environment variable that should allow us to do just that, and was intending to try it out. If that doesn't work out, we should still be able to fetch the full job output from Cirrus CI after the fact. -- Andrea Bolognani / Red Hat / Virtualization
On Mon, 2020-06-08 at 19:07 +0200, Andrea Bolognani wrote:
On Mon, 2020-06-08 at 17:34 +0100, Daniel P. Berrangé wrote:
On Mon, Jun 08, 2020 at 05:42:31PM +0200, Andrea Bolognani wrote:
To see this in action, check out the
https://gitlab.com/abologna/libvirt/-/pipelines/153963978
pipeline, and in particular the
https://gitlab.com/abologna/libvirt/-/jobs/585678408 https://gitlab.com/abologna/libvirt/-/jobs/585678413
Ideally the actual build output from Cirrus CI would be pulled into the build output of the GitLab CI job. That way email notifications from failed builds will include the actual compile failure details. This also avoids having to cut+paste non-clickable links to the Cirrus CI job.
I agree that this would be desirable.
Earlier today, while reviewing the code for cirrus-run, I found a reference to a CIRRUS_LOG_FILE environment variable that should allow us to do just that, and was intending to try it out. If that doesn't work out, we should still be able to fetch the full job output from Cirrus CI after the fact.
I tried setting CIRRUS_LOG_FILE and it doesn't seem to work the way we want. Getting the taks log directly from Cirrus CI is definitely possible - you can do so from the Web UI already - but obtaining the correct URL requires calling the GraphQL API and I need some more time to figure out how that works. Possibly make some changes to cirrus-run as well. Since it's already possible, although not very convenient, to get to the full log, I think this should not block the series and it can come as a follow-up instead. I'll post a v2 that addresses the other issue you pointed out. -- Andrea Bolognani / Red Hat / Virtualization
On Tue, Jun 09, 2020 at 11:53:41AM +0200, Andrea Bolognani wrote:
On Mon, 2020-06-08 at 19:07 +0200, Andrea Bolognani wrote:
On Mon, 2020-06-08 at 17:34 +0100, Daniel P. Berrangé wrote:
On Mon, Jun 08, 2020 at 05:42:31PM +0200, Andrea Bolognani wrote:
To see this in action, check out the
https://gitlab.com/abologna/libvirt/-/pipelines/153963978
pipeline, and in particular the
https://gitlab.com/abologna/libvirt/-/jobs/585678408 https://gitlab.com/abologna/libvirt/-/jobs/585678413
Ideally the actual build output from Cirrus CI would be pulled into the build output of the GitLab CI job. That way email notifications from failed builds will include the actual compile failure details. This also avoids having to cut+paste non-clickable links to the Cirrus CI job.
I agree that this would be desirable.
Earlier today, while reviewing the code for cirrus-run, I found a reference to a CIRRUS_LOG_FILE environment variable that should allow us to do just that, and was intending to try it out. If that doesn't work out, we should still be able to fetch the full job output from Cirrus CI after the fact.
I tried setting CIRRUS_LOG_FILE and it doesn't seem to work the way we want. Getting the taks log directly from Cirrus CI is definitely possible - you can do so from the Web UI already - but obtaining the correct URL requires calling the GraphQL API and I need some more time to figure out how that works. Possibly make some changes to cirrus-run as well.
Since it's already possible, although not very convenient, to get to the full log, I think this should not block the series and it can come as a follow-up instead. I'll post a v2 that addresses the other issue you pointed out.
Yep, that's fine Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (3)
-
Andrea Bolognani -
Daniel P. Berrangé -
Erik Skultety