[libvirt PATCH 1/2] ci: Enable Cirrus CI integration

We use cirrus-run to trigger Cirrus CI jobs from GitLab CI jobs, making it possible to extend our platform coverage to include FreeBSD without having to maintain our own runners; additionally, we'll be able to ditch Travis CI and, since results for Cirrus CI jobs are reflected back to the GitLab CI jobs that triggered them, we will be able to get all information from a single dashboard. The FreeBSD and macOS job definitions can be improved further: for example, we will want to enable caching to speed up builds, and ultimately we should figure out a way to generate at least part of them, notably the list of packages to be installed, using lcitool. All of that will happen in later patches: for now, this is good enough to start using Cirrus CI. Signed-off-by: Andrea Bolognani <abologna(a)redhat.com&gt; --- .gitlab-ci.yml | 36 ++++++++++++++++++ ci/README.rst | 59 ++++++++++++++++++++++++++++++ ci/cirrus/freebsd-11.yml.j2 | 73 +++++++++++++++++++++++++++++++++++++ ci/cirrus/freebsd-12.yml.j2 | 73 +++++++++++++++++++++++++++++++++++++ ci/cirrus/macos-1015.yml.j2 | 38 +++++++++++++++++++ 5 files changed, 279 insertions(+) create mode 100644 ci/README.rst create mode 100644 ci/cirrus/freebsd-11.yml.j2 create mode 100644 ci/cirrus/freebsd-12.yml.j2 create mode 100644 ci/cirrus/macos-1015.yml.j2 diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 7113a1283c..bded69224b 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -38,6 +38,27 @@ stages: - master - /^ci-full-.*$/ +# Jobs that we delegate to Cirrus CI because they require an operating +# system other than Linux +.cirrus_build_default_job_template: &cirrus_build_default_job_definition + stage: native_build + image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master + script: + - | + if test -z "$CIRRUS_GITHUB_REPO" || test -z "$CIRRUS_API_TOKEN"; then + echo "WARN: Cirrus CI integration not set up correctly for your account" + echo "WARN: This job will be skipped" + echo "WARN: See ci/README.rst for more information" + exit 0 + fi + - cirrus-run ci/cirrus/$NAME.yml.j2 + +.cirrus_build_extra_job_template: &cirrus_build_extra_job_definition + <<: *cirrus_build_default_job_definition + only: + - master + - /^ci-full-.*$/ + # Default cross build jobs that are always run .cross_build_default_job_template: &cross_build_default_job_definition @@ -109,6 +130,21 @@ x64-ubuntu-2004: <<: *native_build_default_job_definition image: quay.io/libvirt/buildenv-libvirt-ubuntu-2004:latest +x64-freebsd-11-build: + <<: *cirrus_build_extra_job_definition + variables: + NAME: freebsd-11 + +x64-freebsd-12-build: + <<: *cirrus_build_default_job_definition + variables: + NAME: freebsd-12 + +x64-macos-1015-build: + <<: *cirrus_build_default_job_definition + variables: + NAME: macos-1015 + # Cross compiled build jobs diff --git a/ci/README.rst b/ci/README.rst new file mode 100644 index 0000000000..033db1c847 --- /dev/null +++ b/ci/README.rst @@ -0,0 +1,59 @@ +============== +CI for libvirt +============== + +This document provides some information related to the CI capabilities for the +libvirt project. + + +Cirrus CI integration +===================== + +libvirt currently supports three non-Linux operating systems: Windows, FreeBSD +and macOS. Windows cross-builds can be prepared on Linux by using `MinGW`_, but +for both FreeBSD and macOS we need to use the actual operating system, and +unfortunately GitLab shared runners are currently not available for either. + +To work around this limitation, we take advantage of `Cirrus CI`_'s free +offering: more specifically, we use the `cirrus-run`_ script to trigger Cirrus +CI jobs from GitLab CI jobs so that the workaround is almost entirely +transparent to users and there's no need to constantly check two separate CI +dashboards. + +There is, however, some one-time setup required. If you want FreeBSD and macOS +builds to happen when you push to your GitLab repository, you need to + +* set up a GitHub repository for the project, eg. ``yourusername/libvirt``. + This repository needs to exist for cirrus-run to work, but it doesn't need to + be kept up to date, so you can create it and then forget about it; + +* enable the `Cirrus CI GitHub app`_ for your GitHub account; + +* sign up for Cirrus CI. It's enough to log into the website using your GitHub + account; + +* grab an API token from the `Cirrus CI settings`_ page; + +* in the *CI/CD / Variables* section of the settings page for your GitLab + repository, create two new variables: + + * ``CIRRUS_GITHUB_REPO``, containing the name of the GitHub repository + created earlier, eg. ``yourusername/libvirt``; + + * ``CIRRUS_API_TOKEN``, containing the Cirrus CI API token generated earlier. + This variable **must** be marked as *Masked*, because anyone with knowledge + of it can impersonate you as far as Cirrus CI is concerned. + + Neither of these variables should be marked as *Protected*, because in + general you'll want to be able to trigger Cirrus CI builds from non-protected + branches. + +Once this one-time setup is complete, you can just keep pushing to your GitLab +repository as usual and you'll automatically get the additional CI coverage. + + +.. _Cirrus CI GitHub app: https://github.com/marketplace/cirrus-ci +.. _Cirrus CI settings: https://cirrus-ci.com/settings/profile/ +.. _Cirrus CI: https://cirrus-ci.com/ +.. _MinGW: http://mingw.org/ +.. _cirrus-run: https://github.com/sio/cirrus-run/ diff --git a/ci/cirrus/freebsd-11.yml.j2 b/ci/cirrus/freebsd-11.yml.j2 new file mode 100644 index 0000000000..e8df443d21 --- /dev/null +++ b/ci/cirrus/freebsd-11.yml.j2 @@ -0,0 +1,73 @@ +freebsd_instance: + image_family: freebsd-11-3 + +env: + CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }} + CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }} + CI_COMMIT_SHA: {{ CI_COMMIT_SHA }} + +freebsd_11_task: + install_script: + - pkg install -y + augeas + autoconf + automake + avahi + bash + bash-completion + ca_root_nss + ccache + chrony + cppi + curl + cyrus-sasl + dbus + diskscrub + dnsmasq + fusefs-libs + gdb + gettext + gettext-tools + git + glib + gmake + gnutls + hal + libpcap + libpciaccess + libssh + libssh2 + libtool + libxml2 + libxslt + lsof + meson + ncurses + ninja + patch + perl5 + pkgconf + polkit + py37-docutils + py37-flake8 + py37-setuptools + py37-wheel + python3 + qemu-utils + radvd + readline + screen + sudo + vim + yajl + clone_script: + - git clone --depth 100 "$CI_REPOSITORY_URL" . + - git fetch origin "$CI_COMMIT_REF_NAME" + - git reset --hard "$CI_COMMIT_SHA" + build_script: + - mkdir build + - cd build + - ../autogen.sh --prefix=$(pwd)/install-root + - gmake -j3 + - gmake -j3 install + - gmake -j3 dist diff --git a/ci/cirrus/freebsd-12.yml.j2 b/ci/cirrus/freebsd-12.yml.j2 new file mode 100644 index 0000000000..a653996850 --- /dev/null +++ b/ci/cirrus/freebsd-12.yml.j2 @@ -0,0 +1,73 @@ +freebsd_instance: + image_family: freebsd-12-1 + +env: + CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }} + CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }} + CI_COMMIT_SHA: {{ CI_COMMIT_SHA }} + +freebsd_12_task: + install_script: + - pkg install -y + augeas + autoconf + automake + avahi + bash + bash-completion + ca_root_nss + ccache + chrony + cppi + curl + cyrus-sasl + dbus + diskscrub + dnsmasq + fusefs-libs + gdb + gettext + gettext-tools + git + glib + gmake + gnutls + hal + libpcap + libpciaccess + libssh + libssh2 + libtool + libxml2 + libxslt + lsof + meson + ncurses + ninja + patch + perl5 + pkgconf + polkit + py37-docutils + py37-flake8 + py37-setuptools + py37-wheel + python3 + qemu-utils + radvd + readline + screen + sudo + vim + yajl + clone_script: + - git clone --depth 100 "$CI_REPOSITORY_URL" . + - git fetch origin "$CI_COMMIT_REF_NAME" + - git reset --hard "$CI_COMMIT_SHA" + build_script: + - mkdir build + - cd build + - ../autogen.sh --prefix=$(pwd)/install-root + - gmake -j3 + - gmake -j3 install + - gmake -j3 dist diff --git a/ci/cirrus/macos-1015.yml.j2 b/ci/cirrus/macos-1015.yml.j2 new file mode 100644 index 0000000000..d8aa9715d6 --- /dev/null +++ b/ci/cirrus/macos-1015.yml.j2 @@ -0,0 +1,38 @@ +osx_instance: + image: catalina-base + +env: + CI_REPOSITORY_URL: {{ CI_REPOSITORY_URL }} + CI_COMMIT_REF_NAME: {{ CI_COMMIT_REF_NAME }} + CI_COMMIT_SHA: {{ CI_COMMIT_SHA }} + PATH: /usr/local/opt/gettext/bin:/usr/local/opt/ccache/libexec:/usr/local/opt/rpcgen/bin:$PATH + PKG_CONFIG_PATH: /usr/local/opt/libxml2/lib/pkgconfig + +macos_1015_task: + install_script: + - brew install + autoconf + automake + ccache + docutils + glib + gnutls + libtool + libxml2 + make + pkg-config + python + rpcgen + xz + yajl + clone_script: + - git clone --depth 100 "$CI_REPOSITORY_URL" . + - git fetch origin "$CI_COMMIT_REF_NAME" + - git reset --hard "$CI_COMMIT_SHA" + build_script: + - mkdir build + - cd build + - ../autogen.sh --prefix=$(pwd)/install-root + - gmake -j3 + - gmake -j3 install + - gmake -j3 dist -- 2.25.4