This introduces CI jobs that replace the current jobs used on Jenkins for every platform except FreeBSD. A merge request workflow requires the user to fork the primary git repo into their personal namespace. In general the changes need to be tested against the current libvirt git master. If the user has a fork of the main libvirt repo, we don't want to use that by default as it may be out of date. The general goal is that the CI jobs are self-contained and don't depend on the build artifacts from the libvirt repo. We also want to avoid having an explicit dependency on the libvirt-ci repo, or on the Quay.io service. Contributors to the PHP module need to be able to make code changes which imply CI environment changes and be able to test them in isolation. Thus, the dockerfile recipes for each distro are added in the ci/ sub-directory. The first stage of the CI jobs is to use these recipes to build and publish a container image. These images are then used in the second stage to perform the actual build. The container image build is cached, inheriting from both the primary libvirt project namespace, and the user's private project namespace. Thus the performance hit of building container images will only be felt the first time the project is forked, or when the parent Docker images are rebuilt. The dockerfiles were originally generated using lcitool, but if the user makes a change that introduces new build dependencies, the corresponding packages can be added to the dockerfile recipes directly in the same commit. The change can be propagated back into the libvirt-ci.git repo asynchronously. The build job will do a minimal(-ish) build of libvirt git master and then build the rest of the code against that. Ideally the main libvirt configure script would have a way to request a minimal build of just the API and test driver, but for now we settle for just --without-libvirtd which culls a large number of the drivers fairly easily. Neither CentOS nor Ubuntu have the required perl deps to run the TCK so those platforms are currently skipped. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- See example pipeline run: https://gitlab.com/berrange/libvirt-tck/-/pipelines/153948849 .gitlab-ci.yml | 115 +++++++++++++++++++++++++++ ci/libvirt-debian-10.Dockerfile | 82 +++++++++++++++++++ ci/libvirt-debian-9.Dockerfile | 86 ++++++++++++++++++++ ci/libvirt-debian-sid.Dockerfile | 82 +++++++++++++++++++ ci/libvirt-fedora-31.Dockerfile | 81 +++++++++++++++++++ ci/libvirt-fedora-32.Dockerfile | 81 +++++++++++++++++++ ci/libvirt-fedora-rawhide.Dockerfile | 82 +++++++++++++++++++ ci/refresh | 22 +++++ 8 files changed, 631 insertions(+) create mode 100644 ci/libvirt-debian-10.Dockerfile create mode 100644 ci/libvirt-debian-9.Dockerfile create mode 100644 ci/libvirt-debian-sid.Dockerfile create mode 100644 ci/libvirt-fedora-31.Dockerfile create mode 100644 ci/libvirt-fedora-32.Dockerfile create mode 100644 ci/libvirt-fedora-rawhide.Dockerfile create mode 100755 ci/refresh diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 50dae92..3d4d7db 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -1,6 +1,60 @@ stages: - prebuild + - containers + - builds + +.container_job_template: &container_job_definition + image: docker:stable + stage: containers + services: + - docker:dind + before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest" + - export COMMON_TAG="$CI_REGISTRY/libvirt/libvirt-perl/ci-$NAME:latest" + - docker info + - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" + script: + - docker pull "$TAG" || docker pull "$COMMON_TAG" || true + - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "ci/libvirt-$NAME.Dockerfile" ci + - docker push "$TAG" + after_script: + - docker logout + +.git_build_job_template: &git_build_job_definition + image: $CI_REGISTRY_IMAGE/ci-$NAME:latest + stage: builds + before_script: + - export MAKEFLAGS="-j$(getconf _NPROCESSORS_ONLN)" + - export SCRATCH_DIR="/tmp/scratch" + - export VROOT="$SCRATCH_DIR/vroot" + - export LD_LIBRARY_PATH="$VROOT/lib" + - export PATH="$VROOT/bin:$PATH" + - export PKG_CONFIG_PATH="$VROOT/lib/pkgconfig" + - export TEST_MAINTAINER=1 + - eval `perl -V:archname` + - export PERL5LIB="$VROOT/lib/perl5/$archname" + script: + - pushd "$PWD" + - mkdir -p "$SCRATCH_DIR" + - cd "$SCRATCH_DIR" + - git clone --depth 1 https://gitlab.com/libvirt/libvirt.git + - git clone --depth 1 https://gitlab.com/libvirt/libvirt-perl.git + - mkdir libvirt/build + - cd libvirt/build + - ../autogen.sh --prefix="$VROOT" --without-libvirtd + - $MAKE install + - cd ../../libvirt-perl + - perl Build.PL install_base=$VROOT + - perl Build + - perl Build install + - popd + - perl Build.PL + - perl Build + - perl Build test + - perl Build install + - perl Build dist + - if test -x /usr/bin/rpmbuild ; then rpmbuild --nodeps -ta Sys-Virt-TCK*.tar.gz ; fi # Check that all commits are signed-off for the DCO. # Skip on "libvirt" namespace, since we only need to run @@ -14,3 +68,64 @@ check-dco: except: variables: - $CI_PROJECT_NAMESPACE == 'libvirt' + +debian-9-container: + <<: *container_job_definition + variables: + NAME: debian-9 + +debian-10-container: + <<: *container_job_definition + variables: + NAME: debian-10 + +debian-sid-container: + <<: *container_job_definition + variables: + NAME: debian-sid + +fedora-31-container: + <<: *container_job_definition + variables: + NAME: fedora-31 + +fedora-32-container: + <<: *container_job_definition + variables: + NAME: fedora-32 + +fedora-rawhide-container: + <<: *container_job_definition + variables: + NAME: fedora-rawhide + + +debian-9-git-build: + <<: *git_build_job_definition + variables: + NAME: debian-9 + +debian-10-git-build: + <<: *git_build_job_definition + variables: + NAME: debian-10 + +debian-sid-git-build: + <<: *git_build_job_definition + variables: + NAME: debian-sid + +fedora-31-git-build: + <<: *git_build_job_definition + variables: + NAME: fedora-31 + +fedora-32-git-build: + <<: *git_build_job_definition + variables: + NAME: fedora-32 + +fedora-rawhide-git-build: + <<: *git_build_job_definition + variables: + NAME: fedora-rawhide diff --git a/ci/libvirt-debian-10.Dockerfile b/ci/libvirt-debian-10.Dockerfile new file mode 100644 index 0000000..f881069 --- /dev/null +++ b/ci/libvirt-debian-10.Dockerfile @@ -0,0 +1,82 @@ +FROM debian:10 + +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get dist-upgrade -y && \ + apt-get install --no-install-recommends -y \ + autoconf \ + automake \ + autopoint \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + git \ + libaccessors-perl \ + libc-dev-bin \ + libc6-dev \ + libconfig-record-perl \ + libcpan-changes-perl \ + libdigest-perl-md5-perl \ + libfile-slurp-perl \ + libglib2.0-dev \ + libgnutls28-dev \ + libio-compress-perl \ + libio-string-perl \ + libmodule-build-perl \ + libnl-3-dev \ + libnl-route-3-dev \ + libsub-uplevel-perl \ + libtap-formatter-html-perl \ + libtap-formatter-junit-perl \ + libtap-harness-archive-perl \ + libtest-exception-perl \ + libtest-lwp-useragent-perl \ + libtest-pod-coverage-perl \ + libtest-pod-perl \ + libtime-hr-perl \ + libtirpc-dev \ + libtool \ + libtool-bin \ + libxml-twig-perl \ + libxml-writer-perl \ + libxml-xpath-perl \ + libxml2-dev \ + libxml2-utils \ + locales \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconf \ + python3 \ + python3-docutils \ + python3-setuptools \ + python3-wheel \ + screen \ + strace \ + sudo \ + vim \ + xsltproc && \ + apt-get autoremove -y && \ + apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-debian-9.Dockerfile b/ci/libvirt-debian-9.Dockerfile new file mode 100644 index 0000000..3a161fe --- /dev/null +++ b/ci/libvirt-debian-9.Dockerfile @@ -0,0 +1,86 @@ +FROM debian:9 + +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get dist-upgrade -y && \ + apt-get install --no-install-recommends -y \ + autoconf \ + automake \ + autopoint \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + git \ + libaccessors-perl \ + libc-dev-bin \ + libc6-dev \ + libconfig-record-perl \ + libcpan-changes-perl \ + libdigest-perl \ + libdigest-perl-md5-perl \ + libfile-slurp-perl \ + libglib2.0-dev \ + libgnutls28-dev \ + libio-compress-perl \ + libio-string-perl \ + libmodule-build-perl \ + libnl-3-dev \ + libnl-route-3-dev \ + libsub-uplevel-perl \ + libtap-formatter-html-perl \ + libtap-formatter-junit-perl \ + libtap-harness-archive-perl \ + libtest-exception-perl \ + libtest-lwp-useragent-perl \ + libtest-pod-coverage-perl \ + libtest-pod-perl \ + libtime-hr-perl \ + libtirpc-dev \ + libtool \ + libtool-bin \ + libxml-twig-perl \ + libxml-writer-perl \ + libxml-xpath-perl \ + libxml2-dev \ + libxml2-utils \ + locales \ + lsof \ + make \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconf \ + python3 \ + python3-docutils \ + python3-pip \ + python3-setuptools \ + python3-wheel \ + screen \ + strace \ + sudo \ + vim \ + xsltproc && \ + apt-get autoremove -y && \ + apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +RUN pip3 install \ + meson==0.49.0 + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-debian-sid.Dockerfile b/ci/libvirt-debian-sid.Dockerfile new file mode 100644 index 0000000..bed01a0 --- /dev/null +++ b/ci/libvirt-debian-sid.Dockerfile @@ -0,0 +1,82 @@ +FROM debian:sid + +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get dist-upgrade -y && \ + apt-get install --no-install-recommends -y \ + autoconf \ + automake \ + autopoint \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + git \ + libaccessors-perl \ + libc-dev-bin \ + libc6-dev \ + libconfig-record-perl \ + libcpan-changes-perl \ + libdigest-perl-md5-perl \ + libfile-slurp-perl \ + libglib2.0-dev \ + libgnutls28-dev \ + libio-compress-perl \ + libio-string-perl \ + libmodule-build-perl \ + libnl-3-dev \ + libnl-route-3-dev \ + libsub-uplevel-perl \ + libtap-formatter-html-perl \ + libtap-formatter-junit-perl \ + libtap-harness-archive-perl \ + libtest-exception-perl \ + libtest-lwp-useragent-perl \ + libtest-pod-coverage-perl \ + libtest-pod-perl \ + libtime-hr-perl \ + libtirpc-dev \ + libtool \ + libtool-bin \ + libxml-twig-perl \ + libxml-writer-perl \ + libxml-xpath-perl \ + libxml2-dev \ + libxml2-utils \ + locales \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconf \ + python3 \ + python3-docutils \ + python3-setuptools \ + python3-wheel \ + screen \ + strace \ + sudo \ + vim \ + xsltproc && \ + apt-get autoremove -y && \ + apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-fedora-31.Dockerfile b/ci/libvirt-fedora-31.Dockerfile new file mode 100644 index 0000000..e72682c --- /dev/null +++ b/ci/libvirt-fedora-31.Dockerfile @@ -0,0 +1,81 @@ +FROM fedora:31 + +RUN dnf update -y && \ + dnf install -y \ + autoconf \ + automake \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + cppi \ + gcc \ + gdb \ + gettext \ + gettext-devel \ + git \ + glib2-devel \ + glibc-devel \ + glibc-langpack-en \ + gnutls-devel \ + libnl3-devel \ + libtirpc-devel \ + libtool \ + libxml2 \ + libxml2-devel \ + libxslt \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + perl-Archive-Tar \ + perl-CPAN-Changes \ + perl-Config-Record \ + perl-Digest \ + perl-Digest-MD5 \ + perl-File-Slurp \ + perl-IO-Compress-Bzip2 \ + perl-IO-String \ + perl-Module-Build \ + perl-Sub-Uplevel \ + perl-TAP-Formatter-HTML \ + perl-TAP-Formatter-JUnit \ + perl-TAP-Harness-Archive \ + perl-Test-Exception \ + perl-Test-LWP-UserAgent \ + perl-Test-Pod \ + perl-Test-Pod-Coverage \ + perl-Time-HiRes \ + perl-XML-Twig \ + perl-XML-Writer \ + perl-XML-XPath \ + perl-accessors \ + perl-generators \ + pkgconfig \ + python3 \ + python3-docutils \ + python3-setuptools \ + python3-wheel \ + rpcgen \ + rpm-build \ + screen \ + strace \ + sudo \ + vim && \ + dnf autoremove -y && \ + dnf clean all -y && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-fedora-32.Dockerfile b/ci/libvirt-fedora-32.Dockerfile new file mode 100644 index 0000000..9770d53 --- /dev/null +++ b/ci/libvirt-fedora-32.Dockerfile @@ -0,0 +1,81 @@ +FROM fedora:32 + +RUN dnf update -y && \ + dnf install -y \ + autoconf \ + automake \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + cppi \ + gcc \ + gdb \ + gettext \ + gettext-devel \ + git \ + glib2-devel \ + glibc-devel \ + glibc-langpack-en \ + gnutls-devel \ + libnl3-devel \ + libtirpc-devel \ + libtool \ + libxml2 \ + libxml2-devel \ + libxslt \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + perl-Archive-Tar \ + perl-CPAN-Changes \ + perl-Config-Record \ + perl-Digest \ + perl-Digest-MD5 \ + perl-File-Slurp \ + perl-IO-Compress-Bzip2 \ + perl-IO-String \ + perl-Module-Build \ + perl-Sub-Uplevel \ + perl-TAP-Formatter-HTML \ + perl-TAP-Formatter-JUnit \ + perl-TAP-Harness-Archive \ + perl-Test-Exception \ + perl-Test-LWP-UserAgent \ + perl-Test-Pod \ + perl-Test-Pod-Coverage \ + perl-Time-HiRes \ + perl-XML-Twig \ + perl-XML-Writer \ + perl-XML-XPath \ + perl-accessors \ + perl-generators \ + pkgconfig \ + python3 \ + python3-docutils \ + python3-setuptools \ + python3-wheel \ + rpcgen \ + rpm-build \ + screen \ + strace \ + sudo \ + vim && \ + dnf autoremove -y && \ + dnf clean all -y && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-fedora-rawhide.Dockerfile b/ci/libvirt-fedora-rawhide.Dockerfile new file mode 100644 index 0000000..4fd8b1e --- /dev/null +++ b/ci/libvirt-fedora-rawhide.Dockerfile @@ -0,0 +1,82 @@ +FROM fedora:rawhide + +RUN dnf update -y --nogpgcheck fedora-gpg-keys && \ + dnf update -y && \ + dnf install -y \ + autoconf \ + automake \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + cppi \ + gcc \ + gdb \ + gettext \ + gettext-devel \ + git \ + glib2-devel \ + glibc-devel \ + glibc-langpack-en \ + gnutls-devel \ + libnl3-devel \ + libtirpc-devel \ + libtool \ + libxml2 \ + libxml2-devel \ + libxslt \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + perl-Archive-Tar \ + perl-CPAN-Changes \ + perl-Config-Record \ + perl-Digest \ + perl-Digest-MD5 \ + perl-File-Slurp \ + perl-IO-Compress-Bzip2 \ + perl-IO-String \ + perl-Module-Build \ + perl-Sub-Uplevel \ + perl-TAP-Formatter-HTML \ + perl-TAP-Formatter-JUnit \ + perl-TAP-Harness-Archive \ + perl-Test-Exception \ + perl-Test-LWP-UserAgent \ + perl-Test-Pod \ + perl-Test-Pod-Coverage \ + perl-Time-HiRes \ + perl-XML-Twig \ + perl-XML-Writer \ + perl-XML-XPath \ + perl-accessors \ + perl-generators \ + pkgconfig \ + python3 \ + python3-docutils \ + python3-setuptools \ + python3-wheel \ + rpcgen \ + rpm-build \ + screen \ + strace \ + sudo \ + vim && \ + dnf autoremove -y && \ + dnf clean all -y && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/refresh b/ci/refresh new file mode 100755 index 0000000..657f4d0 --- /dev/null +++ b/ci/refresh @@ -0,0 +1,22 @@ +#!/bin/sh + +if test -z "$1" +then + echo "syntax: $0 PATH-TO-LCITOOL" + exit 1 +fi + +LCITOOL=$1 + +if ! test -x "$LCITOOL" +then + echo "$LCITOOL is not executable" + exit 1 +fi + +HOSTS=$($LCITOOL hosts | grep -v -E '(freebsd|ubuntu|suse|centos)') + +for host in $HOSTS +do + $LCITOOL dockerfile $host libvirt+minimal,libvirt-perl,libvirt-tck > $host.Dockerfile +done -- 2.26.2