[libvirt-python PATCH v3 0/3] gitlab: introduce CI jobs for validating python build

This introduces GitLab CI to the python module. Traditional Jenkins has validated the python build across all distros, using libvirt git. This tested one axis - a variety of python versions - but failed to test the other interesting axis - a variety of libvirt versions. This new CI setup fixes that mistake validating both axis. Daniel P. Berrangé (3): test: workaround missing VIR_TYPED_PARAM enums in API definition gitlab: introduce CI jobs testing git master & distro libvirt travis: delete redundant configuration .gitlab-ci.yml | 185 +++++++++++++++++++++++++++ .travis.yml | 55 -------- ci/README.rst | 13 ++ ci/libvirt-centos-7.Dockerfile | 86 +++++++++++++ ci/libvirt-centos-8.Dockerfile | 64 +++++++++ ci/libvirt-debian-10.Dockerfile | 56 ++++++++ ci/libvirt-debian-9.Dockerfile | 59 +++++++++ ci/libvirt-debian-sid.Dockerfile | 56 ++++++++ ci/libvirt-fedora-31.Dockerfile | 53 ++++++++ ci/libvirt-fedora-32.Dockerfile | 53 ++++++++ ci/libvirt-fedora-rawhide.Dockerfile | 54 ++++++++ ci/libvirt-opensuse-151.Dockerfile | 55 ++++++++ ci/libvirt-ubuntu-1804.Dockerfile | 59 +++++++++ ci/libvirt-ubuntu-2004.Dockerfile | 56 ++++++++ ci/refresh | 27 ++++ sanitytest.py | 6 +- 16 files changed, 881 insertions(+), 56 deletions(-) create mode 100644 .gitlab-ci.yml delete mode 100644 .travis.yml create mode 100644 ci/README.rst create mode 100644 ci/libvirt-centos-7.Dockerfile create mode 100644 ci/libvirt-centos-8.Dockerfile create mode 100644 ci/libvirt-debian-10.Dockerfile create mode 100644 ci/libvirt-debian-9.Dockerfile create mode 100644 ci/libvirt-debian-sid.Dockerfile create mode 100644 ci/libvirt-fedora-31.Dockerfile create mode 100644 ci/libvirt-fedora-32.Dockerfile create mode 100644 ci/libvirt-fedora-rawhide.Dockerfile create mode 100644 ci/libvirt-opensuse-151.Dockerfile create mode 100644 ci/libvirt-ubuntu-1804.Dockerfile create mode 100644 ci/libvirt-ubuntu-2004.Dockerfile create mode 100755 ci/refresh -- 2.25.4

On Ubuntu 18.04 with libvirt 4.0.0 libvirt-python build fails running test /usr/bin/python3 sanitytest.py build/lib.linux-x86_64-3.6 /usr/share/libvirt/api/libvirt-api.xml Cannot get a value of enum VIR_TYPED_PARAM_BOOLEAN (originally VIR_DOMAIN_BLKIO_PARAM_BOOLEAN) Cannot get a value of enum VIR_TYPED_PARAM_DOUBLE (originally VIR_DOMAIN_BLKIO_PARAM_DOUBLE) Cannot get a value of enum VIR_TYPED_PARAM_INT (originally VIR_DOMAIN_BLKIO_PARAM_INT) ...snip... The code generated for the binding is still correct and so we can just whitelist this error scenario. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- sanitytest.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/sanitytest.py b/sanitytest.py index 1bedd55..f187c15 100644 --- a/sanitytest.py +++ b/sanitytest.py @@ -73,7 +73,11 @@ for n in second_pass: val = int(v[val]) break - if type(val) != int: + # Version 4.0.0 was broken as missing VIR_TYPED_PARAM enums + # constants. This is harmless from POV of validating API + # coverage so ignore this error. + if (type(val) != int and + not val.startswith("VIR_TYPED_PARAM_")): fail = True print("Cannot get a value of enum %s (originally %s)" % (val, name)) enumvals[typ][name] = val -- 2.25.4

The python build needs to validate two axis - A variety of libvirt versions - A variety of python versions We get coverage for both these axis by running a build against the distro provided libvirt packages. All that is then missing is a build against the latest libvirt git master, which only needs to be run on a single distro, for which CentOS 8 is picked as a stable long life base. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 185 +++++++++++++++++++++++++++ ci/README.rst | 13 ++ ci/libvirt-centos-7.Dockerfile | 86 +++++++++++++ ci/libvirt-centos-8.Dockerfile | 64 +++++++++ ci/libvirt-debian-10.Dockerfile | 56 ++++++++ ci/libvirt-debian-9.Dockerfile | 59 +++++++++ ci/libvirt-debian-sid.Dockerfile | 56 ++++++++ ci/libvirt-fedora-31.Dockerfile | 53 ++++++++ ci/libvirt-fedora-32.Dockerfile | 53 ++++++++ ci/libvirt-fedora-rawhide.Dockerfile | 54 ++++++++ ci/libvirt-opensuse-151.Dockerfile | 55 ++++++++ ci/libvirt-ubuntu-1804.Dockerfile | 59 +++++++++ ci/libvirt-ubuntu-2004.Dockerfile | 56 ++++++++ ci/refresh | 27 ++++ 14 files changed, 876 insertions(+) create mode 100644 .gitlab-ci.yml create mode 100644 ci/README.rst create mode 100644 ci/libvirt-centos-7.Dockerfile create mode 100644 ci/libvirt-centos-8.Dockerfile create mode 100644 ci/libvirt-debian-10.Dockerfile create mode 100644 ci/libvirt-debian-9.Dockerfile create mode 100644 ci/libvirt-debian-sid.Dockerfile create mode 100644 ci/libvirt-fedora-31.Dockerfile create mode 100644 ci/libvirt-fedora-32.Dockerfile create mode 100644 ci/libvirt-fedora-rawhide.Dockerfile create mode 100644 ci/libvirt-opensuse-151.Dockerfile create mode 100644 ci/libvirt-ubuntu-1804.Dockerfile create mode 100644 ci/libvirt-ubuntu-2004.Dockerfile create mode 100755 ci/refresh diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 0000000..bc95d41 --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,185 @@ + +stages: + - prebuild + - containers + - builds + - docs + +.container_job_template: &container_job_definition + image: docker:stable + stage: containers + services: + - docker:dind + before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest" + - export COMMON_TAG="$CI_REGISTRY/libvirt/libvirt-perl/ci-$NAME:latest" + - docker info + - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" + script: + - docker pull "$TAG" || docker pull "$COMMON_TAG" || true + - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "ci/libvirt-$NAME.Dockerfile" ci + - docker push "$TAG" + after_script: + - docker logout + +.build_git_job_template: &build_git_job_definition + image: $CI_REGISTRY_IMAGE/ci-$NAME:latest + stage: builds + before_script: + - export MAKEFLAGS="-j$(getconf _NPROCESSORS_ONLN)" + - export SCRATCH_DIR="/tmp/scratch" + - export VROOT="$SCRATCH_DIR/vroot" + - export LD_LIBRARY_PATH="$VROOT/lib" + - export PATH="$VROOT/bin:$PATH" + - export PKG_CONFIG_PATH="$VROOT/lib/pkgconfig" + script: + - pushd "$PWD" + - mkdir -p "$SCRATCH_DIR" + - cd "$SCRATCH_DIR" + - git clone --depth 1 https://gitlab.com/libvirt/libvirt.git + - mkdir libvirt/build + - cd libvirt/build + - ../autogen.sh --prefix="$VROOT" --without-libvirtd + - $MAKE install + - popd + - $PYTHON setup.py build + - $PYTHON setup.py test + +.build_dist_job_template: &build_dist_job_definition + image: $CI_REGISTRY_IMAGE/ci-$NAME:latest + stage: builds + before_script: + - export MAKEFLAGS="-j$(getconf _NPROCESSORS_ONLN)" + script: + - $PYTHON setup.py build + - $PYTHON setup.py test + +# Check that all commits are signed-off for the DCO. +# Skip on "libvirt" namespace, since we only need to run +# this test on developer's personal forks from which +# merge requests are submitted +check-dco: + stage: prebuild + image: registry.gitlab.com/libvirt/libvirt-ci/check-dco:master + script: + - /check-dco + except: + variables: + - $CI_PROJECT_NAMESPACE == 'libvirt' + +centos-7-container: + <<: *container_job_definition + variables: + NAME: centos-7 + +centos-8-container: + <<: *container_job_definition + variables: + NAME: centos-8 + +debian-9-container: + <<: *container_job_definition + variables: + NAME: debian-9 + +debian-10-container: + <<: *container_job_definition + variables: + NAME: debian-10 + +debian-sid-container: + <<: *container_job_definition + variables: + NAME: debian-sid + +fedora-31-container: + <<: *container_job_definition + variables: + NAME: fedora-31 + +fedora-32-container: + <<: *container_job_definition + variables: + NAME: fedora-32 + +fedora-rawhide-container: + <<: *container_job_definition + variables: + NAME: fedora-rawhide + +opensuse-151-container: + <<: *container_job_definition + variables: + NAME: opensuse-151 + +ubuntu-1804-container: + <<: *container_job_definition + variables: + NAME: ubuntu-1804 + +ubuntu-2004-container: + <<: *container_job_definition + variables: + NAME: ubuntu-2004 + + +centos-8-git-build: + <<: *build_git_job_definition + variables: + NAME: centos-8 + + +centos-7-dist-build: + <<: *build_dist_job_definition + variables: + NAME: centos-7 + +centos-8-dist-build: + <<: *build_dist_job_definition + variables: + NAME: centos-8 + +debian-9-dist-build: + <<: *build_dist_job_definition + variables: + NAME: debian-9 + +debian-10-dist-build: + <<: *build_dist_job_definition + variables: + NAME: debian-10 + +debian-sid-dist-build: + <<: *build_dist_job_definition + variables: + NAME: debian-sid + +fedora-31-dist-build: + <<: *build_dist_job_definition + variables: + NAME: fedora-31 + +fedora-32-dist-build: + <<: *build_dist_job_definition + variables: + NAME: fedora-32 + +fedora-rawhide-dist-build: + <<: *build_dist_job_definition + variables: + NAME: fedora-rawhide + +opensuse-151-dist-build: + <<: *build_dist_job_definition + variables: + NAME: opensuse-151 + +ubuntu-1804-dist-build: + <<: *build_dist_job_definition + variables: + NAME: ubuntu-1804 + +ubuntu-2004-dist-build: + <<: *build_dist_job_definition + variables: + NAME: ubuntu-2004 diff --git a/ci/README.rst b/ci/README.rst new file mode 100644 index 0000000..009a981 --- /dev/null +++ b/ci/README.rst @@ -0,0 +1,13 @@ +CI job assets +============= + +This directory contains assets used in the automated CI jobs, most +notably the Dockerfiles used to build container images in which the +CI jobs then run. + +The ``refresh`` script is used to re-create the Dockerfiles using the +``lcitool`` that is provided by repo https://gitlab.com/libvirt/libvirt-ci + +The containers are built during the CI process and cached in the GitLab +container registry of the project doing the build. The cached containers +can be deleted at any time and will be correctly rebuilt. diff --git a/ci/libvirt-centos-7.Dockerfile b/ci/libvirt-centos-7.Dockerfile new file mode 100644 index 0000000..74eba4e --- /dev/null +++ b/ci/libvirt-centos-7.Dockerfile @@ -0,0 +1,86 @@ +FROM centos:7 + +RUN echo -e '[openvz]\n\ +name=OpenVZ addons\n\ +baseurl=https://download.openvz.org/virtuozzo/releases/openvz-7.0.11-235/x86_64/os/\n\ +enabled=1\n\ +gpgcheck=1\n\ +skip_if_unavailable=0\n\ +metadata_expire=6h\n\ +priority=90\n\ +includepkgs=libprl*' > /etc/yum.repos.d/openvz.repo && \ + echo -e '-----BEGIN PGP PUBLIC KEY BLOCK-----\n\ +Version: GnuPG v2.0.22 (GNU/Linux)\n\ +\n\ +mI0EVl80nQEEAKrEeyeTCwrzS9kYedZ/sAc/GUqlb81C7pA9SaR3fyck5mVw1Ogk\n\ +YdmNBPM2kY7QDxR9F0EpSpnxSCAXZXugsQ8KzZ0DRLVeBDQyGs9IGK5hI0zzxIil\n\ +BzfvIexLiQQhLy7YlIi8Jt/uUqKkW0pIMNMGcduY97VATtczpncpkmSzABEBAAG0\n\ +SFZpcnR1b3p6byBUZWFtIChHUEcga2V5IHNpZ25hdHVyZSBmb3IgcGFja2FnZXMp\n\ +IDxzZWN1cml0eUB2aXJ0dW96em8uY29tPoi5BBMBAgAjBQJWXzSdAhsDBwsJCAcD\n\ +AgEGFQgCCQoLBBYCAwECHgECF4AACgkQygt9GUTNrSruIgP/er70Eyo73A1gfrjv\n\ +oPUkyo4rslVRZu3qqCwoMFtJc/Z/UxWgEka1buorlcGLa6eO/EZ49c0n+KGa4Kvt\n\ +EUboIq0yEu5i0FyAj92ifm+hNhoAbGfm0cZ4/fD0oGr3l8OsQo4+iHX4xAPwFe7Y\n\ +zABuB8I1ZDZ4OIp5tDfTTuF2LT24jQRWXzSdAQQAog2Aqb+Ptl68O7cQhWLjVGkj\n\ +yyigZrdeReLx3HloKJPBeQ/kA6uvMJc/IYS3uppMWXv9v+QenS6uhP1TUJ2k9FvM\n\ +t94MQZfALN7Vpf8AF+UeWu4Ru+y4BNzcFhrPhIFNFChOR2QqW6FkgE57D9I177NC\n\ +oJMyrlNe8wcGa178An8AEQEAAYifBBgBAgAJBQJWXzSdAhsMAAoJEMoLfRlEza0q\n\ +bKwD/3+OFVIEXnIv5XgdGRNX5fHggsUN1bb8gva7HANRlKdd4LD8foDM3F/yv/3V\n\ +igG14D5EjKz56SaBDNgiI4++hOzb2M8jhAsR86jxkXFrrP1U3ZNRKg6av9DPFAPS\n\ +WEiJKtQrZDJloqtyi/mmRa1VsV7RYR0VPJjhK/R8EQ7Ysshy\n\ +=fRMg\n\ +-----END PGP PUBLIC KEY BLOCK-----' > /etc/pki/rpm-gpg/RPM-GPG-KEY-OpenVZ && \ + rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-OpenVZ && \ + yum install -y epel-release && \ + yum update -y && \ + yum install -y \ + autoconf \ + automake \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + gettext-devel \ + git \ + glibc-common \ + glibc-devel \ + libtool \ + libvirt-devel \ + lsof \ + make \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconfig \ + python3 \ + python3-pip \ + python3-setuptools \ + python3-wheel \ + python36-devel \ + python36-lxml \ + python36-nose \ + rpm-build \ + screen \ + strace \ + sudo \ + vim && \ + yum autoremove -y && \ + yum clean all -y && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +RUN pip3 install \ + meson==0.49.0 + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja-build" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-centos-8.Dockerfile b/ci/libvirt-centos-8.Dockerfile new file mode 100644 index 0000000..3d0eb68 --- /dev/null +++ b/ci/libvirt-centos-8.Dockerfile @@ -0,0 +1,64 @@ +FROM centos:8 + +RUN dnf install 'dnf-command(config-manager)' -y && \ + dnf config-manager --set-enabled PowerTools -y && \ + dnf install -y epel-release && \ + dnf update -y && \ + dnf install -y \ + autoconf \ + automake \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + gettext-devel \ + git \ + glib2-devel \ + glibc-devel \ + glibc-langpack-en \ + gnutls-devel \ + libnl3-devel \ + libtirpc-devel \ + libtool \ + libvirt-devel \ + libxml2 \ + libxml2-devel \ + libxslt \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconfig \ + python3 \ + python3-devel \ + python3-docutils \ + python3-lxml \ + python3-nose \ + python3-setuptools \ + python3-wheel \ + rpcgen \ + rpm-build \ + screen \ + strace \ + sudo \ + vim && \ + dnf autoremove -y && \ + dnf clean all -y && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-debian-10.Dockerfile b/ci/libvirt-debian-10.Dockerfile new file mode 100644 index 0000000..0237461 --- /dev/null +++ b/ci/libvirt-debian-10.Dockerfile @@ -0,0 +1,56 @@ +FROM debian:10 + +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get dist-upgrade -y && \ + apt-get install --no-install-recommends -y \ + autoconf \ + automake \ + autopoint \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + git \ + libc6-dev \ + libtool \ + libtool-bin \ + libvirt-dev \ + locales \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconf \ + python3 \ + python3-dev \ + python3-lxml \ + python3-nose \ + python3-setuptools \ + python3-wheel \ + screen \ + strace \ + sudo \ + vim && \ + apt-get autoremove -y && \ + apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-debian-9.Dockerfile b/ci/libvirt-debian-9.Dockerfile new file mode 100644 index 0000000..4b25f55 --- /dev/null +++ b/ci/libvirt-debian-9.Dockerfile @@ -0,0 +1,59 @@ +FROM debian:9 + +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get dist-upgrade -y && \ + apt-get install --no-install-recommends -y \ + autoconf \ + automake \ + autopoint \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + git \ + libc6-dev \ + libtool \ + libtool-bin \ + libvirt-dev \ + locales \ + lsof \ + make \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconf \ + python3 \ + python3-dev \ + python3-lxml \ + python3-nose \ + python3-pip \ + python3-setuptools \ + python3-wheel \ + screen \ + strace \ + sudo \ + vim && \ + apt-get autoremove -y && \ + apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +RUN pip3 install \ + meson==0.49.0 + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-debian-sid.Dockerfile b/ci/libvirt-debian-sid.Dockerfile new file mode 100644 index 0000000..e75c478 --- /dev/null +++ b/ci/libvirt-debian-sid.Dockerfile @@ -0,0 +1,56 @@ +FROM debian:sid + +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get dist-upgrade -y && \ + apt-get install --no-install-recommends -y \ + autoconf \ + automake \ + autopoint \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + git \ + libc6-dev \ + libtool \ + libtool-bin \ + libvirt-dev \ + locales \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconf \ + python3 \ + python3-dev \ + python3-lxml \ + python3-nose \ + python3-setuptools \ + python3-wheel \ + screen \ + strace \ + sudo \ + vim && \ + apt-get autoremove -y && \ + apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-fedora-31.Dockerfile b/ci/libvirt-fedora-31.Dockerfile new file mode 100644 index 0000000..c1c7b01 --- /dev/null +++ b/ci/libvirt-fedora-31.Dockerfile @@ -0,0 +1,53 @@ +FROM fedora:31 + +RUN dnf update -y && \ + dnf install -y \ + autoconf \ + automake \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + cppi \ + gcc \ + gdb \ + gettext \ + gettext-devel \ + git \ + glibc-devel \ + glibc-langpack-en \ + libtool \ + libvirt-devel \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconfig \ + python3 \ + python3-devel \ + python3-lxml \ + python3-nose \ + python3-setuptools \ + python3-wheel \ + rpm-build \ + screen \ + strace \ + sudo \ + vim && \ + dnf autoremove -y && \ + dnf clean all -y && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-fedora-32.Dockerfile b/ci/libvirt-fedora-32.Dockerfile new file mode 100644 index 0000000..8de2ad7 --- /dev/null +++ b/ci/libvirt-fedora-32.Dockerfile @@ -0,0 +1,53 @@ +FROM fedora:32 + +RUN dnf update -y && \ + dnf install -y \ + autoconf \ + automake \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + cppi \ + gcc \ + gdb \ + gettext \ + gettext-devel \ + git \ + glibc-devel \ + glibc-langpack-en \ + libtool \ + libvirt-devel \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconfig \ + python3 \ + python3-devel \ + python3-lxml \ + python3-nose \ + python3-setuptools \ + python3-wheel \ + rpm-build \ + screen \ + strace \ + sudo \ + vim && \ + dnf autoremove -y && \ + dnf clean all -y && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-fedora-rawhide.Dockerfile b/ci/libvirt-fedora-rawhide.Dockerfile new file mode 100644 index 0000000..0d5ca91 --- /dev/null +++ b/ci/libvirt-fedora-rawhide.Dockerfile @@ -0,0 +1,54 @@ +FROM fedora:rawhide + +RUN dnf update -y --nogpgcheck fedora-gpg-keys && \ + dnf update -y && \ + dnf install -y \ + autoconf \ + automake \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + cppi \ + gcc \ + gdb \ + gettext \ + gettext-devel \ + git \ + glibc-devel \ + glibc-langpack-en \ + libtool \ + libvirt-devel \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconfig \ + python3 \ + python3-devel \ + python3-lxml \ + python3-nose \ + python3-setuptools \ + python3-wheel \ + rpm-build \ + screen \ + strace \ + sudo \ + vim && \ + dnf autoremove -y && \ + dnf clean all -y && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-opensuse-151.Dockerfile b/ci/libvirt-opensuse-151.Dockerfile new file mode 100644 index 0000000..6a5bce5 --- /dev/null +++ b/ci/libvirt-opensuse-151.Dockerfile @@ -0,0 +1,55 @@ +FROM opensuse/leap:15.1 + +RUN zypper update -y && \ + zypper install -y \ + autoconf \ + automake \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + cppi \ + gcc \ + gdb \ + gettext \ + gettext-devel \ + git \ + glibc-devel \ + glibc-locale \ + libtool \ + libvirt-devel \ + lsof \ + make \ + net-tools \ + ninja \ + patch \ + perl \ + pkgconfig \ + python3 \ + python3-devel \ + python3-lxml \ + python3-nose \ + python3-pip \ + python3-setuptools \ + python3-wheel \ + rpm-build \ + screen \ + strace \ + sudo \ + vim && \ + zypper clean --all && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +RUN pip3 install \ + meson==0.49.0 + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-ubuntu-1804.Dockerfile b/ci/libvirt-ubuntu-1804.Dockerfile new file mode 100644 index 0000000..7186fac --- /dev/null +++ b/ci/libvirt-ubuntu-1804.Dockerfile @@ -0,0 +1,59 @@ +FROM ubuntu:18.04 + +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get dist-upgrade -y && \ + apt-get install --no-install-recommends -y \ + autoconf \ + automake \ + autopoint \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + git \ + libc6-dev \ + libtool \ + libtool-bin \ + libvirt-dev \ + locales \ + lsof \ + make \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconf \ + python3 \ + python3-dev \ + python3-lxml \ + python3-nose \ + python3-pip \ + python3-setuptools \ + python3-wheel \ + screen \ + strace \ + sudo \ + vim && \ + apt-get autoremove -y && \ + apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +RUN pip3 install \ + meson==0.49.0 + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-ubuntu-2004.Dockerfile b/ci/libvirt-ubuntu-2004.Dockerfile new file mode 100644 index 0000000..de959f5 --- /dev/null +++ b/ci/libvirt-ubuntu-2004.Dockerfile @@ -0,0 +1,56 @@ +FROM ubuntu:20.04 + +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get dist-upgrade -y && \ + apt-get install --no-install-recommends -y \ + autoconf \ + automake \ + autopoint \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + git \ + libc6-dev \ + libtool \ + libtool-bin \ + libvirt-dev \ + locales \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconf \ + python3 \ + python3-dev \ + python3-lxml \ + python3-nose \ + python3-setuptools \ + python3-wheel \ + screen \ + strace \ + sudo \ + vim && \ + apt-get autoremove -y && \ + apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/refresh b/ci/refresh new file mode 100755 index 0000000..011afdc --- /dev/null +++ b/ci/refresh @@ -0,0 +1,27 @@ +#!/bin/sh + +if test -z "$1" +then + echo "syntax: $0 PATH-TO-LCITOOL" + exit 1 +fi + +LCITOOL=$1 + +if ! test -x "$LCITOOL" +then + echo "$LCITOOL is not executable" + exit 1 +fi + +HOSTS=$($LCITOOL hosts | grep -v freebsd) + +for host in $HOSTS +do + if test "$host" = "libvirt-centos-8" + then + $LCITOOL dockerfile $host libvirt+minimal,libvirt+dist,libvirt-python > $host.Dockerfile + else + $LCITOOL dockerfile $host libvirt+dist,libvirt-python > $host.Dockerfile + fi +done -- 2.25.4

On Fri, 2020-05-01 at 10:20 +0100, Daniel P. Berrangé wrote:
+.build_git_job_template: &build_git_job_definition + image: $CI_REGISTRY_IMAGE/ci-$NAME:latest + stage: builds
This should be git_build_job_template...
+.build_dist_job_template: &build_dist_job_definition + image: $CI_REGISTRY_IMAGE/ci-$NAME:latest + stage: builds
... and this should be dist_build_job_template now, with all uses below updated accordingly of course.
+++ b/ci/README.rst @@ -0,0 +1,13 @@ +CI job assets +============= + +This directory contains assets used in the automated CI jobs, most +notably the Dockerfiles used to build container images in which the +CI jobs then run. + +The ``refresh`` script is used to re-create the Dockerfiles using the +``lcitool`` that is provided by repo https://gitlab.com/libvirt/libvirt-ci
This should read either "the lcitool command", or "the lcitool script", or something else along those lines. With these last remaining nits addressed, Reviewed-by: Andrea Bolognani <abologna@redhat.com> I still don't feel comfortable ACKing the first patch, but you can look at it this way: once you've pushed the last two patches, the first one will suddenly fall squarely under the CI fix rule ;) -- Andrea Bolognani / Red Hat / Virtualization

On Fri, May 01, 2020 at 10:20:03AM +0100, Daniel P. Berrangé wrote:
The python build needs to validate two axis
- A variety of libvirt versions - A variety of python versions
We get coverage for both these axis by running a build against the distro provided libvirt packages. All that is then missing is a build against the latest libvirt git master, which only needs to be run on a single distro, for which CentOS 8 is picked as a stable long life base.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 185 +++++++++++++++++++++++++++ ci/README.rst | 13 ++ ci/libvirt-centos-7.Dockerfile | 86 +++++++++++++ ci/libvirt-centos-8.Dockerfile | 64 +++++++++ ci/libvirt-debian-10.Dockerfile | 56 ++++++++ ci/libvirt-debian-9.Dockerfile | 59 +++++++++ ci/libvirt-debian-sid.Dockerfile | 56 ++++++++ ci/libvirt-fedora-31.Dockerfile | 53 ++++++++ ci/libvirt-fedora-32.Dockerfile | 53 ++++++++ ci/libvirt-fedora-rawhide.Dockerfile | 54 ++++++++ ci/libvirt-opensuse-151.Dockerfile | 55 ++++++++ ci/libvirt-ubuntu-1804.Dockerfile | 59 +++++++++ ci/libvirt-ubuntu-2004.Dockerfile | 56 ++++++++ ci/refresh | 27 ++++ 14 files changed, 876 insertions(+) create mode 100644 .gitlab-ci.yml create mode 100644 ci/README.rst create mode 100644 ci/libvirt-centos-7.Dockerfile create mode 100644 ci/libvirt-centos-8.Dockerfile create mode 100644 ci/libvirt-debian-10.Dockerfile create mode 100644 ci/libvirt-debian-9.Dockerfile create mode 100644 ci/libvirt-debian-sid.Dockerfile create mode 100644 ci/libvirt-fedora-31.Dockerfile create mode 100644 ci/libvirt-fedora-32.Dockerfile create mode 100644 ci/libvirt-fedora-rawhide.Dockerfile create mode 100644 ci/libvirt-opensuse-151.Dockerfile create mode 100644 ci/libvirt-ubuntu-1804.Dockerfile create mode 100644 ci/libvirt-ubuntu-2004.Dockerfile create mode 100755 ci/refresh
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 0000000..bc95d41 --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,185 @@ + +stages: + - prebuild + - containers + - builds + - docs + +.container_job_template: &container_job_definition + image: docker:stable + stage: containers + services: + - docker:dind + before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest" + - export COMMON_TAG="$CI_REGISTRY/libvirt/libvirt-perl/ci-$NAME:latest"
This doesn't look right, shouldn't we use libvirt-python here? Pavel

On Fri, May 01, 2020 at 05:56:23PM +0200, Pavel Hrdina wrote:
On Fri, May 01, 2020 at 10:20:03AM +0100, Daniel P. Berrangé wrote:
The python build needs to validate two axis
- A variety of libvirt versions - A variety of python versions
We get coverage for both these axis by running a build against the distro provided libvirt packages. All that is then missing is a build against the latest libvirt git master, which only needs to be run on a single distro, for which CentOS 8 is picked as a stable long life base.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 185 +++++++++++++++++++++++++++ ci/README.rst | 13 ++ ci/libvirt-centos-7.Dockerfile | 86 +++++++++++++ ci/libvirt-centos-8.Dockerfile | 64 +++++++++ ci/libvirt-debian-10.Dockerfile | 56 ++++++++ ci/libvirt-debian-9.Dockerfile | 59 +++++++++ ci/libvirt-debian-sid.Dockerfile | 56 ++++++++ ci/libvirt-fedora-31.Dockerfile | 53 ++++++++ ci/libvirt-fedora-32.Dockerfile | 53 ++++++++ ci/libvirt-fedora-rawhide.Dockerfile | 54 ++++++++ ci/libvirt-opensuse-151.Dockerfile | 55 ++++++++ ci/libvirt-ubuntu-1804.Dockerfile | 59 +++++++++ ci/libvirt-ubuntu-2004.Dockerfile | 56 ++++++++ ci/refresh | 27 ++++ 14 files changed, 876 insertions(+) create mode 100644 .gitlab-ci.yml create mode 100644 ci/README.rst create mode 100644 ci/libvirt-centos-7.Dockerfile create mode 100644 ci/libvirt-centos-8.Dockerfile create mode 100644 ci/libvirt-debian-10.Dockerfile create mode 100644 ci/libvirt-debian-9.Dockerfile create mode 100644 ci/libvirt-debian-sid.Dockerfile create mode 100644 ci/libvirt-fedora-31.Dockerfile create mode 100644 ci/libvirt-fedora-32.Dockerfile create mode 100644 ci/libvirt-fedora-rawhide.Dockerfile create mode 100644 ci/libvirt-opensuse-151.Dockerfile create mode 100644 ci/libvirt-ubuntu-1804.Dockerfile create mode 100644 ci/libvirt-ubuntu-2004.Dockerfile create mode 100755 ci/refresh
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 0000000..bc95d41 --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,185 @@ + +stages: + - prebuild + - containers + - builds + - docs + +.container_job_template: &container_job_definition + image: docker:stable + stage: containers + services: + - docker:dind + before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest" + - export COMMON_TAG="$CI_REGISTRY/libvirt/libvirt-perl/ci-$NAME:latest"
This doesn't look right, shouldn't we use libvirt-python here?
Err, yes, of course. This would have meant the cache never matched as the dockerfile is different. I'll push the obvious fix. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

Now that we're standardizing on GitLab CI for both official gating CI and developer CI, there's no compelling reason to continue to support Travis CI. Reviewed-by: Andrea Bolognani <abologna@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .travis.yml | 55 ----------------------------------------------------- 1 file changed, 55 deletions(-) delete mode 100644 .travis.yml diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index fe24957..0000000 --- a/.travis.yml +++ /dev/null @@ -1,55 +0,0 @@ -language: python -os: linux -dist: xenial - -python: - - 3.6 - -env: - - LIBVIRT=1.2.0 EXT=gz - - LIBVIRT=2.0.0 EXT=xz - - LIBVIRT=3.6.0 EXT=xz - - LIBVIRT=4.5.0 EXT=xz - - LIBVIRT=5.0.0 EXT=xz - -install: - - sudo apt-get -qqy build-dep libvirt libxml2-dev - - sudo apt-get -qqy install curl - - pip install -r requirements-test.txt - - curl -O -s https://libvirt.org/sources/libvirt-${LIBVIRT}.tar.${EXT} - - tar -xf libvirt-${LIBVIRT}.tar.${EXT} - - pushd libvirt-${LIBVIRT} - - | - ./configure --prefix=`pwd`/../libvirt-vroot \ - --without-libvirtd \ - --without-esx \ - --without-vbox \ - --without-libxl \ - --without-xen \ - --without-qemu \ - --without-lxc \ - --without-hyperv \ - --without-macvtap \ - --disable-werror - - make - - make install - - popd - -script: - - LD_LIBRARY_PATH=`pwd`/libvirt-vroot/lib PKG_CONFIG_PATH=`pwd`/libvirt-vroot/lib/pkgconfig python setup.py build sdist test - -notifications: - irc: - # The channel name "irc.oftc.net#virt" is encrypted against libvirt/libvirt-python - # to prevent IRC notifications from github forks. This was created using: - # $ travis encrypt -r "libvirt/libvirt-python" "irc.oftc.net#virt" - channels: - - secure: "K4JrbRpz4CHtZ1vjthVwseT8K6INJgjtZethP4DN1jOpm1uC5esbe1Q1qJOfB92JbMcdM6DNjrVg5eyTJj35aD9UoGpTUcPMsYrhlTPHZtfAuLv/at2eB2XRmETlhiXHgI6LizX6gTiwGW5ZHYwGChzumWxu141d/L9harNh9R6z8XH9uJpkNdOAIsJcwS56XGZ74CKsrqF5dK6ZYPIyP+i7gPO67gEWo0oD6TiJKR908fw03ZiXarIFmLRlk4MbHywLRF0byfD0gg2Ht/tDX73+59QXjLKo/GvQecwoU8UuuFRJlyhUfvm1JYYydnS+O7fPJvI0FWlYFY7i76aeVqkARHRpHknFueT6kZADOmiyMLuvdr+gWVuyIdX33vVJtDm4T1OtNMG/wy9EUZUU1vEu+gHhaRkf/O0GkMj0Hac4I14BGyd/Wdhto6zWojFiMEG/HRHey6l15MBQu49QyW/YMyWi/LeBWXuCUgwQ/ij5EPgsn36OxCafV9zMz0oXZskwX6rJGQRZsdgdwYvt2hP3muLaJbwVyT0bGlOJDJieOa/LVKOXPcQm26aGfyMuLgm0//E9v++6W1IDKh6+BNsfTKAwTxlAvJyz6Bns3XuUJUxUz2+uQVSS6S3EwEZUJ+yHDd2F4sX5OP1L7TWIOWFbI4vQK90ZZ7/jgiYQbwo=" - on_success: change - on_failure: always - email: - # The list name 'libvirt-ci@redhat.com" is encrypted against libvirt/libvirt-python - # to prevent IRC notifications from github forks. This was created using: - # $ travis encrypt -r "libvirt/libvirt-python" "libvirt-ci@redhat.com" - recipients: - - secure: "l6TTLcEcXdDEldHE2NgSIdt6a0k99ug3hp2W4IlnqJWJfIk/87nysJtLNrA0va20pPApCa3iJfMq4PUmBGiIIimTN0/KgC7tONDraogXhCbgfZp9Ejy/57TXxygSp4oum2kDw/c5uLnfrFV/xcn1fk6hvH6CD3bVcJPOQ/mc5FSKLqN5UzwqNnMpMTtG9qxCwfXJ/Bdm9fbURfezC7djcYRwRfPUe3TSD0L76G2HnQnSy4RqR3KFSjQHFPnSGM5IbsokbOaFKCyp/pHOt7QomQaY7YAPX/K9O+eP+hkkp6DGADkkumHctcgnMoyxpahf7pNKw9S8JYabH2NwREIq8whbp9Mo+R4rYO2ozroLWHaboYs/pBLrs606ivTwOmWGRCpJdCmmKTiZNyo6MRrwiOM6x+2YHUTMOa2kVheRNzaaxMFzHPW2kZ20bujPhfViJsRYj9flo5GJXJLyjluGZK5RjrguNJeIh8VJNBiSHW37uj7drmNBsqMad+65mf/4xtGITBqhz5Spx5R9UMZbuiJvcm8GasJMMdQ+bCfuWYjF2nZvSvFEr54Ii1YrDp6FKQ8YG1aD1/D8Z0/b3pLd/8Pn+M9yIWyO/Sto5TbSUjxBTmTStuDmtYE5uu1miYebvgJH5MovWPBegYgrfI417kPJgCG3q/R0YcZFMKFfQyo=" -- 2.25.4
participants (3)
-
Andrea Bolognani
-
Daniel P. Berrangé
-
Pavel Hrdina