[libvirt-python PATCH v2 0/3] gitlab: introduce CI jobs for validating python build

This introduces GitLab CI to the python module. Traditional Jenkins has validated the python build across all distros, using libvirt git. This tested one axis - a variety of python versions - but failed to test the other interesting axis - a variety of libvirt versions. This new CI setup fixes that mistake validating both axis. Daniel P. Berrangé (3): test: workaround missing VIR_TYPED_PARAM enums in API definition gitlab: introduce CI jobs testing git master & distro libvirt travis: delete redundant configuration .gitlab-ci.yml | 171 +++++++++++++++++++++++++++ .travis.yml | 55 --------- ci/README.rst | 12 ++ ci/libvirt-centos-7.Dockerfile | 86 ++++++++++++++ ci/libvirt-centos-8.Dockerfile | 64 ++++++++++ ci/libvirt-debian-10.Dockerfile | 56 +++++++++ ci/libvirt-debian-9.Dockerfile | 59 +++++++++ ci/libvirt-debian-sid.Dockerfile | 56 +++++++++ ci/libvirt-fedora-30.Dockerfile | 53 +++++++++ ci/libvirt-fedora-31.Dockerfile | 53 +++++++++ ci/libvirt-fedora-rawhide.Dockerfile | 54 +++++++++ ci/libvirt-opensuse-151.Dockerfile | 55 +++++++++ ci/libvirt-ubuntu-1604.Dockerfile | 59 +++++++++ ci/libvirt-ubuntu-1804.Dockerfile | 59 +++++++++ ci/refresh | 27 +++++ sanitytest.py | 6 +- 16 files changed, 869 insertions(+), 56 deletions(-) create mode 100644 .gitlab-ci.yml delete mode 100644 .travis.yml create mode 100644 ci/README.rst create mode 100644 ci/libvirt-centos-7.Dockerfile create mode 100644 ci/libvirt-centos-8.Dockerfile create mode 100644 ci/libvirt-debian-10.Dockerfile create mode 100644 ci/libvirt-debian-9.Dockerfile create mode 100644 ci/libvirt-debian-sid.Dockerfile create mode 100644 ci/libvirt-fedora-30.Dockerfile create mode 100644 ci/libvirt-fedora-31.Dockerfile create mode 100644 ci/libvirt-fedora-rawhide.Dockerfile create mode 100644 ci/libvirt-opensuse-151.Dockerfile create mode 100644 ci/libvirt-ubuntu-1604.Dockerfile create mode 100644 ci/libvirt-ubuntu-1804.Dockerfile create mode 100755 ci/refresh -- 2.25.4

On Ubuntu 18.04 with libvirt 4.0.0 libvirt-python build fails running test /usr/bin/python3 sanitytest.py build/lib.linux-x86_64-3.6 /usr/share/libvirt/api/libvirt-api.xml Cannot get a value of enum VIR_TYPED_PARAM_BOOLEAN (originally VIR_DOMAIN_BLKIO_PARAM_BOOLEAN) Cannot get a value of enum VIR_TYPED_PARAM_DOUBLE (originally VIR_DOMAIN_BLKIO_PARAM_DOUBLE) Cannot get a value of enum VIR_TYPED_PARAM_INT (originally VIR_DOMAIN_BLKIO_PARAM_INT) ...snip... The code generated for the binding is still correct and so we can just whitelist this error scenario. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- sanitytest.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/sanitytest.py b/sanitytest.py index 1bedd55..f187c15 100644 --- a/sanitytest.py +++ b/sanitytest.py @@ -73,7 +73,11 @@ for n in second_pass: val = int(v[val]) break - if type(val) != int: + # Version 4.0.0 was broken as missing VIR_TYPED_PARAM enums + # constants. This is harmless from POV of validating API + # coverage so ignore this error. + if (type(val) != int and + not val.startswith("VIR_TYPED_PARAM_")): fail = True print("Cannot get a value of enum %s (originally %s)" % (val, name)) enumvals[typ][name] = val -- 2.25.4

The python build needs to validate two axis - A variety of libvirt versions - A variety of python versions We get coverage for both these axis by running a build against the distro provided libvirt packages. All that is then missing is a build against the latest libvirt git master, which only needs to be run on a single distro, for which CentOS 8 is picked as a stable long life base. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 171 +++++++++++++++++++++++++++ ci/README.rst | 12 ++ ci/libvirt-centos-7.Dockerfile | 86 ++++++++++++++ ci/libvirt-centos-8.Dockerfile | 64 ++++++++++ ci/libvirt-debian-10.Dockerfile | 56 +++++++++ ci/libvirt-debian-9.Dockerfile | 59 +++++++++ ci/libvirt-debian-sid.Dockerfile | 56 +++++++++ ci/libvirt-fedora-30.Dockerfile | 53 +++++++++ ci/libvirt-fedora-31.Dockerfile | 53 +++++++++ ci/libvirt-fedora-rawhide.Dockerfile | 54 +++++++++ ci/libvirt-opensuse-151.Dockerfile | 55 +++++++++ ci/libvirt-ubuntu-1604.Dockerfile | 59 +++++++++ ci/libvirt-ubuntu-1804.Dockerfile | 59 +++++++++ ci/refresh | 27 +++++ 14 files changed, 864 insertions(+) create mode 100644 .gitlab-ci.yml create mode 100644 ci/README.rst create mode 100644 ci/libvirt-centos-7.Dockerfile create mode 100644 ci/libvirt-centos-8.Dockerfile create mode 100644 ci/libvirt-debian-10.Dockerfile create mode 100644 ci/libvirt-debian-9.Dockerfile create mode 100644 ci/libvirt-debian-sid.Dockerfile create mode 100644 ci/libvirt-fedora-30.Dockerfile create mode 100644 ci/libvirt-fedora-31.Dockerfile create mode 100644 ci/libvirt-fedora-rawhide.Dockerfile create mode 100644 ci/libvirt-opensuse-151.Dockerfile create mode 100644 ci/libvirt-ubuntu-1604.Dockerfile create mode 100644 ci/libvirt-ubuntu-1804.Dockerfile create mode 100755 ci/refresh diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 0000000..a54c499 --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,171 @@ + +stages: + - prebuild + - containers + - build + - docs + +.container_job_template: &container_job_definition + image: docker:stable + stage: containers + services: + - docker:dind + before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest" + - export COMMON_TAG="$CI_REGISTRY/libvirt/libvirt-perl/ci-$NAME:latest" + - docker info + - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" + script: + - docker pull "$TAG" || docker pull "$COMMON_TAG" || true + - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "ci/libvirt-$NAME.Dockerfile" ci + - docker push "$TAG" + after_script: + - docker logout + +.build_git_job_template: &build_git_job_definition + stage: build + before_script: + - export MAKEFLAGS="-j$(getconf _NPROCESSORS_ONLN)" + - export SCRATCH_DIR="/tmp/scratch" + - export VROOT="$SCRATCH_DIR/vroot" + - export LD_LIBRARY_PATH="$VROOT/lib" + - export PATH="$PATH:$VROOT/bin" + - export PKG_CONFIG_PATH="$VROOT/lib/pkgconfig" + script: + - pushd "$PWD" + - mkdir -p "$SCRATCH_DIR" + - cd "$SCRATCH_DIR" + - git clone --depth 1 https://gitlab.com/libvirt/libvirt.git src + - mkdir build + - cd build + - ../src/autogen.sh --prefix="$VROOT" --without-libvirtd + - make install + - popd + - $PYTHON setup.py build + - $PYTHON setup.py test + +.build_dist_job_template: &build_dist_job_definition + stage: build + before_script: + - export MAKEFLAGS="-j$(getconf _NPROCESSORS_ONLN)" + script: + - $PYTHON setup.py build + - $PYTHON setup.py test + +# Check that all commits are signed-off for the DCO. +# Skip on "libvirt" namespace, since we only need to run +# this test on developer's personal forks from which +# merge requests are submitted +check-dco: + stage: prebuild + image: registry.gitlab.com/libvirt/libvirt-ci/check-dco:master + script: + - /check-dco + except: + variables: + - $CI_PROJECT_NAMESPACE == 'libvirt' + +centos-7-container: + <<: *container_job_definition + variables: + NAME: centos-7 + +centos-8-container: + <<: *container_job_definition + variables: + NAME: centos-8 + +debian-9-container: + <<: *container_job_definition + variables: + NAME: debian-9 + +debian-10-container: + <<: *container_job_definition + variables: + NAME: debian-10 + +debian-sid-container: + <<: *container_job_definition + variables: + NAME: debian-sid + +fedora-30-container: + <<: *container_job_definition + variables: + NAME: fedora-30 + +fedora-31-container: + <<: *container_job_definition + variables: + NAME: fedora-31 + +fedora-rawhide-container: + <<: *container_job_definition + variables: + NAME: fedora-rawhide + +opensuse-151-container: + <<: *container_job_definition + variables: + NAME: opensuse-151 + +ubuntu-1604-container: + <<: *container_job_definition + variables: + NAME: ubuntu-1604 + +ubuntu-1804-container: + <<: *container_job_definition + variables: + NAME: ubuntu-1804 + + +centos-8-build-git: + <<: *build_git_job_definition + image: $CI_REGISTRY_IMAGE/ci-centos-8:latest + + +centos-7-build-dist: + <<: *build_dist_job_definition + image: $CI_REGISTRY_IMAGE/ci-centos-7:latest + +centos-8-build-dist: + <<: *build_dist_job_definition + image: $CI_REGISTRY_IMAGE/ci-centos-8:latest + +debian-9-build-dist: + <<: *build_dist_job_definition + image: $CI_REGISTRY_IMAGE/ci-debian-9:latest + +debian-10-build-dist: + <<: *build_dist_job_definition + image: $CI_REGISTRY_IMAGE/ci-debian-10:latest + +debian-sid-build-dist: + <<: *build_dist_job_definition + image: $CI_REGISTRY_IMAGE/ci-debian-sid:latest + +fedora-30-build-dist: + <<: *build_dist_job_definition + image: $CI_REGISTRY_IMAGE/ci-fedora-30:latest + +fedora-31-build-dist: + <<: *build_dist_job_definition + image: $CI_REGISTRY_IMAGE/ci-fedora-31:latest + +fedora-rawhide-build-dist: + <<: *build_dist_job_definition + image: $CI_REGISTRY_IMAGE/ci-fedora-rawhide:latest + +opensuse-151-build-dist: + <<: *build_dist_job_definition + image: $CI_REGISTRY_IMAGE/ci-opensuse-151:latest + +ubuntu-1604-build-dist: + <<: *build_dist_job_definition + image: $CI_REGISTRY_IMAGE/ci-ubuntu-1604:latest + +ubuntu-1804-build-dist: + <<: *build_dist_job_definition + image: $CI_REGISTRY_IMAGE/ci-ubuntu-1804:latest diff --git a/ci/README.rst b/ci/README.rst new file mode 100644 index 0000000..fef7a95 --- /dev/null +++ b/ci/README.rst @@ -0,0 +1,12 @@ +CI job assets +============= + +This directory contains assets used in the automated CI jobs, most +notably the Dockerfiles used to build container images in which the +CI jobs then run. + +The `refresh` script is used to re-create the Dockerfiles using the +`lcitool` that is provided by repo `https://gitlab.com/libvirt/libvirt-ci` +The containers are built during the CI process and cached in the GitLab +container registry of the project doing the build. The cached containers +can be deleted at any time and will be correctly rebuilt. diff --git a/ci/libvirt-centos-7.Dockerfile b/ci/libvirt-centos-7.Dockerfile new file mode 100644 index 0000000..74eba4e --- /dev/null +++ b/ci/libvirt-centos-7.Dockerfile @@ -0,0 +1,86 @@ +FROM centos:7 + +RUN echo -e '[openvz]\n\ +name=OpenVZ addons\n\ +baseurl=https://download.openvz.org/virtuozzo/releases/openvz-7.0.11-235/x86_64/os/\n\ +enabled=1\n\ +gpgcheck=1\n\ +skip_if_unavailable=0\n\ +metadata_expire=6h\n\ +priority=90\n\ +includepkgs=libprl*' > /etc/yum.repos.d/openvz.repo && \ + echo -e '-----BEGIN PGP PUBLIC KEY BLOCK-----\n\ +Version: GnuPG v2.0.22 (GNU/Linux)\n\ +\n\ +mI0EVl80nQEEAKrEeyeTCwrzS9kYedZ/sAc/GUqlb81C7pA9SaR3fyck5mVw1Ogk\n\ +YdmNBPM2kY7QDxR9F0EpSpnxSCAXZXugsQ8KzZ0DRLVeBDQyGs9IGK5hI0zzxIil\n\ +BzfvIexLiQQhLy7YlIi8Jt/uUqKkW0pIMNMGcduY97VATtczpncpkmSzABEBAAG0\n\ +SFZpcnR1b3p6byBUZWFtIChHUEcga2V5IHNpZ25hdHVyZSBmb3IgcGFja2FnZXMp\n\ +IDxzZWN1cml0eUB2aXJ0dW96em8uY29tPoi5BBMBAgAjBQJWXzSdAhsDBwsJCAcD\n\ +AgEGFQgCCQoLBBYCAwECHgECF4AACgkQygt9GUTNrSruIgP/er70Eyo73A1gfrjv\n\ +oPUkyo4rslVRZu3qqCwoMFtJc/Z/UxWgEka1buorlcGLa6eO/EZ49c0n+KGa4Kvt\n\ +EUboIq0yEu5i0FyAj92ifm+hNhoAbGfm0cZ4/fD0oGr3l8OsQo4+iHX4xAPwFe7Y\n\ +zABuB8I1ZDZ4OIp5tDfTTuF2LT24jQRWXzSdAQQAog2Aqb+Ptl68O7cQhWLjVGkj\n\ +yyigZrdeReLx3HloKJPBeQ/kA6uvMJc/IYS3uppMWXv9v+QenS6uhP1TUJ2k9FvM\n\ +t94MQZfALN7Vpf8AF+UeWu4Ru+y4BNzcFhrPhIFNFChOR2QqW6FkgE57D9I177NC\n\ +oJMyrlNe8wcGa178An8AEQEAAYifBBgBAgAJBQJWXzSdAhsMAAoJEMoLfRlEza0q\n\ +bKwD/3+OFVIEXnIv5XgdGRNX5fHggsUN1bb8gva7HANRlKdd4LD8foDM3F/yv/3V\n\ +igG14D5EjKz56SaBDNgiI4++hOzb2M8jhAsR86jxkXFrrP1U3ZNRKg6av9DPFAPS\n\ +WEiJKtQrZDJloqtyi/mmRa1VsV7RYR0VPJjhK/R8EQ7Ysshy\n\ +=fRMg\n\ +-----END PGP PUBLIC KEY BLOCK-----' > /etc/pki/rpm-gpg/RPM-GPG-KEY-OpenVZ && \ + rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-OpenVZ && \ + yum install -y epel-release && \ + yum update -y && \ + yum install -y \ + autoconf \ + automake \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + gettext-devel \ + git \ + glibc-common \ + glibc-devel \ + libtool \ + libvirt-devel \ + lsof \ + make \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconfig \ + python3 \ + python3-pip \ + python3-setuptools \ + python3-wheel \ + python36-devel \ + python36-lxml \ + python36-nose \ + rpm-build \ + screen \ + strace \ + sudo \ + vim && \ + yum autoremove -y && \ + yum clean all -y && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +RUN pip3 install \ + meson==0.49.0 + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja-build" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-centos-8.Dockerfile b/ci/libvirt-centos-8.Dockerfile new file mode 100644 index 0000000..3d0eb68 --- /dev/null +++ b/ci/libvirt-centos-8.Dockerfile @@ -0,0 +1,64 @@ +FROM centos:8 + +RUN dnf install 'dnf-command(config-manager)' -y && \ + dnf config-manager --set-enabled PowerTools -y && \ + dnf install -y epel-release && \ + dnf update -y && \ + dnf install -y \ + autoconf \ + automake \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + gettext-devel \ + git \ + glib2-devel \ + glibc-devel \ + glibc-langpack-en \ + gnutls-devel \ + libnl3-devel \ + libtirpc-devel \ + libtool \ + libvirt-devel \ + libxml2 \ + libxml2-devel \ + libxslt \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconfig \ + python3 \ + python3-devel \ + python3-docutils \ + python3-lxml \ + python3-nose \ + python3-setuptools \ + python3-wheel \ + rpcgen \ + rpm-build \ + screen \ + strace \ + sudo \ + vim && \ + dnf autoremove -y && \ + dnf clean all -y && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-debian-10.Dockerfile b/ci/libvirt-debian-10.Dockerfile new file mode 100644 index 0000000..0237461 --- /dev/null +++ b/ci/libvirt-debian-10.Dockerfile @@ -0,0 +1,56 @@ +FROM debian:10 + +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get dist-upgrade -y && \ + apt-get install --no-install-recommends -y \ + autoconf \ + automake \ + autopoint \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + git \ + libc6-dev \ + libtool \ + libtool-bin \ + libvirt-dev \ + locales \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconf \ + python3 \ + python3-dev \ + python3-lxml \ + python3-nose \ + python3-setuptools \ + python3-wheel \ + screen \ + strace \ + sudo \ + vim && \ + apt-get autoremove -y && \ + apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-debian-9.Dockerfile b/ci/libvirt-debian-9.Dockerfile new file mode 100644 index 0000000..4b25f55 --- /dev/null +++ b/ci/libvirt-debian-9.Dockerfile @@ -0,0 +1,59 @@ +FROM debian:9 + +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get dist-upgrade -y && \ + apt-get install --no-install-recommends -y \ + autoconf \ + automake \ + autopoint \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + git \ + libc6-dev \ + libtool \ + libtool-bin \ + libvirt-dev \ + locales \ + lsof \ + make \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconf \ + python3 \ + python3-dev \ + python3-lxml \ + python3-nose \ + python3-pip \ + python3-setuptools \ + python3-wheel \ + screen \ + strace \ + sudo \ + vim && \ + apt-get autoremove -y && \ + apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +RUN pip3 install \ + meson==0.49.0 + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-debian-sid.Dockerfile b/ci/libvirt-debian-sid.Dockerfile new file mode 100644 index 0000000..e75c478 --- /dev/null +++ b/ci/libvirt-debian-sid.Dockerfile @@ -0,0 +1,56 @@ +FROM debian:sid + +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get dist-upgrade -y && \ + apt-get install --no-install-recommends -y \ + autoconf \ + automake \ + autopoint \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + git \ + libc6-dev \ + libtool \ + libtool-bin \ + libvirt-dev \ + locales \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconf \ + python3 \ + python3-dev \ + python3-lxml \ + python3-nose \ + python3-setuptools \ + python3-wheel \ + screen \ + strace \ + sudo \ + vim && \ + apt-get autoremove -y && \ + apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-fedora-30.Dockerfile b/ci/libvirt-fedora-30.Dockerfile new file mode 100644 index 0000000..3368b99 --- /dev/null +++ b/ci/libvirt-fedora-30.Dockerfile @@ -0,0 +1,53 @@ +FROM fedora:30 + +RUN dnf update -y && \ + dnf install -y \ + autoconf \ + automake \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + cppi \ + gcc \ + gdb \ + gettext \ + gettext-devel \ + git \ + glibc-devel \ + glibc-langpack-en \ + libtool \ + libvirt-devel \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconfig \ + python3 \ + python3-devel \ + python3-lxml \ + python3-nose \ + python3-setuptools \ + python3-wheel \ + rpm-build \ + screen \ + strace \ + sudo \ + vim && \ + dnf autoremove -y && \ + dnf clean all -y && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-fedora-31.Dockerfile b/ci/libvirt-fedora-31.Dockerfile new file mode 100644 index 0000000..c1c7b01 --- /dev/null +++ b/ci/libvirt-fedora-31.Dockerfile @@ -0,0 +1,53 @@ +FROM fedora:31 + +RUN dnf update -y && \ + dnf install -y \ + autoconf \ + automake \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + cppi \ + gcc \ + gdb \ + gettext \ + gettext-devel \ + git \ + glibc-devel \ + glibc-langpack-en \ + libtool \ + libvirt-devel \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconfig \ + python3 \ + python3-devel \ + python3-lxml \ + python3-nose \ + python3-setuptools \ + python3-wheel \ + rpm-build \ + screen \ + strace \ + sudo \ + vim && \ + dnf autoremove -y && \ + dnf clean all -y && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-fedora-rawhide.Dockerfile b/ci/libvirt-fedora-rawhide.Dockerfile new file mode 100644 index 0000000..0d5ca91 --- /dev/null +++ b/ci/libvirt-fedora-rawhide.Dockerfile @@ -0,0 +1,54 @@ +FROM fedora:rawhide + +RUN dnf update -y --nogpgcheck fedora-gpg-keys && \ + dnf update -y && \ + dnf install -y \ + autoconf \ + automake \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + cppi \ + gcc \ + gdb \ + gettext \ + gettext-devel \ + git \ + glibc-devel \ + glibc-langpack-en \ + libtool \ + libvirt-devel \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconfig \ + python3 \ + python3-devel \ + python3-lxml \ + python3-nose \ + python3-setuptools \ + python3-wheel \ + rpm-build \ + screen \ + strace \ + sudo \ + vim && \ + dnf autoremove -y && \ + dnf clean all -y && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-opensuse-151.Dockerfile b/ci/libvirt-opensuse-151.Dockerfile new file mode 100644 index 0000000..6a5bce5 --- /dev/null +++ b/ci/libvirt-opensuse-151.Dockerfile @@ -0,0 +1,55 @@ +FROM opensuse/leap:15.1 + +RUN zypper update -y && \ + zypper install -y \ + autoconf \ + automake \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + cppi \ + gcc \ + gdb \ + gettext \ + gettext-devel \ + git \ + glibc-devel \ + glibc-locale \ + libtool \ + libvirt-devel \ + lsof \ + make \ + net-tools \ + ninja \ + patch \ + perl \ + pkgconfig \ + python3 \ + python3-devel \ + python3-lxml \ + python3-nose \ + python3-pip \ + python3-setuptools \ + python3-wheel \ + rpm-build \ + screen \ + strace \ + sudo \ + vim && \ + zypper clean --all && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +RUN pip3 install \ + meson==0.49.0 + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-ubuntu-1604.Dockerfile b/ci/libvirt-ubuntu-1604.Dockerfile new file mode 100644 index 0000000..cce6507 --- /dev/null +++ b/ci/libvirt-ubuntu-1604.Dockerfile @@ -0,0 +1,59 @@ +FROM ubuntu:16.04 + +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get dist-upgrade -y && \ + apt-get install --no-install-recommends -y \ + autoconf \ + automake \ + autopoint \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + git \ + libc6-dev \ + libtool \ + libtool-bin \ + libvirt-dev \ + locales \ + lsof \ + make \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconf \ + python3 \ + python3-dev \ + python3-lxml \ + python3-nose \ + python3-pip \ + python3-setuptools \ + python3-wheel \ + screen \ + strace \ + sudo \ + vim && \ + apt-get autoremove -y && \ + apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +RUN pip3 install \ + meson==0.49.0 + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/libvirt-ubuntu-1804.Dockerfile b/ci/libvirt-ubuntu-1804.Dockerfile new file mode 100644 index 0000000..7186fac --- /dev/null +++ b/ci/libvirt-ubuntu-1804.Dockerfile @@ -0,0 +1,59 @@ +FROM ubuntu:18.04 + +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get dist-upgrade -y && \ + apt-get install --no-install-recommends -y \ + autoconf \ + automake \ + autopoint \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + git \ + libc6-dev \ + libtool \ + libtool-bin \ + libvirt-dev \ + locales \ + lsof \ + make \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconf \ + python3 \ + python3-dev \ + python3-lxml \ + python3-nose \ + python3-pip \ + python3-setuptools \ + python3-wheel \ + screen \ + strace \ + sudo \ + vim && \ + apt-get autoremove -y && \ + apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +RUN pip3 install \ + meson==0.49.0 + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/refresh b/ci/refresh new file mode 100755 index 0000000..011afdc --- /dev/null +++ b/ci/refresh @@ -0,0 +1,27 @@ +#!/bin/sh + +if test -z "$1" +then + echo "syntax: $0 PATH-TO-LCITOOL" + exit 1 +fi + +LCITOOL=$1 + +if ! test -x "$LCITOOL" +then + echo "$LCITOOL is not executable" + exit 1 +fi + +HOSTS=$($LCITOOL hosts | grep -v freebsd) + +for host in $HOSTS +do + if test "$host" = "libvirt-centos-8" + then + $LCITOOL dockerfile $host libvirt+minimal,libvirt+dist,libvirt-python > $host.Dockerfile + else + $LCITOOL dockerfile $host libvirt+dist,libvirt-python > $host.Dockerfile + fi +done -- 2.25.4

On Thu, Apr 30, 2020 at 03:14:15PM +0100, Daniel P. Berrangé wrote:
The python build needs to validate two axis
- A variety of libvirt versions - A variety of python versions
We get coverage for both these axis by running a build against the distro provided libvirt packages. All that is then missing is a build against the latest libvirt git master, which only needs to be run on a single distro, for which CentOS 8 is picked as a stable long life base.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 171 +++++++++++++++++++++++++++ ci/README.rst | 12 ++ ci/libvirt-centos-7.Dockerfile | 86 ++++++++++++++ ci/libvirt-centos-8.Dockerfile | 64 ++++++++++ ci/libvirt-debian-10.Dockerfile | 56 +++++++++ ci/libvirt-debian-9.Dockerfile | 59 +++++++++ ci/libvirt-debian-sid.Dockerfile | 56 +++++++++ ci/libvirt-fedora-30.Dockerfile | 53 +++++++++ ci/libvirt-fedora-31.Dockerfile | 53 +++++++++ ci/libvirt-fedora-rawhide.Dockerfile | 54 +++++++++ ci/libvirt-opensuse-151.Dockerfile | 55 +++++++++ ci/libvirt-ubuntu-1604.Dockerfile | 59 +++++++++ ci/libvirt-ubuntu-1804.Dockerfile | 59 +++++++++ ci/refresh | 27 +++++
This will need the obvious changes to replace F30 -> F31 and Ubuntu 1604->2004. I won't repost the series just for that. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Thu, 2020-04-30 at 15:14 +0100, Daniel P. Berrangé wrote:
+++ b/.gitlab-ci.yml @@ -0,0 +1,171 @@ + +stages: + - prebuild + - containers + - build
Should this stage be called "builds", just like the previous one is called "containers"? I'm also thinking whether it would be interesting to have a two separate "dist_builds" and "git_builds" stages, just like we have a stage for native builds and another one for cross builds in libvirt. All of this is just me thinking out loud. I'm happy enough with the current solution :)
+.build_git_job_template: &build_git_job_definition + stage: build + before_script: + - export MAKEFLAGS="-j$(getconf _NPROCESSORS_ONLN)" + - export SCRATCH_DIR="/tmp/scratch" + - export VROOT="$SCRATCH_DIR/vroot" + - export LD_LIBRARY_PATH="$VROOT/lib" + - export PATH="$PATH:$VROOT/bin"
$VROOT/bin should go first, to make sure we use the binaries we just built and not the ones from the distro packags.
+ script: + - pushd "$PWD" + - mkdir -p "$SCRATCH_DIR" + - cd "$SCRATCH_DIR" + - git clone --depth 1 https://gitlab.com/libvirt/libvirt.git src + - mkdir build + - cd build + - ../src/autogen.sh --prefix="$VROOT" --without-libvirtd
I'd just let 'git clone' do its own thing here, especially since for projects like libvirt-dbus that have multiple dependencies when building from source we'll need to differentiate them.
+fedora-30-container: + <<: *container_job_definition + variables: + NAME: fedora-30
As you already mentioned, this should be replaced with fedora-32...
+ubuntu-1604-container: + <<: *container_job_definition + variables: + NAME: ubuntu-1604
... and this with ubuntu-2004.
+centos-8-build-git: + <<: *build_git_job_definition + image: $CI_REGISTRY_IMAGE/ci-centos-8:latest
We could invert the two parts of the suffix, in the interest of having the most relevant piece of information first, and call this centos-8-git-build...
+centos-7-build-dist: + <<: *build_dist_job_definition + image: $CI_REGISTRY_IMAGE/ci-centos-7:latest
and all the other ones *-dist-build. Regarless, you can use the same trick used for container jobs also for build jobs: that is, tweak the templates above so that they look like .build_dist_job_template: &build_dist_job_definition stage: build image: $CI_REGISTRY_IMAGE/ci-$NAME:latest ... .build_git_job_template: &build_git_job_definition stage: build image: $CI_REGISTRY_IMAGE/ci-$NAME:latest ... and then redefine the build jobs as centos-8-build-git: <<: *build_git_job_definition variables: NAME: centos-8 centos-7-build-dist: <<: *build_dist_job_definition variables: NAME: centos-7 and so on. Test pipeline to prove this actually works: https://gitlab.com/abologna/libvirt-python/pipelines/141549490
+++ b/ci/README.rst @@ -0,0 +1,12 @@ +CI job assets +============= + +This directory contains assets used in the automated CI jobs, most +notably the Dockerfiles used to build container images in which the +CI jobs then run. + +The `refresh` script is used to re-create the Dockerfiles
Monospace in reStructuredText looks ``like this``, not `like this`.
using the `lcitool` that is provided by
s/that/command that/ or s/that/script that/
Remove the backticks and this will be a clickable link. Then leave a blank line between this and the next paragraph. -- Andrea Bolognani / Red Hat / Virtualization

On Thu, Apr 30, 2020 at 07:19:07PM +0200, Andrea Bolognani wrote:
On Thu, 2020-04-30 at 15:14 +0100, Daniel P. Berrangé wrote:
+++ b/.gitlab-ci.yml @@ -0,0 +1,171 @@ + +stages: + - prebuild + - containers + - build
Should this stage be called "builds", just like the previous one is called "containers"?
I'm also thinking whether it would be interesting to have a two separate "dist_builds" and "git_builds" stages, just like we have a stage for native builds and another one for cross builds in libvirt.
I don't want to separate the stages, as then you have to do extra work to get them to run in parallel. I've done all the other changes. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

Now that we're standardizing on GitLab CI for both official gating CI and developer CI, there's no compelling reason to continue to support Travis CI. Reviewed-by: Andrea Bolognani <abologna@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .travis.yml | 55 ----------------------------------------------------- 1 file changed, 55 deletions(-) delete mode 100644 .travis.yml diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index fe24957..0000000 --- a/.travis.yml +++ /dev/null @@ -1,55 +0,0 @@ -language: python -os: linux -dist: xenial - -python: - - 3.6 - -env: - - LIBVIRT=1.2.0 EXT=gz - - LIBVIRT=2.0.0 EXT=xz - - LIBVIRT=3.6.0 EXT=xz - - LIBVIRT=4.5.0 EXT=xz - - LIBVIRT=5.0.0 EXT=xz - -install: - - sudo apt-get -qqy build-dep libvirt libxml2-dev - - sudo apt-get -qqy install curl - - pip install -r requirements-test.txt - - curl -O -s https://libvirt.org/sources/libvirt-${LIBVIRT}.tar.${EXT} - - tar -xf libvirt-${LIBVIRT}.tar.${EXT} - - pushd libvirt-${LIBVIRT} - - | - ./configure --prefix=`pwd`/../libvirt-vroot \ - --without-libvirtd \ - --without-esx \ - --without-vbox \ - --without-libxl \ - --without-xen \ - --without-qemu \ - --without-lxc \ - --without-hyperv \ - --without-macvtap \ - --disable-werror - - make - - make install - - popd - -script: - - LD_LIBRARY_PATH=`pwd`/libvirt-vroot/lib PKG_CONFIG_PATH=`pwd`/libvirt-vroot/lib/pkgconfig python setup.py build sdist test - -notifications: - irc: - # The channel name "irc.oftc.net#virt" is encrypted against libvirt/libvirt-python - # to prevent IRC notifications from github forks. This was created using: - # $ travis encrypt -r "libvirt/libvirt-python" "irc.oftc.net#virt" - channels: - - secure: "K4JrbRpz4CHtZ1vjthVwseT8K6INJgjtZethP4DN1jOpm1uC5esbe1Q1qJOfB92JbMcdM6DNjrVg5eyTJj35aD9UoGpTUcPMsYrhlTPHZtfAuLv/at2eB2XRmETlhiXHgI6LizX6gTiwGW5ZHYwGChzumWxu141d/L9harNh9R6z8XH9uJpkNdOAIsJcwS56XGZ74CKsrqF5dK6ZYPIyP+i7gPO67gEWo0oD6TiJKR908fw03ZiXarIFmLRlk4MbHywLRF0byfD0gg2Ht/tDX73+59QXjLKo/GvQecwoU8UuuFRJlyhUfvm1JYYydnS+O7fPJvI0FWlYFY7i76aeVqkARHRpHknFueT6kZADOmiyMLuvdr+gWVuyIdX33vVJtDm4T1OtNMG/wy9EUZUU1vEu+gHhaRkf/O0GkMj0Hac4I14BGyd/Wdhto6zWojFiMEG/HRHey6l15MBQu49QyW/YMyWi/LeBWXuCUgwQ/ij5EPgsn36OxCafV9zMz0oXZskwX6rJGQRZsdgdwYvt2hP3muLaJbwVyT0bGlOJDJieOa/LVKOXPcQm26aGfyMuLgm0//E9v++6W1IDKh6+BNsfTKAwTxlAvJyz6Bns3XuUJUxUz2+uQVSS6S3EwEZUJ+yHDd2F4sX5OP1L7TWIOWFbI4vQK90ZZ7/jgiYQbwo=" - on_success: change - on_failure: always - email: - # The list name 'libvirt-ci@redhat.com" is encrypted against libvirt/libvirt-python - # to prevent IRC notifications from github forks. This was created using: - # $ travis encrypt -r "libvirt/libvirt-python" "libvirt-ci@redhat.com" - recipients: - - secure: "l6TTLcEcXdDEldHE2NgSIdt6a0k99ug3hp2W4IlnqJWJfIk/87nysJtLNrA0va20pPApCa3iJfMq4PUmBGiIIimTN0/KgC7tONDraogXhCbgfZp9Ejy/57TXxygSp4oum2kDw/c5uLnfrFV/xcn1fk6hvH6CD3bVcJPOQ/mc5FSKLqN5UzwqNnMpMTtG9qxCwfXJ/Bdm9fbURfezC7djcYRwRfPUe3TSD0L76G2HnQnSy4RqR3KFSjQHFPnSGM5IbsokbOaFKCyp/pHOt7QomQaY7YAPX/K9O+eP+hkkp6DGADkkumHctcgnMoyxpahf7pNKw9S8JYabH2NwREIq8whbp9Mo+R4rYO2ozroLWHaboYs/pBLrs606ivTwOmWGRCpJdCmmKTiZNyo6MRrwiOM6x+2YHUTMOa2kVheRNzaaxMFzHPW2kZ20bujPhfViJsRYj9flo5GJXJLyjluGZK5RjrguNJeIh8VJNBiSHW37uj7drmNBsqMad+65mf/4xtGITBqhz5Spx5R9UMZbuiJvcm8GasJMMdQ+bCfuWYjF2nZvSvFEr54Ii1YrDp6FKQ8YG1aD1/D8Z0/b3pLd/8Pn+M9yIWyO/Sto5TbSUjxBTmTStuDmtYE5uu1miYebvgJH5MovWPBegYgrfI417kPJgCG3q/R0YcZFMKFfQyo=" -- 2.25.4
participants (2)
-
Andrea Bolognani
-
Daniel P. Berrangé