[libvirt PATCH v2 0/9] gitlab: expand the CI job coverage (RESEND)
There are main goals with this series - Introduce a minimal job building the website and publishing an artifact which can be deployed onto libvirt.org - Introduce a minimal job building the libvirt.pot for import into Weblate (only runs on git master branch) - Expanding CI jobs to get coverage closer to Travis/Jenkins - Reducing cross-build jobs to just interesting variants, since the full set hasn't shown value in detecting failures The Linux native job coverage is now a superset of that achieved by Travis/Jenkins. For post-merge testing the full set of jobs are run on git master (measured approx 50 minutes total duration) For pre-merge testing the Linux job count is reduced for quicker turnaround time for developers (measured ~35 minutes total duration) Changed in v2: - Add more native test jobs to run on git master - Restrict git clone depth - User variable for "make" command name - Add test job to build the master pot file - Remove extra configure args for website job - Re-ordered patches to reduce repeated changes Daniel P. Berrangé (9): gitlab: use CI for building website contents gitlab: reduce number of cross-build CI jobs gitlab: group jobs into stages gitlab: rename the cross build jobs gitlab: add mingw cross build CI jobs gitlab: add x86_64 native CI jobs gitlab: add variable for make command name gitlab: restrict git history to 100 commits gitlab: add job for building latest potfile .gitlab-ci.yml | 178 ++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 148 insertions(+), 30 deletions(-) -- 2.24.1
Run the bare minimum build that is possible to create the docs. Ideally the '--without-remote' arg would be passed, but there are several bugs preventing a build from succeeding without the remote driver built. The generated website is published as an artifact and thus is browsable on build completion and can be downloaded as a zip file. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index ea49c6178b..180861f082 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -44,3 +44,24 @@ debian-sid-cross-i686: debian-sid-cross-mipsel: <<: *job_definition image: quay.io/libvirt/buildenv-libvirt-debian-sid-cross-mipsel:latest + +# This artifact published by this job is downloaded by libvirt.org to +# be deployed to the web root: +# https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?job=webs... +website: + script: + - mkdir build + - cd build + - ../autogen.sh --prefix=$(pwd)/../vroot || (cat config.log && exit 1) + - make -j $(getconf _NPROCESSORS_ONLN) -C docs + - make -j $(getconf _NPROCESSORS_ONLN) -C docs install + - cd .. + - mv vroot/share/doc/libvirt/html/ website + image: quay.io/libvirt/buildenv-libvirt-fedora-31:latest + artifacts: + expose_as: 'Website' + name: 'website' + when: on_success + expire_in: 30 days + paths: + - website -- 2.24.1
On Tue, 2020-03-24 at 16:23 +0000, Daniel P. Berrangé wrote:
Run the bare minimum build that is possible to create the docs. Ideally the '--without-remote' arg would be passed, but there are several bugs preventing a build from succeeding without the remote driver built.
This comment is no longer necessary, is it? [...]
+# This artifact published by this job is downloaded by libvirt.org to +# be deployed to the web root: +# https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?job=webs... +website: + script: + - mkdir build + - cd build + - ../autogen.sh --prefix=$(pwd)/../vroot || (cat config.log && exit 1) + - make -j $(getconf _NPROCESSORS_ONLN) -C docs + - make -j $(getconf _NPROCESSORS_ONLN) -C docs install
Can you please make patch 7/9 the first in the series, so that you don't need to go back and change this to $MAKE later?
+ - cd .. + - mv vroot/share/doc/libvirt/html/ website + image: quay.io/libvirt/buildenv-libvirt-fedora-31:latest
Since the build job is running on Fedora 31 (at the moment), would it make sense to call it website-fedora-31? The name of the artifact would remain the same, so eg. the cron job we're going to have on libvirt.org will not need to be modified every time we bump this.
+ artifacts: + expose_as: 'Website' + name: 'website'
I'm not sure the quotes are necessary. Either way, we can remove them later if they turn out not to be. -- Andrea Bolognani / Red Hat / Virtualization
On Tue, Mar 24, 2020 at 06:26:15PM +0100, Andrea Bolognani wrote:
On Tue, 2020-03-24 at 16:23 +0000, Daniel P. Berrangé wrote:
Run the bare minimum build that is possible to create the docs. Ideally the '--without-remote' arg would be passed, but there are several bugs preventing a build from succeeding without the remote driver built.
This comment is no longer necessary, is it?
[...]
+# This artifact published by this job is downloaded by libvirt.org to +# be deployed to the web root: +# https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?job=webs... +website: + script: + - mkdir build + - cd build + - ../autogen.sh --prefix=$(pwd)/../vroot || (cat config.log && exit 1) + - make -j $(getconf _NPROCESSORS_ONLN) -C docs + - make -j $(getconf _NPROCESSORS_ONLN) -C docs install
Can you please make patch 7/9 the first in the series, so that you don't need to go back and change this to $MAKE later?
I concur.
+ - cd .. + - mv vroot/share/doc/libvirt/html/ website + image: quay.io/libvirt/buildenv-libvirt-fedora-31:latest
Since the build job is running on Fedora 31 (at the moment), would it make sense to call it website-fedora-31? The name of the artifact would remain the same, so eg. the cron job we're going to have on libvirt.org will not need to be modified every time we bump this.
The website job is generic, meaning we don't care which distro it is being run on, so I'd rather not force the naming conventionby including the distro name. Reviewed-by: Erik Skultety <eskultet@redhat.com>
We're going to add more build jobs to CI, and users have limited time granted on the shared CI runners. The number of cross-build jobs currently present is not sustainable, so cut it down to two interesting jobs to cover big endian and 32-bit platform variants. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 37 ++++++------------------------------- 1 file changed, 6 insertions(+), 31 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 180861f082..1249ec6df5 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -5,29 +5,12 @@ - ../autogen.sh $CONFIGURE_OPTS || (cat config.log && exit 1) - make -j $(getconf _NPROCESSORS_ONLN) -# We could run every arch on every versions, but it is a little -# overkill. Instead we split jobs evenly across 9, 10 and sid -# to achieve reasonable cross-coverage. - -debian-9-cross-armv6l: - <<: *job_definition - image: quay.io/libvirt/buildenv-libvirt-debian-9-cross-armv6l:latest - -debian-9-cross-mips64el: - <<: *job_definition - image: quay.io/libvirt/buildenv-libvirt-debian-9-cross-mips64el:latest - -debian-9-cross-mips: - <<: *job_definition - image: quay.io/libvirt/buildenv-libvirt-debian-9-cross-mips:latest - -debian-10-cross-aarch64: - <<: *job_definition - image: quay.io/libvirt/buildenv-libvirt-debian-10-cross-aarch64:latest - -debian-10-cross-ppc64le: - <<: *job_definition - image: quay.io/libvirt/buildenv-libvirt-debian-10-cross-ppc64le:latest +# There are many possible cross-arch jobs we could do, but to preserve +# limited CI resource time allocated to users, we cut it down to two +# interesting variants. The default jobs are x86_64, which means 64-bit +# and little endian. We thus pick armv7l as an interesting 32-bit +# platform, and s390x as an interesting big endian platform. We split +# between Debian 10 and sid to help detect problems on the horizon. debian-10-cross-s390x: <<: *job_definition @@ -37,14 +20,6 @@ debian-sid-cross-armv7l: <<: *job_definition image: quay.io/libvirt/buildenv-libvirt-debian-sid-cross-armv7l:latest -debian-sid-cross-i686: - <<: *job_definition - image: quay.io/libvirt/buildenv-libvirt-debian-sid-cross-i686:latest - -debian-sid-cross-mipsel: - <<: *job_definition - image: quay.io/libvirt/buildenv-libvirt-debian-sid-cross-mipsel:latest - # This artifact published by this job is downloaded by libvirt.org to # be deployed to the web root: # https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?job=webs... -- 2.24.1
On Tue, 2020-03-24 at 16:23 +0000, Daniel P. Berrangé wrote:
We're going to add more build jobs to CI, and users have limited time granted on the shared CI runners. The number of cross-build jobs currently present is not sustainable, so cut it down to two interesting jobs to cover big endian and 32-bit platform variants.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 37 ++++++------------------------------- 1 file changed, 6 insertions(+), 31 deletions(-)
Reviewed-by: Andrea Bolognani <abologna@redhat.com> -- Andrea Bolognani / Red Hat / Virtualization
On Tue, Mar 24, 2020 at 04:23:58PM +0000, Daniel P. Berrangé wrote:
We're going to add more build jobs to CI, and users have limited time granted on the shared CI runners. The number of cross-build jobs currently present is not sustainable, so cut it down to two interesting jobs to cover big endian and 32-bit platform variants.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Reviewed-by: Erik Skultety <eskultet@redhat.com>
Within a stage all jobs run in parallel. Stages are ordered so later stages are only executed if previous stages succeeded. By using separate stages for the cross builds, we can avoid wasting CI resources if the relatively simple website build fails. Later we can avoid running cross builds, if the native build fails too. Reviewed-by: Erik Skultety <eskultet@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 1249ec6df5..8d6b5a0787 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -1,4 +1,9 @@ -.job_template: &job_definition +stages: + - prebuild + - cross_build + +.cross_build_job_template: &cross_build_job_definition + stage: cross_build script: - mkdir build - cd build @@ -13,17 +18,18 @@ # between Debian 10 and sid to help detect problems on the horizon. debian-10-cross-s390x: - <<: *job_definition + <<: *cross_build_job_definition image: quay.io/libvirt/buildenv-libvirt-debian-10-cross-s390x:latest debian-sid-cross-armv7l: - <<: *job_definition + <<: *cross_build_job_definition image: quay.io/libvirt/buildenv-libvirt-debian-sid-cross-armv7l:latest # This artifact published by this job is downloaded by libvirt.org to # be deployed to the web root: # https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?job=webs... website: + stage: prebuild script: - mkdir build - cd build -- 2.24.1
On Tue, 2020-03-24 at 16:23 +0000, Daniel P. Berrangé wrote:
Within a stage all jobs run in parallel. Stages are ordered so later stages are only executed if previous stages succeeded. By using separate stages for the cross builds, we can avoid wasting CI resources if the relatively simple website build fails. Later we can avoid running cross builds, if the native build fails too.
Reviewed-by: Erik Skultety <eskultet@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
Reviewed-by: Andrea Bolognani <abologna@redhat.com> -- Andrea Bolognani / Red Hat / Virtualization
The pipeline UI will truncate the names of jobs after about 15 characters. As a result with the cross-builds, we truncate the most important part of the job name. Putting the most important part first is robust against truncation, and we can drop the redundant "-cross" stub. Reviewed-by: Erik Skultety <skultety.erik@gmail.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 8d6b5a0787..563b126662 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -17,11 +17,11 @@ stages: # platform, and s390x as an interesting big endian platform. We split # between Debian 10 and sid to help detect problems on the horizon. -debian-10-cross-s390x: +s390x-debian-10: <<: *cross_build_job_definition image: quay.io/libvirt/buildenv-libvirt-debian-10-cross-s390x:latest -debian-sid-cross-armv7l: +armv7l-debian-sid: <<: *cross_build_job_definition image: quay.io/libvirt/buildenv-libvirt-debian-sid-cross-armv7l:latest -- 2.24.1
On Tue, 2020-03-24 at 16:24 +0000, Daniel P. Berrangé wrote:
The pipeline UI will truncate the names of jobs after about 15 characters. As a result with the cross-builds, we truncate the most important part of the job name. Putting the most important part first is robust against truncation, and we can drop the redundant "-cross" stub.
Reviewed-by: Erik Skultety <skultety.erik@gmail.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Reviewed-by: Andrea Bolognani <abologna@redhat.com> -- Andrea Bolognani / Red Hat / Virtualization
This pulls in the mingw cross build jobs using Fedora 30 as a base, matching what is done on Jenkins and Travis. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 563b126662..329374a34f 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -25,6 +25,14 @@ armv7l-debian-sid: <<: *cross_build_job_definition image: quay.io/libvirt/buildenv-libvirt-debian-sid-cross-armv7l:latest +mingw32-fedora-30: + <<: *cross_build_job_definition + image: quay.io/libvirt/buildenv-libvirt-fedora-30-cross-mingw32:latest + +mingw64-fedora-30: + <<: *cross_build_job_definition + image: quay.io/libvirt/buildenv-libvirt-fedora-30-cross-mingw64:latest + # This artifact published by this job is downloaded by libvirt.org to # be deployed to the web root: # https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?job=webs... -- 2.24.1
On Tue, 2020-03-24 at 16:24 +0000, Daniel P. Berrangé wrote:
This pulls in the mingw cross build jobs using Fedora 30 as a base, matching what is done on Jenkins and Travis.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 8 ++++++++ 1 file changed, 8 insertions(+)
Reviewed-by: Andrea Bolognani <abologna@redhat.com> -- Andrea Bolognani / Red Hat / Virtualization
On Tue, Mar 24, 2020 at 04:24:01PM +0000, Daniel P. Berrangé wrote:
This pulls in the mingw cross build jobs using Fedora 30 as a base, matching what is done on Jenkins and Travis.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Reviewed-by: Erik Skultety <eskultet@redhat.com>
This patch adds x86_64 native CI jobs for all distros that we currently build container images for. This is a superset of the Linux jobs run on current Jenkins and Travis platforms. The remaining missing platforms are FreeBSD and macOS, neither of which can use the shared runner container based infrastructure. We may add further native jobs in the future which are not x86_64 based, if we get access to suitable hardware, thus the jobs all have an arch prefix in their name, just like the cross-built jobs do. To control the total CI execution time, we split the native jobs into two distinct stages. A representative set of distros are used as the primary native build sanity test, run for everyone regardless of whether pre/post merge, and on any branch. The remaining distros are set to run after the cross builds, and only execute for master branch, and thus will only run for post-merge. When we switch to using a merge request workflow, these extra jobs can be triggered when the merge request is opened. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 329374a34f..58abcbe1f3 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -1,6 +1,34 @@ stages: - prebuild + - native_build - cross_build + - native_build_extra + + +# Common templates + +# Native jobs that run every time, pre/post merge and on any branch +.native_build_job_template: &native_build_job_definition + stage: native_build + script: + - mkdir build + - cd build + - ../autogen.sh $CONFIGURE_OPTS || (cat config.log && exit 1) + - make -j $(getconf _NPROCESSORS_ONLN) syntax-check + - make -j $(getconf _NPROCESSORS_ONLN) distcheck + +# Native jobs that will only run post merge on master branch +# Switch to running against merge requests later +.native_build_extra_job_template: &native_build_extra_job_definition + stage: native_build_extra + script: + - mkdir build + - cd build + - ../autogen.sh $CONFIGURE_OPTS || (cat config.log && exit 1) + - $MAKE -j $(getconf _NPROCESSORS_ONLN) check + only: + refs: + - master .cross_build_job_template: &cross_build_job_definition stage: cross_build @@ -10,6 +38,56 @@ stages: - ../autogen.sh $CONFIGURE_OPTS || (cat config.log && exit 1) - make -j $(getconf _NPROCESSORS_ONLN) + +# Native architecture build + test jobs + +x64-debian-9: + <<: *native_build_extra_job_definition + image: quay.io/libvirt/buildenv-libvirt-debian-9:latest + +x64-debian-10: + <<: *native_build_job_definition + image: quay.io/libvirt/buildenv-libvirt-debian-10:latest + +x64-debian-sid: + <<: *native_build_extra_job_definition + image: quay.io/libvirt/buildenv-libvirt-debian-sid:latest + +x64-centos-7: + <<: *native_build_job_definition + image: quay.io/libvirt/buildenv-libvirt-centos-7:latest + +x64-centos-8: + <<: *native_build_extra_job_definition + image: quay.io/libvirt/buildenv-libvirt-centos-8:latest + +x64-fedora-30: + <<: *native_build_job_definition + image: quay.io/libvirt/buildenv-libvirt-fedora-30:latest + +x64-fedora-31: + <<: *native_build_extra_job_definition + image: quay.io/libvirt/buildenv-libvirt-fedora-31:latest + +x64-fedora-rawhide: + <<: *native_build_job_definition + image: quay.io/libvirt/buildenv-libvirt-fedora-rawhide:latest + +x64-opensuse-151: + <<: *native_build_job_definition + image: quay.io/libvirt/buildenv-libvirt-opensuse-151:latest + +x64-ubuntu-1604: + <<: *native_build_job_definition + image: quay.io/libvirt/buildenv-libvirt-ubuntu-1604:latest + +x64-ubuntu-1804: + <<: *native_build_extra_job_definition + image: quay.io/libvirt/buildenv-libvirt-ubuntu-1804:latest + + +# Cross compiled build jobs + # There are many possible cross-arch jobs we could do, but to preserve # limited CI resource time allocated to users, we cut it down to two # interesting variants. The default jobs are x86_64, which means 64-bit -- 2.24.1
On Tue, 2020-03-24 at 16:24 +0000, Daniel P. Berrangé wrote:
To control the total CI execution time, we split the native jobs into two distinct stages. A representative set of distros are used as the primary native build sanity test, run for everyone regardless of whether pre/post merge, and on any branch. The remaining distros are set to run after the cross builds, and only execute for master branch, and thus will only run for post-merge. When we switch to using a merge request workflow, these extra jobs can be triggered when the merge request is opened.
I don't get the rationale behind the split. Right now we're not using merge requests, but we're limiting the number of jobs for the merge request case; at the same time, we say that once we switch to a MR-based workflow, we're going to run the extra jobs on each merge request as well. So what does the distinction buy us? I think a better split would be: * pick a selection of jobs that includes both native and cross build, across various distros of different vintage, such that they can all run without waiting on shared runners. This can be used by developers as a reasonably quick (~20 minutes) smoke test while working on a feature; * run every other build as a second stage in the pipeline, both for master and (later) for merge requests, to validate patches as thoroughly as possible before they get into libvirt; * run the website and potfile jobs for master only, as their purpose is not to validate the build (the regular Fedora 31 job will have done that already) but to publish the artifacts. Does that sound reasonable? -- Andrea Bolognani / Red Hat / Virtualization
On Tue, Mar 24, 2020 at 06:47:01PM +0100, Andrea Bolognani wrote:
On Tue, 2020-03-24 at 16:24 +0000, Daniel P. Berrangé wrote:
To control the total CI execution time, we split the native jobs into two distinct stages. A representative set of distros are used as the primary native build sanity test, run for everyone regardless of whether pre/post merge, and on any branch. The remaining distros are set to run after the cross builds, and only execute for master branch, and thus will only run for post-merge. When we switch to using a merge request workflow, these extra jobs can be triggered when the merge request is opened.
I don't get the rationale behind the split.
Right now we're not using merge requests, but we're limiting the number of jobs for the merge request case; at the same time, we say that once we switch to a MR-based workflow, we're going to run the extra jobs on each merge request as well. So what does the distinction buy us?
With this split today, if I push to my private fork, then the reduced set of jobs run. This gives quick turnaround for developers developing patches. When it gets reviewed & pushed to master, the full set run post merge. In the future, when we switch to merge requests, we'll change it so that the full set run when the merge request is opened, instad of post-merge. What is run for developers private branches will remain the same
I think a better split would be:
* pick a selection of jobs that includes both native and cross build, across various distros of different vintage, such that they can all run without waiting on shared runners. This can be used by developers as a reasonably quick (~20 minutes) smoke test while working on a feature;
"without waiting on shared runners" is undefined, as whether you wait & how long, is dependant on many things outside our control. As notedin the cover letter though, this minimal set of native + cross build jobs gives about a 35 min turn around based on load I see today. I think that's acceptably fast.
* run every other build as a second stage in the pipeline, both for master and (later) for merge requests, to validate patches as thoroughly as possible before they get into libvirt;
There's no point in doing both master & merge requests - only one or the other. Right now since we're not doing merge reuqests, I picked master. Later we'll switch it to be on merge requests.
* run the website and potfile jobs for master only, as their purpose is not to validate the build (the regular Fedora 31 job will have done that already) but to publish the artifacts.
Doing the website job on all branches was intentionale because by publishing the artifact, developers can now browse the website for their job during development to see the effect on their changes. eg for my most recent job I can view: https://berrange.gitlab.io/-/libvirt/-/jobs/483635204/artifacts/website/inde... Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Mar 24, 2020 at 06:21:24PM +0000, Daniel P. Berrangé wrote:
On Tue, Mar 24, 2020 at 06:47:01PM +0100, Andrea Bolognani wrote:
On Tue, 2020-03-24 at 16:24 +0000, Daniel P. Berrangé wrote:
To control the total CI execution time, we split the native jobs into two distinct stages. A representative set of distros are used as the primary native build sanity test, run for everyone regardless of whether pre/post merge, and on any branch. The remaining distros are set to run after the cross builds, and only execute for master branch, and thus will only run for post-merge. When we switch to using a merge request workflow, these extra jobs can be triggered when the merge request is opened.
I don't get the rationale behind the split.
Right now we're not using merge requests, but we're limiting the number of jobs for the merge request case; at the same time, we say that once we switch to a MR-based workflow, we're going to run the extra jobs on each merge request as well. So what does the distinction buy us?
With this split today, if I push to my private fork, then the reduced set of jobs run. This gives quick turnaround for developers developing patches.
When it gets reviewed & pushed to master, the full set run post merge.
In the future, when we switch to merge requests, we'll change it so that the full set run when the merge request is opened, instad of post-merge.
What is run for developers private branches will remain the same
I think a better split would be:
* pick a selection of jobs that includes both native and cross build, across various distros of different vintage, such that they can all run without waiting on shared runners. This can be used by developers as a reasonably quick (~20 minutes) smoke test while working on a feature;
"without waiting on shared runners" is undefined, as whether you wait & how long, is dependant on many things outside our control. As notedin the cover letter though, this minimal set of native + cross build jobs gives about a 35 min turn around based on load I see today. I think that's acceptably fast.
Related to the availability of shared runners, for merge_requests/post_merge builds only, the PSI/OpenShift infra may actually help in order to achieve more stable execution times due to the usage of private runners. Erik
On Wed, 2020-03-25 at 09:35 +0100, Erik Skultety wrote:
On Tue, Mar 24, 2020 at 06:21:24PM +0000, Daniel P. Berrangé wrote:
On Tue, Mar 24, 2020 at 06:47:01PM +0100, Andrea Bolognani wrote:
* pick a selection of jobs that includes both native and cross build, across various distros of different vintage, such that they can all run without waiting on shared runners. This can be used by developers as a reasonably quick (~20 minutes) smoke test while working on a feature;
"without waiting on shared runners" is undefined, as whether you wait & how long, is dependant on many things outside our control. As notedin the cover letter though, this minimal set of native + cross build jobs gives about a 35 min turn around based on load I see today. I think that's acceptably fast.
Related to the availability of shared runners, for merge_requests/post_merge builds only, the PSI/OpenShift infra may actually help in order to achieve more stable execution times due to the usage of private runners.
As explained by Dan, that will not help for the merge request builds, because those are executed in the context of the submitter's fork and as such don't have access to any runners we might deploy ourselves. It would help speed up the post merge phase, but once we move to a merge request workflow we'll want to test as much as possible at the time the merge request is submitted, and there won't be much that needs to happen after the changes hit master... So dedicated runners might actually not be useful outside of supporting non-Linux targets. -- Andrea Bolognani / Red Hat / Virtualization
On Wed, Mar 25, 2020 at 11:27:31AM +0100, Andrea Bolognani wrote:
On Wed, 2020-03-25 at 09:35 +0100, Erik Skultety wrote:
On Tue, Mar 24, 2020 at 06:21:24PM +0000, Daniel P. Berrangé wrote:
On Tue, Mar 24, 2020 at 06:47:01PM +0100, Andrea Bolognani wrote:
* pick a selection of jobs that includes both native and cross build, across various distros of different vintage, such that they can all run without waiting on shared runners. This can be used by developers as a reasonably quick (~20 minutes) smoke test while working on a feature;
"without waiting on shared runners" is undefined, as whether you wait & how long, is dependant on many things outside our control. As notedin the cover letter though, this minimal set of native + cross build jobs gives about a 35 min turn around based on load I see today. I think that's acceptably fast.
Related to the availability of shared runners, for merge_requests/post_merge builds only, the PSI/OpenShift infra may actually help in order to achieve more stable execution times due to the usage of private runners.
As explained by Dan, that will not help for the merge request builds, because those are executed in the context of the submitter's fork and as such don't have access to any runners we might deploy ourselves.
It would help speed up the post merge phase, but once we move to a merge request workflow we'll want to test as much as possible at the time the merge request is submitted, and there won't be much that needs to happen after the changes hit master... So dedicated runners might actually not be useful outside of supporting non-Linux targets.
NB, our current testing is all just build+unit testing. We want to expand to do integration testing, and that will also benefit from dedicated runners so that we can actually run stuff in a real OS, as opposed to a container where our functionality is much reduced. Also, even with merge request testing, we'll still want to test everything post-merge, as there is still a gap there. IIUC, the merge requests will test against the git master than that the merge request branch was based on. The fast-forward to current git master only happens after that. So there is a risk that merge request can pass testing but still result in breakage due to change to git master affecting the fast forward result. There might be ways to mitigate this by having a CI job itself try to do a fast-forwar to latest git master, rather than honouring the git state given to it, but I've not investigated. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, 2020-03-25 at 10:44 +0000, Daniel P. Berrangé wrote:
On Wed, Mar 25, 2020 at 11:27:31AM +0100, Andrea Bolognani wrote:
On Wed, 2020-03-25 at 09:35 +0100, Erik Skultety wrote:
Related to the availability of shared runners, for merge_requests/post_merge builds only, the PSI/OpenShift infra may actually help in order to achieve more stable execution times due to the usage of private runners.
As explained by Dan, that will not help for the merge request builds, because those are executed in the context of the submitter's fork and as such don't have access to any runners we might deploy ourselves.
It would help speed up the post merge phase, but once we move to a merge request workflow we'll want to test as much as possible at the time the merge request is submitted, and there won't be much that needs to happen after the changes hit master... So dedicated runners might actually not be useful outside of supporting non-Linux targets.
NB, our current testing is all just build+unit testing. We want to expand to do integration testing, and that will also benefit from dedicated runners so that we can actually run stuff in a real OS, as opposed to a container where our functionality is much reduced.
Good point, I was just thinking in terms of migrating what we already have to GitLab rather than building upon it. This kind of testing will still be limited by the fact that merge requests will not be able to use these dedicated runners. Compare this to something like KubeVirt's CI, where proper integrations tests are executed for each and every merge request...
Also, even with merge request testing, we'll still want to test everything post-merge, as there is still a gap there. IIUC, the merge requests will test against the git master than that the merge request branch was based on. The fast-forward to current git master only happens after that.
So there is a risk that merge request can pass testing but still result in breakage due to change to git master affecting the fast forward result.
There might be ways to mitigate this by having a CI job itself try to do a fast-forwar to latest git master, rather than honouring the git state given to it, but I've not investigated.
There has to be a way to tell GitLab to re-test the actual code that is about to hit master when a merge request is approved... It would be kind of insane otherwise, since both the base branch and the set of CI tests themselves might be weeks, if not months old by the time a merge request is approved. -- Andrea Bolognani / Red Hat / Virtualization
On Wed, Mar 25, 2020 at 12:27:33PM +0100, Andrea Bolognani wrote:
On Wed, 2020-03-25 at 10:44 +0000, Daniel P. Berrangé wrote:
On Wed, Mar 25, 2020 at 11:27:31AM +0100, Andrea Bolognani wrote:
On Wed, 2020-03-25 at 09:35 +0100, Erik Skultety wrote:
Related to the availability of shared runners, for merge_requests/post_merge builds only, the PSI/OpenShift infra may actually help in order to achieve more stable execution times due to the usage of private runners.
As explained by Dan, that will not help for the merge request builds, because those are executed in the context of the submitter's fork and as such don't have access to any runners we might deploy ourselves.
It would help speed up the post merge phase, but once we move to a merge request workflow we'll want to test as much as possible at the time the merge request is submitted, and there won't be much that needs to happen after the changes hit master... So dedicated runners might actually not be useful outside of supporting non-Linux targets.
NB, our current testing is all just build+unit testing. We want to expand to do integration testing, and that will also benefit from dedicated runners so that we can actually run stuff in a real OS, as opposed to a container where our functionality is much reduced.
Good point, I was just thinking in terms of migrating what we already have to GitLab rather than building upon it.
This kind of testing will still be limited by the fact that merge requests will not be able to use these dedicated runners. Compare this to something like KubeVirt's CI, where proper integrations tests are executed for each and every merge request...
I didn't know that merge requests are run in the context of the submitter, I thought that once submitted, it moves over to the original project. Anyhow, yes, the test coverage (more specifically the integration tests) are going to be limited in this regard, however, even if we could achieve what kubevirt is doing, we certainly don't have the capacity to cover all the existing forks out there. That's where lcitool comes in ;), before you submit a pull request, here, run the test suite via lcitool in an automated fashion and then submit the pull request - not perfect, but IMO a massive improvement. Erik
On Wed, Mar 25, 2020 at 12:27:33PM +0100, Andrea Bolognani wrote:
On Wed, 2020-03-25 at 10:44 +0000, Daniel P. Berrangé wrote:
On Wed, Mar 25, 2020 at 11:27:31AM +0100, Andrea Bolognani wrote: Also, even with merge request testing, we'll still want to test everything post-merge, as there is still a gap there. IIUC, the merge requests will test against the git master than that the merge request branch was based on. The fast-forward to current git master only happens after that.
So there is a risk that merge request can pass testing but still result in breakage due to change to git master affecting the fast forward result.
There might be ways to mitigate this by having a CI job itself try to do a fast-forwar to latest git master, rather than honouring the git state given to it, but I've not investigated.
There has to be a way to tell GitLab to re-test the actual code that is about to hit master when a merge request is approved... It would be kind of insane otherwise, since both the base branch and the set of CI tests themselves might be weeks, if not months old by the time a merge request is approved.
It isn't as big a deal as you think, because the older something is, the less likely a fast-forward merge is going to suceed without hitting conflicts. When there are conflicts, you're forced to rebase the merge, which will trigger tests again. So this is only a problem on simple patches which are fast forwardable but then semantically incorrect. And of course the reviewer will often notice incompatibilities themselves. GitLab does do the correct thing in testing the post-merge result for a merge request when using a branch from the same repo - just doens't do it for forked repos. This is related to the previous issue I mentioned with forks / runners. There's a variety of issues talking about it - the key phrases to look for are "merge train" which is GitLab terminology for this feature. https://gitlab.com/gitlab-org/gitlab/-/issues/9713 https://gitlab.com/gitlab-org/gitlab/-/issues/11935 https://gitlab.com/gitlab-org/gitlab/-/issues/11934 Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, Mar 25, 2020 at 12:27:33PM +0100, Andrea Bolognani wrote:
On Wed, 2020-03-25 at 10:44 +0000, Daniel P. Berrangé wrote:
On Wed, Mar 25, 2020 at 11:27:31AM +0100, Andrea Bolognani wrote:
On Wed, 2020-03-25 at 09:35 +0100, Erik Skultety wrote:
Related to the availability of shared runners, for merge_requests/post_merge builds only, the PSI/OpenShift infra may actually help in order to achieve more stable execution times due to the usage of private runners.
As explained by Dan, that will not help for the merge request builds, because those are executed in the context of the submitter's fork and as such don't have access to any runners we might deploy ourselves.
It would help speed up the post merge phase, but once we move to a merge request workflow we'll want to test as much as possible at the time the merge request is submitted, and there won't be much that needs to happen after the changes hit master... So dedicated runners might actually not be useful outside of supporting non-Linux targets.
NB, our current testing is all just build+unit testing. We want to expand to do integration testing, and that will also benefit from dedicated runners so that we can actually run stuff in a real OS, as opposed to a container where our functionality is much reduced.
Good point, I was just thinking in terms of migrating what we already have to GitLab rather than building upon it.
This kind of testing will still be limited by the fact that merge requests will not be able to use these dedicated runners. Compare this to something like KubeVirt's CI, where proper integrations tests are executed for each and every merge request...
BTW, when comparing to CI done in GitHub be aware that you're comparing apples & oranges. Traditionally most/all GitHub CI setups were done via external services / bots, feeding results back with the GitHub API. This approach is possible with the GitLab API too if you want to ignore the built-in CI facilities. IOW, it would be entirely possible to ignore the GitLab runners / CI system, and have a integration test harness that just watched for new Merge requests, runs tests on them, and then adds a comment and/or approval mark to the merge request. My preference is to do as much as possible using the GitLab CI setup though, since that's the more commonly expected approach. If we hit roadblocks that are important, we can take the external service/bot approach where needed. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, 2020-03-25 at 12:19 +0000, Daniel P. Berrangé wrote:
On Wed, Mar 25, 2020 at 12:27:33PM +0100, Andrea Bolognani wrote:
This kind of testing will still be limited by the fact that merge requests will not be able to use these dedicated runners. Compare this to something like KubeVirt's CI, where proper integrations tests are executed for each and every merge request...
BTW, when comparing to CI done in GitHub be aware that you're comparing apples & oranges. Traditionally most/all GitHub CI setups were done via external services / bots, feeding results back with the GitHub API. This approach is possible with the GitLab API too if you want to ignore the built-in CI facilities.
IOW, it would be entirely possible to ignore the GitLab runners / CI system, and have a integration test harness that just watched for new Merge requests, runs tests on them, and then adds a comment and/or approval mark to the merge request.
My preference is to do as much as possible using the GitLab CI setup though, since that's the more commonly expected approach. If we hit roadblocks that are important, we can take the external service/bot approach where needed.
I don't think it's really an unfair comparison. My point was that the CI setup we're building simply can't do things that other setups can. Of course everything is a trade-off to some extent, and as of today the value proposition of not having to maintain your own runners can't be ignored; once we start maintaining our own FreeBSD runners, as well as Linux runners for real functional tests, then the balance shifts again. Anyway, I'm not saying that we should stall the current efforts, which already represent an improvement over the status quo, but merely pointing out that there are some limitation that we should be aware of and that might result in us re-evaluating our choices further down the line. -- Andrea Bolognani / Red Hat / Virtualization
On Tue, 2020-03-24 at 18:21 +0000, Daniel P. Berrangé wrote:
On Tue, Mar 24, 2020 at 06:47:01PM +0100, Andrea Bolognani wrote:
I don't get the rationale behind the split.
Right now we're not using merge requests, but we're limiting the number of jobs for the merge request case; at the same time, we say that once we switch to a MR-based workflow, we're going to run the extra jobs on each merge request as well. So what does the distinction buy us?
With this split today, if I push to my private fork, then the reduced set of jobs run. This gives quick turnaround for developers developing patches.
When it gets reviewed & pushed to master, the full set run post merge.
In the future, when we switch to merge requests, we'll change it so that the full set run when the merge request is opened, instad of post-merge.
What is run for developers private branches will remain the same
Okay, I understand the rationale now, and I can see that we were arguing for very similar approaches the entire time - it's just that I could not see that. Thanks for taking the time to spell it out for me :)
* pick a selection of jobs that includes both native and cross build, across various distros of different vintage, such that they can all run without waiting on shared runners. This can be used by developers as a reasonably quick (~20 minutes) smoke test while working on a feature;
"without waiting on shared runners" is undefined, as whether you wait & how long, is dependant on many things outside our control. As notedin the cover letter though, this minimal set of native + cross build jobs gives about a 35 min turn around based on load I see today. I think that's acceptably fast.
Given the intentions, I think the current split can be improved further: having cross_build as a separate stage from native_build means that the former has to wait before the latter is done to even start, and that makes wait times longer. I suggest we instead have the following stages: artifacts: - website - potfile smoke: - armv7l-debian-sid - mingw32-fedora-30 - mingw64-fedora-30 - s390x-debian-10 - x64-centos-7 - x64-debian-10 - x64-fedora-30 - x64-fedora-rawhide - x64-opensuse-151 - x64-ubuntu-1604 full_coverage: - x64-centos-8 - x64-debian-9 - x64-debian-sid - x64-fedora-31 - x64-ubuntu-1804 Aside from the name change, this maintains the same job split, except for merging the native_build and cross_build stages together. When I tested this split, the artifacts + smoke builds only took ~20 minutes thanks to the increased parallelism; even if for load balancing reasons the smoke builds were not able to start all at the same time, I reckon this setup would still fare better than forcing the second batch to start only after the first one has completed. Another thing that I would like to point out is that, while it makes sense that we'd only have a small number of cross-builds happening in the smoke test phase, when it comes to post-merge (and later merge request) time we don't really need to limit ourselves as much, and we could easily leave most of the existing cross-build jobs enabled as part of the full_coverage stage. -- Andrea Bolognani / Red Hat / Virtualization
On Wed, Mar 25, 2020 at 11:48:35AM +0100, Andrea Bolognani wrote:
On Tue, 2020-03-24 at 18:21 +0000, Daniel P. Berrangé wrote:
On Tue, Mar 24, 2020 at 06:47:01PM +0100, Andrea Bolognani wrote:
I don't get the rationale behind the split.
Right now we're not using merge requests, but we're limiting the number of jobs for the merge request case; at the same time, we say that once we switch to a MR-based workflow, we're going to run the extra jobs on each merge request as well. So what does the distinction buy us?
With this split today, if I push to my private fork, then the reduced set of jobs run. This gives quick turnaround for developers developing patches.
When it gets reviewed & pushed to master, the full set run post merge.
In the future, when we switch to merge requests, we'll change it so that the full set run when the merge request is opened, instad of post-merge.
What is run for developers private branches will remain the same
Okay, I understand the rationale now, and I can see that we were arguing for very similar approaches the entire time - it's just that I could not see that. Thanks for taking the time to spell it out for me :)
* pick a selection of jobs that includes both native and cross build, across various distros of different vintage, such that they can all run without waiting on shared runners. This can be used by developers as a reasonably quick (~20 minutes) smoke test while working on a feature;
"without waiting on shared runners" is undefined, as whether you wait & how long, is dependant on many things outside our control. As notedin the cover letter though, this minimal set of native + cross build jobs gives about a 35 min turn around based on load I see today. I think that's acceptably fast.
Given the intentions, I think the current split can be improved further: having cross_build as a separate stage from native_build means that the former has to wait before the latter is done to even start, and that makes wait times longer.
By default jobs in a stage get given a dependency "needs: [.... jobs from prev stage...]" which serializes each stage, but that is not mandatory. We can set the stages to allow parallel execution across stages using an explicit 'needs: []'. So we can keep stage as a conceptual grouping without affecting parallelism. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, 2020-03-25 at 10:54 +0000, Daniel P. Berrangé wrote:
On Wed, Mar 25, 2020 at 11:48:35AM +0100, Andrea Bolognani wrote:
Given the intentions, I think the current split can be improved further: having cross_build as a separate stage from native_build means that the former has to wait before the latter is done to even start, and that makes wait times longer.
By default jobs in a stage get given a dependency
"needs: [.... jobs from prev stage...]"
which serializes each stage, but that is not mandatory. We can set the stages to allow parallel execution across stages using an explicit 'needs: []'. So we can keep stage as a conceptual grouping without affecting parallelism.
I didn't know about that! Anyway, even at the conceptual level splitting the jobs across smoke tests and full coverage makes more sense to me than splitting them across native builds, cross builds and... More native builds? That's basically surfacing an implementation details at the UI level, so I'm not too keen on that. If you feel strongly about having the cross builds separated from the native ones, maybe we can adopt an hybrid approach where we have stages: - artifacts - smoke_native + smoke_cross # these run in parallel using the - full_native + full_cross # trick you described above but I'm not sure this is easy, or even possible, to implement. -- Andrea Bolognani / Red Hat / Virtualization
On Wed, Mar 25, 2020 at 12:18:00PM +0100, Andrea Bolognani wrote:
On Wed, 2020-03-25 at 10:54 +0000, Daniel P. Berrangé wrote:
On Wed, Mar 25, 2020 at 11:48:35AM +0100, Andrea Bolognani wrote:
Given the intentions, I think the current split can be improved further: having cross_build as a separate stage from native_build means that the former has to wait before the latter is done to even start, and that makes wait times longer.
By default jobs in a stage get given a dependency
"needs: [.... jobs from prev stage...]"
which serializes each stage, but that is not mandatory. We can set the stages to allow parallel execution across stages using an explicit 'needs: []'. So we can keep stage as a conceptual grouping without affecting parallelism.
I didn't know about that!
Anyway, even at the conceptual level splitting the jobs across smoke tests and full coverage makes more sense to me than splitting them across native builds, cross builds and... More native builds? That's basically surfacing an implementation details at the UI level, so I'm not too keen on that.
The reason I think these should be different is that we're doing different levels of testing on them. Native builds are full tests (make distcheck), where as cross builds are build only (make all). Similarly the extra native builds are not repeating the full distcheck as that's overkill (just make check).
If you feel strongly about having the cross builds separated from the native ones, maybe we can adopt an hybrid approach where we have
stages: - artifacts - smoke_native + smoke_cross # these run in parallel using the - full_native + full_cross # trick you described above
but I'm not sure this is easy, or even possible, to implement.
-- Andrea Bolognani / Red Hat / Virtualization
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, 2020-03-25 at 11:30 +0000, Daniel P. Berrangé wrote:
On Wed, Mar 25, 2020 at 12:18:00PM +0100, Andrea Bolognani wrote:
On Wed, 2020-03-25 at 10:54 +0000, Daniel P. Berrangé wrote:
By default jobs in a stage get given a dependency
"needs: [.... jobs from prev stage...]"
which serializes each stage, but that is not mandatory. We can set the stages to allow parallel execution across stages using an explicit 'needs: []'. So we can keep stage as a conceptual grouping without affecting parallelism.
I didn't know about that!
Anyway, even at the conceptual level splitting the jobs across smoke tests and full coverage makes more sense to me than splitting them across native builds, cross builds and... More native builds? That's basically surfacing an implementation details at the UI level, so I'm not too keen on that.
The reason I think these should be different is that we're doing different levels of testing on them. Native builds are full tests (make distcheck), where as cross builds are build only (make all).
Similarly the extra native builds are not repeating the full distcheck as that's overkill (just make check).
Okay, that's somewhat reasonable, though I'm still not entirely convinced this is something that we need to put quite as front and center as you have. Can you provide an implementation where the first batch of native and cross builds are executed in parallel, to demonstrate that goal can be achieved with code that's not too complicated or brittle? That's my main concern at this point. -- Andrea Bolognani / Red Hat / Virtualization
On Wed, Mar 25, 2020 at 12:42:41PM +0100, Andrea Bolognani wrote:
On Wed, 2020-03-25 at 11:30 +0000, Daniel P. Berrangé wrote:
On Wed, Mar 25, 2020 at 12:18:00PM +0100, Andrea Bolognani wrote:
On Wed, 2020-03-25 at 10:54 +0000, Daniel P. Berrangé wrote:
By default jobs in a stage get given a dependency
"needs: [.... jobs from prev stage...]"
which serializes each stage, but that is not mandatory. We can set the stages to allow parallel execution across stages using an explicit 'needs: []'. So we can keep stage as a conceptual grouping without affecting parallelism.
I didn't know about that!
Anyway, even at the conceptual level splitting the jobs across smoke tests and full coverage makes more sense to me than splitting them across native builds, cross builds and... More native builds? That's basically surfacing an implementation details at the UI level, so I'm not too keen on that.
The reason I think these should be different is that we're doing different levels of testing on them. Native builds are full tests (make distcheck), where as cross builds are build only (make all).
Similarly the extra native builds are not repeating the full distcheck as that's overkill (just make check).
Okay, that's somewhat reasonable, though I'm still not entirely convinced this is something that we need to put quite as front and center as you have.
Can you provide an implementation where the first batch of native and cross builds are executed in parallel, to demonstrate that goal can be achieved with code that's not too complicated or brittle? That's my main concern at this point.
I tried this & it could work but then I looked a ccache and determined that it cuts the build time by about 50%, down to < 20 minutes. With this in mind, I don't think we need to worry about playing games with dependencies. The build will be faster again when we have meson because the meson build is more efficient, but also because there's no need for a heavyweight "distcheck" concept with meson. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Mar 24, 2020 at 04:24:02PM +0000, Daniel P. Berrangé wrote:
This patch adds x86_64 native CI jobs for all distros that we currently build container images for. This is a superset of the Linux jobs run on current Jenkins and Travis platforms.
The remaining missing platforms are FreeBSD and macOS, neither of which can use the shared runner container based infrastructure.
We may add further native jobs in the future which are not x86_64 based, if we get access to suitable hardware, thus the jobs all have an arch prefix in their name, just like the cross-built jobs do.
To control the total CI execution time, we split the native jobs into two distinct stages. A representative set of distros are used as the primary native build sanity test, run for everyone regardless of whether pre/post merge, and on any branch. The remaining distros are set to run after the cross builds, and only execute for master branch, and thus will only run for post-merge. When we switch to using a merge request workflow, these extra jobs can be triggered when the merge request is opened.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 329374a34f..58abcbe1f3 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -1,6 +1,34 @@ stages: - prebuild + - native_build - cross_build + - native_build_extra + + +# Common templates + +# Native jobs that run every time, pre/post merge and on any branch +.native_build_job_template: &native_build_job_definition + stage: native_build + script: + - mkdir build + - cd build + - ../autogen.sh $CONFIGURE_OPTS || (cat config.log && exit 1) + - make -j $(getconf _NPROCESSORS_ONLN) syntax-check + - make -j $(getconf _NPROCESSORS_ONLN) distcheck + +# Native jobs that will only run post merge on master branch +# Switch to running against merge requests later +.native_build_extra_job_template: &native_build_extra_job_definition + stage: native_build_extra + script: + - mkdir build + - cd build + - ../autogen.sh $CONFIGURE_OPTS || (cat config.log && exit 1) + - $MAKE -j $(getconf _NPROCESSORS_ONLN) check + only: + refs: + - master
only: - master would be enough, or is there a special need for the "refs" keyword? Erik
To facilitate future jobs that will use FreeBSD Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 58abcbe1f3..9814b6580a 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -4,6 +4,9 @@ stages: - cross_build - native_build_extra +variables: + MAKE: make + # Common templates @@ -14,8 +17,8 @@ stages: - mkdir build - cd build - ../autogen.sh $CONFIGURE_OPTS || (cat config.log && exit 1) - - make -j $(getconf _NPROCESSORS_ONLN) syntax-check - - make -j $(getconf _NPROCESSORS_ONLN) distcheck + - $MAKE -j $(getconf _NPROCESSORS_ONLN) syntax-check + - $MAKE -j $(getconf _NPROCESSORS_ONLN) distcheck # Native jobs that will only run post merge on master branch # Switch to running against merge requests later @@ -36,7 +39,7 @@ stages: - mkdir build - cd build - ../autogen.sh $CONFIGURE_OPTS || (cat config.log && exit 1) - - make -j $(getconf _NPROCESSORS_ONLN) + - $MAKE -j $(getconf _NPROCESSORS_ONLN) # Native architecture build + test jobs @@ -120,8 +123,8 @@ website: - mkdir build - cd build - ../autogen.sh --prefix=$(pwd)/../vroot || (cat config.log && exit 1) - - make -j $(getconf _NPROCESSORS_ONLN) -C docs - - make -j $(getconf _NPROCESSORS_ONLN) -C docs install + - $MAKE -j $(getconf _NPROCESSORS_ONLN) -C docs + - $MAKE -j $(getconf _NPROCESSORS_ONLN) -C docs install - cd .. - mv vroot/share/doc/libvirt/html/ website image: quay.io/libvirt/buildenv-libvirt-fedora-31:latest -- 2.24.1
On Tue, 2020-03-24 at 16:24 +0000, Daniel P. Berrangé wrote:
To facilitate future jobs that will use FreeBSD
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
As mentioned, this should be the very first patch in the series. Reviewed-by: Andrea Bolognani <abologna@redhat.com> -- Andrea Bolognani / Red Hat / Virtualization
On Tue, Mar 24, 2020 at 06:47:50PM +0100, Andrea Bolognani wrote:
On Tue, 2020-03-24 at 16:24 +0000, Daniel P. Berrangé wrote:
To facilitate future jobs that will use FreeBSD
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
As mentioned, this should be the very first patch in the series.
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Agreed. Reviewed-by: Erik Skultety <eskultet@redhat.com>
We don't need the full git history when running CI jobs. From a code POV we only need the most recent commit, but we want to be able to run checks on the commits too. In particular to validate the DCO signoff for each commit. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 9814b6580a..2286d28707 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -6,6 +6,7 @@ stages: variables: MAKE: make + GIT_DEPTH: 100 # Common templates -- 2.24.1
On Tue, 2020-03-24 at 16:24 +0000, Daniel P. Berrangé wrote:
We don't need the full git history when running CI jobs. From a code POV we only need the most recent commit, but we want to be able to run checks on the commits too. In particular to validate the DCO signoff for each commit.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 1 + 1 file changed, 1 insertion(+)
This would also make more sense right at the beginning of the series instead of in the middle of it. Reviewed-by: Andrea Bolognani <abologna@redhat.com> -- Andrea Bolognani / Red Hat / Virtualization
On Tue, Mar 24, 2020 at 06:48:52PM +0100, Andrea Bolognani wrote:
On Tue, 2020-03-24 at 16:24 +0000, Daniel P. Berrangé wrote:
We don't need the full git history when running CI jobs. From a code POV we only need the most recent commit, but we want to be able to run checks on the commits too. In particular to validate the DCO signoff for each commit.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 1 + 1 file changed, 1 insertion(+)
This would also make more sense right at the beginning of the series instead of in the middle of it.
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
With Andrea's proposed change: Reviewed-by: Erik Skultety <eskultet@redhat.com>
Whenever there is a change to the translatable strings we need to push a new libvirt.pot to weblate. This only needs to be done when code merges into git master, so the job is restricted to that branch. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 2286d28707..08e8df5d25 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -136,3 +136,29 @@ website: expire_in: 30 days paths: - website + + +# This artifact published by this job is downloaded to push to Weblate +# for translation usage: +# https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?job=potf... +potfile: + stage: prebuild + only: + refs: + - master + script: + - mkdir build + - cd build + - ../autogen.sh || (cat config.log && exit 1) + - $MAKE -j $(getconf _NPROCESSORS_ONLN) -C src generated-sources + - $MAKE -j $(getconf _NPROCESSORS_ONLN) -C po libvirt.pot + - cd .. + - mv build/po/libvirt.pot libvirt.pot + image: quay.io/libvirt/buildenv-libvirt-fedora-31:latest + artifacts: + expose_as: 'Potfile' + name: 'potfile' + when: on_success + expire_in: 30 days + paths: + - libvirt.pot -- 2.24.1
On Tue, 2020-03-24 at 16:24 +0000, Daniel P. Berrangé wrote:
+# This artifact published by this job is downloaded to push to Weblate +# for translation usage: +# https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?job=potf... +potfile: + stage: prebuild + only: + refs: + - master
As discussed in patch 6/9, it would make more sense for this to be in a postbuild stage, after all other builds have succeeded, than in a prebuild stage. -- Andrea Bolognani / Red Hat / Virtualization
On Tue, Mar 24, 2020 at 06:51:10PM +0100, Andrea Bolognani wrote:
On Tue, 2020-03-24 at 16:24 +0000, Daniel P. Berrangé wrote:
+# This artifact published by this job is downloaded to push to Weblate +# for translation usage: +# https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?job=potf... +potfile: + stage: prebuild + only: + refs: + - master
As discussed in patch 6/9, it would make more sense for this to be in a postbuild stage, after all other builds have succeeded, than in a prebuild stage.
We want the pot file build to run regardless of whether the other builds succeed. There may be transient failures with builds and we want to publish the pot & website content regardless, once it is in master. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Mar 24, 2020 at 06:23:19PM +0000, Daniel P. Berrangé wrote:
On Tue, Mar 24, 2020 at 06:51:10PM +0100, Andrea Bolognani wrote:
On Tue, 2020-03-24 at 16:24 +0000, Daniel P. Berrangé wrote:
+# This artifact published by this job is downloaded to push to Weblate +# for translation usage: +# https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?job=potf... +potfile: + stage: prebuild + only: + refs: + - master
As discussed in patch 6/9, it would make more sense for this to be in a postbuild stage, after all other builds have succeeded, than in a prebuild stage.
We want the pot file build to run regardless of whether the other builds succeed. There may be transient failures with builds and we want to publish the pot & website content regardless, once it is in master.
That explanation makes sense. Reviewed-by: Erik Skultety <eskultet@redhat.com>
On Tue, 2020-03-24 at 18:23 +0000, Daniel P. Berrangé wrote:
On Tue, Mar 24, 2020 at 06:51:10PM +0100, Andrea Bolognani wrote:
On Tue, 2020-03-24 at 16:24 +0000, Daniel P. Berrangé wrote:
+# This artifact published by this job is downloaded to push to Weblate +# for translation usage: +# https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?job=potf... +potfile: + stage: prebuild + only: + refs: + - master
As discussed in patch 6/9, it would make more sense for this to be in a postbuild stage, after all other builds have succeeded, than in a prebuild stage.
We want the pot file build to run regardless of whether the other builds succeed. There may be transient failures with builds and we want to publish the pot & website content regardless, once it is in master.
Makes sense. Reviewed-by: Andrea Bolognani <abologna@redhat.com> -- Andrea Bolognani / Red Hat / Virtualization
On Tue, Mar 24, 2020 at 04:24:05PM +0000, Daniel P. Berrangé wrote:
Whenever there is a change to the translatable strings we need to push a new libvirt.pot to weblate. This only needs to be done when code merges into git master, so the job is restricted to that branch.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 2286d28707..08e8df5d25 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -136,3 +136,29 @@ website: expire_in: 30 days paths: - website + + +# This artifact published by this job is downloaded to push to Weblate +# for translation usage: +# https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?job=potf... +potfile: + stage: prebuild + only: + refs: + - master
Actually, my fingers were too fast...same as in one of the previous patches: only: - master should be sufficient... Erik
On Tue, Mar 24, 2020 at 04:23:56PM +0000, Daniel P. Berrangé wrote:
There are main goals with this series
- Introduce a minimal job building the website and publishing an artifact which can be deployed onto libvirt.org - Introduce a minimal job building the libvirt.pot for import into Weblate (only runs on git master branch) - Expanding CI jobs to get coverage closer to Travis/Jenkins - Reducing cross-build jobs to just interesting variants, since the full set hasn't shown value in detecting failures
The Linux native job coverage is now a superset of that achieved by Travis/Jenkins.
For post-merge testing the full set of jobs are run on git master (measured approx 50 minutes total duration)
For pre-merge testing the Linux job count is reduced for quicker turnaround time for developers (measured ~35 minutes total duration)
I meant to say you can see an illustration of the CI pipeline in this URL. https://gitlab.com/berrange/libvirt/pipelines/129242976 Published artifacts are downloaded from the "jobs" tab, and if we had unit tests published in junit data format, they'd appear in the "tests" tab IIUC Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (3)
-
Andrea Bolognani -
Daniel P. Berrangé -
Erik Skultety