[libvirt PATCH 0/4] gitlab: improve CI speed
In several of the jobs we are running the libvirt build twice. We also uncessarily run the syntax-check job multiple times which causes more work for contributors looking at failed jobs. Daniel P. Berrangé (4): gitlab: only print failed tests in codestyle job gitlab: remove ability to skip 'ninja dist' gitlab: run "meson" directly instead of "ninja" indirection gitlab: avoid building libvirt twice and running syntax-check twice .gitlab-ci.yml | 31 ++++++++++++++----------------- ci/cirrus/build.yml | 6 ++++-- 2 files changed, 18 insertions(+), 19 deletions(-) -- 2.31.1
"meson test" will only print a list of which tests fail by default, so we were sending the full test log to stdout on failure. This makes it really hard to see the errors though as the test log has all succcesful tests too. "ninja test" will print the same as "meson test", following by details of each failure. It does this using the "--print-errorlog" flag, so lets use that in the codestyle job. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 89f618e678..cf0ce268b8 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -555,7 +555,7 @@ codestyle: script: - meson build --werror || (cat build/meson-logs/meson-log.txt && exit 1) - ninja -C build libvirt-pot-dep - - meson test -C build --suite syntax-check --no-rebuild || (cat build/meson-logs/testlog.txt && exit 1) + - meson test -C build --suite syntax-check --no-rebuild --print-errorlogs # This artifact published by this job is downloaded to push to Weblate -- 2.31.1
The "DIST=skip" flag as used in CentOS 7 jobs to workaround a problem with older git versions. This is no longer required since commit d35003aee7d057e66d935cfd919ea86d5d450c4c Author: Andrea Bolognani <abologna@redhat.com> Date: Tue May 4 10:45:29 2021 +0200 ci: Drop CentOS 7 Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index cf0ce268b8..a3b0e9ebba 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -61,13 +61,7 @@ stages: - *script_variables script: - meson build --werror || (cat build/meson-logs/meson-log.txt && exit 1) - - if test "$DIST" != "skip"; - then - ninja -C build dist; - else - ninja -C build; - ninja -C build test; - fi + - ninja -C build dist; - if test -x /usr/bin/rpmbuild && test "$RPM" != "skip"; then rpmbuild --nodeps -ta build/meson-dist/libvirt-*.tar.xz; -- 2.31.1
The "dist" and "test" targets in ninja end up calling back into the equivalent meson commands. The meson commands support various arguments that are not accessible when invoked via ninja, so it is preferrable to use meson directly. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 20 ++++++++++---------- ci/cirrus/build.yml | 4 ++-- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index a3b0e9ebba..9dfbd0ca54 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -60,8 +60,8 @@ stages: before_script: - *script_variables script: - - meson build --werror || (cat build/meson-logs/meson-log.txt && exit 1) - - ninja -C build dist; + - meson setup build --werror || (cat build/meson-logs/meson-log.txt && exit 1) + - meson dist -C build - if test -x /usr/bin/rpmbuild && test "$RPM" != "skip"; then rpmbuild --nodeps -ta build/meson-dist/libvirt-*.tar.xz; @@ -122,9 +122,9 @@ stages: before_script: - *script_variables script: - - meson build --werror $MESON_OPTS || (cat build/meson-logs/meson-log.txt && exit 1) - - ninja -C build - - if test "$CROSS" = "i686" ; then ninja -C build test ; fi + - meson setup build --werror $MESON_OPTS || (cat build/meson-logs/meson-log.txt && exit 1) + - meson compile -C build + - if test "$CROSS" = "i686" ; then meson test -C build --print-errorlogs ; fi # Native container build jobs @@ -527,7 +527,7 @@ website: before_script: - *script_variables script: - - meson build --werror --prefix=$(pwd)/vroot || (cat build/meson-logs/meson-log.txt && exit 1) + - meson setup build --werror --prefix=$(pwd)/vroot || (cat build/meson-logs/meson-log.txt && exit 1) - ninja -C build install-web - mv vroot/share/doc/libvirt/html/ website artifacts: @@ -547,7 +547,7 @@ codestyle: before_script: - *script_variables script: - - meson build --werror || (cat build/meson-logs/meson-log.txt && exit 1) + - meson setup build --werror || (cat build/meson-logs/meson-log.txt && exit 1) - ninja -C build libvirt-pot-dep - meson test -C build --suite syntax-check --no-rebuild --print-errorlogs @@ -565,7 +565,7 @@ potfile: before_script: - *script_variables script: - - meson build --werror || (cat build/meson-logs/meson-log.txt && exit 1) + - meson setup build --werror || (cat build/meson-logs/meson-log.txt && exit 1) - ninja -C build libvirt-pot-dep - ninja -C build libvirt-pot - cp po/libvirt.pot libvirt.pot @@ -603,8 +603,8 @@ coverity: script: - curl https://scan.coverity.com/download/linux64 --form project=$COVERITY_SCAN_PROJECT_NAME --form token=$COVERITY_SCAN_TOKEN -o /tmp/cov-analysis-linux64.tgz - tar xfz /tmp/cov-analysis-linux64.tgz - - meson build --werror || (cat build/meson-logs/meson-log.txt && exit 1) - - cov-analysis-linux64-*/bin/cov-build --dir cov-int ninja -C build + - meson setup build --werror || (cat build/meson-logs/meson-log.txt && exit 1) + - cov-analysis-linux64-*/bin/cov-build --dir cov-int meson compile -C build - tar cfz cov-int.tar.gz cov-int - curl https://scan.coverity.com/builds?project=$COVERITY_SCAN_PROJECT_NAME --form token=$COVERITY_SCAN_TOKEN --form email=$GITLAB_USER_EMAIL --form file=@cov-int.tar.gz --form version="$(git describe --tags)" --form description="$(git describe --tags) / $CI_COMMIT_TITLE / $CI_COMMIT_REF_NAME:$CI_PIPELINE_ID" rules: diff --git a/ci/cirrus/build.yml b/ci/cirrus/build.yml index f7ef5c2d13..c9420a8d91 100644 --- a/ci/cirrus/build.yml +++ b/ci/cirrus/build.yml @@ -22,5 +22,5 @@ build_task: - git fetch origin "$CI_COMMIT_REF_NAME" - git reset --hard "$CI_COMMIT_SHA" build_script: - - meson build - - ninja -C build dist + - meson setup build + - meson dist -C build -- 2.31.1
In the distros using RPMs, we build libvirt once as a side effect of running "ninja dist", and once via rpmbuild. In addition "ninja dist" will run all tests including the "syntax-check" suite, despite use having a separate "codestyle" job for for that. There is no way to pass "--no-suite" when creating the dist, but if we switch to invoking "meson dist", we can skip the build+test part entirely using "--no-tests". When doing this we then run explicit "meson compile" and "meson test" commands for the distros that don't build the RPMs, and in the latter case we can now skip the "syntax-check" suite. The RPM builds already skipped the "syntax-check" suite. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.yml | 7 +++++-- ci/cirrus/build.yml | 4 +++- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 9dfbd0ca54..11a201563f 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -61,10 +61,13 @@ stages: - *script_variables script: - meson setup build --werror || (cat build/meson-logs/meson-log.txt && exit 1) - - meson dist -C build + - meson dist -C build --no-tests - if test -x /usr/bin/rpmbuild && test "$RPM" != "skip"; then rpmbuild --nodeps -ta build/meson-dist/libvirt-*.tar.xz; + else + meson compile -C build; + meson test -C build --no-suite syntax-check --print-errorlogs; fi # Jobs that we delegate to Cirrus CI because they require an operating @@ -124,7 +127,7 @@ stages: script: - meson setup build --werror $MESON_OPTS || (cat build/meson-logs/meson-log.txt && exit 1) - meson compile -C build - - if test "$CROSS" = "i686" ; then meson test -C build --print-errorlogs ; fi + - if test "$CROSS" = "i686" ; then meson test -C build --no-suite syntax-check --print-errorlogs ; fi # Native container build jobs diff --git a/ci/cirrus/build.yml b/ci/cirrus/build.yml index c9420a8d91..39c17dc08a 100644 --- a/ci/cirrus/build.yml +++ b/ci/cirrus/build.yml @@ -23,4 +23,6 @@ build_task: - git reset --hard "$CI_COMMIT_SHA" build_script: - meson setup build - - meson dist -C build + - meson dist -C build --no-tests + - meson compile -C build + - meson test -C build --no-suite syntax-check -- 2.31.1
On 5/10/21 9:07 PM, Daniel P. Berrangé wrote:
In several of the jobs we are running the libvirt build twice.
We also uncessarily run the syntax-check job multiple times which causes more work for contributors looking at failed jobs.
Daniel P. Berrangé (4): gitlab: only print failed tests in codestyle job gitlab: remove ability to skip 'ninja dist' gitlab: run "meson" directly instead of "ninja" indirection gitlab: avoid building libvirt twice and running syntax-check twice
.gitlab-ci.yml | 31 ++++++++++++++----------------- ci/cirrus/build.yml | 6 ++++-- 2 files changed, 18 insertions(+), 19 deletions(-)
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Michal
participants (2)
-
Daniel P. Berrangé -
Michal Prívozník