[libvirt] [jenkins-ci PATCH v2 0/3] Enable out-of-the-box parallel make
Changes from [v1]: * turns out some versions of ExtUtils::MakeMaker output Makefiles that are not entirely compatible with parallel make, which forces us to introduce an exception in the relevant template and shuffle patches around. [v1] https://www.redhat.com/archives/libvir-list/2018-May/msg00732.html Andrea Bolognani (3): jobs: Enable parallel make everywhere jobs: Drop explicit parallel make usage guests: Set MAKEFLAGS for out-of-the-box parallel make guests/templates/bashrc | 2 ++ jobs/autotools.yaml | 10 +++++----- jobs/defaults.yaml | 1 - jobs/perl-makemaker.yaml | 12 +++++++----- projects/libvirt.yaml | 4 ++-- projects/osinfo-db.yaml | 4 ++-- 6 files changed, 18 insertions(+), 15 deletions(-) -- 2.17.0
A few jobs were running serial make for no apparent reason. Almost all Makefiles have been tested and found to work just fine even with parallel execution, the only exception being those generated by some versions of ExtUtils::MakeMaker; ensure parallel make is used everywhere else. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- jobs/perl-makemaker.yaml | 2 ++ projects/osinfo-db.yaml | 6 +++--- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/jobs/perl-makemaker.yaml b/jobs/perl-makemaker.yaml index 84b368e..08f7162 100644 --- a/jobs/perl-makemaker.yaml +++ b/jobs/perl-makemaker.yaml @@ -43,6 +43,8 @@ perl Makefile.PL PREFIX="$VIRT_PREFIX" {global_env} {local_env} + # Makefiles generated by some versions of ExtUtils::MakeMaker + # fail when parallel make is used, so we have to avoid it here $MAKE $MAKE -j{smp} install $MAKE -j{smp} manifest diff --git a/projects/osinfo-db.yaml b/projects/osinfo-db.yaml index 0d0a972..b30a541 100644 --- a/projects/osinfo-db.yaml +++ b/projects/osinfo-db.yaml @@ -8,7 +8,7 @@ parent_jobs: 'osinfo-db-tools-master-build' command: | $MAKE -j{smp} all - $MAKE install OSINFO_DB_TARGET="--system" + $MAKE -j{smp} install OSINFO_DB_TARGET="--system" - generic-check-job: parent_jobs: 'osinfo-db-master-build' command: | @@ -18,6 +18,6 @@ machines: '{rpm_machines}' command: | rm -f *.tar.xz - $MAKE osinfo-db.spec + $MAKE -j{smp} osinfo-db.spec perl -i -p -e 's/BuildRequires: osinfo-db-tools.*//' osinfo-db.spec - $MAKE rpm + $MAKE -j{smp} rpm -- 2.17.0
We're going to enable it by default throught $MAKEFLAGS in the next commit, which will make using '-j{smp}' explicitly with each invocation entirely unnecessary. Since the 'smp' variable is no longer used anywhere, we can drop it. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- jobs/autotools.yaml | 10 +++++----- jobs/defaults.yaml | 1 - jobs/perl-makemaker.yaml | 10 +++++----- projects/libvirt.yaml | 4 ++-- projects/osinfo-db.yaml | 10 +++++----- 5 files changed, 17 insertions(+), 18 deletions(-) diff --git a/jobs/autotools.yaml b/jobs/autotools.yaml index 9868573..a07d806 100644 --- a/jobs/autotools.yaml +++ b/jobs/autotools.yaml @@ -45,8 +45,8 @@ mkdir build cd build ../autogen.sh --prefix=$VIRT_PREFIX {autogen_args} - $MAKE -j{smp} - $MAKE -j{smp} install + $MAKE + $MAKE install publishers: - email: recipients: '{obj:spam}' @@ -85,7 +85,7 @@ {global_env} {local_env} cd build - $MAKE -j{smp} syntax-check + $MAKE syntax-check publishers: - email: recipients: '{obj:spam}' @@ -124,7 +124,7 @@ {global_env} {local_env} cd build - if ! $MAKE -j{smp} check + if ! $MAKE check then cat tests/test-suite.log || true exit 1 @@ -172,7 +172,7 @@ sed -i -e 's/BuildRequires: libvirt.*devel.*//' {name}.spec sed -i -e 's/BuildRequires: pkgconfig(libvirt.*).*//' {name}.spec rm -f *.tar.{archive_format} - $MAKE -j{smp} dist + $MAKE dist rpmbuild --define "_topdir `pwd`/rpmbuild" -ta {name}-*.tar.{archive_format} publishers: - email: diff --git a/jobs/defaults.yaml b/jobs/defaults.yaml index 45eedb7..de56226 100644 --- a/jobs/defaults.yaml +++ b/jobs/defaults.yaml @@ -32,5 +32,4 @@ export PKG_CONFIG_PATH="$VIRT_PREFIX/lib/pkgconfig" export PKG_CONFIG_LIBDIR="/usr/x86_64-w64-mingw32/sys-root/mingw/lib/pkgconfig:/usr/x86_64-w64-mingw32/sys-root/mingw/share/pkgconfig" mingw64_autogen_args: --host=x86_64-w64-mingw32 - smp: 3 spam: ymankad@redhat.com libvirt-ci@redhat.com diff --git a/jobs/perl-makemaker.yaml b/jobs/perl-makemaker.yaml index 08f7162..fc25e53 100644 --- a/jobs/perl-makemaker.yaml +++ b/jobs/perl-makemaker.yaml @@ -45,9 +45,9 @@ {local_env} # Makefiles generated by some versions of ExtUtils::MakeMaker # fail when parallel make is used, so we have to avoid it here - $MAKE - $MAKE -j{smp} install - $MAKE -j{smp} manifest + MAKEFLAGS= $MAKE + $MAKE install + $MAKE manifest publishers: - email: recipients: '{obj:spam}' @@ -87,7 +87,7 @@ - shell: | {global_env} {local_env} - $MAKE -j{smp} test {test_args} + $MAKE test {test_args} publishers: - email: recipients: '{obj:spam}' @@ -129,7 +129,7 @@ sed -i -e 's/BuildRequires: *perl-Sys-Virt.*//' *.spec sed -i -e 's/BuildRequires: *libvirt.*devel.*//' *.spec rm -f *.tar.{archive_format} - $MAKE -j{smp} dist + $MAKE dist rpmbuild --define "_topdir `pwd`/rpmbuild" -ta *.tar.{archive_format} publishers: - email: diff --git a/projects/libvirt.yaml b/projects/libvirt.yaml index 6f5464b..4bba21e 100644 --- a/projects/libvirt.yaml +++ b/projects/libvirt.yaml @@ -43,7 +43,7 @@ mkdir build cd build ../autogen.sh --without-libvirtd --without-macvtap - $MAKE -j{smp} -C docs/ - $MAKE -j{smp} dist + $MAKE -C docs/ + $MAKE dist machines: - libvirt-centos-6 diff --git a/projects/osinfo-db.yaml b/projects/osinfo-db.yaml index b30a541..9f52e9e 100644 --- a/projects/osinfo-db.yaml +++ b/projects/osinfo-db.yaml @@ -7,17 +7,17 @@ - generic-build-job: parent_jobs: 'osinfo-db-tools-master-build' command: | - $MAKE -j{smp} all - $MAKE -j{smp} install OSINFO_DB_TARGET="--system" + $MAKE all + $MAKE install OSINFO_DB_TARGET="--system" - generic-check-job: parent_jobs: 'osinfo-db-master-build' command: | - $MAKE -j{smp} check + $MAKE check - generic-rpm-job: parent_jobs: 'osinfo-db-master-check' machines: '{rpm_machines}' command: | rm -f *.tar.xz - $MAKE -j{smp} osinfo-db.spec + $MAKE osinfo-db.spec perl -i -p -e 's/BuildRequires: osinfo-db-tools.*//' osinfo-db.spec - $MAKE -j{smp} rpm + $MAKE rpm -- 2.17.0
We know all jobs are compatible with parallel make, and we also know exactly how many vCPUs guests have, since we created them in the first place: that allows us to default to parallel make instead of making it opt-in. The main advantage of doing this is that it will make life more convenient for developers running one-off jobs or debugging interactively. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/templates/bashrc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/guests/templates/bashrc b/guests/templates/bashrc index 866087e..2b3542c 100644 --- a/guests/templates/bashrc +++ b/guests/templates/bashrc @@ -14,6 +14,8 @@ export PYTHONPATH="{{ PYTHONPATH }}" export MAKE="{{ make }}" export PYTHON="{{ python }}" +export MAKEFLAGS="-j{{ install_vcpus|int + 1 }}" + # Enable bash completion. Only needed on FreeBSD, the system-wide # shell profile will take care of it for us everywhere else bash_completion=/usr/local/share/bash-completion/bash_completion.sh -- 2.17.0
On Tue, May 15, 2018 at 03:23:21PM +0200, Andrea Bolognani wrote:
Changes from [v1]:
* turns out some versions of ExtUtils::MakeMaker output Makefiles that are not entirely compatible with parallel make, which forces us to introduce an exception in the relevant template and shuffle patches around.
Oh, if that's a problem I'll happily drop use of MakeMaker in favour of Module::Build. Looks like it is only the core libvirt perl binding that actually still uses MakeMaker.
[v1] https://www.redhat.com/archives/libvir-list/2018-May/msg00732.html
Andrea Bolognani (3): jobs: Enable parallel make everywhere jobs: Drop explicit parallel make usage guests: Set MAKEFLAGS for out-of-the-box parallel make
guests/templates/bashrc | 2 ++ jobs/autotools.yaml | 10 +++++----- jobs/defaults.yaml | 1 - jobs/perl-makemaker.yaml | 12 +++++++----- projects/libvirt.yaml | 4 ++-- projects/osinfo-db.yaml | 4 ++-- 6 files changed, 18 insertions(+), 15 deletions(-)
-- 2.17.0
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, 2018-05-15 at 14:31 +0100, Daniel P. Berrangé wrote:
On Tue, May 15, 2018 at 03:23:21PM +0200, Andrea Bolognani wrote:
Changes from [v1]:
* turns out some versions of ExtUtils::MakeMaker output Makefiles that are not entirely compatible with parallel make, which forces us to introduce an exception in the relevant template and shuffle patches around.
Oh, if that's a problem I'll happily drop use of MakeMaker in favour of Module::Build. Looks like it is only the core libvirt perl binding that actually still uses MakeMaker.
Standardizing on a single build system for Perl sounds great :) Once you've done that, we can use v1 unchanged; or we can push v2 now and remove the tiny hack after you've switched libvirt-perl to Module::Build. Either one is fine with me. -- Andrea Bolognani / Red Hat / Virtualization
participants (2)
-
Andrea Bolognani -
Daniel P. Berrangé