[libvirt] [jenkins-ci PATCH v3 0/3] Enable out-of-the-box parallel make
Changes from [v2]: * now that libvirt-perl uses Module::Build and ExtUtils::MakeMaker support has been dropped, we don't need to special case any job, so revert back to [v1] and rebase on top of master because the original series doesn't apply anymore. Changes from [v1]: * turns out some versions of ExtUtils::MakeMaker output Makefiles that are not entirely compatible with parallel make, which forces us to introduce an exception in the relevant template and shuffle patches around. [v1] https://www.redhat.com/archives/libvir-list/2018-May/msg00732.html [v2] https://www.redhat.com/archives/libvir-list/2018-May/msg01070.html Andrea Bolognani (3): jobs: Enable parallel make everywhere guests: Set MAKEFLAGS for out-of-the-box parallel make jobs: Drop explicit parallel make usage guests/templates/bashrc | 2 ++ jobs/autotools.yaml | 10 +++++----- jobs/defaults.yaml | 1 - projects/libvirt.yaml | 4 ++-- projects/osinfo-db.yaml | 4 ++-- 5 files changed, 11 insertions(+), 10 deletions(-) -- 2.17.0
A few jobs were running serial make for no apparent reason. Since all Makefiles have been tested and found to work just fine even with parallel execution, make sure that's what's used everywhere. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- projects/osinfo-db.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/projects/osinfo-db.yaml b/projects/osinfo-db.yaml index 9e22c93..df5bd3e 100644 --- a/projects/osinfo-db.yaml +++ b/projects/osinfo-db.yaml @@ -9,7 +9,7 @@ parent_jobs: 'osinfo-db-tools-master-build' command: | $MAKE -j{smp} all - $MAKE install OSINFO_DB_TARGET="--system" + $MAKE -j{smp} install OSINFO_DB_TARGET="--system" - generic-check-job: parent_jobs: 'osinfo-db-master-build' command: | @@ -19,6 +19,6 @@ machines: '{rpm_machines}' command: | rm -f *.tar.{archive_format} - $MAKE osinfo-db.spec + $MAKE -j{smp} osinfo-db.spec perl -i -p -e 's/BuildRequires: osinfo-db-tools.*//' osinfo-db.spec - $MAKE rpm + $MAKE -j{smp} rpm -- 2.17.0
We know all jobs are compatible with parallel make, and we also know exactly how many vCPUs guests have, since we created them in the first place: that allows us to default to parallel make instead of making it opt-in. The main advantage of doing this is that it will make life more convenient for developers running one-off jobs or debugging interactively. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/templates/bashrc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/guests/templates/bashrc b/guests/templates/bashrc index 866087e..2b3542c 100644 --- a/guests/templates/bashrc +++ b/guests/templates/bashrc @@ -14,6 +14,8 @@ export PYTHONPATH="{{ PYTHONPATH }}" export MAKE="{{ make }}" export PYTHON="{{ python }}" +export MAKEFLAGS="-j{{ install_vcpus|int + 1 }}" + # Enable bash completion. Only needed on FreeBSD, the system-wide # shell profile will take care of it for us everywhere else bash_completion=/usr/local/share/bash-completion/bash_completion.sh -- 2.17.0
It's enabled by default through $MAKEFLAGS now, so enabling it explicitly in jobs is entirely unnecessary. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- jobs/autotools.yaml | 10 +++++----- jobs/defaults.yaml | 1 - projects/libvirt.yaml | 4 ++-- projects/osinfo-db.yaml | 10 +++++----- 4 files changed, 12 insertions(+), 13 deletions(-) diff --git a/jobs/autotools.yaml b/jobs/autotools.yaml index 638be5e..f526ed2 100644 --- a/jobs/autotools.yaml +++ b/jobs/autotools.yaml @@ -45,8 +45,8 @@ mkdir build cd build ../autogen.sh --prefix=$VIRT_PREFIX {autogen_args} - $MAKE -j{smp} - $MAKE -j{smp} install + $MAKE + $MAKE install publishers: - email: recipients: '{obj:spam}' @@ -85,7 +85,7 @@ {global_env} {local_env} cd build - $MAKE -j{smp} syntax-check + $MAKE syntax-check publishers: - email: recipients: '{obj:spam}' @@ -124,7 +124,7 @@ {global_env} {local_env} cd build - if ! $MAKE -j{smp} check + if ! $MAKE check then cat tests/test-suite.log || true exit 1 @@ -171,7 +171,7 @@ sed -i -e 's/BuildRequires: libvirt.*devel.*//' {name}.spec sed -i -e 's/BuildRequires: pkgconfig(libvirt.*).*//' {name}.spec rm -f *.tar.{archive_format} - $MAKE -j{smp} dist + $MAKE dist rpmbuild --define "_topdir `pwd`/rpmbuild" -ta {name}-*.tar.{archive_format} publishers: - email: diff --git a/jobs/defaults.yaml b/jobs/defaults.yaml index 0a1c757..eb04f32 100644 --- a/jobs/defaults.yaml +++ b/jobs/defaults.yaml @@ -33,5 +33,4 @@ export PKG_CONFIG_PATH="$VIRT_PREFIX/lib/pkgconfig" export PKG_CONFIG_LIBDIR="/usr/x86_64-w64-mingw32/sys-root/mingw/lib/pkgconfig:/usr/x86_64-w64-mingw32/sys-root/mingw/share/pkgconfig" mingw64_autogen_args: --host=x86_64-w64-mingw32 - smp: 3 spam: ymankad@redhat.com libvirt-ci@redhat.com diff --git a/projects/libvirt.yaml b/projects/libvirt.yaml index 3273ebf..3d893dd 100644 --- a/projects/libvirt.yaml +++ b/projects/libvirt.yaml @@ -47,7 +47,7 @@ mkdir build cd build ../autogen.sh --without-libvirtd --without-macvtap - $MAKE -j{smp} -C docs/ - $MAKE -j{smp} dist + $MAKE -C docs/ + $MAKE dist machines: - libvirt-centos-6 diff --git a/projects/osinfo-db.yaml b/projects/osinfo-db.yaml index df5bd3e..a197bbd 100644 --- a/projects/osinfo-db.yaml +++ b/projects/osinfo-db.yaml @@ -8,17 +8,17 @@ - generic-build-job: parent_jobs: 'osinfo-db-tools-master-build' command: | - $MAKE -j{smp} all - $MAKE -j{smp} install OSINFO_DB_TARGET="--system" + $MAKE all + $MAKE install OSINFO_DB_TARGET="--system" - generic-check-job: parent_jobs: 'osinfo-db-master-build' command: | - $MAKE -j{smp} check + $MAKE check - generic-rpm-job: parent_jobs: 'osinfo-db-master-check' machines: '{rpm_machines}' command: | rm -f *.tar.{archive_format} - $MAKE -j{smp} osinfo-db.spec + $MAKE osinfo-db.spec perl -i -p -e 's/BuildRequires: osinfo-db-tools.*//' osinfo-db.spec - $MAKE -j{smp} rpm + $MAKE rpm -- 2.17.0
On Tue, May 22, 2018 at 08:45:54AM +0200, Andrea Bolognani wrote:
Changes from [v2]:
* now that libvirt-perl uses Module::Build and ExtUtils::MakeMaker support has been dropped, we don't need to special case any job, so revert back to [v1] and rebase on top of master because the original series doesn't apply anymore.
Changes from [v1]:
* turns out some versions of ExtUtils::MakeMaker output Makefiles that are not entirely compatible with parallel make, which forces us to introduce an exception in the relevant template and shuffle patches around.
[v1] https://www.redhat.com/archives/libvir-list/2018-May/msg00732.html [v2] https://www.redhat.com/archives/libvir-list/2018-May/msg01070.html
Andrea Bolognani (3): jobs: Enable parallel make everywhere guests: Set MAKEFLAGS for out-of-the-box parallel make jobs: Drop explicit parallel make usage
guests/templates/bashrc | 2 ++ jobs/autotools.yaml | 10 +++++----- jobs/defaults.yaml | 1 - projects/libvirt.yaml | 4 ++-- projects/osinfo-db.yaml | 4 ++-- 5 files changed, 11 insertions(+), 10 deletions(-)
For all patches Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (2)
-
Andrea Bolognani -
Daniel P. Berrangé