[libvirt] [jenkins-ci PATCH 0/5] Fix environment handling
My own attempt at fixing libosinfo-master-rpm's failing status, and then some. This is an extension of Pavel's attempt[1] in a way; except of course I think my approach is superior, otherwise I wouldn't have bothered ;) It also sets the groundwork for finally moving $VIRT_PREFIX and friends out of the Jenkins server configuration, where they are invisible to anyone but the CI admins, and into the git repository so that they can be tracked alongside the jobs using them. [1] https://www.redhat.com/archives/libvir-list/2017-November/msg00045.html Andrea Bolognani (5): jobs: Rename {make_env} to {global_env} jobs: Include {global_env} everywhere jobs: Rename {check_env} to {env} jobs: Include {env} everywhere jobs: Include $PATH definition in {global_env} jobs/autotools.yaml | 17 ++++++++++++----- jobs/defaults.yaml | 5 +++-- jobs/generic.yaml | 20 ++++++++++++++++---- jobs/go.yaml | 4 ++++ jobs/perl-makemaker.yaml | 9 ++++++--- jobs/perl-modulebuild.yaml | 6 ++++++ jobs/python-distutils.yaml | 6 ++++++ projects/libosinfo.yaml | 2 +- projects/libvirt.yaml | 2 +- projects/osinfo-db.yaml | 4 ---- 10 files changed, 55 insertions(+), 20 deletions(-) -- 2.14.3
Right now, this environment only contains definitions that are useful when running make, but eventually it will be expanded and include all sort of variables that are not specific to a single project, eg. $VIRT_PREFIX and the like. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- jobs/autotools.yaml | 8 ++++---- jobs/defaults.yaml | 2 +- jobs/perl-makemaker.yaml | 6 +++--- projects/osinfo-db.yaml | 4 ++-- 4 files changed, 10 insertions(+), 10 deletions(-) diff --git a/jobs/autotools.yaml b/jobs/autotools.yaml index 74378ce..a4a5a5b 100644 --- a/jobs/autotools.yaml +++ b/jobs/autotools.yaml @@ -40,7 +40,7 @@ values: '{obj:machines}' builders: - shell: | - {make_env} + {global_env} mkdir build cd build ../autogen.sh --prefix=$VIRT_PREFIX {autogen_args} @@ -81,7 +81,7 @@ values: '{obj:machines}' builders: - shell: | - {make_env} + {global_env} cd build $MAKE -j{smp} syntax-check publishers: @@ -119,7 +119,7 @@ values: '{obj:machines}' builders: - shell: | - {make_env} + {global_env} {check_env} cd build if ! $MAKE -j{smp} check @@ -163,7 +163,7 @@ values: '{obj:machines}' builders: - shell: | - {make_env} + {global_env} cd build sed -i -e 's/BuildRequires: osinfo-db.*//' {name}.spec sed -i -e 's/BuildRequires: libvirt.*devel.*//' {name}.spec diff --git a/jobs/defaults.yaml b/jobs/defaults.yaml index 0e628bb..e2a5b1a 100644 --- a/jobs/defaults.yaml +++ b/jobs/defaults.yaml @@ -3,7 +3,7 @@ name: global branch: master node: libvirt - make_env: | + global_env: | MAKE='make' unamestr=`uname` if [ "$unamestr" = 'FreeBSD' ]; then diff --git a/jobs/perl-makemaker.yaml b/jobs/perl-makemaker.yaml index 0a3227a..930d4ee 100644 --- a/jobs/perl-makemaker.yaml +++ b/jobs/perl-makemaker.yaml @@ -41,7 +41,7 @@ builders: - shell: | perl Makefile.PL PREFIX="$VIRT_PREFIX" - {make_env} + {global_env} $MAKE $MAKE -j{smp} install $MAKE -j{smp} manifest @@ -82,7 +82,7 @@ values: '{obj:machines}' builders: - shell: | - {make_env} + {global_env} $MAKE -j{smp} test {test_args} publishers: - email: @@ -120,7 +120,7 @@ values: '{obj:machines}' builders: - shell: | - {make_env} + {global_env} sed -i -e 's/BuildRequires: *perl-Sys-Virt.*//' *.spec sed -i -e 's/BuildRequires: *libvirt.*devel.*//' *.spec rm -f *.tar.{archive_format} diff --git a/projects/osinfo-db.yaml b/projects/osinfo-db.yaml index 6fa1895..9d70000 100644 --- a/projects/osinfo-db.yaml +++ b/projects/osinfo-db.yaml @@ -15,7 +15,7 @@ - generic-build-job: parent_jobs: 'osinfo-db-tools-master-build' command: | - {make_env} + {global_env} export PATH=$VIRT_PREFIX/bin:$PATH $MAKE -j{smp} all $MAKE install OSINFO_DB_TARGET="--system" @@ -27,7 +27,7 @@ - libvirt-fedora-26 - libvirt-fedora-rawhide command: | - {make_env} + {global_env} export PATH=$VIRT_PREFIX/bin:$PATH rm -f *.tar.xz $MAKE osinfo-db.spec -- 2.14.3
This environment will later include variables that are relevant to more than make, so it should be included every time a shell command is going to be executed. Including {global_env} in the generic-*-job templates allows us to drop the ad-hoc inclusions from the osinfo-db project. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- jobs/autotools.yaml | 2 ++ jobs/generic.yaml | 16 ++++++++++++---- jobs/go.yaml | 2 ++ jobs/perl-modulebuild.yaml | 3 +++ jobs/python-distutils.yaml | 3 +++ projects/osinfo-db.yaml | 2 -- 6 files changed, 22 insertions(+), 6 deletions(-) diff --git a/jobs/autotools.yaml b/jobs/autotools.yaml index a4a5a5b..8a37d00 100644 --- a/jobs/autotools.yaml +++ b/jobs/autotools.yaml @@ -218,6 +218,7 @@ values: '{obj:machines}' builders: - shell: | + {global_env} mkdir -p build32 cd build32 @@ -228,6 +229,7 @@ $MAKE -j{smp} $MAKE -j{smp} install - shell: | + {global_env} mkdir -p build64 cd build64 diff --git a/jobs/generic.yaml b/jobs/generic.yaml index 050e35d..20ef5da 100644 --- a/jobs/generic.yaml +++ b/jobs/generic.yaml @@ -39,7 +39,9 @@ type: slave values: '{obj:machines}' builders: - - shell: '{command}' + - shell: | + {global_env} + {command} publishers: - email: recipients: '{obj:spam}' @@ -74,7 +76,9 @@ type: slave values: '{obj:machines}' builders: - - shell: '{command}' + - shell: | + {global_env} + {command} publishers: - email: recipients: '{obj:spam}' @@ -109,7 +113,9 @@ type: slave values: '{obj:machines}' builders: - - shell: '{command}' + - shell: | + {global_env} + {command} publishers: - email: recipients: '{obj:spam}' @@ -145,7 +151,9 @@ type: slave values: '{obj:machines}' builders: - - shell: '{command}' + - shell: | + {global_env} + {command} publishers: - email: recipients: '{obj:spam}' diff --git a/jobs/go.yaml b/jobs/go.yaml index 8eee6af..8764395 100644 --- a/jobs/go.yaml +++ b/jobs/go.yaml @@ -40,6 +40,7 @@ values: '{obj:machines}' builders: - shell: | + {global_env} go build -v publishers: - email: @@ -76,6 +77,7 @@ values: '{obj:machines}' builders: - shell: | + {global_env} go test -tags api publishers: - email: diff --git a/jobs/perl-modulebuild.yaml b/jobs/perl-modulebuild.yaml index cb81de1..188b153 100644 --- a/jobs/perl-modulebuild.yaml +++ b/jobs/perl-modulebuild.yaml @@ -40,6 +40,7 @@ values: '{obj:machines}' builders: - shell: | + {global_env} perl Build.PL install_base="$VIRT_PREFIX" perl Build perl Build manifest @@ -80,6 +81,7 @@ values: '{obj:machines}' builders: - shell: | + {global_env} perl Build test publishers: - email: @@ -117,6 +119,7 @@ values: '{obj:machines}' builders: - shell: | + {global_env} sed -i -e 's/BuildRequires: *perl-Sys-Virt.*//' *.spec sed -i -e 's/BuildRequires: *perl(Sys::Virt).*//' *.spec sed -i -e 's/BuildRequires: *libvirt.*devel.*//' *.spec diff --git a/jobs/python-distutils.yaml b/jobs/python-distutils.yaml index 7055d26..b0e3f5f 100644 --- a/jobs/python-distutils.yaml +++ b/jobs/python-distutils.yaml @@ -40,6 +40,7 @@ values: '{obj:machines}' builders: - shell: | + {global_env} {command_pre_build} python2 setup.py build python2 setup.py install --prefix=$VIRT_PREFIX @@ -79,6 +80,7 @@ values: '{obj:machines}' builders: - shell: | + {global_env} python2 setup.py test publishers: - email: @@ -115,6 +117,7 @@ values: '{obj:machines}' builders: - shell: | + {global_env} sed -i -e 's/BuildRequires: libvirt.*devel.*//' *.spec.in python2 setup.py rpm publishers: diff --git a/projects/osinfo-db.yaml b/projects/osinfo-db.yaml index 9d70000..95ad845 100644 --- a/projects/osinfo-db.yaml +++ b/projects/osinfo-db.yaml @@ -15,7 +15,6 @@ - generic-build-job: parent_jobs: 'osinfo-db-tools-master-build' command: | - {global_env} export PATH=$VIRT_PREFIX/bin:$PATH $MAKE -j{smp} all $MAKE install OSINFO_DB_TARGET="--system" @@ -27,7 +26,6 @@ - libvirt-fedora-26 - libvirt-fedora-rawhide command: | - {global_env} export PATH=$VIRT_PREFIX/bin:$PATH rm -f *.tar.xz $MAKE osinfo-db.spec -- 2.14.3
There's no reason to have an environment which is tied, at least in name, to *-check-job only: having a local environment is something that can be useful for any kind of job, so it's better to use a neutral name. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- jobs/autotools.yaml | 2 +- jobs/defaults.yaml | 2 +- projects/libosinfo.yaml | 2 +- projects/libvirt.yaml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/jobs/autotools.yaml b/jobs/autotools.yaml index 8a37d00..f6a2af0 100644 --- a/jobs/autotools.yaml +++ b/jobs/autotools.yaml @@ -120,7 +120,7 @@ builders: - shell: | {global_env} - {check_env} + {env} cd build if ! $MAKE -j{smp} check then diff --git a/jobs/defaults.yaml b/jobs/defaults.yaml index e2a5b1a..86e7100 100644 --- a/jobs/defaults.yaml +++ b/jobs/defaults.yaml @@ -9,6 +9,6 @@ if [ "$unamestr" = 'FreeBSD' ]; then MAKE='gmake' fi - check_env: | + env: | smp: 3 spam: ymankad@redhat.com libvirt-ci@redhat.com diff --git a/projects/libosinfo.yaml b/projects/libosinfo.yaml index 9ab2281..311ca71 100644 --- a/projects/libosinfo.yaml +++ b/projects/libosinfo.yaml @@ -11,7 +11,7 @@ - libvirt-freebsd-10 - libvirt-freebsd-11 title: libosinfo - check_env: | + env: | export OSINFO_SYSTEM_DIR=$VIRT_PREFIX/share/osinfo jobs: - autotools-build-job: diff --git a/projects/libvirt.yaml b/projects/libvirt.yaml index 5661066..52f8740 100644 --- a/projects/libvirt.yaml +++ b/projects/libvirt.yaml @@ -28,7 +28,7 @@ parent_jobs: 'libvirt-master-build' - autotools-check-job: parent_jobs: 'libvirt-master-syntax-check' - check_env: | + env: | export VIR_TEST_EXPENSIVE=1 export VIR_TEST_DEBUG=2 - autotools-rpm-job: -- 2.14.3
On Fri, Nov 17, 2017 at 07:26:10PM +0100, Andrea Bolognani wrote:
There's no reason to have an environment which is tied, at least in name, to *-check-job only: having a local environment is something that can be useful for any kind of job, so it's better to use a neutral name.
My proposal was "job_env" and you as a reply suggested "local_env". I would probably prefer using "local_env" over "env" since it is local environment as you've described it in commit message :). Pavel
On Mon, 2017-11-20 at 14:29 +0100, Pavel Hrdina wrote:
On Fri, Nov 17, 2017 at 07:26:10PM +0100, Andrea Bolognani wrote:
There's no reason to have an environment which is tied, at least in name, to *-check-job only: having a local environment is something that can be useful for any kind of job, so it's better to use a neutral name.
My proposal was "job_env" and you as a reply suggested "local_env". I would probably prefer using "local_env" over "env" since it is local environment as you've described it in commit message :).
'env' looks more natural when defining projects, but 'local_env' mirrors 'global_env' nicely and it's way more greppable, so I've changed it and pushed :) -- Andrea Bolognani / Red Hat / Virtualization
Any job might need to define a custom environment, so it should be included everywhere to make it possible. A side-effect of this change is that $OSINFO_SYSTEM_DIR will be defined in libosinfo-master-rpm's environment, and the job should stop failing. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- jobs/autotools.yaml | 5 +++++ jobs/generic.yaml | 4 ++++ jobs/go.yaml | 2 ++ jobs/perl-makemaker.yaml | 3 +++ jobs/perl-modulebuild.yaml | 3 +++ jobs/python-distutils.yaml | 3 +++ 6 files changed, 20 insertions(+) diff --git a/jobs/autotools.yaml b/jobs/autotools.yaml index f6a2af0..d71d0ce 100644 --- a/jobs/autotools.yaml +++ b/jobs/autotools.yaml @@ -41,6 +41,7 @@ builders: - shell: | {global_env} + {env} mkdir build cd build ../autogen.sh --prefix=$VIRT_PREFIX {autogen_args} @@ -82,6 +83,7 @@ builders: - shell: | {global_env} + {env} cd build $MAKE -j{smp} syntax-check publishers: @@ -164,6 +166,7 @@ builders: - shell: | {global_env} + {env} cd build sed -i -e 's/BuildRequires: osinfo-db.*//' {name}.spec sed -i -e 's/BuildRequires: libvirt.*devel.*//' {name}.spec @@ -219,6 +222,7 @@ builders: - shell: | {global_env} + {env} mkdir -p build32 cd build32 @@ -230,6 +234,7 @@ $MAKE -j{smp} install - shell: | {global_env} + {env} mkdir -p build64 cd build64 diff --git a/jobs/generic.yaml b/jobs/generic.yaml index 20ef5da..9b4569f 100644 --- a/jobs/generic.yaml +++ b/jobs/generic.yaml @@ -41,6 +41,7 @@ builders: - shell: | {global_env} + {env} {command} publishers: - email: @@ -78,6 +79,7 @@ builders: - shell: | {global_env} + {env} {command} publishers: - email: @@ -115,6 +117,7 @@ builders: - shell: | {global_env} + {env} {command} publishers: - email: @@ -153,6 +156,7 @@ builders: - shell: | {global_env} + {env} {command} publishers: - email: diff --git a/jobs/go.yaml b/jobs/go.yaml index 8764395..22d3459 100644 --- a/jobs/go.yaml +++ b/jobs/go.yaml @@ -41,6 +41,7 @@ builders: - shell: | {global_env} + {env} go build -v publishers: - email: @@ -78,6 +79,7 @@ builders: - shell: | {global_env} + {env} go test -tags api publishers: - email: diff --git a/jobs/perl-makemaker.yaml b/jobs/perl-makemaker.yaml index 930d4ee..ea05b61 100644 --- a/jobs/perl-makemaker.yaml +++ b/jobs/perl-makemaker.yaml @@ -42,6 +42,7 @@ - shell: | perl Makefile.PL PREFIX="$VIRT_PREFIX" {global_env} + {env} $MAKE $MAKE -j{smp} install $MAKE -j{smp} manifest @@ -83,6 +84,7 @@ builders: - shell: | {global_env} + {env} $MAKE -j{smp} test {test_args} publishers: - email: @@ -121,6 +123,7 @@ builders: - shell: | {global_env} + {env} sed -i -e 's/BuildRequires: *perl-Sys-Virt.*//' *.spec sed -i -e 's/BuildRequires: *libvirt.*devel.*//' *.spec rm -f *.tar.{archive_format} diff --git a/jobs/perl-modulebuild.yaml b/jobs/perl-modulebuild.yaml index 188b153..e556237 100644 --- a/jobs/perl-modulebuild.yaml +++ b/jobs/perl-modulebuild.yaml @@ -41,6 +41,7 @@ builders: - shell: | {global_env} + {env} perl Build.PL install_base="$VIRT_PREFIX" perl Build perl Build manifest @@ -82,6 +83,7 @@ builders: - shell: | {global_env} + {env} perl Build test publishers: - email: @@ -120,6 +122,7 @@ builders: - shell: | {global_env} + {env} sed -i -e 's/BuildRequires: *perl-Sys-Virt.*//' *.spec sed -i -e 's/BuildRequires: *perl(Sys::Virt).*//' *.spec sed -i -e 's/BuildRequires: *libvirt.*devel.*//' *.spec diff --git a/jobs/python-distutils.yaml b/jobs/python-distutils.yaml index b0e3f5f..fc969ad 100644 --- a/jobs/python-distutils.yaml +++ b/jobs/python-distutils.yaml @@ -41,6 +41,7 @@ builders: - shell: | {global_env} + {env} {command_pre_build} python2 setup.py build python2 setup.py install --prefix=$VIRT_PREFIX @@ -81,6 +82,7 @@ builders: - shell: | {global_env} + {env} python2 setup.py test publishers: - email: @@ -118,6 +120,7 @@ builders: - shell: | {global_env} + {env} sed -i -e 's/BuildRequires: libvirt.*devel.*//' *.spec.in python2 setup.py rpm publishers: -- 2.14.3
$PATH is generic enough that it shouldn't be limited to the osinfo-db project. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- jobs/defaults.yaml | 1 + projects/osinfo-db.yaml | 2 -- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/jobs/defaults.yaml b/jobs/defaults.yaml index 86e7100..7b2c040 100644 --- a/jobs/defaults.yaml +++ b/jobs/defaults.yaml @@ -9,6 +9,7 @@ if [ "$unamestr" = 'FreeBSD' ]; then MAKE='gmake' fi + export PATH=$VIRT_PREFIX/bin:$PATH env: | smp: 3 spam: ymankad@redhat.com libvirt-ci@redhat.com diff --git a/projects/osinfo-db.yaml b/projects/osinfo-db.yaml index 95ad845..2a9cab2 100644 --- a/projects/osinfo-db.yaml +++ b/projects/osinfo-db.yaml @@ -15,7 +15,6 @@ - generic-build-job: parent_jobs: 'osinfo-db-tools-master-build' command: | - export PATH=$VIRT_PREFIX/bin:$PATH $MAKE -j{smp} all $MAKE install OSINFO_DB_TARGET="--system" - generic-rpm-job: @@ -26,7 +25,6 @@ - libvirt-fedora-26 - libvirt-fedora-rawhide command: | - export PATH=$VIRT_PREFIX/bin:$PATH rm -f *.tar.xz $MAKE osinfo-db.spec perl -i -p -e 's/BuildRequires: osinfo-db-tools.*//' osinfo-db.spec -- 2.14.3
On Fri, Nov 17, 2017 at 07:26:07PM +0100, Andrea Bolognani wrote:
My own attempt at fixing libosinfo-master-rpm's failing status, and then some.
This is an extension of Pavel's attempt[1] in a way; except of course I think my approach is superior, otherwise I wouldn't have bothered ;)
It also sets the groundwork for finally moving $VIRT_PREFIX and friends out of the Jenkins server configuration, where they are invisible to anyone but the CI admins, and into the git repository so that they can be tracked alongside the jobs using them.
I've looked into the jenkins-job-builder documentation and was not able to find a better way so we might use {global_env} for all the guest specific environment variables.
[1] https://www.redhat.com/archives/libvir-list/2017-November/msg00045.html
Andrea Bolognani (5): jobs: Rename {make_env} to {global_env} jobs: Include {global_env} everywhere jobs: Rename {check_env} to {env} jobs: Include {env} everywhere jobs: Include $PATH definition in {global_env}
See my note for PATCH 3. Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
participants (2)
-
Andrea Bolognani -
Pavel Hrdina