7:45 a.m.
Pavel Hrdina (2):
jobs: rename check_env into job_env
jobs: use job_env in all job templates
jobs/autotools.yaml | 5 ++++-
jobs/defaults.yaml | 2 +-
jobs/perl-makemaker.yaml | 3 +++
projects/libosinfo.yaml | 2 +-
projects/libvirt.yaml | 2 +-
5 files changed, 10 insertions(+), 4 deletions(-)
--
2.13.6
7:45 a.m.
New subject: [libvirt] [jenkins-ci PATCH 1/2] jobs: rename check_env into job_env
The check_env was used only for "make check" but there might be some
environment variables that needs to be configured for more than one
job of the same project.
Signed-off-by: Pavel Hrdina <phrdina(a)redhat.com>
---
jobs/autotools.yaml | 2 +-
jobs/defaults.yaml | 2 +-
projects/libosinfo.yaml | 2 +-
projects/libvirt.yaml | 2 +-
4 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/jobs/autotools.yaml b/jobs/autotools.yaml
index 19f62be..9ed5efe 100644
--- a/jobs/autotools.yaml
+++ b/jobs/autotools.yaml
@@ -120,7 +120,7 @@
builders:
- shell: |
{make_env}
- {check_env}
+ {job_env}
cd build
if ! $MAKE -j{smp} check
then
diff --git a/jobs/defaults.yaml b/jobs/defaults.yaml
index 0e628bb..43face2 100644
--- a/jobs/defaults.yaml
+++ b/jobs/defaults.yaml
@@ -9,6 +9,6 @@
if [ "$unamestr" = 'FreeBSD' ]; then
MAKE='gmake'
fi
- check_env: |
+ job_env: |
smp: 3
spam: ymankad(a)redhat.com libvirt-ci(a)redhat.com
diff --git a/projects/libosinfo.yaml b/projects/libosinfo.yaml
index 9ab2281..a02246f 100644
--- a/projects/libosinfo.yaml
+++ b/projects/libosinfo.yaml
@@ -11,7 +11,7 @@
- libvirt-freebsd-10
- libvirt-freebsd-11
title: libosinfo
- check_env: |
+ job_env: |
export OSINFO_SYSTEM_DIR=$VIRT_PREFIX/share/osinfo
jobs:
- autotools-build-job:
diff --git a/projects/libvirt.yaml b/projects/libvirt.yaml
index 5661066..459aa17 100644
--- a/projects/libvirt.yaml
+++ b/projects/libvirt.yaml
@@ -28,7 +28,7 @@
parent_jobs: 'libvirt-master-build'
- autotools-check-job:
parent_jobs: 'libvirt-master-syntax-check'
- check_env: |
+ job_env: |
export VIR_TEST_EXPENSIVE=1
export VIR_TEST_DEBUG=2
- autotools-rpm-job:
--
2.13.6
7:45 a.m.
New subject: [libvirt] [jenkins-ci PATCH 2/2] jobs: use job_env in all job templates
This fixes libosinfo RPM build, we need to set "OSINFO_SYSTEM_DIR" for
RPM build as well since the spec file runs tests and they need valid
osinfo-db. Broken by <359b069f7e4700bee337588eab0a9d83cac94010>.
Signed-off-by: Pavel Hrdina <phrdina(a)redhat.com>
---
jobs/autotools.yaml | 3 +++
jobs/perl-makemaker.yaml | 3 +++
2 files changed, 6 insertions(+)
diff --git a/jobs/autotools.yaml b/jobs/autotools.yaml
index 9ed5efe..e41d5ab 100644
--- a/jobs/autotools.yaml
+++ b/jobs/autotools.yaml
@@ -41,6 +41,7 @@
builders:
- shell: |
{make_env}
+ {job_env}
mkdir build
cd build
../autogen.sh --prefix=$VIRT_PREFIX {autogen_args}
@@ -82,6 +83,7 @@
builders:
- shell: |
{make_env}
+ {job_env}
cd build
$MAKE -j{smp} syntax-check
publishers:
@@ -164,6 +166,7 @@
builders:
- shell: |
{make_env}
+ {job_env}
cd build
sed -i -e 's/BuildRequires: osinfo-db.*//' {name}.spec
sed -i -e 's/BuildRequires: libvirt.*devel.*//' {name}.spec
diff --git a/jobs/perl-makemaker.yaml b/jobs/perl-makemaker.yaml
index 0a3227a..3e53a29 100644
--- a/jobs/perl-makemaker.yaml
+++ b/jobs/perl-makemaker.yaml
@@ -42,6 +42,7 @@
- shell: |
perl Makefile.PL PREFIX="$VIRT_PREFIX"
{make_env}
+ {job_env}
$MAKE
$MAKE -j{smp} install
$MAKE -j{smp} manifest
@@ -83,6 +84,7 @@
builders:
- shell: |
{make_env}
+ {job_env}
$MAKE -j{smp} test {test_args}
publishers:
- email:
@@ -121,6 +123,7 @@
builders:
- shell: |
{make_env}
+ {job_env}
sed -i -e 's/BuildRequires: *perl-Sys-Virt.*//' *.spec
sed -i -e 's/BuildRequires: *libvirt.*devel.*//' *.spec
rm -f *.tar.{archive_format}
--
2.13.6
10:42 a.m.
New subject: [libvirt] [jenkins-ci PATCH 2/2] jobs: use job_env in all job templates
On Thu, Nov 02, 2017 at 01:45:51PM +0100, Pavel Hrdina wrote:
This fixes libosinfo RPM build, we need to set
"OSINFO_SYSTEM_DIR" for
RPM build as well since the spec file runs tests and they need valid
osinfo-db. Broken by <359b069f7e4700bee337588eab0a9d83cac94010>.
I'm not suggesting you fix this right now, but our current RPM build
process is very broken by design, because we are completely ignoring
dependancies between packages. We've hacked around this by culling the
BuildRequires lines from the spec file before running rpmbuild, and
relying on a bit of luck to have the rpmbuild find the stuff we just
built into $VIRT_PREFIX. As this test suite problem shows though,
this is very fragile.
One way to fix this is to create a yum repo populated with all RPMs
we build during CI. Then setup a mock vroot to do the builds in,
which includes this local yum repo. That way we can honour the RPM
deps correctly getting a more accurate test of RPM build process.
My main concern with this is that mock is kind of slow to bootstrap
the initial vroot, so this would slow our CI somewhat.
Signed-off-by: Pavel Hrdina <phrdina(a)redhat.com>
---
jobs/autotools.yaml | 3 +++
jobs/perl-makemaker.yaml | 3 +++
2 files changed, 6 insertions(+)
diff --git a/jobs/autotools.yaml b/jobs/autotools.yaml
index 9ed5efe..e41d5ab 100644
--- a/jobs/autotools.yaml
+++ b/jobs/autotools.yaml
@@ -41,6 +41,7 @@
builders:
- shell: |
{make_env}
+ {job_env}
mkdir build
cd build
../autogen.sh --prefix=$VIRT_PREFIX {autogen_args}
@@ -82,6 +83,7 @@
builders:
- shell: |
{make_env}
+ {job_env}
cd build
$MAKE -j{smp} syntax-check
publishers:
@@ -164,6 +166,7 @@
builders:
- shell: |
{make_env}
+ {job_env}
cd build
sed -i -e 's/BuildRequires: osinfo-db.*//' {name}.spec
sed -i -e 's/BuildRequires: libvirt.*devel.*//' {name}.spec
diff --git a/jobs/perl-makemaker.yaml b/jobs/perl-makemaker.yaml
index 0a3227a..3e53a29 100644
--- a/jobs/perl-makemaker.yaml
+++ b/jobs/perl-makemaker.yaml
@@ -42,6 +42,7 @@
- shell: |
perl Makefile.PL PREFIX="$VIRT_PREFIX"
{make_env}
+ {job_env}
$MAKE
$MAKE -j{smp} install
$MAKE -j{smp} manifest
@@ -83,6 +84,7 @@
builders:
- shell: |
{make_env}
+ {job_env}
$MAKE -j{smp} test {test_args}
publishers:
- email:
@@ -121,6 +123,7 @@
builders:
- shell: |
{make_env}
+ {job_env}
sed -i -e 's/BuildRequires: *perl-Sys-Virt.*//' *.spec
sed -i -e 's/BuildRequires: *libvirt.*devel.*//' *.spec
rm -f *.tar.{archive_format}
--
2.13.6
--
libvir-list mailing list
libvir-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
10:54 a.m.
New subject: [libvirt] [jenkins-ci PATCH 2/2] jobs: use job_env in all job templates
On Thu, Nov 02, 2017 at 03:42:56PM +0000, Daniel P. Berrange wrote:
On Thu, Nov 02, 2017 at 01:45:51PM +0100, Pavel Hrdina wrote:
> This fixes libosinfo RPM build, we need to set "OSINFO_SYSTEM_DIR" for
> RPM build as well since the spec file runs tests and they need valid
> osinfo-db. Broken by <359b069f7e4700bee337588eab0a9d83cac94010>.
I'm not suggesting you fix this right now, but our current RPM build
process is very broken by design, because we are completely ignoring
dependancies between packages. We've hacked around this by culling the
BuildRequires lines from the spec file before running rpmbuild, and
relying on a bit of luck to have the rpmbuild find the stuff we just
built into $VIRT_PREFIX. As this test suite problem shows though,
this is very fragile.
I completely agree with it, it's fragile and it uses all the environment
variables to make sure that all the dependencies can be resolved.
One way to fix this is to create a yum repo populated with all RPMs
we build during CI. Then setup a mock vroot to do the builds in,
which includes this local yum repo. That way we can honour the RPM
deps correctly getting a more accurate test of RPM build process.
My main concern with this is that mock is kind of slow to bootstrap
the initial vroot, so this would slow our CI somewhat.
We need to look into it how much it will slow down our CI since we have
very limited resources. Maybe it's time to ask whether we can have more
resources.
Pavel
10:57 a.m.
New subject: [libvirt] [jenkins-ci PATCH 2/2] jobs: use job_env in all job templates
On Thu, Nov 02, 2017 at 04:54:38PM +0100, Pavel Hrdina wrote:
On Thu, Nov 02, 2017 at 03:42:56PM +0000, Daniel P. Berrange wrote:
> On Thu, Nov 02, 2017 at 01:45:51PM +0100, Pavel Hrdina wrote:
> > This fixes libosinfo RPM build, we need to set "OSINFO_SYSTEM_DIR"
for
> > RPM build as well since the spec file runs tests and they need valid
> > osinfo-db. Broken by <359b069f7e4700bee337588eab0a9d83cac94010>.
>
> I'm not suggesting you fix this right now, but our current RPM build
> process is very broken by design, because we are completely ignoring
> dependancies between packages. We've hacked around this by culling the
> BuildRequires lines from the spec file before running rpmbuild, and
> relying on a bit of luck to have the rpmbuild find the stuff we just
> built into $VIRT_PREFIX. As this test suite problem shows though,
> this is very fragile.
I completely agree with it, it's fragile and it uses all the environment
variables to make sure that all the dependencies can be resolved.
> One way to fix this is to create a yum repo populated with all RPMs
> we build during CI. Then setup a mock vroot to do the builds in,
> which includes this local yum repo. That way we can honour the RPM
> deps correctly getting a more accurate test of RPM build process.
>
> My main concern with this is that mock is kind of slow to bootstrap
> the initial vroot, so this would slow our CI somewhat.
We need to look into it how much it will slow down our CI since we have
very limited resources. Maybe it's time to ask whether we can have more
resources.
I think it is well overdue for us to acquire more hardware for Jenkins
slaves. If CentOS have a 2nd machine they can spare for us then great,
otherwise we should try to push RH to fund some...
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
10:32 a.m.
On Thu, 2017-11-02 at 13:45 +0100, Pavel Hrdina wrote:
Pavel Hrdina (2):
jobs: rename check_env into job_env
jobs: use job_env in all job templates
jobs/autotools.yaml | 5 ++++-
jobs/defaults.yaml | 2 +-
jobs/perl-makemaker.yaml | 3 +++
projects/libosinfo.yaml | 2 +-
projects/libvirt.yaml | 2 +-
5 files changed, 10 insertions(+), 4 deletions(-)
I'm not sure this approach is the best one.
I like the idea of having a place where we can stick common
environment variables - for us that mostly means paths, and we
should definitely move guest configuration out of its definition
in Jenkins and into this repository, as we prototyped yesterday.
But here you're still keeping that info local. Moreover, you're
moving the VIR_TEST_* variables from check_env to job_env, which
means they end up being defined even during regular 'make'. I seem
to recall Dan speaking up against that earlier.
I think we should have:
global_env - as the name implies, global; for $VIRT_PREFIX and all
other paths that affect one or more build jobs, like
$OSINFO_SYSTEM_DIR and $MAKE
build_env - local; for variables that affect the build step of a
single job (can't think of any right now)
test_env - local; for variables that affect the test suite of a
single job, like $VIR_TEST_DEBUG
For projects that inherit from anything but generic-*-job,
global_env will be included automatically; projects that don't use
any of the known build system will have to take care of that
themselves, but then again they already have to do that for make_env.
Moreover, we should be able to convert libvirt-cim to use
autotools-*-job by tweaking its autogen.sh script, which would leave
osinfo-db as the only user of generic-*-job.
--
Andrea Bolognani / Red Hat / Virtualization
10:49 a.m.
On Thu, Nov 02, 2017 at 04:32:12PM +0100, Andrea Bolognani wrote:
On Thu, 2017-11-02 at 13:45 +0100, Pavel Hrdina wrote:
> Pavel Hrdina (2):
> jobs: rename check_env into job_env
> jobs: use job_env in all job templates
>
> jobs/autotools.yaml | 5 ++++-
> jobs/defaults.yaml | 2 +-
> jobs/perl-makemaker.yaml | 3 +++
> projects/libosinfo.yaml | 2 +-
> projects/libvirt.yaml | 2 +-
> 5 files changed, 10 insertions(+), 4 deletions(-)
I'm not sure this approach is the best one.
I like the idea of having a place where we can stick common
environment variables - for us that mostly means paths, and we
should definitely move guest configuration out of its definition
in Jenkins and into this repository, as we prototyped yesterday.
This is a fix of the commit that broke it, I'm not trying to move all
environment variables out of Jenkins.
But here you're still keeping that info local. Moreover,
you're
moving the VIR_TEST_* variables from check_env to job_env, which
means they end up being defined even during regular 'make'. I seem
to recall Dan speaking up against that earlier.
This is not true, the VIR_TEST_* variables are defined only for
"autotools-check-job", see <projects/libvirt.yaml>.
I think we should have:
global_env - as the name implies, global; for $VIRT_PREFIX and all
other paths that affect one or more build jobs, like
$OSINFO_SYSTEM_DIR and $MAKE
I'm not sure whether there is some different way how to do it but I
would rather use different approach than using these defines since
you can easily overwrite them and forget to include it.
build_env - local; for variables that affect the build step of a
single job (can't think of any right now)
test_env - local; for variables that affect the test suite of a
single job, like $VIR_TEST_DEBUG
I thought about splitting it into {build,test} but we don't need to have
it that way and it feels like an overkill.
For projects that inherit from anything but generic-*-job,
global_env will be included automatically; projects that don't use
any of the known build system will have to take care of that
themselves, but then again they already have to do that for make_env.
And this is exactly why I don't want to do it this way, the "global_env"
needs to be present always, like it is now with all the environment
variables configured in Jenkins.
Moreover, we should be able to convert libvirt-cim to use
autotools-*-job by tweaking its autogen.sh script, which would leave
osinfo-db as the only user of generic-*-job.
+1
Pavel
10:56 a.m.
On Thu, Nov 02, 2017 at 04:49:38PM +0100, Pavel Hrdina wrote:
On Thu, Nov 02, 2017 at 04:32:12PM +0100, Andrea Bolognani wrote:
> Moreover, we should be able to convert libvirt-cim to use
> autotools-*-job by tweaking its autogen.sh script, which would leave
> osinfo-db as the only user of generic-*-job.
+1
The only reason libvirt-cim didn't use the autotools job was that its
setup script was called autoconfiscate insteadof autogen.sh. I see that
Andrea has fixed that crazyness now so we can use the autotools job
easily enough i expect :-)
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
11:21 a.m.
On Thu, 2017-11-02 at 15:56 +0000, Daniel P. Berrange wrote:
On Thu, Nov 02, 2017 at 04:49:38PM +0100, Pavel Hrdina wrote:
> On Thu, Nov 02, 2017 at 04:32:12PM +0100, Andrea Bolognani wrote:
> > Moreover, we should be able to convert libvirt-cim to use
> > autotools-*-job by tweaking its autogen.sh script, which would leave
> > osinfo-db as the only user of generic-*-job.
>
> +1
The only reason libvirt-cim didn't use the autotools job was that its
setup script was called autoconfiscate insteadof autogen.sh. I see that
Andrea has fixed that crazyness now so we can use the autotools job
easily enough i expect :-)
There's one more difference still: libvirt-cim's autogen.sh doesn't
automatically call configure, which autotools-*-job expect. Easily
fixed, though.
--
Andrea Bolognani / Red Hat / Virtualization
11:33 a.m.
On Thu, 2017-11-02 at 16:49 +0100, Pavel Hrdina wrote:
On Thu, Nov 02, 2017 at 04:32:12PM +0100, Andrea Bolognani wrote:
> I'm not sure this approach is the best one.
>
> I like the idea of having a place where we can stick common
> environment variables - for us that mostly means paths, and we
> should definitely move guest configuration out of its definition
> in Jenkins and into this repository, as we prototyped yesterday.
This is a fix of the commit that broke it, I'm not trying to move all
environment variables out of Jenkins.
I understand that, and I didn't expect you to of course, but I was
considering your changes with that frame of reference and it looked
to me like you were changing some things that would have to be
changed again for that to happen, so I thought it would be better
to go in that direction right away.
> But here you're still keeping that info local. Moreover,
you're
> moving the VIR_TEST_* variables from check_env to job_env, which
> means they end up being defined even during regular 'make'. I seem
> to recall Dan speaking up against that earlier.
This is not true, the VIR_TEST_* variables are defined only for
"autotools-check-job", see <projects/libvirt.yaml>.
You're right, sorry for the noise.
> I think we should have:
>
> global_env - as the name implies, global; for $VIRT_PREFIX and all
> other paths that affect one or more build jobs, like
> $OSINFO_SYSTEM_DIR and $MAKE
I'm not sure whether there is some different way how to do it but I
would rather use different approach than using these defines since
you can easily overwrite them and forget to include it.
See below.
> build_env - local; for variables that affect the build step
of a
> single job (can't think of any right now)
> test_env - local; for variables that affect the test suite of a
> single job, like $VIR_TEST_DEBUG
I thought about splitting it into {build,test} but we don't need to have
it that way and it feels like an overkill.
After looking into it better, I agree that splitting them doesn't
make much sense at all. How about local_env instead of job_env, to
complement global_env defined above? Assuming we don't discard my
proposal entirely, that is :)
> For projects that inherit from anything but generic-*-job,
> global_env will be included automatically; projects that don't use
> any of the known build system will have to take care of that
> themselves, but then again they already have to do that for make_env.
And this is exactly why I don't want to do it this way, the "global_env"
needs to be present always, like it is now with all the environment
variables configured in Jenkins.
Is there any way we can have that guarantee without asking users
of generic-*-job (which shouldn't be many at all, ideally just the
one after we've fixed libvirt-cim) to include it themselves?
I'm really not an expert in jenkins-job-builder but since we're
setting the environment using export in the 'command' section of
the job definition, I don't see much of a way around it.
So while I agree with your concern about forgetting to include
global_env when defining new jobs, I think that it would still be
a better approach than keeping the environment variables in the
per-worker Jenkins configuration, where they are not tracked and
basically entirely invisible to anyone but the CI administrators.
--
Andrea Bolognani / Red Hat / Virtualization
2607
days inactive
2607
days old
10 comments
3 participants
participants (3)
-
Andrea Bolognani -
Daniel P. Berrange -
Pavel Hrdina