[libvirt] [jenkins-ci PATCH 0/5] guests: Make Perl and Python paths dynamic
Plus other cleanups. See patches 3/5 and 4/5 for the good stuff. Andrea Bolognani (5): guests: Add .j2 extension to existing templates guests: Remove distinction between files and templates guests: Make PERL5LIB dynamic guests: Make PYTHONPATH dynamic guests: Move remaining paths out of the inventory guests/group_vars/all/main.yml | 13 -------- guests/host_vars/libvirt-centos-6/main.yml | 3 -- guests/host_vars/libvirt-centos-7/main.yml | 3 -- guests/host_vars/libvirt-debian-8/main.yml | 3 -- guests/host_vars/libvirt-debian-9/main.yml | 3 -- guests/host_vars/libvirt-debian-sid/main.yml | 3 -- guests/host_vars/libvirt-fedora-27/main.yml | 3 -- guests/host_vars/libvirt-fedora-28/main.yml | 3 -- .../host_vars/libvirt-fedora-rawhide/main.yml | 3 -- guests/host_vars/libvirt-freebsd-10/main.yml | 3 -- guests/host_vars/libvirt-freebsd-11/main.yml | 3 -- .../libvirt-freebsd-current/main.yml | 3 -- guests/host_vars/libvirt-ubuntu-14/main.yml | 3 -- guests/host_vars/libvirt-ubuntu-16/main.yml | 3 -- guests/host_vars/libvirt-ubuntu-18/main.yml | 3 -- guests/tasks/base.yml | 12 +++---- guests/tasks/users.yml | 6 ++-- .../{bash_profile => bash_profile.j2} | 0 guests/templates/bashrc | 20 ------------ guests/templates/bashrc.j2 | 32 +++++++++++++++++++ .../ccache.conf => templates/ccache.conf.j2} | 0 .../fedora-rawhide-kernel-nodebug.repo.j2} | 0 .../jessie-backports.preferences.j2} | 0 .../jessie-backports.sources.j2} | 0 24 files changed, 41 insertions(+), 84 deletions(-) rename guests/templates/{bash_profile => bash_profile.j2} (100%) delete mode 100644 guests/templates/bashrc create mode 100644 guests/templates/bashrc.j2 rename guests/{files/ccache.conf => templates/ccache.conf.j2} (100%) rename guests/{files/fedora-rawhide-kernel-nodebug.repo => templates/fedora-rawhide-kernel-nodebug.repo.j2} (100%) rename guests/{files/jessie-backports.preferences => templates/jessie-backports.preferences.j2} (100%) rename guests/{files/jessie-backports.sources => templates/jessie-backports.sources.j2} (100%) -- 2.17.0
By convention, Jinja2 templates should use the .j2 extension. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/tasks/users.yml | 2 +- guests/templates/{bash_profile => bash_profile.j2} | 0 guests/templates/{bashrc => bashrc.j2} | 0 3 files changed, 1 insertion(+), 1 deletion(-) rename guests/templates/{bash_profile => bash_profile.j2} (100%) rename guests/templates/{bashrc => bashrc.j2} (100%) diff --git a/guests/tasks/users.yml b/guests/tasks/users.yml index 033ca2a..7b37842 100644 --- a/guests/tasks/users.yml +++ b/guests/tasks/users.yml @@ -98,7 +98,7 @@ - name: '{{ flavor }}: Create shell profile' template: - src: templates/{{ item }} + src: templates/{{ item }}.j2 dest: /home/{{ flavor }}/.{{ item }} owner: '{{ flavor }}' group: '{{ flavor }}' diff --git a/guests/templates/bash_profile b/guests/templates/bash_profile.j2 similarity index 100% rename from guests/templates/bash_profile rename to guests/templates/bash_profile.j2 diff --git a/guests/templates/bashrc b/guests/templates/bashrc.j2 similarity index 100% rename from guests/templates/bashrc rename to guests/templates/bashrc.j2 -- 2.17.0
On Mon, May 21, 2018 at 06:53:23PM +0200, Andrea Bolognani wrote:
By convention, Jinja2 templates should use the .j2 extension.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/tasks/users.yml | 2 +- guests/templates/{bash_profile => bash_profile.j2} | 0 guests/templates/{bashrc => bashrc.j2} | 0 3 files changed, 1 insertion(+), 1 deletion(-) rename guests/templates/{bash_profile => bash_profile.j2} (100%) rename guests/templates/{bashrc => bashrc.j2} (100%)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
The main difference between files and templates is that the former can can be uploaded as-is, while the latter has to go through Jinja2 first. Since the overhead of template processing is negligible, and there's a fair chance we will need to start using templating for files that currently don't need it or viceversa, let's get rid of the distinction altogether. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/tasks/base.yml | 12 ++++++------ guests/tasks/users.yml | 4 ++-- .../{files/ccache.conf => templates/ccache.conf.j2} | 0 .../fedora-rawhide-kernel-nodebug.repo.j2} | 0 .../jessie-backports.preferences.j2} | 0 .../jessie-backports.sources.j2} | 0 6 files changed, 8 insertions(+), 8 deletions(-) rename guests/{files/ccache.conf => templates/ccache.conf.j2} (100%) rename guests/{files/fedora-rawhide-kernel-nodebug.repo => templates/fedora-rawhide-kernel-nodebug.repo.j2} (100%) rename guests/{files/jessie-backports.preferences => templates/jessie-backports.preferences.j2} (100%) rename guests/{files/jessie-backports.sources => templates/jessie-backports.sources.j2} (100%) diff --git a/guests/tasks/base.yml b/guests/tasks/base.yml index 70d5abc..aac5d4f 100644 --- a/guests/tasks/base.yml +++ b/guests/tasks/base.yml @@ -67,8 +67,8 @@ os_name == 'Fedora' ) - name: Enable jessie-backports repository - copy: - src: files/jessie-backports.sources + template: + src: templates/jessie-backports.sources.j2 dest: /etc/apt/sources.list.d/jessie-backports.list owner: root group: root @@ -78,8 +78,8 @@ - flavor == 'jenkins' - name: Configure APT pinning for jessie-backports - copy: - src: files/jessie-backports.preferences + template: + src: templates/jessie-backports.preferences.j2 dest: /etc/apt/preferences.d/jessie-backports owner: root group: root @@ -89,8 +89,8 @@ - flavor == 'jenkins' - name: Enable fedora-rawhide-kernel-nodebug repository - copy: - src: files/fedora-rawhide-kernel-nodebug.repo + template: + src: templates/fedora-rawhide-kernel-nodebug.repo.j2 dest: /etc/yum.repos.d/fedora-rawhide-kernel-nodebug.repo owner: root group: root diff --git a/guests/tasks/users.yml b/guests/tasks/users.yml index 7b37842..9c5c34d 100644 --- a/guests/tasks/users.yml +++ b/guests/tasks/users.yml @@ -62,8 +62,8 @@ - ccache/bin - name: '{{ flavor }}: Configure ccache' - copy: - src: files/ccache.conf + template: + src: templates/ccache.conf.j2 dest: /home/{{ flavor }}/.ccache/ccache.conf owner: '{{ flavor }}' group: '{{ flavor }}' diff --git a/guests/files/ccache.conf b/guests/templates/ccache.conf.j2 similarity index 100% rename from guests/files/ccache.conf rename to guests/templates/ccache.conf.j2 diff --git a/guests/files/fedora-rawhide-kernel-nodebug.repo b/guests/templates/fedora-rawhide-kernel-nodebug.repo.j2 similarity index 100% rename from guests/files/fedora-rawhide-kernel-nodebug.repo rename to guests/templates/fedora-rawhide-kernel-nodebug.repo.j2 diff --git a/guests/files/jessie-backports.preferences b/guests/templates/jessie-backports.preferences.j2 similarity index 100% rename from guests/files/jessie-backports.preferences rename to guests/templates/jessie-backports.preferences.j2 diff --git a/guests/files/jessie-backports.sources b/guests/templates/jessie-backports.sources.j2 similarity index 100% rename from guests/files/jessie-backports.sources rename to guests/templates/jessie-backports.sources.j2 -- 2.17.0
On Mon, May 21, 2018 at 06:53:24PM +0200, Andrea Bolognani wrote:
The main difference between files and templates is that the former can can be uploaded as-is, while the latter has to go through Jinja2 first.
Since the overhead of template processing is negligible, and there's a fair chance we will need to start using templating for files that currently don't need it or viceversa, let's get rid of the distinction altogether.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/tasks/base.yml | 12 ++++++------ guests/tasks/users.yml | 4 ++-- .../{files/ccache.conf => templates/ccache.conf.j2} | 0 .../fedora-rawhide-kernel-nodebug.repo.j2} | 0 .../jessie-backports.preferences.j2} | 0 .../jessie-backports.sources.j2} | 0 6 files changed, 8 insertions(+), 8 deletions(-) rename guests/{files/ccache.conf => templates/ccache.conf.j2} (100%) rename guests/{files/fedora-rawhide-kernel-nodebug.repo => templates/fedora-rawhide-kernel-nodebug.repo.j2} (100%) rename guests/{files/jessie-backports.preferences => templates/jessie-backports.preferences.j2} (100%) rename guests/{files/jessie-backports.sources => templates/jessie-backports.sources.j2} (100%)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Instead of having to specify this separately for every guest due to minor differences in the environment, figure it out at login time by asking Perl itself for the information we need. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/group_vars/all/main.yml | 1 - guests/host_vars/libvirt-centos-6/main.yml | 1 - guests/host_vars/libvirt-centos-7/main.yml | 1 - guests/host_vars/libvirt-debian-8/main.yml | 1 - guests/host_vars/libvirt-debian-9/main.yml | 1 - guests/host_vars/libvirt-debian-sid/main.yml | 1 - guests/host_vars/libvirt-fedora-27/main.yml | 1 - guests/host_vars/libvirt-fedora-28/main.yml | 1 - guests/host_vars/libvirt-fedora-rawhide/main.yml | 1 - guests/host_vars/libvirt-freebsd-10/main.yml | 1 - guests/host_vars/libvirt-freebsd-11/main.yml | 1 - guests/host_vars/libvirt-freebsd-current/main.yml | 1 - guests/host_vars/libvirt-ubuntu-14/main.yml | 1 - guests/host_vars/libvirt-ubuntu-16/main.yml | 1 - guests/host_vars/libvirt-ubuntu-18/main.yml | 1 - guests/templates/bashrc.j2 | 6 +++++- 16 files changed, 5 insertions(+), 16 deletions(-) diff --git a/guests/group_vars/all/main.yml b/guests/group_vars/all/main.yml index cd8ef25..840c80b 100644 --- a/guests/group_vars/all/main.yml +++ b/guests/group_vars/all/main.yml @@ -16,5 +16,4 @@ PKG_CONFIG_PATH: $VIRT_PREFIX/lib/pkgconfig XDG_DATA_DIRS: $VIRT_PREFIX/share:/usr/share:/usr/local/share GI_TYPELIB_PATH: $VIRT_PREFIX/lib/girepository-1.0 OSINFO_SYSTEM_DIR: $VIRT_PREFIX/share/osinfo -PERL5LIB: PYTHONPATH: diff --git a/guests/host_vars/libvirt-centos-6/main.yml b/guests/host_vars/libvirt-centos-6/main.yml index 8b18c19..c27c1ad 100644 --- a/guests/host_vars/libvirt-centos-6/main.yml +++ b/guests/host_vars/libvirt-centos-6/main.yml @@ -1,5 +1,4 @@ --- -PERL5LIB: $VIRT_PREFIX/lib/perl5/x86_64-linux-thread-multi PYTHONPATH: $VIRT_PREFIX/lib64/python2.6/site-packages projects: diff --git a/guests/host_vars/libvirt-centos-7/main.yml b/guests/host_vars/libvirt-centos-7/main.yml index 8f88615..6cb8995 100644 --- a/guests/host_vars/libvirt-centos-7/main.yml +++ b/guests/host_vars/libvirt-centos-7/main.yml @@ -1,5 +1,4 @@ --- -PERL5LIB: $VIRT_PREFIX/lib/perl5/x86_64-linux-thread-multi PYTHONPATH: $VIRT_PREFIX/lib64/python2.7/site-packages projects: diff --git a/guests/host_vars/libvirt-debian-8/main.yml b/guests/host_vars/libvirt-debian-8/main.yml index d0b3601..64f3255 100644 --- a/guests/host_vars/libvirt-debian-8/main.yml +++ b/guests/host_vars/libvirt-debian-8/main.yml @@ -1,5 +1,4 @@ --- -PERL5LIB: $VIRT_PREFIX/lib/perl5/x86_64-linux-gnu-thread-multi PYTHONPATH: $VIRT_PREFIX/lib/python3.4/site-packages projects: diff --git a/guests/host_vars/libvirt-debian-9/main.yml b/guests/host_vars/libvirt-debian-9/main.yml index caf513e..04d5132 100644 --- a/guests/host_vars/libvirt-debian-9/main.yml +++ b/guests/host_vars/libvirt-debian-9/main.yml @@ -1,5 +1,4 @@ --- -PERL5LIB: $VIRT_PREFIX/lib/perl5/x86_64-linux-gnu-thread-multi PYTHONPATH: $VIRT_PREFIX/lib/python3.5/site-packages projects: diff --git a/guests/host_vars/libvirt-debian-sid/main.yml b/guests/host_vars/libvirt-debian-sid/main.yml index 88b8b9b..cbf57aa 100644 --- a/guests/host_vars/libvirt-debian-sid/main.yml +++ b/guests/host_vars/libvirt-debian-sid/main.yml @@ -1,5 +1,4 @@ --- -PERL5LIB: $VIRT_PREFIX/lib/perl5/x86_64-linux-gnu-thread-multi PYTHONPATH: $VIRT_PREFIX/lib/python3.6/site-packages projects: diff --git a/guests/host_vars/libvirt-fedora-27/main.yml b/guests/host_vars/libvirt-fedora-27/main.yml index 20d90c5..9ed392b 100644 --- a/guests/host_vars/libvirt-fedora-27/main.yml +++ b/guests/host_vars/libvirt-fedora-27/main.yml @@ -1,5 +1,4 @@ --- -PERL5LIB: $VIRT_PREFIX/lib/perl5/x86_64-linux-thread-multi PYTHONPATH: $VIRT_PREFIX/lib64/python3.6/site-packages projects: diff --git a/guests/host_vars/libvirt-fedora-28/main.yml b/guests/host_vars/libvirt-fedora-28/main.yml index 20d90c5..9ed392b 100644 --- a/guests/host_vars/libvirt-fedora-28/main.yml +++ b/guests/host_vars/libvirt-fedora-28/main.yml @@ -1,5 +1,4 @@ --- -PERL5LIB: $VIRT_PREFIX/lib/perl5/x86_64-linux-thread-multi PYTHONPATH: $VIRT_PREFIX/lib64/python3.6/site-packages projects: diff --git a/guests/host_vars/libvirt-fedora-rawhide/main.yml b/guests/host_vars/libvirt-fedora-rawhide/main.yml index 202eccf..91b4158 100644 --- a/guests/host_vars/libvirt-fedora-rawhide/main.yml +++ b/guests/host_vars/libvirt-fedora-rawhide/main.yml @@ -1,5 +1,4 @@ --- -PERL5LIB: $VIRT_PREFIX/lib/perl5/x86_64-linux-thread-multi PYTHONPATH: $VIRT_PREFIX/lib64/python3.6/site-packages projects: diff --git a/guests/host_vars/libvirt-freebsd-10/main.yml b/guests/host_vars/libvirt-freebsd-10/main.yml index ac5f12b..e37c845 100644 --- a/guests/host_vars/libvirt-freebsd-10/main.yml +++ b/guests/host_vars/libvirt-freebsd-10/main.yml @@ -1,7 +1,6 @@ --- ansible_python_interpreter: /usr/local/bin/python2 -PERL5LIB: $VIRT_PREFIX/lib/perl5/amd64-freebsd-thread-multi PYTHONPATH: $VIRT_PREFIX/lib/python3.6/site-packages projects: diff --git a/guests/host_vars/libvirt-freebsd-11/main.yml b/guests/host_vars/libvirt-freebsd-11/main.yml index ac5f12b..e37c845 100644 --- a/guests/host_vars/libvirt-freebsd-11/main.yml +++ b/guests/host_vars/libvirt-freebsd-11/main.yml @@ -1,7 +1,6 @@ --- ansible_python_interpreter: /usr/local/bin/python2 -PERL5LIB: $VIRT_PREFIX/lib/perl5/amd64-freebsd-thread-multi PYTHONPATH: $VIRT_PREFIX/lib/python3.6/site-packages projects: diff --git a/guests/host_vars/libvirt-freebsd-current/main.yml b/guests/host_vars/libvirt-freebsd-current/main.yml index ac5f12b..e37c845 100644 --- a/guests/host_vars/libvirt-freebsd-current/main.yml +++ b/guests/host_vars/libvirt-freebsd-current/main.yml @@ -1,7 +1,6 @@ --- ansible_python_interpreter: /usr/local/bin/python2 -PERL5LIB: $VIRT_PREFIX/lib/perl5/amd64-freebsd-thread-multi PYTHONPATH: $VIRT_PREFIX/lib/python3.6/site-packages projects: diff --git a/guests/host_vars/libvirt-ubuntu-14/main.yml b/guests/host_vars/libvirt-ubuntu-14/main.yml index fccd0ef..bfbdda6 100644 --- a/guests/host_vars/libvirt-ubuntu-14/main.yml +++ b/guests/host_vars/libvirt-ubuntu-14/main.yml @@ -1,5 +1,4 @@ --- -PERL5LIB: $VIRT_PREFIX/lib/perl5/x86_64-linux-gnu-thread-multi PYTHONPATH: $VIRT_PREFIX/lib/python3.4/site-packages projects: diff --git a/guests/host_vars/libvirt-ubuntu-16/main.yml b/guests/host_vars/libvirt-ubuntu-16/main.yml index caf513e..04d5132 100644 --- a/guests/host_vars/libvirt-ubuntu-16/main.yml +++ b/guests/host_vars/libvirt-ubuntu-16/main.yml @@ -1,5 +1,4 @@ --- -PERL5LIB: $VIRT_PREFIX/lib/perl5/x86_64-linux-gnu-thread-multi PYTHONPATH: $VIRT_PREFIX/lib/python3.5/site-packages projects: diff --git a/guests/host_vars/libvirt-ubuntu-18/main.yml b/guests/host_vars/libvirt-ubuntu-18/main.yml index 88b8b9b..cbf57aa 100644 --- a/guests/host_vars/libvirt-ubuntu-18/main.yml +++ b/guests/host_vars/libvirt-ubuntu-18/main.yml @@ -1,5 +1,4 @@ --- -PERL5LIB: $VIRT_PREFIX/lib/perl5/x86_64-linux-gnu-thread-multi PYTHONPATH: $VIRT_PREFIX/lib/python3.6/site-packages projects: diff --git a/guests/templates/bashrc.j2 b/guests/templates/bashrc.j2 index 866087e..cf2679e 100644 --- a/guests/templates/bashrc.j2 +++ b/guests/templates/bashrc.j2 @@ -8,12 +8,16 @@ export PKG_CONFIG_PATH="{{ PKG_CONFIG_PATH }}" export XDG_DATA_DIRS="{{ XDG_DATA_DIRS }}" export GI_TYPELIB_PATH="{{ GI_TYPELIB_PATH }}" export OSINFO_SYSTEM_DIR="{{ OSINFO_SYSTEM_DIR }}" -export PERL5LIB="{{ PERL5LIB }}" export PYTHONPATH="{{ PYTHONPATH }}" export MAKE="{{ make }}" export PYTHON="{{ python }}" +# We need to ask Perl for this information, since it's used to +# construct installation paths +plarch=$(perl -e 'use Config; print $Config{archname}') +export PERL5LIB="$VIRT_PREFIX/lib/perl5/$plarch" + # Enable bash completion. Only needed on FreeBSD, the system-wide # shell profile will take care of it for us everywhere else bash_completion=/usr/local/share/bash-completion/bash_completion.sh -- 2.17.0
On Mon, May 21, 2018 at 06:53:25PM +0200, Andrea Bolognani wrote:
Instead of having to specify this separately for every guest due to minor differences in the environment, figure it out at login time by asking Perl itself for the information we need.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/group_vars/all/main.yml | 1 - guests/host_vars/libvirt-centos-6/main.yml | 1 - guests/host_vars/libvirt-centos-7/main.yml | 1 - guests/host_vars/libvirt-debian-8/main.yml | 1 - guests/host_vars/libvirt-debian-9/main.yml | 1 - guests/host_vars/libvirt-debian-sid/main.yml | 1 - guests/host_vars/libvirt-fedora-27/main.yml | 1 - guests/host_vars/libvirt-fedora-28/main.yml | 1 - guests/host_vars/libvirt-fedora-rawhide/main.yml | 1 - guests/host_vars/libvirt-freebsd-10/main.yml | 1 - guests/host_vars/libvirt-freebsd-11/main.yml | 1 - guests/host_vars/libvirt-freebsd-current/main.yml | 1 - guests/host_vars/libvirt-ubuntu-14/main.yml | 1 - guests/host_vars/libvirt-ubuntu-16/main.yml | 1 - guests/host_vars/libvirt-ubuntu-18/main.yml | 1 - guests/templates/bashrc.j2 | 6 +++++- 16 files changed, 5 insertions(+), 16 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, May 21, 2018 at 06:53:25PM +0200, Andrea Bolognani wrote:
Instead of having to specify this separately for every guest due to minor differences in the environment, figure it out at login time by asking Perl itself for the information we need.
We still have the problem that any time a package update is done it might pull in a newer Perl, which will change the path, but our Jenkins slave never picks it up until the VM is rebooted. I wonder if we can solve this by instead of putting all these env vars in the main bash profile, we create a 'jenkins-ci.sh' that contains all the env variables. We can then source that from the bash profile, and also source it as the first thing we do in every build job too. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, 2018-06-05 at 11:42 +0100, Daniel P. Berrangé wrote:
On Mon, May 21, 2018 at 06:53:25PM +0200, Andrea Bolognani wrote:
Instead of having to specify this separately for every guest due to minor differences in the environment, figure it out at login time by asking Perl itself for the information we need.
We still have the problem that any time a package update is done it might pull in a newer Perl, which will change the path, but our Jenkins slave never picks it up until the VM is rebooted.
Is that the case, though? I'm not entirely clear on the lifecycle, but it might very well be the case that the Jenkins agent spawns a separate session for each job, in which case changes to the shell profile would be picked up immediately. We can perhaps leave it as is for now, and see whether it breaks at some point down the line.
I wonder if we can solve this by instead of putting all these env vars in the main bash profile, we create a
'jenkins-ci.sh'
that contains all the env variables. We can then source that from the bash profile, and also source it as the first thing we do in every build job too.
If we really need to do that, might as well source ~/.bashrc right at the start of every build job instead of having a separate script: the only unnecessary bits are setting up the prompt and bash completion, but there's no harm in having them in there. -- Andrea Bolognani / Red Hat / Virtualization
On Tue, Jun 05, 2018 at 01:10:45PM +0200, Andrea Bolognani wrote:
On Tue, 2018-06-05 at 11:42 +0100, Daniel P. Berrangé wrote:
On Mon, May 21, 2018 at 06:53:25PM +0200, Andrea Bolognani wrote:
Instead of having to specify this separately for every guest due to minor differences in the environment, figure it out at login time by asking Perl itself for the information we need.
We still have the problem that any time a package update is done it might pull in a newer Perl, which will change the path, but our Jenkins slave never picks it up until the VM is rebooted.
Is that the case, though? I'm not entirely clear on the lifecycle, but it might very well be the case that the Jenkins agent spawns a separate session for each job, in which case changes to the shell profile would be picked up immediately.
Since we move env variables out of jenkins host config and into the ansible host config, any time we've made env variable changes we've had to restart the agent or host to pick it up.
I wonder if we can solve this by instead of putting all these env vars in the main bash profile, we create a
'jenkins-ci.sh'
that contains all the env variables. We can then source that from the bash profile, and also source it as the first thing we do in every build job too.
If we really need to do that, might as well source ~/.bashrc right at the start of every build job instead of having a separate script: the only unnecessary bits are setting up the prompt and bash completion, but there's no harm in having them in there.
Oh true, yes, we can do that. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Instead of having to specify this separately for every guest due to minor differences in the environment, figure it out at login time by asking Python itself for the information we need. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/group_vars/all/main.yml | 1 - guests/host_vars/libvirt-centos-6/main.yml | 2 -- guests/host_vars/libvirt-centos-7/main.yml | 2 -- guests/host_vars/libvirt-debian-8/main.yml | 2 -- guests/host_vars/libvirt-debian-9/main.yml | 2 -- guests/host_vars/libvirt-debian-sid/main.yml | 2 -- guests/host_vars/libvirt-fedora-27/main.yml | 2 -- guests/host_vars/libvirt-fedora-28/main.yml | 2 -- guests/host_vars/libvirt-fedora-rawhide/main.yml | 2 -- guests/host_vars/libvirt-freebsd-10/main.yml | 2 -- guests/host_vars/libvirt-freebsd-11/main.yml | 2 -- guests/host_vars/libvirt-freebsd-current/main.yml | 2 -- guests/host_vars/libvirt-ubuntu-14/main.yml | 2 -- guests/host_vars/libvirt-ubuntu-16/main.yml | 2 -- guests/host_vars/libvirt-ubuntu-18/main.yml | 2 -- guests/templates/bashrc.j2 | 10 +++++++++- 16 files changed, 9 insertions(+), 30 deletions(-) diff --git a/guests/group_vars/all/main.yml b/guests/group_vars/all/main.yml index 840c80b..2641c33 100644 --- a/guests/group_vars/all/main.yml +++ b/guests/group_vars/all/main.yml @@ -16,4 +16,3 @@ PKG_CONFIG_PATH: $VIRT_PREFIX/lib/pkgconfig XDG_DATA_DIRS: $VIRT_PREFIX/share:/usr/share:/usr/local/share GI_TYPELIB_PATH: $VIRT_PREFIX/lib/girepository-1.0 OSINFO_SYSTEM_DIR: $VIRT_PREFIX/share/osinfo -PYTHONPATH: diff --git a/guests/host_vars/libvirt-centos-6/main.yml b/guests/host_vars/libvirt-centos-6/main.yml index c27c1ad..265c622 100644 --- a/guests/host_vars/libvirt-centos-6/main.yml +++ b/guests/host_vars/libvirt-centos-6/main.yml @@ -1,5 +1,3 @@ --- -PYTHONPATH: $VIRT_PREFIX/lib64/python2.6/site-packages - projects: - libvirt+website diff --git a/guests/host_vars/libvirt-centos-7/main.yml b/guests/host_vars/libvirt-centos-7/main.yml index 6cb8995..c147830 100644 --- a/guests/host_vars/libvirt-centos-7/main.yml +++ b/guests/host_vars/libvirt-centos-7/main.yml @@ -1,6 +1,4 @@ --- -PYTHONPATH: $VIRT_PREFIX/lib64/python2.7/site-packages - projects: - libosinfo - libvirt diff --git a/guests/host_vars/libvirt-debian-8/main.yml b/guests/host_vars/libvirt-debian-8/main.yml index 64f3255..954a037 100644 --- a/guests/host_vars/libvirt-debian-8/main.yml +++ b/guests/host_vars/libvirt-debian-8/main.yml @@ -1,6 +1,4 @@ --- -PYTHONPATH: $VIRT_PREFIX/lib/python3.4/site-packages - projects: - libosinfo - libvirt diff --git a/guests/host_vars/libvirt-debian-9/main.yml b/guests/host_vars/libvirt-debian-9/main.yml index 04d5132..a588c09 100644 --- a/guests/host_vars/libvirt-debian-9/main.yml +++ b/guests/host_vars/libvirt-debian-9/main.yml @@ -1,6 +1,4 @@ --- -PYTHONPATH: $VIRT_PREFIX/lib/python3.5/site-packages - projects: - libosinfo - libvirt diff --git a/guests/host_vars/libvirt-debian-sid/main.yml b/guests/host_vars/libvirt-debian-sid/main.yml index cbf57aa..a588c09 100644 --- a/guests/host_vars/libvirt-debian-sid/main.yml +++ b/guests/host_vars/libvirt-debian-sid/main.yml @@ -1,6 +1,4 @@ --- -PYTHONPATH: $VIRT_PREFIX/lib/python3.6/site-packages - projects: - libosinfo - libvirt diff --git a/guests/host_vars/libvirt-fedora-27/main.yml b/guests/host_vars/libvirt-fedora-27/main.yml index 9ed392b..00b0848 100644 --- a/guests/host_vars/libvirt-fedora-27/main.yml +++ b/guests/host_vars/libvirt-fedora-27/main.yml @@ -1,6 +1,4 @@ --- -PYTHONPATH: $VIRT_PREFIX/lib64/python3.6/site-packages - projects: - libosinfo - libvirt diff --git a/guests/host_vars/libvirt-fedora-28/main.yml b/guests/host_vars/libvirt-fedora-28/main.yml index 9ed392b..00b0848 100644 --- a/guests/host_vars/libvirt-fedora-28/main.yml +++ b/guests/host_vars/libvirt-fedora-28/main.yml @@ -1,6 +1,4 @@ --- -PYTHONPATH: $VIRT_PREFIX/lib64/python3.6/site-packages - projects: - libosinfo - libvirt diff --git a/guests/host_vars/libvirt-fedora-rawhide/main.yml b/guests/host_vars/libvirt-fedora-rawhide/main.yml index 91b4158..5044594 100644 --- a/guests/host_vars/libvirt-fedora-rawhide/main.yml +++ b/guests/host_vars/libvirt-fedora-rawhide/main.yml @@ -1,6 +1,4 @@ --- -PYTHONPATH: $VIRT_PREFIX/lib64/python3.6/site-packages - projects: - libosinfo - libosinfo+mingw diff --git a/guests/host_vars/libvirt-freebsd-10/main.yml b/guests/host_vars/libvirt-freebsd-10/main.yml index e37c845..2ad4584 100644 --- a/guests/host_vars/libvirt-freebsd-10/main.yml +++ b/guests/host_vars/libvirt-freebsd-10/main.yml @@ -1,8 +1,6 @@ --- ansible_python_interpreter: /usr/local/bin/python2 -PYTHONPATH: $VIRT_PREFIX/lib/python3.6/site-packages - projects: - libosinfo - libvirt diff --git a/guests/host_vars/libvirt-freebsd-11/main.yml b/guests/host_vars/libvirt-freebsd-11/main.yml index e37c845..2ad4584 100644 --- a/guests/host_vars/libvirt-freebsd-11/main.yml +++ b/guests/host_vars/libvirt-freebsd-11/main.yml @@ -1,8 +1,6 @@ --- ansible_python_interpreter: /usr/local/bin/python2 -PYTHONPATH: $VIRT_PREFIX/lib/python3.6/site-packages - projects: - libosinfo - libvirt diff --git a/guests/host_vars/libvirt-freebsd-current/main.yml b/guests/host_vars/libvirt-freebsd-current/main.yml index e37c845..2ad4584 100644 --- a/guests/host_vars/libvirt-freebsd-current/main.yml +++ b/guests/host_vars/libvirt-freebsd-current/main.yml @@ -1,8 +1,6 @@ --- ansible_python_interpreter: /usr/local/bin/python2 -PYTHONPATH: $VIRT_PREFIX/lib/python3.6/site-packages - projects: - libosinfo - libvirt diff --git a/guests/host_vars/libvirt-ubuntu-14/main.yml b/guests/host_vars/libvirt-ubuntu-14/main.yml index bfbdda6..839c668 100644 --- a/guests/host_vars/libvirt-ubuntu-14/main.yml +++ b/guests/host_vars/libvirt-ubuntu-14/main.yml @@ -1,6 +1,4 @@ --- -PYTHONPATH: $VIRT_PREFIX/lib/python3.4/site-packages - projects: - libosinfo - libvirt diff --git a/guests/host_vars/libvirt-ubuntu-16/main.yml b/guests/host_vars/libvirt-ubuntu-16/main.yml index 04d5132..a588c09 100644 --- a/guests/host_vars/libvirt-ubuntu-16/main.yml +++ b/guests/host_vars/libvirt-ubuntu-16/main.yml @@ -1,6 +1,4 @@ --- -PYTHONPATH: $VIRT_PREFIX/lib/python3.5/site-packages - projects: - libosinfo - libvirt diff --git a/guests/host_vars/libvirt-ubuntu-18/main.yml b/guests/host_vars/libvirt-ubuntu-18/main.yml index cbf57aa..a588c09 100644 --- a/guests/host_vars/libvirt-ubuntu-18/main.yml +++ b/guests/host_vars/libvirt-ubuntu-18/main.yml @@ -1,6 +1,4 @@ --- -PYTHONPATH: $VIRT_PREFIX/lib/python3.6/site-packages - projects: - libosinfo - libvirt diff --git a/guests/templates/bashrc.j2 b/guests/templates/bashrc.j2 index cf2679e..a4183ff 100644 --- a/guests/templates/bashrc.j2 +++ b/guests/templates/bashrc.j2 @@ -8,7 +8,6 @@ export PKG_CONFIG_PATH="{{ PKG_CONFIG_PATH }}" export XDG_DATA_DIRS="{{ XDG_DATA_DIRS }}" export GI_TYPELIB_PATH="{{ GI_TYPELIB_PATH }}" export OSINFO_SYSTEM_DIR="{{ OSINFO_SYSTEM_DIR }}" -export PYTHONPATH="{{ PYTHONPATH }}" export MAKE="{{ make }}" export PYTHON="{{ python }}" @@ -18,6 +17,15 @@ export PYTHON="{{ python }}" plarch=$(perl -e 'use Config; print $Config{archname}') export PERL5LIB="$VIRT_PREFIX/lib/perl5/$plarch" +# For Python we need the version number (major and minor) and +# to know whether "lib64" paths are searched +pylib=lib +if $PYTHON -c 'import sys; print("\n".join(sys.path))' | grep -q lib64; then + pylib=lib64 +fi +pyver=$($PYTHON -c 'import sys; print(".".join(map(lambda x: str(sys.version_info[x]), [0,1])))') +export PYTHONPATH="$VIRT_PREFIX/$pylib/python$pyver/site-packages" + # Enable bash completion. Only needed on FreeBSD, the system-wide # shell profile will take care of it for us everywhere else bash_completion=/usr/local/share/bash-completion/bash_completion.sh -- 2.17.0
On Mon, May 21, 2018 at 06:53:26PM +0200, Andrea Bolognani wrote:
Instead of having to specify this separately for every guest due to minor differences in the environment, figure it out at login time by asking Python itself for the information we need.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/group_vars/all/main.yml | 1 - guests/host_vars/libvirt-centos-6/main.yml | 2 -- guests/host_vars/libvirt-centos-7/main.yml | 2 -- guests/host_vars/libvirt-debian-8/main.yml | 2 -- guests/host_vars/libvirt-debian-9/main.yml | 2 -- guests/host_vars/libvirt-debian-sid/main.yml | 2 -- guests/host_vars/libvirt-fedora-27/main.yml | 2 -- guests/host_vars/libvirt-fedora-28/main.yml | 2 -- guests/host_vars/libvirt-fedora-rawhide/main.yml | 2 -- guests/host_vars/libvirt-freebsd-10/main.yml | 2 -- guests/host_vars/libvirt-freebsd-11/main.yml | 2 -- guests/host_vars/libvirt-freebsd-current/main.yml | 2 -- guests/host_vars/libvirt-ubuntu-14/main.yml | 2 -- guests/host_vars/libvirt-ubuntu-16/main.yml | 2 -- guests/host_vars/libvirt-ubuntu-18/main.yml | 2 -- guests/templates/bashrc.j2 | 10 +++++++++- 16 files changed, 9 insertions(+), 30 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
None of these ever needed per-host overrides, so we can avoid one layer of indirection and store them directly in the shell profile. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/group_vars/all/main.yml | 11 ----------- guests/templates/bashrc.j2 | 18 +++++++++--------- 2 files changed, 9 insertions(+), 20 deletions(-) diff --git a/guests/group_vars/all/main.yml b/guests/group_vars/all/main.yml index 2641c33..b73795e 100644 --- a/guests/group_vars/all/main.yml +++ b/guests/group_vars/all/main.yml @@ -5,14 +5,3 @@ ansible_ssh_pass: root jenkins_url: https://ci.centos.org/computer/{{ inventory_hostname }}/slave-agent.jnlp - -# Environment variables and paths to commands/files that might be -# OS-dependent. Can be overridden on a per-host basis -VIRT_PREFIX: $HOME/build/libvirt -PATH: $VIRT_PREFIX/bin:$HOME/.ccache/bin:$PATH -C_INCLUDE_PATH: $VIRT_PREFIX/include -LD_LIBRARY_PATH: $VIRT_PREFIX/lib -PKG_CONFIG_PATH: $VIRT_PREFIX/lib/pkgconfig -XDG_DATA_DIRS: $VIRT_PREFIX/share:/usr/share:/usr/local/share -GI_TYPELIB_PATH: $VIRT_PREFIX/lib/girepository-1.0 -OSINFO_SYSTEM_DIR: $VIRT_PREFIX/share/osinfo diff --git a/guests/templates/bashrc.j2 b/guests/templates/bashrc.j2 index a4183ff..7807fc6 100644 --- a/guests/templates/bashrc.j2 +++ b/guests/templates/bashrc.j2 @@ -1,17 +1,17 @@ export PS1="[\u@\h \w]\$ " -export VIRT_PREFIX="{{ VIRT_PREFIX }}" -export PATH="{{ PATH }}" -export C_INCLUDE_PATH="{{ C_INCLUDE_PATH }}" -export LD_LIBRARY_PATH="{{ LD_LIBRARY_PATH }}" -export PKG_CONFIG_PATH="{{ PKG_CONFIG_PATH }}" -export XDG_DATA_DIRS="{{ XDG_DATA_DIRS }}" -export GI_TYPELIB_PATH="{{ GI_TYPELIB_PATH }}" -export OSINFO_SYSTEM_DIR="{{ OSINFO_SYSTEM_DIR }}" - export MAKE="{{ make }}" export PYTHON="{{ python }}" +export VIRT_PREFIX="$HOME/build/libvirt" +export PATH="$VIRT_PREFIX/bin:$HOME/.ccache/bin:$PATH" +export C_INCLUDE_PATH="$VIRT_PREFIX/include" +export LD_LIBRARY_PATH="$VIRT_PREFIX/lib" +export PKG_CONFIG_PATH="$VIRT_PREFIX/lib/pkgconfig" +export XDG_DATA_DIRS="$VIRT_PREFIX/share:/usr/share:/usr/local/share" +export GI_TYPELIB_PATH="$VIRT_PREFIX/lib/girepository-1.0" +export OSINFO_SYSTEM_DIR="$VIRT_PREFIX/share/osinfo" + # We need to ask Perl for this information, since it's used to # construct installation paths plarch=$(perl -e 'use Config; print $Config{archname}') -- 2.17.0
On Mon, May 21, 2018 at 06:53:27PM +0200, Andrea Bolognani wrote:
None of these ever needed per-host overrides, so we can avoid one layer of indirection and store them directly in the shell profile.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/group_vars/all/main.yml | 11 ----------- guests/templates/bashrc.j2 | 18 +++++++++--------- 2 files changed, 9 insertions(+), 20 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (2)
-
Andrea Bolognani -
Daniel P. Berrangé