10:25 a.m.
This allows us to drop the ugly workaround from the Go and
MinGW templates; as a nice side-effect, we also remove the
need to hardcode most OS-dependent paths in the inventory.
Andrea Bolognani (5):
guests: Reorder tasks
guests: Figure out OS-dependent paths automatically
guests: Figure out more OS-dependent paths ahead of time
guests: Create ccache symlink farm
jobs: Remove $CC override
guests/group_vars/all/main.yml | 7 +--
guests/host_vars/libvirt-freebsd-10/main.yml | 5 --
guests/host_vars/libvirt-freebsd-11/main.yml | 5 --
guests/host_vars/libvirt-freebsd-current/main.yml | 5 --
guests/site.yml | 17 +++---
guests/tasks/paths.yml | 69 +++++++++++++++++++++++
guests/tasks/users.yml | 33 ++++++++++-
guests/templates/bashrc | 5 +-
jobs/defaults.yaml | 6 --
jobs/go.yaml | 8 ---
10 files changed, 111 insertions(+), 49 deletions(-)
create mode 100644 guests/tasks/paths.yml
--
2.14.3
10:25 a.m.
New subject: [libvirt] [jenkins-ci PATCH 1/5] guests: Reorder tasks
We're going to introduce more customization steps that need to
be performed after installing some packages. To avoid hacking
too much around the core issue, just move all environment
customization after package installation and be done with it.
Signed-off-by: Andrea Bolognani <abologna(a)redhat.com>
---
guests/site.yml | 16 ++++++----------
1 file changed, 6 insertions(+), 10 deletions(-)
diff --git a/guests/site.yml b/guests/site.yml
index a13e0e9..46d2db6 100644
--- a/guests/site.yml
+++ b/guests/site.yml
@@ -31,12 +31,6 @@
project: blacklist
state: absent
- # Configure environment. Needs to happen after installing base packages
- - include: tasks/bootloader.yml
- - include: tasks/services.yml
- - include: tasks/kludges.yml
- - include: tasks/users.yml
-
# Install build dependencies for each project
- include: tasks/packages.yml
with_items:
@@ -53,11 +47,13 @@
when:
- flavor == "jenkins"
+ # Configure environment. Needs to happen after installing packages
+ - include: tasks/bootloader.yml
+ - include: tasks/services.yml
+ - include: tasks/kludges.yml
+ - include: tasks/users.yml
+
# Configure the Jenkins agent
- include: tasks/jenkins.yml
when:
- flavor == 'jenkins'
-
- # Some of the kludges involve tweaking files that are included in
- # the packages we just installed, so go through them again here
- - include: tasks/kludges.yml
--
2.14.3
10:25 a.m.
New subject: [libvirt] [jenkins-ci PATCH 2/5] guests: Figure out OS-dependent paths automatically
When configuring a guest, we need to know the absolute path for
some commands and files; unfortunately, while most of them are
the same regardless of the OS, some of them don't.
Right now we hardcode such paths in the inventory, but it would
be better if we didn't have to do that: this commit introduces
a way to let Ansible figure out the paths for itself instead.
We're going to use it for more paths later on.
Signed-off-by: Andrea Bolognani <abologna(a)redhat.com>
---
guests/group_vars/all/main.yml | 5 ---
guests/host_vars/libvirt-freebsd-10/main.yml | 5 ---
guests/host_vars/libvirt-freebsd-11/main.yml | 5 ---
guests/host_vars/libvirt-freebsd-current/main.yml | 5 ---
guests/site.yml | 1 +
guests/tasks/paths.yml | 37 +++++++++++++++++++++++
6 files changed, 38 insertions(+), 20 deletions(-)
create mode 100644 guests/tasks/paths.yml
diff --git a/guests/group_vars/all/main.yml b/guests/group_vars/all/main.yml
index 4e05c66..b364acf 100644
--- a/guests/group_vars/all/main.yml
+++ b/guests/group_vars/all/main.yml
@@ -18,8 +18,3 @@ GI_TYPELIB_PATH: $VIRT_PREFIX/lib/girepository-1.0
OSINFO_SYSTEM_DIR: $VIRT_PREFIX/share/osinfo
PERL5LIB:
PYTHONPATH:
-
-bash: /bin/bash
-java: /usr/bin/java
-su: /bin/su
-sudoers: /etc/sudoers
diff --git a/guests/host_vars/libvirt-freebsd-10/main.yml
b/guests/host_vars/libvirt-freebsd-10/main.yml
index 487b3e8..ab5e367 100644
--- a/guests/host_vars/libvirt-freebsd-10/main.yml
+++ b/guests/host_vars/libvirt-freebsd-10/main.yml
@@ -4,11 +4,6 @@ ansible_python_interpreter: /usr/local/bin/python2
PERL5LIB: $VIRT_PREFIX/lib/perl5/site_perl/mach/5.26
PYTHONPATH: $VIRT_PREFIX/lib/python3.6/site-packages
-bash: /usr/local/bin/bash
-java: /usr/local/bin/java
-su: /usr/bin/su
-sudoers: /usr/local/etc/sudoers
-
projects:
- libosinfo
- libvirt
diff --git a/guests/host_vars/libvirt-freebsd-11/main.yml
b/guests/host_vars/libvirt-freebsd-11/main.yml
index 487b3e8..ab5e367 100644
--- a/guests/host_vars/libvirt-freebsd-11/main.yml
+++ b/guests/host_vars/libvirt-freebsd-11/main.yml
@@ -4,11 +4,6 @@ ansible_python_interpreter: /usr/local/bin/python2
PERL5LIB: $VIRT_PREFIX/lib/perl5/site_perl/mach/5.26
PYTHONPATH: $VIRT_PREFIX/lib/python3.6/site-packages
-bash: /usr/local/bin/bash
-java: /usr/local/bin/java
-su: /usr/bin/su
-sudoers: /usr/local/etc/sudoers
-
projects:
- libosinfo
- libvirt
diff --git a/guests/host_vars/libvirt-freebsd-current/main.yml
b/guests/host_vars/libvirt-freebsd-current/main.yml
index 487b3e8..ab5e367 100644
--- a/guests/host_vars/libvirt-freebsd-current/main.yml
+++ b/guests/host_vars/libvirt-freebsd-current/main.yml
@@ -4,11 +4,6 @@ ansible_python_interpreter: /usr/local/bin/python2
PERL5LIB: $VIRT_PREFIX/lib/perl5/site_perl/mach/5.26
PYTHONPATH: $VIRT_PREFIX/lib/python3.6/site-packages
-bash: /usr/local/bin/bash
-java: /usr/local/bin/java
-su: /usr/bin/su
-sudoers: /usr/local/etc/sudoers
-
projects:
- libosinfo
- libvirt
diff --git a/guests/site.yml b/guests/site.yml
index 46d2db6..063b0c6 100644
--- a/guests/site.yml
+++ b/guests/site.yml
@@ -48,6 +48,7 @@
- flavor == "jenkins"
# Configure environment. Needs to happen after installing packages
+ - include: tasks/paths.yml
- include: tasks/bootloader.yml
- include: tasks/services.yml
- include: tasks/kludges.yml
diff --git a/guests/tasks/paths.yml b/guests/tasks/paths.yml
new file mode 100644
index 0000000..36b699f
--- /dev/null
+++ b/guests/tasks/paths.yml
@@ -0,0 +1,37 @@
+---
+- set_fact:
+ commands: {}
+ files: {}
+
+- name: 'Look for commands'
+ shell: 'which {{ item }} 2>/dev/null || true'
+ register: which
+ with_items:
+ - bash
+ - java
+ - su
+
+- name: 'Look for files'
+ shell: 'find /etc /usr/local/etc -name {{ item }} 2>/dev/null'
+ register: find
+ with_items:
+ - sudoers
+
+- set_fact:
+ commands: '{{ commands|combine({ item.item: item.stdout }) }}'
+ no_log: True
+ with_items:
+ '{{ which.results }}'
+
+- set_fact:
+ files: '{{ files|combine({ item.item: item.stdout }) }}'
+ no_log: True
+ with_items:
+ '{{ find.results }}'
+
+- name: 'Export paths'
+ set_fact:
+ bash: '{{ commands["bash"] }}'
+ java: '{{ commands["java"] }}'
+ su: '{{ commands["su"] }}'
+ sudoers: '{{ files["sudoers"] }}'
--
2.14.3
10:25 a.m.
New subject: [libvirt] [jenkins-ci PATCH 3/5] guests: Figure out more OS-dependent paths ahead of time
We will need to know ccache's absolute path at the playbook
level later on, so while we're at it we might as well treat
all OS-dependent paths the same way and figure them out when
configuring the guest rather than at login time.
Signed-off-by: Andrea Bolognani <abologna(a)redhat.com>
---
guests/tasks/paths.yml | 32 ++++++++++++++++++++++++++++++++
guests/templates/bashrc | 8 +++++---
2 files changed, 37 insertions(+), 3 deletions(-)
diff --git a/guests/tasks/paths.yml b/guests/tasks/paths.yml
index 36b699f..00decc9 100644
--- a/guests/tasks/paths.yml
+++ b/guests/tasks/paths.yml
@@ -8,7 +8,12 @@
register: which
with_items:
- bash
+ - ccache
+ - gmake
- java
+ - make
+ - python2
+ - python3
- su
- name: 'Look for files'
@@ -32,6 +37,33 @@
- name: 'Export paths'
set_fact:
bash: '{{ commands["bash"] }}'
+ ccache: '{{ commands["ccache"] }}'
java: '{{ commands["java"] }}'
su: '{{ commands["su"] }}'
sudoers: '{{ files["sudoers"] }}'
+
+# Prefer gmake, fall back to make
+- name: 'Export paths'
+ set_fact:
+ make: '{{ commands["gmake"] }}'
+ when:
+ - commands["gmake"] != ''
+
+- name: 'Export paths'
+ set_fact:
+ make: '{{ commands["make"] }}'
+ when:
+ - make is undefined
+
+# Prefer python3, fall back to python2
+- name: 'Export paths'
+ set_fact:
+ python: '{{ commands["python3"] }}'
+ when:
+ - commands["python3"] != ''
+
+- name: 'Export paths'
+ set_fact:
+ python: '{{ commands["python2"] }}'
+ when:
+ - python is undefined
diff --git a/guests/templates/bashrc b/guests/templates/bashrc
index e925cd9..65c82cd 100644
--- a/guests/templates/bashrc
+++ b/guests/templates/bashrc
@@ -11,9 +11,11 @@ export OSINFO_SYSTEM_DIR="{{ OSINFO_SYSTEM_DIR }}"
export PERL5LIB="{{ PERL5LIB }}"
export PYTHONPATH="{{ PYTHONPATH }}"
-which python3 >/dev/null 2>&1 && export PYTHON="python3" ||
export PYTHON="python2"
-which gmake >/dev/null 2>&1 && export MAKE="gmake" || export
MAKE="make"
-which ccache >/dev/null 2>&1 && export CC="ccache cc"
+export CCACHE="{{ ccache }}"
+export MAKE="{{ make }}"
+export PYTHON="{{ python }}"
+
+test "$CCACHE" && export CC="$CCACHE cc"
# Enable bash completion. Only needed on FreeBSD, the system-wide
# shell profile will take care of it for us everywhere else
--
2.14.3
10:25 a.m.
New subject: [libvirt] [jenkins-ci PATCH 4/5] guests: Create ccache symlink farm
ccache can detect the name it's been called with and
transparently invoke the compiler if it matches. The Fedora
package uses this trick to enable ccache out of the box, but
other operating systems and distributions leave the task up
to the user.
Replicate the setup in the user's ~/.ccache directory so
that it works across all guests. With the linke farm in
place, setting $CC in the user's shell profile is no longer
necessary.
Signed-off-by: Andrea Bolognani <abologna(a)redhat.com>
---
guests/group_vars/all/main.yml | 2 +-
guests/tasks/users.yml | 33 ++++++++++++++++++++++++++++++++-
guests/templates/bashrc | 3 ---
3 files changed, 33 insertions(+), 5 deletions(-)
diff --git a/guests/group_vars/all/main.yml b/guests/group_vars/all/main.yml
index b364acf..cd8ef25 100644
--- a/guests/group_vars/all/main.yml
+++ b/guests/group_vars/all/main.yml
@@ -9,7 +9,7 @@ jenkins_url: https://ci.centos.org/computer/{{ inventory_hostname
}}/slave-agent
# Environment variables and paths to commands/files that might be
# OS-dependent. Can be overridden on a per-host basis
VIRT_PREFIX: $HOME/build/libvirt
-PATH: $VIRT_PREFIX/bin:$PATH
+PATH: $VIRT_PREFIX/bin:$HOME/.ccache/bin:$PATH
C_INCLUDE_PATH: $VIRT_PREFIX/include
LD_LIBRARY_PATH: $VIRT_PREFIX/lib
PKG_CONFIG_PATH: $VIRT_PREFIX/lib/pkgconfig
diff --git a/guests/tasks/users.yml b/guests/tasks/users.yml
index e691b32..033ca2a 100644
--- a/guests/tasks/users.yml
+++ b/guests/tasks/users.yml
@@ -53,10 +53,13 @@
- name: '{{ flavor }}: Configure ccache'
file:
- path: /home/{{ flavor }}/.ccache
+ path: /home/{{ flavor }}/.{{ item }}
state: directory
owner: '{{ flavor }}'
group: '{{ flavor }}'
+ with_items:
+ - ccache
+ - ccache/bin
- name: '{{ flavor }}: Configure ccache'
copy:
@@ -65,6 +68,34 @@
owner: '{{ flavor }}'
group: '{{ flavor }}'
+- name: '{{ flavor }}: Create ccache wrappers'
+ file:
+ src: '{{ ccache }}'
+ dest: /home/{{ flavor }}/.ccache/bin/{{ item }}
+ state: link
+ owner: '{{ flavor }}'
+ group: '{{ flavor }}'
+ with_items:
+ - cc
+ - clang
+ when:
+ - ccache != ''
+ - os_name == 'FreeBSD'
+
+- name: '{{ flavor }}: Create ccache wrappers'
+ file:
+ src: '{{ ccache }}'
+ dest: /home/{{ flavor }}/.ccache/bin/{{ item }}
+ state: link
+ owner: '{{ flavor }}'
+ group: '{{ flavor }}'
+ with_items:
+ - cc
+ - gcc
+ when:
+ - ccache != ''
+ - os_name != 'FreeBSD'
+
- name: '{{ flavor }}: Create shell profile'
template:
src: templates/{{ item }}
diff --git a/guests/templates/bashrc b/guests/templates/bashrc
index 65c82cd..866087e 100644
--- a/guests/templates/bashrc
+++ b/guests/templates/bashrc
@@ -11,12 +11,9 @@ export OSINFO_SYSTEM_DIR="{{ OSINFO_SYSTEM_DIR }}"
export PERL5LIB="{{ PERL5LIB }}"
export PYTHONPATH="{{ PYTHONPATH }}"
-export CCACHE="{{ ccache }}"
export MAKE="{{ make }}"
export PYTHON="{{ python }}"
-test "$CCACHE" && export CC="$CCACHE cc"
-
# Enable bash completion. Only needed on FreeBSD, the system-wide
# shell profile will take care of it for us everywhere else
bash_completion=/usr/local/share/bash-completion/bash_completion.sh
--
2.14.3
10:36 a.m.
New subject: [libvirt] [jenkins-ci PATCH 4/5] guests: Create ccache symlink farm
Disclaimer: I don't feel like I can review the patches properly. Not
only because I never heard the term "symlink farm" before, but mostly
because jenkins-ci. However I noticed two nits here and there. I hope
it won't discourage others from review. I apologize in advance if it
does.
On Wed, Apr 18, 2018 at 05:25:21PM +0200, Andrea Bolognani wrote:
ccache can detect the name it's been called with and
transparently invoke the compiler if it matches. The Fedora
package uses this trick to enable ccache out of the box, but
other operating systems and distributions leave the task up
to the user.
Replicate the setup in the user's ~/.ccache directory so
that it works across all guests. With the linke farm in
s/linke/link/ ?
place, setting $CC in the user's shell profile is no longer
necessary.
Signed-off-by: Andrea Bolognani <abologna(a)redhat.com>
---
guests/group_vars/all/main.yml | 2 +-
guests/tasks/users.yml | 33 ++++++++++++++++++++++++++++++++-
guests/templates/bashrc | 3 ---
3 files changed, 33 insertions(+), 5 deletions(-)
diff --git a/guests/group_vars/all/main.yml b/guests/group_vars/all/main.yml
index b364acf..cd8ef25 100644
--- a/guests/group_vars/all/main.yml
+++ b/guests/group_vars/all/main.yml
@@ -9,7 +9,7 @@ jenkins_url: https://ci.centos.org/computer/{{ inventory_hostname
}}/slave-agent
# Environment variables and paths to commands/files that might be
# OS-dependent. Can be overridden on a per-host basis
VIRT_PREFIX: $HOME/build/libvirt
-PATH: $VIRT_PREFIX/bin:$PATH
+PATH: $VIRT_PREFIX/bin:$HOME/.ccache/bin:$PATH
C_INCLUDE_PATH: $VIRT_PREFIX/include
LD_LIBRARY_PATH: $VIRT_PREFIX/lib
PKG_CONFIG_PATH: $VIRT_PREFIX/lib/pkgconfig
diff --git a/guests/tasks/users.yml b/guests/tasks/users.yml
index e691b32..033ca2a 100644
--- a/guests/tasks/users.yml
+++ b/guests/tasks/users.yml
@@ -53,10 +53,13 @@
- name: '{{ flavor }}: Configure ccache'
file:
- path: /home/{{ flavor }}/.ccache
+ path: /home/{{ flavor }}/.{{ item }}
state: directory
owner: '{{ flavor }}'
group: '{{ flavor }}'
+ with_items:
+ - ccache
+ - ccache/bin
With new enough ansible, you don't need the 'ccache' entry as this
behaves similarly to `mkdir -p`:
"If directory, all immediate subdirectories will be created if they do
not exist, since 1.7 they will be created with the supplied
permissions." [1]
You can also use 'recurse' if doing more than just creating them.
[1] https://docs.ansible.com/ansible/latest/modules/file_module.html
Have a nice day
Martin
8:06 a.m.
New subject: [libvirt] [jenkins-ci PATCH 4/5] guests: Create ccache symlink farm
On Thu, 2018-04-19 at 17:36 +0200, Martin Kletzander wrote:
> Replicate the setup in the user's ~/.ccache directory so
> that it works across all guests. With the linke farm in
s/linke/link/ ?
Close: I actually meant to write Linkle[1] :P
> - name: '{{ flavor }}: Configure ccache'
> file:
> - path: /home/{{ flavor }}/.ccache
> + path: /home/{{ flavor }}/.{{ item }}
> state: directory
> owner: '{{ flavor }}'
> group: '{{ flavor }}'
> + with_items:
> + - ccache
> + - ccache/bin
With new enough ansible, you don't need the 'ccache' entry as this
behaves similarly to `mkdir -p`:
"If directory, all immediate subdirectories will be created if they do
not exist, since 1.7 they will be created with the supplied
permissions." [1]
If you dig through the playbooks, you'll certainly find no lack of
suboptimal or roundabout approaches to the problem at hand, mostly
as a consequence of me pretty much figuring out Ansible as I went.
I'll be make to get rid of this specific instance before pushing.
More patches in the spirit would be very much welcome O:-)
[1] https://zelda.gamepedia.com/Linkle
--
Andrea Bolognani / Red Hat / Virtualization
10:25 a.m.
New subject: [libvirt] [jenkins-ci PATCH 5/5] jobs: Remove $CC override
Neither Go nor MinGW builds were able to deal with our
previous way of enabling ccache correctly, which forced us
to work around the issue at the job template level.
Now that ccache is enabled in a completely transparent
fashion through a symlink farm, we can drop the workaround.
Signed-off-by: Andrea Bolognani <abologna(a)redhat.com>
---
jobs/defaults.yaml | 6 ------
jobs/go.yaml | 8 --------
2 files changed, 14 deletions(-)
diff --git a/jobs/defaults.yaml b/jobs/defaults.yaml
index 99e8b62..43686c5 100644
--- a/jobs/defaults.yaml
+++ b/jobs/defaults.yaml
@@ -23,17 +23,11 @@
global_env: |
local_env: |
mingw32_local_env: |
- # The MinGW build needs to use the MinGW compiler toolchain,
- # while $CC is pointing to the native toolchain, so we have
- # to unset it here.
- export CC=
export VIRT_PREFIX="$VIRT_PREFIX/i686-w64-mingw32/sys-root/mingw"
export PKG_CONFIG_PATH="$VIRT_PREFIX/lib/pkgconfig"
export
PKG_CONFIG_LIBDIR="/usr/i686-w64-mingw32/sys-root/mingw/lib/pkgconfig:/usr/i686-w64-mingw32/sys-root/mingw/share/pkgconfig"
mingw32_autogen_args: --host=i686-w64-mingw32
mingw64_local_env: |
- # See above
- export CC=
export VIRT_PREFIX="$VIRT_PREFIX/x86_64-w64-mingw32/sys-root/mingw"
export PKG_CONFIG_PATH="$VIRT_PREFIX/lib/pkgconfig"
export
PKG_CONFIG_LIBDIR="/usr/x86_64-w64-mingw32/sys-root/mingw/lib/pkgconfig:/usr/x86_64-w64-mingw32/sys-root/mingw/share/pkgconfig"
diff --git a/jobs/go.yaml b/jobs/go.yaml
index 9a349ca..bffe56e 100644
--- a/jobs/go.yaml
+++ b/jobs/go.yaml
@@ -42,11 +42,6 @@
- shell: |
{global_env}
{local_env}
- # go doesn't handle the $CC variable containing whitespace
- # correctly, so we have to make sure it's unset when building.
- # See https://github.com/golang/go/issues/11685
- export CC=
-
go build -v
publishers:
- email:
@@ -85,9 +80,6 @@
- shell: |
{global_env}
{local_env}
- # See above
- export CC=
-
go test $TEST_ARGS
publishers:
- email:
--
2.14.3
9:23 a.m.
On Wed, Apr 18, 2018 at 05:25:17PM +0200, Andrea Bolognani wrote:
This allows us to drop the ugly workaround from the Go and
MinGW templates; as a nice side-effect, we also remove the
need to hardcode most OS-dependent paths in the inventory.
Andrea Bolognani (5):
guests: Reorder tasks
guests: Figure out OS-dependent paths automatically
guests: Figure out more OS-dependent paths ahead of time
guests: Create ccache symlink farm
jobs: Remove $CC override
Reviewed-by: Pavel Hrdina <phrdina(a)redhat.com>
2:47 a.m.
On Thu, 2018-04-26 at 16:23 +0200, Pavel Hrdina wrote:
On Wed, Apr 18, 2018 at 05:25:17PM +0200, Andrea Bolognani wrote:
> This allows us to drop the ugly workaround from the Go and
> MinGW templates; as a nice side-effect, we also remove the
> need to hardcode most OS-dependent paths in the inventory.
>
> Andrea Bolognani (5):
> guests: Reorder tasks
> guests: Figure out OS-dependent paths automatically
> guests: Figure out more OS-dependent paths ahead of time
> guests: Create ccache symlink farm
> jobs: Remove $CC override
Reviewed-by: Pavel Hrdina <phrdina(a)redhat.com>
Thanks, I've pushed them now. Feel free to deploy the changes to
the CI environment whenever it's convenient for you :)
--
Andrea Bolognani / Red Hat / Virtualization
2401
days inactive
2410
days old
9 comments
3 participants
participants (3)
-
Andrea Bolognani -
Martin Kletzander -
Pavel Hrdina