[libvirt-jenkins-ci PATCH 0/3] lcitool: Include some paths in the generated Dockerfiles

Andrea Bolognani (3): lcitool: Include paths in the inventory lcitool: Refactor varmap generation a bit lcitool: Include some paths in the generated Dockerfiles guests/host_vars/libvirt-centos-7/main.yml | 9 +++ guests/host_vars/libvirt-centos-8/main.yml | 9 +++ guests/host_vars/libvirt-debian-10/main.yml | 9 +++ guests/host_vars/libvirt-debian-9/main.yml | 9 +++ guests/host_vars/libvirt-debian-sid/main.yml | 9 +++ guests/host_vars/libvirt-fedora-30/main.yml | 9 +++ guests/host_vars/libvirt-fedora-31/main.yml | 9 +++ .../host_vars/libvirt-fedora-rawhide/main.yml | 9 +++ guests/host_vars/libvirt-freebsd-11/main.yml | 9 +++ guests/host_vars/libvirt-freebsd-12/main.yml | 9 +++ .../libvirt-freebsd-current/main.yml | 9 +++ .../host_vars/libvirt-opensuse-151/main.yml | 9 +++ guests/host_vars/libvirt-ubuntu-1604/main.yml | 9 +++ guests/host_vars/libvirt-ubuntu-1804/main.yml | 9 +++ guests/lcitool | 22 ++++-- guests/playbooks/update/main.yml | 1 - guests/playbooks/update/tasks/paths.yml | 71 ------------------- 17 files changed, 142 insertions(+), 78 deletions(-) delete mode 100644 guests/playbooks/update/tasks/paths.yml -- 2.25.1

Figuring them out at runtime is neat, but in practice they don't change frequently enough for it to be necessary; more importantly, including them in the inventory means we can use them in the Dockerfile generator in addition to the Ansible playbooks. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/host_vars/libvirt-centos-7/main.yml | 9 +++ guests/host_vars/libvirt-centos-8/main.yml | 9 +++ guests/host_vars/libvirt-debian-10/main.yml | 9 +++ guests/host_vars/libvirt-debian-9/main.yml | 9 +++ guests/host_vars/libvirt-debian-sid/main.yml | 9 +++ guests/host_vars/libvirt-fedora-30/main.yml | 9 +++ guests/host_vars/libvirt-fedora-31/main.yml | 9 +++ .../host_vars/libvirt-fedora-rawhide/main.yml | 9 +++ guests/host_vars/libvirt-freebsd-11/main.yml | 9 +++ guests/host_vars/libvirt-freebsd-12/main.yml | 9 +++ .../libvirt-freebsd-current/main.yml | 9 +++ .../host_vars/libvirt-opensuse-151/main.yml | 9 +++ guests/host_vars/libvirt-ubuntu-1604/main.yml | 9 +++ guests/host_vars/libvirt-ubuntu-1804/main.yml | 9 +++ guests/playbooks/update/main.yml | 1 - guests/playbooks/update/tasks/paths.yml | 71 ------------------- 16 files changed, 126 insertions(+), 72 deletions(-) delete mode 100644 guests/playbooks/update/tasks/paths.yml diff --git a/guests/host_vars/libvirt-centos-7/main.yml b/guests/host_vars/libvirt-centos-7/main.yml index becd297..7f22d9d 100644 --- a/guests/host_vars/libvirt-centos-7/main.yml +++ b/guests/host_vars/libvirt-centos-7/main.yml @@ -20,4 +20,13 @@ package_manager: 'yum' os_name: 'CentOS' os_version: '7' +bash: /bin/bash +ccache: /usr/bin/ccache +java: /usr/bin/java +make: /usr/bin/make +ninja: /usr/bin/ninja-build +python: /usr/bin/python3 +su: /bin/su +sudoers: /etc/sudoers + ansible_python_interpreter: /usr/bin/python2 diff --git a/guests/host_vars/libvirt-centos-8/main.yml b/guests/host_vars/libvirt-centos-8/main.yml index cb49e9c..bcf4224 100644 --- a/guests/host_vars/libvirt-centos-8/main.yml +++ b/guests/host_vars/libvirt-centos-8/main.yml @@ -20,4 +20,13 @@ package_manager: 'dnf' os_name: 'CentOS' os_version: '8' +bash: /bin/bash +ccache: /usr/bin/ccache +java: /usr/bin/java +make: /usr/bin/make +ninja: /usr/bin/ninja +python: /usr/bin/python3 +su: /bin/su +sudoers: /etc/sudoers + ansible_python_interpreter: /usr/bin/python3 diff --git a/guests/host_vars/libvirt-debian-10/main.yml b/guests/host_vars/libvirt-debian-10/main.yml index c5d0047..88e1634 100644 --- a/guests/host_vars/libvirt-debian-10/main.yml +++ b/guests/host_vars/libvirt-debian-10/main.yml @@ -22,4 +22,13 @@ package_manager: 'apt-get' os_name: 'Debian' os_version: '10' +bash: /bin/bash +ccache: /usr/bin/ccache +java: /usr/bin/java +make: /usr/bin/make +ninja: /usr/bin/ninja +python: /usr/bin/python3 +su: /bin/su +sudoers: /etc/sudoers + ansible_python_interpreter: /usr/bin/python3 diff --git a/guests/host_vars/libvirt-debian-9/main.yml b/guests/host_vars/libvirt-debian-9/main.yml index 3addcd4..a6f1a56 100644 --- a/guests/host_vars/libvirt-debian-9/main.yml +++ b/guests/host_vars/libvirt-debian-9/main.yml @@ -22,4 +22,13 @@ package_manager: 'apt-get' os_name: 'Debian' os_version: '9' +bash: /bin/bash +ccache: /usr/bin/ccache +java: /usr/bin/java +make: /usr/bin/make +ninja: /usr/bin/ninja +python: /usr/bin/python3 +su: /bin/su +sudoers: /etc/sudoers + ansible_python_interpreter: /usr/bin/python3 diff --git a/guests/host_vars/libvirt-debian-sid/main.yml b/guests/host_vars/libvirt-debian-sid/main.yml index a60dd1b..5dd51ff 100644 --- a/guests/host_vars/libvirt-debian-sid/main.yml +++ b/guests/host_vars/libvirt-debian-sid/main.yml @@ -22,4 +22,13 @@ package_manager: 'apt-get' os_name: 'Debian' os_version: 'Sid' +bash: /bin/bash +ccache: /usr/bin/ccache +java: /usr/bin/java +make: /usr/bin/make +ninja: /usr/bin/ninja +python: /usr/bin/python3 +su: /bin/su +sudoers: /etc/sudoers + ansible_python_interpreter: /usr/bin/python3 diff --git a/guests/host_vars/libvirt-fedora-30/main.yml b/guests/host_vars/libvirt-fedora-30/main.yml index e4ad7ea..a15aff2 100644 --- a/guests/host_vars/libvirt-fedora-30/main.yml +++ b/guests/host_vars/libvirt-fedora-30/main.yml @@ -35,4 +35,13 @@ package_manager: 'dnf' os_name: 'Fedora' os_version: '30' +bash: /bin/bash +ccache: /usr/bin/ccache +java: /usr/bin/java +make: /usr/bin/make +ninja: /usr/bin/ninja +python: /usr/bin/python3 +su: /bin/su +sudoers: /etc/sudoers + ansible_python_interpreter: /usr/bin/python3 diff --git a/guests/host_vars/libvirt-fedora-31/main.yml b/guests/host_vars/libvirt-fedora-31/main.yml index e605873..4be1a19 100644 --- a/guests/host_vars/libvirt-fedora-31/main.yml +++ b/guests/host_vars/libvirt-fedora-31/main.yml @@ -23,4 +23,13 @@ package_manager: 'dnf' os_name: 'Fedora' os_version: '31' +bash: /bin/bash +ccache: /usr/bin/ccache +java: /usr/bin/java +make: /usr/bin/make +ninja: /usr/bin/ninja +python: /usr/bin/python3 +su: /bin/su +sudoers: /etc/sudoers + ansible_python_interpreter: /usr/bin/python3 diff --git a/guests/host_vars/libvirt-fedora-rawhide/main.yml b/guests/host_vars/libvirt-fedora-rawhide/main.yml index 1a3338c..3a79a3a 100644 --- a/guests/host_vars/libvirt-fedora-rawhide/main.yml +++ b/guests/host_vars/libvirt-fedora-rawhide/main.yml @@ -23,4 +23,13 @@ package_manager: 'dnf' os_name: 'Fedora' os_version: 'Rawhide' +bash: /bin/bash +ccache: /usr/bin/ccache +java: /usr/bin/java +make: /usr/bin/make +ninja: /usr/bin/ninja +python: /usr/bin/python3 +su: /bin/su +sudoers: /etc/sudoers + ansible_python_interpreter: /usr/bin/python3 diff --git a/guests/host_vars/libvirt-freebsd-11/main.yml b/guests/host_vars/libvirt-freebsd-11/main.yml index 2de64ef..6234e8a 100644 --- a/guests/host_vars/libvirt-freebsd-11/main.yml +++ b/guests/host_vars/libvirt-freebsd-11/main.yml @@ -21,4 +21,13 @@ package_manager: 'pkg' os_name: 'FreeBSD' os_version: '11' +bash: /usr/local/bin/bash +ccache: /usr/local/bin/ccache +java: /usr/local/bin/java +make: /usr/local/bin/gmake +ninja: /usr/local/bin/ninja +python: /usr/local/bin/python3 +su: /usr/bin/su +sudoers: /usr/local/etc/sudoers + ansible_python_interpreter: /usr/local/bin/python3 diff --git a/guests/host_vars/libvirt-freebsd-12/main.yml b/guests/host_vars/libvirt-freebsd-12/main.yml index 2e3b935..22a3697 100644 --- a/guests/host_vars/libvirt-freebsd-12/main.yml +++ b/guests/host_vars/libvirt-freebsd-12/main.yml @@ -21,4 +21,13 @@ package_manager: 'pkg' os_name: 'FreeBSD' os_version: '12' +bash: /usr/local/bin/bash +ccache: /usr/local/bin/ccache +java: /usr/local/bin/java +make: /usr/local/bin/gmake +ninja: /usr/local/bin/ninja +python: /usr/local/bin/python3 +su: /usr/bin/su +sudoers: /usr/local/etc/sudoers + ansible_python_interpreter: /usr/local/bin/python3 diff --git a/guests/host_vars/libvirt-freebsd-current/main.yml b/guests/host_vars/libvirt-freebsd-current/main.yml index 9b63d30..fdc7e9b 100644 --- a/guests/host_vars/libvirt-freebsd-current/main.yml +++ b/guests/host_vars/libvirt-freebsd-current/main.yml @@ -21,4 +21,13 @@ package_manager: 'pkg' os_name: 'FreeBSD' os_version: 'Current' +bash: /usr/local/bin/bash +ccache: /usr/local/bin/ccache +java: /usr/local/bin/java +make: /usr/local/bin/gmake +ninja: /usr/local/bin/ninja +python: /usr/local/bin/python3 +su: /usr/bin/su +sudoers: /usr/local/etc/sudoers + ansible_python_interpreter: /usr/local/bin/python3 diff --git a/guests/host_vars/libvirt-opensuse-151/main.yml b/guests/host_vars/libvirt-opensuse-151/main.yml index 88d5dfd..bcc5ca3 100644 --- a/guests/host_vars/libvirt-opensuse-151/main.yml +++ b/guests/host_vars/libvirt-opensuse-151/main.yml @@ -21,4 +21,13 @@ package_manager: 'zypper' os_name: 'OpenSUSE' os_version: '151' +bash: /bin/bash +ccache: /usr/bin/ccache +java: /usr/bin/java +make: /usr/bin/make +ninja: /usr/bin/ninja +python: /usr/bin/python3 +su: /bin/su +sudoers: /etc/sudoers + ansible_python_interpreter: /usr/bin/python3 diff --git a/guests/host_vars/libvirt-ubuntu-1604/main.yml b/guests/host_vars/libvirt-ubuntu-1604/main.yml index 2754282..b474264 100644 --- a/guests/host_vars/libvirt-ubuntu-1604/main.yml +++ b/guests/host_vars/libvirt-ubuntu-1604/main.yml @@ -21,4 +21,13 @@ package_manager: 'apt-get' os_name: 'Ubuntu' os_version: '1604' +bash: /bin/bash +ccache: /usr/bin/ccache +java: /usr/bin/java +make: /usr/bin/make +ninja: /usr/bin/ninja +python: /usr/bin/python3 +su: /bin/su +sudoers: /etc/sudoers + ansible_python_interpreter: /usr/bin/python3 diff --git a/guests/host_vars/libvirt-ubuntu-1804/main.yml b/guests/host_vars/libvirt-ubuntu-1804/main.yml index 027acba..ddfeef2 100644 --- a/guests/host_vars/libvirt-ubuntu-1804/main.yml +++ b/guests/host_vars/libvirt-ubuntu-1804/main.yml @@ -22,4 +22,13 @@ package_manager: 'apt-get' os_name: 'Ubuntu' os_version: '1804' +bash: /bin/bash +ccache: /usr/bin/ccache +java: /usr/bin/java +make: /usr/bin/make +ninja: /usr/bin/ninja +python: /usr/bin/python3 +su: /bin/su +sudoers: /etc/sudoers + ansible_python_interpreter: /usr/bin/python3 diff --git a/guests/playbooks/update/main.yml b/guests/playbooks/update/main.yml index e82055b..107a78a 100644 --- a/guests/playbooks/update/main.yml +++ b/guests/playbooks/update/main.yml @@ -49,7 +49,6 @@ # Configure environment. Needs to happen after installing packages - include: '{{ playbook_base }}/tasks/kludges.yml' - - include: '{{ playbook_base }}/tasks/paths.yml' - include: '{{ playbook_base }}/tasks/bootloader.yml' - include: '{{ playbook_base }}/tasks/services.yml' - include: '{{ playbook_base }}/tasks/users.yml' diff --git a/guests/playbooks/update/tasks/paths.yml b/guests/playbooks/update/tasks/paths.yml deleted file mode 100644 index c204462..0000000 --- a/guests/playbooks/update/tasks/paths.yml +++ /dev/null @@ -1,71 +0,0 @@ ---- -- set_fact: - commands: {} - files: {} - -- name: 'Look for commands' - shell: 'which {{ item }} 2>/dev/null || true' - register: which - with_items: - - bash - - ccache - - gmake - - java - - make - - ninja - - ninja-build - - python3 - - su - -- name: 'Look for files' - shell: '(find /usr/local/etc -name {{ item }}; find /etc -name {{ item }}) 2>/dev/null' - register: find - with_items: - - sudoers - -- set_fact: - commands: '{{ commands|combine({ item.item: item.stdout }) }}' - no_log: True - with_items: - '{{ which.results }}' - -- set_fact: - files: '{{ files|combine({ item.item: item.stdout }) }}' - no_log: True - with_items: - '{{ find.results }}' - -- name: 'Export paths' - set_fact: - bash: '{{ commands["bash"] }}' - ccache: '{{ commands["ccache"] }}' - java: '{{ commands["java"] }}' - python: '{{ commands["python3"] }}' - su: '{{ commands["su"] }}' - sudoers: '{{ files["sudoers"] }}' - -# Prefer gmake, fall back to make -- name: 'Export paths' - set_fact: - make: '{{ commands["gmake"] }}' - when: - - commands["gmake"] != '' - -- name: 'Export paths' - set_fact: - make: '{{ commands["make"] }}' - when: - - make is undefined - -# Prefer ninja, fall back to ninja-build -- name: 'Export paths' - set_fact: - ninja: '{{ commands["ninja"] }}' - when: - - commands["ninja"] != '' - -- name: 'Export paths' - set_fact: - ninja: '{{ commands["ninja-build"] }}' - when: - - ninja is undefined -- 2.25.1

On Fri, Mar 27, 2020 at 06:26:25PM +0100, Andrea Bolognani wrote:
Figuring them out at runtime is neat, but in practice they don't change frequently enough for it to be necessary; more importantly, including them in the inventory means we can use them in the Dockerfile generator in addition to the Ansible playbooks.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/host_vars/libvirt-centos-7/main.yml | 9 +++ guests/host_vars/libvirt-centos-8/main.yml | 9 +++ guests/host_vars/libvirt-debian-10/main.yml | 9 +++ guests/host_vars/libvirt-debian-9/main.yml | 9 +++ guests/host_vars/libvirt-debian-sid/main.yml | 9 +++ guests/host_vars/libvirt-fedora-30/main.yml | 9 +++ guests/host_vars/libvirt-fedora-31/main.yml | 9 +++ .../host_vars/libvirt-fedora-rawhide/main.yml | 9 +++ guests/host_vars/libvirt-freebsd-11/main.yml | 9 +++ guests/host_vars/libvirt-freebsd-12/main.yml | 9 +++ .../libvirt-freebsd-current/main.yml | 9 +++ .../host_vars/libvirt-opensuse-151/main.yml | 9 +++ guests/host_vars/libvirt-ubuntu-1604/main.yml | 9 +++ guests/host_vars/libvirt-ubuntu-1804/main.yml | 9 +++ guests/playbooks/update/main.yml | 1 - guests/playbooks/update/tasks/paths.yml | 71 ------------------- 16 files changed, 126 insertions(+), 72 deletions(-) delete mode 100644 guests/playbooks/update/tasks/paths.yml
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

Some values don't depend on whether the underlying OS user deb or rpm as the package format, so we can set them in common code. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/lcitool | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index 209380a..b72abec 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -646,9 +646,16 @@ class Application: def _dockerfile_build_varmap(self, facts, mappings, pip_mappings, projects, cross_arch): if facts["package_format"] == "deb": - return self._dockerfile_build_varmap_deb(facts, mappings, pip_mappings, projects, cross_arch) + varmap = self._dockerfile_build_varmap_deb(facts, mappings, pip_mappings, projects, cross_arch) if facts["package_format"] == "rpm": - return self._dockerfile_build_varmap_rpm(facts, mappings, pip_mappings, projects, cross_arch) + varmap = self._dockerfile_build_varmap_rpm(facts, mappings, pip_mappings, projects, cross_arch) + + varmap["package_manager"] = facts["package_manager"] + + if cross_arch: + varmap["cross_abi"] = Util.native_arch_to_abi(cross_arch) + + return varmap def _dockerfile_build_varmap_deb(self, facts, mappings, pip_mappings, projects, cross_arch): package_format = facts["package_format"] @@ -700,7 +707,6 @@ class Application: pip_pkg_align = " \\\n" + (" " * len("RUN pip3 ")) varmap = {} - varmap["package_manager"] = package_manager varmap["pkgs"] = pkg_align[1:] + pkg_align.join(sorted(set(pkgs.values()))) if cross_arch: @@ -712,7 +718,6 @@ class Application: pkg_names = [p + ":" + deb_arch for p in cross_pkgs.values()] pkg_names.append(gcc) varmap["cross_pkgs"] = pkg_align[1:] + pkg_align.join(sorted(set(pkg_names))) - varmap["cross_abi"] = abi varmap["cross_lib"] = lib if pip_pkgs: @@ -771,11 +776,9 @@ class Application: pip_pkg_align = " \\\n" + (" " * len("RUN pip3 ")) varmap = {} - varmap["package_manager"] = package_manager varmap["pkgs"] = pkg_align[1:] + pkg_align.join(sorted(set(pkgs.values()))) if cross_arch: - varmap["cross_abi"] = Util.native_arch_to_abi(cross_arch) varmap["cross_pkgs"] = pkg_align[1:] + pkg_align.join(sorted(set(cross_pkgs.values()))) if pip_pkgs: -- 2.25.1

On Fri, Mar 27, 2020 at 06:26:26PM +0100, Andrea Bolognani wrote:
Some values don't depend on whether the underlying OS user deb or rpm as the package format, so we can set them in common code.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/lcitool | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

These paths are a reflections of the contents of the container image, so it makes sense to expose them in the container's environment. This will allow the GitLab CI integration for both libvirt and libosinfo to stop setting these values themselves and start relying on them being already present in the environment instead. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/lcitool | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/guests/lcitool b/guests/lcitool index b72abec..9559002 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -651,6 +651,9 @@ class Application: varmap = self._dockerfile_build_varmap_rpm(facts, mappings, pip_mappings, projects, cross_arch) varmap["package_manager"] = facts["package_manager"] + varmap["make"] = facts["make"] + varmap["ninja"] = facts["ninja"] + varmap["python"] = facts["python"] if cross_arch: varmap["cross_abi"] = Util.native_arch_to_abi(cross_arch) @@ -895,6 +898,10 @@ class Application: sys.stdout.write(textwrap.dedent(""" ENV LANG "en_US.UTF-8" + + ENV MAKE "{make}" + ENV NINJA "{ninja}" + ENV PYTHON "{python}" """).format(**varmap)) if cross_arch: -- 2.25.1

On Fri, Mar 27, 2020 at 06:26:27PM +0100, Andrea Bolognani wrote:
These paths are a reflections of the contents of the container image, so it makes sense to expose them in the container's environment.
This will allow the GitLab CI integration for both libvirt and libosinfo to stop setting these values themselves and start relying on them being already present in the environment instead.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/lcitool | 7 +++++++ 1 file changed, 7 insertions(+)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (2)
-
Andrea Bolognani
-
Daniel P. Berrangé