Instead of having the same code snippet twice, once for Linux and once for FreeBSD, the only actual difference being the name of the compiler, include the compiler's path in the inventory among other paths and then figure out the correct name for the symlink based on that information. Signed-off-by: Andrea Bolognani <abologna(a)redhat.com&gt; --- guests/host_vars/libvirt-centos-7/main.yml | 1 + guests/host_vars/libvirt-centos-8/main.yml | 1 + guests/host_vars/libvirt-debian-10/main.yml | 1 + guests/host_vars/libvirt-debian-9/main.yml | 1 + guests/host_vars/libvirt-debian-sid/main.yml | 1 + guests/host_vars/libvirt-fedora-30/main.yml | 1 + guests/host_vars/libvirt-fedora-31/main.yml | 1 + .../host_vars/libvirt-fedora-rawhide/main.yml | 1 + guests/host_vars/libvirt-freebsd-11/main.yml | 1 + guests/host_vars/libvirt-freebsd-12/main.yml | 1 + .../libvirt-freebsd-current/main.yml | 1 + .../host_vars/libvirt-opensuse-151/main.yml | 1 + guests/host_vars/libvirt-ubuntu-1604/main.yml | 1 + guests/host_vars/libvirt-ubuntu-1804/main.yml | 1 + guests/playbooks/update/tasks/users.yml | 19 +------------------ 15 files changed, 15 insertions(+), 18 deletions(-) diff --git a/guests/host_vars/libvirt-centos-7/main.yml b/guests/host_vars/libvirt-centos-7/main.yml index 7f22d9d..f1fef44 100644 --- a/guests/host_vars/libvirt-centos-7/main.yml +++ b/guests/host_vars/libvirt-centos-7/main.yml @@ -21,6 +21,7 @@ os_name: 'CentOS' os_version: '7' bash: /bin/bash +cc: /usr/bin/gcc ccache: /usr/bin/ccache java: /usr/bin/java make: /usr/bin/make diff --git a/guests/host_vars/libvirt-centos-8/main.yml b/guests/host_vars/libvirt-centos-8/main.yml index bcf4224..48edb78 100644 --- a/guests/host_vars/libvirt-centos-8/main.yml +++ b/guests/host_vars/libvirt-centos-8/main.yml @@ -21,6 +21,7 @@ os_name: 'CentOS' os_version: '8' bash: /bin/bash +cc: /usr/bin/gcc ccache: /usr/bin/ccache java: /usr/bin/java make: /usr/bin/make diff --git a/guests/host_vars/libvirt-debian-10/main.yml b/guests/host_vars/libvirt-debian-10/main.yml index 88e1634..37036e6 100644 --- a/guests/host_vars/libvirt-debian-10/main.yml +++ b/guests/host_vars/libvirt-debian-10/main.yml @@ -23,6 +23,7 @@ os_name: 'Debian' os_version: '10' bash: /bin/bash +cc: /usr/bin/gcc ccache: /usr/bin/ccache java: /usr/bin/java make: /usr/bin/make diff --git a/guests/host_vars/libvirt-debian-9/main.yml b/guests/host_vars/libvirt-debian-9/main.yml index a6f1a56..b46efd1 100644 --- a/guests/host_vars/libvirt-debian-9/main.yml +++ b/guests/host_vars/libvirt-debian-9/main.yml @@ -23,6 +23,7 @@ os_name: 'Debian' os_version: '9' bash: /bin/bash +cc: /usr/bin/gcc ccache: /usr/bin/ccache java: /usr/bin/java make: /usr/bin/make diff --git a/guests/host_vars/libvirt-debian-sid/main.yml b/guests/host_vars/libvirt-debian-sid/main.yml index 5dd51ff..36045cd 100644 --- a/guests/host_vars/libvirt-debian-sid/main.yml +++ b/guests/host_vars/libvirt-debian-sid/main.yml @@ -23,6 +23,7 @@ os_name: 'Debian' os_version: 'Sid' bash: /bin/bash +cc: /usr/bin/gcc ccache: /usr/bin/ccache java: /usr/bin/java make: /usr/bin/make diff --git a/guests/host_vars/libvirt-fedora-30/main.yml b/guests/host_vars/libvirt-fedora-30/main.yml index a15aff2..71b8e7b 100644 --- a/guests/host_vars/libvirt-fedora-30/main.yml +++ b/guests/host_vars/libvirt-fedora-30/main.yml @@ -36,6 +36,7 @@ os_name: 'Fedora' os_version: '30' bash: /bin/bash +cc: /usr/bin/gcc ccache: /usr/bin/ccache java: /usr/bin/java make: /usr/bin/make diff --git a/guests/host_vars/libvirt-fedora-31/main.yml b/guests/host_vars/libvirt-fedora-31/main.yml index 4be1a19..c692715 100644 --- a/guests/host_vars/libvirt-fedora-31/main.yml +++ b/guests/host_vars/libvirt-fedora-31/main.yml @@ -24,6 +24,7 @@ os_name: 'Fedora' os_version: '31' bash: /bin/bash +cc: /usr/bin/gcc ccache: /usr/bin/ccache java: /usr/bin/java make: /usr/bin/make diff --git a/guests/host_vars/libvirt-fedora-rawhide/main.yml b/guests/host_vars/libvirt-fedora-rawhide/main.yml index 3a79a3a..bea4ad1 100644 --- a/guests/host_vars/libvirt-fedora-rawhide/main.yml +++ b/guests/host_vars/libvirt-fedora-rawhide/main.yml @@ -24,6 +24,7 @@ os_name: 'Fedora' os_version: 'Rawhide' bash: /bin/bash +cc: /usr/bin/gcc ccache: /usr/bin/ccache java: /usr/bin/java make: /usr/bin/make diff --git a/guests/host_vars/libvirt-freebsd-11/main.yml b/guests/host_vars/libvirt-freebsd-11/main.yml index 6234e8a..7fbab39 100644 --- a/guests/host_vars/libvirt-freebsd-11/main.yml +++ b/guests/host_vars/libvirt-freebsd-11/main.yml @@ -22,6 +22,7 @@ os_name: 'FreeBSD' os_version: '11' bash: /usr/local/bin/bash +cc: /usr/bin/clang ccache: /usr/local/bin/ccache java: /usr/local/bin/java make: /usr/local/bin/gmake diff --git a/guests/host_vars/libvirt-freebsd-12/main.yml b/guests/host_vars/libvirt-freebsd-12/main.yml index 22a3697..abedd95 100644 --- a/guests/host_vars/libvirt-freebsd-12/main.yml +++ b/guests/host_vars/libvirt-freebsd-12/main.yml @@ -22,6 +22,7 @@ os_name: 'FreeBSD' os_version: '12' bash: /usr/local/bin/bash +cc: /usr/bin/clang ccache: /usr/local/bin/ccache java: /usr/local/bin/java make: /usr/local/bin/gmake diff --git a/guests/host_vars/libvirt-freebsd-current/main.yml b/guests/host_vars/libvirt-freebsd-current/main.yml index fdc7e9b..9b0ba1f 100644 --- a/guests/host_vars/libvirt-freebsd-current/main.yml +++ b/guests/host_vars/libvirt-freebsd-current/main.yml @@ -22,6 +22,7 @@ os_name: 'FreeBSD' os_version: 'Current' bash: /usr/local/bin/bash +cc: /usr/bin/clang ccache: /usr/local/bin/ccache java: /usr/local/bin/java make: /usr/local/bin/gmake diff --git a/guests/host_vars/libvirt-opensuse-151/main.yml b/guests/host_vars/libvirt-opensuse-151/main.yml index bcc5ca3..3e674cc 100644 --- a/guests/host_vars/libvirt-opensuse-151/main.yml +++ b/guests/host_vars/libvirt-opensuse-151/main.yml @@ -22,6 +22,7 @@ os_name: 'OpenSUSE' os_version: '151' bash: /bin/bash +cc: /usr/bin/gcc ccache: /usr/bin/ccache java: /usr/bin/java make: /usr/bin/make diff --git a/guests/host_vars/libvirt-ubuntu-1604/main.yml b/guests/host_vars/libvirt-ubuntu-1604/main.yml index b474264..1eb2345 100644 --- a/guests/host_vars/libvirt-ubuntu-1604/main.yml +++ b/guests/host_vars/libvirt-ubuntu-1604/main.yml @@ -22,6 +22,7 @@ os_name: 'Ubuntu' os_version: '1604' bash: /bin/bash +cc: /usr/bin/gcc ccache: /usr/bin/ccache java: /usr/bin/java make: /usr/bin/make diff --git a/guests/host_vars/libvirt-ubuntu-1804/main.yml b/guests/host_vars/libvirt-ubuntu-1804/main.yml index ddfeef2..c9515b6 100644 --- a/guests/host_vars/libvirt-ubuntu-1804/main.yml +++ b/guests/host_vars/libvirt-ubuntu-1804/main.yml @@ -23,6 +23,7 @@ os_name: 'Ubuntu' os_version: '1804' bash: /bin/bash +cc: /usr/bin/gcc ccache: /usr/bin/ccache java: /usr/bin/java make: /usr/bin/make diff --git a/guests/playbooks/update/tasks/users.yml b/guests/playbooks/update/tasks/users.yml index a6fb0aa..5d202b3 100644 --- a/guests/playbooks/update/tasks/users.yml +++ b/guests/playbooks/update/tasks/users.yml @@ -79,24 +79,7 @@ group: '{{ flavor }}' with_items: - cc - - clang - when: - - ccache != '' - - os_name == 'FreeBSD' - -- name: '{{ flavor }}: Create ccache wrappers' - file: - src: '{{ ccache }}' - dest: /home/{{ flavor }}/.ccache/bin/{{ item }} - state: link - owner: '{{ flavor }}' - group: '{{ flavor }}' - with_items: - - cc - - gcc - when: - - ccache != '' - - os_name != 'FreeBSD' + - '{{ cc | basename }}' - name: '{{ flavor }}: Create shell profile' template: -- 2.25.1

We're going to stop creating $HOME/.ccache soon, so we need an alternative way to configure ccache; environment variables work perfectly fine. Signed-off-by: Andrea Bolognani <abologna(a)redhat.com&gt; --- guests/playbooks/update/tasks/users.yml | 7 ------- guests/playbooks/update/templates/bashrc.j2 | 1 + guests/playbooks/update/templates/ccache.conf.j2 | 1 - 3 files changed, 1 insertion(+), 8 deletions(-) delete mode 100644 guests/playbooks/update/templates/ccache.conf.j2 diff --git a/guests/playbooks/update/tasks/users.yml b/guests/playbooks/update/tasks/users.yml index 5d202b3..931d79b 100644 --- a/guests/playbooks/update/tasks/users.yml +++ b/guests/playbooks/update/tasks/users.yml @@ -63,13 +63,6 @@ - ccache - ccache/bin -- name: '{{ flavor }}: Configure ccache' - template: - src: '{{ playbook_base }}/templates/ccache.conf.j2' - dest: /home/{{ flavor }}/.ccache/ccache.conf - owner: '{{ flavor }}' - group: '{{ flavor }}' - - name: '{{ flavor }}: Create ccache wrappers' file: src: '{{ ccache }}' diff --git a/guests/playbooks/update/templates/bashrc.j2 b/guests/playbooks/update/templates/bashrc.j2 index 9cea90c..898d30b 100644 --- a/guests/playbooks/update/templates/bashrc.j2 +++ b/guests/playbooks/update/templates/bashrc.j2 @@ -5,6 +5,7 @@ export NINJA="{{ ninja }}" export PYTHON="{{ python }}" export MAKEFLAGS="-j{{ install_vcpus|int + 1 }}" +export CCACHE_MAXSIZE="2G" export VIRT_PREFIX="$HOME/build/libvirt" export PATH="$VIRT_PREFIX/bin:$HOME/.ccache/bin:$PATH" diff --git a/guests/playbooks/update/templates/ccache.conf.j2 b/guests/playbooks/update/templates/ccache.conf.j2 deleted file mode 100644 index 48c3756..0000000 --- a/guests/playbooks/update/templates/ccache.conf.j2 +++ /dev/null @@ -1 +0,0 @@ -max_size = 2G -- 2.25.1

On Fri, Mar 27, 2020 at 08:34:55PM +0100, Andrea Bolognani wrote: I weakly suggest "/usr/local/share/ccache/bin" Reviewed-by: Daniel P. Berrangé <berrange(a)redhat.com&gt; Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

It will make further changes easier. This commit is better viewed with 'git show -w'. Signed-off-by: Andrea Bolognani <abologna(a)redhat.com&gt; --- guests/lcitool | 34 ++++++++++++++++------------------ 1 file changed, 16 insertions(+), 18 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index 9559002..98ef2a0 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -808,20 +808,6 @@ class Application: sed -Ei 's,^# (en_US\\.UTF-8 .*)$,\\1,' /etc/locale.gen && \\ dpkg-reconfigure locales """).format(**varmap)) - if cross_arch: - # Intentionally a separate RUN command from the above - # so that the common packages of all cross-built images - # share a Docker image layer. - sys.stdout.write(textwrap.dedent(""" - RUN export DEBIAN_FRONTEND=noninteractive && \\ - dpkg --add-architecture {cross_arch} && \\ - {package_manager} update && \\ - {package_manager} dist-upgrade -y && \\ - {package_manager} install --no-install-recommends -y dpkg-dev && \\ - {package_manager} install --no-install-recommends -y {cross_pkgs} && \\ - {package_manager} autoremove -y && \\ - {package_manager} autoclean -y - """).format(**varmap)) elif package_format == "rpm": commands = [] @@ -882,10 +868,22 @@ class Application: sys.stdout.write(script.format(**varmap)) - if cross_arch: - # Intentionally a separate RUN command from the above - # so that the common packages of all cross-built images - # share a Docker image layer. + if cross_arch: + # Intentionally a separate RUN command from the above + # so that the common packages of all cross-built images + # share a Docker image layer. + if package_format == "deb": + sys.stdout.write(textwrap.dedent(""" + RUN export DEBIAN_FRONTEND=noninteractive && \\ + dpkg --add-architecture {cross_arch} && \\ + {package_manager} update && \\ + {package_manager} dist-upgrade -y && \\ + {package_manager} install --no-install-recommends -y dpkg-dev && \\ + {package_manager} install --no-install-recommends -y {cross_pkgs} && \\ + {package_manager} autoremove -y && \\ + {package_manager} autoclean -y + """).format(**varmap)) + elif package_format == "rpm": sys.stdout.write(textwrap.dedent(""" RUN {package_manager} install -y {cross_pkgs} && \\ {package_manager} clean all -y -- 2.25.1

It's nicer to use, and will make further changes easier. This commit is better viewed with 'git show -w'. Signed-off-by: Andrea Bolognani <abologna(a)redhat.com&gt; --- guests/lcitool | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index 98ef2a0..fb0219b 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -797,20 +797,20 @@ class Application: print("FROM {}".format(facts["docker_base"])) + commands = [] + if package_format == "deb": - sys.stdout.write(textwrap.dedent(""" - RUN export DEBIAN_FRONTEND=noninteractive && \\ - {package_manager} update && \\ - {package_manager} dist-upgrade -y && \\ - {package_manager} install --no-install-recommends -y {pkgs} && \\ - {package_manager} autoremove -y && \\ - {package_manager} autoclean -y && \\ - sed -Ei 's,^# (en_US\\.UTF-8 .*)$,\\1,' /etc/locale.gen && \\ - dpkg-reconfigure locales - """).format(**varmap)) + commands.extend([ + "export DEBIAN_FRONTEND=noninteractive", + "{package_manager} update", + "{package_manager} dist-upgrade -y", + "{package_manager} install --no-install-recommends -y {pkgs}", + "{package_manager} autoremove -y", + "{package_manager} autoclean -y", + "sed -Ei 's,^# (en_US\\.UTF-8 .*)$,\\1,' /etc/locale.gen", + "dpkg-reconfigure locales", + ]) elif package_format == "rpm": - commands = [] - # Rawhide needs this because the keys used to sign packages are # cycled from time to time if os_name == "Fedora" and os_version == "Rawhide": @@ -864,9 +864,8 @@ class Application: "{package_manager} clean all -y", ]) - script = "\nRUN " + (" && \\\n ".join(commands)) + "\n" - - sys.stdout.write(script.format(**varmap)) + script = "\nRUN " + (" && \\\n ".join(commands)) + "\n" + sys.stdout.write(script.format(**varmap)) if cross_arch: # Intentionally a separate RUN command from the above -- 2.25.1

It's nicer to use, and consistent with how we're building the native part of the Dockerfile. Signed-off-by: Andrea Bolognani <abologna(a)redhat.com&gt; --- guests/lcitool | 33 +++++++++++++++++++-------------- 1 file changed, 19 insertions(+), 14 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index fb0219b..117e1a5 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -868,25 +868,30 @@ class Application: sys.stdout.write(script.format(**varmap)) if cross_arch: + cross_commands = [] + # Intentionally a separate RUN command from the above # so that the common packages of all cross-built images # share a Docker image layer. if package_format == "deb": - sys.stdout.write(textwrap.dedent(""" - RUN export DEBIAN_FRONTEND=noninteractive && \\ - dpkg --add-architecture {cross_arch} && \\ - {package_manager} update && \\ - {package_manager} dist-upgrade -y && \\ - {package_manager} install --no-install-recommends -y dpkg-dev && \\ - {package_manager} install --no-install-recommends -y {cross_pkgs} && \\ - {package_manager} autoremove -y && \\ - {package_manager} autoclean -y - """).format(**varmap)) + cross_commands.extend([ + "export DEBIAN_FRONTEND=noninteractive", + "dpkg --add-architecture {cross_arch}", + "{package_manager} update", + "{package_manager} dist-upgrade -y", + "{package_manager} install --no-install-recommends -y dpkg-dev", + "{package_manager} install --no-install-recommends -y {cross_pkgs}", + "{package_manager} autoremove -y", + "{package_manager} autoclean -y", + ]) elif package_format == "rpm": - sys.stdout.write(textwrap.dedent(""" - RUN {package_manager} install -y {cross_pkgs} && \\ - {package_manager} clean all -y - """).format(**varmap)) + cross_commands.extend([ + "{package_manager} install -y {cross_pkgs}", + "{package_manager} clean all -y", + ]) + + cross_script = "\nRUN " + (" && \\\n ".join(cross_commands)) + "\n" + sys.stdout.write(cross_script.format(**varmap)) if "pip_pkgs" in varmap: sys.stdout.write(textwrap.dedent(""" -- 2.25.1

VM-based builds have used ccache by default for a very long time, and now container-based builds will too. Signed-off-by: Andrea Bolognani <abologna(a)redhat.com&gt; --- guests/lcitool | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/guests/lcitool b/guests/lcitool index 117e1a5..011fc07 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -651,6 +651,8 @@ class Application: varmap = self._dockerfile_build_varmap_rpm(facts, mappings, pip_mappings, projects, cross_arch) varmap["package_manager"] = facts["package_manager"] + varmap["cc"] = facts["cc"] + varmap["ccache"] = facts["ccache"] varmap["make"] = facts["make"] varmap["ninja"] = facts["ninja"] varmap["python"] = facts["python"] @@ -864,6 +866,21 @@ class Application: "{package_manager} clean all -y", ]) + commands.extend([ + "mkdir -p /usr/local/share/ccache-wrappers", + ]) + + if cross_arch: + commands.extend([ + "ln -s {ccache} /usr/local/share/ccache-wrappers/{cross_abi}-cc", + "ln -s {ccache} /usr/local/share/ccache-wrappers/{cross_abi}-$(basename {cc})", + ]) + else: + commands.extend([ + "ln -s {ccache} /usr/local/share/ccache-wrappers/cc", + "ln -s {ccache} /usr/local/share/ccache-wrappers/$(basename {cc})", + ]) + script = "\nRUN " + (" && \\\n ".join(commands)) + "\n" sys.stdout.write(script.format(**varmap)) @@ -900,6 +917,7 @@ class Application: sys.stdout.write(textwrap.dedent(""" ENV LANG "en_US.UTF-8" + ENV PATH "/usr/local/share/ccache-wrappers:$PATH" ENV MAKE "{make}" ENV NINJA "{ninja}" -- 2.25.1

On Fri, Mar 27, 2020 at 08:34:59PM +0100, Andrea Bolognani wrote: I've just realized that this addition has prevented the caching and reuse the base layer across the cross images. The first "RUN" cmmand was supposed to have stuff that is common across cross images, but we've accidentally included the "cross_abi" in the path for ccache here. We need to put the ccache setup in the second RUN command, for cross containers. Only native images can have it in the first RUN command. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|