The libpcap-dev package is a temporary backcompat package until everything switches to the new libpcap0.8-dev pacakge name. Reviewed-by: Andrea Bolognani <abologna(a)redhat.com&gt; Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com&gt; --- guests/vars/mappings.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guests/vars/mappings.yml b/guests/vars/mappings.yml index 84543c4..f211169 100644 --- a/guests/vars/mappings.yml +++ b/guests/vars/mappings.yml @@ -281,7 +281,7 @@ mappings: rpm: parted-devel libpcap: - deb: libpcap-dev + deb: libpcap0.8-dev pkg: libpcap rpm: libpcap-devel -- 2.20.1

We want the development headers not the client binary Reviewed-by: Andrea Bolognani <abologna(a)redhat.com&gt; Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com&gt; --- guests/vars/mappings.yml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/guests/vars/mappings.yml b/guests/vars/mappings.yml index e10e3da..0945e36 100644 --- a/guests/vars/mappings.yml +++ b/guests/vars/mappings.yml @@ -141,8 +141,12 @@ mappings: rpm: glibc-static glusterfs: - deb: glusterfs-client + deb: libglusterfs-dev rpm: glusterfs-api-devel + Debian8: glusterfs-common + Debian9: glusterfs-common + Ubuntu16: glusterfs-common + Ubuntu18: glusterfs-common gnome-common: default: gnome-common -- 2.20.1

On Wed, 2019-02-13 at 19:03 +0000, Daniel P. Berrangé wrote: [...] flake8 complains about this line being too long, so please rewrite it as raise Error( "Can't read inventory location in ansible.cfg: {}".format(ex) ) to make it happy. [...] This line is also too long and needs to be reformatted. [...] 'ex' should be an argument to format(), not Error(). This kind of stuff is exactly why Python is so much fun! :P You also need to reformat it to prevent flake8 from complaining about its length. With those issues addressed, Reviewed-by: Andrea Bolognani <abologna(a)redhat.com&gt; -- Andrea Bolognani / Red Hat / Virtualization

Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com&gt; --- guests/lcitool | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index 1271954..0978c40 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -529,15 +529,18 @@ class Application: if os_full in mappings[package]: temp[package] = mappings[package][os_full] - flattened = [] + pkgs = [] for item in temp: - if temp[item] is not None and temp[item] not in flattened: - flattened += [temp[item]] + pkgname = temp[item] + if pkgname is None: + continue + if pkgname not in pkgs: + pkgs.append(pkgname) print("FROM {}".format(facts["docker_base"])) sys.stdout.write("ENV PACKAGES ") - sys.stdout.write(" \\\n ".join(sorted(flattened))) + sys.stdout.write(" \\\n ".join(sorted(pkgs))) if package_format == "deb": sys.stdout.write(textwrap.dedent(""" -- 2.20.1

Every statement in a dockerfile results in a new layer in the image. There is no need for an env var to store the package list when it can be included inline. This avoids the env variable being later exposed to the container at runtime. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com&gt; --- guests/lcitool | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index 0978c40..8252dc2 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -539,34 +539,34 @@ class Application: print("FROM {}".format(facts["docker_base"])) - sys.stdout.write("ENV PACKAGES ") - sys.stdout.write(" \\\n ".join(sorted(pkgs))) - + varmap = {} + varmap["pkgs"] = " \\\n ".join(sorted(pkgs)) if package_format == "deb": sys.stdout.write(textwrap.dedent(""" RUN export DEBIAN_FRONTEND=noninteractive && \\ apt-get update && \\ apt-get dist-upgrade -y && \\ - apt-get install --no-install-recommends -y ${PACKAGES} && \\ + apt-get install --no-install-recommends -y \\ + {pkgs} && \\ apt-get autoremove -y && \\ apt-get autoclean -y - """)) + """).format(**varmap)) elif package_format == "rpm": if os_name == "Fedora" and os_version == "Rawhide": sys.stdout.write(textwrap.dedent(""" RUN yum update -y --nogpgcheck fedora-gpg-keys && \\ yum update -y && \\ - yum install -y ${PACKAGES} && \\ + yum install -y %(pkgs)s && \\ yum autoremove -y && \\ yum clean all -y - """)) + """).format(**varmap)) else: sys.stdout.write(textwrap.dedent(""" RUN yum update -y && \\ - yum install -y ${PACKAGES} && \\ + yum install -y %(pkgs)s && \\ yum autoremove -y && \\ yum clean all -y - """)) + """).format(**varmap)) def run(self): cmdline = self._parser.parse_args() -- 2.20.1

Debian's filesystem layout has a nice advantage over Fedora which is that it can install non-native RPMs in the main root filesystem. It is thus possible to prepare an x86_64 filesystem containing -dev packages for a foreign architecture, along with a GCC cross compiler. QEMU has used this technique to facilitate developer build testing of non-x86 architectures, since few people have access to physical hardware for most of these architectures. For the same reason it would be helpful to libvirt developers. This patch extends the 'dockerfile' command to 'lcitool' so that it accepts a '-x $ARCH' argument. $ lcitool -a dockerfile -h libvirt-debian-9 -p libvirt -x s390x This is only valid when using a 'deb' based distro. With the Debian 9 distro, this supports arm64, armel, armhf, mips, mipsel, mips64el, ppc64el, s390x, which are all the official archs that Debian maintainers currently build libvirt for. With Debian Sid, this is extended to include i386. Debian also builds libvirt on hppa, powerpcspe, sparc64 and x32 which are unofficial ports. Unfortunately it is not possible to reliably add foreign arch packages from the unofficial ports in parallel with those from the native arch, without hitting dependency problems from apt. When an architecture is given, any package name ending in '-dev' will be installed using that architecture variant, while all remaining packages will have their native variant installed. For various reasons (commented inline) a few blacklists are required on a per-arch basis. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com&gt; --- .../libvirt-debian-9/cross-build.yml | 50 ++++++++++++++ .../libvirt-debian-sid/cross-build.yml | 52 +++++++++++++++ guests/lcitool | 66 ++++++++++++++++--- 3 files changed, 159 insertions(+), 9 deletions(-) create mode 100644 guests/host_vars/libvirt-debian-9/cross-build.yml create mode 100644 guests/host_vars/libvirt-debian-sid/cross-build.yml diff --git a/guests/host_vars/libvirt-debian-9/cross-build.yml b/guests/host_vars/libvirt-debian-9/cross-build.yml new file mode 100644 index 0000000..9c6e511 --- /dev/null +++ b/guests/host_vars/libvirt-debian-9/cross-build.yml @@ -0,0 +1,50 @@ +--- +cross_build: + blacklist: + # apt doesn't properly resolve from foreign arch package + # to the native package of augeas-lenses + - libnetcf-dev + # apt fails to resolve deps from foreign arch + - wireshark-dev + # dtrace tool doesn't know how to cross-compile + - systemtap-sdt-dev + arches: + arm64: + target: aarch64-linux-gnu + armel: + target: arm-linux-gnueabi + blacklist: + # Not built on this arch + - libxen-dev + # Arch does not support NUMA concept + - libnuma-dev + armhf: + target: arm-linux-gnueabihf + blacklist: + # Arch does not support NUMA concept + - libnuma-dev + mips64el: + target: mips64el-linux-gnuabi64 + blacklist: + # Not built on this arch + - libxen-dev + mips: + target: mips-linux-gnu + blacklist: + # Not built on this arch + - libxen-dev + mipsel: + target: mipsel-linux-gnu + blacklist: + # Not built on this arch + - libxen-dev + ppc64el: + target: powerpc64le-linux-gnu + blacklist: + # Not built on this arch + - libxen-dev + s390x: + target: s390x-linux-gnu + blacklist: + # Not built on this arch + - libxen-dev diff --git a/guests/host_vars/libvirt-debian-sid/cross-build.yml b/guests/host_vars/libvirt-debian-sid/cross-build.yml new file mode 100644 index 0000000..7184083 --- /dev/null +++ b/guests/host_vars/libvirt-debian-sid/cross-build.yml @@ -0,0 +1,52 @@ +--- +cross_build: + blacklist: + # apt doesn't properly resolve from foreign arch package + # to the native package of augeas-lenses + - libnetcf-dev + # apt fails to resolve deps from foreign arch + - wireshark-dev + # dtrace tool doesn't know how to cross-compile + - systemtap-sdt-dev + arches: + arm64: + target: aarch64-linux-gnu + armel: + target: arm-linux-gnueabi + blacklist: + # Not built on this arch + - libxen-dev + # Arch does not support NUMA concept + - libnuma-dev + armhf: + target: arm-linux-gnueabihf + blacklist: + # Arch does not support NUMA concept + - libnuma-dev + mips64el: + target: mips64el-linux-gnuabi64 + blacklist: + # Not built on this arch + - libxen-dev + mips: + target: mips-linux-gnu + blacklist: + # Not built on this arch + - libxen-dev + mipsel: + target: mipsel-linux-gnu + blacklist: + # Not built on this arch + - libxen-dev + ppc64el: + target: powerpc64le-linux-gnu + blacklist: + # Not built on this arch + - libxen-dev + s390x: + target: s390x-linux-gnu + blacklist: + # Not built on this arch + - libxen-dev + i386: + target: i686-linux-gnu diff --git a/guests/lcitool b/guests/lcitool index 8252dc2..cc123a0 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -342,6 +342,11 @@ class Application: metavar="GIT_REVISION", help="git revision to build (remote/branch)", ) + self._parser.add_argument( + "-x", + metavar="CROSS_ARCH", + help="cross compiler architecture (dockerfile action only)", + ) def _execute_playbook(self, playbook, hosts, projects, git_revision): base = Util.get_base() @@ -401,15 +406,15 @@ class Application: except Exception as ex: raise Error("Failed to run {} on '{}': {}".format(playbook, hosts), ex) - def _action_hosts(self, _hosts, _projects, _revision): + def _action_hosts(self, _hosts, _projects, _revision, _cross_arch): for host in self._inventory.expand_pattern("all"): print(host) - def _action_projects(self, _hosts, _projects, _revision): + def _action_projects(self, _hosts, _projects, _revision, _cross_arch): for project in self._projects.expand_pattern("all"): print(project) - def _action_install(self, hosts, _projects, _revision): + def _action_install(self, hosts, _projects, _revision, _cross_arch): base = Util.get_base() flavor = self._config.get_flavor() @@ -481,13 +486,13 @@ class Application: except Exception as ex: raise Error("Failed to install '{}': {}".format(host, ex)) - def _action_update(self, hosts, projects, git_revision): + def _action_update(self, hosts, projects, git_revision, _cross_arch): self._execute_playbook("update", hosts, projects, git_revision) - def _action_build(self, hosts, projects, git_revision): + def _action_build(self, hosts, projects, git_revision, _cross_arch): self._execute_playbook("build", hosts, projects, git_revision) - def _action_dockerfile(self, hosts, projects, _revision): + def _action_dockerfile(self, hosts, projects, _revision, cross_arch): mappings = self._projects.get_mappings() hosts = self._inventory.expand_pattern(hosts) @@ -500,10 +505,22 @@ class Application: os_name = facts["os_name"] os_version = facts["os_version"] os_full = os_name + os_version + blacklist = [] + cross_build_facts = None if package_format not in ["deb", "rpm"]: raise Error("Host {} doesn't support Dockerfiles".format(host)) + if cross_arch is not None: + if "cross_build" not in facts: + raise Error("cross compilers not supported for this host") + if cross_arch not in facts["cross_build"]["arches"]: + raise Error("unsupported cross compiler architecture, use one of {}".format( + ", ".join(facts["cross_build"]["arches"].keys()))) + cross_build_facts = facts["cross_build"]["arches"][cross_arch] + blacklist.extend(facts["cross_build"].get("blacklist", [])) + blacklist.extend(cross_build_facts.get("blacklist", [])) + projects = self._projects.expand_pattern(projects) for project in projects: if project not in facts["projects"]: @@ -530,17 +547,29 @@ class Application: temp[package] = mappings[package][os_full] pkgs = [] + cross_pkgs = [] for item in temp: pkgname = temp[item] if pkgname is None: continue - if pkgname not in pkgs: - pkgs.append(pkgname) + if pkgname in blacklist: + continue + if cross_arch and pkgname[-4:] == "-dev": + if pkgname not in cross_pkgs: + cross_pkgs.append(pkgname + ":" + cross_arch) + else: + if pkgname not in pkgs: + pkgs.append(pkgname) print("FROM {}".format(facts["docker_base"])) varmap = {} varmap["pkgs"] = " \\\n ".join(sorted(pkgs)) + if cross_arch: + varmap["cross_arch"] = cross_arch + cross_pkgs.append("gcc-" + cross_build_facts["target"]) + varmap["cross_pkgs"] = " \\\n ".join(sorted(cross_pkgs)) + varmap["cross_target"] = cross_build_facts["target"] if package_format == "deb": sys.stdout.write(textwrap.dedent(""" RUN export DEBIAN_FRONTEND=noninteractive && \\ @@ -551,6 +580,24 @@ class Application: apt-get autoremove -y && \\ apt-get autoclean -y """).format(**varmap)) + if cross_arch: + # Intentionally a separate RUN command from the above + # so that the common packages of all cross-built images + # share a docker image layer. + sys.stdout.write(textwrap.dedent(""" + RUN export DEBIAN_FRONTEND=noninteractive && \\ + dpkg --add-architecture {cross_arch} && \\ + apt-get update && \\ + apt-get dist-upgrade -y && \\ + apt-get install --no-install-recommends -y \\ + {cross_pkgs} && \\ + apt-get autoremove -y && \\ + apt-get autoclean -y + + ENV TARGET "{cross_target}" + ENV CONFIGURE_OPTS "--host={cross_target} --target={cross_target}" + ENV PKG_CONFIG_LIBDIR "/usr/lib/{cross_target}/pkgconfig" + """).format(**varmap)) elif package_format == "rpm": if os_name == "Fedora" and os_version == "Rawhide": sys.stdout.write(textwrap.dedent(""" @@ -574,11 +621,12 @@ class Application: hosts = cmdline.h projects = cmdline.p git_revision = cmdline.g + cross_arch = cmdline.x method = "_action_{}".format(action.replace("-", "_")) if hasattr(self, method): - getattr(self, method).__call__(hosts, projects, git_revision) + getattr(self, method).__call__(hosts, projects, git_revision, cross_arch) else: raise Error("Invalid action '{}'".format(action)) -- 2.20.1