The libpcap-dev package is a temporary backcompat package everything switches to the new libpcap0.8-dev pacakge name.x Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com&gt; --- guests/vars/mappings.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guests/vars/mappings.yml b/guests/vars/mappings.yml index 84543c4..f211169 100644 --- a/guests/vars/mappings.yml +++ b/guests/vars/mappings.yml @@ -281,7 +281,7 @@ mappings: rpm: parted-devel libpcap: - deb: libpcap-dev + deb: libpcap0.8-dev pkg: libpcap rpm: libpcap-devel -- 2.20.1

On Tue, 2019-02-05 at 17:53 +0000, Daniel P. Berrangé wrote: [...] Did you actually run into a situation where this was useful? It's one of those diagnostics that I didn't really expect to trigger in practice... Either way, I don't really have a problem with adding it, but what I don't like is using a completely different way to format strings than the rest of the code. I don't use Python nearly enough to have an opinion on the merits of either syntax compared to the other, so if the one you're using here is considered a best practice then we can definitely switch to it; however, it will have to be done in a separate commit that converts the entire script at once rather than in a piecemail fashion. -- Andrea Bolognani / Red Hat / Virtualization

On Wed, Feb 06, 2019 at 03:56:37PM +0000, Daniel P. Berrangé wrote: I kind of agree with Andrea, the preferred formatting in python is to use {} and .format() method, the '%' formatting is old and should not be used. In this case you have these options: "Can't load facts for '{}': {}".format(host, ex) "Can't load facts for '{0}': {1}".format(host, ex) "Can't load facts for '{host}': {ex}".format(host=host, ex=ex) Pavel

We know exactly which packages we need and don't want apt picking extra "recommended" ones for us. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com&gt; --- guests/lcitool | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guests/lcitool b/guests/lcitool index 1d5cbd0..eb111b8 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -546,7 +546,7 @@ class Application: ( \\ apt-get update && \\ apt-get dist-upgrade -y && \\ - apt-get install -y ${PACKAGES} && \\ + apt-get install --no-install-recommends -y ${PACKAGES} && \\ apt-get autoremove -y && \\ apt-get autoclean -y \\ ) -- 2.20.1

Every statement in a dockerfile results in a new layer in the image. There is no need for an env var to store the package list when it can be included inline. This avoids the env variable being later exposed to the container at runtime. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com&gt; --- guests/lcitool | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index eb111b8..cd757eb 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -530,43 +530,46 @@ class Application: if os_full in mappings[package]: temp[package] = mappings[package][os_full] - flattened = [] + pkgs = [] for item in temp: - if temp[item] is not None and temp[item] not in flattened: - flattened += [temp[item]] + pkgname = temp[item] + if pkgname is None: + continue + if pkgname not in pkgs: + pkgs.append(pkgname) print("FROM {}".format(facts["docker_base"])) - sys.stdout.write("ENV PACKAGES ") - sys.stdout.write(" \\\n ".join(sorted(flattened))) - + varmap = {} + varmap["pkgs"] = "".join([" \\\n " + pkgname + for pkgname in sorted(pkgs)]) if package_format == "deb": sys.stdout.write(textwrap.dedent(""" RUN DEBIAN_FRONTEND=noninteractive && \\ ( \\ apt-get update && \\ apt-get dist-upgrade -y && \\ - apt-get install --no-install-recommends -y ${PACKAGES} && \\ + apt-get install --no-install-recommends -y %(pkgs)s && \\ apt-get autoremove -y && \\ apt-get autoclean -y \\ ) - """)) + """) % varmap ) elif package_format == "rpm": if os_name == "Fedora" and os_version == "Rawhide": sys.stdout.write(textwrap.dedent(""" RUN yum update -y --nogpgcheck fedora-gpg-keys && \\ yum update -y && \\ - yum install -y ${PACKAGES} && \\ + yum install -y %(pkgs)s && \\ yum autoremove -y && \\ yum clean all -y - """)) + """) % varmap ) else: sys.stdout.write(textwrap.dedent(""" RUN yum update -y && \\ - yum install -y ${PACKAGES} && \\ + yum install -y %(pkgs)s && \\ yum autoremove -y && \\ yum clean all -y - """)) + """) % varmap ) def run(self): cmdline = self._parser.parse_args() -- 2.20.1

Debian's filesystem layout has a nice advantage over Fedora which is that it can install non-native RPMs in the main root filesystem. It is thus possible to prepare an x86_64 filesystem containing -dev packages for a foreign architecture, along with a GCC cross compiler. QEMU has used this technique to facilitate developer build testing of non-x86 architectures, since few people have access to physical hardware for most of these architectures. For the same reason it would be helpful to libvirt developers. This patch extends the 'dockerfile' command to 'lcitool' so that it accepts a '-x $ARCH' argument. $ lcitool -a dockerfile -h libvirt-debian-9 -p libvirt -x s390x This is only valid when using a 'deb' based distro. With the Debian 9 distro, this supports arm64, armel, armhf, mips, mipsel, mips64el, ppc64el, s390x, which are all the official archs that Debian maintainers currently build libvirt for. With Debian Sid, this is extended to include i386. Debian also builds libvirt on hppa, powerpcspe, sparc64 and x32 which are unofficial ports. Unfortunately it is not possible to reliably add foreign arch packages from the unofficial ports in parallel with those from the native arch, without hitting dependency problems from apt. When an architecture is given, any package name ending in '-dev' will be installed using that architecture variant, while all remaining packages will have their native variant installed. For various reasons (commented inline) a few blacklists are required on a per-arch basis. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com&gt; --- guests/host_vars/libvirt-debian-9/docker.yml | 57 +++++++++++++++ .../host_vars/libvirt-debian-sid/docker.yml | 62 +++++++++++++++++ guests/lcitool | 69 ++++++++++++++++--- 3 files changed, 179 insertions(+), 9 deletions(-) diff --git a/guests/host_vars/libvirt-debian-9/docker.yml b/guests/host_vars/libvirt-debian-9/docker.yml index 0b4ccee..0e613da 100644 --- a/guests/host_vars/libvirt-debian-9/docker.yml +++ b/guests/host_vars/libvirt-debian-9/docker.yml @@ -1,2 +1,59 @@ --- docker_base: debian:9 +cross_build: + blacklist: + # Doesn't properly resolve from foreign arch package + # to the native package of augeas-lenses + - libnetcf-dev + # Fails to resolve deps from foreign arch + - wireshark-dev + # dtrace tool doesn't know how to cross-compile + - systemtap-sdt-dev + arches: + arm64: + gcc-pkg-prefix: crossbuild-essential-arm64 + target-prefix: aarch64-linux-gnu + armel: + gcc-pkg-prefix: crossbuild-essential-armel + target-prefix: arm-linux-gnueabi + blacklist: + # Not built on this arch + - libxen-dev + # Arch does not support NUMA concept + - libnuma-dev + armhf: + gcc-pkg-prefix: crossbuild-essential-armhf + target-prefix: arm-linux-gnueabihf + blacklist: + # Arch does not support NUMA concept + - libnuma-dev + mips64el: + gcc-pkg-prefix: gcc-mips64el-linux-gnuabi64 + target-prefix: mips64el-linux-gnuabi64 + blacklist: + # Not built on this arch + - libxen-dev + mips: + gcc-pkg-prefix: gcc-mips-linux-gnu + target-prefix: mips-linux-gnu + blacklist: + # Not built on this arch + - libxen-dev + mipsel: + gcc-pkg-prefix: gcc-mipsel-linux-gnu + target-prefix: mipsel-linux-gnu + blacklist: + # Not built on this arch + - libxen-dev + ppc64el: + gcc-pkg-prefix: crossbuild-essential-ppc64el + target-prefix: powerpc64le-linux-gnu + blacklist: + # Not built on this arch + - libxen-dev + s390x: + gcc-pkg-prefix: gcc-multilib-s390x-linux-gnu + target-prefix: s390x-linux-gnu + blacklist: + # Not built on this arch + - libxen-dev diff --git a/guests/host_vars/libvirt-debian-sid/docker.yml b/guests/host_vars/libvirt-debian-sid/docker.yml index e20a37e..61ecc1f 100644 --- a/guests/host_vars/libvirt-debian-sid/docker.yml +++ b/guests/host_vars/libvirt-debian-sid/docker.yml @@ -1,2 +1,64 @@ --- docker_base: debian:sid +cross_build: + blacklist: + # Doesn't properly resolve from foreign arch package + # to the native package of augeas-lenses + - libnetcf-dev + # Fails to resolve deps from foreign arch + - wireshark-dev + # dtrace tool doesn't know how to cross-compile + - systemtap-sdt-dev + # appears to be dropped from the 'sid' tree + - sheepdog + arches: + arm64: + gcc-pkg-prefix: crossbuild-essential-arm64 + target-prefix: aarch64-linux-gnu + armel: + gcc-pkg-prefix: crossbuild-essential-armel + target-prefix: arm-linux-gnueabi + blacklist: + # Not built on this arch + - libxen-dev + # Arch does not support NUMA concept + - libnuma-dev + armhf: + gcc-pkg-prefix: crossbuild-essential-armhf + target-prefix: arm-linux-gnueabihf + blacklist: + # Arch does not support NUMA concept + - libnuma-dev + mips64el: + gcc-pkg-prefix: gcc-mips64el-linux-gnuabi64 + target-prefix: mips64el-linux-gnuabi64 + blacklist: + # Not built on this arch + - libxen-dev + mips: + gcc-pkg-prefix: gcc-mips-linux-gnu + target-prefix: mips-linux-gnu + blacklist: + # Not built on this arch + - libxen-dev + mipsel: + gcc-pkg-prefix: gcc-mipsel-linux-gnu + target-prefix: mipsel-linux-gnu + blacklist: + # Not built on this arch + - libxen-dev + ppc64el: + gcc-pkg-prefix: crossbuild-essential-ppc64el + target-prefix: powerpc64le-linux-gnu + blacklist: + # Not built on this arch + - libxen-dev + s390x: + gcc-pkg-prefix: gcc-multilib-s390x-linux-gnu + target-prefix: s390x-linux-gnu + blacklist: + # Not built on this arch + - libxen-dev + i386: + gcc-pkg-prefix: gcc-multilib-i686-linux-gnu + target-prefix: i686-linux-gnu diff --git a/guests/lcitool b/guests/lcitool index cd757eb..dc5741e 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -343,6 +343,11 @@ class Application: metavar="GIT_REVISION", help="git revision to build (remote/branch)", ) + self._parser.add_argument( + "-x", + metavar="CROSS-ARCH", + help="cross compiler architecture (dockerfile action only)", + ) def _execute_playbook(self, playbook, hosts, projects, git_revision): base = Util.get_base() @@ -402,15 +407,15 @@ class Application: except Exception: raise Error("Failed to run {} on '{}'".format(playbook, hosts)) - def _action_hosts(self, _hosts, _projects, _revision): + def _action_hosts(self, _hosts, _projects, _revision, _cross_arch): for host in self._inventory.expand_pattern("all"): print(host) - def _action_projects(self, _hosts, _projects, _revision): + def _action_projects(self, _hosts, _projects, _revision, _cross_arch): for project in self._projects.expand_pattern("all"): print(project) - def _action_install(self, hosts, _projects, _revision): + def _action_install(self, hosts, _projects, _revision, _cross_arch): base = Util.get_base() flavor = self._config.get_flavor() @@ -482,13 +487,13 @@ class Application: except Exception: raise Error("Failed to install '{}'".format(host)) - def _action_update(self, hosts, projects, git_revision): + def _action_update(self, hosts, projects, git_revision, _cross_arch): self._execute_playbook("update", hosts, projects, git_revision) - def _action_build(self, hosts, projects, git_revision): + def _action_build(self, hosts, projects, git_revision, _cross_arch): self._execute_playbook("build", hosts, projects, git_revision) - def _action_dockerfile(self, hosts, projects, _revision): + def _action_dockerfile(self, hosts, projects, _revision, cross_arch): mappings = self._projects.get_mappings() hosts = self._inventory.expand_pattern(hosts) @@ -501,10 +506,24 @@ class Application: os_name = facts["os_name"] os_version = facts["os_version"] os_full = os_name + os_version + blacklist = [] + cross_build_facts = None if package_format not in ["deb", "rpm"]: raise Error("Host {} doesn't support Dockerfiles".format(host)) + if cross_arch is not None: + if package_format != "deb": + raise Error("cross compilers only supported for debian packages") + if "cross_build" not in facts: + raise Error("cross compilers not supported for this host") + if cross_arch not in facts["cross_build"]["arches"]: + raise Error("unsupported cross compiler architecture, use one of {}".format( + ", ".join(facts["cross_build"]["arches"].keys()))) + cross_build_facts = facts["cross_build"]["arches"][cross_arch] + blacklist.extend(facts["cross_build"].get("blacklist", [])) + blacklist.extend(cross_build_facts.get("blacklist", [])) + projects = self._projects.expand_pattern(projects) for project in projects: if project not in facts["projects"]: @@ -531,18 +550,31 @@ class Application: temp[package] = mappings[package][os_full] pkgs = [] + cross_pkgs = [] for item in temp: pkgname = temp[item] if pkgname is None: continue - if pkgname not in pkgs: - pkgs.append(pkgname) + if pkgname in blacklist: + continue + if cross_arch and pkgname[-4:] == "-dev": + if pkgname not in cross_pkgs: + cross_pkgs.append(pkgname + ":" + cross_arch) + else: + if pkgname not in pkgs: + pkgs.append(pkgname) print("FROM {}".format(facts["docker_base"])) varmap = {} varmap["pkgs"] = "".join([" \\\n " + pkgname for pkgname in sorted(pkgs)]) + if cross_arch: + varmap["cross_arch"] = cross_arch + varmap["cross_pkgs"] = "".join([" \\\n " + pkgname + for pkgname in sorted(cross_pkgs)]) + varmap["cross_gcc"] = cross_build_facts["gcc-pkg-prefix"] + varmap["cross_prefix"] = cross_build_facts["target-prefix"] if package_format == "deb": sys.stdout.write(textwrap.dedent(""" RUN DEBIAN_FRONTEND=noninteractive && \\ @@ -554,6 +586,24 @@ class Application: apt-get autoclean -y \\ ) """) % varmap ) + if cross_arch: + # Intentionally a separate RUN command from the above + # so that the common packages of all cross-built images + # share a docker image layer + sys.stdout.write(textwrap.dedent(""" + RUN DEBIAN_FRONTEND=noninteractive && \\ + ( \\ + dpkg --add-architecture %(cross_arch)s && \\ + apt-get update && \\ + apt-get dist-upgrade -y && \\ + apt-get install --no-install-recommends -y %(cross_gcc)s %(cross_pkgs)s && \\ + apt-get autoremove -y && \\ + apt-get autoclean -y \\ + ) + ENV TARGET "%(cross_prefix)s" + ENV CONFIGURE_OPTS "--host=%(cross_prefix)s --target=%(cross_prefix)s" + ENV PKG_CONFIG_LIBDIR "/usr/lib/%(cross_prefix)s/pkgconfig" + """) % varmap ) elif package_format == "rpm": if os_name == "Fedora" and os_version == "Rawhide": sys.stdout.write(textwrap.dedent(""" @@ -577,11 +627,12 @@ class Application: hosts = cmdline.h projects = cmdline.p git_revision = cmdline.g + cross_arch = cmdline.x method = "_action_{}".format(action.replace("-", "_")) if hasattr(self, method): - getattr(self, method).__call__(hosts, projects, git_revision) + getattr(self, method).__call__(hosts, projects, git_revision, cross_arch) else: raise Error("Invalid action '{}'".format(action)) -- 2.20.1

On Thu, Feb 07, 2019 at 04:21:34PM +0100, Andrea Bolognani wrote: Yes, the i386 gcc simply doesn't exist in 9. No idea why Ok. In practice this split doesn't seem to actually do anything since we load all files & take the union of their contents. dpkg fails to resolve the deps. It wants augeas-lenses:s390x when it should be satisfied with augeas-lenses:$NATIVE. You can't install both augeas-lenses as they conflict on files. I don't find that desirable. This is a debian specific config file, and it is clearer to list the actual debian packages we have trouble with, as that's what apt-get is complaining about when it fails. Using the generic mapping obscures what is going on. Actually this entire variable can be eliminated, as we can derive it from the target name. Using "cross_" is redundant as this already inside a parent "cross-build" block. Actually, I've eliminated all usage of the crossbuild-essential-* packages. These needlessly pull in the C++ compiler too. What's missing that we've not seen this problem already ? Is it just because we don't regularlyl rebuild the CI hosts / images ? Using the libvirt arch names adds no value. There's no friction as there is no context in which we're using the libvirt arch names & need a mapping to the cross build names. These image configs are distro specific and using the distro's native arch is directly relevant to the build results. The difference is the usage context. The ENV statement is intended to provide environment variables to runtime users of the image. The previous ENV usage was merely used internally to the docker build and was polluting the runtime env where it was not required. These ENV statements are intended for direct usage in the docker images at runtime, so you can launch the image with docker and simply type "$TARGET-gcc" or "./configure $CONFIGURE_OPTS" and have it do the right thing. The mingw images are slightly different in that the Fedora mingw RPMs have invented a "mingw32-configure" program that sets all these env variables already. If that program didn't exist, then it would be desirable to have the same env vars set for the mingw images too. Creating a single image would mean a significantly larger download image size when you only care about testing a single architecture. Because of the way image layer sharing works, having lots of separate images ensures you can have a minimal download size for a single image, while not giving duplication if you need to download all images. We ought to have a separate Fedora image for each MinGW target for this same reason in fact. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

Add a simple makefile that can generate dockerfiles and run local builds of them. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com&gt; --- .gitignore | 1 + dockerfiles/Makefile | 33 +++++++++++++++++++++++++++++++++ guests/lcitool | 15 +++++++++++++-- 3 files changed, 47 insertions(+), 2 deletions(-) create mode 100644 dockerfiles/Makefile diff --git a/.gitignore b/.gitignore index b25c15b..c92cb0e 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,2 @@ *~ +dockerfiles/*.docker diff --git a/dockerfiles/Makefile b/dockerfiles/Makefile new file mode 100644 index 0000000..0075840 --- /dev/null +++ b/dockerfiles/Makefile @@ -0,0 +1,33 @@ + +LCITOOL = ../guests/lcitool + +HOSTS = $(shell $(LCITOOL) -a hosts) + +CROSS_HOST = libvirt-debian-9 + +ARCHES = $(shell $(LCITOOL) -a arches -h $(CROSS_HOST)) + +CONFIGS = $(wildcard ../guests/host_vars/*/docker.yml) + +DOCKERFILES = $(shell $(LCITOOL) -a dockerfiles) + +DOCKERIMAGES = $(DOCKERFILES:%.docker=%) + +all: $(DOCKERFILES) + +buildenv-%.docker: $(LCITOOL) $(CONFIGS) + case $@ in \ + *cross*) \ + HOST=`echo $@ | sed -e 's/-cross-.*//' -e 's/buildenv-//' -e 's/.docker//'` && \ + ARCH=`echo $@ | sed -e 's/^.*-cross-//' -e 's/.docker//'` && \ + $(LCITOOL) -a dockerfile -p libvirt -h $$HOST -x $$ARCH > $@ ;; \ + *) \ + HOST=`echo $@ | sed -e 's/buildenv-//' -e 's/.docker//'` && \ + $(LCITOOL) -a dockerfile -p libvirt -h $$HOST > $@ ;; \ + esac + +buildenv-%: buildenv-%.docker + docker build --tag $@ -f $< . + +clean: + rm -f *.docker *~ diff --git a/guests/lcitool b/guests/lcitool index dc5741e..a5ae817 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -313,8 +313,9 @@ class Application: build build projects on hosts informational actions: - hosts list all known hosts - projects list all known projects + hosts list all known hosts + projects list all known projects + dockerfiles list all known dockerfiles uncommon actions: dockerfile generate Dockerfile (doesn't access the host) @@ -415,6 +416,16 @@ class Application: for project in self._projects.expand_pattern("all"): print(project) + def _action_dockerfiles(self, _hosts, _projects, _revision, _cross_arch): + for host in self._inventory.expand_pattern("all"): + facts = self._inventory.get_facts(host) + package_format = facts["package_format"] + if package_format not in ["deb", "rpm"]: + continue + print ("buildenv-%s.docker" % host) + for arch in facts.get("cross_build", {}).get("arches", {}).keys(): + print ("buildenv-%s-cross-%s.docker" % (host, arch)) + def _action_install(self, hosts, _projects, _revision, _cross_arch): base = Util.get_base() -- 2.20.1