10:33 a.m.
Changed in v4:
- Pull in change to use argparse sub-parsers
- Refactor way architecture specific package rules are stored
to be in the main package mappings data
Changed in v3:
- Remove sheepdog more generally
- Use .format() style printf
- Split config to cross-build.yml
- Make glusterfs name per-distro customized
- Misc code style changes
- Rename fields in cross-build.yml
- Don't use crossbuild-essential packages
Changed in v2:
- Fix multiple package name mistakes
- Modify lcitool to generate cross-arch docker files
- Add --no-install-recommended flag to apt-get
- Add DEBIAN_FRONTEND=noninteractive env to apt-get
- Improve error reporting in lcitool
- Add make rule for generating dockerfiles locally
Daniel P. Berrangé (5):
lcitool: use subparsers for commands
lcitool: avoid repetition when expanding package mappings
lcitool: avoid intermediate list of packages
mappings: extend mapping to allow per-arch entries
lcitool: support generating cross compiler dockerfiles
guests/host_vars/libvirt-debian-9/main.yml | 44 ++++
guests/host_vars/libvirt-debian-sid/main.yml | 45 ++++
guests/lcitool | 236 +++++++++++++------
guests/playbooks/update/tasks/packages.yml | 32 +++
guests/vars/mappings.yml | 101 +++++++-
5 files changed, 379 insertions(+), 79 deletions(-)
--
2.20.1
10:33 a.m.
New subject: [libvirt] [jenkins-ci PATCH v4 1/5] lcitool: use subparsers for commands
Currently only a single global parser is used for all commands. This
means that every command accepts every argument which is undesirable as
users don't know what to pass. It also prevents the parser from
generating useful help information.
Python's argparse module supports multi-command binaries in a very easy
way by adding subparsers, each of which has their own distinct
arguments. It can then generate suitable help text on a per command
basis. This also means commands can use positional arguments for data
items that are always passed.
$ ./guests/lcitool -h
usage: lcitool [-h] {install,update,build,hosts,projects,dockerfile} ...
libvirt CI guest management tool
positional arguments:
{install,update,build,hosts,projects,dockerfile}
commands
install perform unattended host installation
update prepare hosts and keep them updated
build build projects on hosts
hosts list all known hosts
projects list all known projects
dockerfile generate a host docker file
optional arguments:
-h, --help show this help message and exit
$ ./guests/lcitool install -h
usage: lcitool install [-h] hosts
positional arguments:
hosts list of hosts to act on (accepts globs)
optional arguments:
-h, --help show this help message and exit
$ ./guests/lcitool dockerfile -h
usage: lcitool dockerfile [-h] hosts projects
positional arguments:
hosts list of hosts to act on (accepts globs)
projects list of projects to consider (accepts globs)
optional arguments:
-h, --help show this help message and exit
Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com>
---
guests/lcitool | 135 ++++++++++++++++++++++++++++---------------------
1 file changed, 78 insertions(+), 57 deletions(-)
diff --git a/guests/lcitool b/guests/lcitool
index 726e3bb..35a6b68 100755
--- a/guests/lcitool
+++ b/guests/lcitool
@@ -307,43 +307,72 @@ class Application:
conflict_handler="resolve",
formatter_class=argparse.RawDescriptionHelpFormatter,
description="libvirt CI guest management tool",
- epilog=textwrap.dedent("""
- common actions:
- install perform unattended host installation
- update prepare hosts and keep them updated
- build build projects on hosts
+ )
- informational actions:
- hosts list all known hosts
- projects list all known projects
+ subparser = self._parser.add_subparsers(help="commands")
+ subparser.required = True
+ subparser.dest = "command"
- uncommon actions:
- dockerfile generate Dockerfile (doesn't access the host)
+ def add_hosts_arg(parser):
+ parser.add_argument(
+ "hosts",
+ help="list of hosts to act on (accepts globs)",
+ )
- glob patterns are supported for HOSTS and PROJECTS
- """),
- )
- self._parser.add_argument(
- "-a",
- metavar="ACTION",
- required=True,
- help="action to perform (see below)",
- )
- self._parser.add_argument(
- "-h",
- metavar="HOSTS",
- help="list of hosts to act on",
- )
- self._parser.add_argument(
- "-p",
- metavar="PROJECTS",
- help="list of projects to consider",
- )
- self._parser.add_argument(
- "-g",
- metavar="GIT_REVISION",
- help="git revision to build (remote/branch)",
- )
+ def add_projects_arg(parser):
+ parser.add_argument(
+ "projects",
+ help="list of projects to consider (accepts globs)",
+ )
+
+ def add_gitrev_arg(parser):
+ parser.add_argument(
+ "-g", "--git-revision",
+ help="git revision to build (remote/branch)",
+ )
+
+ installparser = subparser.add_parser(
+ "install", help="perform unattended host installation",
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ installparser.set_defaults(func=self._action_install)
+
+ add_hosts_arg(installparser)
+
+ updateparser = subparser.add_parser(
+ "update", help="prepare hosts and keep them updated",
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ updateparser.set_defaults(func=self._action_update)
+
+ add_hosts_arg(updateparser)
+ add_projects_arg(updateparser)
+ add_gitrev_arg(updateparser)
+
+ buildparser = subparser.add_parser(
+ "build", help="build projects on hosts",
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ buildparser.set_defaults(func=self._action_build)
+
+ add_hosts_arg(buildparser)
+ add_projects_arg(buildparser)
+ add_gitrev_arg(buildparser)
+
+ hostsparser = subparser.add_parser(
+ "hosts", help="list all known hosts",
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ hostsparser.set_defaults(func=self._action_hosts)
+
+ projectsparser = subparser.add_parser(
+ "projects", help="list all known projects",
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ projectsparser.set_defaults(func=self._action_projects)
+
+ dockerfileparser = subparser.add_parser(
+ "dockerfile", help="generate a host docker file",
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ dockerfileparser.set_defaults(func=self._action_dockerfile)
+
+ add_hosts_arg(dockerfileparser)
+ add_projects_arg(dockerfileparser)
def _execute_playbook(self, playbook, hosts, projects, git_revision):
base = Util.get_base()
@@ -404,20 +433,20 @@ class Application:
raise Error(
"Failed to run {} on '{}': {}".format(playbook, hosts,
ex))
- def _action_hosts(self, _hosts, _projects, _revision):
+ def _action_hosts(self, args):
for host in self._inventory.expand_pattern("all"):
print(host)
- def _action_projects(self, _hosts, _projects, _revision):
+ def _action_projects(self, args):
for project in self._projects.expand_pattern("all"):
print(project)
- def _action_install(self, hosts, _projects, _revision):
+ def _action_install(self, args):
base = Util.get_base()
flavor = self._config.get_flavor()
- for host in self._inventory.expand_pattern(hosts):
+ for host in self._inventory.expand_pattern(args.hosts):
facts = self._inventory.get_facts(host)
# Both memory size and disk size are stored as GiB in the
@@ -484,16 +513,18 @@ class Application:
except Exception as ex:
raise Error("Failed to install '{}': {}".format(host,
ex))
- def _action_update(self, hosts, projects, git_revision):
- self._execute_playbook("update", hosts, projects, git_revision)
+ def _action_update(self, args):
+ self._execute_playbook("update", args.hosts, args.projects,
+ args.git_revision)
- def _action_build(self, hosts, projects, git_revision):
- self._execute_playbook("build", hosts, projects, git_revision)
+ def _action_build(self, args):
+ self._execute_playbook("build", args.hosts, args.projects,
+ args.git_revision)
- def _action_dockerfile(self, hosts, projects, _revision):
+ def _action_dockerfile(self, args):
mappings = self._projects.get_mappings()
- hosts = self._inventory.expand_pattern(hosts)
+ hosts = self._inventory.expand_pattern(args.hosts)
if len(hosts) > 1:
raise Error("Can't generate Dockerfile for multiple hosts")
host = hosts[0]
@@ -507,7 +538,7 @@ class Application:
if package_format not in ["deb", "rpm"]:
raise Error("Host {} doesn't support
Dockerfiles".format(host))
- projects = self._projects.expand_pattern(projects)
+ projects = self._projects.expand_pattern(args.projects)
for project in projects:
if project not in facts["projects"]:
raise Error(
@@ -573,18 +604,8 @@ class Application:
""").format(**varmap))
def run(self):
- cmdline = self._parser.parse_args()
- action = cmdline.a
- hosts = cmdline.h
- projects = cmdline.p
- git_revision = cmdline.g
-
- method = "_action_{}".format(action.replace("-",
"_"))
-
- if hasattr(self, method):
- getattr(self, method).__call__(hosts, projects, git_revision)
- else:
- raise Error("Invalid action '{}'".format(action))
+ args = self._parser.parse_args()
+ args.func(args)
if __name__ == "__main__":
--
2.20.1
7:43 a.m.
New subject: [libvirt] [jenkins-ci PATCH v4 1/5] lcitool: use subparsers for commands
On Thu, 2019-02-21 at 16:33 +0000, Daniel P. Berrangé wrote:
Currently only a single global parser is used for all commands. This
means that every command accepts every argument which is undesirable as
users don't know what to pass. It also prevents the parser from
generating useful help information.
Python's argparse module supports multi-command binaries in a very easy
way by adding subparsers, each of which has their own distinct
arguments. It can then generate suitable help text on a per command
basis. This also means commands can use positional arguments for data
items that are always passed.
$ ./guests/lcitool -h
usage: lcitool [-h] {install,update,build,hosts,projects,dockerfile} ...
libvirt CI guest management tool
positional arguments:
{install,update,build,hosts,projects,dockerfile}
commands
install perform unattended host installation
update prepare hosts and keep them updated
build build projects on hosts
hosts list all known hosts
projects list all known projects
dockerfile generate a host docker file
optional arguments:
-h, --help show this help message and exit
$ ./guests/lcitool install -h
usage: lcitool install [-h] hosts
positional arguments:
hosts list of hosts to act on (accepts globs)
optional arguments:
-h, --help show this help message and exit
$ ./guests/lcitool dockerfile -h
usage: lcitool dockerfile [-h] hosts projects
positional arguments:
hosts list of hosts to act on (accepts globs)
projects list of projects to consider (accepts globs)
optional arguments:
-h, --help show this help message and exit
I feel like the usage examples are not really adding much to the
commit message, so I'd just leave them out.
On the other hand, all the examples in README.markdown are now
invalid, so you should make sure that file is updated along with
the script.
[...]
+++ b/guests/lcitool
@@ -307,43 +307,72 @@ class Application:
conflict_handler="resolve",
formatter_class=argparse.RawDescriptionHelpFormatter,
Since we're no longer providing our own, pre-formatted help text
and we're leaving its generation up to argparse instead, setting
formatter_class is no longer needed.
The same applies to all callers you introduce later.
[...]
+ subparser =
self._parser.add_subparsers(help="commands")
Please rename the variable to 'subparsers'. At the same time, drop
use of the 'help' parameter and set 'metavar' to 'ACTION'
instead:
that will change the main help output from
usage: lcitool [-h] {install,update,build,hosts,projects,dockerfile} ...
libvirt CI guest management tool
positional arguments:
{install,update,build,hosts,projects,dockerfile}
commands
install perform unattended host installation
...
to a less unwieldy
usage: lcitool [-h] ACTION ...
libvirt CI guest management tool
positional arguments:
ACTION
install perform unattended host installation
...
As an aside, I don't much like how argparse takes over the '-h'
option as a shorthand for '--help', especially since up until now
it has been used to specify hosts, but I haven't found a way to
disable that behavior. I guess I can live with it :)
+ subparser.required = True
+ subparser.dest = "command"
Setting the 'dest' parameter is not necessary - you're not using
it anywhere.
[...]
+ installparser.set_defaults(func=self._action_install)
Well, isn't this clever :)
[...]
+ dockerfileparser = subparser.add_parser(
+ "dockerfile", help="generate a host docker file",
I'd prefer if you kept the original help text here.
Everything else looks good.
--
Andrea Bolognani / Red Hat / Virtualization
10:33 a.m.
New subject: [libvirt] [jenkins-ci PATCH v4 2/5] lcitool: avoid repetition when expanding package mappings
Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com>
---
guests/lcitool | 12 ++++--------
1 file changed, 4 insertions(+), 8 deletions(-)
diff --git a/guests/lcitool b/guests/lcitool
index 35a6b68..5c2b785 100755
--- a/guests/lcitool
+++ b/guests/lcitool
@@ -550,18 +550,14 @@ class Application:
temp = {}
+ keys = ["default", package_format, os_name, os_full]
# We need to add the base project manually here: the standard
# machinery hides it because it's an implementation detail
for project in projects + ["base"]:
for package in self._projects.get_packages(project):
- if "default" in mappings[package]:
- temp[package] = mappings[package]["default"]
- if package_format in mappings[package]:
- temp[package] = mappings[package][package_format]
- if os_name in mappings[package]:
- temp[package] = mappings[package][os_name]
- if os_full in mappings[package]:
- temp[package] = mappings[package][os_full]
+ for key in keys:
+ if key in mappings[package]:
+ temp[package] = mappings[package][key]
pkgs = []
for item in temp:
--
2.20.1
9:57 a.m.
New subject: [libvirt] [jenkins-ci PATCH v4 2/5] lcitool: avoid repetition when expanding package mappings
On Thu, 2019-02-21 at 16:33 +0000, Daniel P. Berrangé wrote:
[...]
+ keys = ["default", package_format, os_name,
os_full]
# We need to add the base project manually here: the standard
# machinery hides it because it's an implementation detail
for project in projects + ["base"]:
for package in self._projects.get_packages(project):
- if "default" in mappings[package]:
- temp[package] = mappings[package]["default"]
- if package_format in mappings[package]:
- temp[package] = mappings[package][package_format]
- if os_name in mappings[package]:
- temp[package] = mappings[package][os_name]
- if os_full in mappings[package]:
- temp[package] = mappings[package][os_full]
+ for key in keys:
+ if key in mappings[package]:
+ temp[package] = mappings[package][key]
Little historical note: the reason why the code looked like this in
the first place[1] is that its structure was supposed to mirror that
of playbooks/update/tasks/packages.yml as closely as possible - the
idea being that, if the Ansible implementation was correct, then the
Python one would most likely be as well.
Of course that's no longer the case as of dcded110e102, so it makes
perfect sense to go further down this road and make the code more
compact.
Reviewed-by: Andrea Bolognani <abologna(a)redhat.com>
[1] In addition to Python being admittedly not my forte :)
--
Andrea Bolognani / Red Hat / Virtualization
10:33 a.m.
New subject: [libvirt] [jenkins-ci PATCH v4 3/5] lcitool: avoid intermediate list of packages
We can purge any packages which expand to None straight away, and
simply convert to a set to get rid of duplicates.
Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com>
---
guests/lcitool | 17 +++++------------
1 file changed, 5 insertions(+), 12 deletions(-)
diff --git a/guests/lcitool b/guests/lcitool
index 5c2b785..2af6878 100755
--- a/guests/lcitool
+++ b/guests/lcitool
@@ -548,8 +548,7 @@ class Application:
)
)
- temp = {}
-
+ pkgs = {}
keys = ["default", package_format, os_name, os_full]
# We need to add the base project manually here: the standard
# machinery hides it because it's an implementation detail
@@ -557,21 +556,15 @@ class Application:
for package in self._projects.get_packages(project):
for key in keys:
if key in mappings[package]:
- temp[package] = mappings[package][key]
+ pkgs[package] = mappings[package][key]
- pkgs = []
- for item in temp:
- pkgname = temp[item]
- if pkgname is None:
- continue
- if pkgname in pkgs:
- continue
- pkgs.append(pkgname)
+ if pkgs[package] is None:
+ del pkgs[package]
print("FROM {}".format(facts["docker_base"]))
varmap = {}
- varmap["pkgs"] = " \\\n ".join(sorted(pkgs))
+ varmap["pkgs"] = " \\\n
".join(sorted(set(pkgs.values())))
if package_format == "deb":
sys.stdout.write(textwrap.dedent("""
RUN export DEBIAN_FRONTEND=noninteractive && \\
--
2.20.1
11:06 a.m.
New subject: [libvirt] [jenkins-ci PATCH v4 3/5] lcitool: avoid intermediate list of packages
On Thu, 2019-02-21 at 16:33 +0000, Daniel P. Berrangé wrote:
We can purge any packages which expand to None straight away, and
simply convert to a set to get rid of duplicates.
Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com>
---
guests/lcitool | 17 +++++------------
1 file changed, 5 insertions(+), 12 deletions(-)
This doesn't work:
$ ./lcitool dockerfile libvirt-centos-7 libvirt
Traceback (most recent call last):
File "./lcitool", line 601, in <module>
Application().run()
File "./lcitool", line 596, in run
args.func(args)
File "./lcitool", line 560, in _action_dockerfile
if pkgs[package] is None:
KeyError: 'apparmor'
That's because...
@@ -557,21 +556,15 @@ class Application:
for package in self._projects.get_packages(project):
for key in keys:
if key in mappings[package]:
- temp[package] = mappings[package][key]
+ pkgs[package] = mappings[package][key]
... the hunk
if package not in pkgs:
continue
from the following patch should have been included right about here.
It would be nice if the Ansible part would be converted to a similar
logic, which should be possible using the appropriate Jinja2 filters,
but that can wait until later.
If you include the hunk mentioned above,
Reviewed-by: Andrea Bolognani <abologna(a)redhat.com>
--
Andrea Bolognani / Red Hat / Virtualization
10:33 a.m.
New subject: [libvirt] [jenkins-ci PATCH v4 4/5] mappings: extend mapping to allow per-arch entries
For example to prevent Xen being installed on any s390x
xen:
default-s390x:
deb: libxen-dev
Fedora: xen-devel
Or the inverse to only install Xen on x86_64 on Debian, but allow
it on all archs on Fedora
xen:
deb-x86_64: libxen-dev
Fedora: xen-devel
Note that the architecture specific matches are considered after
all the non-architcture matches.
The mappings are updated to blacklist libnuma on arm6/7 for Debian
since it is not built for those archs. xen is whitelisted to only
be used on x86_64, arm7 and aarch64 for Debian, since the majority
of other architectures don't support it.
The dockerfile generator is updated to apply arch filtering based
on the host arch.
The ansible playbook is not actually implementing support for
non-x86_64 architectures in this commit. It is just hardcoding
x86_64, which is enough to ensure the changes in the mappings.yml
file are a no-op initially.
Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com>
---
guests/lcitool | 17 ++++++++-
guests/playbooks/update/tasks/packages.yml | 32 +++++++++++++++++
guests/vars/mappings.yml | 42 +++++++++++++++++++---
3 files changed, 86 insertions(+), 5 deletions(-)
diff --git a/guests/lcitool b/guests/lcitool
index 2af6878..ae7e4ee 100755
--- a/guests/lcitool
+++ b/guests/lcitool
@@ -21,6 +21,7 @@ import crypt
import fnmatch
import json
import os
+import platform
import random
import string
import subprocess
@@ -303,6 +304,8 @@ class Application:
self._inventory = Inventory()
self._projects = Projects()
+ self._native_arch = self._libvirt_host_arch()
+
self._parser = argparse.ArgumentParser(
conflict_handler="resolve",
formatter_class=argparse.RawDescriptionHelpFormatter,
@@ -433,6 +436,15 @@ class Application:
raise Error(
"Failed to run {} on '{}': {}".format(playbook, hosts,
ex))
+ def _libvirt_host_arch(self):
+ # Same canonicalization as libvirt virArchFromHost
+ arch = platform.machine()
+ if arch in ["i386", "i486", "i586"]:
+ arch = "i686"
+ if arch == "amd64":
+ arch = "x86_64"
+ return arch
+
def _action_hosts(self, args):
for host in self._inventory.expand_pattern("all"):
print(host)
@@ -549,7 +561,8 @@ class Application:
)
pkgs = {}
- keys = ["default", package_format, os_name, os_full]
+ base_keys = ["default", package_format, os_name, os_full]
+ keys = base_keys + [k + "-" + self._native_arch for k in base_keys]
# We need to add the base project manually here: the standard
# machinery hides it because it's an implementation detail
for project in projects + ["base"]:
@@ -558,6 +571,8 @@ class Application:
if key in mappings[package]:
pkgs[package] = mappings[package][key]
+ if package not in pkgs:
+ continue
if pkgs[package] is None:
del pkgs[package]
diff --git a/guests/playbooks/update/tasks/packages.yml
b/guests/playbooks/update/tasks/packages.yml
index 7fdfc45..b3b8a27 100644
--- a/guests/playbooks/update/tasks/packages.yml
+++ b/guests/playbooks/update/tasks/packages.yml
@@ -52,6 +52,38 @@
when:
- mappings[item][os_name + os_version] is defined
+- name: '{{ project }}: Look up mappings (default with arch)'
+ set_fact:
+ temp: '{{ temp|combine({ item: mappings[item]["default" + "-"
+ "x86_64"] }) }}'
+ with_items:
+ '{{ packages }}'
+ when:
+ - mappings[item]["default" + "-" + "x86_64"] is
defined
+
+- name: '{{ project }}: Look up mappings (package format with arch)'
+ set_fact:
+ temp: '{{ temp|combine({ item: mappings[item][package_format + "-" +
"x86_64"] }) }}'
+ with_items:
+ '{{ packages }}'
+ when:
+ - mappings[item][package_format + "-" + "x86_64"] is defined
+
+- name: '{{ project }}: Look up mappings (OS name with arch)'
+ set_fact:
+ temp: '{{ temp|combine({ item: mappings[item][os_name + "-" +
"x86_64"] }) }}'
+ with_items:
+ '{{ packages }}'
+ when:
+ - mappings[item][os_name + "-" + "x86_64"] is defined
+
+- name: '{{ project }}: Look up mappings (OS version with arch)'
+ set_fact:
+ temp: '{{ temp|combine({ item: mappings[item][os_name + os_version +
"-" + "x86_64"] }) }}'
+ with_items:
+ '{{ packages }}'
+ when:
+ - mappings[item][os_name + os_version + "-" + "x86_64"] is
defined
+
- set_fact:
flattened: []
diff --git a/guests/vars/mappings.yml b/guests/vars/mappings.yml
index f31b460..4ca8fae 100644
--- a/guests/vars/mappings.yml
+++ b/guests/vars/mappings.yml
@@ -7,9 +7,25 @@
# priority:
#
# - default
-# - package format (deb, pkg, rpm)
-# - OS name (CentOS, Debian, Fedora, FreeBSD, Ubuntu)
-# - OS version (CentOS7, Debian9, FedoraRawhide, Ubuntu18 and so on)
+# - package format
+# - OS name
+# - OS version
+# - default with arch
+# - package format with arch
+# - OS name with arch
+# - OS version with arch
+#
+# Valid package formats are
+# - deb, pkg, rpm
+#
+# Valid OS names are:
+# - CentOS, Debian, Fedora, FreeBSD, Ubuntu
+#
+# Valid OS versions are:
+# - CentOS7, Debian9, FedoraRawhide, Ubuntu18 and so on
+#
+# The 'with arch' levels take a suffix "-$ARCH" where $ARCH
+# is a libvirt arch name.
#
# So something like
#
@@ -27,6 +43,20 @@
#
# will result in the 'ccache' package being installed everywhere except
# for CentOS, where nothing will be installed.
+#
+# For example to prevent Xen being installed on any s390x
+#
+# xen:
+# default-s390x:
+# deb: libxen-dev
+# Fedora: xen-devel
+#
+# Or the inverse to only install Xen on x86_64
+#
+# xen:
+# deb-x86_64: libxen-dev
+# Fedora-x86_64: xen-devel
+#
mappings:
@@ -278,6 +308,8 @@ mappings:
libnuma:
deb: libnuma-dev
+ deb-armv6l:
+ deb-armv7l:
rpm: numactl-devel
libparted:
@@ -817,7 +849,9 @@ mappings:
Debian8:
xen:
- deb: libxen-dev
+ deb-x86_64: libxen-dev
+ deb-armv7l: libxen-dev
+ deb-aarch64: libxen-dev
Fedora: xen-devel
xfsprogs:
--
2.20.1
10:33 a.m.
New subject: [libvirt] [jenkins-ci PATCH v4 5/5] lcitool: support generating cross compiler dockerfiles
Debian's filesystem layout has a nice advantage over Fedora which is
that it can install non-native RPMs in the main root filesystem. It is
thus possible to prepare an x86_64 filesystem containing -dev packages
for a foreign architecture, along with a GCC cross compiler.
QEMU has used this technique to facilitate developer build testing of
non-x86 architectures, since few people have access to physical
hardware for most of these architectures. For the same reason it would
be helpful to libvirt developers.
This patch extends the 'dockerfile' command to 'lcitool' so that it
accepts a '-x $ARCH' argument.
$ lcitool dockerfile -x s390x libvirt-debian-9 libvirt
This is only valid when using a 'deb' based distro.
Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com>
---
guests/host_vars/libvirt-debian-9/main.yml | 44 +++++++++++++
guests/host_vars/libvirt-debian-sid/main.yml | 45 ++++++++++++++
guests/lcitool | 65 +++++++++++++++++++-
guests/vars/mappings.yml | 59 ++++++++++++++++++
4 files changed, 211 insertions(+), 2 deletions(-)
diff --git a/guests/host_vars/libvirt-debian-9/main.yml
b/guests/host_vars/libvirt-debian-9/main.yml
index ec7e6b4..3bf4ae1 100644
--- a/guests/host_vars/libvirt-debian-9/main.yml
+++ b/guests/host_vars/libvirt-debian-9/main.yml
@@ -21,3 +21,47 @@ os_name: 'Debian'
os_version: '9'
ansible_python_interpreter: /usr/bin/python3
+
+arches:
+ aarch64:
+ package_arch: arm64
+ abi: aarch64-linux-gnu
+ cross_gcc: gcc-aarch64-linux-gnu
+ armv6l:
+ package_arch: armel
+ abi: arm-linux-gnueabi
+ cross_gcc: gcc-arm-linux-gnueabi
+ armv7l:
+ package_arch: armhf
+ abi: arm-linux-gnueabihf
+ cross_gcc: gcc-arm-linux-gnueabihf
+ i686:
+ package_arch: i386
+ abi: i386-linux-gnu
+ mips:
+ package_arch: mips
+ abi: mips-linux-gnu
+ cross_gcc: gcc-mips-linux-gnu
+ mipsel:
+ package_arch: mipsel
+ abi: mipsel-linux-gnu
+ cross_gcc: gcc-mipsel-linux-gnu
+ mips64:
+ package_arch: mips64
+ abi: mips64-linux-gnu
+ cross_gcc: gcc-mips64-linux-gnu
+ mips64el:
+ package_arch: mips64el
+ abi: mips64el-linux-gnu
+ cross_gcc: gcc-mips64el-linux-gnu
+ ppc64el:
+ package_arch: ppc64el
+ abi: ppc64el-linux-gnu
+ cross_gcc: gcc-ppc64el-linux-gnu
+ s390x:
+ package_arch: s390x
+ abi: s390x-linux-gnu
+ cross_gcc: gcc-s390x-linux-gnu
+ x86_64:
+ package_arch: amd64
+ abi: x86_64-linux-gnu
diff --git a/guests/host_vars/libvirt-debian-sid/main.yml
b/guests/host_vars/libvirt-debian-sid/main.yml
index 1c7a29b..b14a564 100644
--- a/guests/host_vars/libvirt-debian-sid/main.yml
+++ b/guests/host_vars/libvirt-debian-sid/main.yml
@@ -21,3 +21,48 @@ os_name: 'Debian'
os_version: 'Sid'
ansible_python_interpreter: /usr/bin/python3
+
+arches:
+ aarch64:
+ package_arch: arm64
+ abi: aarch64-linux-gnu
+ cross_gcc: gcc-aarch64-linux-gnu
+ armv6l:
+ package_arch: armel
+ abi: arm-linux-gnueabi
+ cross_gcc: gcc-arm-linux-gnueabi
+ armv7l:
+ package_arch: armhf
+ abi: arm-linux-gnueabihf
+ cross_gcc: gcc-arm-linux-gnueabihf
+ i686:
+ package_arch: i386
+ abi: i686-linux-gnu
+ cross_gcc: gcc-i686-linux-gnu
+ mips:
+ package_arch: mips
+ abi: mips-linux-gnu
+ cross_gcc: gcc-mips-linux-gnu
+ mipsel:
+ package_arch: mipsel
+ abi: mipsel-linux-gnu
+ cross_gcc: gcc-mipsel-linux-gnu
+ mips64:
+ package_arch: mips64
+ abi: mips64-linux-gnu
+ cross_gcc: gcc-mips64-linux-gnu
+ mips64el:
+ package_arch: mips64el
+ abi: mips64el-linux-gnu
+ cross_gcc: gcc-mips64el-linux-gnu
+ ppc64el:
+ package_arch: ppc64el
+ abi: ppc64el-linux-gnu
+ cross_gcc: gcc-ppc64el-linux-gnu
+ s390x:
+ package_arch: s390x
+ abi: s390x-linux-gnu
+ cross_gcc: gcc-s390x-linux-gnu
+ x86_64:
+ package_arch: amd64
+ abi: x86_64-linux-gnu
diff --git a/guests/lcitool b/guests/lcitool
index ae7e4ee..b431b93 100755
--- a/guests/lcitool
+++ b/guests/lcitool
@@ -376,6 +376,10 @@ class Application:
add_hosts_arg(dockerfileparser)
add_projects_arg(dockerfileparser)
+ dockerfileparser.add_argument(
+ "-x", "--cross-arch",
+ help="cross compiler architecture",
+ )
def _execute_playbook(self, playbook, hosts, projects, git_revision):
base = Util.get_base()
@@ -546,10 +550,23 @@ class Application:
os_name = facts["os_name"]
os_version = facts["os_version"]
os_full = os_name + os_version
+ cross_facts = None
if package_format not in ["deb", "rpm"]:
raise Error("Host {} doesn't support
Dockerfiles".format(host))
+ if args.cross_arch is not None:
+ if "arches" not in facts:
+ raise Error("Non x86_64 arches not supported for this host")
+ if args.cross_arch not in facts["arches"]:
+ raise Error("Arch {} not supported for this host".format(
+ args.cross_arch))
+ if "cross_gcc" not in facts["arches"][args.cross_arch]:
+ raise Error("Arch {} cross compiler not supported for this
host".format
+ (args.cross_arch))
+
+ cross_build_facts = facts["arches"][args.cross_arch]
+
projects = self._projects.expand_pattern(args.projects)
for project in projects:
if project not in facts["projects"]:
@@ -561,25 +578,50 @@ class Application:
)
pkgs = {}
+ cross_pkgs = {}
base_keys = ["default", package_format, os_name, os_full]
- keys = base_keys + [k + "-" + self._native_arch for k in base_keys]
+ cross_keys = []
+ if args.cross_arch:
+ keys = base_keys + [k + "-" + args.cross_arch for k in base_keys]
+ cross_keys = [k + "-cross-arch" for k in base_keys]
+ else:
+ keys = base_keys + [k + "-" + self._native_arch for k in
base_keys]
+
# We need to add the base project manually here: the standard
# machinery hides it because it's an implementation detail
for project in projects + ["base"]:
for package in self._projects.get_packages(project):
+ policy = "native"
+ for key in cross_keys:
+ if key in mappings[package]:
+ policy = mappings[package][key]
+ if policy not in ["native", "foreign",
"skip"]:
+ raise Error("Unexpected policy {} for {}",
+ policy, package)
+
for key in keys:
if key in mappings[package]:
pkgs[package] = mappings[package][key]
if package not in pkgs:
continue
- if pkgs[package] is None:
+ if policy == "foreign" and pkgs[package] is not None:
+ cross_pkgs[package] = pkgs[package]
+ if pkgs[package] is None or policy in ["skip",
"foreign"]:
del pkgs[package]
print("FROM {}".format(facts["docker_base"]))
varmap = {}
varmap["pkgs"] = " \\\n
".join(sorted(set(pkgs.values())))
+ if args.cross_arch:
+ if package_format != "deb":
+ raise Error("Cannot install foreign {}
packages".format(package_format))
+ varmap["cross_arch"] = cross_build_facts["package_arch"]
+ pkg_names = [p + ":" + cross_build_facts["package_arch"]
for p in cross_pkgs.values()]
+ pkg_names.append(cross_build_facts["cross_gcc"])
+ varmap["cross_pkgs"] = " \\\n
".join(sorted(set(pkg_names)))
+ varmap["cross_target"] = cross_build_facts["abi"]
if package_format == "deb":
sys.stdout.write(textwrap.dedent("""
RUN export DEBIAN_FRONTEND=noninteractive && \\
@@ -590,6 +632,25 @@ class Application:
apt-get autoremove -y && \\
apt-get autoclean -y
""").format(**varmap))
+ if args.cross_arch:
+ # Intentionally a separate RUN command from the above
+ # so that the common packages of all cross-built images
+ # share a docker image layer.
+ sys.stdout.write(textwrap.dedent("""
+ RUN export DEBIAN_FRONTEND=noninteractive && \\
+ dpkg --add-architecture {cross_arch} && \\
+ apt-get update && \\
+ apt-get dist-upgrade -y && \\
+ apt-get install --no-install-recommends -y \\
+ {cross_pkgs} && \\
+ apt-get autoremove -y && \\
+ apt-get autoclean -y
+
+ ENV TARGET "{cross_target}"
+ ENV CONFIGURE_OPTS "--host={cross_target} \\
+ --target={cross_target}"
+ ENV PKG_CONFIG_LIBDIR "/usr/lib/{cross_target}/pkgconfig"
+ """).format(**varmap))
elif package_format == "rpm":
if os_name == "Fedora" and os_version == "Rawhide":
sys.stdout.write(textwrap.dedent("""
diff --git a/guests/vars/mappings.yml b/guests/vars/mappings.yml
index 4ca8fae..a70a6a5 100644
--- a/guests/vars/mappings.yml
+++ b/guests/vars/mappings.yml
@@ -57,11 +57,15 @@
# deb-x86_64: libxen-dev
# Fedora-x86_64: xen-devel
#
+# In parallel with this 'cross-arch: native|foreign|skip' entries can
+# used to indicate the policy when setting up a cross-architecture
+# build environment. If omitted 'native' is assumed
mappings:
apparmor:
deb: libapparmor-dev
+ deb-cross-arch: foreign
augeas:
default: augeas
@@ -80,6 +84,7 @@ mappings:
avahi:
deb: libavahi-client-dev
+ deb-cross-arch: foreign
pkg: avahi
rpm: avahi-devel
@@ -106,6 +111,7 @@ mappings:
cyrus-sasl:
deb: libsasl2-dev
+ deb-cross-arch: foreign
pkg: cyrus-sasl
rpm: cyrus-sasl-devel
@@ -116,6 +122,7 @@ mappings:
device-mapper:
deb: libdevmapper-dev
+ deb-cross-arch: foreign
rpm: device-mapper-devel
dnsmasq:
@@ -124,6 +131,7 @@ mappings:
dtrace:
deb: systemtap-sdt-dev
+ deb-cross-arch: skip
rpm: systemtap-sdt-devel
dwarves:
@@ -144,6 +152,7 @@ mappings:
fuse:
deb: libfuse-dev
+ deb-cross-arch: foreign
pkg: fusefs-libs
rpm: fuse-devel
@@ -159,19 +168,23 @@ mappings:
glib2:
deb: libglib2.0-dev
+ deb-cross-arch: foreign
pkg: glib
rpm: glib2-devel
glibc:
deb: libc6-dev
+ deb-cross-arch: foreign
rpm: glibc-devel
glibc-static:
deb: libc6-dev
+ deb-cross-arch: foreign
rpm: glibc-static
glusterfs:
deb: libglusterfs-dev
+ deb-cross-arch: foreign
rpm: glusterfs-api-devel
Debian8: glusterfs-common
Debian9: glusterfs-common
@@ -183,6 +196,7 @@ mappings:
gnutls:
deb: libgnutls28-dev
+ deb-cross-arch: foreign
pkg: gnutls
rpm: gnutls-devel
@@ -192,11 +206,13 @@ mappings:
gobject-introspection:
deb: libgirepository1.0-dev
+ deb-cross-arch: foreign
pkg: gobject-introspection
rpm: gobject-introspection-devel
gtk3:
deb: libgtk-3-dev
+ deb-cross-arch: foreign
pkg: gtk3
rpm: gtk3-devel
@@ -211,6 +227,7 @@ mappings:
gtk-vnc2:
deb: libgtk-vnc-2.0-dev
+ deb-cross-arch: foreign
pkg: gtk-vnc
rpm: gtk-vnc2-devel
@@ -243,32 +260,39 @@ mappings:
json-glib:
deb: libjson-glib-dev
+ deb-cross-arch: foreign
pkg: json-glib
rpm: json-glib-devel
libacl:
deb: libacl1-dev
+ deb-cross-arch: foreign
rpm: libacl-devel
libarchive:
deb: libarchive-dev
+ deb-cross-arch: foreign
pkg: libarchive
rpm: libarchive-devel
libattr:
deb: libattr1-dev
+ deb-cross-arch: foreign
rpm: libattr-devel
libaudit:
deb: libaudit-dev
+ deb-cross-arch: foreign
rpm: audit-libs-devel
libblkid:
deb: libblkid-dev
+ deb-cross-arch: foreign
rpm: libblkid-devel
libcap-ng:
deb: libcap-ng-dev
+ deb-cross-arch: foreign
rpm: libcap-ng-devel
libcmpiutil:
@@ -276,67 +300,81 @@ mappings:
libconfig:
deb: libconfig-dev
+ deb-cross-arch: foreign
pkg: libconfig
rpm: libconfig-devel
libcurl:
deb: libcurl4-gnutls-dev
+ deb-cross-arch: foreign
pkg: curl
rpm: libcurl-devel
libdbus:
deb: libdbus-1-dev
+ deb-cross-arch: foreign
pkg: dbus
rpm: dbus-devel
libgovirt:
rpm: libgovirt-devel
Debian: libgovirt-dev
+ Debian-cross-arch: foreign
Debian8:
libiscsi:
deb: libiscsi-dev
+ deb-cross-arch: foreign
rpm: libiscsi-devel
libnl3:
deb: libnl-3-dev
+ deb-cross-arch: foreign
rpm: libnl3-devel
libnlroute3:
deb: libnl-route-3-dev
+ deb-cross-arch: foreign
rpm: libnl3-devel
libnuma:
deb: libnuma-dev
+ deb-cross-arch: foreign
deb-armv6l:
deb-armv7l:
rpm: numactl-devel
libparted:
deb: libparted-dev
+ deb-cross-arch: foreign
rpm: parted-devel
libpcap:
deb: libpcap0.8-dev
+ deb-cross-arch: foreign
pkg: libpcap
rpm: libpcap-devel
libpciaccess:
deb: libpciaccess-dev
+ deb-cross-arch: foreign
pkg: libpciaccess
rpm: libpciaccess-devel
librbd:
deb: librbd-dev
+ deb-cross-arch: foreign
Fedora: librbd-devel
CentOS7: librbd1-devel
libselinux:
deb: libselinux1-dev
+ deb-cross-arch: foreign
rpm: libselinux-devel
libsoup:
deb: libsoup2.4-dev
+ deb-cross-arch: foreign
pkg: libsoup
rpm: libsoup-devel
@@ -344,15 +382,18 @@ mappings:
pkg: libssh
rpm: libssh-devel
Debian: libssh-gcrypt-dev
+ Debian-cross-arch: foreign
Ubuntu: libssh-dev
libssh2:
deb: libssh2-1-dev
+ deb-cross-arch: foreign
pkg: libssh2
rpm: libssh2-devel
libtirpc:
deb: libtirpc-dev
+ deb-cross-arch: foreign
rpm: libtirpc-devel
libtool:
@@ -364,20 +405,24 @@ mappings:
libudev:
deb: libudev-dev
+ deb-cross-arch: foreign
rpm: libudev-devel
libuuid:
deb: uuid-dev
+ deb-cross-arch: foreign
pkg: e2fsprogs-libuuid
rpm: libuuid-devel
libxml2:
deb: libxml2-dev
+ deb-cross-arch: foreign
pkg: libxml2
rpm: libxml2-devel
libxslt:
deb: libxslt1-dev
+ deb-cross-arch: foreign
pkg: libxslt
rpm: libxslt-devel
@@ -550,6 +595,7 @@ mappings:
netcf:
deb: libnetcf-dev
+ deb-cross-arch: skip
rpm: netcf-devel
numad:
@@ -710,6 +756,7 @@ mappings:
python2-devel:
deb: python-dev
+ deb-cross-arch: foreign
pkg: python2
rpm: python2-devel
@@ -734,6 +781,7 @@ mappings:
python3-devel:
deb: python3-dev
+ deb-cross-arch: foreign
pkg: python3
Fedora: python3-devel
@@ -779,6 +827,7 @@ mappings:
readline:
deb: libreadline-dev
+ deb-cross-arch: foreign
pkg: readline
rpm: readline-devel
@@ -793,6 +842,7 @@ mappings:
sanlock:
deb: libsanlock-dev
+ deb-cross-arch: foreign
rpm: sanlock-devel
screen:
@@ -814,6 +864,7 @@ mappings:
spice-gtk3:
deb: libspice-client-gtk-3.0-dev
+ deb-cross-arch: foreign
pkg: spice-gtk
rpm: spice-gtk3-devel
@@ -845,10 +896,12 @@ mappings:
wireshark:
deb: wireshark-dev
+ deb-cross-arch: skip
Fedora: wireshark-devel
Debian8:
xen:
+ deb-cross-arch: foreign
deb-x86_64: libxen-dev
deb-armv7l: libxen-dev
deb-aarch64: libxen-dev
@@ -856,6 +909,7 @@ mappings:
xfsprogs:
deb: xfslibs-dev
+ deb-cross-arch: foreign
rpm: xfsprogs-devel
xmllint:
@@ -868,14 +922,17 @@ mappings:
xz:
deb: liblzma-dev
+ deb-cross-arch: foreign
rpm: xz-devel
xz-static:
deb: liblzma-dev
+ deb-cross-arch: foreign
Fedora: xz-static
yajl:
deb: libyajl-dev
+ deb-cross-arch: foreign
pkg: yajl
rpm: yajl-devel
@@ -886,8 +943,10 @@ mappings:
zlib:
deb: zlib1g-dev
+ deb-cross-arch: foreign
rpm: zlib-devel
zlib-static:
deb: zlib1g-dev
+ deb-cross-arch: foreign
rpm: zlib-static
--
2.20.1
2098
days inactive
2102
days old
8 comments
2 participants
participants (2)
-
Andrea Bolognani -
Daniel P. Berrangé