[libvirt] [jenkins-ci POC 0/2] guests: Add Dockerfile generator

See 2/2 for details. Andrea Bolognani (2): guests: Remove OS detection guests: Add Dockerfile generator guests/dockerfile-generator | 72 +++++++++++++++++++ guests/host_vars/libvirt-centos-7/docker.yml | 2 + guests/host_vars/libvirt-centos-7/main.yml | 4 ++ guests/host_vars/libvirt-debian-8/docker.yml | 2 + guests/host_vars/libvirt-debian-8/main.yml | 4 ++ guests/host_vars/libvirt-debian-9/docker.yml | 2 + guests/host_vars/libvirt-debian-9/main.yml | 4 ++ .../host_vars/libvirt-debian-sid/docker.yml | 2 + guests/host_vars/libvirt-debian-sid/main.yml | 4 ++ guests/host_vars/libvirt-fedora-27/docker.yml | 2 + guests/host_vars/libvirt-fedora-27/main.yml | 4 ++ guests/host_vars/libvirt-fedora-28/docker.yml | 2 + guests/host_vars/libvirt-fedora-28/main.yml | 4 ++ .../libvirt-fedora-rawhide/docker.yml | 2 + .../host_vars/libvirt-fedora-rawhide/main.yml | 4 ++ guests/host_vars/libvirt-freebsd-10/main.yml | 4 ++ guests/host_vars/libvirt-freebsd-11/main.yml | 4 ++ .../libvirt-freebsd-current/main.yml | 4 ++ guests/host_vars/libvirt-ubuntu-16/docker.yml | 2 + guests/host_vars/libvirt-ubuntu-16/main.yml | 4 ++ guests/host_vars/libvirt-ubuntu-18/docker.yml | 2 + guests/host_vars/libvirt-ubuntu-18/main.yml | 4 ++ guests/tasks/base.yml | 51 ------------- 23 files changed, 138 insertions(+), 51 deletions(-) create mode 100755 guests/dockerfile-generator create mode 100644 guests/host_vars/libvirt-centos-7/docker.yml create mode 100644 guests/host_vars/libvirt-debian-8/docker.yml create mode 100644 guests/host_vars/libvirt-debian-9/docker.yml create mode 100644 guests/host_vars/libvirt-debian-sid/docker.yml create mode 100644 guests/host_vars/libvirt-fedora-27/docker.yml create mode 100644 guests/host_vars/libvirt-fedora-28/docker.yml create mode 100644 guests/host_vars/libvirt-fedora-rawhide/docker.yml create mode 100644 guests/host_vars/libvirt-ubuntu-16/docker.yml create mode 100644 guests/host_vars/libvirt-ubuntu-18/docker.yml -- 2.17.1

We're going to need information about the OS outside of Ansible soon, so drop the OS detection code and start storing the information as host variables instead. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/host_vars/libvirt-centos-7/main.yml | 4 ++ guests/host_vars/libvirt-debian-8/main.yml | 4 ++ guests/host_vars/libvirt-debian-9/main.yml | 4 ++ guests/host_vars/libvirt-debian-sid/main.yml | 4 ++ guests/host_vars/libvirt-fedora-27/main.yml | 4 ++ guests/host_vars/libvirt-fedora-28/main.yml | 4 ++ .../host_vars/libvirt-fedora-rawhide/main.yml | 4 ++ guests/host_vars/libvirt-freebsd-10/main.yml | 4 ++ guests/host_vars/libvirt-freebsd-11/main.yml | 4 ++ .../libvirt-freebsd-current/main.yml | 4 ++ guests/host_vars/libvirt-ubuntu-16/main.yml | 4 ++ guests/host_vars/libvirt-ubuntu-18/main.yml | 4 ++ guests/tasks/base.yml | 51 ------------------- 13 files changed, 48 insertions(+), 51 deletions(-) diff --git a/guests/host_vars/libvirt-centos-7/main.yml b/guests/host_vars/libvirt-centos-7/main.yml index c147830..4c80e22 100644 --- a/guests/host_vars/libvirt-centos-7/main.yml +++ b/guests/host_vars/libvirt-centos-7/main.yml @@ -13,3 +13,7 @@ projects: - osinfo-db - osinfo-db-tools - virt-viewer + +package_format: rpm +os_name: CentOS +os_version: 7 diff --git a/guests/host_vars/libvirt-debian-8/main.yml b/guests/host_vars/libvirt-debian-8/main.yml index 954a037..a6875a4 100644 --- a/guests/host_vars/libvirt-debian-8/main.yml +++ b/guests/host_vars/libvirt-debian-8/main.yml @@ -11,3 +11,7 @@ projects: - osinfo-db - osinfo-db-tools - virt-viewer + +package_format: deb +os_name: Debian +os_version: 8 diff --git a/guests/host_vars/libvirt-debian-9/main.yml b/guests/host_vars/libvirt-debian-9/main.yml index a588c09..cdc5a57 100644 --- a/guests/host_vars/libvirt-debian-9/main.yml +++ b/guests/host_vars/libvirt-debian-9/main.yml @@ -14,3 +14,7 @@ projects: - osinfo-db-tools - virt-manager - virt-viewer + +package_format: deb +os_name: Debian +os_version: 9 diff --git a/guests/host_vars/libvirt-debian-sid/main.yml b/guests/host_vars/libvirt-debian-sid/main.yml index a588c09..5dd887b 100644 --- a/guests/host_vars/libvirt-debian-sid/main.yml +++ b/guests/host_vars/libvirt-debian-sid/main.yml @@ -14,3 +14,7 @@ projects: - osinfo-db-tools - virt-manager - virt-viewer + +package_format: deb +os_name: Debian +os_version: Sid diff --git a/guests/host_vars/libvirt-fedora-27/main.yml b/guests/host_vars/libvirt-fedora-27/main.yml index 00b0848..f131953 100644 --- a/guests/host_vars/libvirt-fedora-27/main.yml +++ b/guests/host_vars/libvirt-fedora-27/main.yml @@ -15,3 +15,7 @@ projects: - osinfo-db-tools - virt-manager - virt-viewer + +package_format: rpm +os_name: Fedora +os_version: 27 diff --git a/guests/host_vars/libvirt-fedora-28/main.yml b/guests/host_vars/libvirt-fedora-28/main.yml index 00b0848..f115598 100644 --- a/guests/host_vars/libvirt-fedora-28/main.yml +++ b/guests/host_vars/libvirt-fedora-28/main.yml @@ -15,3 +15,7 @@ projects: - osinfo-db-tools - virt-manager - virt-viewer + +package_format: rpm +os_name: Fedora +os_version: 28 diff --git a/guests/host_vars/libvirt-fedora-rawhide/main.yml b/guests/host_vars/libvirt-fedora-rawhide/main.yml index 5044594..1bd3332 100644 --- a/guests/host_vars/libvirt-fedora-rawhide/main.yml +++ b/guests/host_vars/libvirt-fedora-rawhide/main.yml @@ -20,3 +20,7 @@ projects: - virt-manager - virt-viewer - virt-viewer+mingw + +package_format: rpm +os_name: Fedora +os_version: Rawhide diff --git a/guests/host_vars/libvirt-freebsd-10/main.yml b/guests/host_vars/libvirt-freebsd-10/main.yml index 2ad4584..005acb4 100644 --- a/guests/host_vars/libvirt-freebsd-10/main.yml +++ b/guests/host_vars/libvirt-freebsd-10/main.yml @@ -15,3 +15,7 @@ projects: - osinfo-db-tools - virt-manager - virt-viewer + +package_format: pkg +os_name: FreeBSD +os_version: 10 diff --git a/guests/host_vars/libvirt-freebsd-11/main.yml b/guests/host_vars/libvirt-freebsd-11/main.yml index 2ad4584..dd06fc5 100644 --- a/guests/host_vars/libvirt-freebsd-11/main.yml +++ b/guests/host_vars/libvirt-freebsd-11/main.yml @@ -15,3 +15,7 @@ projects: - osinfo-db-tools - virt-manager - virt-viewer + +package_format: pkg +os_name: FreeBSD +os_version: 11 diff --git a/guests/host_vars/libvirt-freebsd-current/main.yml b/guests/host_vars/libvirt-freebsd-current/main.yml index 2ad4584..07f3f25 100644 --- a/guests/host_vars/libvirt-freebsd-current/main.yml +++ b/guests/host_vars/libvirt-freebsd-current/main.yml @@ -15,3 +15,7 @@ projects: - osinfo-db-tools - virt-manager - virt-viewer + +package_format: pkg +os_name: FreeBSD +os_version: Current diff --git a/guests/host_vars/libvirt-ubuntu-16/main.yml b/guests/host_vars/libvirt-ubuntu-16/main.yml index a588c09..92296b4 100644 --- a/guests/host_vars/libvirt-ubuntu-16/main.yml +++ b/guests/host_vars/libvirt-ubuntu-16/main.yml @@ -14,3 +14,7 @@ projects: - osinfo-db-tools - virt-manager - virt-viewer + +package_format: deb +os_name: Ubuntu +os_version: 16 diff --git a/guests/host_vars/libvirt-ubuntu-18/main.yml b/guests/host_vars/libvirt-ubuntu-18/main.yml index a588c09..8a45854 100644 --- a/guests/host_vars/libvirt-ubuntu-18/main.yml +++ b/guests/host_vars/libvirt-ubuntu-18/main.yml @@ -14,3 +14,7 @@ projects: - osinfo-db-tools - virt-manager - virt-viewer + +package_format: deb +os_name: Ubuntu +os_version: 18 diff --git a/guests/tasks/base.yml b/guests/tasks/base.yml index 13d02f8..6310d96 100644 --- a/guests/tasks/base.yml +++ b/guests/tasks/base.yml @@ -1,55 +1,4 @@ --- -- name: Set additional facts (OS) - set_fact: - os_name: '{{ ansible_distribution }}' - -- name: Set additional facts (OS) - set_fact: - os_version: '{{ ansible_distribution_major_version }}' - when: - - ansible_distribution_major_version is defined - -- name: Set additional facts (OS) - set_fact: - os_version: Rawhide - when: - - os_name == 'Fedora' - - ansible_distribution_release == 'Rawhide' - -- name: Set additional facts (OS) - set_fact: - os_version: Sid - when: - - os_name == 'Debian' - - ansible_distribution_major_version is match("^.*/sid$") - -- name: Set additional facts (OS) - set_fact: - os_version: Current - when: - - ansible_distribution == 'FreeBSD' - - ansible_distribution_release is match("^.*-CURRENT$") - -- name: Set additional facts (package format) - set_fact: - package_format: deb - when: - - ( os_name == 'Debian' or - os_name == 'Ubuntu' ) - -- name: Set additional facts (package format) - set_fact: - package_format: pkg - when: - - os_name == 'FreeBSD' - -- name: Set additional facts (package format) - set_fact: - package_format: rpm - when: - - ( os_name == 'CentOS' or - os_name == 'Fedora' ) - - name: Bootstrap the package module command: apt-get install -y python-apt args: -- 2.17.1

On Fri, Jun 15, 2018 at 08:53:43AM +0200, Andrea Bolognani wrote:
We're going to need information about the OS outside of Ansible soon, so drop the OS detection code and start storing the information as host variables instead.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/host_vars/libvirt-centos-7/main.yml | 4 ++ guests/host_vars/libvirt-debian-8/main.yml | 4 ++ guests/host_vars/libvirt-debian-9/main.yml | 4 ++ guests/host_vars/libvirt-debian-sid/main.yml | 4 ++ guests/host_vars/libvirt-fedora-27/main.yml | 4 ++ guests/host_vars/libvirt-fedora-28/main.yml | 4 ++ .../host_vars/libvirt-fedora-rawhide/main.yml | 4 ++ guests/host_vars/libvirt-freebsd-10/main.yml | 4 ++ guests/host_vars/libvirt-freebsd-11/main.yml | 4 ++ .../libvirt-freebsd-current/main.yml | 4 ++ guests/host_vars/libvirt-ubuntu-16/main.yml | 4 ++ guests/host_vars/libvirt-ubuntu-18/main.yml | 4 ++ guests/tasks/base.yml | 51 ------------------- 13 files changed, 48 insertions(+), 51 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

This is basically the exact same algorithm used by the Ansible playbooks to process package mappings, implemented as a standalone script. There's little to none error handling, and some information is hardcoded instead of being configurable at runtime; more importantly, before it can be considered for merging it needs to be integrated into lcitool, which in turn requires lcitool to be significantly reworked. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/dockerfile-generator | 72 +++++++++++++++++++ guests/host_vars/libvirt-centos-7/docker.yml | 2 + guests/host_vars/libvirt-debian-8/docker.yml | 2 + guests/host_vars/libvirt-debian-9/docker.yml | 2 + .../host_vars/libvirt-debian-sid/docker.yml | 2 + guests/host_vars/libvirt-fedora-27/docker.yml | 2 + guests/host_vars/libvirt-fedora-28/docker.yml | 2 + .../libvirt-fedora-rawhide/docker.yml | 2 + guests/host_vars/libvirt-ubuntu-16/docker.yml | 2 + guests/host_vars/libvirt-ubuntu-18/docker.yml | 2 + 10 files changed, 90 insertions(+) create mode 100755 guests/dockerfile-generator create mode 100644 guests/host_vars/libvirt-centos-7/docker.yml create mode 100644 guests/host_vars/libvirt-debian-8/docker.yml create mode 100644 guests/host_vars/libvirt-debian-9/docker.yml create mode 100644 guests/host_vars/libvirt-debian-sid/docker.yml create mode 100644 guests/host_vars/libvirt-fedora-27/docker.yml create mode 100644 guests/host_vars/libvirt-fedora-28/docker.yml create mode 100644 guests/host_vars/libvirt-fedora-rawhide/docker.yml create mode 100644 guests/host_vars/libvirt-ubuntu-16/docker.yml create mode 100644 guests/host_vars/libvirt-ubuntu-18/docker.yml diff --git a/guests/dockerfile-generator b/guests/dockerfile-generator new file mode 100755 index 0000000..e1212da --- /dev/null +++ b/guests/dockerfile-generator @@ -0,0 +1,72 @@ +#!/usr/bin/env python3 + +import sys +import yaml + +if len(sys.argv) < 2: + print("Usage: {} GUEST".format(sys.argv[0])) + sys.exit(1) + +guest = sys.argv[1] + +# Load data + +with open("vars/mappings.yml", "r") as f: + mappings = yaml.load(f) + mappings = mappings["mappings"] + +with open("host_vars/{}/main.yml".format(guest), "r") as f: + main = yaml.load(f) + package_format = main["package_format"] + os_name = main["os_name"] + os_version = main["os_version"] + os_full = "{}{}".format(os_name, os_version) + +with open("host_vars/{}/docker.yml".format(guest), "r") as f: + docker = yaml.load(f) + docker_base = docker["docker_base"] + +projects = ["base", "libvirt"] +if os_name == "Fedora" and os_version == "Rawhide": + projects += ["libvirt+mingw"] + +# Process mappings + +temp = {} + +for project in projects: + with open("vars/projects/{}.yml".format(project), "r") as f: + packages = yaml.load(f) + packages = packages["packages"] + for package in packages: + if "default" in mappings[package]: + temp[package] = mappings[package]["default"] + if package_format in mappings[package]: + temp[package] = mappings[package][package_format] + if os_name in mappings[package]: + temp[package] = mappings[package][os_name] + if os_full in mappings[package]: + temp[package] = mappings[package][os_full] + +flattened = [] + +for item in temp: + if temp[item] != None and temp[item] not in flattened: + flattened += [ temp[item] ] + +# Output Dockerfile + +print("FROM {}".format(docker_base)) + +sys.stdout.write("ENV PACKAGES ") +print(" \\\n ".join(sorted(flattened))) + +if package_format == "deb": + print("RUN apt-get update && \\") + print(" apt-get install -y ${PACKAGES} && \\") + print(" apt-get autoremove -y && \\") + print(" apt-get autoclean -y") +elif package_format == "rpm": + print("RUN yum install -y ${PACKAGES} && \\") + print(" yum autoremove -y && \\") + print(" yum clean all -y") diff --git a/guests/host_vars/libvirt-centos-7/docker.yml b/guests/host_vars/libvirt-centos-7/docker.yml new file mode 100644 index 0000000..59f7f12 --- /dev/null +++ b/guests/host_vars/libvirt-centos-7/docker.yml @@ -0,0 +1,2 @@ +--- +docker_base: centos:centos7 diff --git a/guests/host_vars/libvirt-debian-8/docker.yml b/guests/host_vars/libvirt-debian-8/docker.yml new file mode 100644 index 0000000..235f0fd --- /dev/null +++ b/guests/host_vars/libvirt-debian-8/docker.yml @@ -0,0 +1,2 @@ +--- +docker_base: debian:8 diff --git a/guests/host_vars/libvirt-debian-9/docker.yml b/guests/host_vars/libvirt-debian-9/docker.yml new file mode 100644 index 0000000..0b4ccee --- /dev/null +++ b/guests/host_vars/libvirt-debian-9/docker.yml @@ -0,0 +1,2 @@ +--- +docker_base: debian:9 diff --git a/guests/host_vars/libvirt-debian-sid/docker.yml b/guests/host_vars/libvirt-debian-sid/docker.yml new file mode 100644 index 0000000..e20a37e --- /dev/null +++ b/guests/host_vars/libvirt-debian-sid/docker.yml @@ -0,0 +1,2 @@ +--- +docker_base: debian:sid diff --git a/guests/host_vars/libvirt-fedora-27/docker.yml b/guests/host_vars/libvirt-fedora-27/docker.yml new file mode 100644 index 0000000..dcada18 --- /dev/null +++ b/guests/host_vars/libvirt-fedora-27/docker.yml @@ -0,0 +1,2 @@ +--- +docker_base: fedora:27 diff --git a/guests/host_vars/libvirt-fedora-28/docker.yml b/guests/host_vars/libvirt-fedora-28/docker.yml new file mode 100644 index 0000000..a874018 --- /dev/null +++ b/guests/host_vars/libvirt-fedora-28/docker.yml @@ -0,0 +1,2 @@ +--- +docker_base: fedora:28 diff --git a/guests/host_vars/libvirt-fedora-rawhide/docker.yml b/guests/host_vars/libvirt-fedora-rawhide/docker.yml new file mode 100644 index 0000000..39dda1c --- /dev/null +++ b/guests/host_vars/libvirt-fedora-rawhide/docker.yml @@ -0,0 +1,2 @@ +--- +docker_base: fedora:rawhide diff --git a/guests/host_vars/libvirt-ubuntu-16/docker.yml b/guests/host_vars/libvirt-ubuntu-16/docker.yml new file mode 100644 index 0000000..2d4eb25 --- /dev/null +++ b/guests/host_vars/libvirt-ubuntu-16/docker.yml @@ -0,0 +1,2 @@ +--- +docker_base: ubuntu:16.04 diff --git a/guests/host_vars/libvirt-ubuntu-18/docker.yml b/guests/host_vars/libvirt-ubuntu-18/docker.yml new file mode 100644 index 0000000..13d6cc1 --- /dev/null +++ b/guests/host_vars/libvirt-ubuntu-18/docker.yml @@ -0,0 +1,2 @@ +--- +docker_base: ubuntu:18.04 -- 2.17.1

On Fri, Jun 15, 2018 at 08:53:44AM +0200, Andrea Bolognani wrote:
This is basically the exact same algorithm used by the Ansible playbooks to process package mappings, implemented as a standalone script.
There's little to none error handling, and some information is hardcoded instead of being configurable at runtime; more importantly, before it can be considered for merging it needs to be integrated into lcitool, which in turn requires lcitool to be significantly reworked.
I'm not understanding why it needs to be integrated into lcitool ? Generating dockerfiles has no dependancy / interaction with ansible updating the jenkins slaves. Why won't we just run this script separately when needed, ideally as non-root ? Looks good enough to commit now IMHO Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Fri, 2018-06-15 at 10:12 +0100, Daniel P. Berrangé wrote:
On Fri, Jun 15, 2018 at 08:53:44AM +0200, Andrea Bolognani wrote:
This is basically the exact same algorithm used by the Ansible playbooks to process package mappings, implemented as a standalone script.
There's little to none error handling, and some information is hardcoded instead of being configurable at runtime; more importantly, before it can be considered for merging it needs to be integrated into lcitool, which in turn requires lcitool to be significantly reworked.
I'm not understanding why it needs to be integrated into lcitool ? Generating dockerfiles has no dependancy / interaction with ansible updating the jenkins slaves.
Strictly speaking there is no interaction; however, in order to generate Dockerfiles you need to poke into the Ansible inventory, so the two are not completely independent either. To a degree it's similar to how installation and setup of guests, while performed in isolation from each other and using completely different tools, both require information stored in the Ansible inventory and are exposed by lcitool as top-level actions with a consistent interface. Plus it's a good excuse to go back and make lcitool somewhat less of a horrible hack ;)
Why won't we just run this script separately when needed, ideally as non-root ? Looks good enough to commit now IMHO
lcitool is already designed to run as non-root, so no need to change anything there. -- Andrea Bolognani / Red Hat / Virtualization
participants (2)
-
Andrea Bolognani
-
Daniel P. Berrangé