[libvirt] [jenkins-ci PATCH 0/8] lcitool: Cleanups and improvements

This series makes the guests/ directory less crowded by moving a bunch of files to subdirectories, enhances the lcitool script so that it works when called from any directory, and introduces the ability to skip installation of build dependencies for projects you don't care about. Some of these changes pave the way for introducing [secret feature redacted] later on. Andrea Bolognani (8): lcitool: Make the script location-independent lcitool: Use JSON to pass extra-vars lcitool: Pass base and playbook_base to Ansible guests: Move update playbook and related resources guests: Drop install_config from inventory guests: Move install configs lcitool: Allow installing a subset of build dependencies guests: Update documentation guests/README.markdown | 38 ++++++-- guests/{ => configs}/kickstart.cfg | 0 guests/{ => configs}/preseed.cfg | 0 guests/host_vars/libvirt-centos-7/install.yml | 1 - guests/host_vars/libvirt-debian-8/install.yml | 1 - guests/host_vars/libvirt-debian-9/install.yml | 1 - .../host_vars/libvirt-debian-sid/install.yml | 1 - .../host_vars/libvirt-fedora-27/install.yml | 1 - .../host_vars/libvirt-fedora-28/install.yml | 1 - .../libvirt-fedora-rawhide/install.yml | 1 - .../host_vars/libvirt-ubuntu-16/install.yml | 1 - .../host_vars/libvirt-ubuntu-18/install.yml | 1 - guests/lcitool | 96 ++++++++++++++----- .../{site.yml => playbooks/update/main.yml} | 30 +++--- guests/{ => playbooks/update}/tasks/base.yml | 6 +- .../update}/tasks/bootloader.yml | 0 .../update}/tasks/bootstrap.yml | 0 .../{ => playbooks/update}/tasks/jenkins.yml | 4 +- .../{ => playbooks/update}/tasks/kludges.yml | 0 .../{ => playbooks/update}/tasks/packages.yml | 2 +- guests/{ => playbooks/update}/tasks/paths.yml | 0 .../{ => playbooks/update}/tasks/services.yml | 0 guests/{ => playbooks/update}/tasks/users.yml | 4 +- .../update}/templates/bash_profile.j2 | 0 .../update}/templates/bashrc.j2 | 0 .../update}/templates/ccache.conf.j2 | 0 .../fedora-rawhide-kernel-nodebug.repo.j2 | 0 .../update}/templates/jenkins.service.j2 | 0 .../templates/jessie-backports.preferences.j2 | 0 .../templates/jessie-backports.sources.j2 | 0 30 files changed, 128 insertions(+), 61 deletions(-) rename guests/{ => configs}/kickstart.cfg (100%) rename guests/{ => configs}/preseed.cfg (100%) rename guests/{site.yml => playbooks/update/main.yml} (52%) rename guests/{ => playbooks/update}/tasks/base.yml (92%) rename guests/{ => playbooks/update}/tasks/bootloader.yml (100%) rename guests/{ => playbooks/update}/tasks/bootstrap.yml (100%) rename guests/{ => playbooks/update}/tasks/jenkins.yml (92%) rename guests/{ => playbooks/update}/tasks/kludges.yml (100%) rename guests/{ => playbooks/update}/tasks/packages.yml (96%) rename guests/{ => playbooks/update}/tasks/paths.yml (100%) rename guests/{ => playbooks/update}/tasks/services.yml (100%) rename guests/{ => playbooks/update}/tasks/users.yml (95%) rename guests/{ => playbooks/update}/templates/bash_profile.j2 (100%) rename guests/{ => playbooks/update}/templates/bashrc.j2 (100%) rename guests/{ => playbooks/update}/templates/ccache.conf.j2 (100%) rename guests/{ => playbooks/update}/templates/fedora-rawhide-kernel-nodebug.repo.j2 (100%) rename guests/{ => playbooks/update}/templates/jenkins.service.j2 (100%) rename guests/{ => playbooks/update}/templates/jessie-backports.preferences.j2 (100%) rename guests/{ => playbooks/update}/templates/jessie-backports.sources.j2 (100%) -- 2.17.1

Up until now, the only way to run lcitool has been from the same directory it lives. After this patch, the script is able to figure out its own location and adjust all paths accordingly. Suggested-by: Katerina Koukiou <kkoukiou@redhat.com> Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/lcitool | 61 +++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 13 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index 22b08dd..5526a27 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -43,6 +43,10 @@ class Error(Exception): class Util: + @staticmethod + def get_base(): + return os.path.dirname(os.path.abspath(__file__)) + @staticmethod def mksalt(): alphabeth = string.ascii_letters + string.digits @@ -83,8 +87,8 @@ class Config: try: config_dir = os.environ["XDG_CONFIG_HOME"] except KeyError: - config_dir = os.path.join(os.environ["HOME"], ".config/") - config_dir = os.path.join(config_dir, "lcitool/") + config_dir = os.path.join(os.environ["HOME"], ".config") + config_dir = os.path.join(config_dir, "lcitool") # Create the directory if it doesn't already exist if not os.path.exists(config_dir): @@ -179,13 +183,18 @@ class Config: class Inventory: def __init__(self): + base = Util.get_base() + ansible_cfg_path = os.path.join(base, "ansible.cfg") + try: parser = configparser.SafeConfigParser() - parser.read("./ansible.cfg") + parser.read(ansible_cfg_path) inventory_path = parser.get("defaults", "inventory") except Exception: raise Error("Can't find inventory location in ansible.cfg") + inventory_path = os.path.join(base, inventory_path) + self._facts = {} try: # We can only deal with trivial inventories, but that's @@ -217,12 +226,19 @@ class Inventory: facts[fact] = some_facts[fact] def _read_all_facts(self, host): + base = Util.get_base() + + sources = [ + os.path.join(base, "group_vars", "all"), + os.path.join(base, "host_vars", host), + ] + facts = {} # We load from group_vars/ first and host_vars/ second, sorting # files alphabetically; doing so should result in our view of # the facts matching Ansible's - for source in ["./group_vars/all/", "./host_vars/{}/".format(host)]: + for source in sources: for item in sorted(os.listdir(source)): yaml_path = os.path.join(source, item) if not os.path.isfile(yaml_path): @@ -243,15 +259,20 @@ class Inventory: class Projects: def __init__(self): + base = Util.get_base() + + mappings_path = os.path.join(base, "vars", "mappings.yml") + try: - with open("./vars/mappings.yml", "r") as infile: + with open(mappings_path, "r") as infile: mappings = yaml.load(infile) self._mappings = mappings["mappings"] except Exception: raise Error("Can't load mappings") + source = os.path.join(base, "vars", "projects") + self._packages = {} - source = "./vars/projects/" for item in os.listdir(source): yaml_path = os.path.join(source, item) if not os.path.isfile(yaml_path): @@ -338,6 +359,8 @@ class Application: print(project) def _action_install(self, hosts, _projects): + base = Util.get_base() + flavor = self._config.get_flavor() for host in self._inventory.expand_pattern(hosts): @@ -358,6 +381,8 @@ class Application: facts["install_network"], ) + install_config = os.path.join(base, facts["install_config"]) + # preseed files must use a well-known name to be picked up by # d-i; for kickstart files, we can use whatever name we please # but we need to point anaconda in the right direction through @@ -381,7 +406,7 @@ class Application: "--graphics", "none", "--console", "pty", "--sound", "none", - "--initrd-inject", facts["install_config"], + "--initrd-inject", install_config, "--extra-args", extra_arg, "--wait", "0", ] @@ -396,28 +421,38 @@ class Application: raise Error("Failed to install '{}'".format(host)) def _action_update(self, hosts, _projects): + base = Util.get_base() + flavor = self._config.get_flavor() vault_pass_file = self._config.get_vault_password_file() root_pass_file = self._config.get_root_password_file() ansible_hosts = ",".join(self._inventory.expand_pattern(hosts)) + ansible_cfg_path = os.path.join(base, "ansible.cfg") + playbook_path = os.path.join(base, "site.yml") + extra_vars = "flavor={} root_password_file={}".format( flavor, root_pass_file, ) - cmd = ["ansible-playbook"] + cmd = [ + "ansible-playbook", + "--limit", ansible_hosts, + "--extra-vars", extra_vars, + ] # Provide the vault password if available if vault_pass_file is not None: cmd += ["--vault-password-file", vault_pass_file] - cmd += [ - "--limit", ansible_hosts, - "--extra-vars", extra_vars, - "./site.yml", - ] + cmd += [playbook_path] + + # We need to point Ansible to the correct configuration file, + # and for some reason this has to be done using the environment + # rather than through the command line + os.environ["ANSIBLE_CONFIG"] = ansible_cfg_path try: subprocess.check_call(cmd) -- 2.17.1

On Thu, Jul 19, 2018 at 06:32:01PM +0200, Andrea Bolognani wrote:
Up until now, the only way to run lcitool has been from the same directory it lives. After this patch, the script is able to figure out its own location and adjust all paths accordingly.
Suggested-by: Katerina Koukiou <kkoukiou@redhat.com> Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- Reviewed-by: Erik Skultety <eskultet@redhat.com>

We're going to add more extra-vars later on, and they will be in some cases more complex than simple strings; using JSON allows us to do that and also be less verbose. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/lcitool | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index 5526a27..4acb076 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -19,6 +19,7 @@ import argparse import crypt import fnmatch +import json import os import random import string @@ -432,10 +433,10 @@ class Application: ansible_cfg_path = os.path.join(base, "ansible.cfg") playbook_path = os.path.join(base, "site.yml") - extra_vars = "flavor={} root_password_file={}".format( - flavor, - root_pass_file, - ) + extra_vars = json.dumps({ + "root_password_file": root_pass_file, + "flavor": flavor, + }) cmd = [ "ansible-playbook", -- 2.17.1

On Thu, Jul 19, 2018 at 06:32:02PM +0200, Andrea Bolognani wrote:
We're going to add more extra-vars later on, and they will be in some cases more complex than simple strings; using JSON allows us to do that and also be less verbose.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- Reviewed-by: Erik Skultety <eskultet@redhat.com>

We want to get rid of relative paths in playbooks and tasks, and in order to do that we have to provide Ansible with some more information. base is the directory where lcitool lives, and playbook_base is the directory where a playbook should look for its private resources: they match for the time being, but that will no longer be the case very shortly. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/lcitool | 5 ++++- guests/site.yml | 26 +++++++++++++------------- guests/tasks/base.yml | 6 +++--- guests/tasks/jenkins.yml | 4 ++-- guests/tasks/packages.yml | 2 +- guests/tasks/users.yml | 4 ++-- 6 files changed, 25 insertions(+), 22 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index 4acb076..206a014 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -431,9 +431,12 @@ class Application: ansible_hosts = ",".join(self._inventory.expand_pattern(hosts)) ansible_cfg_path = os.path.join(base, "ansible.cfg") - playbook_path = os.path.join(base, "site.yml") + playbook_base = base + playbook_path = os.path.join(playbook_base, "site.yml") extra_vars = json.dumps({ + "base": base, + "playbook_base": playbook_base, "root_password_file": root_pass_file, "flavor": flavor, }) diff --git a/guests/site.yml b/guests/site.yml index 063b0c6..4de759b 100644 --- a/guests/site.yml +++ b/guests/site.yml @@ -6,33 +6,33 @@ tasks: # Bootstrap Ansible itself - - include: tasks/bootstrap.yml + - include: '{{ playbook_base }}/tasks/bootstrap.yml' - hosts: all remote_user: root vars_files: - - vars/mappings.yml + - '{{ base }}/vars/mappings.yml' tasks: # Prepare environment. None of the actions performed here might # depend on packages being installed - - include: tasks/base.yml + - include: '{{ playbook_base }}/tasks/base.yml' # Install base packages - - include: tasks/packages.yml + - include: '{{ playbook_base }}/tasks/packages.yml' vars: project: base # Remove blacklisted packages - - include: tasks/packages.yml + - include: '{{ playbook_base }}/tasks/packages.yml' vars: project: blacklist state: absent # Install build dependencies for each project - - include: tasks/packages.yml + - include: '{{ playbook_base }}/tasks/packages.yml' with_items: '{{ projects }}' loop_control: @@ -41,20 +41,20 @@ - projects is defined # Install packages needed for the Jenkins agent - - include: tasks/packages.yml + - include: '{{ playbook_base }}/tasks/packages.yml' vars: project: jenkins when: - flavor == "jenkins" # Configure environment. Needs to happen after installing packages - - include: tasks/paths.yml - - include: tasks/bootloader.yml - - include: tasks/services.yml - - include: tasks/kludges.yml - - include: tasks/users.yml + - include: '{{ playbook_base }}/tasks/paths.yml' + - include: '{{ playbook_base }}/tasks/bootloader.yml' + - include: '{{ playbook_base }}/tasks/services.yml' + - include: '{{ playbook_base }}/tasks/kludges.yml' + - include: '{{ playbook_base }}/tasks/users.yml' # Configure the Jenkins agent - - include: tasks/jenkins.yml + - include: '{{ playbook_base }}/tasks/jenkins.yml' when: - flavor == 'jenkins' diff --git a/guests/tasks/base.yml b/guests/tasks/base.yml index 6310d96..11f600f 100644 --- a/guests/tasks/base.yml +++ b/guests/tasks/base.yml @@ -25,7 +25,7 @@ - name: Enable jessie-backports repository template: - src: templates/jessie-backports.sources.j2 + src: '{{ playbook_base }}/templates/jessie-backports.sources.j2' dest: /etc/apt/sources.list.d/jessie-backports.list owner: root group: root @@ -36,7 +36,7 @@ - name: Configure APT pinning for jessie-backports template: - src: templates/jessie-backports.preferences.j2 + src: '{{ playbook_base }}/templates/jessie-backports.preferences.j2' dest: /etc/apt/preferences.d/jessie-backports owner: root group: root @@ -47,7 +47,7 @@ - name: Enable fedora-rawhide-kernel-nodebug repository template: - src: templates/fedora-rawhide-kernel-nodebug.repo.j2 + src: '{{ playbook_base }}/templates/fedora-rawhide-kernel-nodebug.repo.j2' dest: /etc/yum.repos.d/fedora-rawhide-kernel-nodebug.repo owner: root group: root diff --git a/guests/tasks/jenkins.yml b/guests/tasks/jenkins.yml index 9076c34..479e5d6 100644 --- a/guests/tasks/jenkins.yml +++ b/guests/tasks/jenkins.yml @@ -1,7 +1,7 @@ --- - name: Open vault include_vars: - file: vars/vault.yml + file: '{{ base }}/vars/vault.yml' - name: Look up Jenkins secret set_fact: @@ -32,7 +32,7 @@ - name: Configure Jenkins agent template: - src: templates/jenkins.service.j2 + src: '{{ playbook_base }}/templates/jenkins.service.j2' dest: /etc/systemd/system/jenkins.service when: - jenkins_secret is defined diff --git a/guests/tasks/packages.yml b/guests/tasks/packages.yml index 718ef47..a725b07 100644 --- a/guests/tasks/packages.yml +++ b/guests/tasks/packages.yml @@ -7,7 +7,7 @@ - name: '{{ project }}: Load variables' include_vars: - file: 'vars/projects/{{ project }}.yml' + file: '{{ base }}/vars/projects/{{ project }}.yml' - set_fact: temp: {} diff --git a/guests/tasks/users.yml b/guests/tasks/users.yml index dd6c39d..ec7f798 100644 --- a/guests/tasks/users.yml +++ b/guests/tasks/users.yml @@ -61,7 +61,7 @@ - name: '{{ flavor }}: Configure ccache' template: - src: templates/ccache.conf.j2 + src: '{{ playbook_base }}/templates/ccache.conf.j2' dest: /home/{{ flavor }}/.ccache/ccache.conf owner: '{{ flavor }}' group: '{{ flavor }}' @@ -96,7 +96,7 @@ - name: '{{ flavor }}: Create shell profile' template: - src: templates/{{ item }}.j2 + src: '{{ playbook_base }}/templates/{{ item }}.j2' dest: /home/{{ flavor }}/.{{ item }} owner: '{{ flavor }}' group: '{{ flavor }}' -- 2.17.1

On Thu, Jul 19, 2018 at 06:32:03PM +0200, Andrea Bolognani wrote:
We want to get rid of relative paths in playbooks and tasks, and in order to do that we have to provide Ansible with some more information.
base is the directory where lcitool lives, and playbook_base is the directory where a playbook should look for its private resources: they match for the time being, but that will no longer be the case very shortly.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- Reviewed-by: Erik Skultety <eskultet@redhat.com>

This has two advantages: it unclutters the top-level directory, and opens the door for adding more playbooks down the line without turning it into a mess. Thanks to the changes made earlier, moving the playbook is almost entirely transparent to lcitool. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/lcitool | 4 ++-- guests/{site.yml => playbooks/update/main.yml} | 0 guests/{ => playbooks/update}/tasks/base.yml | 0 guests/{ => playbooks/update}/tasks/bootloader.yml | 0 guests/{ => playbooks/update}/tasks/bootstrap.yml | 0 guests/{ => playbooks/update}/tasks/jenkins.yml | 0 guests/{ => playbooks/update}/tasks/kludges.yml | 0 guests/{ => playbooks/update}/tasks/packages.yml | 0 guests/{ => playbooks/update}/tasks/paths.yml | 0 guests/{ => playbooks/update}/tasks/services.yml | 0 guests/{ => playbooks/update}/tasks/users.yml | 0 guests/{ => playbooks/update}/templates/bash_profile.j2 | 0 guests/{ => playbooks/update}/templates/bashrc.j2 | 0 guests/{ => playbooks/update}/templates/ccache.conf.j2 | 0 .../update}/templates/fedora-rawhide-kernel-nodebug.repo.j2 | 0 guests/{ => playbooks/update}/templates/jenkins.service.j2 | 0 .../update}/templates/jessie-backports.preferences.j2 | 0 .../update}/templates/jessie-backports.sources.j2 | 0 18 files changed, 2 insertions(+), 2 deletions(-) rename guests/{site.yml => playbooks/update/main.yml} (100%) rename guests/{ => playbooks/update}/tasks/base.yml (100%) rename guests/{ => playbooks/update}/tasks/bootloader.yml (100%) rename guests/{ => playbooks/update}/tasks/bootstrap.yml (100%) rename guests/{ => playbooks/update}/tasks/jenkins.yml (100%) rename guests/{ => playbooks/update}/tasks/kludges.yml (100%) rename guests/{ => playbooks/update}/tasks/packages.yml (100%) rename guests/{ => playbooks/update}/tasks/paths.yml (100%) rename guests/{ => playbooks/update}/tasks/services.yml (100%) rename guests/{ => playbooks/update}/tasks/users.yml (100%) rename guests/{ => playbooks/update}/templates/bash_profile.j2 (100%) rename guests/{ => playbooks/update}/templates/bashrc.j2 (100%) rename guests/{ => playbooks/update}/templates/ccache.conf.j2 (100%) rename guests/{ => playbooks/update}/templates/fedora-rawhide-kernel-nodebug.repo.j2 (100%) rename guests/{ => playbooks/update}/templates/jenkins.service.j2 (100%) rename guests/{ => playbooks/update}/templates/jessie-backports.preferences.j2 (100%) rename guests/{ => playbooks/update}/templates/jessie-backports.sources.j2 (100%) diff --git a/guests/lcitool b/guests/lcitool index 206a014..2cfb0e9 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -431,8 +431,8 @@ class Application: ansible_hosts = ",".join(self._inventory.expand_pattern(hosts)) ansible_cfg_path = os.path.join(base, "ansible.cfg") - playbook_base = base - playbook_path = os.path.join(playbook_base, "site.yml") + playbook_base = os.path.join(base, "playbooks", "update") + playbook_path = os.path.join(playbook_base, "main.yml") extra_vars = json.dumps({ "base": base, diff --git a/guests/site.yml b/guests/playbooks/update/main.yml similarity index 100% rename from guests/site.yml rename to guests/playbooks/update/main.yml diff --git a/guests/tasks/base.yml b/guests/playbooks/update/tasks/base.yml similarity index 100% rename from guests/tasks/base.yml rename to guests/playbooks/update/tasks/base.yml diff --git a/guests/tasks/bootloader.yml b/guests/playbooks/update/tasks/bootloader.yml similarity index 100% rename from guests/tasks/bootloader.yml rename to guests/playbooks/update/tasks/bootloader.yml diff --git a/guests/tasks/bootstrap.yml b/guests/playbooks/update/tasks/bootstrap.yml similarity index 100% rename from guests/tasks/bootstrap.yml rename to guests/playbooks/update/tasks/bootstrap.yml diff --git a/guests/tasks/jenkins.yml b/guests/playbooks/update/tasks/jenkins.yml similarity index 100% rename from guests/tasks/jenkins.yml rename to guests/playbooks/update/tasks/jenkins.yml diff --git a/guests/tasks/kludges.yml b/guests/playbooks/update/tasks/kludges.yml similarity index 100% rename from guests/tasks/kludges.yml rename to guests/playbooks/update/tasks/kludges.yml diff --git a/guests/tasks/packages.yml b/guests/playbooks/update/tasks/packages.yml similarity index 100% rename from guests/tasks/packages.yml rename to guests/playbooks/update/tasks/packages.yml diff --git a/guests/tasks/paths.yml b/guests/playbooks/update/tasks/paths.yml similarity index 100% rename from guests/tasks/paths.yml rename to guests/playbooks/update/tasks/paths.yml diff --git a/guests/tasks/services.yml b/guests/playbooks/update/tasks/services.yml similarity index 100% rename from guests/tasks/services.yml rename to guests/playbooks/update/tasks/services.yml diff --git a/guests/tasks/users.yml b/guests/playbooks/update/tasks/users.yml similarity index 100% rename from guests/tasks/users.yml rename to guests/playbooks/update/tasks/users.yml diff --git a/guests/templates/bash_profile.j2 b/guests/playbooks/update/templates/bash_profile.j2 similarity index 100% rename from guests/templates/bash_profile.j2 rename to guests/playbooks/update/templates/bash_profile.j2 diff --git a/guests/templates/bashrc.j2 b/guests/playbooks/update/templates/bashrc.j2 similarity index 100% rename from guests/templates/bashrc.j2 rename to guests/playbooks/update/templates/bashrc.j2 diff --git a/guests/templates/ccache.conf.j2 b/guests/playbooks/update/templates/ccache.conf.j2 similarity index 100% rename from guests/templates/ccache.conf.j2 rename to guests/playbooks/update/templates/ccache.conf.j2 diff --git a/guests/templates/fedora-rawhide-kernel-nodebug.repo.j2 b/guests/playbooks/update/templates/fedora-rawhide-kernel-nodebug.repo.j2 similarity index 100% rename from guests/templates/fedora-rawhide-kernel-nodebug.repo.j2 rename to guests/playbooks/update/templates/fedora-rawhide-kernel-nodebug.repo.j2 diff --git a/guests/templates/jenkins.service.j2 b/guests/playbooks/update/templates/jenkins.service.j2 similarity index 100% rename from guests/templates/jenkins.service.j2 rename to guests/playbooks/update/templates/jenkins.service.j2 diff --git a/guests/templates/jessie-backports.preferences.j2 b/guests/playbooks/update/templates/jessie-backports.preferences.j2 similarity index 100% rename from guests/templates/jessie-backports.preferences.j2 rename to guests/playbooks/update/templates/jessie-backports.preferences.j2 diff --git a/guests/templates/jessie-backports.sources.j2 b/guests/playbooks/update/templates/jessie-backports.sources.j2 similarity index 100% rename from guests/templates/jessie-backports.sources.j2 rename to guests/playbooks/update/templates/jessie-backports.sources.j2 -- 2.17.1

On Thu, Jul 19, 2018 at 06:32:04PM +0200, Andrea Bolognani wrote:
This has two advantages: it unclutters the top-level directory, and opens the door for adding more playbooks down the line without turning it into a mess.
Thanks to the changes made earlier, moving the playbook is almost entirely transparent to lcitool.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- Reviewed-by: Erik Skultety <eskultet@redhat.com>

The information is mostly duplicated and can be easily inferred in a programmatic manner, so storing it in the inventory is far from the cleanest solution. As a side-effect, we reintroduce the error message that was supposed to be displayed when attempting to install a FreeBSD guest but was lost in the Python rewrite. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/host_vars/libvirt-centos-7/install.yml | 1 - guests/host_vars/libvirt-debian-8/install.yml | 1 - guests/host_vars/libvirt-debian-9/install.yml | 1 - .../host_vars/libvirt-debian-sid/install.yml | 1 - .../host_vars/libvirt-fedora-27/install.yml | 1 - .../host_vars/libvirt-fedora-28/install.yml | 1 - .../libvirt-fedora-rawhide/install.yml | 1 - .../host_vars/libvirt-ubuntu-16/install.yml | 1 - .../host_vars/libvirt-ubuntu-18/install.yml | 1 - guests/lcitool | 19 ++++++++++++++----- 10 files changed, 14 insertions(+), 14 deletions(-) diff --git a/guests/host_vars/libvirt-centos-7/install.yml b/guests/host_vars/libvirt-centos-7/install.yml index f003b89..2164ac5 100644 --- a/guests/host_vars/libvirt-centos-7/install.yml +++ b/guests/host_vars/libvirt-centos-7/install.yml @@ -1,3 +1,2 @@ --- install_url: http://mirror.centos.org/centos/7/os/x86_64/ -install_config: kickstart.cfg diff --git a/guests/host_vars/libvirt-debian-8/install.yml b/guests/host_vars/libvirt-debian-8/install.yml index a2c8341..299a1a6 100644 --- a/guests/host_vars/libvirt-debian-8/install.yml +++ b/guests/host_vars/libvirt-debian-8/install.yml @@ -1,3 +1,2 @@ --- install_url: http://deb.debian.org/debian/dists/jessie/main/installer-amd64/ -install_config: preseed.cfg diff --git a/guests/host_vars/libvirt-debian-9/install.yml b/guests/host_vars/libvirt-debian-9/install.yml index 5b1da76..7641753 100644 --- a/guests/host_vars/libvirt-debian-9/install.yml +++ b/guests/host_vars/libvirt-debian-9/install.yml @@ -1,3 +1,2 @@ --- install_url: http://deb.debian.org/debian/dists/stretch/main/installer-amd64/ -install_config: preseed.cfg diff --git a/guests/host_vars/libvirt-debian-sid/install.yml b/guests/host_vars/libvirt-debian-sid/install.yml index da1c7a8..46c6366 100644 --- a/guests/host_vars/libvirt-debian-sid/install.yml +++ b/guests/host_vars/libvirt-debian-sid/install.yml @@ -1,3 +1,2 @@ --- install_url: http://deb.debian.org/debian/dists/sid/main/installer-amd64/ -install_config: preseed.cfg diff --git a/guests/host_vars/libvirt-fedora-27/install.yml b/guests/host_vars/libvirt-fedora-27/install.yml index 66ce38e..f7a45af 100644 --- a/guests/host_vars/libvirt-fedora-27/install.yml +++ b/guests/host_vars/libvirt-fedora-27/install.yml @@ -1,3 +1,2 @@ --- install_url: https://download.fedoraproject.org/pub/fedora/linux/releases/27/Everything/x... -install_config: kickstart.cfg diff --git a/guests/host_vars/libvirt-fedora-28/install.yml b/guests/host_vars/libvirt-fedora-28/install.yml index 4b2b9f0..73433f1 100644 --- a/guests/host_vars/libvirt-fedora-28/install.yml +++ b/guests/host_vars/libvirt-fedora-28/install.yml @@ -1,3 +1,2 @@ --- install_url: https://download.fedoraproject.org/pub/fedora/linux/releases/28/Everything/x... -install_config: kickstart.cfg diff --git a/guests/host_vars/libvirt-fedora-rawhide/install.yml b/guests/host_vars/libvirt-fedora-rawhide/install.yml index 2216e81..5c67562 100644 --- a/guests/host_vars/libvirt-fedora-rawhide/install.yml +++ b/guests/host_vars/libvirt-fedora-rawhide/install.yml @@ -1,3 +1,2 @@ --- install_url: https://download.fedoraproject.org/pub/fedora/linux/development/rawhide/Ever... -install_config: kickstart.cfg diff --git a/guests/host_vars/libvirt-ubuntu-16/install.yml b/guests/host_vars/libvirt-ubuntu-16/install.yml index a7bb2da..d8ce841 100644 --- a/guests/host_vars/libvirt-ubuntu-16/install.yml +++ b/guests/host_vars/libvirt-ubuntu-16/install.yml @@ -1,3 +1,2 @@ --- install_url: http://archive.ubuntu.com/ubuntu/dists/xenial/main/installer-amd64/ -install_config: preseed.cfg diff --git a/guests/host_vars/libvirt-ubuntu-18/install.yml b/guests/host_vars/libvirt-ubuntu-18/install.yml index bd3e1d9..544b3f2 100644 --- a/guests/host_vars/libvirt-ubuntu-18/install.yml +++ b/guests/host_vars/libvirt-ubuntu-18/install.yml @@ -1,3 +1,2 @@ --- install_url: http://archive.ubuntu.com/ubuntu/dists/bionic/main/installer-amd64/ -install_config: preseed.cfg diff --git a/guests/lcitool b/guests/lcitool index 2cfb0e9..13f0392 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -382,15 +382,24 @@ class Application: facts["install_network"], ) - install_config = os.path.join(base, facts["install_config"]) + # Different operating systems require different configuration + # files for unattended installation to work, but some operating + # systems simply don't support unattended installation at all + if facts["os_name"] in ["Debian", "Ubuntu"]: + install_config = "preseed.cfg" + elif facts["os_name"] in ["CentOS", "Fedora"]: + install_config = "kickstart.cfg" + else: + raise Error( + "Host {} doesn't support installation".format(host) + ) + initrd_inject = os.path.join(base, install_config) # preseed files must use a well-known name to be picked up by # d-i; for kickstart files, we can use whatever name we please # but we need to point anaconda in the right direction through # a kernel argument - extra_arg = "console=ttyS0 ks=file:/{}".format( - facts["install_config"], - ) + extra_arg = "console=ttyS0 ks=file:/{}".format(install_config) cmd = [ "virt-install", @@ -407,7 +416,7 @@ class Application: "--graphics", "none", "--console", "pty", "--sound", "none", - "--initrd-inject", install_config, + "--initrd-inject", initrd_inject, "--extra-args", extra_arg, "--wait", "0", ] -- 2.17.1

On Thu, Jul 19, 2018 at 06:32:05PM +0200, Andrea Bolognani wrote:
The information is mostly duplicated and can be easily inferred in a programmatic manner, so storing it in the inventory is far from the cleanest solution.
As a side-effect, we reintroduce the error message that was supposed to be displayed when attempting to install a FreeBSD guest but was lost in the Python rewrite.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/host_vars/libvirt-centos-7/install.yml | 1 - guests/host_vars/libvirt-debian-8/install.yml | 1 - guests/host_vars/libvirt-debian-9/install.yml | 1 - .../host_vars/libvirt-debian-sid/install.yml | 1 - .../host_vars/libvirt-fedora-27/install.yml | 1 - .../host_vars/libvirt-fedora-28/install.yml | 1 - .../libvirt-fedora-rawhide/install.yml | 1 - .../host_vars/libvirt-ubuntu-16/install.yml | 1 - .../host_vars/libvirt-ubuntu-18/install.yml | 1 - guests/lcitool | 19 ++++++++++++++----- 10 files changed, 14 insertions(+), 14 deletions(-)
diff --git a/guests/host_vars/libvirt-centos-7/install.yml b/guests/host_vars/libvirt-centos-7/install.yml index f003b89..2164ac5 100644 --- a/guests/host_vars/libvirt-centos-7/install.yml +++ b/guests/host_vars/libvirt-centos-7/install.yml @@ -1,3 +1,2 @@ --- install_url: http://mirror.centos.org/centos/7/os/x86_64/ -install_config: kickstart.cfg diff --git a/guests/host_vars/libvirt-debian-8/install.yml b/guests/host_vars/libvirt-debian-8/install.yml index a2c8341..299a1a6 100644 --- a/guests/host_vars/libvirt-debian-8/install.yml +++ b/guests/host_vars/libvirt-debian-8/install.yml @@ -1,3 +1,2 @@ --- install_url: http://deb.debian.org/debian/dists/jessie/main/installer-amd64/ -install_config: preseed.cfg diff --git a/guests/host_vars/libvirt-debian-9/install.yml b/guests/host_vars/libvirt-debian-9/install.yml index 5b1da76..7641753 100644 --- a/guests/host_vars/libvirt-debian-9/install.yml +++ b/guests/host_vars/libvirt-debian-9/install.yml @@ -1,3 +1,2 @@ --- install_url: http://deb.debian.org/debian/dists/stretch/main/installer-amd64/ -install_config: preseed.cfg diff --git a/guests/host_vars/libvirt-debian-sid/install.yml b/guests/host_vars/libvirt-debian-sid/install.yml index da1c7a8..46c6366 100644 --- a/guests/host_vars/libvirt-debian-sid/install.yml +++ b/guests/host_vars/libvirt-debian-sid/install.yml @@ -1,3 +1,2 @@ --- install_url: http://deb.debian.org/debian/dists/sid/main/installer-amd64/ -install_config: preseed.cfg diff --git a/guests/host_vars/libvirt-fedora-27/install.yml b/guests/host_vars/libvirt-fedora-27/install.yml index 66ce38e..f7a45af 100644 --- a/guests/host_vars/libvirt-fedora-27/install.yml +++ b/guests/host_vars/libvirt-fedora-27/install.yml @@ -1,3 +1,2 @@ --- install_url: https://download.fedoraproject.org/pub/fedora/linux/releases/27/Everything/x... -install_config: kickstart.cfg diff --git a/guests/host_vars/libvirt-fedora-28/install.yml b/guests/host_vars/libvirt-fedora-28/install.yml index 4b2b9f0..73433f1 100644 --- a/guests/host_vars/libvirt-fedora-28/install.yml +++ b/guests/host_vars/libvirt-fedora-28/install.yml @@ -1,3 +1,2 @@ --- install_url: https://download.fedoraproject.org/pub/fedora/linux/releases/28/Everything/x... -install_config: kickstart.cfg diff --git a/guests/host_vars/libvirt-fedora-rawhide/install.yml b/guests/host_vars/libvirt-fedora-rawhide/install.yml index 2216e81..5c67562 100644 --- a/guests/host_vars/libvirt-fedora-rawhide/install.yml +++ b/guests/host_vars/libvirt-fedora-rawhide/install.yml @@ -1,3 +1,2 @@ --- install_url: https://download.fedoraproject.org/pub/fedora/linux/development/rawhide/Ever... -install_config: kickstart.cfg diff --git a/guests/host_vars/libvirt-ubuntu-16/install.yml b/guests/host_vars/libvirt-ubuntu-16/install.yml index a7bb2da..d8ce841 100644 --- a/guests/host_vars/libvirt-ubuntu-16/install.yml +++ b/guests/host_vars/libvirt-ubuntu-16/install.yml @@ -1,3 +1,2 @@ --- install_url: http://archive.ubuntu.com/ubuntu/dists/xenial/main/installer-amd64/ -install_config: preseed.cfg diff --git a/guests/host_vars/libvirt-ubuntu-18/install.yml b/guests/host_vars/libvirt-ubuntu-18/install.yml index bd3e1d9..544b3f2 100644 --- a/guests/host_vars/libvirt-ubuntu-18/install.yml +++ b/guests/host_vars/libvirt-ubuntu-18/install.yml @@ -1,3 +1,2 @@ --- install_url: http://archive.ubuntu.com/ubuntu/dists/bionic/main/installer-amd64/ -install_config: preseed.cfg diff --git a/guests/lcitool b/guests/lcitool index 2cfb0e9..13f0392 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -382,15 +382,24 @@ class Application: facts["install_network"], )
- install_config = os.path.join(base, facts["install_config"]) + # Different operating systems require different configuration + # files for unattended installation to work, but some operating + # systems simply don't support unattended installation at all + if facts["os_name"] in ["Debian", "Ubuntu"]: + install_config = "preseed.cfg" + elif facts["os_name"] in ["CentOS", "Fedora"]: + install_config = "kickstart.cfg" + else: + raise Error( + "Host {} doesn't support installation".format(host) + ) + initrd_inject = os.path.join(base, install_config)
# preseed files must use a well-known name to be picked up by # d-i; for kickstart files, we can use whatever name we please # but we need to point anaconda in the right direction through # a kernel argument - extra_arg = "console=ttyS0 ks=file:/{}".format( - facts["install_config"], - ) + extra_arg = "console=ttyS0 ks=file:/{}".format(install_config)
Pre-existing, but ks= should be replaced by inst.ks, as using the inst. prefix is the current preferred way of passing installer-specific options, the legacy syntax is still accepted, however, this may be a subject to future changes - the inst. prefix is supported since fedora 17 and RHEL/CentOS 7 and since we dropped CentOS 6 from the supported distros, we don't need to keep the legacy option :). The patch is fine though. Reviewed-by: Erik Skultety <eskultet@redhat.com>

On Wed, 2018-08-01 at 14:46 +0200, Erik Skultety wrote:
On Thu, Jul 19, 2018 at 06:32:05PM +0200, Andrea Bolognani wrote:
# preseed files must use a well-known name to be picked up by # d-i; for kickstart files, we can use whatever name we please # but we need to point anaconda in the right direction through # a kernel argument - extra_arg = "console=ttyS0 ks=file:/{}".format( - facts["install_config"], - ) + extra_arg = "console=ttyS0 ks=file:/{}".format(install_config)
Pre-existing, but ks= should be replaced by inst.ks, as using the inst. prefix is the current preferred way of passing installer-specific options, the legacy syntax is still accepted, however, this may be a subject to future changes - the inst. prefix is supported since fedora 17 and RHEL/CentOS 7 and since we dropped CentOS 6 from the supported distros, we don't need to keep the legacy option :).
Today I learned! :) We can definitely switch to the non-legacy version, and I suspect that if we spent some time on it we might be able to modernize the kickstart file itself a bit as well... -- Andrea Bolognani / Red Hat / Virtualization

The rationale is the same as for moving playbooks. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/{ => configs}/kickstart.cfg | 0 guests/{ => configs}/preseed.cfg | 0 guests/lcitool | 2 +- 3 files changed, 1 insertion(+), 1 deletion(-) rename guests/{ => configs}/kickstart.cfg (100%) rename guests/{ => configs}/preseed.cfg (100%) diff --git a/guests/kickstart.cfg b/guests/configs/kickstart.cfg similarity index 100% rename from guests/kickstart.cfg rename to guests/configs/kickstart.cfg diff --git a/guests/preseed.cfg b/guests/configs/preseed.cfg similarity index 100% rename from guests/preseed.cfg rename to guests/configs/preseed.cfg diff --git a/guests/lcitool b/guests/lcitool index 13f0392..96a59ac 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -393,7 +393,7 @@ class Application: raise Error( "Host {} doesn't support installation".format(host) ) - initrd_inject = os.path.join(base, install_config) + initrd_inject = os.path.join(base, "configs", install_config) # preseed files must use a well-known name to be picked up by # d-i; for kickstart files, we can use whatever name we please -- 2.17.1

On Thu, Jul 19, 2018 at 06:32:06PM +0200, Andrea Bolognani wrote:
The rationale is the same as for moving playbooks.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- Reviewed-by: Erik Skultety <eskultet@redhat.com>

For CentOS CI, we need build dependencies for all known projects to be installed; however, when using lcitool for development purposes, it is very convenient to install just the subset relevant to the project that's being worked on, as doing so reduces the storage requirements and makes the update procedure quite a bit faster. The previous behavior can still be obtained by using $ lcitool -a update -p all ... Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/lcitool | 4 +++- guests/playbooks/update/main.yml | 4 ++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index 96a59ac..2aa9674 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -430,7 +430,7 @@ class Application: except Exception: raise Error("Failed to install '{}'".format(host)) - def _action_update(self, hosts, _projects): + def _action_update(self, hosts, projects): base = Util.get_base() flavor = self._config.get_flavor() @@ -438,6 +438,7 @@ class Application: root_pass_file = self._config.get_root_password_file() ansible_hosts = ",".join(self._inventory.expand_pattern(hosts)) + selected_projects = self._projects.expand_pattern(projects) ansible_cfg_path = os.path.join(base, "ansible.cfg") playbook_base = os.path.join(base, "playbooks", "update") @@ -448,6 +449,7 @@ class Application: "playbook_base": playbook_base, "root_password_file": root_pass_file, "flavor": flavor, + "selected_projects": selected_projects, }) cmd = [ diff --git a/guests/playbooks/update/main.yml b/guests/playbooks/update/main.yml index 4de759b..753bac4 100644 --- a/guests/playbooks/update/main.yml +++ b/guests/playbooks/update/main.yml @@ -34,11 +34,11 @@ # Install build dependencies for each project - include: '{{ playbook_base }}/tasks/packages.yml' with_items: - '{{ projects }}' + '{{ selected_projects }}' loop_control: loop_var: project when: - - projects is defined + - project in projects # Install packages needed for the Jenkins agent - include: '{{ playbook_base }}/tasks/packages.yml' -- 2.17.1

On Thu, Jul 19, 2018 at 06:32:07PM +0200, Andrea Bolognani wrote:
For CentOS CI, we need build dependencies for all known projects to be installed; however, when using lcitool for development purposes, it is very convenient to install just the subset relevant to the project that's being worked on, as doing so reduces the storage requirements and makes the update procedure quite a bit faster.
The previous behavior can still be obtained by using
$ lcitool -a update -p all ...
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- Reviewed-by: Erik Skultety <eskultet@redhat.com>

The usage has once again changed slightly; additionally, a few concrete examples are now provided. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/README.markdown | 38 ++++++++++++++++++++++++++++++++------ 1 file changed, 32 insertions(+), 6 deletions(-) diff --git a/guests/README.markdown b/guests/README.markdown index 4a40619..ddf0149 100644 --- a/guests/README.markdown +++ b/guests/README.markdown @@ -4,22 +4,48 @@ libvirt CI - guest management tools The tools contained in this directory simplify and automate the management of the guests used by the Jenkins-based libvirt CI environment. + +Usage and examples +------------------ + There are two steps to bringing up a guest: -* `./lcitool -a install -h $guest` will perform an unattended installation +* `lcitool -a install -h $guest` will perform an unattended installation of `$guest`. Not all guests can be installed this way: see the "FreeBSD" section below; -* `./lcitool -a update -h $guest` will go through all the post-installation - configuration steps required to make the newly-created guest usable; +* `lcitool -a update -h $guest -p $project` will go through all the + post-installation configuration steps required to make the newly-created + guest usable and ready to be used for building `$project`; Once those steps have been performed, maintainance will involve running: -* `./lcitool -a update -h $guest` + lcitool -a update -h $guest -p $project periodically to ensure the guest configuration is sane and all installed packages are updated. +To get a list of known guests and projects, run + + lcitool -a hosts + +and + + lcitool -a projects + +respectively. You can run operations involving multiple guests and projects +at once by providing a list on the command line: for example, running + + lcitool -a update -h '*fedora*' -p '*osinfo*' + +will update all Fedora guests and get them ready to build libosinfo and +related projects, while running + + lcitool -a update -h all -p libvirt,libvirt+mingw + +will update all hosts and prepare them to build libvirt both as a native +library and, where supported, as a Windows library using MinGW. + Host setup ---------- @@ -40,13 +66,13 @@ you'll want to use the `libvirt_guest` variant of the plugin. To keep guests up to date over time, it's recommended to have an entry along the lines of - 0 0 * * * cd ~/libvirt-jenkins-ci/guests && ./lcitool -a update -h all + 0 0 * * * ~/libvirt-jenkins-ci/guests/lcitool -a update -h all -p all in your crontab. Test use ---------------- +-------- If you are a developer trying to reproduce a bug on some OS you don't have easy access to, you can use these tools to create a suitable test -- 2.17.1

On Thu, Jul 19, 2018 at 06:32:08PM +0200, Andrea Bolognani wrote:
The usage has once again changed slightly; additionally, a few concrete examples are now provided.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- Reviewed-by: Erik Skultety <eskultet@redhat.com>
participants (2)
-
Andrea Bolognani
-
Erik Skultety