[libvirt] [jenkins-ci PATCH 0/5] do testing of the libvirt-jenkins-ci project

This sets things up so that we do testing of the libvirt-jenkins-ci via flake8, and also validate the docker file builder works. Daniel P. Berrangé (5): make: add a rule for running a 'syntax-check' with flake8 lcitool: use subparsers for commands lcitool: allow restricting host list to those supporting docker make: add a simple build target that generates all dockerfiles Add project for CI testing libvirt-jenkins-ci Makefile | 23 +++ guests/lcitool | 145 +++++++++++------- guests/playbooks/build/jobs/defaults.yml | 2 + .../build/projects/libvirt-jenkins-ci.yml | 15 ++ guests/vars/projects/libvirt-jenkins-ci.yml | 4 + jobs/defaults.yaml | 2 + projects/libvirt-jenkins-ci.yaml | 14 ++ 7 files changed, 148 insertions(+), 57 deletions(-) create mode 100644 Makefile create mode 100644 guests/playbooks/build/projects/libvirt-jenkins-ci.yml create mode 100644 guests/vars/projects/libvirt-jenkins-ci.yml create mode 100644 projects/libvirt-jenkins-ci.yaml -- 2.20.1

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 Makefile diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..1ba6b21 --- /dev/null +++ b/Makefile @@ -0,0 +1,4 @@ + + +syntax-check: + flake8 guests/lcitool -- 2.20.1

Currently only a single global parser is used for all commands. This means that every command accepts every argument which is undesirable as users don't know what to pass. It also prevents the parser from generating useful help information. Python's argparse module supports multi-command binaries in a very easy way by adding subparsers, each of which has their own distinct arguments. It can then generate suitable help text on a per command basis. This also means commands can use positional arguments for data items that are always passed. $ ./guests/lcitool -h usage: lcitool [-h] {install,update,build,hosts,projects,dockerfile} ... libvirt CI guest management tool positional arguments: {install,update,build,hosts,projects,dockerfile} commands install perform unattended host installation update prepare hosts and keep them updated build build projects on hosts hosts list all known hosts projects list all known projects dockerfile generate a host docker file optional arguments: -h, --help show this help message and exit $ ./guests/lcitool install -h usage: lcitool install [-h] hosts positional arguments: hosts list of hosts to act on (accepts globs) optional arguments: -h, --help show this help message and exit $ ./guests/lcitool dockerfile -h usage: lcitool dockerfile [-h] hosts projects positional arguments: hosts list of hosts to act on (accepts globs) projects list of projects to consider (accepts globs) optional arguments: -h, --help show this help message and exit Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- guests/lcitool | 135 ++++++++++++++++++++++++++++--------------------- 1 file changed, 78 insertions(+), 57 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index 726e3bb..35a6b68 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -307,43 +307,72 @@ class Application: conflict_handler="resolve", formatter_class=argparse.RawDescriptionHelpFormatter, description="libvirt CI guest management tool", - epilog=textwrap.dedent(""" - common actions: - install perform unattended host installation - update prepare hosts and keep them updated - build build projects on hosts + ) - informational actions: - hosts list all known hosts - projects list all known projects + subparser = self._parser.add_subparsers(help="commands") + subparser.required = True + subparser.dest = "command" - uncommon actions: - dockerfile generate Dockerfile (doesn't access the host) + def add_hosts_arg(parser): + parser.add_argument( + "hosts", + help="list of hosts to act on (accepts globs)", + ) - glob patterns are supported for HOSTS and PROJECTS - """), - ) - self._parser.add_argument( - "-a", - metavar="ACTION", - required=True, - help="action to perform (see below)", - ) - self._parser.add_argument( - "-h", - metavar="HOSTS", - help="list of hosts to act on", - ) - self._parser.add_argument( - "-p", - metavar="PROJECTS", - help="list of projects to consider", - ) - self._parser.add_argument( - "-g", - metavar="GIT_REVISION", - help="git revision to build (remote/branch)", - ) + def add_projects_arg(parser): + parser.add_argument( + "projects", + help="list of projects to consider (accepts globs)", + ) + + def add_gitrev_arg(parser): + parser.add_argument( + "-g", "--git-revision", + help="git revision to build (remote/branch)", + ) + + installparser = subparser.add_parser( + "install", help="perform unattended host installation", + formatter_class=argparse.RawDescriptionHelpFormatter) + installparser.set_defaults(func=self._action_install) + + add_hosts_arg(installparser) + + updateparser = subparser.add_parser( + "update", help="prepare hosts and keep them updated", + formatter_class=argparse.RawDescriptionHelpFormatter) + updateparser.set_defaults(func=self._action_update) + + add_hosts_arg(updateparser) + add_projects_arg(updateparser) + add_gitrev_arg(updateparser) + + buildparser = subparser.add_parser( + "build", help="build projects on hosts", + formatter_class=argparse.RawDescriptionHelpFormatter) + buildparser.set_defaults(func=self._action_build) + + add_hosts_arg(buildparser) + add_projects_arg(buildparser) + add_gitrev_arg(buildparser) + + hostsparser = subparser.add_parser( + "hosts", help="list all known hosts", + formatter_class=argparse.RawDescriptionHelpFormatter) + hostsparser.set_defaults(func=self._action_hosts) + + projectsparser = subparser.add_parser( + "projects", help="list all known projects", + formatter_class=argparse.RawDescriptionHelpFormatter) + projectsparser.set_defaults(func=self._action_projects) + + dockerfileparser = subparser.add_parser( + "dockerfile", help="generate a host docker file", + formatter_class=argparse.RawDescriptionHelpFormatter) + dockerfileparser.set_defaults(func=self._action_dockerfile) + + add_hosts_arg(dockerfileparser) + add_projects_arg(dockerfileparser) def _execute_playbook(self, playbook, hosts, projects, git_revision): base = Util.get_base() @@ -404,20 +433,20 @@ class Application: raise Error( "Failed to run {} on '{}': {}".format(playbook, hosts, ex)) - def _action_hosts(self, _hosts, _projects, _revision): + def _action_hosts(self, args): for host in self._inventory.expand_pattern("all"): print(host) - def _action_projects(self, _hosts, _projects, _revision): + def _action_projects(self, args): for project in self._projects.expand_pattern("all"): print(project) - def _action_install(self, hosts, _projects, _revision): + def _action_install(self, args): base = Util.get_base() flavor = self._config.get_flavor() - for host in self._inventory.expand_pattern(hosts): + for host in self._inventory.expand_pattern(args.hosts): facts = self._inventory.get_facts(host) # Both memory size and disk size are stored as GiB in the @@ -484,16 +513,18 @@ class Application: except Exception as ex: raise Error("Failed to install '{}': {}".format(host, ex)) - def _action_update(self, hosts, projects, git_revision): - self._execute_playbook("update", hosts, projects, git_revision) + def _action_update(self, args): + self._execute_playbook("update", args.hosts, args.projects, + args.git_revision) - def _action_build(self, hosts, projects, git_revision): - self._execute_playbook("build", hosts, projects, git_revision) + def _action_build(self, args): + self._execute_playbook("build", args.hosts, args.projects, + args.git_revision) - def _action_dockerfile(self, hosts, projects, _revision): + def _action_dockerfile(self, args): mappings = self._projects.get_mappings() - hosts = self._inventory.expand_pattern(hosts) + hosts = self._inventory.expand_pattern(args.hosts) if len(hosts) > 1: raise Error("Can't generate Dockerfile for multiple hosts") host = hosts[0] @@ -507,7 +538,7 @@ class Application: if package_format not in ["deb", "rpm"]: raise Error("Host {} doesn't support Dockerfiles".format(host)) - projects = self._projects.expand_pattern(projects) + projects = self._projects.expand_pattern(args.projects) for project in projects: if project not in facts["projects"]: raise Error( @@ -573,18 +604,8 @@ class Application: """).format(**varmap)) def run(self): - cmdline = self._parser.parse_args() - action = cmdline.a - hosts = cmdline.h - projects = cmdline.p - git_revision = cmdline.g - - method = "_action_{}".format(action.replace("-", "_")) - - if hasattr(self, method): - getattr(self, method).__call__(hosts, projects, git_revision) - else: - raise Error("Invalid action '{}'".format(action)) + args = self._parser.parse_args() + args.func(args) if __name__ == "__main__": -- 2.20.1

Add a '-d' arg to the 'hosts' command to restrict the output to only include hosts on which dockerfiles can be built Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- guests/lcitool | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/guests/lcitool b/guests/lcitool index 35a6b68..374ba50 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -361,6 +361,11 @@ class Application: formatter_class=argparse.RawDescriptionHelpFormatter) hostsparser.set_defaults(func=self._action_hosts) + hostsparser.add_argument( + "-d", "--dockerfiles", + action="store_true", + help="Only list hosts supporting dockerfiles") + projectsparser = subparser.add_parser( "projects", help="list all known projects", formatter_class=argparse.RawDescriptionHelpFormatter) @@ -435,6 +440,11 @@ class Application: def _action_hosts(self, args): for host in self._inventory.expand_pattern("all"): + if args.dockerfiles: + facts = self._inventory.get_facts(host) + package_format = facts["package_format"] + if package_format not in ["deb", "rpm"]: + continue print(host) def _action_projects(self, args): -- 2.20.1

This is useful for local testing of docker changes, and can also be called to rebuild the content needed for the libvirt-dockerfiles repository. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- Makefile | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/Makefile b/Makefile index 1ba6b21..d2c89e9 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,23 @@ +LCITOOL = guests/lcitool + +HOST_VARS = $(wildcard guests/host_vars/*/*.yml) +PROJECT_VARS = $(wildcard guests/vars/projects/*.yml) + +all: dockerfiles syntax-check: flake8 guests/lcitool + +.PHONY: dockerfiles + +DOCKERHOSTS=$(shell $(LCITOOL) hosts -d) + +build/dockerfiles/%.docker: Makefile $(LCITOOL) $(HOST_VARS) $(PROJECT_VARS) + mkdir -p build/dockerfiles + $(LCITOOL) dockerfile $* libvirt > $@ + +dockerfiles: $(DOCKERHOSTS:%=build/dockerfiles/%.docker) + +clean: + rm -rf build/ -- 2.20.1

A build task will validate that the dockerfile generator is able to create output, and a syntax-check task will run flake8 Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- guests/playbooks/build/jobs/defaults.yml | 2 ++ .../build/projects/libvirt-jenkins-ci.yml | 15 +++++++++++++++ guests/vars/projects/libvirt-jenkins-ci.yml | 4 ++++ jobs/defaults.yaml | 2 ++ projects/libvirt-jenkins-ci.yaml | 14 ++++++++++++++ 5 files changed, 37 insertions(+) create mode 100644 guests/playbooks/build/projects/libvirt-jenkins-ci.yml create mode 100644 guests/vars/projects/libvirt-jenkins-ci.yml create mode 100644 projects/libvirt-jenkins-ci.yaml diff --git a/guests/playbooks/build/jobs/defaults.yml b/guests/playbooks/build/jobs/defaults.yml index 1c241fe..73cef08 100644 --- a/guests/playbooks/build/jobs/defaults.yml +++ b/guests/playbooks/build/jobs/defaults.yml @@ -52,6 +52,8 @@ git_urls: default: https://github.com/libvirt/libvirt-go-xml.git libvirt-go: default: https://github.com/libvirt/libvirt-go.git + libvirt-jenkins-ci: + default: https://github.com/libvirt/libvirt-jenkins-ci.git libvirt-ocaml: default: https://github.com/libvirt/libvirt-ocaml.git libvirt-perl: diff --git a/guests/playbooks/build/projects/libvirt-jenkins-ci.yml b/guests/playbooks/build/projects/libvirt-jenkins-ci.yml new file mode 100644 index 0000000..0314009 --- /dev/null +++ b/guests/playbooks/build/projects/libvirt-jenkins-ci.yml @@ -0,0 +1,15 @@ +--- +- set_fact: + name: libvirt-jenkins-ci + machines: '{{ rpm_machines }}' + git_url: '{{ git_urls["libvirt-jenkins-ci"][git_remote] }}' + +- include: '{{ playbook_base }}/jobs/prepare.yml' +- include: '{{ playbook_base }}/jobs/generic-build-job.yml' + vars: + command: | + $MAKE all +- include: '{{ playbook_base }}/jobs/generic-syntax-check-job.yml' + vars: + command: | + $MAKE syntax-check diff --git a/guests/vars/projects/libvirt-jenkins-ci.yml b/guests/vars/projects/libvirt-jenkins-ci.yml new file mode 100644 index 0000000..e8c6fa5 --- /dev/null +++ b/guests/vars/projects/libvirt-jenkins-ci.yml @@ -0,0 +1,4 @@ +--- +packages: + - python3 + - python3-flake8 diff --git a/jobs/defaults.yaml b/jobs/defaults.yaml index 6b39724..19abb78 100644 --- a/jobs/defaults.yaml +++ b/jobs/defaults.yaml @@ -51,6 +51,8 @@ default: https://github.com/libvirt/libvirt-go-xml.git libvirt-go: default: https://github.com/libvirt/libvirt-go.git + libvirt-jenkins-ci: + default: https://github.com/libvirt/libvirt-jenkins-ci.git libvirt-ocaml: default: https://github.com/libvirt/libvirt-ocaml.git libvirt-perl: diff --git a/projects/libvirt-jenkins-ci.yaml b/projects/libvirt-jenkins-ci.yaml new file mode 100644 index 0000000..04d13c4 --- /dev/null +++ b/projects/libvirt-jenkins-ci.yaml @@ -0,0 +1,14 @@ + +- project: + name: libvirt-jenkins-ci + machines: '{rpm_machines}' + title: libvirt Jenkins CI + git_url: '{git_urls[libvirt-jenkins-ci][default]}' + jobs: + - generic-build-job: + command: | + $MAKE all + - generic-syntax-check-job: + parent_jobs: 'libvirt-jenkins-ci-build' + command: | + $MAKE syntax-check -- 2.20.1
participants (1)
-
Daniel P. Berrangé