[libvirt PATCH 0/4] RFC: tests: introduce lavocado
lavocado aims to be an alternative test framework for the libvirt project using Python, python-libvirt and Avocado. This can be used to write unit, functional and integration tests and it is inspired by the libvirt-tck framework. This series introduces the basic framework along with some basic test examples that I got from libvirt-tck. I would appreciate your comments on this RFC, to see if this fits this project's needs. Also, names and locations are just a proposal and can be changed. For now, this framework assumes that you are going to run the tests in a fresh clean environment, i.e. a VM. If you decide to use your local system, beware that execution of the tests may affect your system. One of the future goals of this framework is to utilize nested virtualization technologies and hence make sure an L1 guest is provisioned automatically for the tests to be executed in this environment and not tamper with your main system. I'm adding more information with some details inside the README file. Beraldo Leal (4): tests: introduce lavocado: initial code structure tests.lavocado: adding basic transient domain tests tests.lavocado: adding a .gitignore tests.lavocado: adding a README and Makefile for convenience tests/lavocado/.gitignore | 3 + tests/lavocado/Makefile | 2 + tests/lavocado/README.md | 124 +++++++++++++++++ tests/lavocado/lavocado/__init__.py | 0 tests/lavocado/lavocado/defaults.py | 11 ++ tests/lavocado/lavocado/exceptions.py | 20 +++ tests/lavocado/lavocado/helpers/__init__.py | 0 tests/lavocado/lavocado/helpers/domains.py | 75 ++++++++++ tests/lavocado/lavocado/test.py | 144 ++++++++++++++++++++ tests/lavocado/requirements.txt | 3 + tests/lavocado/templates/domain.xml.jinja | 20 +++ tests/lavocado/tests/domain/transient.py | 102 ++++++++++++++ 12 files changed, 504 insertions(+) create mode 100644 tests/lavocado/.gitignore create mode 100644 tests/lavocado/Makefile create mode 100644 tests/lavocado/README.md create mode 100644 tests/lavocado/lavocado/__init__.py create mode 100644 tests/lavocado/lavocado/defaults.py create mode 100644 tests/lavocado/lavocado/exceptions.py create mode 100644 tests/lavocado/lavocado/helpers/__init__.py create mode 100644 tests/lavocado/lavocado/helpers/domains.py create mode 100644 tests/lavocado/lavocado/test.py create mode 100644 tests/lavocado/requirements.txt create mode 100644 tests/lavocado/templates/domain.xml.jinja create mode 100644 tests/lavocado/tests/domain/transient.py -- 2.26.3
lavocado aims to be an alternative test framework for the libvirt project using Python, python-libvirt and Avocado. This can be used to write unit, functional and integration tests and it is inspired by the libvirt-tck framework. Documentation, helper classes and templates will be provided to speed up common test writing scenarios. Signed-off-by: Beraldo Leal <bleal@redhat.com> --- tests/lavocado/lavocado/__init__.py | 0 tests/lavocado/lavocado/defaults.py | 11 ++ tests/lavocado/lavocado/exceptions.py | 20 +++ tests/lavocado/lavocado/helpers/__init__.py | 0 tests/lavocado/lavocado/helpers/domains.py | 75 ++++++++++ tests/lavocado/lavocado/test.py | 144 ++++++++++++++++++++ tests/lavocado/requirements.txt | 3 + tests/lavocado/templates/domain.xml.jinja | 20 +++ 8 files changed, 273 insertions(+) create mode 100644 tests/lavocado/lavocado/__init__.py create mode 100644 tests/lavocado/lavocado/defaults.py create mode 100644 tests/lavocado/lavocado/exceptions.py create mode 100644 tests/lavocado/lavocado/helpers/__init__.py create mode 100644 tests/lavocado/lavocado/helpers/domains.py create mode 100644 tests/lavocado/lavocado/test.py create mode 100644 tests/lavocado/requirements.txt create mode 100644 tests/lavocado/templates/domain.xml.jinja diff --git a/tests/lavocado/lavocado/__init__.py b/tests/lavocado/lavocado/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/lavocado/lavocado/defaults.py b/tests/lavocado/lavocado/defaults.py new file mode 100644 index 0000000000..47f1299cf4 --- /dev/null +++ b/tests/lavocado/lavocado/defaults.py @@ -0,0 +1,11 @@ +LIBVIRT_URI = "qemu:///system" + +TEMPLATE_PATH = "./templates/domain.xml.jinja" + +VMIMAGE = { + 'provider': 'fedora', + 'version': '33', + 'checksum': '67daa956d8c82ef799f8b0a191c1753c9bda3bca' + } + +CACHE_DIR = '/tmp/lavocado-cache' diff --git a/tests/lavocado/lavocado/exceptions.py b/tests/lavocado/lavocado/exceptions.py new file mode 100644 index 0000000000..d89cbb3eef --- /dev/null +++ b/tests/lavocado/lavocado/exceptions.py @@ -0,0 +1,20 @@ +# Copyright (C) 2021 Red Hat, Inc. +# Author: Beraldo Leal <bleal@redhat.com> +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + + +class TestSetupException(Exception): + pass diff --git a/tests/lavocado/lavocado/helpers/__init__.py b/tests/lavocado/lavocado/helpers/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/lavocado/lavocado/helpers/domains.py b/tests/lavocado/lavocado/helpers/domains.py new file mode 100644 index 0000000000..cddee1b4b7 --- /dev/null +++ b/tests/lavocado/lavocado/helpers/domains.py @@ -0,0 +1,75 @@ +# Copyright (C) 2021 Red Hat, Inc. +# Author: Beraldo Leal <bleal@redhat.com> +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + + +import os + +from avocado.utils.genio import read_file +from jinja2 import Template + +from lavocado import defaults +from lavocado.exceptions import TestSetupException + + +class Domain: + @classmethod + def from_xml_path(cls, conn, xml_path): + """Create a new domain from a XML file. + + :param conn: a connection object to the Hypervisor. + :type conn: libvirt.virConnect + :param xml_path: XML file path. + :type xml_path: str + :returns: : the created domain object + :rtype: libvirt.virDomain + """ + xml_content = read_file(xml_path) + return conn.createXML(xml_content) + + @classmethod + def from_xml_template(cls, conn, suffix, arguments=None): + """Create a new domain from the default XML template. + + This will use the `defaults.TEMPLATE_PATH` file, parsing some arguments + defined there. + + :param conn: a connection object to the Hypervisor. + :type conn: libvirt.virConnect + :param suffix: A suffix string to be added to the domain domain. + :type suffix: str + :param arguments: a key/value dict to be used during + template parse. currently supported keys are: name, + memory, vcpus, arch, machine and image. Visit the + template file for details. + :rtype arguments: dict + :returns: : the created domain object + :rtype: libvirt.virDomain + """ + template_path = defaults.TEMPLATE_PATH + arguments = arguments or {} + + if not os.path.isfile(template_path): + error = f"Template {template_path} not found." + raise TestSetupException(error) + + # Adding a suffix to the name + name = arguments.get('name', 'lavocado-test') + arguments['name'] = f"{name}-{suffix}" + + template = Template(read_file(template_path)) + xml_content = template.render(**arguments) + return conn.createXML(xml_content) diff --git a/tests/lavocado/lavocado/test.py b/tests/lavocado/lavocado/test.py new file mode 100644 index 0000000000..b77ecaed58 --- /dev/null +++ b/tests/lavocado/lavocado/test.py @@ -0,0 +1,144 @@ +# Copyright (C) 2021 Red Hat, Inc. +# Author: Beraldo Leal <bleal@redhat.com> +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +"""Basic test helper module to avoid code redundancy.""" + +import os +import libvirt + +from avocado import Test +from avocado.utils import vmimage + +from lavocado import defaults +from lavocado.exceptions import TestSetupException +from lavocado.helpers.domains import Domain + + +class LibvirtTest(Test): + """Main class helper for tests. + + Any test that inherits from this class, will have some methods and + properties to assist on their jobs. + """ + def setUp(self): + """Setup to be executed before each test. + + Currently, this method is creating just a basic hypervisor connection. + Please, extend this method when writing your tests for your own needs. + + Any error that happens here *will not* flag the test as "FAIL", instead + tests will be flagged as "ERROR", meaning that some bootstrap error has + happened. + """ + self.defaults = defaults + self.conn = self.connect() + + def connect(self): + """Try to open a new connection with the hypervisor. + + This method uses the value defined at `defaults.LIBVIRT_URI` as URI. + + :returns: a libvirt connection. + :rtype: libvirt.virConnect + """ + try: + return libvirt.open(self.defaults.LIBVIRT_URI) + except libvirt.libvirtError: + msg = ("Failed to open connection with the hypervisor using " + + self.defaults.LIBVIRT_URI) + self.cancel(msg) + + def create_domain(self, arguments=None): + """Creates a libvirt domain based on the default template. + + This will receive some arguments that will be rendered on the + template. For more information about the arguments, see + templates/domain.xml.jinja. For now, at least the 'image' argument must + be informed, with the path for the image to boot. + + If you are using this method from a test method (different from + setUp()), AND you would like to count its call as a "setup/bootstrap" + stage, consider using the following Avocado decorator: + + from avocado.core.decorators import cancel_on + + @cancel_on(TestSetupException) + def test_foo(self): + ... + + In that way, your test will not FAIL, instead it will be cancelled in + case of any problem during this bootstrap. + + :param dict arguments: A key,value dictionary with the arguments + to be replaced on the template. If + any missing argument, template will be + rendered with default values. + """ + try: + return Domain.from_xml_template(self.conn, self.id(), arguments) + # This will catch any avocado exception plus any os error + except Exception as ex: + msg = f"Failed to create domain: {ex}" + raise TestSetupException(msg) from ex + + def get_generic_image(self): + """Ask Avocado to fetch an VM image snapshot. + + Avocado will handle if image is already downloaded into the + cache dir and also will make sure the checksum is matching. + + This will return an Image object pointing to a snapshot file. So + multiple calls of this method will never return the same object. + + If you are using this method from a test method (different from + setUp()), AND you would like to count its call as a "setup/bootstrap" + stage, consider using the following Avocado decorator: + + from avocado.core.decorators import cancel_on + + @cancel_on(TestSetupException) + def test_foo(self): + ... + + In that way, your test will not FAIL, instead it will be cancelled in + case of any problem during this bootstrap. + """ + image = self.defaults.VMIMAGE + try: + return vmimage.get(name=image.get('provider'), + version=image.get('version'), + cache_dir=self.defaults.CACHE_DIR, + checksum=image.get('checksum')) + # This will catch any error, including avocado exceptions + OS errors + except Exception as ex: + msg = f"Failed to get a generic image: {ex}" + raise TestSetupException(msg) from ex + + def tearDown(self): + """Shutdown after each test. + + This will destroy all previously created domains by this test, and + remove any image snapshot if created. + """ + for domain in self.conn.listAllDomains(): + if domain.name().endswith(self.id()): + domain.destroy() + + if hasattr(self, 'image') and isinstance(self.image, vmimage.Image): + if os.path.exists(self.image.path): + os.remove(self.image.path) + self.conn.close() diff --git a/tests/lavocado/requirements.txt b/tests/lavocado/requirements.txt new file mode 100644 index 0000000000..6927528323 --- /dev/null +++ b/tests/lavocado/requirements.txt @@ -0,0 +1,3 @@ +git+https://github.com/avocado-framework/avocado@8c87bfe5e8a1895d77226064433453f... +libvirt-python +Jinja2 diff --git a/tests/lavocado/templates/domain.xml.jinja b/tests/lavocado/templates/domain.xml.jinja new file mode 100644 index 0000000000..a7bd57e5b0 --- /dev/null +++ b/tests/lavocado/templates/domain.xml.jinja @@ -0,0 +1,20 @@ +<domain type='kvm'> + <name>{{name|default("lavocado-test")}}</name> + <memory unit='KiB'>{{memory|default(4194304)}}</memory> + <vcpu placement='static'>{{vcpus|default(4)}}</vcpu> + <os> + <type arch='{{arch|default("x86_64")}}' machine='{{machine|default("q35")}}'>hvm</type> + <boot dev='hd'/> + </os> + <devices> + <emulator>/usr/bin/qemu-system-x86_64</emulator> + <disk type='file' device='disk'> + <driver name='qemu' type='qcow2'/> + <source file='{{image}}'/> + <target dev='vda' bus='virtio'/> + </disk> + <console type='pty'> + <target type='serial' port='0'/> + </console> + </devices> +</domain> -- 2.26.3
This basic test case validates the core lifecycle and operations on transient domains and came from libvirt-tck as an exercise in porting tests. Signed-off-by: Beraldo Leal <bleal@redhat.com> --- tests/lavocado/tests/domain/transient.py | 102 +++++++++++++++++++++++ 1 file changed, 102 insertions(+) create mode 100644 tests/lavocado/tests/domain/transient.py diff --git a/tests/lavocado/tests/domain/transient.py b/tests/lavocado/tests/domain/transient.py new file mode 100644 index 0000000000..f96906c41b --- /dev/null +++ b/tests/lavocado/tests/domain/transient.py @@ -0,0 +1,102 @@ +# Copyright (C) 2021 Red Hat, Inc. +# Author: Beraldo Leal <bleal@redhat.com> +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# <http://www.gnu.org/licenses/>. + +import libvirt + +from lavocado.test import LibvirtTest + + +class TransientDomain(LibvirtTest): + """Transient domain basic operations. + + The test case validates the core lifecycle and operations on transient + domains. + """ + + def setUp(self): + super().setUp() + self.image = self.get_generic_image() + self.domain = self.create_domain(arguments={'image': self.image.path}) + + def test_lifecycle(self): + """Creating and destroying a new transient domain. + + A transient domain has no configuration file so, once destroyed, all + trace of the domain should disappear. + """ + name = self.domain.name() + state, _ = self.domain.state() + self.assertTrue(state, libvirt.VIR_DOMAIN_RUNNING) + + try: + self.domain.destroy() + except Exception as ex: + self.fail(f"destroy() raised an exception: {ex}") + + with self.assertRaises(libvirt.libvirtError) as context: + self.conn.lookupByName(name) + + expected = f"no domain with matching name '{self.domain.name()}'" + self.assertTrue(expected in str(context.exception)) + + def test_autostart(self): + """Makes sure autostart is disabled and not allowed on a transient domain. + + The test case validates that the autostart command returns a suitable + error message when used on a transient domain. + """ + self.assertFalse(self.domain.autostart()) + + with self.assertRaises(libvirt.libvirtError) as context: + self.domain.setAutostart(True) + + expected = "cannot set autostart for transient domain" + self.assertTrue(expected in str(context.exception)) + + def test_convert_transient_to_persistent(self): + """Converting transient to persistent should work fine. + + This test case validates that a transient domain can be converted to a + persistent one. This is achieved by defining a configuration file while + the transient domain is running. + """ + name = self.domain.name() + + # Make sure is running and not persistent + state, _ = self.domain.state() + self.assertTrue(state, libvirt.VIR_DOMAIN_RUNNING) + self.assertFalse(self.domain.isPersistent()) + + # Defining config for transient guest + new_domain = self.conn.defineXML(self.domain.XMLDesc()) + self.assertTrue(new_domain.isPersistent()) + + # Destroying active domain + self.domain.destroy() + + # Checking that an inactive domain config still exists + found = self.conn.lookupByName(name) + self.assertEqual(found.name(), name) + + # Removing inactive domain config + new_domain.undefine() + + # Checking that inactive domain has really gone + with self.assertRaises(libvirt.libvirtError) as context: + self.conn.lookupByName(name) + expected = f"no domain with matching name '{name}'" + self.assertTrue(expected in str(context.exception)) -- 2.26.3
This will prevent unnecessary Python object and temporary files to be tracked by git. Signed-off-by: Beraldo Leal <bleal@redhat.com> --- tests/lavocado/.gitignore | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 tests/lavocado/.gitignore diff --git a/tests/lavocado/.gitignore b/tests/lavocado/.gitignore new file mode 100644 index 0000000000..59c71f50f9 --- /dev/null +++ b/tests/lavocado/.gitignore @@ -0,0 +1,3 @@ +*.pyc +__pycache__ +*~ -- 2.26.3
Signed-off-by: Beraldo Leal <bleal@redhat.com> --- tests/lavocado/Makefile | 2 + tests/lavocado/README.md | 124 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 126 insertions(+) create mode 100644 tests/lavocado/Makefile create mode 100644 tests/lavocado/README.md diff --git a/tests/lavocado/Makefile b/tests/lavocado/Makefile new file mode 100644 index 0000000000..33df604105 --- /dev/null +++ b/tests/lavocado/Makefile @@ -0,0 +1,2 @@ +libvirt-tests: + avocado run --test-runner='nrunner' tests/ diff --git a/tests/lavocado/README.md b/tests/lavocado/README.md new file mode 100644 index 0000000000..b22015fe46 --- /dev/null +++ b/tests/lavocado/README.md @@ -0,0 +1,124 @@ +# lavocado - libvirt test framework based on Avocado + +The lavocado aims to be an alternative test framework for the libvirt project +using as muscles only Avocado and python-libvirt. This can be used to write +unit, functional and integration tests and it is inspired on the libvirt-tck +framework, but instead of Perl it written in Python. + +The idea is to provide to the test writers helper classes to avoid boilerplate +code, improve code readability and maintenance. + +## Disclaimer + +**For now, this framework assumes that you are going to run the tests in a fresh +clean environment, i.e. a VM. If you decide to use your local system, beware +that execution of the tests may affect your system.** + +One of the future goals of this framework is to utilize nested virtualization +technologies and hence make sure an L1 guest is provisioned automatically for +the tests to be executed in this environment and not tamper with your main +system. + +## Requirements + +The libvirt interactions will all be done via the libvirt Python bindings +(libvirt-python), we also rely on Avocado, not only to execute the tests but +also to generate multiple output format artifacts (that can be uploaded to CI +environments) and also the image fetch, so we don't need to handle with local +cache, checksum and any other test's requirement task. + +In order to install the requirements, please execute the following command: + +```bash + $ pip3 install -r requirements.txt +``` + +If you wish you can run this command under a virtual environment. + +## Features + +Bellow, we list some features that can be explored from this proposal: + + * Parallel execution: If your tests are independent of each other, you can run + those in parallel, when getting images, Avocado will create a snapshot to + avoid conflicts. Also domains created by tests have unique names to avoid + collisions. + + * Creating domains easily: You can use the `.create_domain()` method to create + a generic domain which is based on a Jinja2 template. If you want to use an + already existing XML domain description, no problem, use + `Domain.from_xml_path()` instead. + + * Multiple output formats: TAP, HTML, JSON, xUnit (Gitlab CI ready). + + * Test tags: Ability to mark tests with tags, for filtering during execution + and/or results. + + * Different isolation models: By default the avocado runner will spawn each + test on local processes (subprocess), but we can configure for specific use + cases, running the tests on different isolation models (containers for + instance). + + * Requirements resolver: Avocado has implemented the 'requirements resolver' + feature that makes it easy to define test requirements. Currently, The + RequirementsResolver can handle `assets` (any local or remote files) and + `packages` but new requirements are on the roadmap, including `ansible + playbooks`. For now, it is still only an experimental feature though. + + * Test result artifacts for future analysis: you can take a look at `avocado + jobs` command or inspect your `~/avocado/test-results/` folder for more + detailed results. + + * Multiple test execution with variants: the variants subsystem is what allows + the creation of multiple variations of parameters, and the execution of + tests with those parameter variations. + + * Clear separation between tests and bootstrap stages: If something fails + during the setUp() metod execution, your test will be not marked as FAIL, + instead it will be flagged as ERROR. You can also use some decorators + (@cancel_on, @skipUnless, ...) around your test to avoid false positives. + + * Ready-to-use utility libraries: `avocado.utils` are rich, and cover + common operating system level needs, such as service managment, + hardware introspection, storage management, networking, etc. + + * Avocado Job API: a programmable interface for defining test jobs, which + allows one to create custom test jobs that are suitable to developer's + systems and CI enviroments. + + * Output artifacts for debug: Avocado stores all job results in + `~/avocado/job-results/` (you can also see details with `avocado jobs` + command). If any test fails we can debug the files there. Besides that, using + `sysinfo` plugin, we could collect additional system information (such as + libvirt/qemu logs). + +## Running + +After installing the requirements, you can run the tests with the following +commands: + +```bash + $ export PYTHONPATH=.:$PYTHONPATH + $ avocado run --test-runner='nrunner' ./tests/domain/*.py +``` + +Please note that the Next Runner (nrunner) will be the default runner soon in +Avocado, and so the `--test-runner='nrunner'` option will no longer be needed. + +Or, if you prefer, you can also execute the tests with `make`: + +```bash + $ make libvirt-tests +``` + +## Writing Tests + +You can write your tests here the same way you write for the [Avocado +Framework](https://avocado-framework.readthedocs.io/en/latest/). +Avocado supports "simple tests" (just executables) and "instrumented tests" +(Python tests). + +See the `tests/` folder for some references and ideas. In addition, feel free +to read the [Avocado Test Writer’s +Guide](https://avocado-framework.readthedocs.io/en/latest/guides/writer/) to +play with some advanced features of the framework. -- 2.26.3
On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
lavocado aims to be an alternative test framework for the libvirt project using Python, python-libvirt and Avocado. This can be used to write unit, functional and integration tests and it is inspired by the libvirt-tck framework.
This series introduces the basic framework along with some basic test examples that I got from libvirt-tck. I would appreciate your comments on this RFC, to see if this fits this project's needs. Also, names and locations are just a proposal and can be changed.
Some high level thoughts - More extensive functional integration testing coverage is good - We need to actually run the functional tests regularly reporting via GitLab pipelines in some way - Using Python is way more contributor friendly than Perl - This does not need to live in libvirt.git as we don't follow a monolithic repo approach in libvirt, and it already depends on functionality provided by other repos. When it comes to testing, I feel like there are several distinct pieces to think about - The execution & reporting harness - Supporting infrastructure to aid writing tests - The set of tests themselves If I look at the TCK - The harness is essentially the standard Perl harness with a thin CLI wrapper, thus essentially works with any test emitting TAP format - The support infra is all custom APIs using libvirt-perl - The tests are mostly written in Perl, but some are written in shell (yuk). They all output TAP format. One key thing here is that the test harness is fairly loosely coupled to the support infra & tests. The TAP data format bridged the two, letting us write tests in essentially any language. Of course writing tests in non-Perl was/is tedious today, since there's no support infra for that which exists today. The TAP data format bridge also means we can easily throw away the current TCK harness and replace it with anything else that can consume tests emitting TAP data. If I look at Avocado, I think (correct me if i'm wrong) 1. The harness is essentially the standard Python harness with a thin CLI wrapper. Thus needs all tests to implement the Python test APIs 2. The support infra is all custom APIs using libvirt-python 3. The tests are to be written entirely in Python, to integrate with the python test harness IIUC, 90% of this patch series is essentially about (2), since (1) is fairly trivial and for (3) there's just simple demo tests. Libvirt has consumers writing applications in a variety of languages, and periodically reporting bugs. My general wish for a test harness would be for something that can invoke and consume arbitrary test cases. Essentially if someone can provide us a piece of code that demonstrates a problem in libvirt, I would like to be able to use that directly as a functional test, regardless of language it was written in. In theory the libvirt TCK allowed for that, and indeed the few tests written in shell essentially came to exist because someone at IBM had written some shell code to test firewalls and contributed that as a functional test. They just hacked their script to emit TAP data.
For now, this framework assumes that you are going to run the tests in a fresh clean environment, i.e. a VM. If you decide to use your local system, beware that execution of the tests may affect your system.
It is always good to be wary of functional test frameworks trashing your system, but at the same time I think makes things more flexible if they are able to be reasonably safely run on /any/ host. For the TCK we tried to be quite conservative in this regard, because it was actually an explicit goal that you can run it on any Fedora host to validate it correctly functioning for KVM. To achieve that we tried to use some standard naming conventions for any resources that tests created, and if we saw pre-existing resources named differently we didn't touch them. ie all VMs were named 'tck-XXX', and were free to be deleted, but other named VMs were ignored. For anything special, such as testing PCI device assignment, the test configuration file had to explicitly contain details of host devices that were safe to use. This was also done for host block devices or NICs, etc. Thus a default invokation only ran a subset of tests which were safe. The more dangerous tests required you to modify the config file to grant access. I think it would be good if the Avocado supporting test APIs had a similar conceptual approach, especially wrt to a config file granting access to host resources such as NICs/Block devs/PCI devs, where you need prior explicit admin permission to avoid danger.
One of the future goals of this framework is to utilize nested virtualization technologies and hence make sure an L1 guest is provisioned automatically for the tests to be executed in this environment and not tamper with your main system.
I think the test framework should not concern itself with this kind of thing directly. We already have libvirt-ci, which has tools for provisioning VMs with pre-defined package sets. The test harness should just expect that this has been done, and that it is already running in such an environment if the user wanted it to be. In the TCK config file we provided setting for a URI to connect to other hosts, to enable multi-hosts tests like live migration to be done.
I'm adding more information with some details inside the README file.
Overall, I'm more enthusiastic about writing tests in Python than Perl, for the long term, but would also potentially like to write tests in Go too. I'm wondering if we can't bridge the divide between what we have already in libvirt-tck, and what you're bringing to the table with avocado here. While we've not done much development with the TCK recently, there are some very valuable tests there, especially related to firewall support and I don't fancy rewriting them. Thus my suggestion is that we: - Put this avocado code into the libvirt-tck repository, with focus on the supporting infra for making it easy to write Python tests - Declare that all tests need a way to emit TAP format, no matter what language they're written in. This could either be the test directly emitting TAP, or it could be via use of a plugin. For example 'tappy' can make existing Python tests emit TAP, with no modifications to the tests themselves. https://tappy.readthedocs.io/en/latest/consumers.html IOW, you can still invoke the python tests using the standard Python test runner, and still invoke the perl tests using the stnadard Perl test runner if desired. - Switch the TCK configuration file to use JSON/YAML instead of its current custom format. This is to enable the Python tests to share a config file with the Perl and Shell tests. Thus they can all have the same way to figure out what block devs, NICs, PCI devs, etc they are permitted to use, and what dirs it is safe to create VM images in, etc. - Replace the current Perl based execution harness with something using Python, so that it is less of a pain to install in terms of dependancies. Looks like we could just do something using the 'tappy' Python APIs to consume the TAP format from all the tests and generate a junit report that GitLab can consume. - Rename the RPMs so they match the project name "libvirt-tck", instead of "perl-Sys-Virt-TCK" to emphasis that this is not a Perl project, it is a testing project, of which some parts were historically Perl, but new parts will be mostly Python (and possibly Go/etc in future if desired). Potentially have RPMs split: * libvirt-tck - just the execution harness, currently the Perl one, but to be replaced with a Python one. * libvirt-tck-tests-python - supporting APIs and tests written in Python - essentially all of this series, and future tests written * libvirt-tck-tests-perl - supporting APIs and tests written in Perl - essentially most of existing TCK stuff * libvirt-tck-tests-shell - supporting APIs and tests written in shell - mostly the network/firewall related TCK pieces Given the existance of the 'tappy' tool, it feels like the first two points ought to be reasonably doable without any significant changes to what you've already written. Just need to import it as is, and then using tappy as a shim to let it invoked from libvirt-tck. Can still have the existing way to invoke it directly too as an alternative. The common JSON/YAML config file, and adopting some standard naming conventions and resource usage conventions are probably where the real work would lie. That's likely not too bad at this stage though, since you've not actually written a large number of test cases.
Beraldo Leal (4): tests: introduce lavocado: initial code structure tests.lavocado: adding basic transient domain tests tests.lavocado: adding a .gitignore tests.lavocado: adding a README and Makefile for convenience
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
If I look at Avocado, I think (correct me if i'm wrong)
1. The harness is essentially the standard Python harness with a thin CLI wrapper. Thus needs all tests to implement the Python test APIs 2. The support infra is all custom APIs using libvirt-python 3. The tests are to be written entirely in Python, to integrate with the python test harness
The harness proposed here would be the Avocado itself. That doesn't just run Python tests, it can run any executable. And also has support for multiple test results, including TAP. However, for developers trying to write Python tests will receive some support infrastructure classes that will help them.
IIUC, 90% of this patch series is essentially about (2), since (1) is fairly trivial and for (3) there's just simple demo tests.
Yes, and (1) is trivial because I’m delegating this to Avocado.
Libvirt has consumers writing applications in a variety of languages, and periodically reporting bugs. My general wish for a test harness would be for something that can invoke and consume arbitrary test cases. Essentially if someone can provide us a piece of code that demonstrates a problem in libvirt, I would like to be able to use that directly as a functional test, regardless of language it was written in.
In theory the libvirt TCK allowed for that, and indeed the few tests written in shell essentially came to exist because someone at IBM had written some shell code to test firewalls and contributed that as a functional test. They just hacked their script to emit TAP data.
For sure, I completely agree and understand. We take that into account as well, but maybe it wasn't so clear in the RFC. With Avocado you can do that, and there is no need for hacking the scripts to emit TAP data. Avocado has a --tap option that will output results in TAP format: $ avocado run --tap - /bin/true /bin/false 1..2 ok 1 /bin/true not ok 2 /bin/false
For now, this framework assumes that you are going to run the tests in a fresh clean environment, i.e. a VM. If you decide to use your local system, beware that execution of the tests may affect your system.
It is always good to be wary of functional test frameworks trashing your system, but at the same time I think makes things more flexible if they are able to be reasonably safely run on /any/ host.
For the TCK we tried to be quite conservative in this regard, because it was actually an explicit goal that you can run it on any Fedora host to validate it correctly functioning for KVM. To achieve that we tried to use some standard naming conventions for any resources that tests created, and if we saw pre-existing resources named differently we didn't touch them. ie all VMs were named 'tck-XXX', and were free to be deleted, but other named VMs were ignored.
For anything special, such as testing PCI device assignment, the test configuration file had to explicitly contain details of host devices that were safe to use. This was also done for host block devices or NICs, etc. Thus a default invokation only ran a subset of tests which were safe. The more dangerous tests required you to modify the config file to grant access.
I think it would be good if the Avocado supporting test APIs had a similar conceptual approach, especially wrt to a config file granting access to host resources such as NICs/Block devs/PCI devs, where you need prior explicit admin permission to avoid danger.
Indeed. It is a good practice to isolate those things. Again, it was my fault that wasn't clear on the RFC, and yes we have that in mind. As mentioned lavocado it was inspired by libvirt-tck and I'm already using the same approach with the prefix: when creating domains I'm prefixing the `test.id` to the domain name (and this can be customizable). On top of that, Avocado has the spawners concept, where we support right now subprocess and podman. But a new spawner could be developed to create a more isolated env. We also have the Job API feature, where, using python code, you can create custom job suites based on your logic.
One of the future goals of this framework is to utilize nested virtualization technologies and hence make sure an L1 guest is provisioned automatically for the tests to be executed in this environment and not tamper with your main system.
I think the test framework should not concern itself with this kind of thing directly. We already have libvirt-ci, which has tools for provisioning VMs with pre-defined package sets. The test harness should just expect that this has been done, and that it is already running in such an environment if the user wanted it to be.
Sure, again I wasn't clear. We intend to delegate this to tools like lcitool.
In the TCK config file we provided setting for a URI to connect to other hosts, to enable multi-hosts tests like live migration to be done.
I notice that, and I did the same here on LIBVIRT_URI config option. By default it is pointing to "qemu:///system". But for sure, could be customized. It is on my plans to extend this to support > 1 URI to support live migration.
I'm adding more information with some details inside the README file.
Overall, I'm more enthusiastic about writing tests in Python than Perl, for the long term, but would also potentially like to write tests in Go too.
I'm wondering if we can't bridge the divide between what we have already in libvirt-tck, and what you're bringing to the table with avocado here. While we've not done much development with the TCK recently, there are some very valuable tests there, especially related to firewall support and I don't fancy rewriting them.
Thus my suggestion is that we:
- Put this avocado code into the libvirt-tck repository, with focus on the supporting infra for making it easy to write Python tests
- Declare that all tests need a way to emit TAP format, no matter what language they're written in. This could either be the test directly emitting TAP, or it could be via use of a plugin. For example 'tappy' can make existing Python tests emit TAP, with no modifications to the tests themselves.
https://tappy.readthedocs.io/en/latest/consumers.html
IOW, you can still invoke the python tests using the standard Python test runner, and still invoke the perl tests using the stnadard Perl test runner if desired.
This is supported already: $ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py 1..3 ok 1 tests/domain/transient.py:TransientDomain.test_autostart ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent Thank you for your comments and review here. -- Beraldo
On Thu, Jul 01, 2021 at 06:09:47PM -0300, Beraldo Leal wrote:
On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
If I look at Avocado, I think (correct me if i'm wrong)
1. The harness is essentially the standard Python harness with a thin CLI wrapper. Thus needs all tests to implement the Python test APIs 2. The support infra is all custom APIs using libvirt-python 3. The tests are to be written entirely in Python, to integrate with the python test harness
This is supported already:
$ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py 1..3 ok 1 tests/domain/transient.py:TransientDomain.test_autostart ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent
BTW, where are you getting 'avocado' from ? AFAICT, it still is not made available in Fedora repositories - I found a copr repo only. It will need to graduate into official Fedora repositories before we can depend on it really. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Fri, Jul 02, 2021 at 12:43:57PM +0100, Daniel P. Berrangé wrote:
On Thu, Jul 01, 2021 at 06:09:47PM -0300, Beraldo Leal wrote:
On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
If I look at Avocado, I think (correct me if i'm wrong)
1. The harness is essentially the standard Python harness with a thin CLI wrapper. Thus needs all tests to implement the Python test APIs 2. The support infra is all custom APIs using libvirt-python 3. The tests are to be written entirely in Python, to integrate with the python test harness
This is supported already:
$ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py 1..3 ok 1 tests/domain/transient.py:TransientDomain.test_autostart ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent
BTW, where are you getting 'avocado' from ? AFAICT, it still is not made available in Fedora repositories - I found a copr repo only. It will need to graduate into official Fedora repositories before we can depend on it really.
ACK. Right now, for this RFC, I'm getting from Github on requirements.txt. But we could get it from Pypi or any other repo. Regards, -- Beraldo
On Fri, Jul 2, 2021 at 7:48 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Thu, Jul 01, 2021 at 06:09:47PM -0300, Beraldo Leal wrote:
On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
If I look at Avocado, I think (correct me if i'm wrong)
1. The harness is essentially the standard Python harness with a thin CLI wrapper. Thus needs all tests to implement the Python test APIs 2. The support infra is all custom APIs using libvirt-python 3. The tests are to be written entirely in Python, to integrate with the python test harness
This is supported already:
$ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py 1..3 ok 1 tests/domain/transient.py:TransientDomain.test_autostart ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent
BTW, where are you getting 'avocado' from ? AFAICT, it still is not made available in Fedora repositories - I found a copr repo only. It will need to graduate into official Fedora repositories before we can depend on it really.
Hi Daniel, It has been in Fedora for a long time. But, during the modularization effort, it became part of the "avocado" module. Now, we're bringing back the non-modular packages. For now: $ dnf module enable avocado:latest $ dnf module install avocado:latest Should work. Thanks, - Cleber.
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Jul 01, 2021 at 06:09:47PM -0300, Beraldo Leal wrote:
On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
I'm adding more information with some details inside the README file.
Overall, I'm more enthusiastic about writing tests in Python than Perl, for the long term, but would also potentially like to write tests in Go too.
I'm wondering if we can't bridge the divide between what we have already in libvirt-tck, and what you're bringing to the table with avocado here. While we've not done much development with the TCK recently, there are some very valuable tests there, especially related to firewall support and I don't fancy rewriting them.
Thus my suggestion is that we:
- Put this avocado code into the libvirt-tck repository, with focus on the supporting infra for making it easy to write Python tests
- Declare that all tests need a way to emit TAP format, no matter what language they're written in. This could either be the test directly emitting TAP, or it could be via use of a plugin. For example 'tappy' can make existing Python tests emit TAP, with no modifications to the tests themselves.
https://tappy.readthedocs.io/en/latest/consumers.html
IOW, you can still invoke the python tests using the standard Python test runner, and still invoke the perl tests using the stnadard Perl test runner if desired.
This is supported already:
$ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py 1..3 ok 1 tests/domain/transient.py:TransientDomain.test_autostart ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent
This is nice, showing fine grained TAP output lines for each individual test within the test program I tried using the hints file that Cleber pointed to make avocado *consume* TAP format for the Perl/Shell scripts: $ cd libvirt-tck $ cat .avocado.hint [kinds] tap = scripts/*/*.t [tap] uri = $testpath And I can indeed invoke the scripts: $ avocado run ./scripts/domain/05*.t JOB ID : b5d596d909dc8024d986957c909fc8fb6b31e2dd JOB LOG : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/job.log (1/2) ./scripts/domain/050-transient-lifecycle.t: PASS (0.70 s) (2/2) ./scripts/domain/051-transient-autostart.t: PASS (0.76 s) RESULTS : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB HTML : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/results.html JOB TIME : 1.90 s which is good. And also I can ask it to produce tap output too: $ avocado run --tap - ./scripts/domain/05*.t 1..2 ok 1 ./scripts/domain/050-transient-lifecycle.t ok 2 ./scripts/domain/051-transient-autostart.t But this output isn't entirely what I was after. This is just summarizing the results of each test program. I can't find a way to make it show the fine grained tap output for the individual tests, like it does for the python program eg I'd like to be able to see something similar to: $ ./scripts/domain/050-transient-lifecycle.t 1..2 # Creating a new transient domain ok 1 - created transient domain object # Destroying the transient domain # Checking that transient domain has gone away ok 2 - NO_DOMAIN error raised from missing domain $ ./scripts/domain/051-transient-autostart.t 1..4 # Creating a new transient domain ok 1 - created transient domain object ok 2 - autostart is disabled for transient VMs ok 3 - Set autostart not supported on transient VMs # Destroying the transient domain # Checking that transient domain has gone away ok 4 - NO_DOMAIN error raised from missing domain None the less this does seem like we're pretty close to being able to do something useful in integration Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, Jul 21, 2021 at 06:50:03PM +0100, Daniel P. Berrangé wrote:
On Thu, Jul 01, 2021 at 06:09:47PM -0300, Beraldo Leal wrote:
On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
I'm adding more information with some details inside the README file.
Overall, I'm more enthusiastic about writing tests in Python than Perl, for the long term, but would also potentially like to write tests in Go too.
I'm wondering if we can't bridge the divide between what we have already in libvirt-tck, and what you're bringing to the table with avocado here. While we've not done much development with the TCK recently, there are some very valuable tests there, especially related to firewall support and I don't fancy rewriting them.
Thus my suggestion is that we:
- Put this avocado code into the libvirt-tck repository, with focus on the supporting infra for making it easy to write Python tests
- Declare that all tests need a way to emit TAP format, no matter what language they're written in. This could either be the test directly emitting TAP, or it could be via use of a plugin. For example 'tappy' can make existing Python tests emit TAP, with no modifications to the tests themselves.
https://tappy.readthedocs.io/en/latest/consumers.html
IOW, you can still invoke the python tests using the standard Python test runner, and still invoke the perl tests using the stnadard Perl test runner if desired.
This is supported already:
$ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py 1..3 ok 1 tests/domain/transient.py:TransientDomain.test_autostart ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent
This is nice, showing fine grained TAP output lines for each individual test within the test program
I tried using the hints file that Cleber pointed to make avocado *consume* TAP format for the Perl/Shell scripts:
$ cd libvirt-tck $ cat .avocado.hint [kinds] tap = scripts/*/*.t
[tap] uri = $testpath
And I can indeed invoke the scripts:
$ avocado run ./scripts/domain/05*.t JOB ID : b5d596d909dc8024d986957c909fc8fb6b31e2dd JOB LOG : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/job.log (1/2) ./scripts/domain/050-transient-lifecycle.t: PASS (0.70 s) (2/2) ./scripts/domain/051-transient-autostart.t: PASS (0.76 s) RESULTS : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB HTML : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/results.html JOB TIME : 1.90 s
which is good.
And also I can ask it to produce tap output too:
$ avocado run --tap - ./scripts/domain/05*.t 1..2 ok 1 ./scripts/domain/050-transient-lifecycle.t ok 2 ./scripts/domain/051-transient-autostart.t
But this output isn't entirely what I was after. This is just summarizing the results of each test program.
I can't find a way to make it show the fine grained tap output for the individual tests, like it does for the python program
Actually, the first Python TAP output example is showing a coarse TAP result. I have combined three transient tests into one single file but splitting them into three different methods, so we could execute each one individually. i.e: tests/domain/transient.py:TransientDomain.test_autostart. So, I did some reorg when migrating to Python test. In order to archive the same with Perl, we could do the same there, because the way individual tests are written there, doesn't allow for individual execution. Yes, we could do some tricks, to parse and combine outputs and list as it was a more fine graned, but afaict, we could not individually execute those. This is part of Avocado test definition where in order to be called a test, we need to be able to execute those individually as well. -- Beraldo
On Wed, Jul 21, 2021 at 04:22:19PM -0300, Beraldo Leal wrote:
On Wed, Jul 21, 2021 at 06:50:03PM +0100, Daniel P. Berrangé wrote:
On Thu, Jul 01, 2021 at 06:09:47PM -0300, Beraldo Leal wrote:
On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
I'm adding more information with some details inside the README file.
Overall, I'm more enthusiastic about writing tests in Python than Perl, for the long term, but would also potentially like to write tests in Go too.
I'm wondering if we can't bridge the divide between what we have already in libvirt-tck, and what you're bringing to the table with avocado here. While we've not done much development with the TCK recently, there are some very valuable tests there, especially related to firewall support and I don't fancy rewriting them.
Thus my suggestion is that we:
- Put this avocado code into the libvirt-tck repository, with focus on the supporting infra for making it easy to write Python tests
- Declare that all tests need a way to emit TAP format, no matter what language they're written in. This could either be the test directly emitting TAP, or it could be via use of a plugin. For example 'tappy' can make existing Python tests emit TAP, with no modifications to the tests themselves.
https://tappy.readthedocs.io/en/latest/consumers.html
IOW, you can still invoke the python tests using the standard Python test runner, and still invoke the perl tests using the stnadard Perl test runner if desired.
This is supported already:
$ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py 1..3 ok 1 tests/domain/transient.py:TransientDomain.test_autostart ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent
This is nice, showing fine grained TAP output lines for each individual test within the test program
I tried using the hints file that Cleber pointed to make avocado *consume* TAP format for the Perl/Shell scripts:
$ cd libvirt-tck $ cat .avocado.hint [kinds] tap = scripts/*/*.t
[tap] uri = $testpath
And I can indeed invoke the scripts:
$ avocado run ./scripts/domain/05*.t JOB ID : b5d596d909dc8024d986957c909fc8fb6b31e2dd JOB LOG : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/job.log (1/2) ./scripts/domain/050-transient-lifecycle.t: PASS (0.70 s) (2/2) ./scripts/domain/051-transient-autostart.t: PASS (0.76 s) RESULTS : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB HTML : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/results.html JOB TIME : 1.90 s
which is good.
And also I can ask it to produce tap output too:
$ avocado run --tap - ./scripts/domain/05*.t 1..2 ok 1 ./scripts/domain/050-transient-lifecycle.t ok 2 ./scripts/domain/051-transient-autostart.t
But this output isn't entirely what I was after. This is just summarizing the results of each test program.
I can't find a way to make it show the fine grained tap output for the individual tests, like it does for the python program
Actually, the first Python TAP output example is showing a coarse TAP result. I have combined three transient tests into one single file but splitting them into three different methods, so we could execute each one individually. i.e: tests/domain/transient.py:TransientDomain.test_autostart.
So, I did some reorg when migrating to Python test.
In order to archive the same with Perl, we could do the same there, because the way individual tests are written there, doesn't allow for individual execution.
Yes, we could do some tricks, to parse and combine outputs and list as it was a more fine graned, but afaict, we could not individually execute those. This is part of Avocado test definition where in order to be called a test, we need to be able to execute those individually as well.
Ok, I'm not so fussed about whether avocado can ultimately preserve the fine grained TAP output. Mostly I'm looking to understand how you should debug failures when they go wrong, becuase the default output I see from avocado is very terse giving no indiciation of the the failure - merely that there has been a failure. In an interactive environment you can just re-rnu the individual failed test directly. In an automated CI environment you need the test harness to display enough info to debug the failure directly. Ideally it should dump the full stdout+stderr either to the console or a log file that can be publish as an artifact from CI. Meson tends todo the latter creating a log file will full stdout/err from all tests. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Jul 22, 2021 at 09:44:28AM +0100, Daniel P. Berrangé wrote:
On Wed, Jul 21, 2021 at 04:22:19PM -0300, Beraldo Leal wrote:
On Wed, Jul 21, 2021 at 06:50:03PM +0100, Daniel P. Berrangé wrote:
On Thu, Jul 01, 2021 at 06:09:47PM -0300, Beraldo Leal wrote:
On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
I'm adding more information with some details inside the README file.
Overall, I'm more enthusiastic about writing tests in Python than Perl, for the long term, but would also potentially like to write tests in Go too.
I'm wondering if we can't bridge the divide between what we have already in libvirt-tck, and what you're bringing to the table with avocado here. While we've not done much development with the TCK recently, there are some very valuable tests there, especially related to firewall support and I don't fancy rewriting them.
Thus my suggestion is that we:
- Put this avocado code into the libvirt-tck repository, with focus on the supporting infra for making it easy to write Python tests
- Declare that all tests need a way to emit TAP format, no matter what language they're written in. This could either be the test directly emitting TAP, or it could be via use of a plugin. For example 'tappy' can make existing Python tests emit TAP, with no modifications to the tests themselves.
https://tappy.readthedocs.io/en/latest/consumers.html
IOW, you can still invoke the python tests using the standard Python test runner, and still invoke the perl tests using the stnadard Perl test runner if desired.
This is supported already:
$ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py 1..3 ok 1 tests/domain/transient.py:TransientDomain.test_autostart ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent
This is nice, showing fine grained TAP output lines for each individual test within the test program
I tried using the hints file that Cleber pointed to make avocado *consume* TAP format for the Perl/Shell scripts:
$ cd libvirt-tck $ cat .avocado.hint [kinds] tap = scripts/*/*.t
[tap] uri = $testpath
And I can indeed invoke the scripts:
$ avocado run ./scripts/domain/05*.t JOB ID : b5d596d909dc8024d986957c909fc8fb6b31e2dd JOB LOG : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/job.log (1/2) ./scripts/domain/050-transient-lifecycle.t: PASS (0.70 s) (2/2) ./scripts/domain/051-transient-autostart.t: PASS (0.76 s) RESULTS : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB HTML : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/results.html JOB TIME : 1.90 s
which is good.
And also I can ask it to produce tap output too:
$ avocado run --tap - ./scripts/domain/05*.t 1..2 ok 1 ./scripts/domain/050-transient-lifecycle.t ok 2 ./scripts/domain/051-transient-autostart.t
But this output isn't entirely what I was after. This is just summarizing the results of each test program.
I can't find a way to make it show the fine grained tap output for the individual tests, like it does for the python program
Actually, the first Python TAP output example is showing a coarse TAP result. I have combined three transient tests into one single file but splitting them into three different methods, so we could execute each one individually. i.e: tests/domain/transient.py:TransientDomain.test_autostart.
So, I did some reorg when migrating to Python test.
In order to archive the same with Perl, we could do the same there, because the way individual tests are written there, doesn't allow for individual execution.
Yes, we could do some tricks, to parse and combine outputs and list as it was a more fine graned, but afaict, we could not individually execute those. This is part of Avocado test definition where in order to be called a test, we need to be able to execute those individually as well.
Ok, I'm not so fussed about whether avocado can ultimately preserve the fine grained TAP output. Mostly I'm looking to understand how you should debug failures when they go wrong, becuase the default output I see from avocado is very terse giving no indiciation of the the failure - merely that there has been a failure.
In an interactive environment you can just re-rnu the individual failed test directly. In an automated CI environment you need the test harness to display enough info to debug the failure directly. Ideally it should dump the full stdout+stderr either to the console or a log file that can be publish as an artifact from CI. Meson tends todo the latter creating a log file will full stdout/err from all tests.
Sure, I completely understand your concerns. In this case, the output here is "very terse" because you are running the "--tap" version. When using the "default/human" output, Avocado outputs some pointers to where you could get more details. We understand that debugging is a critical part of the process, so Avocado stores the job information inside "~/avocado/job-results/[job-id|latest]/" folder by default (regardless of the output format used). Inside this folder, you can find a "debug.log" file and a "test-results" sub-folder (with per-test stdout/stderr files). The entire folder could be uploaded as an artifact for any CI. It is also generated by default a results.xml (xUnit format) for better debugging on CI that supports this format (i.e: Gitlab). If you would like to see an example, QEMU project's test list[1] on Gitlab is based on this XML file. I understand that there is some room for improvements here and we are glad to address them. In meanwhile I was working on a proposal for the libvirt-tck based on the progress of this thread. Hopefully, this will help with the Avocado bootstrap phase. [1] - https://gitlab.com/qemu-project/qemu/-/pipelines/340613922/test_report [2] - https://gitlab.com/beraldoleal/libvirt-tck/-/tree/avocado-libvirt/ Regards, -- Beraldo
Daniel P. Berrangé writes:
On Thu, Jul 01, 2021 at 06:09:47PM -0300, Beraldo Leal wrote:
This is supported already:
$ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py 1..3 ok 1 tests/domain/transient.py:TransientDomain.test_autostart ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent
This is nice, showing fine grained TAP output lines for each individual test within the test program
I tried using the hints file that Cleber pointed to make avocado *consume* TAP format for the Perl/Shell scripts:
$ cd libvirt-tck $ cat .avocado.hint [kinds] tap = scripts/*/*.t
[tap] uri = $testpath
And I can indeed invoke the scripts:
$ avocado run ./scripts/domain/05*.t JOB ID : b5d596d909dc8024d986957c909fc8fb6b31e2dd JOB LOG : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/job.log (1/2) ./scripts/domain/050-transient-lifecycle.t: PASS (0.70 s) (2/2) ./scripts/domain/051-transient-autostart.t: PASS (0.76 s) RESULTS : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB HTML : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/results.html JOB TIME : 1.90 s
which is good.
And also I can ask it to produce tap output too:
$ avocado run --tap - ./scripts/domain/05*.t 1..2 ok 1 ./scripts/domain/050-transient-lifecycle.t ok 2 ./scripts/domain/051-transient-autostart.t
But this output isn't entirely what I was after. This is just summarizing the results of each test program.
From the user experience perspective, I completely understand your
Hi Daniel, point. I would side with this expectation and output any day if acting purely as a user. But, as a framework, Avocado came up with some well defined concepts, and one of them is what a test is. To be able to "replay only the failed tests of the latest job" or to "run a test completely isolated in a container or VM", there needs to be an specific and indistinguishable entry point for anything that is considered a test.
I can't find a way to make it show the fine grained tap output for the individual tests, like it does for the python program
eg I'd like to be able to see something similar to:
$ ./scripts/domain/050-transient-lifecycle.t 1..2 # Creating a new transient domain ok 1 - created transient domain object # Destroying the transient domain # Checking that transient domain has gone away ok 2 - NO_DOMAIN error raised from missing domain
$ ./scripts/domain/051-transient-autostart.t 1..4 # Creating a new transient domain ok 1 - created transient domain object ok 2 - autostart is disabled for transient VMs ok 3 - Set autostart not supported on transient VMs # Destroying the transient domain # Checking that transient domain has gone away ok 4 - NO_DOMAIN error raised from missing domain
Results for tests that produce TAP can be seen in the granularity you list here, given some of the conditions like the one explained before. For instance, the GLib plugin[1] knows that it's possible to run a single test case on tests that use the g_test_*() APIs, by providing a "-p /test/case/name". We could in theory reorganize the current Perl based tests, so that a test matches a function or some other type of code marker that can be used as an entry point. Given that the overall result will always be valid and the log files will always contain the more granular information, I have mixed feelings about the cost/benefit of doing that, but you are surely a much better judge for that.
None the less this does seem like we're pretty close to being able to do something useful in integration
Regards, Daniel
Thanks for the feedback! - Cleber. PS: The GLib plugin was deprecated because we observed that most users would do fine with the TAP plugin by itself, but it can be reinstated if there is a demand. PS2: If a test needs to produce more detailed information about its status, such as for a long running test with different stages, we recommend that advanced logging is used[2], instead of calling the different stages of a single test (from Avocado's PoV). [1] - https://avocado-framework.readthedocs.io/en/82.0/plugins/optional/glib.html [2] - https://avocado-framework.readthedocs.io/en/89.0/guides/writer/chapters/logg...
On Wed, Jul 21, 2021 at 06:01:34PM -0400, Cleber Rosa wrote:
Daniel P. Berrangé writes:
On Thu, Jul 01, 2021 at 06:09:47PM -0300, Beraldo Leal wrote:
This is supported already:
$ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py 1..3 ok 1 tests/domain/transient.py:TransientDomain.test_autostart ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent
This is nice, showing fine grained TAP output lines for each individual test within the test program
I tried using the hints file that Cleber pointed to make avocado *consume* TAP format for the Perl/Shell scripts:
$ cd libvirt-tck $ cat .avocado.hint [kinds] tap = scripts/*/*.t
[tap] uri = $testpath
And I can indeed invoke the scripts:
$ avocado run ./scripts/domain/05*.t JOB ID : b5d596d909dc8024d986957c909fc8fb6b31e2dd JOB LOG : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/job.log (1/2) ./scripts/domain/050-transient-lifecycle.t: PASS (0.70 s) (2/2) ./scripts/domain/051-transient-autostart.t: PASS (0.76 s) RESULTS : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB HTML : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/results.html JOB TIME : 1.90 s
which is good.
And also I can ask it to produce tap output too:
$ avocado run --tap - ./scripts/domain/05*.t 1..2 ok 1 ./scripts/domain/050-transient-lifecycle.t ok 2 ./scripts/domain/051-transient-autostart.t
But this output isn't entirely what I was after. This is just summarizing the results of each test program.
Hi Daniel,
From the user experience perspective, I completely understand your point. I would side with this expectation and output any day if acting purely as a user.
But, as a framework, Avocado came up with some well defined concepts, and one of them is what a test is. To be able to "replay only the failed tests of the latest job" or to "run a test completely isolated in a container or VM", there needs to be an specific and indistinguishable entry point for anything that is considered a test.
Ok, I see what you mean. Regardless of what its written in, it is common that a test will print arbitrary stuff to stdout or stderr if things go very badly wrong, possibly including stack traces, etc. What is the expected approach for debugging failed tests such that you can see the full stdout/stderr ? Are you supposed to just directly invoke the individual test program without using the avocado harness to see full output ?
I can't find a way to make it show the fine grained tap output for the individual tests, like it does for the python program
eg I'd like to be able to see something similar to:
$ ./scripts/domain/050-transient-lifecycle.t 1..2 # Creating a new transient domain ok 1 - created transient domain object # Destroying the transient domain # Checking that transient domain has gone away ok 2 - NO_DOMAIN error raised from missing domain
$ ./scripts/domain/051-transient-autostart.t 1..4 # Creating a new transient domain ok 1 - created transient domain object ok 2 - autostart is disabled for transient VMs ok 3 - Set autostart not supported on transient VMs # Destroying the transient domain # Checking that transient domain has gone away ok 4 - NO_DOMAIN error raised from missing domain
Results for tests that produce TAP can be seen in the granularity you list here, given some of the conditions like the one explained before. For instance, the GLib plugin[1] knows that it's possible to run a single test case on tests that use the g_test_*() APIs, by providing a "-p /test/case/name".
We could in theory reorganize the current Perl based tests, so that a test matches a function or some other type of code marker that can be used as an entry point. Given that the overall result will always be valid and the log files will always contain the more granular information, I have mixed feelings about the cost/benefit of doing that, but you are surely a much better judge for that.
Yeah, that concept is alien to the Perl tests and not worth the effort to try to change that - you'd be better off just rewriting from scratch at that point. The perl re-execution granularity is purely at an individual script level - the implication is that while in python you might have a single test_foo.py with many tests inside, in Perl you'd probably have many xxxx.t files.
None the less this does seem like we're pretty close to being able to do something useful in integration
PS: The GLib plugin was deprecated because we observed that most users would do fine with the TAP plugin by itself, but it can be reinstated if there is a demand.
GLib's test harness has two output formats - it can output its own custom data format, or it can output in TAP format. The granularity of its TAP format is pretty much the same as the way avocado reports TAP format granularity for python programs. Using TAP format, however, requires the test harness to pass an extra --tap arg to the individual test program. You can also still use the -p arg. If you're deprecating the GLib plugin, is avocado TAP harness have a way to be told to pass --tap, and a way to se -p to select individual tests ? I agre with no needing to support the GLib test output format - it is more typical for GLib apps to be using its TAP format instead these days.
PS2: If a test needs to produce more detailed information about its status, such as for a long running test with different stages, we recommend that advanced logging is used[2], instead of calling the different stages of a single test (from Avocado's PoV).
[1] - https://avocado-framework.readthedocs.io/en/82.0/plugins/optional/glib.html [2] - https://avocado-framework.readthedocs.io/en/89.0/guides/writer/chapters/logg...
That logging assume a python program ? Is there a way for non-python programs to get logging back into avocado ? Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Hi Daniel, On Thu, Jul 1, 2021 at 2:05 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
lavocado aims to be an alternative test framework for the libvirt project using Python, python-libvirt and Avocado. This can be used to write unit, functional and integration tests and it is inspired by the libvirt-tck framework.
This series introduces the basic framework along with some basic test examples that I got from libvirt-tck. I would appreciate your comments on this RFC, to see if this fits this project's needs. Also, names and locations are just a proposal and can be changed.
Some high level thoughts
- More extensive functional integration testing coverage is good
- We need to actually run the functional tests regularly reporting via GitLab pipelines in some way
- Using Python is way more contributor friendly than Perl
- This does not need to live in libvirt.git as we don't follow a monolithic repo approach in libvirt, and it already depends on functionality provided by other repos.
When it comes to testing, I feel like there are several distinct pieces to think about
- The execution & reporting harness - Supporting infrastructure to aid writing tests - The set of tests themselves
If I look at the TCK
- The harness is essentially the standard Perl harness with a thin CLI wrapper, thus essentially works with any test emitting TAP format - The support infra is all custom APIs using libvirt-perl - The tests are mostly written in Perl, but some are written in shell (yuk). They all output TAP format.
One key thing here is that the test harness is fairly loosely coupled to the support infra & tests.
The TAP data format bridged the two, letting us write tests in essentially any language. Of course writing tests in non-Perl was/is tedious today, since there's no support infra for that which exists today.
The TAP data format bridge also means we can easily throw away the current TCK harness and replace it with anything else that can consume tests emitting TAP data.
If I look at Avocado, I think (correct me if i'm wrong)
1. The harness is essentially the standard Python harness with a thin CLI wrapper. Thus needs all tests to implement the Python test APIs
Not really. Even though Avocado is mostly written in Python, there have been provisions for accommodating foreign types of tests (in different forms) since its inception. The most basic way, is, of course, simply treating a test as an executable. But this is far from the only way. For instance, these are other possibilities: a) if an executable generates TAP, it can consume the test's TAP output and determine the test results b) you can write support for completely new test types[1]. If you want your test to be "discoverable" when running `avocado list ...` you will have to write some Python code. But you can skip that part, and have an executable named "avocado-runner-$test-type" written in any language whatsoever, which will be used to execute whatever your tests are and feed the information to Avocado. BTW, we already have a plugin that can list go tests[2] and run them (via a Python wrapper that uses `go test`).
2. The support infra is all custom APIs using libvirt-python 3. The tests are to be written entirely in Python, to integrate with the python test harness
IIUC, 90% of this patch series is essentially about (2), since (1) is fairly trivial and for (3) there's just simple demo tests.
Libvirt has consumers writing applications in a variety of languages, and periodically reporting bugs. My general wish for a test harness would be for something that can invoke and consume arbitrary test cases. Essentially if someone can provide us a piece of code that demonstrates a problem in libvirt, I would like to be able to use that directly as a functional test, regardless of language it was written in.
In theory the libvirt TCK allowed for that, and indeed the few tests written in shell essentially came to exist because someone at IBM had written some shell code to test firewalls and contributed that as a functional test. They just hacked their script to emit TAP data.
I agree there's no reason to *not* continue to allow that. I'd even argue that Avocado can give libvirt even more options wrt integrating different types of tests.
For now, this framework assumes that you are going to run the tests in a fresh clean environment, i.e. a VM. If you decide to use your local system, beware that execution of the tests may affect your system.
It is always good to be wary of functional test frameworks trashing your system, but at the same time I think makes things more flexible if they are able to be reasonably safely run on /any/ host.
I'm on the same boat here. All the QEMU/Libvirt testing infrastructure that I was involved with, evolved from something that needed to run as privileged users and could easily trash your system, from something that is runnable as a regular user after a `pip install --user`. There are challenges, but most situations are handled with tests being canceled if they do not have what they need to run, and by always assuming a "good neighbor" policy (that is, you're not alone in the system). Having said that, I've seen how much care has been dedicated to libvirt-ci so that test execution environments would be predictable, reliable and easy to obtain. I guess the original text from Beraldo tried to make it clear that we envision an obvious/default integration with those environments created by libvirt-ci, while not restricting other environments or assuming every execution will be on those environments.
For the TCK we tried to be quite conservative in this regard, because it was actually an explicit goal that you can run it on any Fedora host to validate it correctly functioning for KVM. To achieve that we tried to use some standard naming conventions for any resources that tests created, and if we saw pre-existing resources named differently we didn't touch them. ie all VMs were named 'tck-XXX', and were free to be deleted, but other named VMs were ignored.
For anything special, such as testing PCI device assignment, the test configuration file had to explicitly contain details of host devices that were safe to use. This was also done for host block devices or NICs, etc. Thus a default invokation only ran a subset of tests which were safe. The more dangerous tests required you to modify the config file to grant access.
I think it would be good if the Avocado supporting test APIs had a similar conceptual approach, especially wrt to a config file granting access to host resources such as NICs/Block devs/PCI devs, where you need prior explicit admin permission to avoid danger.
Absolutely agreed.
One of the future goals of this framework is to utilize nested virtualization technologies and hence make sure an L1 guest is provisioned automatically for the tests to be executed in this environment and not tamper with your main system.
I think the test framework should not concern itself with this kind of thing directly. We already have libvirt-ci, which has tools for provisioning VMs with pre-defined package sets. The test harness should just expect that this has been done, and that it is already running in such an environment if the user wanted it to be.
Right. I know for a fact the original text was not about depending exclusively on those VM environments, but having an obvious default experience based on them.
In the TCK config file we provided setting for a URI to connect to other hosts, to enable multi-hosts tests like live migration to be done.
I'm adding more information with some details inside the README file.
Overall, I'm more enthusiastic about writing tests in Python than Perl, for the long term, but would also potentially like to write tests in Go too.
I'm wondering if we can't bridge the divide between what we have already in libvirt-tck, and what you're bringing to the table with avocado here. While we've not done much development with the TCK recently, there are some very valuable tests there, especially related to firewall support and I don't fancy rewriting them.
Thus my suggestion is that we:
- Put this avocado code into the libvirt-tck repository, with focus on the supporting infra for making it easy to write Python tests
One drawback would be the various supporting files for a perl+Python package. I mean, a Makefile.PL and a setup.py in the same repo can be a source of confusion. But, maybe I'm overreacting.
- Declare that all tests need a way to emit TAP format, no matter what language they're written in. This could either be the test directly emitting TAP, or it could be via use of a plugin. For example 'tappy' can make existing Python tests emit TAP, with no modifications to the tests themselves.
I honestly think Avocado can provide a better range of integration opportunities.
IOW, you can still invoke the python tests using the standard Python test runner, and still invoke the perl tests using the stnadard Perl test runner if desired.
- Switch the TCK configuration file to use JSON/YAML instead of its current custom format. This is to enable the Python tests to share a config file with the Perl and Shell tests. Thus they can all have the same way to figure out what block devs, NICs, PCI devs, etc they are permitted to use, and what dirs it is safe to create VM images in, etc.
Makes sense.
- Replace the current Perl based execution harness with something using Python, so that it is less of a pain to install in terms of dependancies. Looks like we could just do something using the 'tappy' Python APIs to consume the TAP format from all the tests and generate a junit report that GitLab can consume.
These are all things Avocado can do right away.
- Rename the RPMs so they match the project name "libvirt-tck", instead of "perl-Sys-Virt-TCK" to emphasis that this is not a Perl project, it is a testing project, of which some parts were historically Perl, but new parts will be mostly Python (and possibly Go/etc in future if desired). Potentially have RPMs split:
* libvirt-tck - just the execution harness, currently the Perl one, but to be replaced with a Python one.
* libvirt-tck-tests-python - supporting APIs and tests written in Python - essentially all of this series, and future tests written
* libvirt-tck-tests-perl - supporting APIs and tests written in Perl - essentially most of existing TCK stuff
* libvirt-tck-tests-shell - supporting APIs and tests written in shell - mostly the network/firewall related TCK pieces
Given the existance of the 'tappy' tool, it feels like the first two points ought to be reasonably doable without any significant changes to what you've already written. Just need to import it as is, and then using tappy as a shim to let it invoked from libvirt-tck. Can still have the existing way to invoke it directly too as an alternative.
The common JSON/YAML config file, and adopting some standard naming conventions and resource usage conventions are probably where the real work would lie. That's likely not too bad at this stage though, since you've not actually written a large number of test cases.
Beraldo Leal (4): tests: introduce lavocado: initial code structure tests.lavocado: adding basic transient domain tests tests.lavocado: adding a .gitignore tests.lavocado: adding a README and Makefile for convenience
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Cheers, - Cleber. [1] - https://avocado-framework.readthedocs.io/en/89.0/guides/contributor/chapters... [2] - https://avocado-framework.readthedocs.io/en/89.0/plugins/optional/golang.htm...
On Fri, Jul 02, 2021 at 07:23:57AM -0400, Cleber Rosa wrote:
On Thu, Jul 1, 2021 at 2:05 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
Overall, I'm more enthusiastic about writing tests in Python than Perl, for the long term, but would also potentially like to write tests in Go too.
I'm wondering if we can't bridge the divide between what we have already in libvirt-tck, and what you're bringing to the table with avocado here. While we've not done much development with the TCK recently, there are some very valuable tests there, especially related to firewall support and I don't fancy rewriting them.
Thus my suggestion is that we:
- Put this avocado code into the libvirt-tck repository, with focus on the supporting infra for making it easy to write Python tests
One drawback would be the various supporting files for a perl+Python package. I mean, a Makefile.PL and a setup.py in the same repo can be a source of confusion. But, maybe I'm overreacting.
I don't think that's an issue. We're not actually trying to ship Perl modules or Python modules, we're providing an application that happens to use some Perl and Python and $whatever code. As such, I would not use either Makefile.PL or setup.py for this task. Instead I think we should just use meson to deal with it all, keeping it all private to the application. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Fri, Jul 02, 2021 at 07:23:57AM -0400, Cleber Rosa wrote:
Hi Daniel,
On Thu, Jul 1, 2021 at 2:05 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
lavocado aims to be an alternative test framework for the libvirt project using Python, python-libvirt and Avocado. This can be used to write unit, functional and integration tests and it is inspired by the libvirt-tck framework.
This series introduces the basic framework along with some basic test examples that I got from libvirt-tck. I would appreciate your comments on this RFC, to see if this fits this project's needs. Also, names and locations are just a proposal and can be changed.
Some high level thoughts
- More extensive functional integration testing coverage is good
- We need to actually run the functional tests regularly reporting via GitLab pipelines in some way
- Using Python is way more contributor friendly than Perl
- This does not need to live in libvirt.git as we don't follow a monolithic repo approach in libvirt, and it already depends on functionality provided by other repos.
When it comes to testing, I feel like there are several distinct pieces to think about
- The execution & reporting harness - Supporting infrastructure to aid writing tests - The set of tests themselves
If I look at the TCK
- The harness is essentially the standard Perl harness with a thin CLI wrapper, thus essentially works with any test emitting TAP format - The support infra is all custom APIs using libvirt-perl - The tests are mostly written in Perl, but some are written in shell (yuk). They all output TAP format.
One key thing here is that the test harness is fairly loosely coupled to the support infra & tests.
The TAP data format bridged the two, letting us write tests in essentially any language. Of course writing tests in non-Perl was/is tedious today, since there's no support infra for that which exists today.
The TAP data format bridge also means we can easily throw away the current TCK harness and replace it with anything else that can consume tests emitting TAP data.
If I look at Avocado, I think (correct me if i'm wrong)
1. The harness is essentially the standard Python harness with a thin CLI wrapper. Thus needs all tests to implement the Python test APIs
Not really. Even though Avocado is mostly written in Python, there have been provisions for accommodating foreign types of tests (in different forms) since its inception. The most basic way, is, of course, simply treating a test as an executable. But this is far from the only way. For instance, these are other possibilities:
a) if an executable generates TAP, it can consume the test's TAP output and determine the test results
Can you show how to actually make this work, since from the manpage I can only see how to make it emit TAP, not consume it. I see there is a 'tap' plugin but that doesn't seem to provide a runner. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Fri, Jul 2, 2021 at 7:55 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Fri, Jul 02, 2021 at 07:23:57AM -0400, Cleber Rosa wrote:
Hi Daniel,
On Thu, Jul 1, 2021 at 2:05 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
lavocado aims to be an alternative test framework for the libvirt project using Python, python-libvirt and Avocado. This can be used to write unit, functional and integration tests and it is inspired by the libvirt-tck framework.
This series introduces the basic framework along with some basic test examples that I got from libvirt-tck. I would appreciate your comments on this RFC, to see if this fits this project's needs. Also, names and locations are just a proposal and can be changed.
Some high level thoughts
- More extensive functional integration testing coverage is good
- We need to actually run the functional tests regularly reporting via GitLab pipelines in some way
- Using Python is way more contributor friendly than Perl
- This does not need to live in libvirt.git as we don't follow a monolithic repo approach in libvirt, and it already depends on functionality provided by other repos.
When it comes to testing, I feel like there are several distinct pieces to think about
- The execution & reporting harness - Supporting infrastructure to aid writing tests - The set of tests themselves
If I look at the TCK
- The harness is essentially the standard Perl harness with a thin CLI wrapper, thus essentially works with any test emitting TAP format - The support infra is all custom APIs using libvirt-perl - The tests are mostly written in Perl, but some are written in shell (yuk). They all output TAP format.
One key thing here is that the test harness is fairly loosely coupled to the support infra & tests.
The TAP data format bridged the two, letting us write tests in essentially any language. Of course writing tests in non-Perl was/is tedious today, since there's no support infra for that which exists today.
The TAP data format bridge also means we can easily throw away the current TCK harness and replace it with anything else that can consume tests emitting TAP data.
If I look at Avocado, I think (correct me if i'm wrong)
1. The harness is essentially the standard Python harness with a thin CLI wrapper. Thus needs all tests to implement the Python test APIs
Not really. Even though Avocado is mostly written in Python, there have been provisions for accommodating foreign types of tests (in different forms) since its inception. The most basic way, is, of course, simply treating a test as an executable. But this is far from the only way. For instance, these are other possibilities:
a) if an executable generates TAP, it can consume the test's TAP output and determine the test results
Can you show how to actually make this work, since from the manpage I can only see how to make it emit TAP, not consume it. I see there is a 'tap' plugin but that doesn't seem to provide a runner.
There are a couple of ways. The simplest is hinting to Avocado that a file is of kind "tap" with a hintfile. Suppose you have a "test-suite" directory, and in it, you have "test.sh": #!/bin/sh -e echo "1..2" echo "ok 2 /bin/true" echo "ok 3 /bin/uname" And ".avocado.hint": [kinds] tap = *.sh [tap] uri = $testpath If you "cd test-suite", and do "avocado list --resolver *sh" you get: Type Test Tag(s) tap test.sh Resolver Reference Info TEST TYPES SUMMARY ================== tap: 1 And then you can run it with: $ avocado run --test-runner=nrunner ./test.sh JOB ID : 2166ad93ffc25da4d5fc8e7db073f4555d55e81a JOB LOG : /home/cleber/avocado/job-results/job-2021-07-02T08.39-2166ad9/job.log (1/1) ./test.sh: STARTED (1/1) ./test.sh: PASS (0.01 s) RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB HTML : /home/cleber/avocado/job-results/job-2021-07-02T08.39-2166ad9/results.html JOB TIME : 1.31 s This is a relevant documentation pointer: https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/intr... And I'll make sure the man page is updated, thanks for noticing it. Thanks, - Cleber.
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Fri, Jul 2, 2021 at 8:40 AM Cleber Rosa <crosa@redhat.com> wrote:
There are a couple of ways. The simplest is hinting to Avocado that a file is of kind "tap" with a hintfile. Suppose you have a "test-suite" directory, and in it, you have "test.sh":
#!/bin/sh -e echo "1..2" echo "ok 2 /bin/true" echo "ok 3 /bin/uname"
And ".avocado.hint":
[kinds] tap = *.sh
[tap] uri = $testpath
If you "cd test-suite", and do "avocado list --resolver *sh" you get:
Actually, I ran the verbose version, that is, ""avocado -V list --resolver *sh".
Type Test Tag(s) tap test.sh
Resolver Reference Info
TEST TYPES SUMMARY ================== tap: 1
And then you can run it with:
$ avocado run --test-runner=nrunner ./test.sh JOB ID : 2166ad93ffc25da4d5fc8e7db073f4555d55e81a JOB LOG : /home/cleber/avocado/job-results/job-2021-07-02T08.39-2166ad9/job.log (1/1) ./test.sh: STARTED (1/1) ./test.sh: PASS (0.01 s) RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB HTML : /home/cleber/avocado/job-results/job-2021-07-02T08.39-2166ad9/results.html JOB TIME : 1.31 s
This is a relevant documentation pointer: https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/intr...
And I'll make sure the man page is updated, thanks for noticing it.
Thanks, - Cleber.
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
lavocado aims to be an alternative test framework for the libvirt project using Python, python-libvirt and Avocado. This can be used to write unit, functional and integration tests and it is inspired by the libvirt-tck framework.
This series introduces the basic framework along with some basic test examples that I got from libvirt-tck. I would appreciate your comments on this RFC, to see if this fits this project's needs. Also, names and locations are just a proposal and can be changed.
...
Libvirt has consumers writing applications in a variety of languages, and periodically reporting bugs. My general wish for a test harness would be for something that can invoke and consume arbitrary test cases. Essentially if someone can provide us a piece of code that demonstrates a problem in libvirt, I would like to be able to use that directly as a functional test, regardless of language it was written in.
Well, in theory yes - it is all nice to let anyone contribute tests in their language of preference. Realistically, look at the nwfilter stuff in TCK, it's a pile of unintelligible Bash which cannot be debugged. So, I understand why you'd want to imagine or write tests in Go, but someone will have to maintain them otherwise we'd end up with a 3rd framework nobody will want to use. Luckily, Python as a language doesn't suffer from this and we've already made that preference in the libvirt repo already. So, I'd suggest to use our expertise in helping contributors who wish to contribute a test with porting their test case to a Python one - yes, it does consumer human resources and prolongs the time for a contribution to be accepted, but that pays off later down the road when the test cases need to be adapted to new conditions/environments.
In theory the libvirt TCK allowed for that, and indeed the few tests written in shell essentially came to exist because someone at IBM had written some shell code to test firewalls and contributed that as a functional test. They just hacked their script to emit TAP data.
Others have already responded that Avocado can already do the same. What I'd like to say, again in context of arbitrary languages used in tests, the problem with the whole bash machinery IBM contributed is not only that it cannot be debugged, but also that you cannot replay a single failed test, so if an error happened in e.g. test 1148, you have literally no idea how to work only with that test case. Knowing literally zero about Avocado's nrunner and external (non-native) test suite support I cannot tell whether Avocado would have done a better job than TCK (I doubt it), but what I know for sure is that if you write native tests, you get much better support and control over the test suite.
For now, this framework assumes that you are going to run the tests in a fresh clean environment, i.e. a VM. If you decide to use your local system, beware that execution of the tests may affect your system.
It is always good to be wary of functional test frameworks trashing your system, but at the same time I think makes things more flexible if they are able to be reasonably safely run on /any/ host.
For the TCK we tried to be quite conservative in this regard, because it was actually an explicit goal that you can run it on any Fedora host to validate it correctly functioning for KVM. To achieve that we tried to use some standard naming conventions for any resources that tests created, and if we saw pre-existing resources named differently we didn't touch them. ie all VMs were named 'tck-XXX', and were free to be deleted, but other named VMs were ignored.
For anything special, such as testing PCI device assignment, the test configuration file had to explicitly contain details of host devices that were safe to use. This was also done for host block devices or NICs, etc. Thus a default invokation only ran a subset of tests which were safe. The more dangerous tests required you to modify the config file to grant access.
This is what I've always disliked about Avocado-VT - it never cared about your environment and always thrashed your system. Therefore we're approaching this very carefully and even try to think off how could libvirt-ci be integrated into the core Avocado which would help in creating clean environments. IIRC it was still necessary in TCK to run some of the network-related tests with root permissions, but polkit may help (I haven't tried) - I sure don't want to run the test suite with root permissions to overcome some of the issues, because that doesn't simulate the real world usage. Therefore, I'd like libvirt Avocado as a framework to be set up by default in a way where polkit rules are in effect.
I think it would be good if the Avocado supporting test APIs had a similar conceptual approach, especially wrt to a config file granting access to host resources such as NICs/Block devs/PCI devs, where you need prior explicit admin permission to avoid danger.
One of the future goals of this framework is to utilize nested virtualization technologies and hence make sure an L1 guest is provisioned automatically for the tests to be executed in this environment and not tamper with your main system.
I think the test framework should not concern itself with this kind of thing directly. We already have libvirt-ci, which has tools for provisioning VMs with pre-defined package sets. The test harness should just expect that this has been done, and that it is already running in such an environment if the user wanted it to be.
I suggested Beraldo to put this one in so as to make it very clear what our intentions are wrt/ coming up with a new framework and our expectations of it as an end result (like I said, we already have 2 frameworks, we don't want this one to end up the same)
In the TCK config file we provided setting for a URI to connect to other hosts, to enable multi-hosts tests like live migration to be done.
We would not exclude live migration from the test suite, it's simply a beast on its own (mainly because of automation, more specifically preparing a know destination to migrate to). But again, ^this is not something the framework itself should be concerned about, just a background story :).
I'm adding more information with some details inside the README file.
Overall, I'm more enthusiastic about writing tests in Python than Perl, for the long term, but would also potentially like to write tests in Go too.
I'm wondering if we can't bridge the divide between what we have already in libvirt-tck, and what you're bringing to the table with avocado here. While we've not done much development with the TCK recently, there are some very valuable tests there, especially related to firewall support and I don't fancy rewriting them.
Nobody does fancy rewriting those, but there's no way going around that, we'll have to do it at some point because like I said, they're very error prone since they match against the verbatim output of firewall CLI tools which leads to frequent breakages and you figuring out what went wrong is pure torment.
Thus my suggestion is that we:
- Put this avocado code into the libvirt-tck repository, with focus on the supporting infra for making it easy to write Python tests
Clearly I missed the section where you mentioned that we don't need to host the framework in the main repo...Anyway, I disagree here, again, we have a precedent of 2 failed frameworks where nobody really knows where to contribute a test nor wants to deal with any additional repos. A few remarks - it is much easier if the test suite lives alongside our unit test suite which is an "environment" every developer is familiar with. - integration into gitlab CI would be much easier with a single repo than with a multi-repo setup -> yes, we already do that with binding builds, but I'm talking about setups where we'd have to configure gitlab to clone 3 repos and build them in chain, setup the framework config and execute the tests; - keep in mind that we're trying to keep our efforts in sync with QEMU wrt/ (should apply vice-versa as well), so if avocado-qemu was accepted in QEMU upstream as the current framework which also happens to be hosted under the main repo, I don't see a problem why we could not do the same in libvirt - the main selling point IIRC of having TCK hosted in a standalone repo was that you could test ANY version of libvirt -> we didn't have CI back then, but we do now...so if we start running the functional tests early, every upstream release would technically be tested, so there would not be a need to test older versions of libvirt with the same test suite, because it would have already been done before releasing that specific version at some point in time
- Declare that all tests need a way to emit TAP format, no matter what language they're written in. This could either be the test directly emitting TAP, or it could be via use of a plugin. For example 'tappy' can make existing Python tests emit TAP, with no modifications to the tests themselves.
https://tappy.readthedocs.io/en/latest/consumers.html
IOW, you can still invoke the python tests using the standard Python test runner, and still invoke the perl tests using the stnadard Perl test runner if desired.
- Switch the TCK configuration file to use JSON/YAML instead of its current custom format. This is to enable the Python tests to share a config file with the Perl and Shell tests. Thus they can all have the same way to figure out what block devs, NICs, PCI devs, etc they are permitted to use, and what dirs it is safe to create VM images in, etc.
I'm worried that any more time we spend on trying to re-shape TCK into something modern is wasted compared to adopting Avocado. I've been fixing the tests in TCK for a while just to make sure TCK doesn't fail, because the original idea was for Avocado to utilize its nrunner to run TCK in its current shape and start testing early just to give us time to transition from TCK to Avocado by porting a good fistful of tests. Erik
On Tue, Jul 20, 2021 at 02:19:25PM +0200, Erik Skultety wrote:
On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
Libvirt has consumers writing applications in a variety of languages, and periodically reporting bugs. My general wish for a test harness would be for something that can invoke and consume arbitrary test cases. Essentially if someone can provide us a piece of code that demonstrates a problem in libvirt, I would like to be able to use that directly as a functional test, regardless of language it was written in.
Well, in theory yes - it is all nice to let anyone contribute tests in their language of preference. Realistically, look at the nwfilter stuff in TCK, it's a pile of unintelligible Bash which cannot be debugged. So, I understand why you'd want to imagine or write tests in Go, but someone will have to maintain them otherwise we'd end up with a 3rd framework nobody will want to use.
The nwfilter shell script was just badly written, and we should have pushed back on accepting it in that state, but it is was so broadly functional it was preferrable than getting no testing. It could have been just as horrible if written in Perl or Python too - the core approach of the tests themselves was just wrong. The use of shell was just icing on the cake. The main point is that we have applications consuming libvirt in multiple languages. Python and Go are the most important ones in terms of usage today, and we get some quite detailed bug reports with demo code. If we get a bug report with a Go demo program, it can't be assumed to be easy to just convert that to Python, and vica-verca. We do have to consider the maintainability of any tests being contributed, but I don't want to be rejecting bug reproducers that work, just because they're not in a specific language.
Luckily, Python as a language doesn't suffer from this and we've already made that preference in the libvirt repo already. So, I'd suggest to use our expertise in helping contributors who wish to contribute a test with porting their test case to a Python one - yes, it does consumer human resources and prolongs the time for a contribution to be accepted, but that pays off later down the road when the test cases need to be adapted to new conditions/environments.
Certainly I'd say we can consider python as the default or preferred language for writing tests, but I'd like us to be open to accepting tests in other languages.
In theory the libvirt TCK allowed for that, and indeed the few tests written in shell essentially came to exist because someone at IBM had written some shell code to test firewalls and contributed that as a functional test. They just hacked their script to emit TAP data.
Others have already responded that Avocado can already do the same. What I'd like to say, again in context of arbitrary languages used in tests, the problem with the whole bash machinery IBM contributed is not only that it cannot be debugged, but also that you cannot replay a single failed test, so if an error happened in e.g. test 1148, you have literally no idea how to work only with that test case. Knowing literally zero about Avocado's nrunner and external (non-native) test suite support I cannot tell whether Avocado would have done a better job than TCK (I doubt it), but what I know for sure is that if you write native tests, you get much better support and control over the test suite.
As above the shell tests were just bad as an overall conceptual approach. Running commands and then comparing the stdout/err against expected output is way too fragile, and inflexible for debugging too. We see the same problem in QEMU block layer tests too.
IIRC it was still necessary in TCK to run some of the network-related tests with root permissions, but polkit may help (I haven't tried) - I sure don't want to run the test suite with root permissions to overcome some of the issues, because that doesn't simulate the real world usage. Therefore, I'd like libvirt Avocado as a framework to be set up by default in a way where polkit rules are in effect.
In theory you shouldn't need root perimissions directly, because you just have to grant access to libvirtd. In practice of course that is just a facade as a libvirt connection gives you indirect permissions equiva to a root shell. The tests written was supposed to take care when interacting with host resources - so only use NICs that are explicitly listed in the config file, otherise skip the test. This kind of approach is doable for any test framework - just requires someone to define the best practice and reviewers to make sure people actually follow it.
I think the test framework should not concern itself with this kind of thing directly. We already have libvirt-ci, which has tools for provisioning VMs with pre-defined package sets. The test harness should just expect that this has been done, and that it is already running in such an environment if the user wanted it to be.
I suggested Beraldo to put this one in so as to make it very clear what our intentions are wrt/ coming up with a new framework and our expectations of it as an end result (like I said, we already have 2 frameworks, we don't want this one to end up the same)
In the TCK config file we provided setting for a URI to connect to other hosts, to enable multi-hosts tests like live migration to be done.
We would not exclude live migration from the test suite, it's simply a beast on its own (mainly because of automation, more specifically preparing a know destination to migrate to). But again, ^this is not something the framework itself should be concerned about, just a background story :).
I'm adding more information with some details inside the README file.
Overall, I'm more enthusiastic about writing tests in Python than Perl, for the long term, but would also potentially like to write tests in Go too.
I'm wondering if we can't bridge the divide between what we have already in libvirt-tck, and what you're bringing to the table with avocado here. While we've not done much development with the TCK recently, there are some very valuable tests there, especially related to firewall support and I don't fancy rewriting them.
Nobody does fancy rewriting those, but there's no way going around that, we'll have to do it at some point because like I said, they're very error prone since they match against the verbatim output of firewall CLI tools which leads to frequent breakages and you figuring out what went wrong is pure torment.
Agreed, we'll have to bite the bullet eventually. The hard thing is deciding just how we want to validate correctness. Comparing stdout format is an "easy" way. If we don't do that we need to figure out a way to declare what the firewall state is suposed to be, and how to validate a match, as we want to validate that the correct set of rules is actually generated. Possibly we should have used a python binding for libxtables to query the firewall state and thus been in control of the expected data format.
Thus my suggestion is that we:
- Put this avocado code into the libvirt-tck repository, with focus on the supporting infra for making it easy to write Python tests
Clearly I missed the section where you mentioned that we don't need to host the framework in the main repo...Anyway, I disagree here, again, we have a precedent of 2 failed frameworks where nobody really knows where to contribute a test nor wants to deal with any additional repos. A few remarks - it is much easier if the test suite lives alongside our unit test suite which is an "environment" every developer is familiar with.
I don't buy that argument. We already have countless repos and people who are motivated to contribute do so easily. The failures of the existing frameworks is down to a number of reasons. People really didn't want to write Perl code is the big one for the TCK. For the other framework its complexity and fact that it was not safe to run on precious hosts put off ever using it. The biggest thing though is that we never even made an effort to develop a culture of writing functional tests as a prerequisite for merging feature. In QEMU there is a avocado integration in-repo, but very few QEMU contributors ever do anything with it. The block iotests integration is another functional test suite and that is very successful, because the block maintainers have set clear expectation that you must contributor tests to it if you want your features accepted. We can see the same with other projects which have a strong culture of testing. In OpenStack you'll not get your feature accepted unless you've made a credible effort to provide functional tests - which live in a completely separate git repo.
- integration into gitlab CI would be much easier with a single repo than with a multi-repo setup -> yes, we already do that with binding builds, but I'm talking about setups where we'd have to configure gitlab to clone 3 repos and build them in chain, setup the framework config and execute the tests;
As you say, we already deal with that. Cloning extra git repos is a complete non-issue in the grand scheme of everything needed to create and maintain a CI pipeline. If we really wanted to we could create a psuedo-monolithic repo and graft in the individual repos using 'git subtree'.
- keep in mind that we're trying to keep our efforts in sync with QEMU wrt/ (should apply vice-versa as well), so if avocado-qemu was accepted in QEMU upstream as the current framework which also happens to be hosted under the main repo, I don't see a problem why we could not do the same in libvirt
QEMU has a very different maintainance model, with its monolithic repo approach is actually a significant burden to QEMU development. QEMU would be better off in many ways by modularizing its repos. As above, having it in the tree doesn't magically make people contribute to it
- the main selling point IIRC of having TCK hosted in a standalone repo was that you could test ANY version of libvirt -> we didn't have CI back then, but we do now...so if we start running the functional tests early, every upstream release would technically be tested, so there would not be a need to test older versions of libvirt with the same test suite, because it would have already been done before releasing that specific version at some point in time
That's only true of unit tests because those run in isolation only exercising the libvirt code. The functional / integration testing is looking at the results of the entire system. You absolutely need to run those tests in many different scenarios against both pre-release and post-release code, because changes in the OS will frequently break you. Changes in the OS will require fixing the test suite and you then need to run the updated test suite against the existing libvirt release. This is the key reason why it is beneficial to have to cleanly separated from the repo. We see this in Fedora/RHEL CI gating jobs, which run after the RPM builds are done, using a full composed OS environent. For many packages we have to untangle the test suite from the main app code, in order to be able to run it seprately
- Switch the TCK configuration file to use JSON/YAML instead of its current custom format. This is to enable the Python tests to share a config file with the Perl and Shell tests. Thus they can all have the same way to figure out what block devs, NICs, PCI devs, etc they are permitted to use, and what dirs it is safe to create VM images in, etc.
I'm worried that any more time we spend on trying to re-shape TCK into something modern is wasted compared to adopting Avocado. I've been fixing the tests in TCK for a while just to make sure TCK doesn't fail, because the original idea was for Avocado to utilize its nrunner to run TCK in its current shape and start testing early just to give us time to transition from TCK to Avocado by porting a good fistful of tests.
The TCK is frequently broken largely because we never run it, and so it bit rots as the OS we're running it on changes beneath our feet. We see the same problem with any of QEMU's functional tests that don't get run on a frequent basis, includng the avocado tests it has. The only real solution there is to ensure whatever test suite we have is actually run frequently and set an expectation that a maintainers job is more than just writing code - they should be expected to help fix test failures that arise. If the CI is used as a gating check this becomes a strong motivating factor, because contributors code will never get merged if the gating is broken. With the every little experimentation I've done it looks like it should be trivial to get avocado to run all the existing TCK tests by just giving it a hint that they're TAP format. That looks like a small part of work involved in actually developing avocado into a solid test framework for libvirt. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Daniel P. Berrangé writes:
On Tue, Jul 20, 2021 at 02:19:25PM +0200, Erik Skultety wrote:
On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
Libvirt has consumers writing applications in a variety of languages, and periodically reporting bugs. My general wish for a test harness would be for something that can invoke and consume arbitrary test cases. Essentially if someone can provide us a piece of code that demonstrates a problem in libvirt, I would like to be able to use that directly as a functional test, regardless of language it was written in.
Well, in theory yes - it is all nice to let anyone contribute tests in their language of preference. Realistically, look at the nwfilter stuff in TCK, it's a pile of unintelligible Bash which cannot be debugged. So, I understand why you'd want to imagine or write tests in Go, but someone will have to maintain them otherwise we'd end up with a 3rd framework nobody will want to use.
The nwfilter shell script was just badly written, and we should have pushed back on accepting it in that state, but it is was so broadly functional it was preferrable than getting no testing. It could have been just as horrible if written in Perl or Python too - the core approach of the tests themselves was just wrong. The use of shell was just icing on the cake.
Clearly that are exceptions, but I agree that for non-unit tests, the "foreign" aspect of a language is pretty much irrelevant. The suitability of a given language will mostly be based on its own characteristics, and Python seems to do quite well for writing tests indeed. Also, my impression is that the quality of documentation, test templates and relevant functionality that is found in an existing "framework" is generally inversely proportional to people's motivation to write a test in another language. Even if people have another language they strongly prefer, and even if the project will execute without tests written in other languages without major hurdles, documentation/templates/relevant APIs weight very heavily on the other side of the scale. That explains at least partially why Perl is, still to this day, the language of virtually every single test on TCK.
The main point is that we have applications consuming libvirt in multiple languages. Python and Go are the most important ones in terms of usage today, and we get some quite detailed bug reports with demo code. If we get a bug report with a Go demo program, it can't be assumed to be easy to just convert that to Python, and vica-verca.
We do have to consider the maintainability of any tests being contributed, but I don't want to be rejecting bug reproducers that work, just because they're not in a specific language.
QEMU is a close example of tests written in many languages. I agree that it's better to have a test in whatever language than none at all. Having said that, maintaining the quality of tests over time is hard enough when they are mostly uniform (built and/or executed similarly). When tests are written in "one off" languages that require extra steps to be built/executed, this is further aggravated. In short, as long as there's an easy way to run tests, they are run often, results are clear and logs are also somewhat uniform (if not in content at least wrt location), then it's a no-brainer whether to keep a contributed test or not in the tree.
Luckily, Python as a language doesn't suffer from this and we've already made that preference in the libvirt repo already. So, I'd suggest to use our expertise in helping contributors who wish to contribute a test with porting their test case to a Python one - yes, it does consumer human resources and prolongs the time for a contribution to be accepted, but that pays off later down the road when the test cases need to be adapted to new conditions/environments.
Certainly I'd say we can consider python as the default or preferred language for writing tests, but I'd like us to be open to accepting tests in other languages.
In theory the libvirt TCK allowed for that, and indeed the few tests written in shell essentially came to exist because someone at IBM had written some shell code to test firewalls and contributed that as a functional test. They just hacked their script to emit TAP data.
Others have already responded that Avocado can already do the same. What I'd like to say, again in context of arbitrary languages used in tests, the problem with the whole bash machinery IBM contributed is not only that it cannot be debugged, but also that you cannot replay a single failed test, so if an error happened in e.g. test 1148, you have literally no idea how to work only with that test case. Knowing literally zero about Avocado's nrunner and external (non-native) test suite support I cannot tell whether Avocado would have done a better job than TCK (I doubt it), but what I know for sure is that if you write native tests, you get much better support and control over the test suite.
As above the shell tests were just bad as an overall conceptual approach. Running commands and then comparing the stdout/err against expected output is way too fragile, and inflexible for debugging too. We see the same problem in QEMU block layer tests too.
IIRC it was still necessary in TCK to run some of the network-related tests with root permissions, but polkit may help (I haven't tried) - I sure don't want to run the test suite with root permissions to overcome some of the issues, because that doesn't simulate the real world usage. Therefore, I'd like libvirt Avocado as a framework to be set up by default in a way where polkit rules are in effect.
In theory you shouldn't need root perimissions directly, because you just have to grant access to libvirtd. In practice of course that is just a facade as a libvirt connection gives you indirect permissions equiva to a root shell. The tests written was supposed to take care when interacting with host resources - so only use NICs that are explicitly listed in the config file, otherise skip the test. This kind of approach is doable for any test framework - just requires someone to define the best practice and reviewers to make sure people actually follow it.
I think the test framework should not concern itself with this kind of thing directly. We already have libvirt-ci, which has tools for provisioning VMs with pre-defined package sets. The test harness should just expect that this has been done, and that it is already running in such an environment if the user wanted it to be.
I suggested Beraldo to put this one in so as to make it very clear what our intentions are wrt/ coming up with a new framework and our expectations of it as an end result (like I said, we already have 2 frameworks, we don't want this one to end up the same)
In the TCK config file we provided setting for a URI to connect to other hosts, to enable multi-hosts tests like live migration to be done.
We would not exclude live migration from the test suite, it's simply a beast on its own (mainly because of automation, more specifically preparing a know destination to migrate to). But again, ^this is not something the framework itself should be concerned about, just a background story :).
I'm adding more information with some details inside the README file.
Overall, I'm more enthusiastic about writing tests in Python than Perl, for the long term, but would also potentially like to write tests in Go too.
I'm wondering if we can't bridge the divide between what we have already in libvirt-tck, and what you're bringing to the table with avocado here. While we've not done much development with the TCK recently, there are some very valuable tests there, especially related to firewall support and I don't fancy rewriting them.
Nobody does fancy rewriting those, but there's no way going around that, we'll have to do it at some point because like I said, they're very error prone since they match against the verbatim output of firewall CLI tools which leads to frequent breakages and you figuring out what went wrong is pure torment.
Agreed, we'll have to bite the bullet eventually. The hard thing is deciding just how we want to validate correctness. Comparing stdout format is an "easy" way. If we don't do that we need to figure out a way to declare what the firewall state is suposed to be, and how to validate a match, as we want to validate that the correct set of rules is actually generated.
Possibly we should have used a python binding for libxtables to query the firewall state and thus been in control of the expected data format.
Thus my suggestion is that we:
- Put this avocado code into the libvirt-tck repository, with focus on the supporting infra for making it easy to write Python tests
Clearly I missed the section where you mentioned that we don't need to host the framework in the main repo...Anyway, I disagree here, again, we have a precedent of 2 failed frameworks where nobody really knows where to contribute a test nor wants to deal with any additional repos. A few remarks - it is much easier if the test suite lives alongside our unit test suite which is an "environment" every developer is familiar with.
I don't buy that argument. We already have countless repos and people who are motivated to contribute do so easily. The failures of the existing frameworks is down to a number of reasons. People really didn't want to write Perl code is the big one for the TCK. For the other framework its complexity and fact that it was not safe to run on precious hosts put off ever using it.
The biggest thing though is that we never even made an effort to develop a culture of writing functional tests as a prerequisite for merging feature.
In QEMU there is a avocado integration in-repo, but very few QEMU contributors ever do anything with it. The block iotests
Well, while this doesn't have to be the only deciding factor for in-tree or out-of-tree tests, I mentioned exactly the opposite a few days ago as a very positive outcome of having the avocado tests within the QEMU repo, so I'll have to share it here: $ git log --pretty=format:"%an%x09" tests/acceptance/ | sort | uniq | wc -l 36 And this is from a ~3 year old effort. On the other hand, just for comparison on libvirt-tck: $ git log --pretty=format:"%an%x09" scripts/ | sort | uniq | wc -l 17 Within the span of ~12 years. I understand other factors (such as the average number of contributors in each project) apply here. But, I must say that I'm positively impressed by the contributions and number of contributors in "tests/acceptance", and I believe that the ability to send a single patch series with tests included at once have played an significant role here.
integration is another functional test suite and that is very successful, because the block maintainers have set clear expectation that you must contributor tests to it if you want your features accepted.
We can see the same with other projects which have a strong culture of testing. In OpenStack you'll not get your feature accepted unless you've made a credible effort to provide functional tests - which live in a completely separate git repo.
Right. The project's culture and guidelines plays a large role in how successfull testing will be.
- integration into gitlab CI would be much easier with a single repo than with a multi-repo setup -> yes, we already do that with binding builds, but I'm talking about setups where we'd have to configure gitlab to clone 3 repos and build them in chain, setup the framework config and execute the tests;
As you say, we already deal with that. Cloning extra git repos is a complete non-issue in the grand scheme of everything needed to create and maintain a CI pipeline. If we really wanted to we could create a psuedo-monolithic repo and graft in the individual repos using 'git subtree'.
From a completely ignorant standpoint I ask: are there extra steps users need to do do have those tests run on their own GIT clones? I mean, how does a CI system know that it must test "user/libvirt" with "user/libvirt-tck"? I will have my mind blown if this "just works", with the right branch names used across repos, etc.
- keep in mind that we're trying to keep our efforts in sync with QEMU wrt/ (should apply vice-versa as well), so if avocado-qemu was accepted in QEMU upstream as the current framework which also happens to be hosted under the main repo, I don't see a problem why we could not do the same in libvirt
QEMU has a very different maintainance model, with its monolithic repo approach is actually a significant burden to QEMU development. QEMU would be better off in many ways by modularizing its repos. As above, having it in the tree doesn't magically make people contribute to it
- the main selling point IIRC of having TCK hosted in a standalone repo was that you could test ANY version of libvirt -> we didn't have CI back then, but we do now...so if we start running the functional tests early, every upstream release would technically be tested, so there would not be a need to test older versions of libvirt with the same test suite, because it would have already been done before releasing that specific version at some point in time
That's only true of unit tests because those run in isolation only exercising the libvirt code. The functional / integration testing is looking at the results of the entire system. You absolutely need to run those tests in many different scenarios against both pre-release and post-release code, because changes in the OS will frequently break you. Changes in the OS will require fixing the test suite and you then need to run the updated test suite against the existing libvirt release. This is the key reason why it is beneficial to have to cleanly separated from the repo.
These points are true, but in my experience, it's very easy to make them non-issue. In a recent experience, I was using an acceptance tests to find a regression (which slipped through because the test was not being run on CI due to lack of suitable hardware), so Erik's point is also very relevant here. In that case, I had to preserve the test, while going to a point in history where the "avocado_qemu" supporting code had some changes I needed. Either a "avocado run -p qemu_bin=/path/to/other/qemu-bin" or a PYTHONPATH tweak pointing to a stable "avocado_qemu" was enough to solve that. Having important tests "always" running, and/or documenting/scripting how to run newer tests with older repos/binaries is, in my experience, an easy enough remedy to the single repo problem.
We see this in Fedora/RHEL CI gating jobs, which run after the RPM builds are done, using a full composed OS environent. For many packages we have to untangle the test suite from the main app code, in order to be able to run it seprately
- Switch the TCK configuration file to use JSON/YAML instead of its current custom format. This is to enable the Python tests to share a config file with the Perl and Shell tests. Thus they can all have the same way to figure out what block devs, NICs, PCI devs, etc they are permitted to use, and what dirs it is safe to create VM images in, etc.
I'm worried that any more time we spend on trying to re-shape TCK into something modern is wasted compared to adopting Avocado. I've been fixing the tests in TCK for a while just to make sure TCK doesn't fail, because the original idea was for Avocado to utilize its nrunner to run TCK in its current shape and start testing early just to give us time to transition from TCK to Avocado by porting a good fistful of tests.
The TCK is frequently broken largely because we never run it, and so it bit rots as the OS we're running it on changes beneath our feet. We see the same problem with any of QEMU's functional tests that don't get run on a frequent basis, includng the avocado tests it has.
The only real solution there is to ensure whatever test suite we have is actually run frequently and set an expectation that a maintainers job is more than just writing code - they should be expected to help fix test failures that arise. If the CI is used as a gating check this becomes a strong motivating factor, because contributors code will never get merged if the gating is broken.
With the every little experimentation I've done it looks like it should be trivial to get avocado to run all the existing TCK tests by just giving it a hint that they're TAP format. That looks like a small part of work involved in actually developing avocado into a solid test framework for libvirt.
Regards, Daniel
Again, thanks a lot for all your feedback here! - Cleber.
On Wed, Jul 21, 2021 at 07:26:49PM +0100, Daniel P. Berrangé wrote:
On Tue, Jul 20, 2021 at 02:19:25PM +0200, Erik Skultety wrote:
On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
Libvirt has consumers writing applications in a variety of languages, and periodically reporting bugs. My general wish for a test harness would be for something that can invoke and consume arbitrary test cases. Essentially if someone can provide us a piece of code that demonstrates a problem in libvirt, I would like to be able to use that directly as a functional test, regardless of language it was written in.
Well, in theory yes - it is all nice to let anyone contribute tests in their language of preference. Realistically, look at the nwfilter stuff in TCK, it's a pile of unintelligible Bash which cannot be debugged. So, I understand why you'd want to imagine or write tests in Go, but someone will have to maintain them otherwise we'd end up with a 3rd framework nobody will want to use.
The nwfilter shell script was just badly written, and we should have pushed back on accepting it in that state, but it is was so broadly functional it was preferrable than getting no testing. It could have been just as horrible if written in Perl or Python too - the core approach of the tests themselves was just wrong. The use of shell was just icing on the cake.
The main point is that we have applications consuming libvirt in multiple languages. Python and Go are the most important ones in terms of usage today, and we get some quite detailed bug reports with demo code. If we get a bug report with a Go demo program, it can't be assumed to be easy to just convert that to Python, and vica-verca.
If the bug is against the Go/Whatever bindings, the test should be integrated there. If it's against libvirt core, I don't see how the test case could not be ported nor reproduced with Python and so I'd strictly mandate Python - now, I'm not saying we should reject anything, what I'm saying is that the same way we don't accept just any code in any language in libvirt upstream (even if people went the extra mile and made sure the whole project would just work) and people have to re-spin a number of revisions until the community is satisfied enough to merge it because of both stability and maintainability reasons, we should approach the tests with the same care and set clear expectations that must be met if a test case is to be contributed. If a test case is contributed as part of some code changes, then it has to conform to our expecations == Python. On the other hand if it's part of a bug report, than we need to IMO invest the time into porting that test case to our selected language of preference ourselves. ...
I'm wondering if we can't bridge the divide between what we have already in libvirt-tck, and what you're bringing to the table with avocado here. While we've not done much development with the TCK recently, there are some very valuable tests there, especially related to firewall support and I don't fancy rewriting them.
Nobody does fancy rewriting those, but there's no way going around that, we'll have to do it at some point because like I said, they're very error prone since they match against the verbatim output of firewall CLI tools which leads to frequent breakages and you figuring out what went wrong is pure torment.
Agreed, we'll have to bite the bullet eventually. The hard thing is deciding just how we want to validate correctness. Comparing stdout format is an "easy" way. If we don't do that we need to figure out a way to declare what the firewall state is suposed to be, and how to validate a match, as we want to validate that the correct set of rules is actually generated.
Possibly we should have used a python binding for libxtables to query the firewall state and thus been in control of the expected data format.
I don't know anything about libxtables, so I'll happily leave it on the table and hope for the best in terms of relying on a stable data format, but I guess the proper way of handling networking is to come up with helper code that constructs and sends raw packets to test whether the firewall rules we put in place as part of a test case actually work. ...
Thus my suggestion is that we:
- Put this avocado code into the libvirt-tck repository, with focus on the supporting infra for making it easy to write Python tests
Clearly I missed the section where you mentioned that we don't need to host the framework in the main repo...Anyway, I disagree here, again, we have a precedent of 2 failed frameworks where nobody really knows where to contribute a test nor wants to deal with any additional repos. A few remarks - it is much easier if the test suite lives alongside our unit test suite which is an "environment" every developer is familiar with.
I don't buy that argument. We already have countless repos and people who are motivated to contribute do so easily. The failures of the existing frameworks is down to a number of reasons. People really didn't want to write Perl code is the big one for the TCK. For the other framework its complexity and fact that it was not safe to run on precious hosts put off ever using it.
The biggest thing though is that we never even made an effort to develop a culture of writing functional tests as a prerequisite for merging feature.
In QEMU there is a avocado integration in-repo, but very few QEMU contributors ever do anything with it. The block iotests integration is another functional test suite and that is very successful, because the block maintainers have set clear expectation that you must contributor tests to it if you want your features accepted.
We can see the same with other projects which have a strong culture of testing. In OpenStack you'll not get your feature accepted unless you've made a credible effort to provide functional tests - which live in a completely separate git repo.
- integration into gitlab CI would be much easier with a single repo than with a multi-repo setup -> yes, we already do that with binding builds, but I'm talking about setups where we'd have to configure gitlab to clone 3 repos and build them in chain, setup the framework config and execute the tests;
As you say, we already deal with that. Cloning extra git repos is a complete non-issue in the grand scheme of everything needed to create and maintain a CI pipeline. If we really wanted to we could create a psuedo-monolithic repo and graft in the individual repos using 'git subtree'.
Well, it's not about that it cannot be done, it's about developer experience. Let's say I'm working on a feature to which I'm expected to contribute a test case for. I have to clone 2 repos and make changes to both - this is fine, even though having it in a single repo would speed things up. Now I have to test my test case, what then? - I need to build and deploy libvirt myself - I need to build/run the test suite against the libvirt I built in the previous step - all manually! If we instead have everything in the same repo, we can (and we indeed plan to) have automation helpers, that would prepare the environment for you and with a single "make integration_test" an environment would be created, git would be clone, libvirt built and installed, and test run against that. Also the review process would be much more predictable, because a feature could not be merged before the corresponding test was merged which is much easier to do in a single repo than 2 individual ones, especially with a one that is solely based on a merge request workflow which on its own is enough for many core libvirt developers to be discouraged to bother with contributing. All I'm trying to say here is that we need to put the developer's needs and workflow first, because in the end, it's them who are going to both contribute and maintain those tests. And so we should get more developer voices here (so far it was just you Dan, Cleber, Beraldo, and myself) - if the overall consensus is in favour of usage of arbitrary languages for test cases and hosting the repo separately, by all means let's do it as long as the core development team agrees on it, otherwise we'll just end up with another useless framework and thus many human hours wasted on the CI efforts. Erik
On Mon, Jul 26, 2021 at 03:27:36PM +0200, Erik Skultety wrote:
On Wed, Jul 21, 2021 at 07:26:49PM +0100, Daniel P. Berrangé wrote:
On Tue, Jul 20, 2021 at 02:19:25PM +0200, Erik Skultety wrote:
On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
Libvirt has consumers writing applications in a variety of languages, and periodically reporting bugs. My general wish for a test harness would be for something that can invoke and consume arbitrary test cases. Essentially if someone can provide us a piece of code that demonstrates a problem in libvirt, I would like to be able to use that directly as a functional test, regardless of language it was written in.
Well, in theory yes - it is all nice to let anyone contribute tests in their language of preference. Realistically, look at the nwfilter stuff in TCK, it's a pile of unintelligible Bash which cannot be debugged. So, I understand why you'd want to imagine or write tests in Go, but someone will have to maintain them otherwise we'd end up with a 3rd framework nobody will want to use.
The nwfilter shell script was just badly written, and we should have pushed back on accepting it in that state, but it is was so broadly functional it was preferrable than getting no testing. It could have been just as horrible if written in Perl or Python too - the core approach of the tests themselves was just wrong. The use of shell was just icing on the cake.
The main point is that we have applications consuming libvirt in multiple languages. Python and Go are the most important ones in terms of usage today, and we get some quite detailed bug reports with demo code. If we get a bug report with a Go demo program, it can't be assumed to be easy to just convert that to Python, and vica-verca.
If the bug is against the Go/Whatever bindings, the test should be integrated there. If it's against libvirt core, I don't see how the test case could not be ported nor reproduced with Python and so I'd strictly mandate Python - now, I'm not saying we should reject anything, what I'm saying is that the same way we don't accept just any code in any language in libvirt upstream (even if people went the extra mile and made sure the whole project would just work) and people have to re-spin a number of revisions until the community is satisfied enough to merge it because of both stability and maintainability reasons, we should approach the tests with the same care and set clear expectations that must be met if a test case is to be contributed.
If a test case is contributed as part of some code changes, then it has to conform to our expecations == Python. On the other hand if it's part of a bug report, than we need to IMO invest the time into porting that test case to our selected language of preference ourselves.
Porting test cases to another language is busy work that is not likely to be done.
Thus my suggestion is that we:
- Put this avocado code into the libvirt-tck repository, with focus on the supporting infra for making it easy to write Python tests
Clearly I missed the section where you mentioned that we don't need to host the framework in the main repo...Anyway, I disagree here, again, we have a precedent of 2 failed frameworks where nobody really knows where to contribute a test nor wants to deal with any additional repos. A few remarks - it is much easier if the test suite lives alongside our unit test suite which is an "environment" every developer is familiar with.
I don't buy that argument. We already have countless repos and people who are motivated to contribute do so easily. The failures of the existing frameworks is down to a number of reasons. People really didn't want to write Perl code is the big one for the TCK. For the other framework its complexity and fact that it was not safe to run on precious hosts put off ever using it.
The biggest thing though is that we never even made an effort to develop a culture of writing functional tests as a prerequisite for merging feature.
In QEMU there is a avocado integration in-repo, but very few QEMU contributors ever do anything with it. The block iotests integration is another functional test suite and that is very successful, because the block maintainers have set clear expectation that you must contributor tests to it if you want your features accepted.
We can see the same with other projects which have a strong culture of testing. In OpenStack you'll not get your feature accepted unless you've made a credible effort to provide functional tests - which live in a completely separate git repo.
- integration into gitlab CI would be much easier with a single repo than with a multi-repo setup -> yes, we already do that with binding builds, but I'm talking about setups where we'd have to configure gitlab to clone 3 repos and build them in chain, setup the framework config and execute the tests;
As you say, we already deal with that. Cloning extra git repos is a complete non-issue in the grand scheme of everything needed to create and maintain a CI pipeline. If we really wanted to we could create a psuedo-monolithic repo and graft in the individual repos using 'git subtree'.
Well, it's not about that it cannot be done, it's about developer experience. Let's say I'm working on a feature to which I'm expected to contribute a test case for.
I have to clone 2 repos and make changes to both - this is fine, even though having it in a single repo would speed things up. Now I have to test my test case, what then? - I need to build and deploy libvirt myself - I need to build/run the test suite against the libvirt I built in the previous step - all manually!
You're describing a scenario in which we provide zero supporting tools or scripts to aid in running tests. We should of course add a target to the build system which lets you run tests in a single command. That does not imply it all has to be part of the same git repo. None of that need to be manually done. It can all be easily scripted so that it can be invoked from a single command, regardless of whether it is in the same repo or not. Also there is no escape from dealing with multiple repos. If you are testing a new API, we need to clone libvirt-python repo and build it to access the newly added API from the test cases.
If we instead have everything in the same repo, we can (and we indeed plan to) have automation helpers, that would prepare the environment for you and with a single "make integration_test" an environment would be created, git would be clone, libvirt built and installed, and test run against that.
This has no bear on whether the tests framework is in the same repo or not. Any such "make integration_test" can be written to work regardless, it means the helper has to add a git clone which is trivial.
Also the review process would be much more predictable, because a feature could not be merged before the corresponding test was merged which is much easier to do in a single repo than 2 individual ones, especially with a one that is solely based on a merge request workflow which on its own is enough for many core libvirt developers to be discouraged to bother with contributing.
Libvirt.git will move to using merge requests workflow too, so that is not really relevant as a forward looking rational. Tests should not need to be merged in lockstep, as they should be decoupled from specific versions of libvirt. We want to be auto-detecting features which are available in the libvirt version / hypervisor choice and gracefully skip tests if unavailable.
All I'm trying to say here is that we need to put the developer's needs and workflow first, because in the end, it's them who are going to both contribute and maintain those tests. And so we should get more developer voices here (so far it was just you Dan, Cleber, Beraldo, and myself) - if the overall consensus is in favour of usage of arbitrary languages for test cases and hosting the repo separately, by all means let's do it as long as the core development team agrees on it, otherwise we'll just end up with another useless framework and thus many human hours wasted on the CI efforts.
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (4)
-
Beraldo Leal -
Cleber Rosa -
Daniel P. Berrangé -
Erik Skultety