Re: [libvirt PATCH 0/4] RFC: tests: introduce lavocado

On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote: Some high level thoughts - More extensive functional integration testing coverage is good - We need to actually run the functional tests regularly reporting via GitLab pipelines in some way - Using Python is way more contributor friendly than Perl - This does not need to live in libvirt.git as we don't follow a monolithic repo approach in libvirt, and it already depends on functionality provided by other repos. When it comes to testing, I feel like there are several distinct pieces to think about - The execution & reporting harness - Supporting infrastructure to aid writing tests - The set of tests themselves If I look at the TCK - The harness is essentially the standard Perl harness with a thin CLI wrapper, thus essentially works with any test emitting TAP format - The support infra is all custom APIs using libvirt-perl - The tests are mostly written in Perl, but some are written in shell (yuk). They all output TAP format. One key thing here is that the test harness is fairly loosely coupled to the support infra & tests. The TAP data format bridged the two, letting us write tests in essentially any language. Of course writing tests in non-Perl was/is tedious today, since there's no support infra for that which exists today. The TAP data format bridge also means we can easily throw away the current TCK harness and replace it with anything else that can consume tests emitting TAP data. If I look at Avocado, I think (correct me if i'm wrong) 1. The harness is essentially the standard Python harness with a thin CLI wrapper. Thus needs all tests to implement the Python test APIs 2. The support infra is all custom APIs using libvirt-python 3. The tests are to be written entirely in Python, to integrate with the python test harness IIUC, 90% of this patch series is essentially about (2), since (1) is fairly trivial and for (3) there's just simple demo tests. Libvirt has consumers writing applications in a variety of languages, and periodically reporting bugs. My general wish for a test harness would be for something that can invoke and consume arbitrary test cases. Essentially if someone can provide us a piece of code that demonstrates a problem in libvirt, I would like to be able to use that directly as a functional test, regardless of language it was written in. In theory the libvirt TCK allowed for that, and indeed the few tests written in shell essentially came to exist because someone at IBM had written some shell code to test firewalls and contributed that as a functional test. They just hacked their script to emit TAP data. It is always good to be wary of functional test frameworks trashing your system, but at the same time I think makes things more flexible if they are able to be reasonably safely run on /any/ host. For the TCK we tried to be quite conservative in this regard, because it was actually an explicit goal that you can run it on any Fedora host to validate it correctly functioning for KVM. To achieve that we tried to use some standard naming conventions for any resources that tests created, and if we saw pre-existing resources named differently we didn't touch them. ie all VMs were named 'tck-XXX', and were free to be deleted, but other named VMs were ignored. For anything special, such as testing PCI device assignment, the test configuration file had to explicitly contain details of host devices that were safe to use. This was also done for host block devices or NICs, etc. Thus a default invokation only ran a subset of tests which were safe. The more dangerous tests required you to modify the config file to grant access. I think it would be good if the Avocado supporting test APIs had a similar conceptual approach, especially wrt to a config file granting access to host resources such as NICs/Block devs/PCI devs, where you need prior explicit admin permission to avoid danger. I think the test framework should not concern itself with this kind of thing directly. We already have libvirt-ci, which has tools for provisioning VMs with pre-defined package sets. The test harness should just expect that this has been done, and that it is already running in such an environment if the user wanted it to be. In the TCK config file we provided setting for a URI to connect to other hosts, to enable multi-hosts tests like live migration to be done. Overall, I'm more enthusiastic about writing tests in Python than Perl, for the long term, but would also potentially like to write tests in Go too. I'm wondering if we can't bridge the divide between what we have already in libvirt-tck, and what you're bringing to the table with avocado here. While we've not done much development with the TCK recently, there are some very valuable tests there, especially related to firewall support and I don't fancy rewriting them. Thus my suggestion is that we: - Put this avocado code into the libvirt-tck repository, with focus on the supporting infra for making it easy to write Python tests - Declare that all tests need a way to emit TAP format, no matter what language they're written in. This could either be the test directly emitting TAP, or it could be via use of a plugin. For example 'tappy' can make existing Python tests emit TAP, with no modifications to the tests themselves. https://tappy.readthedocs.io/en/latest/consumers.html IOW, you can still invoke the python tests using the standard Python test runner, and still invoke the perl tests using the stnadard Perl test runner if desired. - Switch the TCK configuration file to use JSON/YAML instead of its current custom format. This is to enable the Python tests to share a config file with the Perl and Shell tests. Thus they can all have the same way to figure out what block devs, NICs, PCI devs, etc they are permitted to use, and what dirs it is safe to create VM images in, etc. - Replace the current Perl based execution harness with something using Python, so that it is less of a pain to install in terms of dependancies. Looks like we could just do something using the 'tappy' Python APIs to consume the TAP format from all the tests and generate a junit report that GitLab can consume. - Rename the RPMs so they match the project name "libvirt-tck", instead of "perl-Sys-Virt-TCK" to emphasis that this is not a Perl project, it is a testing project, of which some parts were historically Perl, but new parts will be mostly Python (and possibly Go/etc in future if desired). Potentially have RPMs split: * libvirt-tck - just the execution harness, currently the Perl one, but to be replaced with a Python one. * libvirt-tck-tests-python - supporting APIs and tests written in Python - essentially all of this series, and future tests written * libvirt-tck-tests-perl - supporting APIs and tests written in Perl - essentially most of existing TCK stuff * libvirt-tck-tests-shell - supporting APIs and tests written in shell - mostly the network/firewall related TCK pieces Given the existance of the 'tappy' tool, it feels like the first two points ought to be reasonably doable without any significant changes to what you've already written. Just need to import it as is, and then using tappy as a shim to let it invoked from libvirt-tck. Can still have the existing way to invoke it directly too as an alternative. The common JSON/YAML config file, and adopting some standard naming conventions and resource usage conventions are probably where the real work would lie. That's likely not too bad at this stage though, since you've not actually written a large number of test cases. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|