On Fri, Jul 2, 2021 at 7:55 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Fri, Jul 02, 2021 at 07:23:57AM -0400, Cleber Rosa wrote:
Hi Daniel,
On Thu, Jul 1, 2021 at 2:05 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
lavocado aims to be an alternative test framework for the libvirt project using Python, python-libvirt and Avocado. This can be used to write unit, functional and integration tests and it is inspired by the libvirt-tck framework.
This series introduces the basic framework along with some basic test examples that I got from libvirt-tck. I would appreciate your comments on this RFC, to see if this fits this project's needs. Also, names and locations are just a proposal and can be changed.
Some high level thoughts
- More extensive functional integration testing coverage is good
- We need to actually run the functional tests regularly reporting via GitLab pipelines in some way
- Using Python is way more contributor friendly than Perl
- This does not need to live in libvirt.git as we don't follow a monolithic repo approach in libvirt, and it already depends on functionality provided by other repos.
When it comes to testing, I feel like there are several distinct pieces to think about
- The execution & reporting harness - Supporting infrastructure to aid writing tests - The set of tests themselves
If I look at the TCK
- The harness is essentially the standard Perl harness with a thin CLI wrapper, thus essentially works with any test emitting TAP format - The support infra is all custom APIs using libvirt-perl - The tests are mostly written in Perl, but some are written in shell (yuk). They all output TAP format.
One key thing here is that the test harness is fairly loosely coupled to the support infra & tests.
The TAP data format bridged the two, letting us write tests in essentially any language. Of course writing tests in non-Perl was/is tedious today, since there's no support infra for that which exists today.
The TAP data format bridge also means we can easily throw away the current TCK harness and replace it with anything else that can consume tests emitting TAP data.
If I look at Avocado, I think (correct me if i'm wrong)
1. The harness is essentially the standard Python harness with a thin CLI wrapper. Thus needs all tests to implement the Python test APIs
Not really. Even though Avocado is mostly written in Python, there have been provisions for accommodating foreign types of tests (in different forms) since its inception. The most basic way, is, of course, simply treating a test as an executable. But this is far from the only way. For instance, these are other possibilities:
a) if an executable generates TAP, it can consume the test's TAP output and determine the test results
Can you show how to actually make this work, since from the manpage I can only see how to make it emit TAP, not consume it. I see there is a 'tap' plugin but that doesn't seem to provide a runner.
There are a couple of ways. The simplest is hinting to Avocado that a file is of kind "tap" with a hintfile. Suppose you have a "test-suite" directory, and in it, you have "test.sh": #!/bin/sh -e echo "1..2" echo "ok 2 /bin/true" echo "ok 3 /bin/uname" And ".avocado.hint": [kinds] tap = *.sh [tap] uri = $testpath If you "cd test-suite", and do "avocado list --resolver *sh" you get: Type Test Tag(s) tap test.sh Resolver Reference Info TEST TYPES SUMMARY ================== tap: 1 And then you can run it with: $ avocado run --test-runner=nrunner ./test.sh JOB ID : 2166ad93ffc25da4d5fc8e7db073f4555d55e81a JOB LOG : /home/cleber/avocado/job-results/job-2021-07-02T08.39-2166ad9/job.log (1/1) ./test.sh: STARTED (1/1) ./test.sh: PASS (0.01 s) RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB HTML : /home/cleber/avocado/job-results/job-2021-07-02T08.39-2166ad9/results.html JOB TIME : 1.31 s This is a relevant documentation pointer: https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/intr... And I'll make sure the man page is updated, thanks for noticing it. Thanks, - Cleber.
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|