Re: [libvirt PATCH 0/4] RFC: tests: introduce lavocado

On Thu, Jul 01, 2021 at 06:09:47PM -0300, Beraldo Leal wrote: > On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote: > > On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote: > > > I'm adding more information with some details inside the README file. > > > > Overall, I'm more enthusiastic about writing tests in Python > > than Perl, for the long term, but would also potentially like > > to write tests in Go too. > > > > I'm wondering if we can't bridge the divide between what we > > have already in libvirt-tck, and what you're bringing to the > > table with avocado here. While we've not done much development > > with the TCK recently, there are some very valuable tests > > there, especially related to firewall support and I don't > > fancy rewriting them. > > > > Thus my suggestion is that we: > > > > - Put this avocado code into the libvirt-tck repository, > > with focus on the supporting infra for making it easy to > > write Python tests > > > > - Declare that all tests need a way to emit TAP format, > > no matter what language they're written in. This could > > either be the test directly emitting TAP, or it could > > be via use of a plugin. For example 'tappy' can make > > existing Python tests emit TAP, with no modifications > > to the tests themselves. > > > > https://tappy.readthedocs.io/en/latest/consumers.html > > > > IOW, you can still invoke the python tests using the > > standard Python test runner, and still invoke the perl > > tests using the stnadard Perl test runner if desired. > > This is supported already: > > $ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py > 1..3 > ok 1 tests/domain/transient.py:TransientDomain.test_autostart > ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle > ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent This is nice, showing fine grained TAP output lines for each individual test within the test program I tried using the hints file that Cleber pointed to make avocado *consume* TAP format for the Perl/Shell scripts: $ cd libvirt-tck $ cat .avocado.hint [kinds] tap = scripts/*/*.t [tap] uri = $testpath And I can indeed invoke the scripts: $ avocado run ./scripts/domain/05*.t JOB ID : b5d596d909dc8024d986957c909fc8fb6b31e2dd JOB LOG : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/job.log (1/2) ./scripts/domain/050-transient-lifecycle.t: PASS (0.70 s) (2/2) ./scripts/domain/051-transient-autostart.t: PASS (0.76 s) RESULTS : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB HTML : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/results.html JOB TIME : 1.90 s which is good. And also I can ask it to produce tap output too: $ avocado run --tap - ./scripts/domain/05*.t 1..2 ok 1 ./scripts/domain/050-transient-lifecycle.t ok 2 ./scripts/domain/051-transient-autostart.t But this output isn't entirely what I was after. This is just summarizing the results of each test program. I can't find a way to make it show the fine grained tap output for the individual tests, like it does for the python program