On Wed, Jul 21, 2021 at 04:22:19PM -0300, Beraldo Leal wrote:
On Wed, Jul 21, 2021 at 06:50:03PM +0100, Daniel P. Berrangé wrote:
On Thu, Jul 01, 2021 at 06:09:47PM -0300, Beraldo Leal wrote:
On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
I'm adding more information with some details inside the README file.
Overall, I'm more enthusiastic about writing tests in Python than Perl, for the long term, but would also potentially like to write tests in Go too.
I'm wondering if we can't bridge the divide between what we have already in libvirt-tck, and what you're bringing to the table with avocado here. While we've not done much development with the TCK recently, there are some very valuable tests there, especially related to firewall support and I don't fancy rewriting them.
Thus my suggestion is that we:
- Put this avocado code into the libvirt-tck repository, with focus on the supporting infra for making it easy to write Python tests
- Declare that all tests need a way to emit TAP format, no matter what language they're written in. This could either be the test directly emitting TAP, or it could be via use of a plugin. For example 'tappy' can make existing Python tests emit TAP, with no modifications to the tests themselves.
https://tappy.readthedocs.io/en/latest/consumers.html
IOW, you can still invoke the python tests using the standard Python test runner, and still invoke the perl tests using the stnadard Perl test runner if desired.
This is supported already:
$ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py 1..3 ok 1 tests/domain/transient.py:TransientDomain.test_autostart ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent
This is nice, showing fine grained TAP output lines for each individual test within the test program
I tried using the hints file that Cleber pointed to make avocado *consume* TAP format for the Perl/Shell scripts:
$ cd libvirt-tck $ cat .avocado.hint [kinds] tap = scripts/*/*.t
[tap] uri = $testpath
And I can indeed invoke the scripts:
$ avocado run ./scripts/domain/05*.t JOB ID : b5d596d909dc8024d986957c909fc8fb6b31e2dd JOB LOG : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/job.log (1/2) ./scripts/domain/050-transient-lifecycle.t: PASS (0.70 s) (2/2) ./scripts/domain/051-transient-autostart.t: PASS (0.76 s) RESULTS : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB HTML : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/results.html JOB TIME : 1.90 s
which is good.
And also I can ask it to produce tap output too:
$ avocado run --tap - ./scripts/domain/05*.t 1..2 ok 1 ./scripts/domain/050-transient-lifecycle.t ok 2 ./scripts/domain/051-transient-autostart.t
But this output isn't entirely what I was after. This is just summarizing the results of each test program.
I can't find a way to make it show the fine grained tap output for the individual tests, like it does for the python program
Actually, the first Python TAP output example is showing a coarse TAP result. I have combined three transient tests into one single file but splitting them into three different methods, so we could execute each one individually. i.e: tests/domain/transient.py:TransientDomain.test_autostart.
So, I did some reorg when migrating to Python test.
In order to archive the same with Perl, we could do the same there, because the way individual tests are written there, doesn't allow for individual execution.
Yes, we could do some tricks, to parse and combine outputs and list as it was a more fine graned, but afaict, we could not individually execute those. This is part of Avocado test definition where in order to be called a test, we need to be able to execute those individually as well.
Ok, I'm not so fussed about whether avocado can ultimately preserve the fine grained TAP output. Mostly I'm looking to understand how you should debug failures when they go wrong, becuase the default output I see from avocado is very terse giving no indiciation of the the failure - merely that there has been a failure. In an interactive environment you can just re-rnu the individual failed test directly. In an automated CI environment you need the test harness to display enough info to debug the failure directly. Ideally it should dump the full stdout+stderr either to the console or a log file that can be publish as an artifact from CI. Meson tends todo the latter creating a log file will full stdout/err from all tests. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|