Re: [libvirt PATCH 0/4] RFC: tests: introduce lavocado
Daniel P. Berrangé writes:
On Thu, Jul 01, 2021 at 06:09:47PM -0300, Beraldo Leal wrote:
>
> This is supported already:
>
> $ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py
> 1..3
> ok 1 tests/domain/transient.py:TransientDomain.test_autostart
> ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle
> ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent
This is nice, showing fine grained TAP output lines for each
individual test within the test program
I tried using the hints file that Cleber pointed to make avocado
*consume* TAP format for the Perl/Shell scripts:
$ cd libvirt-tck
$ cat .avocado.hint
[kinds]
tap = scripts/*/*.t
[tap]
uri = $testpath
And I can indeed invoke the scripts:
$ avocado run ./scripts/domain/05*.t
JOB ID : b5d596d909dc8024d986957c909fc8fb6b31e2dd
JOB LOG : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/job.log
(1/2) ./scripts/domain/050-transient-lifecycle.t: PASS (0.70 s)
(2/2) ./scripts/domain/051-transient-autostart.t: PASS (0.76 s)
RESULTS : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML :
/home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/results.html
JOB TIME : 1.90 s
which is good.
And also I can ask it to produce tap output too:
$ avocado run --tap - ./scripts/domain/05*.t
1..2
ok 1 ./scripts/domain/050-transient-lifecycle.t
ok 2 ./scripts/domain/051-transient-autostart.t
But this output isn't entirely what I was after. This is just summarizing
the results of each test program.
Hi Daniel,
From the user experience perspective, I completely understand your
point. I would side with this expectation and output any day if acting
purely as a user.
But, as a framework, Avocado came up with some well defined concepts,
and one of them is what a test is. To be able to "replay only the
failed tests of the latest job" or to "run a test completely isolated in
a container or VM", there needs to be an specific and indistinguishable
entry point for anything that is considered a test.
I can't find a way to make it show the fine grained tap output
for the
individual tests, like it does for the python program
eg I'd like to be able to see something similar to:
$ ./scripts/domain/050-transient-lifecycle.t
1..2
# Creating a new transient domain
ok 1 - created transient domain object
# Destroying the transient domain
# Checking that transient domain has gone away
ok 2 - NO_DOMAIN error raised from missing domain
$ ./scripts/domain/051-transient-autostart.t
1..4
# Creating a new transient domain
ok 1 - created transient domain object
ok 2 - autostart is disabled for transient VMs
ok 3 - Set autostart not supported on transient VMs
# Destroying the transient domain
# Checking that transient domain has gone away
ok 4 - NO_DOMAIN error raised from missing domain
Results for tests that produce TAP can be seen in the granularity you
list here, given some of the conditions like the one explained before.
For instance, the GLib plugin[1] knows that it's possible to run a
single test case on tests that use the g_test_*() APIs, by providing a
"-p /test/case/name".
We could in theory reorganize the current Perl based tests, so that a
test matches a function or some other type of code marker that can be
used as an entry point. Given that the overall result will always be
valid and the log files will always contain the more granular
information, I have mixed feelings about the cost/benefit of doing that,
but you are surely a much better judge for that.
None the less this does seem like we're pretty close to being able
to do something useful in integration
Regards,
Daniel
Thanks for the feedback!
- Cleber.
PS: The GLib plugin was deprecated because we observed that most users
would do fine with the TAP plugin by itself, but it can be reinstated if
there is a demand.
PS2: If a test needs to produce more detailed information about its
status, such as for a long running test with different stages, we
recommend that advanced logging is used[2], instead of calling the
different stages of a single test (from Avocado's PoV).
[1] - https://avocado-framework.readthedocs.io/en/82.0/plugins/optional/glib.html
[2] -
https://avocado-framework.readthedocs.io/en/89.0/guides/writer/chapters/l...