Re: [libvirt PATCH 0/4] RFC: tests: introduce lavocado

On Fri, Jul 02, 2021 at 07:23:57AM -0400, Cleber Rosa wrote: > Hi Daniel, > > On Thu, Jul 1, 2021 at 2:05 PM Daniel P. Berrangé <berrange(a)redhat.com&gt; wrote: > > > > On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote: > > > lavocado aims to be an alternative test framework for the libvirt > > > project using Python, python-libvirt and Avocado. This can be used to > > > write unit, functional and integration tests and it is inspired by the > > > libvirt-tck framework. > > > > > > This series introduces the basic framework along with some basic test > > > examples that I got from libvirt-tck. I would appreciate your comments > > > on this RFC, to see if this fits this project's needs. Also, names and > > > locations are just a proposal and can be changed. > > > > Some high level thoughts > > > > - More extensive functional integration testing coverage is good > > > > - We need to actually run the functional tests regularly reporting > > via GitLab pipelines in some way > > > > - Using Python is way more contributor friendly than Perl > > > > - This does not need to live in libvirt.git as we don't follow > > a monolithic repo approach in libvirt, and it already depends > > on functionality provided by other repos. > > > > > > When it comes to testing, I feel like there are several distinct > > pieces to think about > > > > - The execution & reporting harness > > - Supporting infrastructure to aid writing tests > > - The set of tests themselves > > > > If I look at the TCK > > > > - The harness is essentially the standard Perl harness > > with a thin CLI wrapper, thus essentially works with > > any test emitting TAP format > > - The support infra is all custom APIs using libvirt-perl > > - The tests are mostly written in Perl, but some are > > written in shell (yuk). They all output TAP format. > > > > One key thing here is that the test harness is fairly loosely > > coupled to the support infra & tests. > > > > The TAP data format bridged the two, letting us write tests > > in essentially any language. Of course writing tests in > > non-Perl was/is tedious today, since there's no support > > infra for that which exists today. > > > > The TAP data format bridge also means we can easily throw > > away the current TCK harness and replace it with anything > > else that can consume tests emitting TAP data. > > > > > > If I look at Avocado, I think (correct me if i'm wrong) > > > > 1. The harness is essentially the standard Python harness > > with a thin CLI wrapper. Thus needs all tests to implement > > the Python test APIs > > Not really. Even though Avocado is mostly written in Python, there > have been provisions for accommodating foreign types of tests (in > different forms) since its inception. The most basic way, is, of > course, simply treating a test as an executable. But this is far from > the only way. For instance, these are other possibilities: > > a) if an executable generates TAP, it can consume the test's TAP > output and determine the test results Can you show how to actually make this work, since from the manpage I can only see how to make it emit TAP, not consume it. I see there is a 'tap' plugin but that doesn't seem to provide a runner.

Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|