Re: [libvirt PATCH 0/4] RFC: tests: introduce lavocado

Hi Daniel, On Thu, Jul 1, 2021 at 2:05 PM Daniel P. Berrangé <berrange(a)redhat.com&gt; wrote: > > On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote: > > lavocado aims to be an alternative test framework for the libvirt > > project using Python, python-libvirt and Avocado. This can be used to > > write unit, functional and integration tests and it is inspired by the > > libvirt-tck framework. > > > > This series introduces the basic framework along with some basic test > > examples that I got from libvirt-tck. I would appreciate your comments > > on this RFC, to see if this fits this project's needs. Also, names and > > locations are just a proposal and can be changed. > > Some high level thoughts > > - More extensive functional integration testing coverage is good > > - We need to actually run the functional tests regularly reporting > via GitLab pipelines in some way > > - Using Python is way more contributor friendly than Perl > > - This does not need to live in libvirt.git as we don't follow > a monolithic repo approach in libvirt, and it already depends > on functionality provided by other repos. > > > When it comes to testing, I feel like there are several distinct > pieces to think about > > - The execution & reporting harness > - Supporting infrastructure to aid writing tests > - The set of tests themselves > > If I look at the TCK > > - The harness is essentially the standard Perl harness > with a thin CLI wrapper, thus essentially works with > any test emitting TAP format > - The support infra is all custom APIs using libvirt-perl > - The tests are mostly written in Perl, but some are > written in shell (yuk). They all output TAP format. > > One key thing here is that the test harness is fairly loosely > coupled to the support infra & tests. > > The TAP data format bridged the two, letting us write tests > in essentially any language. Of course writing tests in > non-Perl was/is tedious today, since there's no support > infra for that which exists today. > > The TAP data format bridge also means we can easily throw > away the current TCK harness and replace it with anything > else that can consume tests emitting TAP data. > > > If I look at Avocado, I think (correct me if i'm wrong) > > 1. The harness is essentially the standard Python harness > with a thin CLI wrapper. Thus needs all tests to implement > the Python test APIs Not really. Even though Avocado is mostly written in Python, there have been provisions for accommodating foreign types of tests (in different forms) since its inception. The most basic way, is, of course, simply treating a test as an executable. But this is far from the only way. For instance, these are other possibilities: a) if an executable generates TAP, it can consume the test's TAP output and determine the test results