6:23 a.m.
Hi Daniel,
On Thu, Jul 1, 2021 at 2:05 PM Daniel P. Berrangé <berrange(a)redhat.com> wrote:
On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
> lavocado aims to be an alternative test framework for the libvirt
> project using Python, python-libvirt and Avocado. This can be used to
> write unit, functional and integration tests and it is inspired by the
> libvirt-tck framework.
>
> This series introduces the basic framework along with some basic test
> examples that I got from libvirt-tck. I would appreciate your comments
> on this RFC, to see if this fits this project's needs. Also, names and
> locations are just a proposal and can be changed.
Some high level thoughts
- More extensive functional integration testing coverage is good
- We need to actually run the functional tests regularly reporting
via GitLab pipelines in some way
- Using Python is way more contributor friendly than Perl
- This does not need to live in libvirt.git as we don't follow
a monolithic repo approach in libvirt, and it already depends
on functionality provided by other repos.
When it comes to testing, I feel like there are several distinct
pieces to think about
- The execution & reporting harness
- Supporting infrastructure to aid writing tests
- The set of tests themselves
If I look at the TCK
- The harness is essentially the standard Perl harness
with a thin CLI wrapper, thus essentially works with
any test emitting TAP format
- The support infra is all custom APIs using libvirt-perl
- The tests are mostly written in Perl, but some are
written in shell (yuk). They all output TAP format.
One key thing here is that the test harness is fairly loosely
coupled to the support infra & tests.
The TAP data format bridged the two, letting us write tests
in essentially any language. Of course writing tests in
non-Perl was/is tedious today, since there's no support
infra for that which exists today.
The TAP data format bridge also means we can easily throw
away the current TCK harness and replace it with anything
else that can consume tests emitting TAP data.
If I look at Avocado, I think (correct me if i'm wrong)
1. The harness is essentially the standard Python harness
with a thin CLI wrapper. Thus needs all tests to implement
the Python test APIs
Not really. Even though Avocado is mostly written in Python, there
have been provisions for accommodating foreign types of tests (in
different forms) since its inception. The most basic way, is, of
course, simply treating a test as an executable. But this is far from
the only way. For instance, these are other possibilities:
a) if an executable generates TAP, it can consume the test's TAP
output and determine the test results
b) you can write support for completely new test types[1]. If you
want your test to be "discoverable" when running `avocado list ...`
you will have to write some Python code. But you can skip that part,
and have an executable named "avocado-runner-$test-type" written in
any language whatsoever, which will be used to execute whatever your
tests are and feed the information to Avocado. BTW, we already have a
plugin that can list go tests[2] and run them (via a Python wrapper
that uses `go test`).
2. The support infra is all custom APIs using libvirt-python
3. The tests are to be written entirely in Python, to integrate
with the python test harness
IIUC, 90% of this patch series is essentially about (2),
since (1) is fairly trivial and for (3) there's just
simple demo tests.
Libvirt has consumers writing applications in a variety of
languages, and periodically reporting bugs. My general wish
for a test harness would be for something that can invoke
and consume arbitrary test cases. Essentially if someone
can provide us a piece of code that demonstrates a problem
in libvirt, I would like to be able to use that directly as
a functional test, regardless of language it was written
in.
In theory the libvirt TCK allowed for that, and indeed the
few tests written in shell essentially came to exist because
someone at IBM had written some shell code to test firewalls
and contributed that as a functional test. They just hacked
their script to emit TAP data.
I agree there's no reason to *not* continue to allow that. I'd even
argue that Avocado can give libvirt even more options wrt integrating
different types of tests.
> For now, this framework assumes that you are going to run the tests in a
> fresh clean environment, i.e. a VM. If you decide to use your local
> system, beware that execution of the tests may affect your system.
It is always good to be wary of functional test frameworks trashing
your system, but at the same time I think makes things more flexible
if they are able to be reasonably safely run on /any/ host.
I'm on the same boat here. All the QEMU/Libvirt testing
infrastructure that I was involved with, evolved from something that
needed to run as privileged users and could easily trash your system,
from something that is runnable as a regular user after a `pip install
--user`. There are challenges, but most situations are handled with
tests being canceled if they do not have what they need to run, and by
always assuming a "good neighbor" policy (that is, you're not alone in
the system).
Having said that, I've seen how much care has been dedicated to
libvirt-ci so that test execution environments would be predictable,
reliable and easy to obtain. I guess the original text from Beraldo
tried to make it clear that we envision an obvious/default integration
with those environments created by libvirt-ci, while not restricting
other environments or assuming every execution will be on those
environments.
For the TCK we tried to be quite conservative in this regard,
because
it was actually an explicit goal that you can run it on any Fedora
host to validate it correctly functioning for KVM. To achieve that
we tried to use some standard naming conventions for any resources
that tests created, and if we saw pre-existing resources named
differently we didn't touch them. ie all VMs were named 'tck-XXX',
and were free to be deleted, but other named VMs were ignored.
For anything special, such as testing PCI device assignment, the test
configuration file had to explicitly contain details of host devices
that were safe to use. This was also done for host block devices or
NICs, etc. Thus a default invokation only ran a subset of tests
which were safe. The more dangerous tests required you to modify the
config file to grant access.
I think it would be good if the Avocado supporting test APIs had a
similar conceptual approach, especially wrt to a config file
granting access to host resources such as NICs/Block devs/PCI devs,
where you need prior explicit admin permission to avoid danger.
Absolutely agreed.
> One of the future goals of this framework is to utilize nested
> virtualization technologies and hence make sure an L1 guest is
> provisioned automatically for the tests to be executed in this
> environment and not tamper with your main system.
I think the test framework should not concern itself with this
kind of thing directly. We already have libvirt-ci, which has
tools for provisioning VMs with pre-defined package sets. The
test harness should just expect that this has been done, and
that it is already running in such an environment if the user
wanted it to be.
Right. I know for a fact the original text was not about depending
exclusively on those VM environments, but having an obvious default
experience based on them.
In the TCK config file we provided setting for a URI to connect
to other hosts, to enable multi-hosts tests like live migration
to be done.
> I'm adding more information with some details inside the README file.
Overall, I'm more enthusiastic about writing tests in Python
than Perl, for the long term, but would also potentially like
to write tests in Go too.
I'm wondering if we can't bridge the divide between what we
have already in libvirt-tck, and what you're bringing to the
table with avocado here. While we've not done much development
with the TCK recently, there are some very valuable tests
there, especially related to firewall support and I don't
fancy rewriting them.
Thus my suggestion is that we:
- Put this avocado code into the libvirt-tck repository,
with focus on the supporting infra for making it easy to
write Python tests
One drawback would be the various supporting files for a perl+Python
package. I mean, a Makefile.PL and a setup.py in the same repo can be
a source of confusion. But, maybe I'm overreacting.
- Declare that all tests need a way to emit TAP format,
no matter what language they're written in. This could
either be the test directly emitting TAP, or it could
be via use of a plugin. For example 'tappy' can make
existing Python tests emit TAP, with no modifications
to the tests themselves.
https://tappy.readthedocs.io/en/latest/consumers.html
I honestly think Avocado can provide a better range of integration
opportunities.
IOW, you can still invoke the python tests using the
standard Python test runner, and still invoke the perl
tests using the stnadard Perl test runner if desired.
- Switch the TCK configuration file to use JSON/YAML
instead of its current custom format. This is to enable
the Python tests to share a config file with the Perl
and Shell tests. Thus they can all have the same way
to figure out what block devs, NICs, PCI devs, etc
they are permitted to use, and what dirs it is safe
to create VM images in, etc.
Makes sense.
- Replace the current Perl based execution harness
with something using Python, so that it is less
of a pain to install in terms of dependancies.
Looks like we could just do something using the
'tappy' Python APIs to consume the TAP format
from all the tests and generate a junit report
that GitLab can consume.
These are all things Avocado can do right away.
- Rename the RPMs so they match the project name
"libvirt-tck", instead of "perl-Sys-Virt-TCK"
to emphasis that this is not a Perl project, it
is a testing project, of which some parts were
historically Perl, but new parts will be mostly
Python (and possibly Go/etc in future if desired).
Potentially have RPMs split:
* libvirt-tck - just the execution harness,
currently the Perl one, but to be replaced
with a Python one.
* libvirt-tck-tests-python - supporting APIs
and tests written in Python - essentially
all of this series, and future tests written
* libvirt-tck-tests-perl - supporting APIs
and tests written in Perl - essentially
most of existing TCK stuff
* libvirt-tck-tests-shell - supporting APIs
and tests written in shell - mostly the
network/firewall related TCK pieces
Given the existance of the 'tappy' tool, it feels like the first
two points ought to be reasonably doable without any significant
changes to what you've already written. Just need to import it
as is, and then using tappy as a shim to let it invoked from
libvirt-tck. Can still have the existing way to invoke it
directly too as an alternative.
The common JSON/YAML config file, and adopting some standard
naming conventions and resource usage conventions are probably
where the real work would lie. That's likely not too bad at
this stage though, since you've not actually written a large
number of test cases.
> Beraldo Leal (4):
> tests: introduce lavocado: initial code structure
> tests.lavocado: adding basic transient domain tests
> tests.lavocado: adding a .gitignore
> tests.lavocado: adding a README and Makefile for convenience
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Cheers,
- Cleber.
[1] -
https://avocado-framework.readthedocs.io/en/89.0/guides/contributor/chapt...
[2] - https://avocado-framework.readthedocs.io/en/89.0/plugins/optional/golang....