4:09 p.m.
On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
If I look at Avocado, I think (correct me if i'm wrong)
1. The harness is essentially the standard Python harness
with a thin CLI wrapper. Thus needs all tests to implement
the Python test APIs
2. The support infra is all custom APIs using libvirt-python
3. The tests are to be written entirely in Python, to integrate
with the python test harness
The harness proposed here would be the Avocado itself. That doesn't just
run Python tests, it can run any executable. And also has support for
multiple test results, including TAP.
However, for developers trying to write Python tests will receive some
support infrastructure classes that will help them.
IIUC, 90% of this patch series is essentially about (2),
since (1) is fairly trivial and for (3) there's just
simple demo tests.
Yes, and (1) is trivial because I’m delegating this to Avocado.
Libvirt has consumers writing applications in a variety of
languages, and periodically reporting bugs. My general wish
for a test harness would be for something that can invoke
and consume arbitrary test cases. Essentially if someone
can provide us a piece of code that demonstrates a problem
in libvirt, I would like to be able to use that directly as
a functional test, regardless of language it was written
in.
In theory the libvirt TCK allowed for that, and indeed the
few tests written in shell essentially came to exist because
someone at IBM had written some shell code to test firewalls
and contributed that as a functional test. They just hacked
their script to emit TAP data.
For sure, I completely agree and understand. We take that into account
as well, but maybe it wasn't so clear in the RFC. With Avocado you can
do that, and there is no need for hacking the scripts to emit TAP data.
Avocado has a --tap option that will output results in TAP format:
$ avocado run --tap - /bin/true /bin/false
1..2
ok 1 /bin/true
not ok 2 /bin/false
> For now, this framework assumes that you are going to run the
tests in a
> fresh clean environment, i.e. a VM. If you decide to use your local
> system, beware that execution of the tests may affect your system.
It is always good to be wary of functional test frameworks trashing
your system, but at the same time I think makes things more flexible
if they are able to be reasonably safely run on /any/ host.
For the TCK we tried to be quite conservative in this regard, because
it was actually an explicit goal that you can run it on any Fedora
host to validate it correctly functioning for KVM. To achieve that
we tried to use some standard naming conventions for any resources
that tests created, and if we saw pre-existing resources named
differently we didn't touch them. ie all VMs were named 'tck-XXX',
and were free to be deleted, but other named VMs were ignored.
For anything special, such as testing PCI device assignment, the test
configuration file had to explicitly contain details of host devices
that were safe to use. This was also done for host block devices or
NICs, etc. Thus a default invokation only ran a subset of tests
which were safe. The more dangerous tests required you to modify the
config file to grant access.
I think it would be good if the Avocado supporting test APIs had a
similar conceptual approach, especially wrt to a config file
granting access to host resources such as NICs/Block devs/PCI devs,
where you need prior explicit admin permission to avoid danger.
Indeed. It is a good practice to isolate those things. Again, it was my
fault that wasn't clear on the RFC, and yes we have that in mind. As
mentioned lavocado it was inspired by libvirt-tck and I'm already using
the same approach with the prefix: when creating domains I'm prefixing
the `test.id` to the domain name (and this can be customizable).
On top of that, Avocado has the spawners concept, where we support right
now subprocess and podman. But a new spawner could be developed to
create a more isolated env.
We also have the Job API feature, where, using python code, you can
create custom job suites based on your logic.
> One of the future goals of this framework is to utilize nested
> virtualization technologies and hence make sure an L1 guest is
> provisioned automatically for the tests to be executed in this
> environment and not tamper with your main system.
I think the test framework should not concern itself with this
kind of thing directly. We already have libvirt-ci, which has
tools for provisioning VMs with pre-defined package sets. The
test harness should just expect that this has been done, and
that it is already running in such an environment if the user
wanted it to be.
Sure, again I wasn't clear. We intend to delegate this to tools like
lcitool.
In the TCK config file we provided setting for a URI to connect
to other hosts, to enable multi-hosts tests like live migration
to be done.
I notice that, and I did the same here on LIBVIRT_URI config option. By
default it is pointing to "qemu:///system". But for sure, could be
customized. It is on my plans to extend this to support > 1 URI to
support live migration.
> I'm adding more information with some details inside the
README file.
Overall, I'm more enthusiastic about writing tests in Python
than Perl, for the long term, but would also potentially like
to write tests in Go too.
I'm wondering if we can't bridge the divide between what we
have already in libvirt-tck, and what you're bringing to the
table with avocado here. While we've not done much development
with the TCK recently, there are some very valuable tests
there, especially related to firewall support and I don't
fancy rewriting them.
Thus my suggestion is that we:
- Put this avocado code into the libvirt-tck repository,
with focus on the supporting infra for making it easy to
write Python tests
- Declare that all tests need a way to emit TAP format,
no matter what language they're written in. This could
either be the test directly emitting TAP, or it could
be via use of a plugin. For example 'tappy' can make
existing Python tests emit TAP, with no modifications
to the tests themselves.
https://tappy.readthedocs.io/en/latest/consumers.html
IOW, you can still invoke the python tests using the
standard Python test runner, and still invoke the perl
tests using the stnadard Perl test runner if desired.
This is supported already:
$ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py
1..3
ok 1 tests/domain/transient.py:TransientDomain.test_autostart
ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle
ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent
Thank you for your comments and review here.
--
Beraldo