On Thu, Sep 05, 2019 at 06:15:04PM +0100, Daniel P. Berrangé wrote:
On Thu, Sep 05, 2019 at 12:30:27PM -0400, Laine Stump wrote:
(BTW, what does the removal of perl from libvirt say about continued use of perl for libvirt-tck? There are a lot of useful tests in there that find real bugs, but they tend to languish (both in use and in enhancements) because nobody wants to do anything with perl (or with the big shell scripts that do the nwfilter testing). It would be nice if the tests were all in a language that was more accessible, but they're kind of married to the perl TAP module (and besides, who wants to spend time rewriting a bunch of test scripts when they already work?). This is mostly a moot point, because I think hardly anyone runs the libvirt-tck tests anymore, which is too bad because it has historically caught some regressions that no other testing framework did.)
I believe they are run by the Red Hat virt QE team, as I ave got bug reports against the Perl libvirt bindings, where the reproducer is a TCK script.
The TCK really covered three distinct things
- framework for controlling test execution - library modules/helper APIs to facilitate test creation - the large set of indivdual tests
There is no code binding between the tests and the framework. They communicate exclusively through the TAP protocol. In that way we can throw out the current framework and drop in a different one quite easily - the new framework merely needs to be able to consume the TAP data format. For any framework which does not natively support the TAP format, it is quite easy to write an adapter / shim that would convert to the desired data format. You might loose some finese / fine grained details in the logs/progress reporting but functionally the tests should work fine.
As a point of reference the GLib testing harness has recently switched from using their own custom output format to using the TAP output format. So if we adopt GLib in the libvirt.git and use it in our unit tests those would end up using TAP. GLib provides a shim for automake that allows it to consume the results from GLib based units tests.
QEMU's makefiles recently switched to using TAP for its unit and (some of) its functional tests too.
IOW, I'm perfectly happy to throw away the existing TCK framework impl if someone proposes a better alternative.
I came across this the other day, which looks like it could be a viable replacement for the TCK control framework: https://github.com/python-tap/tappy https://tappy.readthedocs.io/en/latest/
As for the individual TCK tests themselves, there's two distinct questions
- Should they be re-written in a different language - Should they continue to use TAP output format
If we are not going to rewrite the tests, then I think they should continue to use TAP output format. It is the easiest thing to use when writing tests in Perl.
If we are going to rewrite the tests, then the tests should ideally use the output format that best suits the framework invoking them. IOW, if the framework still uses TAP, we should continue to use TAP. If not, then don't. There is an adapter for python that can convert its native unittest output format into TAP format.
I'm pretty ambivalent on rewriting the existing tests. The language of existing tests only matters if you expect people to be modifying them. Assuming a decent support library it should be no harder to define a new test, than to extend an existing test.
IOW, if we want to write TCK tests in Python, we don't really need to rewrite existing ones. We instead need to write a Python variant of the helper APIs, so that you can trivially write new Python tests which operate in the same manner as the Perl tests.
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|