Re: Removal of fake-capabilities test from qemuxml2argvtest
On Fri, Apr 09, 2021 at 17:52:59 +0100, Daniel P. Berrangé wrote:
On Fri, Apr 09, 2021 at 05:14:41PM +0200, Peter Krempa wrote:
> Hi,
>
> recently I've got very annoyed that we still have a very large amount of
> tests in qemuxml2argvtest which use DO_TEST or some other fake-caps
> test.
>
> While I can see value of fake-caps test for negative cases (but I'd
> prefer actually real-caps with capability masking) I don't think there's
> much value in keeping the DO_TEST fake-caps stuff around.
>
> Namely in most cases it doesn't test anything useful which could be
> encountered in real world.
>
> I propose that we convert all DO_TEST cases to:
>
> - DO_TEST_CAPS_LATEST
> - and possibly a amount of version-bound tests based on:
> - oldest supported version?
> - versions in popular distros?
> - none?
> - any ideas?
[...]
Using DO_TEST_CAPS_LATEST gets that coverage, but at a quite
large cost. Basically it is a big hammer approach that lets us
ignore what tests are actually needed to get minimal desired
coverage, and instead just test everything.
Your (snipped) reply sumarizes the "good practices" very well and I
think that in most cases we are already implementing them properly.
Also in cases when code is being refactored I see mostly that we already
do everything described here.
One thing that is clearly bad with plain DO_TEST() is that
due to the individual listed capability flags, we end up
generating an frankenstein argv that has no relation to a
real world QEMU version.
My main question wasn't though how to do it properly because I think
that has been already established at least implicitly without it being
really recorded anywhere.
My original question was what to do with the rest of DO_TEST and that
ideally with the least amount of manual hacking and thinking necessary,
because franky I'm not going to go through all the tests and attempt to
see whether it still makes sense.
I think that in most cases the fake-caps DO_TEST cases don't do much
useful testing and generally we'd make them slightly more useful if we'd
just convert DO_TEST to DO_TEST_CAPS_LATEST.
I'm more curious whether there are any suggestions which could be used
to determine when other than DO_TEST_CAPS_LATEST might make sense.
Again, without going manually through every case.