On Fri, 2018-02-23 at 14:18 -0500, Laine Stump wrote:
On 02/21/2018 09:14 AM, Andrea Bolognani wrote:
The input configurations set all existing options for all PCI controllers, even those that are not valid for the controller. As we implement validation for PCI controller options, we expect these test to start failing.
A noble cause, but since multiple options are being tested for multiple controllers in the same file, once you have all the proper checks in place the tests won't actually be verifying all of the negative tests - only the first failure will be noticed - if one of the others is missed, it won't "fail extra hard" or anything.
I'm well aware of that.
Although I hate to explode the number of tests, I think if you want to have proper negative testing for every options that doesn't belong on a particular controller, then you'll need a separate test case for each combination of option and controller model. And since that would make for a *lot* of test cases if we tried to extrapolate to all other options for all other elements, I don't know that it's worth going down that rabbit hole.
So should I just drop this one, or is it still somewhat valuable to have any sort of test suite coverage for PCI controller options? -- Andrea Bolognani / Red Hat / Virtualization