On Tue, 2018-02-27 at 09:59 -0500, Laine Stump wrote:
So should I just drop this one, or is it still somewhat valuable to have any sort of test suite coverage for PCI controller options?
I was thinking that having this test is better than not having this test. But then I thought about what would happen if there was a regression in just a single one of these validations - the negative test would still "succeed" (i.e. "succeed in detecting a failure") because it would hit the next disallowed attribute. As a matter of fact, the test would continue to "succeed" until there was a regression in the validation of every single attribute, so that no error would be triggered. So having a negative test that has multiple examples of failures actually gives us a false sense of security - we believe that it's verifying we're catching incorrect config, but it won't actually notify us until *all* of the bad config is missed by validation.
So, I think each negative test should have exactly one piece of incorrect data. That necessarily means that it's only testing for proper validation of a single aspect of a single attribute. But making this generally useful with the current test apparatus would mean a huge explosion in the number of test files, and I don't think that's practical. But if we're only testing for one out of a thousand validations, there's really not much point in it.
Okay, I'll drop it and respin then. -- Andrea Bolognani / Red Hat / Virtualization