Re: [PATCH] tests: Record negative tests as pass instead of expected-fail
On Mon, 2025-02-10 at 12:44 +0100, Peter Krempa wrote:
On Mon, Feb 10, 2025 at 11:30:24 +0100, Peter Krempa wrote:
> On Mon, Feb 10, 2025 at 11:25:55 +0100, Tim Wiederhake wrote:
> > meson's "test()" function provides a "should_fail:
bool" argument
> > that
> > checks for a command to exit with a non-zero exit code instead of
> > the usual
> > zero exit code to signal success. If the program under test does
> > so, it is
> > recorded as "EXPECTEDFAIL" instead of "OK". While there is
an
> > argument to be
> > made that the program under test failed as expected, the test in
> > itself is
> > successful and should be recorded as such.
>
> Could you please elaborate what the problem is? The wording of the
I've re-read this and it ineed seems that just the wording is the
issue.
I'm not sold that we need a script to bypass the return value just
for
meson to say "OK".
Expected failures or "xfail" results were used as a tool to write a
test for a bug before committing the fix. You'd write the test first,
mark it as "xfail" to make the issue visible without making the entire
test suite fail. The commit that fixes the issue would then remove the
"xfail" marker and demonstrate that it actually fixes the issue, i.e.
makes the test pass.
Using expected-fail is not the right tool to handle a test case that is
always expected to exit with a non-zero exit code. As I said, the test
is "program exits non-zero", and that test passes. There is no failure
in the test, expected or otherwise.
Tim