On Tue, May 07, 2024 at 14:58:55 +0100, Daniel P. Berrangé wrote:
I was annoyed that
meson build --auto-features=disabled -Dtests=enabled
will fail in unit tests. Rather than just fix that scenario, I went down the rabbit hole and tested (almost) every single minimal build option.
By that I mean I looked at meson_options.txt and got a list of every option that has 'auto' as its default. I then attempted a minimal build with only that enabled. I quickly found some options have mandatory pre-requisite options, so I re-did the tests with such deps present.
This uncovered many scenarios where we didn't use the right conditions for tests. Surprisingly (at first, but not in retrospect) it also found many places where the tests failed to depend on the earlier build artifacts.
My test process was:
opts=$(grep auto meson_options.txt | sed -e 's/option(//' -e 's/,.*//' -e "s/'//g")
for opt in $opts do deps=`grep --before 1 "option('$opt'" meson_options.txt | grep dep: | head -1 | sed -e 's/# //' -e 's/dep:/-D/g' -e 's/$/=enabled/' -e 's/ -D/=enabled -D/g'` echo "Try test $opt with $deps" rm -rf build meson setup build --auto-features=disabled -Dtests=enabled $deps -D$opt=enabled 1> opt-$opt.log 2>&1 meson test -C build --no-suite syntax-check --print-errorlogs 1>> opt-$opt.log 2>&1 ; done done
Then to check results
grep '^Fail:' opt-*.log | grep -v 0
This is slow, but not as slow as you might think, because each build is very minimal, so we're not actually building more than 300-400 files each time, rather than 1000's, and we're not running as many tests either.
Might be interesting to get extensive test into CI.
Strictly speaking we care about the combinatorial expansion of meson_options.txt, but that would be insanely slow and is likely to be overkill.
Whoah, that's rather cool. But as you note I don't think this is anything for CI and even if, this'd be something to run once a month perhaps. Otherwise it's almost as useful as mining cryptocurrencies With the few things I've pointed out inline: Series: Reviewed-by: Peter Krempa <pkrempa@redhat.com>