On Tue, May 07, 2024 at 14:58:55 +0100, Daniel P. Berrangé wrote:
I was annoyed that
meson build --auto-features=disabled -Dtests=enabled
will fail in unit tests. Rather than just fix that scenario, I went
down the rabbit hole and tested (almost) every single minimal build
option.
By that I mean I looked at meson_options.txt and got a list of every
option that has 'auto' as its default. I then attempted a minimal
build with only that enabled. I quickly found some options have
mandatory pre-requisite options, so I re-did the tests with such
deps present.
This uncovered many scenarios where we didn't use the right conditions
for tests. Surprisingly (at first, but not in retrospect) it also
found many places where the tests failed to depend on the earlier
build artifacts.
My test process was:
opts=$(grep auto meson_options.txt | sed -e 's/option(//' -e 's/,.*//'
-e "s/'//g")
for opt in $opts
do
deps=`grep --before 1 "option('$opt'" meson_options.txt | grep dep:
| head -1 | sed -e 's/# //' -e 's/dep:/-D/g' -e 's/$/=enabled/' -e
's/ -D/=enabled -D/g'`
echo "Try test $opt with $deps"
rm -rf build
meson setup build --auto-features=disabled -Dtests=enabled $deps -D$opt=enabled 1>
opt-$opt.log 2>&1
meson test -C build --no-suite syntax-check --print-errorlogs 1>> opt-$opt.log
2>&1 ; done
done
Then to check results
grep '^Fail:' opt-*.log | grep -v 0
This is slow, but not as slow as you might think, because each
build is very minimal, so we're not actually building more than
300-400 files each time, rather than 1000's, and we're not running
as many tests either.
Might be interesting to get extensive test into CI.
Strictly speaking we care about the combinatorial expansion of
meson_options.txt, but that would be insanely slow and is likely
to be overkill.
Whoah, that's rather cool. But as you note I don't think this is
anything for CI and even if, this'd be something to run once a month
perhaps. Otherwise it's almost as useful as mining cryptocurrencies
With the few things I've pointed out inline:
Series:
Reviewed-by: Peter Krempa <pkrempa(a)redhat.com>