On Tue, Jul 28, 2020 at 04:19:56PM +0200, Peter Krempa wrote:
On Tue, Jul 28, 2020 at 15:08:11 +0100, Daniel Berrange wrote:
On Tue, Jul 28, 2020 at 04:04:39PM +0200, Pavel Hrdina wrote:
On Tue, Jul 28, 2020 at 03:48:29PM +0200, Peter Krempa wrote:
On Thu, Jul 16, 2020 at 11:59:38 +0200, Pavel Hrdina wrote:
Signed-off-by: Pavel Hrdina <phrdina@redhat.com> --- build-aux/Makefile.in | 9 +++ .../Makefile.nonreentrant | 0 build-aux/meson.build | 30 +++++++++ build-aux/syntax-check.mk | 62 +++++++++---------- meson.build | 2 + 5 files changed, 72 insertions(+), 31 deletions(-) create mode 100644 build-aux/Makefile.in rename Makefile.nonreentrant => build-aux/Makefile.nonreentrant (100%) create mode 100644 build-aux/meson.build
[...]
+make_prog = find_program('make') + +# There is no way how to pass value to -j using run_target so let's use +# it without any value to run all tests in parallel. +run_target( + 'syntax-check', + command: [ + make_prog, '-C', meson.current_build_dir(), '-j', 'syntax-check', + ],
While I do run syntax check with unlimited '-j'. I don't think it's entirely cool to impose that on everybody. Specifically overcommiting the system is not cool. Since meson is automagically scaling can't we use the meson-detected cpu number here?
Unfortunately no, that was the first thing I was trying to figure out by going through meson code as well. It's not ideal I know.
Other options are to not use -j at all which is no-go or we can add some code to detect the available number of CPUs. But again it would not reflect the fact if user runs 'ninja -j N'.
In libosinfo we put "syntax-check" as part of the unit tests, rather than as a separate meson target. With that you don't need to pass -j to syntax-check, because other unit tests are running in parallel already, and chances are syntax-check will finish first even when serialized.
Unfortunately it's not even close.
Serialized syntax-check: real 0m22.139s user 0m24.209s sys 0m6.788s
test suite: real 0m4.833s user 0m12.408s sys 0m3.918s
syntax-check with -j == ncpus: (24 thread box)
real 0m2.099s user 0m28.558s sys 0m7.739s
As said, I'm a big fan of -jncpus or -j. so I really want to keep it especially given the data above, but on the other hand I don't want to set the CI boxes on fire.
So I was trying to figure out what to do with our syntax-check and this could be one solution: rc = run_command( 'sed', '-n', 's/^\\(sc_[a-zA-Z0-9_-]*\\):.*/\\1/p', meson.current_source_dir() / 'syntax-check.mk', check: true, ) sc_tests = rc.stdout().strip().split() This is how syntax-check.mk gets the list of targets to run for syntax-check target. We can use the same list to define tests like this: foreach target : sc_tests rc = run_command( python3_prog, '-c', 'print("@0@".replace("sc_", ""))'.format(target), check: true, env: runutf8, ) name = rc.stdout().strip() test( name, make_prog, args: [ '-C', meson.current_build_dir(), target ], depends: [ potfiles_dep, ], suite: 'syntax-check', ) endforeach It could be simplified if we don't care that all the syntax-check tests would have 'sc_' prefix. To use it with meson/ninja would be possible with the following commands: ninja test - will run all tests regardless of the test suite, it is not possible to specify test suite meson test - same as ninja test meson test --no-suite syntax-check - this would be equivalent to make check meson test --suite syntax-check - this would be equivalent to make syntax-check In addition we can add `suite: 'check'` to the remaining test() calls to make `meson test --suite check` available as well. With this change running `meson test --suite syntax-check` takes on my machine 3.037s compared to `ninja syntax-check` which takes 2.256s. Pavel