Re: [PATCH] Increase timeout for tests and syntax-check

On Fri, Jan 29, 2021 at 09:30:24AM -0300, Daniel Henrique Barboza wrote: > > > On 1/27/21 2:59 PM, Michal Privoznik wrote: >> Since we've switched to meson our tests run with a timeout (meson >> uses 30 seconds as the default). However, not every machine that >> builds libvirt is fast enough to run every test under 30 seconds >> (each test binary has its own timeout, but still). For instance >> when building a package for distro on a farm that's under load. >> Or on a generally slow ARM hardware. While each developer can >> tune their command line for building by adding >> --timeout-multiplier=10, this is hard to do for aforementioned >> build farms. >> >> It's time to admit that not everybody has the latest, top shelf >> CPU and increase the timeout. >> >> Signed-off-by: Michal Privoznik <mprivozn(a)redhat.com&gt; >> --- > > This sure will help these build farms environments, but what about the cases > where an actual timeout means that there is something wrong with the code? > E.g. commits 46d88d8dba56 and 2ba0b7497ce7 were only possible because I was > seeing tests timing out in Power hosts when the 30 sec timeout was being > enforced. > > A 120 second default timeout for the majority of the test cases is a long time. > virschematest in this laptop I use takes 2.5 sec to complete. If I do something > wrong in the code and now the test is now 4 times slower (10 sec) I will not be > able to detect it (I'll need to start keeping track or something). You'll have to > run the test suit on your RasPi 2B to see that something went wrong because the > timeout is better tuned to your RasPI than this laptop, but then the code is already > upstream. > > And the tests will get more complex and will naturally take longer to complete. > Eventually this timeout might no be enough. Increase the timeout again? > > Meson 0.57 has support for disabling test timeout entirely with --timeout-multiplier=0, > disabling test timeout entirely. Can't we bump the meson version to 0.57 and > then tell the distros to use timeout-multiplier=0? That will solve the problems > for them, I keep the short timeouts for development, and we won't need to keep > bumping the test timeout once every 2 years or something because the s390x > TCG enviroment in Fedora COPR is timing out again. Agreed. The timeout is useful and we can always find a machine where the current timeout will not be good enough. I don't think it is a good idea to increase the timeout like this for only few specific use-cases.