On 1/27/21 2:59 PM, Michal Privoznik wrote: > Since we've switched to meson our tests run with a timeout (meson > uses 30 seconds as the default). However, not every machine that > builds libvirt is fast enough to run every test under 30 seconds > (each test binary has its own timeout, but still). For instance > when building a package for distro on a farm that's under load. > Or on a generally slow ARM hardware. While each developer can > tune their command line for building by adding > --timeout-multiplier=10, this is hard to do for aforementioned > build farms. > > It's time to admit that not everybody has the latest, top shelf > CPU and increase the timeout. > > Signed-off-by: Michal Privoznik <mprivozn(a)redhat.com&gt; > --- This sure will help these build farms environments, but what about the cases where an actual timeout means that there is something wrong with the code? E.g. commits 46d88d8dba56 and 2ba0b7497ce7 were only possible because I was seeing tests timing out in Power hosts when the 30 sec timeout was being enforced. A 120 second default timeout for the majority of the test cases is a long time. virschematest in this laptop I use takes 2.5 sec to complete. If I do something wrong in the code and now the test is now 4 times slower (10 sec) I will not be able to detect it (I'll need to start keeping track or something). You'll have to run the test suit on your RasPi 2B to see that something went wrong because the timeout is better tuned to your RasPI than this laptop, but then the code is already upstream. And the tests will get more complex and will naturally take longer to complete. Eventually this timeout might no be enough. Increase the timeout again? Meson 0.57 has support for disabling test timeout entirely with --timeout-multiplier=0, disabling test timeout entirely. Can't we bump the meson version to 0.57 and then tell the distros to use timeout-multiplier=0? That will solve the problems for them, I keep the short timeouts for development, and we won't need to keep bumping the test timeout once every 2 years or something because the s390x TCG enviroment in Fedora COPR is timing out again.

On 1/29/21 1:45 PM, Pavel Hrdina wrote: > On Fri, Jan 29, 2021 at 09:30:24AM -0300, Daniel Henrique Barboza wrote: > > > > > > On 1/27/21 2:59 PM, Michal Privoznik wrote: > > > Since we've switched to meson our tests run with a timeout (meson > > > uses 30 seconds as the default). However, not every machine that > > > builds libvirt is fast enough to run every test under 30 seconds > > > (each test binary has its own timeout, but still). For instance > > > when building a package for distro on a farm that's under load. > > > Or on a generally slow ARM hardware. While each developer can > > > tune their command line for building by adding > > > --timeout-multiplier=10, this is hard to do for aforementioned > > > build farms. > > > > > > It's time to admit that not everybody has the latest, top shelf > > > CPU and increase the timeout. > > > > > > Signed-off-by: Michal Privoznik <mprivozn(a)redhat.com&gt; > > > --- > > > > This sure will help these build farms environments, but what about the cases > > where an actual timeout means that there is something wrong with the code? > > E.g. commits 46d88d8dba56 and 2ba0b7497ce7 were only possible because I was > > seeing tests timing out in Power hosts when the 30 sec timeout was being > > enforced. > > > > A 120 second default timeout for the majority of the test cases is a long time. > > virschematest in this laptop I use takes 2.5 sec to complete. If I do something > > wrong in the code and now the test is now 4 times slower (10 sec) I will not be > > able to detect it (I'll need to start keeping track or something). You'll have to > > run the test suit on your RasPi 2B to see that something went wrong because the > > timeout is better tuned to your RasPI than this laptop, but then the code is already > > upstream. > > > > And the tests will get more complex and will naturally take longer to complete. > > Eventually this timeout might no be enough. Increase the timeout again? > > > > Meson 0.57 has support for disabling test timeout entirely with --timeout-multiplier=0, > > disabling test timeout entirely. Can't we bump the meson version to 0.57 and > > then tell the distros to use timeout-multiplier=0? That will solve the problems > > for them, I keep the short timeouts for development, and we won't need to keep > > bumping the test timeout once every 2 years or something because the s390x > > TCG enviroment in Fedora COPR is timing out again. > > Agreed. The timeout is useful and we can always find a machine where the > current timeout will not be good enough. I don't think it is a good idea > to increase the timeout like this for only few specific use-cases. So we agree that timeouts are evil :-) Back in the autotools days we did not

use any timeout at all and I don't remember that causing any trouble. But maybe it was so painful that I blocked it in my mind :-) Michal

On 1/27/21 2:59 PM, Michal Privoznik wrote: > Since we've switched to meson our tests run with a timeout (meson > uses 30 seconds as the default). However, not every machine that > builds libvirt is fast enough to run every test under 30 seconds > (each test binary has its own timeout, but still). For instance > when building a package for distro on a farm that's under load. > Or on a generally slow ARM hardware. While each developer can > tune their command line for building by adding > --timeout-multiplier=10, this is hard to do for aforementioned > build farms. > > It's time to admit that not everybody has the latest, top shelf > CPU and increase the timeout. > > Signed-off-by: Michal Privoznik <mprivozn(a)redhat.com&gt; > --- This sure will help these build farms environments, but what about the cases where an actual timeout means that there is something wrong with the code? E.g. commits 46d88d8dba56 and 2ba0b7497ce7 were only possible because I was seeing tests timing out in Power hosts when the 30 sec timeout was being enforced. A 120 second default timeout for the majority of the test cases is a long time. virschematest in this laptop I use takes 2.5 sec to complete. If I do something wrong in the code and now the test is now 4 times slower (10 sec) I will not be able to detect it (I'll need to start keeping track or something). You'll have to run the test suit on your RasPi 2B to see that something went wrong because the timeout is better tuned to your RasPI than this laptop, but then the code is already upstream. And the tests will get more complex and will naturally take longer to complete. Eventually this timeout might no be enough. Increase the timeout again? Meson 0.57 has support for disabling test timeout entirely with --timeout-multiplier=0, disabling test timeout entirely. Can't we bump the meson version to 0.57 and then tell the distros to use timeout-multiplier=0? That will solve the problems for them, I keep the short timeouts for development, and we won't need to keep bumping the test timeout once every 2 years or something because the s390x TCG enviroment in Fedora COPR is timing out again.

On 1/29/21 1:30 PM, Daniel Henrique Barboza wrote: > > > On 1/27/21 2:59 PM, Michal Privoznik wrote: >> Since we've switched to meson our tests run with a timeout (meson >> uses 30 seconds as the default). However, not every machine that >> builds libvirt is fast enough to run every test under 30 seconds >> (each test binary has its own timeout, but still). For instance >> when building a package for distro on a farm that's under load. >> Or on a generally slow ARM hardware. While each developer can >> tune their command line for building by adding >> --timeout-multiplier=10, this is hard to do for aforementioned >> build farms. >> >> It's time to admit that not everybody has the latest, top shelf >> CPU and increase the timeout. >> >> Signed-off-by: Michal Privoznik <mprivozn(a)redhat.com&gt; >> --- > > This sure will help these build farms environments, but what about the cases > where an actual timeout means that there is something wrong with the code? > E.g. commits 46d88d8dba56 and 2ba0b7497ce7 were only possible because I was > seeing tests timing out in Power hosts when the 30 sec timeout was being > enforced. > > A 120 second default timeout for the majority of the test cases is a long time. > virschematest in this laptop I use takes 2.5 sec to complete. If I do something > wrong in the code and now the test is now 4 times slower (10 sec) I will not be > able to detect it (I'll need to start keeping track or something). With 30 second timeout you won't detect that either. Using timeout as an indicator of test failure is wrong IMO. And if I were not lazy and fixed 'check-access' test suite then we would see instantly what tests are accessing paths in the host (=> depend on host configuration).

> You'll have to > run the test suit on your RasPi 2B to see that something went wrong because the > timeout is better tuned to your RasPI than this laptop, but then the code is already > upstream. So should we make timeouts shorter then? Why is 30 seconds sweet spot? > > And the tests will get more complex and will naturally take longer to complete. > Eventually this timeout might no be enough. Increase the timeout again? Sure, why not? We adapt to newer gcc/clang/$whatever, why not timeout? Michal

On Wed, Jan 27, 2021 at 06:59:58PM +0100, Michal Privoznik wrote: > Since we've switched to meson our tests run with a timeout (meson > uses 30 seconds as the default). However, not every machine that > builds libvirt is fast enough to run every test under 30 seconds > (each test binary has its own timeout, but still). For instance > when building a package for distro on a farm that's under load. > Or on a generally slow ARM hardware. While each developer can > tune their command line for building by adding > --timeout-multiplier=10, this is hard to do for aforementioned > build farms. I don't get why it is hard for build farms. Someone, somwhere is writing the script that invokes meson & ninja with some args. Why is it hard to add --timeout-multiplier=10 too ? > It's time to admit that not everybody has the latest, top shelf > CPU and increase the timeout. I'm not convinced we want to optimize for the slowest hardware we can find, especially when there's an easy option of setting --timeout-multiplier=10.

On Fri, Jan 29, 2021 at 03:46:18PM +0100, Andrea Bolognani wrote: > On Fri, 2021-01-29 at 12:48 +0000, Daniel P. Berrangé wrote: >> On Wed, Jan 27, 2021 at 06:59:58PM +0100, Michal Privoznik wrote: >>> Since we've switched to meson our tests run with a timeout (meson >>> uses 30 seconds as the default). However, not every machine that >>> builds libvirt is fast enough to run every test under 30 seconds >>> (each test binary has its own timeout, but still). For instance >>> when building a package for distro on a farm that's under load. >>> Or on a generally slow ARM hardware. While each developer can >>> tune their command line for building by adding >>> --timeout-multiplier=10, this is hard to do for aforementioned >>> build farms. >> >> I don't get why it is hard for build farms. Someone, somwhere >> is writing the script that invokes meson & ninja with some >> args. Why is it hard to add --timeout-multiplier=10 too ? >> >>> It's time to admit that not everybody has the latest, top shelf >>> CPU and increase the timeout. >> >> I'm not convinced we want to optimize for the slowest hardware >> we can find, especially when there's an easy option of setting >> --timeout-multiplier=10. > > It's not complicated to add the option, but the fact that Debian, > SUSE and now Fedora all need to specify a timeout multiplier hints to > the fact that perhaps the default timeout is just too small.

AFAIK, Fedora hasn't set any timeout multiplier in our builds.

On Fri, Jan 29, 2021 at 03:46:18PM +0100, Andrea Bolognani wrote: > On Fri, 2021-01-29 at 12:48 +0000, Daniel P. Berrangé wrote: > > I'm not convinced we want to optimize for the slowest hardware > > we can find, especially when there's an easy option of setting > > --timeout-multiplier=10. > > It's not complicated to add the option, but the fact that Debian, > SUSE and now Fedora all need to specify a timeout multiplier hints to > the fact that perhaps the default timeout is just too small. AFAIK, Fedora hasn't set any timeout multiplier in our builds.

On Fri, Jan 29, 2021 at 04:20:53PM +0100, Andrea Bolognani wrote: > the timeout multiplier has been added to the upstream spec file, > which the Fedora spec file is based on, so while you're correct that > Fedora is not yet using it, it's fair to assume it will soon make its > way there for the benefit of e.g. the virt-preview COPR. I think this is the wrong solution. IMHO RPM should be making th %meson_test macro include the timeout arg automatically when running on a emulated environment. This would fix the problem for all users of meson, without causing unecessarily long timeouts for native builds.