Noah Misch <noah@leadboat.com> writes:
> On Fri, Apr 21, 2017 at 10:34:58PM -0700, Andres Freund wrote:
>> ... I was wondering about adding
>> a loop that simply runs for like 30s and then quits or such, but who
>> knows.
> If the probabilistic test catches the bug even 5% of the time in typical
> configurations, the buildfarm will rapidly identify any regression. I'd
> choose a 7s test that detects the bug 5% of the time over a 30s test that
> detects it 99% of the time. (When I wrote src/bin/pgbench/t/001_pgbench.pl
> for a probabilistic bug, I sized that test to finish in 1s and catch its bug
> half the time. In its case, only two buildfarm members were able to
> demonstrate the original bug, so 5% detection would have been too low.)
30sec is kind of a big lump from a buildfarm standpoint, especially if
you mean "it runs for 30s on my honkin' fast workstation". I'm fine
with individual tests that run for ~ 1sec.
(This is top-of-mind for me right now because I've been looking around
for ways to speed up the regression tests.)
regards, tom lane
