Andrew Dunstan <andrew@dunslane.net> writes:
> Tom Lane wrote:
>> But we've already had a couple of cases of interesting failures going
>> unnoticed because of the noise level. Between duplicate reports about
>> busted patches and transient problems on particular build machines
>> (out of disk space, misconfiguration, etc) it's pretty hard to not miss
>> the once-in-a-while failures. Is there some other way we could attack
>> that problem?
> The real issue is the one you identify of stuff getting lost in the
> noise. But I'm not sure there's any realistic cure for that.
Maybe we should think about filtering the noise. Like, say, discarding
every report from mongoose that involves an icc core dump ...
http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=mongoose&dt=2007-03-20%2006:30:01
That's only semi-serious, but I do think that it's getting harder to
pluck the wheat from the chaff. My investigations over the weekend
showed that we have got basically three categories of reports:
1. genuine code breakage from unportable patches: normally multiple
reports over a short period until we fix or revert the cause.
2. failures on a single buildfarm member due to misconfiguration,
hardware flakiness, etc. These are sometimes repeatable and sometimes
not.
3. all the rest, of which some fraction represents bugs we need to fix,
only we don't know they're there.
In category 1 the buildfarm certainly pays for itself, but we'd hoped
that it would help us spot less-reproducible errors too. The problem
I'm seeing is that category 2 is overwhelming our ability to recognize
patterns within category 3. How can we dial down the noise level?
regards, tom lane
