Peter Eisentraut wrote:
>Andrew Dunstan writes:
>
>
>
>>"Useful" is probably subjective. That list would at least be a good
>>place to start, though. What combinations of variables do you think we
>>would need?
>>
>>
>
>First of all, I don't necessarily think that a large list of CPU/operation
>system combinations is going to help much. IIRC, this round of platform
>testing showed us two real problems, and both happened because the
>operating system version in question came out the previous day, so we
>could not have caught it. Much more problems arise when people use
>different versions of secondary packages, such as Tcl, Perl, Kerberos,
>Flex, Bison. So you would need to compile a large collection of these
>things. The problem again is that it is usually the brand-new or the odd
>intermediate version of such a tool that breaks things, so a "set and
>forget" build farm is not going to catch it. Another real source of
>problems are real systems. Weird combinations of packages, weird network
>setups, weird applications, custom kernels. These cannot be detected on
>out of the box setups. In fact, the regression tests might not detect
>them at all.
>
>Hence the open-source community approach. Closed-source development teams
>can do all the above, with great effort. But by throwing out the code and
>have real people test them on real systems with real applications, you can
>do much better.
>
>
>
The fact that something doesn't find everything doesn't mean it is of no
value. (Thinks of Scott Adams' nice example: "Your theory of gravity
doesn't prove why there are no unicorns, so it is wrong." ;-) )
I don't believe there is a single "open source community" approach -
open source projects all have differing ways of handling problems. At
least 2 very significant open source projects I know of run build farms,
notwithstanding that your objections should apply equally to them.
Mozilla's is fairly centralised and very complex and heavy, but gives
fairly immediate feedback if anything gets broken. Samba's is much
lighter, distributed, and they still apparently see good value in it.
(Samba uses a "torture test" - perhaps we need one of those in addition
to the regression tests.)
Maybe it wouldn't be of great value to PostgreSQL. And maybe it would. I
have an open mind about it. I don't think incompleteness is an argument
against it, though.
cheers
andrew
