Let me add my two cents ..
I think something like PostgreSQL needs two test suites - an
acceptance test (to ensure that checkins don't break functionality)
and a regression test suite.
What we call the regression test suite is really an acceptance
test. Tom Lane is absolutely right in asserting that a test suite that
takes a week to run will mean that people won't test at
all. Personally, I can (and have for many years) tolerate acceptance
test suites that take upto an hour.
The existence of such an acceptance test should not however obviate
the presence of a wider regression test suite. It should be fine to
have an entire suite of regression test buckets take a week to run,
because you only start running them once you have a release candidate
or equivalent.
Of course, setting up a regression test suite takes effort. There is
no need, however, to spend umpteen amounts of time writing the
buckets. What can be done is to incrementally build it up. So whenever
we have a significant new feature, somebody (preferably not the key
developers) could take the time to set up a set of test cases that try
to test it thoroughly. It's okay if such a test bucket takes 10-15
minutes to run. Then this can get rolled up into the regression suite
while a very small "representative" test case makes it to the
acceptance test suite.
Of course, in the open source world, these things take resources and
are not easy to do.
I certainly think that the base regression test suite is great. We
have clearing the pgsql regression test a checkin requirement for
TelegraphCQ developers as our goal is to not break pgsql
functionality.
--
Pip-pip
Sailesh
http://www.cs.berkeley.edu/~sailesh
