On Wed, Aug 26, 2009 at 7:39 PM, Bruce Momjian<bruce@momjian.us> wrote:
> Peter Eisentraut wrote:
>> Much of the delay and uncertainty during beta in my mind comes from the
>> situation that we wait for negative results and don't trust the release
>> until we have seen and fixed enough of them. Instead of waiting for
>> concrete, positive results and producing the release with confidence.
>
> Yep, that is our dilemma.
To get positive results in which you can have confidence, you have to
know that the testing which was done actually did a reasonably good
job exercising the code in a way that would have flushed out bugs, had
any been present. That sounds a lot like the definition of a
regression test suite. Of course, we have that already, but it's
nowhere near comprehensive. Maybe we should be looking at an expanded
test suite that runs on a time scale of hours rather than seconds.
Actually, didn't Peter talk about something like this at PGCon?
I don't think that any test suite is going to eliminate the need for
beta-testing. But if we could say that we had a regression test suite
which covered X% of our code, and it passed on all Y platforms tested,
that would certainly be a confidence booster, especially for large
values of X. What we're basically doing now is hoping that
beta-testers come up with some novel tests, and that's a bit
hit-or-miss.
Part of the question, of course, is how to build up such a regression
test suite. One way to do it is try to add a test every time you fix
a bug, such that the new test fails on the unpatched code and passes
with the fix... or we could have a expanded-regression-test only
commitfest during beta, or something. I think people would be willing
to put in some work on this. I certainly would. I run the regression
tests frequently during development but unless I'm making system
catalog changes they rarely catch anything. I'm not prepared to write
the whole thing myself, but I would definitely be willing to develop
and contribute some tests.
...Robert
