Hi,
Peter Eisentraut <peter_e@gmx.net> writes:
> On ons, 2009-08-26 at 14:26 -0400, Robert Haas wrote:
>> Sure, but an aimless mandate to do testing for 4 (or 8, or 12) months
>> doesn't necessarily buy you much, either. I'm good at focused
>> activity - but there was nothing focused about 8.4 beta that I could
>> see. Maybe we need some kind of TestFest process.
>
> Yeah, exactly. I can't imagine end users would know what to do during
> beta. Even assuming that you have release notes at the beginning of
> beta, you can't expect people to go through every item and do a formal
> test for it. Surely it's been tested before, else it would not be in
> the release, right?
Well we all know that developpers are really bad at testing, because
they tend to test for cases they though about while developping the code
rather than being creative and running a full application against the
overall new product.
Now, it could be that what we miss is some tool suite to enable
application people to test their full code against our new releases and
check results, performance, plans, etc.
I know about a couple of tools to get there, Tsung and Playr. And the
focus is performance testing and scalability, not that it still works.
Is the offering good enough? We might need to run some kind of tutorials
for users to be able to run large tests easily, and maybe think about
some newer tools allowing to compare logs of two application runs in two
database versions (capture all queries and result in a database, then
have a way to diff). Then beta testing would mean having a spare machine
where to run the magic regression test suite against some internal
application.
Regards,
--
dim
Tsung: http://tsung.erlang-projects.org/
Playr: https://area51.myyearbook.com/trac.cgi/wiki/Playr
PS: sql level unit testing isn't an answer here as it means the
application already have the tests when entering beta. Hard sell.
