Re: 8.5 release timetable, again - Mailing list pgsql-hackers

On Thu, Aug 27, 2009 at 12:03:05AM +0200, Dimitri Fontaine wrote:
> Hi,
> 
> Peter Eisentraut <peter_e@gmx.net> writes:
> > On ons, 2009-08-26 at 14:26 -0400, Robert Haas wrote:
> >> Sure, but an aimless mandate to do testing for 4 (or 8, or 12) months
> >> doesn't necessarily buy you much, either.  I'm good at focused
> >> activity - but there was nothing focused about 8.4 beta that I could
> >> see.  Maybe we need some kind of TestFest process.
> >
> > Yeah, exactly.  I can't imagine end users would know what to do during
> > beta.  Even assuming that you have release notes at the beginning of
> > beta, you can't expect people to go through every item and do a formal
> > test for it.  Surely it's been tested before, else it would not be in
> > the release, right?
> 
> Well we all know that developpers are really bad at testing, because
> they tend to test for cases they though about while developping the code
> rather than being creative and running a full application against the
> overall new product.
> 
> Now, it could be that what we miss is some tool suite to enable
> application people to test their full code against our new releases and
> check results, performance, plans, etc.
> 
> I know about a couple of tools to get there, Tsung and Playr. And the
> focus is performance testing and scalability, not that it still works.
> 
> Is the offering good enough? We might need to run some kind of tutorials
> for users to be able to run large tests easily, and maybe think about
> some newer tools allowing to compare logs of two application runs in two
> database versions (capture all queries and result in a database, then
> have a way to diff). Then beta testing would mean having a spare machine
> where to run the magic regression test suite against some internal
> application.

Someone else upthread mentioned it as well; having an easy way to
capture queries for a day and replaying it towards a server with
the beta software would be a totally awesome way to beta test.
Query results would be compared and any anomalies reported. Doing
this in realtime over both servers would be an even more
appealing option (at least to me and my environment).

I could easily capture queries from a dev machine (where
development is still towards a stable release of postgres) and
test them on the new Beta release. I haven't looked at either
Tsung or Playr (but I will), so if anyone knows if this is
possible with these tools or perhaps is already doing it, it
would be nice with a short "tutorial" (documented on the wiki?)
on steps on how to setup such an environment.

Kind regards,   Kristian. 

-- 
Kristian Larsson                                        KLL-RIPE
+46 704 264511                          kll@spritelink.net