On Thu, Aug 27, 2009 at 12:03 AM, Dimitri
Fontaine<dfontaine@hi-media.com> wrote:
> Is the offering good enough? We might need to run some kind of tutorials
> for users to be able to run large tests easily, and maybe think about
> some newer tools allowing to compare logs of two application runs in two
> database versions (capture all queries and result in a database, then
> have a way to diff). Then beta testing would mean having a spare machine
> where to run the magic regression test suite against some internal
> application.
Well, people may recall I spent a lot of time testing 8.3 before and
during beta. The important words are "a lot of time", probably one
month full time spread on 3 months to find *only* 3 problems Tom,
Alvarro and Andrew fixed: yes one month for only 3 problems
identified, reported, discussed and fixed.
The problem isn't to connect your application to the database - it's
the easy part: if you have a large one, you probably won't see the
anomalies.
The application I used for my tests is displaying every SQL query at
the bottom of the page with the time spent executing the query, I was
switching from the 8.1 site to the 8.3 site to check everything
manually.
I also got all the urls of this application (more than one million),
and use a load test tool to load each page and pgFouine to grab any
error from the PostgreSQL logs.
Even with these information and this work, I'm pretty sure I would
have missed a join problem which would have returned 2 or 3 more rows
even with the time I spent working on it.
My plan at the time was to develop an application which would parse
the query logs from the server, replay the queries on 2 PG servers
with both versions and report me the anomalies (difference in the
number of rows, difference in the content, queries slower than with
the old version).
I haven't had the opportunity to work on the 8.4 beta test due to my
daily (and often nightly) job work load but the idea is still there
and IMHO it's really necessary if we want to be able to detect the
problems we only discovered after the release.
--
Guillaume
