Thread: Regression tests
Are there any regression tests or unit tests beyond 'make check', or possibly benchmarks which not only measure performance but also verify that the results are correct? I have patches which I want to test under high load from multiple concurrent clients, so 'make check' isn't enough. Google has tons of hits for articles and RDBMS reviews that mention SuperSmack, but no hits for the actual software. DES -- Dag-Erling Smørgrav - des@des.no
> Are there any regression tests or unit tests beyond 'make check', or > possibly benchmarks which not only measure performance but also verify > that the results are correct? I have patches which I want to test > under high load from multiple concurrent clients, so 'make check' > isn't enough. Google has tons of hits for articles and RDBMS reviews > that mention SuperSmack, but no hits for the actual software. The whole point of make check is to check correctness, not performance. It has concurrent loading as well. Chris
Christopher Kings-Lynne <chriskl@familyhealth.com.au> writes: > The whole point of make check is to check correctness, not > performance. I understand that. > It has concurrent loading as well. It doesn't stress the system anywhere near enough to reveal bugs in, say, the shared memory or semaphore code. DES -- Dag-Erling Smørgrav - des@des.no
Dag-Erling Smørgrav wrote: > It doesn't stress the system anywhere near enough to reveal bugs in, > say, the shared memory or semaphore code. I agree -- I think we definitely need more tests for the concurrent behavior of the system. -Neil
On Wed, 2005-05-04 at 10:56 +1000, Neil Conway wrote: > Dag-Erling Smørgrav wrote: > > It doesn't stress the system anywhere near enough to reveal bugs in, > > say, the shared memory or semaphore code. > > I agree -- I think we definitely need more tests for the concurrent > behavior of the system. > Yes, very much agree. Contributions in that area would be most welcome. Best Regards, Simon Riggs
Neil Conway <neilc@samurai.com> writes: > Dag-Erling Smørgrav wrote: > > It doesn't stress the system anywhere near enough to reveal bugs in, > > say, the shared memory or semaphore code. > I agree -- I think we definitely need more tests for the concurrent > behavior of the system. Quite, but in the meantime, a good benchmark should stress the system enough to cause crashes, lockups or at least incorrect results if a bug is introduced in the shared memory or semaphore code, and will definitely reveal any slowdowns introduced by new code, so my question is: where can I find a good benchmark for PostgreSQL? Note that I don't care about comparing PostgreSQL to other RDBMSes; I just want to a) test PostgreSQL under high concurrent load and b) if possible, measure the performance impact of a patch. DES -- Dag-Erling Smørgrav - des@des.no
> Quite, but in the meantime, a good benchmark should stress the system > enough to cause crashes, lockups or at least incorrect results if a > bug is introduced in the shared memory or semaphore code, and will > definitely reveal any slowdowns introduced by new code, so my question > is: where can I find a good benchmark for PostgreSQL? Note that I > don't care about comparing PostgreSQL to other RDBMSes; I just want to > a) test PostgreSQL under high concurrent load and b) if possible, > measure the performance impact of a patch. You can use contrib/pgbench as a rather simplistic test, or you could try OSDB (http://osdb.sourceforge.net/) Chris
Christopher Kings-Lynne <chriskl@familyhealth.com.au> writes: >> Quite, but in the meantime, a good benchmark should stress the system >> enough to cause crashes, lockups or at least incorrect results if a >> bug is introduced in the shared memory or semaphore code, and will >> definitely reveal any slowdowns introduced by new code, so my question >> is: where can I find a good benchmark for PostgreSQL? Note that I >> don't care about comparing PostgreSQL to other RDBMSes; I just want to >> a) test PostgreSQL under high concurrent load and b) if possible, >> measure the performance impact of a patch. > You can use contrib/pgbench as a rather simplistic test, or you could > try OSDB (http://osdb.sourceforge.net/) For something a little tougher, talk to Mark Wong about using OSDL's testbed. regards, tom lane
On Wednesday 04 May 2005 03:20, Christopher Kings-Lynne wrote: > > Quite, but in the meantime, a good benchmark should stress the system > > enough to cause crashes, lockups or at least incorrect results if a > > bug is introduced in the shared memory or semaphore code, and will > > definitely reveal any slowdowns introduced by new code, so my question > > is: where can I find a good benchmark for PostgreSQL? Note that I > > don't care about comparing PostgreSQL to other RDBMSes; I just want to > > a) test PostgreSQL under high concurrent load and b) if possible, > > measure the performance impact of a patch. > > You can use contrib/pgbench as a rather simplistic test, or you could > try OSDB (http://osdb.sourceforge.net/) > Shouldn't we be getting some of this testing out of the work being done by Mark Wong and the OSDN folks? -- Robert Treat Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL