Tom Lane wrote:
> There's no such thing as a useful performance test that runs quickly
> enough to be sane to incorporate in our standard regression tests.
>
To throw a hard number out: I can get a moderately useful performance
test on a SELECT-only workload from pgbench in about one minute. That's
the bare minimum, stepping up to 5 minutes is really necessary before
I'd want to draw any real conclusions.
More importantly, a large portion of the time I'd expect regression test
runs to be happening with debug/assert on. We've well established this
trashes pgbench performance. One of the uglier bits of code added to
add the "performance farm" feature to the buildfarm code was hacking in
a whole different set of build options for it.
Anyway, what I was envisioning here was that performance farm systems
would also execute the standard buildfarm tests, but not the other way
around. We don't want performance numbers from some platform that is
failing the basic tests. I would just expect that systems running the
performance tests would cycle through regression testing much less
often, as they might miss a commit because they were running a longer
test then.
--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us
