Thread: pgbench: Support Multiple Simultaneous Runs (with Mean and Std. Dev.)

I created a version of this a while back for a comparison of 7.4.x to
8.0.x. Now that 8.1.x has come out, I've decided to go ahead and put
it in patch form for potential inclusion in contrib.

The main addition is the addition of a -x option that allows
specification of a number of successive runs of pgbench for use in
sanity-checking basic benchmark results to reduce the potential for
noise in a single run.

At the end of a series of runs, the mean tps and standard deviations
are reported.

I only patched the English README.

I didn't test compatibility with the new file code, but a cursory
glance at the code makes me think it would work.

The following patch is against a fresh pull from anonymous CVS.


Similar versions of pgbench.c for 7.4.7 and 8.0.2 are available at
the following URL, although the argument was -m originally. I changed
it to -x to follow bonnie++, which I frequently use in conjunction
with pgbench for benchmarking.

http://www.sitening.com/tools/postgresql-benchmark/

--
Thomas F. O'Connell
Database Architecture and Programming
Co-Founder
Sitening, LLC

http://www.sitening.com/
110 30th Avenue North, Suite 6
Nashville, TN 37203-6320
615-260-0005 (cell)
615-469-5150 (office)
615-469-5151 (fax)

"Thomas F. O'Connell" <tfo@sitening.com> writes:
> The main addition is the addition of a -x option that allows
> specification of a number of successive runs of pgbench for use in
> sanity-checking basic benchmark results to reduce the potential for
> noise in a single run.

What exactly does this do that increasing the number of transactions
doesn't do?

Not to say that I'm not all for making pgbench results more trustworthy.
But adding still another parameter that people don't know what to do
with isn't going to create any progress in that direction.

            regards, tom lane

I guess I was thinking that the multiple iterations allow for easier
pinpointing of noise on the system. I think I remember originally
writing this during an episode where successive runs of pgbench were
yielding wildly different results, and I wound up wanting an easy way
to detect this and examine it further, so standard deviations in
smaller sets of runs seemed an easy way to do it.

With large numbers of transactions, any noise in the system would get
smoothed out. I guess I like the ability to see the incremental
behavior of the system when using pgbench, and this lets me run it
for a long enough duration inspecting more often.

If this isn't deemed generally useful functionality, that's fine.

--
Thomas F. O'Connell
Database Architecture and Programming
Co-Founder
Sitening, LLC

http://www.sitening.com/
110 30th Avenue North, Suite 6
Nashville, TN 37203-6320
615-260-0005 (cell)
615-469-5150 (office)
615-469-5151 (fax)

Tom,

I noticed in this thread:

http://archives.postgresql.org/pgsql-performance/2005-12/msg00206.php

you state:

"Try something like "-s 10 -c 10 -t 3000" to get numbers reflecting test
conditions more like what the TPC council had in mind when they designed
this benchmark.  I tend to repeat such a test 3 times to see if the
numbers are repeatable, and quote the middle TPS number as long as
they're not too far apart."

The -x I added is to make your "3 times" easier. Sure, it's just
sugar, but if this is part of the recommended use of the tool, or
even a personal case study from the core team, I figured it might be
of use to others. Anyway, just a followup since I was going through
the archives and found a mild argument in favor it from you. :)

--
Thomas F. O'Connell
Database Architecture and Programming
Co-Founder
Sitening, LLC

http://www.sitening.com/
3004 B Poston Avenue
Nashville, TN 37203-1314
615-260-0005 (cell)
615-469-5150 (office)
615-469-5151 (fax)