Greg Smith <gsmith@gregsmith.com> writes:
> I'd want a set of 3 at each configuration because even with longer runs,
> you occasionally get really odd results. Until you have 3 it can be
> unclear which is the weird one.
Yeah, pgbench results are notoriously unrepeatable. One issue is that
the first run after pgbench -i sees conditions a lot different from
subsequent runs (no free space in tables, indexes are more tightly
packed than they will be later, etc). The recently added option to
initialize the tables with a selected fillfactor might help here,
but personally I've not experimented with it.
There seems to be also some of the "good average but bad worst case"
behavior that Josh and others have pointed out in bigger benchmarks.
I've always assumed this was due to checkpointing (and autovac if
enabled). If your test run isn't long enough to cover one full
checkpoint cycle then the results will be quite variable depending
on whether it included a checkpoint or not. 8.3 might alleviate
this effect to some extent.
regards, tom lane
