Thread: Add accurate option to pgbench

On Thu, Oct 31, 2013 at 6:36 AM, Mitsumasa KONDO
<kondo.mitsumasa@gmail.com> wrote:
> Hi,
>
> I create pgbench patch that adding accurate option in benchmark, and submit
> it in CF3.
> It is simple option to get more accurate benchmark result and to avoid miss
> benchmark result in pgbench.
>
> Logic of this option is under following.
>   1. execute cluster command to sort records.
>   2. execute checkpoint to clear dirty-buffers in shared_buffers.
>   3. execute sync command to clear dirty-file-caches in OS.
>   4. waiting 10 seconds for raid cache is until empty .
>   5. execute checkpoint to init checkpoint_timeout and checkpoint_segments.
>   6. start benchmark.

I have similar logic in some of my benchmarking scripts but I don't
see a compelling reason to include it in pgbench itself.  You can
checkpoint, sync, and clear OS caches in your script before starting
the pgbench run.  Requirements will vary from system to system; e.g.
some people might want to write to /proc/sys/vm/drop_caches, which is
both non-portable and not possible from within pgbench because it
requires additional privileges.  More importantly, not everyone will
want to do it, and not everyone will want to write the same value.
Similarly, waiting 10 seconds for the RAID cache to drain is not
relevant for everyone, nor is necessarily the right amount of time to
wait.  We'll go nuts if we try to anticipate needs in this area in
pgbench; there will be many different right answers on individual
people's systems.

All of which is to say that I'm not in favor of accepting this patch.
As a side node, if we were going to accept it, I think --accurate
isn't a good name; there could be good reasons to want to run without
these behaviors, but who wouldn't want to be accurate?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

>> Logic of this option is under following.
>>   1. execute cluster command to sort records.
>>   2. execute checkpoint to clear dirty-buffers in shared_buffers.
>>   3. execute sync command to clear dirty-file-caches in OS.
>>   4. waiting 10 seconds for raid cache is until empty .
>>   5. execute checkpoint to init checkpoint_timeout and checkpoint_segments.
>>   6. start benchmark.
>
> I have similar logic in some of my benchmarking scripts but I don't
> see a compelling reason to include it in pgbench itself.

I agree that this looks more like script material.

However I think that part of this interesting checklist and discussion 
could make it to some "caveat" section about reproducible performance 
measures in pgbench documentation, though?

-- 
Fabien.

Fabien COELHO <coelho@cri.ensmp.fr> writes:
> However I think that part of this interesting checklist and discussion 
> could make it to some "caveat" section about reproducible performance 
> measures in pgbench documentation, though?

+1.  There's already a section of advice about how to get reproducible
numbers from pgbench --- we could certainly extend that to cover more
things.

BTW, even if we were going to put code for these things into pgbench,
driving them all off a single switch would be very bad design.
I see no reason to think that all and only these issues would be
appropriate to control for any particular user of pgbench.
        regards, tom lane