Thread: Add accurate option to pgbench

Add accurate option to pgbench

From
Mitsumasa KONDO
Date:
Hi,

I create pgbench patch that adding accurate option in benchmark, and submit it in CF3.
It is simple option to get more accurate benchmark result and to avoid miss benchmark result in pgbench.

Logic of this option is under following.
  1. execute cluster command to sort records.
  2. execute checkpoint to clear dirty-buffers in shared_buffers.
  3. execute sync command to clear dirty-file-caches in OS.
  4. waiting 10 seconds for raid cache is until empty .
  5. execute checkpoint to init checkpoint_timeout and checkpoint_segments.
  6. start benchmark.

Sample output is under following.

[mitsu-ko@vm-kondo pgbench]$ ./pgbench -a
starting cluster...end.
starting checkpoint...end.
starting sync all buffers and wait 10 seconds...end.
transaction type: TPC-B (sort of)
scaling factor: 1
query mode: simple
accurate mode: on
number of clients: 1
number of threads: 1
number of transactions per client: 10
number of transactions actually processed: 10/10
latency average: 0.000 ms
tps = 187.677120 (including connections establishing)
tps = 236.417798 (excluding connections establishing)


I hope that it will be reccomended pgbench option in commnity development.
However, it might too carefuly option before starting benchmark.
Please give me comments.

Regards,
--
Mitsumasa KONDO
NTT Open Source Software Center
Attachment

Re: Add accurate option to pgbench

From
Robert Haas
Date:
On Thu, Oct 31, 2013 at 6:36 AM, Mitsumasa KONDO
<kondo.mitsumasa@gmail.com> wrote:
> Hi,
>
> I create pgbench patch that adding accurate option in benchmark, and submit
> it in CF3.
> It is simple option to get more accurate benchmark result and to avoid miss
> benchmark result in pgbench.
>
> Logic of this option is under following.
>   1. execute cluster command to sort records.
>   2. execute checkpoint to clear dirty-buffers in shared_buffers.
>   3. execute sync command to clear dirty-file-caches in OS.
>   4. waiting 10 seconds for raid cache is until empty .
>   5. execute checkpoint to init checkpoint_timeout and checkpoint_segments.
>   6. start benchmark.

I have similar logic in some of my benchmarking scripts but I don't
see a compelling reason to include it in pgbench itself.  You can
checkpoint, sync, and clear OS caches in your script before starting
the pgbench run.  Requirements will vary from system to system; e.g.
some people might want to write to /proc/sys/vm/drop_caches, which is
both non-portable and not possible from within pgbench because it
requires additional privileges.  More importantly, not everyone will
want to do it, and not everyone will want to write the same value.
Similarly, waiting 10 seconds for the RAID cache to drain is not
relevant for everyone, nor is necessarily the right amount of time to
wait.  We'll go nuts if we try to anticipate needs in this area in
pgbench; there will be many different right answers on individual
people's systems.

All of which is to say that I'm not in favor of accepting this patch.
As a side node, if we were going to accept it, I think --accurate
isn't a good name; there could be good reasons to want to run without
these behaviors, but who wouldn't want to be accurate?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: Add accurate option to pgbench

From
Fabien COELHO
Date:
>> Logic of this option is under following.
>>   1. execute cluster command to sort records.
>>   2. execute checkpoint to clear dirty-buffers in shared_buffers.
>>   3. execute sync command to clear dirty-file-caches in OS.
>>   4. waiting 10 seconds for raid cache is until empty .
>>   5. execute checkpoint to init checkpoint_timeout and checkpoint_segments.
>>   6. start benchmark.
>
> I have similar logic in some of my benchmarking scripts but I don't
> see a compelling reason to include it in pgbench itself.

I agree that this looks more like script material.

However I think that part of this interesting checklist and discussion 
could make it to some "caveat" section about reproducible performance 
measures in pgbench documentation, though?

-- 
Fabien.



Re: Add accurate option to pgbench

From
Tom Lane
Date:
Fabien COELHO <coelho@cri.ensmp.fr> writes:
> However I think that part of this interesting checklist and discussion 
> could make it to some "caveat" section about reproducible performance 
> measures in pgbench documentation, though?

+1.  There's already a section of advice about how to get reproducible
numbers from pgbench --- we could certainly extend that to cover more
things.

BTW, even if we were going to put code for these things into pgbench,
driving them all off a single switch would be very bad design.
I see no reason to think that all and only these issues would be
appropriate to control for any particular user of pgbench.
        regards, tom lane