Thread: Add accurate option to pgbench
Hi,
I create pgbench patch that adding accurate option in benchmark, and submit it in CF3.
It is simple option to get more accurate benchmark result and to avoid miss benchmark result in pgbench.
Logic of this option is under following.
1. execute cluster command to sort records.
2. execute checkpoint to clear dirty-buffers in shared_buffers.
3. execute sync command to clear dirty-file-caches in OS.
4. waiting 10 seconds for raid cache is until empty .
5. execute checkpoint to init checkpoint_timeout and checkpoint_segments.
1. execute cluster command to sort records.
2. execute checkpoint to clear dirty-buffers in shared_buffers.
3. execute sync command to clear dirty-file-caches in OS.
4. waiting 10 seconds for raid cache is until empty .
5. execute checkpoint to init checkpoint_timeout and checkpoint_segments.
6. start benchmark.
Sample output is under following.
[mitsu-ko@vm-kondo pgbench]$ ./pgbench -a
starting cluster...end.
starting checkpoint...end.
starting sync all buffers and wait 10 seconds...end.
transaction type: TPC-B (sort of)
scaling factor: 1
query mode: simple
accurate mode: on
number of clients: 1
number of threads: 1
number of transactions per client: 10
number of transactions actually processed: 10/10
latency average: 0.000 ms
tps = 187.677120 (including connections establishing)
tps = 236.417798 (excluding connections establishing)
I hope that it will be reccomended pgbench option in commnity development.
However, it might too carefuly option before starting benchmark.
Please give me comments.
Regards,
--
Mitsumasa KONDO
NTT Open Source Software CenterAttachment
On Thu, Oct 31, 2013 at 6:36 AM, Mitsumasa KONDO <kondo.mitsumasa@gmail.com> wrote: > Hi, > > I create pgbench patch that adding accurate option in benchmark, and submit > it in CF3. > It is simple option to get more accurate benchmark result and to avoid miss > benchmark result in pgbench. > > Logic of this option is under following. > 1. execute cluster command to sort records. > 2. execute checkpoint to clear dirty-buffers in shared_buffers. > 3. execute sync command to clear dirty-file-caches in OS. > 4. waiting 10 seconds for raid cache is until empty . > 5. execute checkpoint to init checkpoint_timeout and checkpoint_segments. > 6. start benchmark. I have similar logic in some of my benchmarking scripts but I don't see a compelling reason to include it in pgbench itself. You can checkpoint, sync, and clear OS caches in your script before starting the pgbench run. Requirements will vary from system to system; e.g. some people might want to write to /proc/sys/vm/drop_caches, which is both non-portable and not possible from within pgbench because it requires additional privileges. More importantly, not everyone will want to do it, and not everyone will want to write the same value. Similarly, waiting 10 seconds for the RAID cache to drain is not relevant for everyone, nor is necessarily the right amount of time to wait. We'll go nuts if we try to anticipate needs in this area in pgbench; there will be many different right answers on individual people's systems. All of which is to say that I'm not in favor of accepting this patch. As a side node, if we were going to accept it, I think --accurate isn't a good name; there could be good reasons to want to run without these behaviors, but who wouldn't want to be accurate? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
>> Logic of this option is under following. >> 1. execute cluster command to sort records. >> 2. execute checkpoint to clear dirty-buffers in shared_buffers. >> 3. execute sync command to clear dirty-file-caches in OS. >> 4. waiting 10 seconds for raid cache is until empty . >> 5. execute checkpoint to init checkpoint_timeout and checkpoint_segments. >> 6. start benchmark. > > I have similar logic in some of my benchmarking scripts but I don't > see a compelling reason to include it in pgbench itself. I agree that this looks more like script material. However I think that part of this interesting checklist and discussion could make it to some "caveat" section about reproducible performance measures in pgbench documentation, though? -- Fabien.
Fabien COELHO <coelho@cri.ensmp.fr> writes: > However I think that part of this interesting checklist and discussion > could make it to some "caveat" section about reproducible performance > measures in pgbench documentation, though? +1. There's already a section of advice about how to get reproducible numbers from pgbench --- we could certainly extend that to cover more things. BTW, even if we were going to put code for these things into pgbench, driving them all off a single switch would be very bad design. I see no reason to think that all and only these issues would be appropriate to control for any particular user of pgbench. regards, tom lane