Re: [PATCH] pgbench --throttle (submission 7 - with lag measurement) - Mailing list pgsql-hackers

From Tatsuo Ishii
Subject Re: [PATCH] pgbench --throttle (submission 7 - with lag measurement)
Date
Msg-id 20130717.221918.1214885621214868490.t-ishii@sraoss.co.jp
Whole thread Raw
In response to Re: [PATCH] pgbench --throttle (submission 7 - with lag measurement)  (Fabien COELHO <coelho@cri.ensmp.fr>)
Responses Re: [PATCH] pgbench --throttle (submission 7 - with lag measurement)
List pgsql-hackers
>>> tps = 9818.741060 (including connections establishing)
>>>
>>> So I thought I could squeeze 10000 TPS from my box.
>>> Then I tried with -R 5000 tps.
>>>
>>> number of transactions actually processed: 1510640
>>> average rate limit lag: 0.304 ms (max 19.101 ms)
>>> tps = 5035.409397 (including connections establishing)
>>>
>>> As you can see, I got about 5000 tps as expected.
> 
> Yep, it works:-)
> 
>>> But I'm confused by the lag:
>>>
>>> 0.304 ms * 1510640 = 459.2 seconds, which is longer than 300 seconds
>>> (specified by -T). Am I missing something?
> 
> The lag is reasonnable, althought no too good. One transaction is
> about 1.2 ms, the lag is much smaller than that, and you are at about
> 50% of the maximum load. I've got similar figures on my box for such
> settings. It improves if your reduce the number of clients.

No, 5000 TPS = 1/5000 = 0.2 ms per transaction, no? However pgbench
says average lag is 0.304 ms. So the lag is longer than transaction
itself.

> If you reduce the number of clients, or add more threads, the lag is
> reduced.
> 
>> BTW, the system was Linux (kernel 3.0.77).
> 
>> tps = 6730.940132 (including connections establishing)
>> $ pgbench -S -n -c 10 -T 10  -R 3000 test
> 
>> average rate limit lag: 0.089 ms (max 27.301 ms)
>> tps = 2983.707895 (including connections establishing)
>>
>> 0.089 ms * 29840 = 2.66 seconds. Not too bad compared with 10
>> seconds.
> 
> Indeed, that is better. Transactions are about 1.5 ms and you run at
> about 45% of the maximum load here.
> 
>> On Linux maybe the overhead to calculate the lag is bigger
>> than Mac OS X? Just my wild guess though...
> 
> I would be surprised that this would be the issue is to compute the
> measure, compared to network connections and the like. With -S the
> bench is cpu bound. Possibly a better scheduler/thread management on
> OSX? Or more available cores?

The number of cores is same.  I don't understand why number of cores
is relatedx, though. Anyway, as you can see in Mac OS X's case, TPS
itself is no better than the Linux box.

>  Well, I do not know! At high load with
> clients running on the same box as the server, and with more clients &
> server than available cores, there is a lot of competition between
> processes, and between clients that share a unique thread, and a log
> context switching whoch will result in a measured lag.

Hmm... I would like to have cleaner explanation/evidence before
committing the patch.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp



pgsql-hackers by date:

Previous
From: Ants Aasma
Date:
Subject: Re: Improvement of checkpoint IO scheduler for stable transaction responses
Next
From: Tom Lane
Date:
Subject: Re: [COMMITTERS] pgsql: Add support for REFRESH MATERIALIZED VIEW CONCURRENTLY.