Thread: HT on or off for E5-26xx ?

HT on or off for E5-26xx ?

From
David Boreham
Date:
I'm bringing up a new type of server using Intel E5-2620 (unisocket)
which was selected for good SpecIntRate performance vs cost/power (201
for $410 and 95W).

Was assuming it was 6-core but I just noticed it has HT which is
currently enabled since I see 12 cores in /proc/cpuinfo

Question for the performance experts : is it better to have HT enabled
or disabled for this generation of Xeon ?
Workload will be moderately concurrent, small OLTP type transactions.
We'll also run a few low-load VMs (using KVM) and a big Java application.

Any thoughts welcome.
Thanks.








Re: HT on or off for E5-26xx ?

From
Mark Kirkwood
Date:
On 07/11/12 16:31, David Boreham wrote:
>
> I'm bringing up a new type of server using Intel E5-2620 (unisocket)
> which was selected for good SpecIntRate performance vs cost/power (201
> for $410 and 95W).
>
> Was assuming it was 6-core but I just noticed it has HT which is
> currently enabled since I see 12 cores in /proc/cpuinfo
>
> Question for the performance experts : is it better to have HT enabled
> or disabled for this generation of Xeon ?
> Workload will be moderately concurrent, small OLTP type transactions.
> We'll also run a few low-load VMs (using KVM) and a big Java application.
>
>
>

I've been benchmarking a E5-4640 (4 socket) and hyperthreading off gave
much better scaling behaviour in pgbench (gentle rise and flatten off),
whereas with hyperthreading on there was a dramatic falloff after approx
number clients = number of (hyperthreaded) cpus. The box is intended to
be a pure db server, so we are running with hyperthreading off.

Cheers

Mark


Re: HT on or off for E5-26xx ?

From
David Boreham
Date:
On 11/6/2012 9:16 PM, Mark Kirkwood wrote:
>
>
> I've been benchmarking a E5-4640 (4 socket) and hyperthreading off
> gave much better scaling behaviour in pgbench (gentle rise and flatten
> off), whereas with hyperthreading on there was a dramatic falloff
> after approx number clients = number of (hyperthreaded) cpus. The box
> is intended to be a pure db server, so we are running with
> hyperthreading off.

It looks like this syndrome is not observed on my box, likely due to the
much lower number of cores system-wide (12).
I see pgbench tps increase nicely until #threads/clients == #cores, then
plateau. I tested up to 96 threads btw.

We're waiting on more memory modules to arrive. I'll post some test
results once we have all 4 memory banks populated.




Re: HT on or off for E5-26xx ?

From
Devrim GÜNDÜZ
Date:
Hi,

On Tue, 2012-11-06 at 20:31 -0700, David Boreham wrote:

> Was assuming it was 6-core but I just noticed it has HT which is
> currently enabled since I see 12 cores in /proc/cpuinfo
>
> Question for the performance experts : is it better to have H enabled
> or disabled for this generation of Xeon ? Workload will be moderately
> concurrent, small OLTP type transactions.  We'll also run a few
> low-load VMs (using KVM) and  big Java application.

HT should be good for file servers, or say many of the app servers, or
small web/mail  servers. PostgreSQL relies on the CPU power, and since
the HT CPUs don't have the same power as the original CPU, when OS
submits a job to that particular HTed CPU, query will run significantly
slow. To avoid issues, I would suggest you to turn HT off on all
PostgreSQL servers. If you can throw some more money, another 6-core CPU
would give more benefit.

Regards,

--
Devrim GÜNDÜZ
Principal Systems Engineer @ EnterpriseDB: http://www.enterprisedb.com
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
Community: devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org  Twitter: http://twitter.com/devrimgunduz

Attachment

Re: HT on or off for E5-26xx ?

From
David Boreham
Date:
On 11/7/2012 6:37 AM, Devrim GÜNDÜZ wrote:
> HT should be good for file servers, or say many of the app servers, or
> small web/mail  servers. PostgreSQL relies on the CPU power, and since
> the HT CPUs don't have the same power as the original CPU, when OS
> submits a job to that particular HTed CPU, query will run significantly
> slow. To avoid issues, I would suggest you to turn HT off on all
> PostgreSQL servers. If you can throw some more money, another 6-core CPU
> would give more benefit.
I realize this is the "received knowledge" but it is not supported by
the evidence before me (which is that I get nearly 2x the throughput
from pgbench using nthreads == nhtcores vs nthreads == nfullcores).
Intel's latest HT implementation seems to suffer less from the kinds of
resource sharing contention issues seen in older generations.

Once I have the machine's full memory installed I'll run pgbench with HT
disabled in the BIOS and post the results.






Re: HT on or off for E5-26xx ?

From
Mark Kirkwood
Date:
On 08/11/12 02:33, David Boreham wrote:
> On 11/6/2012 9:16 PM, Mark Kirkwood wrote:
>>
>>
>> I've been benchmarking a E5-4640 (4 socket) and hyperthreading off
>> gave much better scaling behaviour in pgbench (gentle rise and
>> flatten off), whereas with hyperthreading on there was a dramatic
>> falloff after approx number clients = number of (hyperthreaded) cpus.
>> The box is intended to be a pure db server, so we are running with
>> hyperthreading off.
>
> It looks like this syndrome is not observed on my box, likely due to
> the much lower number of cores system-wide (12).
> I see pgbench tps increase nicely until #threads/clients == #cores,
> then plateau. I tested up to 96 threads btw.
>
> We're waiting on more memory modules to arrive. I'll post some test
> results once we have all 4 memory banks populated.
>
>
>
>

Interesting - I was wondering if a single socket board would behave
differently (immediately after posting of course)...I've got an i3 home
system that scales nicely even with hyperthreading on (2 cores, 4
typerthreads).

Cheers

Mark


Re: HT on or off for E5-26xx ?

From
David Boreham
Date:
Well, the results are in and at least in this particular case
conventional wisdom is overturned. Not a huge benefit, but throughput is
definitely higher with HT enabled and nthreads >> ncores:

HT off :

bash-4.1$ /usr/pgsql-9.2/bin/pgbench -T 600 -j 48 -c 48
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 100
query mode: simple
number of clients: 48
number of threads: 48
duration: 600 s
number of transactions actually processed: 2435711
tps = 4058.667332 (including connections establishing)
tps = 4058.796309 (excluding connections establishing)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           52.50    0.00   14.79    5.07    0.00   27.64

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00  5700.30    0.10 13843.50     0.00 74.78
11.06    48.46    3.50   0.05  65.21

HT on:

bash-4.1$ /usr/pgsql-9.2/bin/pgbench -T 600 -j 48 -c 48
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 100
query mode: simple
number of clients: 48
number of threads: 48
duration: 600 s
number of transactions actually processed: 2832463
tps = 4720.668984 (including connections establishing)
tps = 4720.750477 (excluding connections establishing)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           40.61    0.00   12.71    3.09    0.00   43.59

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00  6197.10   14.80 16389.50     0.14 86.53
10.82    54.11    3.30   0.05  82.35

System details:

E5-2620 (6 core + HT 15Mb LL) 64G (4 channels with 16G 1333 modules),
Intel 710 300G (which is faster than the smaller drives, note),
Supermicro X9SRi-F Motherboard.
CentOS 6.3 64-bit, PG 9.2.1 from the PGDG RPM repository. pgbench
running locally on the server.




Re: HT on or off for E5-26xx ?

From
Shaun Thomas
Date:
On 11/07/2012 09:16 PM, David Boreham wrote:

> bash-4.1$ /usr/pgsql-9.2/bin/pgbench -T 600 -j 48 -c 48

Unfortunately without -S, you're not really testing the processors. A
regular pgbench can fluctuate a more than that due to writing and
checkpoints.

For what it's worth, our X5675's perform about 40-50% better with HT
enabled. Not the 2x you might expect by doubling the amount of
"processors", but it definitely didn't make things worse.

--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas@optionshouse.com

______________________________________________

See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email


Re: HT on or off for E5-26xx ?

From
David Boreham
Date:
On 11/8/2012 6:58 AM, Shaun Thomas wrote:
> On 11/07/2012 09:16 PM, David Boreham wrote:
>
>> bash-4.1$ /usr/pgsql-9.2/bin/pgbench -T 600 -j 48 -c 48
>
> Unfortunately without -S, you're not really testing the processors. A
> regular pgbench can fluctuate a more than that due to writing and
> checkpoints.
Hmm...my goal was to test with a workload close to our application's
(which is heavy OLTP, small transactions and hence sensitive to I/O
commit rate).
The hypothesis I was testing was that enabling HT positively degrades
performance (which in my case it does not). I wasn't to be honest really
testing the additional benefit from HT, rather observing that it is
non-negative :)

If I have time I can run the select-only test for you and post the
results. The DB fits into memory so it will be a good CPU test.




Re: HT on or off for E5-26xx ?

From
David Boreham
Date:
Here are the SELECT only pgbench test results from my E5-2620 machine,
with HT on and off:

HT off:

bash-4.1$ /usr/pgsql-9.2/bin/pgbench -T 600 -j 48 -c 48 -S
starting vacuum...end.
transaction type: SELECT only
scaling factor: 100
query mode: simple
number of clients: 48
number of threads: 48
duration: 600 s
number of transactions actually processed: 25969680
tps = 43281.392273 (including connections establishing)
tps = 43282.476955 (excluding connections establishing)

All 6 cores saturated:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           81.42    0.00   18.21    0.00    0.00    0.37

HT on:

bash-4.1$ /usr/pgsql-9.2/bin/pgbench -T 600 -j 48 -c 48 -S
starting vacuum...end.
transaction type: SELECT only
scaling factor: 100
query mode: simple
number of clients: 48
number of threads: 48
duration: 600 s
number of transactions actually processed: 29934601
tps = 49888.697225 (including connections establishing)
tps = 49889.570754 (excluding connections establishing)

12% of CPU showing as idle (whether that's true or not I'm not sure):

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           71.09    0.00   16.99    0.00    0.00   11.92

So for this particular test HT gives us the equivalent of about one
extra core.
It does not reduce performance, rather increases performance slightly.