Thread: Some performance testing?

All,

I currently have access to a matched pair of 20-core, 128GB RAM servers
with SSD-PCI storage, for about 2 weeks before they go into production.
 Are there any performance tests people would like to see me run on
these?  Otherwise, I'll just do some pgbench and DVDStore.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

It would be interesting to get raw performance benchmarks in addition to
PG specific benchmarks. I’ve been measuring raw I/O performance of a few
of our systems and run the following tests as well:

1. 10 runs of bonnie++
2. 5 runs of hdparm -Tt
3. Using a temp file created on the SSD, dd if=tempfile of=/dev/null bs=1M
count=1024 && echo 3 > /proc/sys/vm/drop_caches; dd
if=tempfileof=/dev/null bs=1M count=1024
4. Using phoronix benchmarks -> stream / ramspeed / compress-7zip


I was curious to measure the magnitude of difference between HDD -> SSD. I
would expect significant differences between SSD -> PCI-E Flash.

I’ve included some metrics from some previous runs vs. different types of
SSDs (OWC Mercury Extreme 6G which is our standard SSD, an Intel S3700
SSD, a Samsung SSD 840 PRO) vs. some standard HDD from  Western Digital
and HGST. I put in a req for a 960Gb Mercury Excelsior PCI-E SSD which
hasn’t yet materialized ...

Thanks, M.

Mel Llaguno • Staff Engineer – Team Lead
Office: +1.403.264.9717 x310
www.coverity.com <http://www.coverity.com/> • Twitter: @coverity
Coverity by Synopsys


On 3/31/15, 1:52 PM, "Josh Berkus" <josh@agliodbs.com> wrote:

>All,
>
>I currently have access to a matched pair of 20-core, 128GB RAM servers
>with SSD-PCI storage, for about 2 weeks before they go into production.
> Are there any performance tests people would like to see me run on
>these?  Otherwise, I'll just do some pgbench and DVDStore.
>
>-- 
>Josh Berkus
>PostgreSQL Experts Inc.
>http://pgexperts.com
>
>
>-- 
>Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
>To make changes to your subscription:
>http://www.postgresql.org/mailpref/pgsql-performance

On 04/01/2015 01:37 AM, Przemysław Deć wrote:
> Maybe you will find time to benchamark xfs vs ext4 (with and without
> journaling enabled on ext4).
>
> Nice comparison also could be rhel 6.5 with its newest kernel 2.6.32-X
> vs RHEL 7.0 and kernel 3.10.

Due to how these are hosted, I can't swap out kernels.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

FYI - all my tests were conducted using Ubuntu 12.04 x64 LTS (which I
believe are all 3.xx series kernels).

Mel Llaguno • Staff Engineer – Team Lead
Office: +1.403.264.9717 x310
www.coverity.com <http://www.coverity.com/> • Twitter: @coverity
Coverity by Synopsys




On 4/6/15, 2:51 PM, "Josh Berkus" <josh@agliodbs.com> wrote:

>On 04/01/2015 01:37 AM, Przemysław Deć wrote:
>> Maybe you will find time to benchamark xfs vs ext4 (with and without
>> journaling enabled on ext4).
>> 
>> Nice comparison also could be rhel 6.5 with its newest kernel 2.6.32-X
>> vs RHEL 7.0 and kernel 3.10.
>
>Due to how these are hosted, I can't swap out kernels.
>
>-- 
>Josh Berkus
>PostgreSQL Experts Inc.
>http://pgexperts.com

On 04/07/2015 09:46 AM, Mel Llaguno wrote:
> FYI - all my tests were conducted using Ubuntu 12.04 x64 LTS (which I
> believe are all 3.xx series kernels).

If it's 3.2 or 3.5, then your tests aren't useful, I'm afraid.  Both of
those kernels have known, severe, memory management issues.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Care to elaborate? We usually do not recommend specific kernel versions
for our customers (who run on a variety of distributions). Thanks, M.

Mel Llaguno • Staff Engineer – Team Lead
Office: +1.403.264.9717 x310
www.coverity.com <http://www.coverity.com/> • Twitter: @coverity
Coverity by Synopsys


On 4/7/15, 11:41 AM, "Josh Berkus" <josh@agliodbs.com> wrote:

>On 04/07/2015 09:46 AM, Mel Llaguno wrote:
>> FYI - all my tests were conducted using Ubuntu 12.04 x64 LTS (which I
>> believe are all 3.xx series kernels).
>
>If it's 3.2 or 3.5, then your tests aren't useful, I'm afraid.  Both of
>those kernels have known, severe, memory management issues.
>
>-- 
>Josh Berkus
>PostgreSQL Experts Inc.
>http://pgexperts.com

On 04/07/2015 11:07 AM, Mel Llaguno wrote:
> Care to elaborate? We usually do not recommend specific kernel versions
> for our customers (who run on a variety of distributions). Thanks, M.

You should.

http://www.databasesoup.com/2014/09/why-you-need-to-avoid-linux-kernel-32.html

Performance is literally 2X to 5X different between kernels.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Cool. Good to know. I'll see if I can replicate these results in my environment. Thanks, M.

________________________________________
From: Josh Berkus <josh@agliodbs.com>
Sent: Wednesday, April 08, 2015 1:05 PM
To: Mel Llaguno; Przemysław Deć
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Some performance testing?

On 04/07/2015 11:07 AM, Mel Llaguno wrote:
> Care to elaborate? We usually do not recommend specific kernel versions
> for our customers (who run on a variety of distributions). Thanks, M.

You should.

http://www.databasesoup.com/2014/09/why-you-need-to-avoid-linux-kernel-32.html

Performance is literally 2X to 5X different between kernels.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

>
> Josh, there seems to be an inconsistency in your blog.  You say 3.10.X is
> safe, but the graph you show with the poor performance seems to be from
> 3.13.X which as I understand it is a later kernel.  Can you clarify which
> 3.X kernels are good to use and which are not?

Sorry to cut in -

So far we've found kernel 3.18 to be excellent for postgres 9.3 performance (pgbench + our own queries run much faster
thanwith the 2.6.32-504 centos 6 kernel, and we haven't encountered random stalls or slowness). 

We use elrepo to get prebuilt rpms of the latest mainline stable kernel (kernel-ml).

http://elrepo.org/tiki/kernel-ml

Graeme Bell

From a measurement I took back when we did the upgrade:

performance with 2.6: (pgbench, size 100, 32 clients)

48 651 transactions per second (read only)
6 504 transactions per second (read-write)


performance with 3.18 (pgbench, size 100, 32 clients)

129 303 transactions per second  (read only)
16 895 transactions (read-write)


So that looks like 2.6x improvement to reads and writes. That was an 8 core xeon server with H710P and 4x crucial M550
SSDsin RAID, pg9.3.   
 

Graeme Bell





On 09 Apr 2015, at 12:39, Przemysław Deć <przemyslaw.dec@linuxpolska.pl> wrote:

> Can you say how much faster it was?
> 
> Przemek Deć
> 
> 2015-04-09 11:04 GMT+02:00 Graeme B. Bell <grb@skogoglandskap.no>:
> >
> > Josh, there seems to be an inconsistency in your blog.  You say 3.10.X is
> > safe, but the graph you show with the poor performance seems to be from
> > 3.13.X which as I understand it is a later kernel.  Can you clarify which
> > 3.X kernels are good to use and which are not?
> 
> Sorry to cut in -
> 
> So far we've found kernel 3.18 to be excellent for postgres 9.3 performance (pgbench + our own queries run much
fasterthan with the 2.6.32-504 centos 6 kernel, and we haven't encountered random stalls or slowness).
 
> 
> We use elrepo to get prebuilt rpms of the latest mainline stable kernel (kernel-ml).
> 
> http://elrepo.org/tiki/kernel-ml
> 
> Graeme Bell
> 
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
> 

ext4 settings

ext4, nobarrier
noatime+nodatime, 
stripe&stride aligned between raid10 & ext4 correctly.


Some other useful things to know

-- h710p
readahead disabled on H710P
writeback cache enabled on H710P  
Direct IO enabled on H710P

-- os filesystem settings
linux readahead enabled (16384), 
nr_requests=975
NOOP scheduler
non-NUMA

-- pg
io_concurrency on
async commit.*** see below!

All settings were kept identical on the server before and after the kernel change, so this performance increase can be
entirelyattributed to the newer kernel  and its synergies with our configuration. 3.18 contains about 5-10 years of
linuxkernel development vs. 2.6 kernels (except where backported).
 

I have conducted quite a lot of plug-pull testing with diskchecker.pl, and rather a lot of testing of
scheduling/IO/RAIDcontroller/etc parameters. The OS/RAID controller/file system settings are as fast as I've been able
toachieve without compromising database  integrity (please note: this server can run async_commit because of the work
weuse it for, but we do not use that setting on our other main production servers). 
 

Our local DBs run extremely nicely for all our normal queries which involve quite a mix of random small IO and
full-tableoperations on e.g. 20GB+ tables , so they're not optimised for pgbench specifically.
 

Graeme Bell



On 09 Apr 2015, at 13:56, Przemysław Deć <przemyslaw.dec@linuxpolska.pl> wrote:

> Wow, thats huge performance gain.
> And it was on ext4?
> 
> -- 
> Linux Polska Sp. z o.o.
> Przemysław Deć - Senior Solutions Architect
> RHCSA, RHCJA, PostgreSQL Professional Certification
> mob: +48 519 130 141
> email: pd@linuxpolska.pl
> www.linuxpolska.pl 
>
___________________________________________________________________________________________________________________________
> Linux Polska Sp. z o. o.  Al. Jerozolimskie 123A (26 p.); 02-017 Warszawa; tel. (+48) 222139571; fax (+48)222139671
> KRS - 0000326158 Sąd Rejonowy dla M. St. Warszawy w Warszawie, XII Wydział Gospodarczy KRS
> Kapitał zakładowy wpłacony 1 000 500PLN;  NIP 7010181018;  REGON 141791601
> <Mail Attachment.jpeg>
> 
> 
> 2015-04-09 13:01 GMT+02:00 Graeme B. Bell <grb@skogoglandskap.no>:
> 
> From a measurement I took back when we did the upgrade:
> 
> performance with 2.6: (pgbench, size 100, 32 clients)
> 
> 48 651 transactions per second (read only)
> 6 504 transactions per second (read-write)
> 
> 
> performance with 3.18 (pgbench, size 100, 32 clients)
> 
> 129 303 transactions per second  (read only)
> 16 895 transactions (read-write)
> 
> 
> So that looks like 2.6x improvement to reads and writes. That was an 8 core xeon server with H710P and 4x crucial
M550SSDs in RAID, pg9.3.
 
> 
> Graeme Bell
> 
> 
> 
> 
> 
> On 09 Apr 2015, at 12:39, Przemysław Deć <przemyslaw.dec@linuxpolska.pl> wrote:
> 
> > Can you say how much faster it was?
> >
> > Przemek Deć
> >
> > 2015-04-09 11:04 GMT+02:00 Graeme B. Bell <grb@skogoglandskap.no>:
> > >
> > > Josh, there seems to be an inconsistency in your blog.  You say 3.10.X is
> > > safe, but the graph you show with the poor performance seems to be from
> > > 3.13.X which as I understand it is a later kernel.  Can you clarify which
> > > 3.X kernels are good to use and which are not?
> >
> > Sorry to cut in -
> >
> > So far we've found kernel 3.18 to be excellent for postgres 9.3 performance (pgbench + our own queries run much
fasterthan with the 2.6.32-504 centos 6 kernel, and we haven't encountered random stalls or slowness).
 
> >
> > We use elrepo to get prebuilt rpms of the latest mainline stable kernel (kernel-ml).
> >
> > http://elrepo.org/tiki/kernel-ml
> >
> > Graeme Bell
> >
> > --
> > Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
> > To make changes to your subscription:
> > http://www.postgresql.org/mailpref/pgsql-performance
> >
> 
> 

On Thu, Apr 9, 2015 at 7:35 AM, Graeme B. Bell <grb@skogoglandskap.no> wrote:
> ext4 settings
>
> ext4, nobarrier
> noatime+nodatime,
> stripe&stride aligned between raid10 & ext4 correctly.
>
>
> Some other useful things to know
>
> -- h710p
> readahead disabled on H710P
> writeback cache enabled on H710P
> Direct IO enabled on H710P
>
> -- os filesystem settings
> linux readahead enabled (16384),
> nr_requests=975
> NOOP scheduler
> non-NUMA
>
> -- pg
> io_concurrency on
> async commit.*** see below!
>
> All settings were kept identical on the server before and after the kernel change, so this performance increase can
beentirely attributed to the newer kernel  and its synergies with our configuration. 3.18 contains about 5-10 years of
linuxkernel development vs. 2.6 kernels (except where backported). 
>
> I have conducted quite a lot of plug-pull testing with diskchecker.pl, and rather a lot of testing of
scheduling/IO/RAIDcontroller/etc parameters. The OS/RAID controller/file system settings are as fast as I've been able
toachieve without compromising database  integrity (please note: this server can run async_commit because of the work
weuse it for, but we do not use that setting on our other main production servers). 
>
> Our local DBs run extremely nicely for all our normal queries which involve quite a mix of random small IO and
full-tableoperations on e.g. 20GB+ tables , so they're not optimised for pgbench specifically. 

It would be handy to see a chart comparing 3.11, 3.13 and 3.18 as
well, to see if most / any of those performance gains came in earlier
kernels, but after 2.6 or 3.2 etc.

Can confirm that for pg purposes, 3.2 is basically broken in some not
to great ways. We've had servers that were overloaded at load factors
of 20 or 30 drop down to 5 or less with just a change from 3.2 to
3.11/3.13 on ubuntu 12.04

The 3.10 mainline had a bug which crashes on boot for us, I think it was the network card driver for the R620. Can't
recall. 
The equipment is in continual use so unfortunately we can't test other kernels at the moment but hopefully someone else
maybe interested. 
It's probably valuable to compare against 3.19 too for anyone who doesn't want to delve into kernel archeology.

Kernel performance gains will be sensitive to things like VM settings, scheduler settings, NUMA, hardware choice, BIOS
settingsetc, use of virtualisation, so it depends which codepaths you end up running. So it may not be meaningful to
comparekernels without a range of system configurations to measure against. ie. Your mileage will probably vary
dependingon how far you've tuned your disk configuration, memory, etc.  

Graeme Bell

On 09 Apr 2015, at 17:15, Scott Marlowe <scott.marlowe@gmail.com> wrote:

> On Thu, Apr 9, 2015 at 7:35 AM, Graeme B. Bell <grb@skogoglandskap.no> wrote:
>> ext4 settings
>>
>> ext4, nobarrier
>> noatime+nodatime,
>> stripe&stride aligned between raid10 & ext4 correctly.
>>
>>
>> Some other useful things to know
>>
>> -- h710p
>> readahead disabled on H710P
>> writeback cache enabled on H710P
>> Direct IO enabled on H710P
>>
>> -- os filesystem settings
>> linux readahead enabled (16384),
>> nr_requests=975
>> NOOP scheduler
>> non-NUMA
>>
>> -- pg
>> io_concurrency on
>> async commit.*** see below!
>>
>> All settings were kept identical on the server before and after the kernel change, so this performance increase can
beentirely attributed to the newer kernel  and its synergies with our configuration. 3.18 contains about 5-10 years of
linuxkernel development vs. 2.6 kernels (except where backported). 
>>
>> I have conducted quite a lot of plug-pull testing with diskchecker.pl, and rather a lot of testing of
scheduling/IO/RAIDcontroller/etc parameters. The OS/RAID controller/file system settings are as fast as I've been able
toachieve without compromising database  integrity (please note: this server can run async_commit because of the work
weuse it for, but we do not use that setting on our other main production servers). 
>>
>> Our local DBs run extremely nicely for all our normal queries which involve quite a mix of random small IO and
full-tableoperations on e.g. 20GB+ tables , so they're not optimised for pgbench specifically. 
>
> It would be handy to see a chart comparing 3.11, 3.13 and 3.18 as
> well, to see if most / any of those performance gains came in earlier
> kernels, but after 2.6 or 3.2 etc.
>
> Can confirm that for pg purposes, 3.2 is basically broken in some not
> to great ways. We've had servers that were overloaded at load factors
> of 20 or 30 drop down to 5 or less with just a change from 3.2 to
> 3.11/3.13 on ubuntu 12.04

Scott,

> Can confirm that for pg purposes, 3.2 is basically broken in some not
> to great ways. We've had servers that were overloaded at load factors
> of 20 or 30 drop down to 5 or less with just a change from 3.2 to
> 3.11/3.13 on ubuntu 12.04

That's correct, and 3.5 shares the same problems.  The underlying issue
was that 3.X was tweaked to be MUCH more aggressive about
cache-clearing, to the point where it would be evicting data from the FS
cache which had just been read in and hadn't even been used yet.  For
some reason, this aggressive eviction got worse the more processes on
the system which were using the FS cache, so where you really see it is
when you have more processes with cache than you have cores.

It's pretty easy to demonstrate just using pgbench, with a database
larger than RAM, and 2X as many clients as cores.  You'll see that
kernels 3.2 and 3.5 will do 3X to 5X as much IO for the same workload as
3.10 and later will do.

Greame,

On 04/09/2015 04:01 AM, Graeme B. Bell wrote:> performance with 2.6:
(pgbench, size 100, 32 clients)
>
> 48 651 transactions per second (read only)
> 6 504 transactions per second (read-write)
>
>
> performance with 3.18 (pgbench, size 100, 32 clients)
>
> 129 303 transactions per second  (read only)
> 16 895 transactions (read-write)

Thanks for that data!  I'm glad to see that 3.18 has improved so much.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com