Thread: Re: Performance with 2 AMD/Opteron 2.6Ghz and 8gig

Luke,

Yeah, I read those results, and I'm very disappointed with my results
from the MSA1500. I would however be interested in other people's
bonnie++ and benchmarksql results using a similar machine (2 cpu dual
core opteron) with other "off the shelf" storage systems
(EMC/Netapp/Xyratex/../). Could you run benchmarksql against that
machine with the 16 SATA disk and 3Ware 9550SX SATA RAID adapters? It
would be *very* interesting to see how the I/O performance correlates to
benchmarksql (postgres) transaction throughout.

/Mikael

-----Original Message-----
From: Luke Lonergan [mailto:LLonergan@greenplum.com]
Sent: den 28 juli 2006 11:17
To: Mikael Carneholm; Kjell Tore Fossbakk;
pgsql-performance@postgresql.org
Subject: RE: [PERFORM] Performance with 2 AMD/Opteron 2.6Ghz and 8gig

Mikael,

> -----Original Message-----
> From: Mikael Carneholm [mailto:Mikael.Carneholm@WirelessCar.com]
> Sent: Friday, July 28, 2006 2:05 AM
>
> My bonnie++ results are found in this message:
> http://archives.postgresql.org/pgsql-performance/2006-07/msg00164.php
>

Apologies if I've already said this, but those bonnie++ results are very
disappointing.  The sequential transfer rates between 20MB/s and 57MB/s
are slower than a single SATA disk, and your SCSI disks might even do
80MB/s sequential transfer rate each.

Random access is also very poor, though perhaps equal to 5 disk drives
at 500/second.

By comparison, we routinely get 950MB/s sequential transfer rate using
16 SATA disks and 3Ware 9550SX SATA RAID adapters on Linux.

On Solaris ZFS on an X4500, we recently got this bonnie++ result on 36
SATA disk drives in RAID10 (single thread first):

Version  1.03       ------Sequential Output------    --Sequential Input-
--Random-
                    -Per Chr-  --Block--  -Rewrite-  -Per Chr-
--Block--  --Seeks--
Machine        Size K/sec  %CP K/sec  %CP K/sec  %CP K/sec  %CP K/sec
%CP /sec %CP
thumperdw-i-1   32G 120453  99 467814  98 290391  58 109371  99 993344
94 1801   4
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
/sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ 30850  99 +++++ +++
+++++ +++

Bumping up the number of concurrent processes to 2, we get about 1.5x
speed reads of RAID10 with a concurrent workload (you have to add the
rates together):

Version  1.03       ------Sequential Output------   --Sequential Input-
--Random-
                    -Per Chr- --Block--  -Rewrite-  -Per Chr-  --Block--
--Seeks--
Machine        Size K/sec  %CP K/sec  %CP K/sec  %CP K/sec  %CP K/sec
%CP  /sec %CP
thumperdw-i-1   32G 111441  95 212536  54 171798  51 106184  98 719472
88  1233   2
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
/sec %CP
                 16 26085  90 +++++ +++  5700  98 21448  97 +++++ +++
4381  97

Version  1.03       ------Sequential Output------   --Sequential Input-
--Random-
                    -Per Chr-  --Block--  -Rewrite-  -Per Chr-
--Block--   --Seeks--
Machine        Size K/sec  %CP K/sec  %CP K/sec  %CP K/sec  %CP K/sec
%CP  /sec %CP
thumperdw-i-1   32G 116355  99 212509  54 171647  50 106112  98 715030
87  1274   3
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
/sec %CP
                 16 26082  99 +++++ +++  5588  98 21399  88 +++++ +++
4272  97

So that's 2500 seeks per second, 1440MB/s sequential block read, 212MB/s
per character sequential read.

- Luke

Hello.

Unfortunately, I'm leaving for my vacation now, gone 3 weeks. When I'm back I'll run benchmarksql and bonnie++ and give the results here.

The spec I will be using:

Prolite DL585
2 x AMD/Opteron 64-bit 2,6GHZ
8G DDR PC3200
4 x 150G SCSI in SmartArray 5i
Running Gentoo 2006.0 AMD_64 Hardened kernel

Then I will remove the SmartArray 5i, and use a simple nonRAID SCSI controller and implement Linux software RAID, and re-run the tests.

I'll give signal in 3 weeks

- Kjell Tore.




--
"Be nice to people on your way up because you meet them on your way down."

On Fri, 28 Jul 2006, Mikael Carneholm wrote:

> Luke,
>
> Yeah, I read those results, and I'm very disappointed with my results
> from the MSA1500. I would however be interested in other people's
> bonnie++ and benchmarksql results using a similar machine (2 cpu dual
> core opteron) with other "off the shelf" storage systems
> (EMC/Netapp/Xyratex/../). Could you run benchmarksql against that
> machine with the 16 SATA disk and 3Ware 9550SX SATA RAID adapters? It
> would be *very* interesting to see how the I/O performance correlates to
> benchmarksql (postgres) transaction throughout.

FWIW, once our vendor gets all the pieces (having some issues with
figuring out which multilane sata cables to get), I'll have a dual-core
opteron box with a 3Ware 9500SX-12MI and 8 drives.  I need to benchmark to
compare this to our xeon/adaptec/scsi build we've been using.

I've also got a 1U with a 9500SX-4 and 4 drives.  I like how the 3Ware
card scales there - started with 2 drives and got "drive speed" mirroring.
Added two more and most of the bonnie numbers doubled.  This is not what
I'm used to with the Adaptec SCSI junk.

These SATA RAID controllers 3Ware is making seem to be leaps and bounds
beyond what the "old guard" is churning out (at much higher prices).

Charles

> /Mikael
>
> -----Original Message-----
> From: Luke Lonergan [mailto:LLonergan@greenplum.com]
> Sent: den 28 juli 2006 11:17
> To: Mikael Carneholm; Kjell Tore Fossbakk;
> pgsql-performance@postgresql.org
> Subject: RE: [PERFORM] Performance with 2 AMD/Opteron 2.6Ghz and 8gig
>
> Mikael,
>
>> -----Original Message-----
>> From: Mikael Carneholm [mailto:Mikael.Carneholm@WirelessCar.com]
>> Sent: Friday, July 28, 2006 2:05 AM
>>
>> My bonnie++ results are found in this message:
>> http://archives.postgresql.org/pgsql-performance/2006-07/msg00164.php
>>
>
> Apologies if I've already said this, but those bonnie++ results are very
> disappointing.  The sequential transfer rates between 20MB/s and 57MB/s
> are slower than a single SATA disk, and your SCSI disks might even do
> 80MB/s sequential transfer rate each.
>
> Random access is also very poor, though perhaps equal to 5 disk drives
> at 500/second.
>
> By comparison, we routinely get 950MB/s sequential transfer rate using
> 16 SATA disks and 3Ware 9550SX SATA RAID adapters on Linux.
>
> On Solaris ZFS on an X4500, we recently got this bonnie++ result on 36
> SATA disk drives in RAID10 (single thread first):
>
> Version  1.03       ------Sequential Output------    --Sequential Input-
> --Random-
>                    -Per Chr-  --Block--  -Rewrite-  -Per Chr-
> --Block--  --Seeks--
> Machine        Size K/sec  %CP K/sec  %CP K/sec  %CP K/sec  %CP K/sec
> %CP /sec %CP
> thumperdw-i-1   32G 120453  99 467814  98 290391  58 109371  99 993344
> 94 1801   4
>                    ------Sequential Create------ --------Random
> Create--------
>                    -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
>              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
> /sec %CP
>                 16 +++++ +++ +++++ +++ +++++ +++ 30850  99 +++++ +++
> +++++ +++
>
> Bumping up the number of concurrent processes to 2, we get about 1.5x
> speed reads of RAID10 with a concurrent workload (you have to add the
> rates together):
>
> Version  1.03       ------Sequential Output------   --Sequential Input-
> --Random-
>                    -Per Chr- --Block--  -Rewrite-  -Per Chr-  --Block--
> --Seeks--
> Machine        Size K/sec  %CP K/sec  %CP K/sec  %CP K/sec  %CP K/sec
> %CP  /sec %CP
> thumperdw-i-1   32G 111441  95 212536  54 171798  51 106184  98 719472
> 88  1233   2
>                    ------Sequential Create------ --------Random
> Create--------
>                    -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
>              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
> /sec %CP
>                 16 26085  90 +++++ +++  5700  98 21448  97 +++++ +++
> 4381  97
>
> Version  1.03       ------Sequential Output------   --Sequential Input-
> --Random-
>                    -Per Chr-  --Block--  -Rewrite-  -Per Chr-
> --Block--   --Seeks--
> Machine        Size K/sec  %CP K/sec  %CP K/sec  %CP K/sec  %CP K/sec
> %CP  /sec %CP
> thumperdw-i-1   32G 116355  99 212509  54 171647  50 106112  98 715030
> 87  1274   3
>                    ------Sequential Create------ --------Random
> Create--------
>                    -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
>              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
> /sec %CP
>                 16 26082  99 +++++ +++  5588  98 21399  88 +++++ +++
> 4272  97
>
> So that's 2500 seeks per second, 1440MB/s sequential block read, 212MB/s
> per character sequential read.
>
> - Luke
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: explain analyze is your friend
>

Hi, Charles,

Charles Sprickman wrote:

> I've also got a 1U with a 9500SX-4 and 4 drives.  I like how the 3Ware
> card scales there - started with 2 drives and got "drive speed"
> mirroring. Added two more and most of the bonnie numbers doubled.  This
> is not what I'm used to with the Adaptec SCSI junk.

Well, for sequential reading, you should be able to get double drive
speed on a 2-disk mirror with a good controller, as it can balance the
reads among the drives.

Markus
--
Markus Schaber | Logical Tracking&Tracing International AG
Dipl. Inf.     | Software Development GIS

Fight against software patents in EU! www.ffii.org www.nosoftwarepatents.org

Although I for one have yet to see a controller that actualy does this (I believe software RAID on linux doesn't either).

Alex.

On Mon, Aug 07, 2006 at 04:02:52PM -0400, Alex Turner wrote:
> Although I for one have yet to see a controller that actualy does this (I
> believe software RAID on linux doesn't either).

Linux' software RAID does. See earlier threads for demonstrations.

/* Steinar */
--
Homepage: http://www.sesse.net/

On Mon, Aug 07, 2006 at 10:20:02PM +0200, Steinar H. Gunderson wrote:
> On Mon, Aug 07, 2006 at 04:02:52PM -0400, Alex Turner wrote:
> > Although I for one have yet to see a controller that actualy does this (I
> > believe software RAID on linux doesn't either).
>
> Linux' software RAID does. See earlier threads for demonstrations.

The real question: will it balance within a single thread?

Cheaper raid setups will balance individual requests between devices,
but good ones should be able to service a single request from both
devices (assuming it's reading more than whatever the stripe size is).
--
Jim C. Nasby, Sr. Engineering Consultant      jnasby@pervasive.com
Pervasive Software      http://pervasive.com    work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf       cell: 512-569-9461