[PERFORM] Suggestions for a HBA controller (6 x SSDs + madam RAID10) - Mailing list pgsql-performance

I configured an IBM X3650 M4 for development and testing purposes. It’s composed by:

 - 2 x Intel Xeon E5-2690 @ 2.90Ghz (2 x 8 physical Cores + HT)

 - 96GB RAM DDR3 1333MHz (12 x 8GB)

 - 2 x 146GB SAS HDDs @ 15k rpm configured in RAID1 (mdadm)

 - 6 x 525GB SATA SSDs (over-provisioned at 25%, so 393GB available)

I’ve done a lot of testing focusing on 4k and 8k workloads and found that the IOPS of those SSDs are half the expected. On serverfault.com someone suggested me that probably the bottle neck is the embedded RAID controller, a IBM ServeRaid m5110e, which mounts a LSI 2008 controller.

I’m using the disks in JBOD mode with mdadm software RAID, which is blazing fast. The CPU is also very fast, so I don’t mind having a little overhead due to software RAID.

My typical workload is Postgres run as a DWH with 1 to 2 billions of rows, big indexes, partitions and so on, but also intensive statistical computations.

Here’s my post on serverfault.com ( http://serverfault.com/questions/833642/slow-ssd-performance-ibm-x3650-m4-7915 )

and here’s a graph of those six SSDs evaluated using fio as stand-alone disks (outside of the RAID):

All those IOPS should be doubled if all was working correctly. The curve trend is correct for increasing IO Depths.

Anyway, I would like to buy a HBA controller that leverages those 6 SSDs. Each SSD should deliver about 80k to 90k IOPS, so in RAID10 I should get ~240k IOPS (6 x 80k / 2) and in RAID0 ~480k IOPS (6 x 80k). I’ve seen that mdadm effectively scales performance, but the controller limits the overal IOPS at ~120k (exactly the half of the expected IOPS).

Thank you everyone

   Pietro Pugni