Thread: Server Performance
Hi
Been having interesting times with an IBM x3650 with 8 15k RPM 73GB drives in RAID 10 and a ServRAID 8K controller with Write-Back cache enabled (battery installed and working). Currently getting a pgbench score of 4.7 transactions per second! After playing with the postgresql configuration file, I'm certain that this is not a postgresql problem. I have tried two different Linux distro's upon the server both with the same problems. I'm fairly certain that this is a problem with the hardware configuration / setup, however I'm still waiting for IBM to contact me!
Initially I started with the OS on a RAID 1 array and a 6 drive RAID 10 array for postgresql. With this setup I got 3tps, altering the RAID configuration to a single 8 drive array, running both the OS and postgresql. I was able to reach 700tps, however after upgrading to the latest RAID controller firmware this has now fallen back to 4tps.
Benchmarking another server I have access to, 4 15k 73GB RPM disks with a Dell Perc 5/i controller. I consistently get a pgbench score of 1400tps. Therefore taking a linear extrapolation I expect the IBM x3650 to manage ~3000tps. Additionally my Laptop with a 5400 RPM sata disk was able to score ~200tps.
I have two of these IBM x3650's running the following configurations:
1) IBM x3650
IBM ServRAID controller (Rebranded Adaptec card, using the aacraid driver)
2 15k RPM 73GB RAID 1 (OS array)
6 15k RPM 73GB RAID 10 (Postgresql data array)
2 quad core 3.0GHz Intel Xeons
8 GB ram
SuSE Linux Enterprise Server 10 (2.6.16 kernel)
Postgresql 8.3.4 (compiled from source)
2)
IBM x3650
IBM ServRaid controller (Rebranded Adaptec card, using the aacraid driver)
8 15k RPM 73GB RAID 10 (OS and Postgres data array)
2 quad core 3.0GHz Intel Xeons
8 GB ram
Mandriva 2009 Free (2.6.27.19 kernel)
Postgresql 8.3.7
As I said, I have the same problem on both machines, I'm expecting that this is caused by the low quality RAID controllers IBM has floged us.
I'm interested to find out whether any one out there has had similar problems with IBM ServRAID controllers, or IBM hardware in general?
What SAS RAID controllers are people using?
What RAID configurations are people using?
What SAS RAID controllers would anyone recommend purchasing?
Any information is gratefully received
Chris Ellis
Shropshire Council
chris.ellis@shropshire.gov.uk
******************************************************************************
If you are not the intended recipient of this email please do not send it on
to others, open any attachments or file the email locally.
Please inform the sender of the error and then delete the original email.
For more information, please refer to http://www.shropshire.gov.uk/privacy.nsf
******************************************************************************
On Tue, Mar 31, 2009 at 3:37 AM, <Chris.Ellis@shropshire.gov.uk> wrote: > > Hi > > Been having interesting times with an IBM x3650 with 8 15k RPM 73GB drives > in RAID 10 and a ServRAID 8K controller with Write-Back cache enabled > (battery installed and working). Currently getting a pgbench score of 4.7 > transactions per second! After playing with the postgresql configuration > file, I'm certain that this is not a postgresql problem. I have tried two > different Linux distro's upon the server both with the same problems. I'm > fairly certain that this is a problem with the hardware configuration / > setup, however I'm still waiting for IBM to contact me! > > Initially I started with the OS on a RAID 1 array and a 6 drive RAID 10 > array for postgresql. With this setup I got 3tps, altering the RAID > configuration to a single 8 drive array, running both the OS and postgresql. > I was able to reach 700tps, however after upgrading to the latest RAID > controller firmware this has now fallen back to 4tps. > > Benchmarking another server I have access to, 4 15k 73GB RPM disks with a > Dell Perc 5/i controller. I consistently get a pgbench score of 1400tps. > Therefore taking a linear extrapolation I expect the IBM x3650 to manage > ~3000tps. Additionally my Laptop with a 5400 RPM sata disk was able to > score ~200tps. SNIP > What SAS RAID controllers are people using? > > What RAID configurations are people using? > > What SAS RAID controllers would anyone recommend purchasing? I am using an Areca 1680 series controller. 16 SAS 15k5 disks. 2 RAID-1, 12 RAID-10, 2 hot spares. 512Meg bbu. RHEL 5.2 I can sustain around 3000 tps with pgbench and 30 minute runs. I'd call IBM and ask them to come pick up their boat anchors.
Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:16:01:
> On Tue, Mar 31, 2009 at 3:37 AM, <Chris.Ellis@shropshire.gov.uk> wrote:
> >
> > Hi
> >
> > Been having interesting times with an IBM x3650 with 8 15k RPM 73GB drives
> > in RAID 10 and a ServRAID 8K controller with Write-Back cache enabled
> > (battery installed and working). Currently getting a pgbench score of 4.7
> > transactions per second! After playing with the postgresql configuration
> > file, I'm certain that this is not a postgresql problem. I have tried two
> > different Linux distro's upon the server both with the same problems. I'm
> > fairly certain that this is a problem with the hardware configuration /
> > setup, however I'm still waiting for IBM to contact me!
> >
> > Initially I started with the OS on a RAID 1 array and a 6 drive RAID 10
> > array for postgresql. With this setup I got 3tps, altering the RAID
> > configuration to a single 8 drive array, running both the OS and postgresql.
> > I was able to reach 700tps, however after upgrading to the latest RAID
> > controller firmware this has now fallen back to 4tps.
> >
> > Benchmarking another server I have access to, 4 15k 73GB RPM disks with a
> > Dell Perc 5/i controller. I consistently get a pgbench score of 1400tps.
> > Therefore taking a linear extrapolation I expect the IBM x3650 to manage
> > ~3000tps. Additionally my Laptop with a 5400 RPM sata disk was able to
> > score ~200tps.
>
> SNIP
>
> > What SAS RAID controllers are people using?
> >
> > What RAID configurations are people using?
> >
> > What SAS RAID controllers would anyone recommend purchasing?
>
> I am using an Areca 1680 series controller. 16 SAS 15k5 disks. 2
> RAID-1, 12 RAID-10, 2 hot spares. 512Meg bbu. RHEL 5.2 I can sustain
> around 3000 tps with pgbench and 30 minute runs.
Thanks for the Info.
>
> I'd call IBM and ask them to come pick up their boat anchors.
My sentiments exactly, unfortunately, I seem stuck with them :(
Chris Ellis
******************************************************************************
If you are not the intended recipient of this email please do not send it on
to others, open any attachments or file the email locally.
Please inform the sender of the error and then delete the original email.
For more information, please refer to http://www.shropshire.gov.uk/privacy.nsf
******************************************************************************
On Tue, Mar 31, 2009 at 8:21 AM, <Chris.Ellis@shropshire.gov.uk> wrote: > > Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:16:01: > >> I'd call IBM and ask them to come pick up their boat anchors. > > My sentiments exactly, unfortunately, I seem stuck with them :( Can you at least source your own RAID controllers?
Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:53:34:
> On Tue, Mar 31, 2009 at 8:21 AM, <Chris.Ellis@shropshire.gov.uk> wrote:
> >
> > Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:16:01:
> >
> >> I'd call IBM and ask them to come pick up their boat anchors.
> >
> > My sentiments exactly, unfortunately, I seem stuck with them :(
>
> Can you at least source your own RAID controllers?
Yes I will be, I never really did trust IBM and I certainly don't now!
I just need to choose the correct RAID card now, good performance at the right price.
Chris Ellis
******************************************************************************
If you are not the intended recipient of this email please do not send it on
to others, open any attachments or file the email locally.
Please inform the sender of the error and then delete the original email.
For more information, please refer to http://www.shropshire.gov.uk/privacy.nsf
******************************************************************************
On Tue, Mar 31, 2009 at 8:54 AM, <Chris.Ellis@shropshire.gov.uk> wrote: > > Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:53:34: > >> On Tue, Mar 31, 2009 at 8:21 AM, <Chris.Ellis@shropshire.gov.uk> wrote: >> > >> > Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:16:01: >> > >> >> I'd call IBM and ask them to come pick up their boat anchors. >> > >> > My sentiments exactly, unfortunately, I seem stuck with them :( >> >> Can you at least source your own RAID controllers? > > Yes I will be, I never really did trust IBM and I certainly don't now! > > I just need to choose the correct RAID card now, good performance at the > right price. The older Areca and 3ware cards are pretty reasonably priced (Areca 12xx series, 3ware 95xx series) and the newer ones aren't too badly priced for the performance you get. What's your budget for the RAID card? Note that you might also be able to get away with the crappy RAID card they sold you if you put it into jbod mode and use software RAID instead. Just guessing.
On Tue, 31 Mar 2009, Chris.Ellis@shropshire.gov.uk wrote: > Been having interesting times with an IBM x3650 with 8 15k RPM 73GB > drives in RAID 10 and a ServRAID 8K controller with Write-Back cache > enabled (battery installed and working). Currently getting a pgbench > score of 4.7 transactions per second! pgbench is a very high level test of your system. It can tell you when a system is doing well, but it's almost useless for figuring out what's wrong if there's a problem. You shouldn't run pgbench until you've first done a lower-level benchmark such as bonnie++ on the hardware. That should give you a better idea what's going on here, and if the badness shows up there it will be much easier to get someone at IBM to pay attention too. -- * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
Chris.Ellis@shropshire.gov.uk wrote: > > Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:53:34: > > > On Tue, Mar 31, 2009 at 8:21 AM, <Chris.Ellis@shropshire.gov.uk> wrote: > > > > > > Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:16:01: > > > > > >> I'd call IBM and ask them to come pick up their boat anchors. > > > > > > My sentiments exactly, unfortunately, I seem stuck with them :( > > > > Can you at least source your own RAID controllers? > > Yes I will be, I never really did trust IBM and I certainly don't now! > > I just need to choose the correct RAID card now, good performance at the > right price. you are jumping to conclusions too quickly - while the 8k is not the worlds fastest raid card available it is really not (that) bad at all. we have plenty of x3650 in production and last time I tested I was easily able to get >>2000tps even on an untuned postgresql install and with fwer disks. So I really think you are looking at another problem here (be it defective hardware or a driver/OS level issue). is your SLES10 install updated to the latest patch levels available and are you running the recommended driver version for that version of SLES? Stefan
Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> wrote on 01/04/2009 06:53:07:
> Chris.Ellis@shropshire.gov.uk wrote:
> >
> > Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:53:34:
> >
> > > On Tue, Mar 31, 2009 at 8:21 AM, <Chris.Ellis@shropshire.gov.uk> wrote:
> > > >
> > > > Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:16:01:
> > > >
> > > >> I'd call IBM and ask them to come pick up their boat anchors.
> > > >
> > > > My sentiments exactly, unfortunately, I seem stuck with them :(
> > >
> > > Can you at least source your own RAID controllers?
> >
> > Yes I will be, I never really did trust IBM and I certainly don't now!
> >
> > I just need to choose the correct RAID card now, good performance at the
> > right price.
>
> you are jumping to conclusions too quickly - while the 8k is not the
> worlds fastest raid card available it is really not (that) bad at all.
> we have plenty of x3650 in production and last time I tested I was
> easily able to get >>2000tps even on an untuned postgresql install and
> with fwer disks.
Could you provide any more information upon your configurations if possible, please?
>
> So I really think you are looking at another problem here (be it
> defective hardware or a driver/OS level issue).
Hardware is always a possiblity, finally managed to get hold of IBM too.
I have tried two different Linux distro's, with different kernels, My current Mandriva test using a fairly upto date kernel.
I may try a custom kernel.
> is your SLES10 install updated to the latest patch levels available and
> are you running the recommended driver version for that version of SLES?
Yes
>
>
> Stefan
******************************************************************************
If you are not the intended recipient of this email please do not send it on
to others, open any attachments or file the email locally.
Please inform the sender of the error and then delete the original email.
For more information, please refer to http://www.shropshire.gov.uk/privacy.nsf
******************************************************************************
Chris.Ellis@shropshire.gov.uk wrote: > > Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> wrote on 01/04/2009 06:53:07: > > > Chris.Ellis@shropshire.gov.uk wrote: > > > > > > Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 15:53:34: > > > > > > > On Tue, Mar 31, 2009 at 8:21 AM, > <Chris.Ellis@shropshire.gov.uk> wrote: > > > > > > > > > > Scott Marlowe <scott.marlowe@gmail.com> wrote on 31/03/2009 > 15:16:01: > > > > > > > > > >> I'd call IBM and ask them to come pick up their boat anchors. > > > > > > > > > > My sentiments exactly, unfortunately, I seem stuck with them :( > > > > > > > > Can you at least source your own RAID controllers? > > > > > > Yes I will be, I never really did trust IBM and I certainly don't now! > > > > > > I just need to choose the correct RAID card now, good performance > at the > > > right price. > > > > you are jumping to conclusions too quickly - while the 8k is not the > > worlds fastest raid card available it is really not (that) bad at all. > > we have plenty of x3650 in production and last time I tested I was > > easily able to get >>2000tps even on an untuned postgresql install and > > with fwer disks. > > Could you provide any more information upon your configurations if > possible, please? x3650, dual quadcore Xeon 5430. Servraid 8k with 256MB-BBWC and likely RAID6 during that testing. OS was/is debian etch/amd64. Don't have the exact (pgbench) test parameters handy anymore though... > > > > > So I really think you are looking at another problem here (be it > > defective hardware or a driver/OS level issue). > > Hardware is always a possiblity, finally managed to get hold of IBM too. > I have tried two different Linux distro's, with different kernels, My > current Mandriva test using a fairly upto date kernel. > I may try a custom kernel. also test with different IO schedulers(especially deadline and noop). Stefan
On Wed, Apr 1, 2009 at 2:01 PM, Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> wrote: > also test with different IO schedulers(especially deadline and noop). But wasn't the OP getting something like 6 tps? I mean, something is so horrifically wrong a simple change like the io scheduler can't hope to fix things.