Thread: Hardware suggestions for high performance 8.3

Hardware suggestions for high performance 8.3

From
Henrik
Date:
Hi list,


We have a database with lots of small simultaneous writes and reads
(millions every day) and are looking at buying a good hardware for this.

What are your suggestions. What we are currently looking at is.

Dual Quad Core Intel
8 - 12 GB RAM

10 disks total.

4 x 146 GB SAS disk in RAID 1+0 for database
6 x 750 GB SATA disks in RAID 1+0 or RAID 5 for OS and transactions
logs.

Good RAID controller with lots of memory and BBU.

Any hints, recommendations would be greatly appreciated.

Cheers,
Henke

Re: Hardware suggestions for high performance 8.3

From
"Claus Guttesen"
Date:
> We have a database with lots of small simultaneous writes and reads
> (millions every day) and are looking at buying a good hardware for this.
>
> What are your suggestions. What we are currently looking at is.
>
> Dual Quad Core Intel
> 8 - 12 GB RAM
>
> 10 disks total.
>
> 4 x 146 GB SAS disk in RAID 1+0 for database
> 6 x 750 GB SATA disks in RAID 1+0 or RAID 5 for OS and transactions logs.
>
> Good RAID controller with lots of memory and BBU.

I have very positive experiences with HP's DL360 and DL380. The latter
slightly more expandable (2U vs. 1U). I have used the internal
p400i-controller with 512 MB cache on the DL380 and bought an external
p800-controller (512 MB cache as well) and a MSA-70-cabinet. I've have
11 disks in raid-6 (one hotspare).

I don't see any reason to mix sas- and sata-disks with different
sizes. I'd go for sas-disks, smaller and faster, less power and heat.
Raid 1+0 or raid-6 does not seem to make much of a difference today as
it used to if you have more than 6-7 disks.

The DL380 is a 4-way woodcrest at 3 GHz and 16 GB ram and the DL360 is
a two-way woodcrest at 2.66 GHz with 16 GB.

My personal preference is FreeBSD and the DL3x0-servers all run
without problems on this platform. But choose your OS depending on
what you're most comfortable with. And choose hardware according to
what your OS supports.

Areca-controllers may also be worth looking into but I haven't tried
these myself.

Our largest table has 85 mill. entries.

--
regards
Claus

When lenity and cruelty play for a kingdom,
the gentlest gamester is the soonest winner.

Shakespeare

Re: Hardware suggestions for high performance 8.3

From
Matthew Wakeling
Date:
On Wed, 25 Jun 2008, Henrik wrote:
> What are your suggestions. What we are currently looking at is.
>
> Dual Quad Core Intel
> 8 - 12 GB RAM

More RAM would be helpful. It's not that expensive, compared to the rest
of your system.

> 10 disks total.
>
> 4 x 146 GB SAS disk in RAID 1+0 for database
> 6 x 750 GB SATA disks in RAID 1+0 or RAID 5 for OS and transactions logs.
>
> Good RAID controller with lots of memory and BBU.

If you have a good RAID controller with BBU cache, then there's no point
splitting the discs into two sets. You're only creating an opportunity to
under-utilise the system. I'd get ten identical discs and put them in a
single array, probably RAID 10.

Also, do you really need 6*750GB for OS and transaction logs? How big can
they be?

However, the most important factor is that you get a good BBU cache.

Matthew

--
I don't want the truth. I want something I can tell parliament!
                                              -- Rt. Hon. Jim Hacker MP

Re: Hardware suggestions for high performance 8.3

From
Henrik
Date:
25 jun 2008 kl. 12.56 skrev Claus Guttesen:

>> We have a database with lots of small simultaneous writes and reads
>> (millions every day) and are looking at buying a good hardware for
>> this.
>>
>> What are your suggestions. What we are currently looking at is.
>>
>> Dual Quad Core Intel
>> 8 - 12 GB RAM
>>
>> 10 disks total.
>>
>> 4 x 146 GB SAS disk in RAID 1+0 for database
>> 6 x 750 GB SATA disks in RAID 1+0 or RAID 5 for OS and transactions
>> logs.
>>
>> Good RAID controller with lots of memory and BBU.
>
> I have very positive experiences with HP's DL360 and DL380. The latter
> slightly more expandable (2U vs. 1U). I have used the internal
> p400i-controller with 512 MB cache on the DL380 and bought an external
> p800-controller (512 MB cache as well) and a MSA-70-cabinet. I've have
> 11 disks in raid-6 (one hotspare).
Mmm I've used DL380 and I also have had good experience with them.

I guess that the nees of splitting up the transactions logs are not
that important if you have enought disks in a raid 10 or raid 6.
>

> My personal preference is FreeBSD and the DL3x0-servers all run
> without problems on this platform. But choose your OS depending on
> what you're most comfortable with. And choose hardware according to
> what your OS supports.
>
I like BDS also but this time its a 64bit Linux system which wil be
used.
>
>
> Our largest table has 85 mill. entries.
>
I believe we will be running in the 200 mill. area.

Thanks for your input!

//Henke

Re: Hardware suggestions for high performance 8.3

From
Henrik
Date:
25 jun 2008 kl. 13.15 skrev Matthew Wakeling:

> On Wed, 25 Jun 2008, Henrik wrote:
>> What are your suggestions. What we are currently looking at is.
>>
>> Dual Quad Core Intel
>> 8 - 12 GB RAM
>
> More RAM would be helpful. It's not that expensive, compared to the
> rest of your system.

True, as long as I can build the system on 2G or 4G modules I can max
out the banks.

>
>
>> 10 disks total.
>>
>> 4 x 146 GB SAS disk in RAID 1+0 for database
>> 6 x 750 GB SATA disks in RAID 1+0 or RAID 5 for OS and transactions
>> logs.
>>
>> Good RAID controller with lots of memory and BBU.
>
> If you have a good RAID controller with BBU cache, then there's no
> point splitting the discs into two sets. You're only creating an
> opportunity to under-utilise the system. I'd get ten identical discs
> and put them in a single array, probably RAID 10.
OK, thats good to know. Really want to keep it as simple as possible.
Would you turn off fsync if you had a controller with BBU? =)

>
>
> Also, do you really need 6*750GB for OS and transaction logs? How
> big can they be?

Ahh, we are going to save a lot of other datafiles on those also but
maybe i'll just get a cabinett.

>
>
> However, the most important factor is that you get a good BBU cache.
Here that!

Thanks for your input!

//Henke

Re: Hardware suggestions for high performance 8.3

From
"Claus Guttesen"
Date:
>> If you have a good RAID controller with BBU cache, then there's no point
>> splitting the discs into two sets. You're only creating an opportunity to
>> under-utilise the system. I'd get ten identical discs and put them in a
>> single array, probably RAID 10.
>
> OK, thats good to know. Really want to keep it as simple as possible. Would
> you turn off fsync if you had a controller with BBU? =)

No, don't do that. Leaving this setting on is *highly* recommended
unless you have data which can easily be reproduced. :-)

--
regards
Claus

When lenity and cruelty play for a kingdom,
the gentlest gamester is the soonest winner.

Shakespeare

Re: Hardware suggestions for high performance 8.3

From
Matthew Wakeling
Date:
On Wed, 25 Jun 2008, Henrik wrote:
> Would you turn off fsync if you had a controller with BBU? =)

No, certainly not. Fsync is what makes the data move from the volatile OS
cache to the non-volatile disc system. It'll just be a lot quicker on a
controller with a BBU cache, because it won't need to actually wait for
the discs. But you still need the fsync to move the data from main OS
cache to BBU cache.

>> Also, do you really need 6*750GB for OS and transaction logs? How big can
>> they be?
>
> Ahh, we are going to save a lot of other datafiles on those also but maybe
> i'll just get a cabinett.

Or you could just get 10 large SATA drives. To be honest, the performance
difference is not large, especially if you ensure the database data is
held compactly on the discs, so the seeks are small.

Matthew

--
It's one of those irregular verbs - "I have an independent mind," "You are
an eccentric," "He is round the twist."
                                      -- Bernard Woolly, Yes Prime Minister

Re: Hardware suggestions for high performance 8.3

From
Greg Smith
Date:
On Wed, 25 Jun 2008, Henrik wrote:

> 4 x 146 GB SAS disk in RAID 1+0 for database
> 6 x 750 GB SATA disks in RAID 1+0 or RAID 5 for OS and transactions logs.

The transaction logs are not that big, and there's very little value to
striping them across even two disks.  You should just get more SAS disks
instead and make them available to the database, adding more spindles for
random I/O is much more important.  Separating out a single RAID-1 pair
from that set to hold the logs is a reasonable practice, with a good
battery-backed controller even that might not buy you anything useful.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

Re: Hardware suggestions for high performance 8.3

From
Greg Smith
Date:
On Wed, 25 Jun 2008, Henrik wrote:

> Would you turn off fsync if you had a controller with BBU? =)

Turning off fsync has some potential to introduce problems even in that
environment, so better not to do that.  The issue is that you might have,
say, 1GB of OS-level cache but 256MB of BBU cache, and if you turn fsync
off it won't force the OS cache out to the controller when it's supposed
to and that can cause corruption.

Also, if you've got a controller with BBU, the overhead of fsync for
regular writes is low enough that you don't really need to turn it off.
If writes are cached the fsync is almost free.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

Re: Hardware suggestions for high performance 8.3

From
Henrik
Date:
25 jun 2008 kl. 17.45 skrev Greg Smith:

> On Wed, 25 Jun 2008, Henrik wrote:
>
>> Would you turn off fsync if you had a controller with BBU? =)
>
> Turning off fsync has some potential to introduce problems even in
> that environment, so better not to do that.  The issue is that you
> might have, say, 1GB of OS-level cache but 256MB of BBU cache, and
> if you turn fsync off it won't force the OS cache out to the
> controller when it's supposed to and that can cause corruption.
>
> Also, if you've got a controller with BBU, the overhead of fsync for
> regular writes is low enough that you don't really need to turn it
> off. If writes are cached the fsync is almost free.
Thanks for a thoroughly answer. I guess I wont be turning of fsync. :)

Thanks Greg!



>
>
> --
> * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com
> Baltimore, MD
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org
> )
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance


Re: Hardware suggestions for high performance 8.3

From
Henrik
Date:
I've seen some concerns about buying database performance hardware
from DELL. Are there at least some of the RAID cards that work well
with Linux or should I stay clear of DELL permanently?

Thanks!

//Henke
25 jun 2008 kl. 17.45 skrev Greg Smith:

> On Wed, 25 Jun 2008, Henrik wrote:
>
>> Would you turn off fsync if you had a controller with BBU? =)
>
> Turning off fsync has some potential to introduce problems even in
> that environment, so better not to do that.  The issue is that you
> might have, say, 1GB of OS-level cache but 256MB of BBU cache, and
> if you turn fsync off it won't force the OS cache out to the
> controller when it's supposed to and that can cause corruption.
>
> Also, if you've got a controller with BBU, the overhead of fsync for
> regular writes is low enough that you don't really need to turn it
> off. If writes are cached the fsync is almost free.
>
> --
> * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com
> Baltimore, MD
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org
> )
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance


Re: Hardware suggestions for high performance 8.3

From
Greg Smith
Date:
On Thu, 26 Jun 2008, Henrik wrote:

> I've seen some concerns about buying database performance hardware from DELL.
> Are there at least some of the RAID cards that work well with Linux or should
> I stay clear of DELL permanently?

People seem to be doing OK if the RAID card is their Perc/6i, which has an
LSI Logic MegaRAID SAS 1078 chipset under the hood.  There's some helpful
benchmark results and follow-up meesages related to one of those at
http://archives.postgresql.org/message-id/47D8B3B6.3010606@emolecules.com

That said, I consider the rebranded LSI cards a pain and hate the quality
of Dell's hardware.  Seems like everybody I talk to lately is buying HP's
DL380 instead of Dells for this level of Linux installs nowadays, I
haven't gotten one of those HP boxes myself yet to comment.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

Re: Hardware suggestions for high performance 8.3

From
"Scott Marlowe"
Date:
On Thu, Jun 26, 2008 at 10:47 PM, Greg Smith <gsmith@gregsmith.com> wrote:
> On Thu, 26 Jun 2008, Henrik wrote:
>
>> I've seen some concerns about buying database performance hardware from
>> DELL. Are there at least some of the RAID cards that work well with Linux or
>> should I stay clear of DELL permanently?
>
> People seem to be doing OK if the RAID card is their Perc/6i, which has an
> LSI Logic MegaRAID SAS 1078 chipset under the hood.  There's some helpful
> benchmark results and follow-up meesages related to one of those at
> http://archives.postgresql.org/message-id/47D8B3B6.3010606@emolecules.com

Yeah, the problems I've had have been with the internal RAID (perc
5???) lsi based controllers.  They kick their drives offline.  Dell
has a firmware update but we haven't had a chance to install it just
yet to see if it fixes the problem with that one.

> That said, I consider the rebranded LSI cards a pain and hate the quality of
> Dell's hardware.

Yeah, I'd just as soon get a regular LSI bios as the remade one Dell
seems intent on pushing.  Also, we just discovered the broadcom
chipsets we have in our Dell 1950s and 1850s will not negotiate to
gigabit with our Nortel switches.  Everything else I've plugged in
just worked.  Went looking at Dell's site, and for the 1950 they
recommend buying a dual port Intel NIC for it.  Why couldn't they just
build in better NICS to start?

Re: Hardware suggestions for high performance 8.3

From
"Joshua D. Drake"
Date:
Greg Smith wrote:
> On Thu, 26 Jun 2008, Henrik wrote:
>
>> I've seen some concerns about buying database performance hardware
>> from DELL. Are there at least some of the RAID cards that work well
>> with Linux or should I stay clear of DELL permanently?
>
> People seem to be doing OK if the RAID card is their Perc/6i, which has
> an LSI Logic MegaRAID SAS 1078 chipset under the hood.  There's some
> helpful benchmark results and follow-up meesages related to one of those
> at
> http://archives.postgresql.org/message-id/47D8B3B6.3010606@emolecules.com
>
> That said, I consider the rebranded LSI cards a pain and hate the
> quality of Dell's hardware.  Seems like everybody I talk to lately is
> buying HP's DL380 instead of Dells for this level of Linux installs
> nowadays, I haven't gotten one of those HP boxes myself yet to comment.
>

The HP P800 controller is a top notch performer.

Joshua D. Drake

P.S. The DL360-380 series is very nice as well

> --
> * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
>