Thread: RAID card recommendation

We're about to purchase a new server to store some of our old databases,
and I was wondering if someone could advise me on a RAID card. We want to
make a 6-drive SATA RAID array out of 2TB drives, and it will be RAID 5 or
6 because there will be zero write traffic. The priority is stuffing as
much storage into a small 2U rack as possible, with performance less
important. We will be running Debian Linux.

People have mentioned Areca as making good RAID controllers. We're looking
at the "Areca ARC-1220 PCI-Express x8 SATA II" as a possibility. Does
anyone have an opinion on whether it is a turkey or a star?

Another possibility is a 3-ware card of some description.

Thanks in advance,

Matthew

--
 Now you see why I said that the first seven minutes of this section will have
 you looking for the nearest brick wall to beat your head against. This is
 why I do it at the end of the lecture - so I can run.
                                        -- Computer Science lecturer

Matthew Wakeling wrote:
>
> We're about to purchase a new server to store some of our old databases,
> and I was wondering if someone could advise me on a RAID card. We want
> to make a 6-drive SATA RAID array out of 2TB drives, and it will be RAID
> 5 or 6 because there will be zero write traffic. The priority is
> stuffing as much storage into a small 2U rack as possible, with
> performance less important. We will be running Debian Linux.
>
> People have mentioned Areca as making good RAID controllers. We're
> looking at the "Areca ARC-1220 PCI-Express x8 SATA II" as a possibility.
> Does anyone have an opinion on whether it is a turkey or a star?
>
> Another possibility is a 3-ware card of some description.
>

Do you actually need a RAID card at all? It's just another point of
failure: the Linux software raid (mdadm) is pretty good.

Also, be very wary of RAID5 for an array that size. It is highly
probable that, if one disk has failed, then during the recovery process,
you may lose a second disk. The unrecoverable error rate on standard
disks is about 1 in 10^14 bits; your disk array is 10^11 bits in size...

We got bitten by this....

Richard

----- "Richard Neill" <rn214@cam.ac.uk> escreveu:

> Matthew Wakeling wrote:
> >
> > We're about to purchase a new server to store some of our old
> databases,
> > and I was wondering if someone could advise me on a RAID card. We
> want
> > to make a 6-drive SATA RAID array out of 2TB drives, and it will be
> RAID
> > 5 or 6 because there will be zero write traffic. The priority is
> > stuffing as much storage into a small 2U rack as possible, with
> > performance less important. We will be running Debian Linux.
> >
> > People have mentioned Areca as making good RAID controllers. We're
> > looking at the "Areca ARC-1220 PCI-Express x8 SATA II" as a
> possibility.
> > Does anyone have an opinion on whether it is a turkey or a star?
> >
> > Another possibility is a 3-ware card of some description.
> >
>
> Do you actually need a RAID card at all? It's just another point of
> failure: the Linux software raid (mdadm) is pretty good.
>
> Also, be very wary of RAID5 for an array that size. It is highly
> probable that, if one disk has failed, then during the recovery
> process,
> you may lose a second disk. The unrecoverable error rate on standard
> disks is about 1 in 10^14 bits; your disk array is 10^11 bits in
> size...
>
> We got bitten by this....
>
> Richard

Linux kernel software RAID is fully supported in Debian Lenny, is quite cheap to implement and powerful.
I would avoid SATA disks but it's just me. SAS controllers and disks are expensive but worth every penny spent on them.

Prefer RAID 1+0 over RAID 5 not only because of the risk of failure of a second disk, but I have 3 cases of performance
issuescaused by RAID 5. 
It's said that performance is not the problem but think twice because a good application tends to scale fast to several
users.
Of course, keep a good continuous backup strategy of your databases and don't trust just the mirroring of disks in a
RAIDfashion. 

Flavio Henrique A. Gurgel
Consultor -- 4Linux
tel. 55-11-2125.4765
fax. 55-11-2125.4777
www.4linux.com.br

On Tue, Nov 24, 2009 at 10:23 AM, Matthew Wakeling <matthew@flymine.org> wrote:
>
> We're about to purchase a new server to store some of our old databases, and
> I was wondering if someone could advise me on a RAID card. We want to make a
> 6-drive SATA RAID array out of 2TB drives, and it will be RAID 5 or 6
> because there will be zero write traffic. The priority is stuffing as much
> storage into a small 2U rack as possible, with performance less important.
> We will be running Debian Linux.
>
> People have mentioned Areca as making good RAID controllers. We're looking
> at the "Areca ARC-1220 PCI-Express x8 SATA II" as a possibility. Does anyone
> have an opinion on whether it is a turkey or a star?

We run a 12xx series on our office server in RAID-6 over 8 1TB 7200RPM
server class SATA drives. Our production server runs the 1680 on top
of 16 15k5 seagates in RAID-10.  The performance difference between
these two are enormous.  Things that take minutes on the production
server can take hours on the office server.  Production handles
1.5Million users, office handles 20 or 30 users.

I've been really happy with the reliability of the 12xx card here at
work.  100% uptime for a year, that machine goes down for kernel
updates and only that.  But it's not worked that hard all day
everyday, so I can't compare its reliability with production in
RAID-10 which has had one drive fail the week it was delivered and
none since in 400+days.  We have two hot spares there.

> Another possibility is a 3-ware card of some description.

They get good reviews as well.  Both manufacturers have their "star"
performers, and their "utility" or work group class controllers.  For
what you're doing the areca 12xx or 3ware 95xx series should do fine.

As far as drives go we've been really happy with WD of late, they make
large enterprise class SATA drives that don't pull a lot of power
(green series) and fast SATA drives that pull a bit more but are
faster (black series).  We've used both and are quite happy with each.
 We use a pair of blacks to build slony read slaves and they're very
fast, with write speeds of ~100MB/second and read speeds double that
in linux under sw RAID-1

Since I'm currently looking at upgrading my own database server, maybe some
of the experts can give a comment on one of the following controllers:

- Promise Technology Supertrak ES4650 + additional BBU
- Adaptec RAID 5405 SGL/256 SATA/SAS + additional BBU
- Adaptec RAID 5405Z SGL/512 SATA/SAS

My personal favourite currently is the 5405Z, since it does not require
regular battery replacements and because it has 512MB of cache.

Since my server only has room for four disks, I'd choose the following
one:

- Seagate Cheetah 15K.6 147GB SAS

Drives would be organized as RAID-0 for fast access, I do not need
terabytes of storage.

The database currently is about 150 GB in size (including indexes), the
main table having a bit less than 1 billion rows (maximum will be about 2
billion) and getting about 10-20 million updates per day, so update speed
is critical.

Currently the database is running on a mdadm raid-0 with four S-ATA drives
(7.2k rpm), which was ok when the database was half this size...

Operating System is Gentoo Linux 2.6.31-r1 on a Fujitsu Siemens Primergy
200 S2 (2xXEON @ 1.6 GHz) with 4 GB of RAM (which also would be increased
to its maximum of 8 GB during the above update)


--
Jochen Erwied     |   home: jochen@erwied.eu     +49-208-38800-18, FAX: -19
Sauerbruchstr. 17 |   work: joe@mbs-software.de  +49-2151-7294-24, FAX: -50
D-45470 Muelheim  | mobile: jochen.erwied@vodafone.de       +49-173-5404164

On Tue, Nov 24, 2009 at 12:28 PM, Jochen Erwied
<jochen@pgsql-performance.erwied.eu> wrote:
>
> Since I'm currently looking at upgrading my own database server, maybe some
> of the experts can give a comment on one of the following controllers:
>
> - Promise Technology Supertrak ES4650 + additional BBU
> - Adaptec RAID 5405 SGL/256 SATA/SAS + additional BBU
> - Adaptec RAID 5405Z SGL/512 SATA/SAS
>
> My personal favourite currently is the 5405Z, since it does not require
> regular battery replacements and because it has 512MB of cache.

Have you searched the -performance archives for references to them?
I'm not that familiar with Adaptec RAID controllers.  Not requiring a
battery check / replacement is nice.

> Since my server only has room for four disks, I'd choose the following
> one:
>
> - Seagate Cheetah 15K.6 147GB SAS

We use the older gen 15k.5 and have been very happy with them.
Nowadays it seems the fastest Seagates and Hitachis own the market for
super fast drives.

> Drives would be organized as RAID-0 for fast access, I do not need
> terabytes of storage.

So, you're willing (or forced by economics) to suffer downtime due to
drive failure every so often.

> The database currently is about 150 GB in size (including indexes), the
> main table having a bit less than 1 billion rows (maximum will be about 2
> billion) and getting about 10-20 million updates per day, so update speed
> is critical.

So, assuming this means an 8 hour work day for ~20M rows, you're
looking at around 700 per second.

> Currently the database is running on a mdadm raid-0 with four S-ATA drives
> (7.2k rpm), which was ok when the database was half this size...
>
> Operating System is Gentoo Linux 2.6.31-r1 on a Fujitsu Siemens Primergy
> 200 S2 (2xXEON @ 1.6 GHz) with 4 GB of RAM (which also would be increased
> to its maximum of 8 GB during the above update)

I'd definitely test the heck out of whatever RAID card you're buying
to make sure it performs well enough.  For some loads and against some
HW RAID cards, SW RAID might be the winner.

Another option might be a JBOD box attached to the machine that holds
12 or so 2.5" 15k like the hitachi ultrastar 147G 2.5" drives.  This
sounds like a problem you need to be able to throw a lot of drives at
at one time.  Is it likely to grow much after this?

Gurgel, Flavio escribió:
> ----- "Richard Neill" <rn214@cam.ac.uk> escreveu:
>
>
>> Matthew Wakeling wrote:
>>
>>> We're about to purchase a new server to store some of our old
>>>
>> databases,
>>
>>> and I was wondering if someone could advise me on a RAID card. We
>>>
>> want
>>
>>> to make a 6-drive SATA RAID array out of 2TB drives, and it will be
>>>
>> RAID
>>
>>> 5 or 6 because there will be zero write traffic. The priority is
>>> stuffing as much storage into a small 2U rack as possible, with
>>> performance less important. We will be running Debian Linux.
>>>
>>> People have mentioned Areca as making good RAID controllers. We're
>>> looking at the "Areca ARC-1220 PCI-Express x8 SATA II" as a
>>>
>> possibility.
>>
>>> Does anyone have an opinion on whether it is a turkey or a star?
>>>
>>> Another possibility is a 3-ware card of some description.
>>>
>>>
>> Do you actually need a RAID card at all? It's just another point of
>> failure: the Linux software raid (mdadm) is pretty good.
>>
>> Also, be very wary of RAID5 for an array that size. It is highly
>> probable that, if one disk has failed, then during the recovery
>> process,
>> you may lose a second disk. The unrecoverable error rate on standard
>> disks is about 1 in 10^14 bits; your disk array is 10^11 bits in
>> size...
>>
>> We got bitten by this....
>>
>> Richard
>>
>
> Linux kernel software RAID is fully supported in Debian Lenny, is quite cheap to implement and powerful.
> I would avoid SATA disks but it's just me. SAS controllers and disks are expensive but worth every penny spent on
them.
>
> Prefer RAID 1+0 over RAID 5 not only because of the risk of failure of a second disk, but I have 3 cases of
performanceissues caused by RAID 5. 
> It's said that performance is not the problem but think twice because a good application tends to scale fast to
severalusers. 
> Of course, keep a good continuous backup strategy of your databases and don't trust just the mirroring of disks in a
RAIDfashion. 
>
> Flavio Henrique A. Gurgel
> Consultor -- 4Linux
> tel. 55-11-2125.4765
> fax. 55-11-2125.4777
> www.4linux.com.br
>
>
>
Do you expose that performance issued caused by RAID 5? Because this is
one of our solutions here on my country to save the data of our
PostgreSQL database. Which model do you recommend ? RAID 0,RAID 1, RAID
5 or RAID 10?

On Tue, Nov 24, 2009 at 1:37 PM, Ing. Marcos Ortiz Valmaseda
<mlortiz@uci.cu> wrote:
> Do you expose that performance issued caused by RAID 5? Because this is one
> of our solutions here on my country to save the data of our PostgreSQL
> database. Which model do you recommend ? RAID 0,RAID 1, RAID 5 or RAID 10?

RAID-1 or RAID-10 are the default, mostly safe choices.

For disposable dbs, RAID-0 is fine.

For very large dbs with very little writing and mostly reading and on
a budget, RAID-6 is ok.

In most instances I never recommend RAID-5 anymore.

Tuesday, November 24, 2009, 9:05:28 PM you wrote:

> Have you searched the -performance archives for references to them?
> I'm not that familiar with Adaptec RAID controllers.  Not requiring a
> battery check / replacement is nice.

Either I searched for the wrong terms, or there isn't really that much
reference on RAID-controllers on this list. Aberdeen is menthioned once and
looks interesting, but I didn't find a reseller in Germany. As far as I see
from the list, Promise and Adaptec both seem to be not too bad choices.

> So, you're willing (or forced by economics) to suffer downtime due to
> drive failure every so often.

I haven't experienced any downtime due to a disk failure for quite a while
now (call me lucky), although I had a really catastrophic experience with a
RAID-5 some time ago (1 drive crashed, the second one during rebuild :-()

But for this application losing one day of updates is not a big deal, and
downtime isn't either. It's a long running project of mine, with growing
storage needs, but not with 100% of integrity or uptime.

> So, assuming this means an 8 hour work day for ~20M rows, you're
> looking at around 700 per second.

It's an automated application running 24/7, so I require 'only' about
200-250 updates per second.

> I'd definitely test the heck out of whatever RAID card you're buying
> to make sure it performs well enough.  For some loads and against some
> HW RAID cards, SW RAID might be the winner.

Well, I haven't got so much opportunities to test out different kind of
hardware, so I have to rely on experience or reports.

> Another option might be a JBOD box attached to the machine that holds
> 12 or so 2.5" 15k like the hitachi ultrastar 147G 2.5" drives.  This
> sounds like a problem you need to be able to throw a lot of drives at
> at one time.  Is it likely to grow much after this?

JBOD in an external casing would be an alternative, especially when using
an external case. And no, the database will not grow too much after
reaching its final size.

But looking at the prices for anything larger than 4+1 drives in an
external casing is not funny at all :-(

--
Jochen Erwied     |   home: jochen@erwied.eu     +49-208-38800-18, FAX: -19
Sauerbruchstr. 17 |   work: joe@mbs-software.de  +49-2151-7294-24, FAX: -50
D-45470 Muelheim  | mobile: jochen.erwied@vodafone.de       +49-173-5404164

----- "Scott Marlowe" <scott.marlowe@gmail.com> escreveu:

> On Tue, Nov 24, 2009 at 1:37 PM, Ing. Marcos Ortiz Valmaseda
> <mlortiz@uci.cu> wrote:
> > Do you expose that performance issued caused by RAID 5? Because this
> is one
> > of our solutions here on my country to save the data of our
> PostgreSQL
> > database. Which model do you recommend ? RAID 0,RAID 1, RAID 5 or
> RAID 10?
>
> RAID-1 or RAID-10 are the default, mostly safe choices.
>
> For disposable dbs, RAID-0 is fine.
>
> For very large dbs with very little writing and mostly reading and on
> a budget, RAID-6 is ok.
>
> In most instances I never recommend RAID-5 anymore.

I would never recommend RAID-5 for database customers (any database system), some of the current ones are using it and
theworst nightmares in disk performance are related to RAID-5. 
As Scott said, RAID-1 is safe, RAID-0 is fast (and accept more request load too), RAID-10 is a great combination of
bothworlds. 

Flavio Henrique A. Gurgel
Consultor -- 4Linux
tel. 55-11-2125.4765
fax. 55-11-2125.4777
www.4linux.com.br

On Tue, Nov 24, 2009 at 1:59 PM, Jochen Erwied
<jochen@pgsql-performance.erwied.eu> wrote:
> Tuesday, November 24, 2009, 9:05:28 PM you wrote:
>
>> Have you searched the -performance archives for references to them?
>> I'm not that familiar with Adaptec RAID controllers.  Not requiring a
>> battery check / replacement is nice.
>
> Either I searched for the wrong terms, or there isn't really that much
> reference on RAID-controllers on this list. Aberdeen is menthioned once and
> looks interesting, but I didn't find a reseller in Germany. As far as I see
> from the list, Promise and Adaptec both seem to be not too bad choices.

Aberdeen is the builder I use.  They'll put any card in you want
(within reason) including our preference here, Areca.  Perhaps you
meant Areca?

>> So, assuming this means an 8 hour work day for ~20M rows, you're
>> looking at around 700 per second.
>
> It's an automated application running 24/7, so I require 'only' about
> 200-250 updates per second.

Oh, much better.  A decent hardware RAID controller with battery
backed cache could handle that load with a pair of spinning 15k drives
in RAID-1 probably.

>> Another option might be a JBOD box attached to the machine that holds
>> 12 or so 2.5" 15k like the hitachi ultrastar 147G 2.5" drives.  This
>> sounds like a problem you need to be able to throw a lot of drives at
>> at one time.  Is it likely to grow much after this?
>
> JBOD in an external casing would be an alternative, especially when using
> an external case. And no, the database will not grow too much after
> reaching its final size.

Yeah, if it's not gonna grow a lot more after the 2B rows, then you
probably won't need an external case.

The problem with RAID-5 or RAID-6 is not the normal speed operation, it's the degraded performance when there is a drive failure. This includes read-only scenarios. A DB server getting any kind of real use will effectively appear to be down to client apps if it loses a drive from that RAID set.

Basically, think of RAID-5/6 as RAID-0 but with much slower writes, and a way to recover the data without going to backup tapes if there is a disc loss. It is NOT a solution for staying up in case of a failure.

Presumably, there is a business reason that you're thinking of using RAID-5/6 with hardware RAID and maybe a hot spare, rather than software RAID-0 which would save you 2-3 spindles of formatted capacity, plus the cost of the RAID card. Whatever that reason is, it's also a reason to use RAID-10.

If you absolutely need it to fit in 2U of rack space, you can get a 2U server with a bunch of 2.5" spindles and with 24x 500GB SATA you can get the same formatted size with RAID-10; or you can use an external SAS expander to put additional 3.5" drives in another enclosure.

If we're taking rackmount server RAID card votes, I've had good experiences with the LSI 8888 under Linux.

Cheers
Dave

Tuesday, November 24, 2009, 10:34:00 PM you wrote:

> Aberdeen is the builder I use.  They'll put any card in you want
> (within reason) including our preference here, Areca.  Perhaps you
> meant Areca?

I knew Areca only for their internal arrays (which one of our customers
uses for his 19" systems), but did not know they manufacture their own
controllers. Added the ARC-1212+BBU to my wishlist :-)


--
Jochen Erwied     |   home: jochen@erwied.eu     +49-208-38800-18, FAX: -19
Sauerbruchstr. 17 |   work: joe@mbs-software.de  +49-2151-7294-24, FAX: -50
D-45470 Muelheim  | mobile: jochen.erwied@vodafone.de       +49-173-5404164

<br /> Jochen Erwied wrote: <blockquote cite="mid:1193522694.20091125000240@erwied.eu" type="cite"><pre
wrap="">Tuesday,November 24, 2009, 10:34:00 PM you wrote:
 
 </pre><blockquote type="cite"><pre wrap="">Aberdeen is the builder I use.  They'll put any card in you want
(within reason) including our preference here, Areca.  Perhaps you
meant Areca?   </pre></blockquote><pre wrap="">
I knew Areca only for their internal arrays (which one of our customers
uses for his 19" systems), but did not know they manufacture their own
controllers. Added the ARC-1212+BBU to my wishlist :-)</pre></blockquote><br /> For what it's worth I'm using the
Adaptec5445Z on my new server (don't have Postgre running on it) and have been happy with it.  For storage on my
serversI use <a class="moz-txt-link-freetext"
href="http://www.pc-pitstop.com/sas_cables_enclosures/scsase16.asp">http://www.pc-pitstop.com/sas_cables_enclosures/scsase16.asp</a>
whichhas an Areca ARC-8020 expander in it. With the 5445Z I use the 4 internal ports for a fast RAID0 "working array"
with450G Seagate 15k6 drives and the external goes to the 16 drive enclosure through the expander.  With 16 drives you
havea lot of possibilities for configuring arrays.  I have another server with an Adaptec 52445 (don't have Postgre
runningon it either) connected to two of the 16 drive enclosures and am happy with it.  I'm running Postgre on my
workstationthat has an Adaptec 52445 hooked up to two EnhanceBox-E8MS (<a class="moz-txt-link-freetext"
href="http://www.enhance-tech.com/products/desktop/E8_Series.html">http://www.enhance-tech.com/products/desktop/E8_Series.html</a>). 
Ihave 8 ST373455SS drives in my tower and 8 in the EnhanceBox so my database is running off 16 drives in RAID5. 
Everyonecomplains about RAID5 but it works for me in my situation.  Very very rarely am I waiting on the disks when
runningqueries. The other EnhanceBox has 8 ST31000640SS drives in RAID5 just for backup images.  All 24 drives run off
the52445 and again, I've been satisfied with it.  I've also been happy with the Enhance Technology products.  Sorry for
beingso long but just wanted to put a plug in for the Adaptec cards and let you know about the external options.  The 5
seriescards are a huge improvement over the 3 series. I had a 3805 and wasn't that impressed.  It's actually sitting on
myshelf now collecting dust.<br /><br /> Bob<br /><br /><br /> 

Matthew Wakeling wrote:
> People have mentioned Areca as making good RAID controllers. We're
> looking at the "Areca ARC-1220 PCI-Express x8 SATA II" as a
> possibility. Does anyone have an opinion on whether it is a turkey or
> a star?
Performance should be OK but not great compared with some of the newer
alternatives (this design is a few years old now).  The main issue I've
had with this series of cards is that the command-line tools are very
hit or miss.  See
http://notemagnet.blogspot.com/2008/08/linux-disk-failures-areca-is-not-so.html
for a long commentary about the things I was disappointed by on the
similar ARC-1210 once I actually ran into a drive failure on one.  As
Scott points out there, they have other cards with a built-in management
NIC that allows an alternate management path, and I believe those have
better performance too.

> Another possibility is a 3-ware card of some description.
I've put a fair number of 9690SA cards in systems with little to
complain about.  Performance was reasonable as long as you make sure to
tweak the read-ahead:  http://www.3ware.com/kb/article.aspx?id=11050
Ignore most of the rest of their advice on that page though--for
example, increasing vm.dirty_background_ratio and vm.dirty_ratio is an
awful idea for PostgreSQL use, where if anything you want to decrease
the defaults.

Also, while they claim you can connect SAS drives to these cards, they
don't support sending SMART commands to them and support seemed pretty
limited overall for them.  Stick with plain on SATA ones.

--
Greg Smith    2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com  www.2ndQuadrant.com

Scott Marlowe wrote:
> As far as drives go we've been really happy with WD of late, they make
> large enterprise class SATA drives that don't pull a lot of power
> (green series) and fast SATA drives that pull a bit more but are
> faster (black series).
Be careful to note the caveat that you need their *enterprise class*
drives.  When you run into an error on their regular consumer drives,
they get distracted for a while trying to cover the whole thing up, in a
way that's exactly the opposite of the behavior you want for a RAID
configuration.  I have a regular consumer WD drive that refuses to admit
that it has a problem such that I can RMA it, but that always generates
an error if I rewrite the whole drive.  The behavior of the firmware is
downright shameful.  As cheap consumer drives go, I feel like WD has
pulled ahead of everybody else on performance and possibly even actual
reliability, but the error handling of their firmware is so bad I'm
still using Seagate drives--when those fail, as least they're honest
about it.

--
Greg Smith    2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com  www.2ndQuadrant.com

Jochen Erwied wrote:
> - Promise Technology Supertrak ES4650 + additional BBU
> - Adaptec RAID 5405 SGL/256 SATA/SAS + additional BBU
> - Adaptec RAID 5405Z SGL/512 SATA/SAS
>
I've never seen a Promise controller that had a Linux driver you would
want to rely on under any circumstances.  Adaptec used to have seriously
bad Linux drivers too.  I've gotten the impression they've cleaned up
their act considerably the last few years, but they've been on my list
of hardware to shun for so long I haven't bothered investigating.
Easier to just buy from a company that has always cared about good Linux
support, like 3ware.  In any case, driver quality is what you want to
research before purchasing any of these; doesn't matter how fast the
cards are if they crash or corrupt your data.

What I like to do is look at what companies who sell high-quality
production servers with Linux preinstalled and see what hardware they
include.  You can find a list of vendors people here like at
http://wiki.postgresql.org/wiki/SCSI_vs._IDE/SATA_Disks#Helpful_vendors_of_SATA_RAID_systems


--
Greg Smith    2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com  www.2ndQuadrant.com

On Tue, Nov 24, 2009 at 7:35 PM, Greg Smith <greg@2ndquadrant.com> wrote:
> Scott Marlowe wrote:
>>
>> As far as drives go we've been really happy with WD of late, they make
>> large enterprise class SATA drives that don't pull a lot of power
>> (green series) and fast SATA drives that pull a bit more but are
>> faster (black series).
>
> Be careful to note the caveat that you need their *enterprise class* drives.
>  When you run into an error on their regular consumer drives, they get
> distracted for a while trying to cover the whole thing up, in a way that's
> exactly the opposite of the behavior you want for a RAID configuration.  I
> have a regular consumer WD drive that refuses to admit that it has a problem
> such that I can RMA it, but that always generates an error if I rewrite the
> whole drive.  The behavior of the firmware is downright shameful.  As cheap
> consumer drives go, I feel like WD has pulled ahead of everybody else on
> performance and possibly even actual reliability, but the error handling of
> their firmware is so bad I'm still using Seagate drives--when those fail, as
> least they're honest about it.

When I inquired earlier this summer about using the consumer WDs in a
new server I was told rather firmly by my sales guy "uhm, no".  They
put the enterprise drives through the wringer before he said they
seemed ok.  They have been great, both green and black series.   For
what they are, big SATA drives in RAID-6 or RAID-10 they're quite
good.  Moderate to quite good performers at a reasonable price.

--- On Tue, 24/11/09, Scott Marlowe <scott.marlowe@gmail.com> wrote:

> Jochen Erwied
> <jochen@pgsql-performance.erwied.eu>
> wrote:
> >
> > Since I'm currently looking at upgrading my own
> database server, maybe some
> > of the experts can give a comment on one of the
> following controllers:
> >
> > - Promise Technology Supertrak ES4650 + additional
> BBU
> > - Adaptec RAID 5405 SGL/256 SATA/SAS + additional BBU
> > - Adaptec RAID 5405Z SGL/512 SATA/SAS
> >
> > My personal favourite currently is the 5405Z, since it
> does not require
> > regular battery replacements and because it has 512MB
> of cache.
>
> Have you searched the -performance archives for references
> to them?
> I'm not that familiar with Adaptec RAID controllers. 
> Not requiring a
> battery check / replacement is nice.
>

We've been running Adaptec 5805s for the past year and I've been pretty happy, I think they have the same dual core
IOP348as the Areca 1680s. 

I've a bunch of 5805Zs on my desk ready to go in some new servers too (that means more perc6 cards to chuck on my smash
pile)and I'm excited to see how they go; I feer the unknown a bit though, and I'm not sure the sight big capacitors is
reassuruingme... 

Only problem I've seen is one controller periodically report it's too hot, but I suspect that may be something to do
withthe server directly above it having fanless power supplies. 

Greg Smith wrote:
> Jochen Erwied wrote:
>> - Promise Technology Supertrak ES4650 + additional BBU
>> - Adaptec RAID 5405 SGL/256 SATA/SAS + additional BBU
>> - Adaptec RAID 5405Z SGL/512 SATA/SAS
>>
> I've never seen a Promise controller that had a Linux driver you would
> want to rely on under any circumstances...Easier to just buy from a
> company that has always cared about good Linux support, like 3ware.
+1

I haven't tried Promise recently, but last time I did I determined that
they got the name because they "Promise" the Linux driver for your card
will be available real-soon-now. Actually got strung along for a couple
months before calling my supplier and telling him to swap it out for a
3ware. The 3ware "just works". I currently have a couple dozen Linux
servers, including some PostgreSQL machines, running the 3ware cards.

Cheers,
Steve

Steve Crawford wrote:
> Greg Smith wrote:
>> Jochen Erwied wrote:
>>> - Promise Technology Supertrak ES4650 + additional BBU
>>>
>> I've never seen a Promise controller that had a Linux driver you would
>> want to rely on under any circumstances...
> +1
>
> I haven't tried Promise recently, but last time I did I determined that
> they got the name because they "Promise" the Linux driver for your card
> will be available real-soon-now.

One more data point, it's not confidence inspiring that google turns up
Promise Technologies customers that are quite vocal about suing them.

http://www.carbonite.com/blog/post/2009/03/Further-clarification-on-our-lawsuit-against-Promise-Technologies.aspx

On 11/24/09 11:13 AM, "Scott Marlowe" <scott.marlowe@gmail.com> wrote:


>
> They get good reviews as well.  Both manufacturers have their "star"
> performers, and their "utility" or work group class controllers.  For
> what you're doing the areca 12xx or 3ware 95xx series should do fine.
>

-1 to 3ware's SATA solutions

3ware 95xx and 96xx had performance somewhere between PERC 5 (horrid) and
PERC 6 (mediocre) when I tested them with large SATA drives with RAID 10.
Haven't tried raid 6 or 5.  Haven't tried the "SA" model that supports SAS.
When a competing card (Areca or Adaptec) gets 3x the sequential throughput
on an 8 disk RAID 10 and only catches up to be 60% the speed after heavy
tuning of readahead value, there's something wrong.
Random access throughput doesn't suffer like that however -- but its nice
when the I/O can sequential scan faser than postgres can read the tuples.


> As far as drives go we've been really happy with WD of late, they make
> large enterprise class SATA drives that don't pull a lot of power
> (green series) and fast SATA drives that pull a bit more but are
> faster (black series).  We've used both and are quite happy with each.
>  We use a pair of blacks to build slony read slaves and they're very
> fast, with write speeds of ~100MB/second and read speeds double that
> in linux under sw RAID-1
>

Scott Carey wrote:

On 11/24/09 11:13 AM, "Scott Marlowe" <scott.marlowe@gmail.com> wrote:

 

They get good reviews as well.  Both manufacturers have their "star"
performers, and their "utility" or work group class controllers.  For
what you're doing the areca 12xx or 3ware 95xx series should do fine.   

-1 to 3ware's SATA solutions

3ware 95xx and 96xx had performance somewhere between PERC 5 (horrid) and
PERC 6 (mediocre) when I tested them with large SATA drives with RAID 10.
Haven't tried raid 6 or 5.  Haven't tried the "SA" model that supports SAS.
When a competing card (Areca or Adaptec) gets 3x the sequential throughput
on an 8 disk RAID 10 and only catches up to be 60% the speed after heavy
tuning of readahead value, there's something wrong.
Random access throughput doesn't suffer like that however -- but its nice
when the I/O can sequential scan faser than postgres can read the tuples. 

What operating system?

I am running under FreeBSD with 96xx series and am getting EXCELLENT performance.  Under Postgres 8.4.x on identical hardware except for the disk controller, I am pulling a literal 3x the iops on the same disks that I do with the Adaptec (!)

I DID note that under Linux the same hardware was a slug. 

Hmmmmm...


-- Karl

Scott Carey wrote:
> 3ware 95xx and 96xx had performance somewhere between PERC 5 (horrid) and
> PERC 6 (mediocre) when I tested them with large SATA drives with RAID 10.
> Haven't tried raid 6 or 5.  Haven't tried the "SA" model that supports SAS
The only models I've tested and recommended lately are exactly those
though.  The 9690SA is the earliest 3ware card I've mentioned as seeming
to have reasonable performance.  The 95XX cards are certainly much
slower than similar models from, say, Areca.  I've never had one of the
earlier 96XX models to test.  Now you've got me wondering what the
difference between the earlier and current 96XX models really is.

--
Greg Smith    2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com  www.2ndQuadrant.com

On 12/1/09 6:08 PM, "Karl Denninger" <karl@denninger.net> wrote:

> Scott Carey wrote:
>>
>> On 11/24/09 11:13 AM, "Scott Marlowe" <scott.marlowe@gmail.com>
>> <mailto:scott.marlowe@gmail.com>  wrote:
>>
>>
>>
>>
>>>
>>> They get good reviews as well.  Both manufacturers have their "star"
>>> performers, and their "utility" or work group class controllers.  For
>>> what you're doing the areca 12xx or 3ware 95xx series should do fine.
>>>
>>>
>>
>>
>> -1 to 3ware's SATA solutions
>>
>> 3ware 95xx and 96xx had performance somewhere between PERC 5 (horrid) and
>> PERC 6 (mediocre) when I tested them with large SATA drives with RAID 10.
>> Haven't tried raid 6 or 5.  Haven't tried the "SA" model that supports SAS.
>> When a competing card (Areca or Adaptec) gets 3x the sequential throughput
>> on an 8 disk RAID 10 and only catches up to be 60% the speed after heavy
>> tuning of readahead value, there's something wrong.
>> Random access throughput doesn't suffer like that however -- but its nice
>> when the I/O can sequential scan faser than postgres can read the tuples.
>>
> What operating system?
>
> I am running under FreeBSD with 96xx series and am getting EXCELLENT
> performance.  Under Postgres 8.4.x on identical hardware except for the disk
> controller, I am pulling a literal 3x the iops on the same disks that I do
> with the Adaptec (!)
>
> I DID note that under Linux the same hardware was a slug.
>
> Hmmmmm...
>

Linux, Centos 5.3.  Drivers/OS can certainly make a big difference.

>
> -- Karl
>

On 12/1/09 6:49 PM, "Greg Smith" <greg@2ndquadrant.com> wrote:

> Scott Carey wrote:
>> 3ware 95xx and 96xx had performance somewhere between PERC 5 (horrid) and
>> PERC 6 (mediocre) when I tested them with large SATA drives with RAID 10.
>> Haven't tried raid 6 or 5.  Haven't tried the "SA" model that supports SAS
> The only models I've tested and recommended lately are exactly those
> though.  The 9690SA is the earliest 3ware card I've mentioned as seeming
> to have reasonable performance.  The 95XX cards are certainly much
> slower than similar models from, say, Areca.  I've never had one of the
> earlier 96XX models to test.  Now you've got me wondering what the
> difference between the earlier and current 96XX models really is.

9650 was made by 3Ware, essentially a PCIe version of the 9550. The 9690SA
was from some sort of acquisition/merger. They are not the same product line
at all.
3Ware, IIRC, has its roots in ATA and SATA RAID.


I gave up on them after the 9650 and 9550 experiences (on Linux) though.

>
> --
> Greg Smith    2ndQuadrant   Baltimore, MD
> PostgreSQL Training, Services and Support
> greg@2ndQuadrant.com  www.2ndQuadrant.com
>
>

Scott Carey wrote:

On 12/1/09 6:49 PM, "Greg Smith" <greg@2ndquadrant.com> wrote:
 

Scott Carey wrote:   

3ware 95xx and 96xx had performance somewhere between PERC 5 (horrid) and
PERC 6 (mediocre) when I tested them with large SATA drives with RAID 10.
Haven't tried raid 6 or 5.  Haven't tried the "SA" model that supports SAS     

The only models I've tested and recommended lately are exactly those
though.  The 9690SA is the earliest 3ware card I've mentioned as seeming
to have reasonable performance.  The 95XX cards are certainly much
slower than similar models from, say, Areca.  I've never had one of the
earlier 96XX models to test.  Now you've got me wondering what the
difference between the earlier and current 96XX models really is.   

9650 was made by 3Ware, essentially a PCIe version of the 9550. The 9690SA
was from some sort of acquisition/merger. They are not the same product line
at all.
3Ware, IIRC, has its roots in ATA and SATA RAID.


I gave up on them after the 9650 and 9550 experiences (on Linux) though. 

My experience under FreeBSD:

1. The Adaptecs suck.  1/3rd to 1/2 the performance of....
2. The 9650s 3ware boards, which under FreeBSD are quite fast.
3. However, the Areca 1680-IX is UNBELIEVABLY fast.  Ridiculously so in fact.

I have a number of 9650s in service and have been happy with them under FreeBSD.  Under Linux, however, they bite in comparison.

The Areca 1680 is not cheap.  However, it comes with out-of-band management (IP-KVM, direct SMTP and SNMP connectivity, etc) which is EXTREMELY nice, especially for colocated machines where you need a way in if things go horribly wrong.

One warning: I have had problems with the Areca under FreeBSD if you set up a passthrough (e.g. JBOD) disc, delete it from the config while running and then either accidentally touch the device nodes OR try to use FreeBSD's "camcontrol" utility to tell it to pick up driver changes.  Either is a great way to panic the machine. 

As such for RAID it's fine but use care if you need to be able to swap NON-RAID disks while the machine is operating (e.g. for backup purposes - run a dump then dismount it and pull the carrier) - it is dangerous to attempt this (the 3Ware card does NOT have this problem.)  I am trying to figure out exactly what provokes this and if I can get around it at this point (in the lab of course!)

No experience with the 9690 3Wares as of yet.

-- Karl

Scott Carey wrote:
> 9650 was made by 3Ware, essentially a PCIe version of the 9550. The 9690SA
> was from some sort of acquisition/merger. They are not the same product line
> at all.
>
3ware became a division of AMCC, which was then bought by LSI.  The
9590SA came out while they were a part of AMCC.

I was under the impression that the differences between the 9650 and the
9690SA were mainly related to adding SAS support, which was sort of a
bridge addition rather than a fundamental change in the design of the
card.  You'll often see people refer to "9650/9690" as if they're the
same card; they may never run the same firmware.  They certainly always
get firmware updates at the same time, and as part of the same download
package.

Another possibility for the difference between Scott's experience and
mine is that I've only evaluated those particular cards recently, and
there seems to be evidence that 3ware did some major firmware overhauls
in late 2008, i.e.
http://unix.derkeiler.com/Mailing-Lists/FreeBSD/performance/2008-10/msg00005.html

Let me try to summarize where things are at a little more clearly, with
the data accumulated during this long thread:

-Areca:  Usually the fastest around.  Management tools are limited
enough that you really want the version with the on-board management
NIC.  May require some testing to find a good driver version.

-3ware:  Performance on current models not as good as Areca, but with a
great set of management tools (unless you're using SAS) and driver
reliability.  Exact magnitude of the performance gap with Areca is
somewhat controversial and may depend on OS--FreeBSD performance might
be better than Linux in particular.  Older 3ware cards were really slow.

One of these days I need to wrangle up enough development cash to buy
current Areca and 3ware cards, an Intel SSD, and disappear into the lab
(already plenty of drives here) until I've sorted this all out to my
satisfaction.

--
Greg Smith    2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com  www.2ndQuadrant.com

Greg Smith wrote:
> Let me try to summarize where things are at a little more clearly, with
> the data accumulated during this long thread:
>
> -Areca:  Usually the fastest around.  Management tools are limited
> enough that you really want the version with the on-board management
> NIC.  May require some testing to find a good driver version.
>
> -3ware:  Performance on current models not as good as Areca, but with a
> great set of management tools (unless you're using SAS) and driver
> reliability.  Exact magnitude of the performance gap with Areca is
> somewhat controversial and may depend on OS--FreeBSD performance might
> be better than Linux in particular.  Older 3ware cards were really slow.
>
> One of these days I need to wrangle up enough development cash to buy
> current Areca and 3ware cards, an Intel SSD, and disappear into the lab
> (already plenty of drives here) until I've sorted this all out to my
> satisfaction.

... and do I hear you saying that no other vendor is worth considering?  Just how far off are they?

Thanks,
Craig

Greg Smith wrote:
> Scott Carey wrote:
>> 9650 was made by 3Ware, essentially a PCIe version of the 9550. The
>> 9690SA
>> was from some sort of acquisition/merger. They are not the same
>> product line
>> at all.
>>
> 3ware became a division of AMCC, which was then bought by LSI.  The
> 9590SA came out while they were a part of AMCC.
>
> I was under the impression that the differences between the 9650 and
> the 9690SA were mainly related to adding SAS support, which was sort
> of a bridge addition rather than a fundamental change in the design of
> the card.  You'll often see people refer to "9650/9690" as if they're
> the same card; they may never run the same firmware.  They certainly
> always get firmware updates at the same time, and as part of the same
> download package.
>
> Another possibility for the difference between Scott's experience and
> mine is that I've only evaluated those particular cards recently, and
> there seems to be evidence that 3ware did some major firmware
> overhauls in late 2008, i.e.
> http://unix.derkeiler.com/Mailing-Lists/FreeBSD/performance/2008-10/msg00005.html
>
>
> Let me try to summarize where things are at a little more clearly,
> with the data accumulated during this long thread:
>
> -Areca:  Usually the fastest around.  Management tools are limited
> enough that you really want the version with the on-board management
> NIC.  May require some testing to find a good driver version.
>
> -3ware:  Performance on current models not as good as Areca, but with
> a great set of management tools (unless you're using SAS) and driver
> reliability.  Exact magnitude of the performance gap with Areca is
> somewhat controversial and may depend on OS--FreeBSD performance might
> be better than Linux in particular.  Older 3ware cards were really slow.
>
> One of these days I need to wrangle up enough development cash to buy
> current Areca and 3ware cards, an Intel SSD, and disappear into the
> lab (already plenty of drives here) until I've sorted this all out to
> my satisfaction.
Most common SSDs will NOT come up on the 3ware cards at present.  Not
sure why as of yet - I've tried several.

Not had the time to screw with them on the ARECA cards yet.

-- Karl

Craig James wrote:
> ... and do I hear you saying that no other vendor is worth
> considering?  Just how far off are they?
I wasn't trying to summarize every possible possibility, just the
complicated ones there's some debate over.

What else is OK besides Areca and 3ware?  HP's P800 is good, albeit not
so easy to buy unless you're getting an HP system.  The LSI Megaraid
stuff and its close relative the Dell PERC6 are OK for some apps too; my
intense hatred of Dell usually results in my forgetting about them.  (As
an example,
http://en.wikipedia.org/wiki/ATX#Issues_with_Dell_power_supplies
documents what I consider the worst design decision ever made by a PC
manufacturer)

I don't think any of the other vendors on the market are viable for a
Linux system due to driver issues and general low quality, which
includes  Adaptec, Promise, Highpoint, and all the motherboard Fake RAID
stuff from Silicon Image, Intel, Via, etc.  I don't feel there's any
justification for using those products instead of using a simple SATA
controller and Linux software RAID in a PostgreSQL context.

--
Greg Smith    2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com  www.2ndQuadrant.com

Karl Denninger wrote:
> Most common SSDs will NOT come up on the 3ware cards at present.  Not
> sure why as of yet - I've tried several.
>
Right, and they're being rather weasly at
http://www.3ware.com/kb/Article.aspx?id=15470 talking about it too.
> Not had the time to screw with them on the ARECA cards yet.
>
I know the situation there is much better, like:
http://hothardware.com/News/24-Samsung-SSDs-Linked-Together-for-2GBSec/

Somebody at Newegg has said they got their Areca 1680 working with one
of the Intel X-25 drives, but wasn't impressed by the write
performance of the result.  Makes me wonder if the Areca card is messing
with the write cache of the drive.

--
Greg Smith    2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com  www.2ndQuadrant.com

Greg Smith wrote:
> Craig James wrote:
>> ... and do I hear you saying that no other vendor is worth
>> considering?  Just how far off are they?
> I wasn't trying to summarize every possible possibility, just the
> complicated ones there's some debate over.
>
> What else is OK besides Areca and 3ware?  HP's P800 is good, albeit
> not so easy to buy unless you're getting an HP system.  The LSI
> Megaraid stuff and its close relative the Dell PERC6 are OK for some
> apps too; my intense hatred of Dell usually results in my forgetting
> about them.  (As an example,
> http://en.wikipedia.org/wiki/ATX#Issues_with_Dell_power_supplies
> documents what I consider the worst design decision ever made by a PC
> manufacturer)
>
> I don't think any of the other vendors on the market are viable for a
> Linux system due to driver issues and general low quality, which
> includes  Adaptec, Promise, Highpoint, and all the motherboard Fake
> RAID stuff from Silicon Image, Intel, Via, etc.  I don't feel there's
> any justification for using those products instead of using a simple
> SATA controller and Linux software RAID in a PostgreSQL context.
The LSI Megaraid (and Intel's repackaging of it, among others) is
reasonably good under FreeBSD.

Performance is slightly worse than the 3ware 95xx series boards, but not
materially so.

Their CLI interface is "interesting" (it drops a log file in the working
directly BY DEFAULT unless you tell it otherwise, among other things.)

-- Karl