Thread: Which RAID Controllers to pick/avoid?
I'm setting up a dedicated linux postgres box with 2x300GB 15k SAS drive in a RAID 1, though if future load dictates we would like to be able to upgrade to RAID 10. The hosting provider offers the following options for a RAID controller (all are the same price):
ADAPTEC 3405 RAID Controller
ADAPTEC 4800 RAID Controller
LSI MegaRaid 8308 RAID Controller
ADAPTEC 5405 RAID Controller
ADAPTEC 5405Z RAID Controller
ADAPTEC 5805 RAID Controller
ADAPTEC 5805Z RAID Controller
ADAPTEC 4800 RAID Controller
LSI MegaRaid 8308 RAID Controller
ADAPTEC 5405 RAID Controller
ADAPTEC 5405Z RAID Controller
ADAPTEC 5805 RAID Controller
ADAPTEC 5805Z RAID Controller
However, they can't guarantee that any particular RAID controller would be in stock when they are building the machine, so basically I would like to know if any of these cards are sufficiently better or worse than the others that I should either a) wait for a particular card or b) avoid a card.
Also, I am planning on replicating this box to a standby machine with streaming replication. Given this, is there any reason not to use the write cache on the RAID controller assuming it has battery backup? My though being even in a worst case scenario if the BBU fails and the master DB gets corrupted, the standby (not using any write cache) would still be ok sans a few seconds of data (assuming the replication was keeping up, which would be monitored).
-Dan
On 03/02/11 07:15, Dan Birken wrote: > However, they can't guarantee that any particular RAID controller would > be in stock when they are building the machine, so basically I would > like to know if any of these cards are sufficiently better or worse than > the others that I should either a) wait for a particular card or b) > avoid a card. I don't know the Adaptec parts above (I avoid them these days) but AFAIK the LSI is at least OK. Whatever RAID controller you get, make sure you have a battery backup unit (BBU) installed so you can safely enable write-back caching. Without that, you might as well use software RAID - it'll generally be faster (and cheaper) than HW RAID w/o a BBU. I get great results with Linux software RAID 10 on my Pg server - but then, I'm not loading it particularly hard. (I continually wish the `md' driver could use a PCIe/SATA battery-backed DRAM cache, because it'd be capable of massively outperforming most HW raid implementation if only it could offer safe write-back caching using persistent cache.) > Also, I am planning on replicating this box to a standby machine with > streaming replication. Given this, is there any reason not to use the > write cache on the RAID controller assuming it has battery backup? My > though being even in a worst case scenario if the BBU fails and the > master DB gets corrupted, the standby (not using any write cache) would > still be ok sans a few seconds of data (assuming the replication was > keeping up, which would be monitored). That sounds about right. The standby would be fine, the master would be totaled. If you're doing write-back caching without a BBU and things go down, it's not going to be neatly time-warped back a few seconds. Your DB will be corrupted, possibly massively. One big advantage of write-back caching is that it lets the controller batch and re-order writes for higher disk throughput as well as lower latencies at the app level. The cost of that is that you lose the safety of ordered writes to WAL then heap; it's quite possible for physical media writes to hit the heap before the WAL, for newer writes to hit the WAL before older writes, etc etc. Because of the cache, the OS and Pg never see or care about the crazily inconsistent state of the actual disk media - unless the cache is lost, in which case you're screwed. Assume that a BBU failure will require restoration from a backup or from a standby server. If you can't afford that, you should operate in write-through cache mode, possibly using synchronous commit and/or commit delay options if you can afford a few seconds data loss on crash. -- System & Network Administrator POST Newspapers
Dan Birken wrote:
The 3405 and 4800 are two of Adaptec's older cards with only 128MB of cache on them. Those are on the slow side compared to the others listed.
The LSI cards are some of the most popular and known to work well with PostgreSQL ones around.
I've recently tested a system based on the 5405, and as much as I've hated Adaptec controllers in the past I have to admit this latest line from them is pretty solid. My own benchmarks and the others I've seen suggest it's easily capable of keeping up with the LSI and Areca controllers Adaptec used to be seriously outrun by. See http://www.tomshardware.com/reviews/adaptec-serial-controllers,1806-14.html for example.
The "Z" variations use their "Zero-Maintenance Cache" instead of a standard battery-backup unit; that's a small amount of flash memory and a supercap, similar to the good caches on SSD: http://www.adaptec.com/NR/rdonlyres/7FD8C372-8231-4727-B12B-5ABF79D9325C/0/6514_Series5Z_1_7.pdf
5405 has 256MB of cache, the others 512MB.
The 5405 and 5805 models do have a known problem where they overheat if you don't have enough cooling in the server box, with the 5805 seeming to be the bigger source of such issues. See the reviews at http://www.newegg.com/Product/Product.aspx?Item=N82E16816103099 for example. Scott Marlowe was griping recently about a similar issue in some of the LSI models, too. I suspect it's a problem impacting several of the larger RAID cards that use the big Intel IOP processors for their RAID computations, given that's the part with the heatsink on it.
Quick summary: avoid the Adaptec 3405 and 4800. Rest are decent cards. Just make sure you monitor the temperatures in your case (and the card too if arcconf lets you, I haven't checked for that yet) if you end up with a 5405/5805.
ADAPTEC 3405 RAID Controller
ADAPTEC 4800 RAID Controller
The 3405 and 4800 are two of Adaptec's older cards with only 128MB of cache on them. Those are on the slow side compared to the others listed.
LSI MegaRaid 8308 RAID Controller
ADAPTEC 5405 RAID Controller
ADAPTEC 5405Z RAID Controller
ADAPTEC 5805 RAID Controller
ADAPTEC 5805Z RAID Controller
The LSI cards are some of the most popular and known to work well with PostgreSQL ones around.
I've recently tested a system based on the 5405, and as much as I've hated Adaptec controllers in the past I have to admit this latest line from them is pretty solid. My own benchmarks and the others I've seen suggest it's easily capable of keeping up with the LSI and Areca controllers Adaptec used to be seriously outrun by. See http://www.tomshardware.com/reviews/adaptec-serial-controllers,1806-14.html for example.
The "Z" variations use their "Zero-Maintenance Cache" instead of a standard battery-backup unit; that's a small amount of flash memory and a supercap, similar to the good caches on SSD: http://www.adaptec.com/NR/rdonlyres/7FD8C372-8231-4727-B12B-5ABF79D9325C/0/6514_Series5Z_1_7.pdf
5405 has 256MB of cache, the others 512MB.
The 5405 and 5805 models do have a known problem where they overheat if you don't have enough cooling in the server box, with the 5805 seeming to be the bigger source of such issues. See the reviews at http://www.newegg.com/Product/Product.aspx?Item=N82E16816103099 for example. Scott Marlowe was griping recently about a similar issue in some of the LSI models, too. I suspect it's a problem impacting several of the larger RAID cards that use the big Intel IOP processors for their RAID computations, given that's the part with the heatsink on it.
Quick summary: avoid the Adaptec 3405 and 4800. Rest are decent cards. Just make sure you monitor the temperatures in your case (and the card too if arcconf lets you, I haven't checked for that yet) if you end up with a 5405/5805.
-- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us "PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books
On Wed, Feb 2, 2011 at 10:46 PM, Greg Smith <greg@2ndquadrant.com> wrote: > example. Scott Marlowe was griping recently about a similar issue in some > of the LSI models, too. I suspect it's a problem impacting several of the > larger RAID cards that use the big Intel IOP processors for their RAID > computations, given that's the part with the heatsink on it. Specifically the LSI 8888 in a case that gave very low amounts of air flow over the RAID card. The case right above the card was quite hot, and the multilane cable was warm enough to almost burn my fingers when I pulled it out the back. I'm not sure any RAID card would have survived there, but the LSI 8888 LP definitely did not.
On Wed, Feb 2, 2011 at 7:00 PM, Craig Ringer <craig@postnewspapers.com.au> wrote: > Whatever RAID controller you get, make sure you have a battery backup > unit (BBU) installed so you can safely enable write-back caching. > Without that, you might as well use software RAID - it'll generally be > faster (and cheaper) than HW RAID w/o a BBU. Recently we had to pull our RAID controllers and go to plain SAS cards. While random access dropped a bit, sequential throughput skyrocketed, saturating the 4 lane cable we use. 4x300Gb/s = 1200Gb/s or right around 1G of data a second off the array. VERY impressive.
Scott Marlowe wrote:
Interesting...that shoots down my theory. Now that I check, the LSI 8888 uses their SAS1078 controller, which is based on a PowerPC 440 processor--it's not one of the Intel IOP processors at all. The 8308 Dan has as an option is using the very popular Intel IOP333 instead, which is also used in some Areca 1200 series cards (1220/1230/1260).
The Adaptec 5405 and 5805 cards both use the Intel IOP348, as does the Areca 1680. Areca puts a fan right on it; Adaptec does not. I suspect the only reason the 5805 cards have gotten more reports of overheating than the 5405 ones is just because having more drives typically connected increases their actual workload. I don't think there's actually any difference between the cooling situation between the two otherwise.
On Wed, Feb 2, 2011 at 10:46 PM, Greg Smith <greg@2ndquadrant.com> wrote:example. Scott Marlowe was griping recently about a similar issue in some of the LSI models, too. I suspect it's a problem impacting several of the larger RAID cards that use the big Intel IOP processors for their RAID computations, given that's the part with the heatsink on it.Specifically the LSI 8888 in a case that gave very low amounts of air flow over the RAID card. The case right above the card was quite hot, and the multilane cable was warm enough to almost burn my fingers whe
Interesting...that shoots down my theory. Now that I check, the LSI 8888 uses their SAS1078 controller, which is based on a PowerPC 440 processor--it's not one of the Intel IOP processors at all. The 8308 Dan has as an option is using the very popular Intel IOP333 instead, which is also used in some Areca 1200 series cards (1220/1230/1260).
The Adaptec 5405 and 5805 cards both use the Intel IOP348, as does the Areca 1680. Areca puts a fan right on it; Adaptec does not. I suspect the only reason the 5805 cards have gotten more reports of overheating than the 5405 ones is just because having more drives typically connected increases their actual workload. I don't think there's actually any difference between the cooling situation between the two otherwise.
-- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us "PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books
Thank you everybody for the detailed answers, the help is well appreciated.
A couple of follow-up questions:
- Is the supercap + flash memory considered superior to the BBU in practice? Is that type of system well tested?
- Is the linux support of the LSI and Adaptec cards comparable?
-Dan
On Wed, Feb 2, 2011 at 10:15 PM, Greg Smith <greg@2ndquadrant.com> wrote:
Scott Marlowe wrote:Interesting...that shoots down my theory. Now that I check, the LSI 8888 uses their SAS1078 controller, which is based on a PowerPC 440 processor--it's not one of the Intel IOP processors at all. The 8308 Dan has as an option is using the very popular Intel IOP333 instead, which is also used in some Areca 1200 series cards (1220/1230/1260).On Wed, Feb 2, 2011 at 10:46 PM, Greg Smith <greg@2ndquadrant.com> wrote:example. Scott Marlowe was griping recently about a similar issue in some of the LSI models, too. I suspect it's a problem impacting several of the larger RAID cards that use the big Intel IOP processors for their RAID computations, given that's the part with the heatsink on it.Specifically the LSI 8888 in a case that gave very low amounts of air flow over the RAID card. The case right above the card was quite hot, and the multilane cable was warm enough to almost burn my fingers whe
The Adaptec 5405 and 5805 cards both use the Intel IOP348, as does the Areca 1680. Areca puts a fan right on it; Adaptec does not. I suspect the only reason the 5805 cards have gotten more reports of overheating than the 5405 ones is just because having more drives typically connected increases their actual workload. I don't think there's actually any difference between the cooling situation between the two otherwise.-- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us "PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books
On Wed, Feb 2, 2011 at 11:15 PM, Greg Smith <greg@2ndquadrant.com> wrote: > Scott Marlowe wrote: > > On Wed, Feb 2, 2011 at 10:46 PM, Greg Smith <greg@2ndquadrant.com> wrote: > > > example. Scott Marlowe was griping recently about a similar issue in some > of the LSI models, too. I suspect it's a problem impacting several of the > larger RAID cards that use the big Intel IOP processors for their RAID > computations, given that's the part with the heatsink on it. > > > Specifically the LSI 8888 in a case that gave very low amounts of air > flow over the RAID card. The case right above the card was quite hot, > and the multilane cable was warm enough to almost burn my fingers whe > > Interesting...that shoots down my theory. Now that I check, the LSI 8888 > uses their SAS1078 controller, which is based on a PowerPC 440 > processor--it's not one of the Intel IOP processors at all. The 8308 Dan > has as an option is using the very popular Intel IOP333 instead, which is > also used in some Areca 1200 series cards (1220/1230/1260). > > The Adaptec 5405 and 5805 cards both use the Intel IOP348, as does the Areca > 1680. Areca puts a fan right on it; Adaptec does not. I suspect the only > reason the 5805 cards have gotten more reports of overheating than the 5405 > ones is just because having more drives typically connected increases their > actual workload. I don't think there's actually any difference between the > cooling situation between the two otherwise. The LSI 8888 has a fan right on it. But it was just moving 90C air around in place.
On Thu, Feb 3, 2011 at 07:30, Dan Birken <birken@gmail.com> wrote: > Thank you everybody for the detailed answers, the help is well appreciated. > A couple of follow-up questions: > - Is the supercap + flash memory considered superior to the BBU in practice? I think it's considered about equivalent. The advantages of the flash one are no battery maintenance (=no downtime every <n> years to replace it) and no "timeout" if you loose power (but the battery backed ones are usually good for 48 hours or so, and if you don't have a server up by then...) > Is that type of system well tested? Yes. > - Is the linux support of the LSI and Adaptec cards comparable? Can't comment on that one, sorry. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/
--- On Thu, 3/2/11, Greg Smith <greg@2ndquadrant.com> wrote: > The 5405 and 5805 models do have a known problem where they overheat if > you don't have enough cooling in the server box, with the 5805 seeming > to be the bigger source of such issues. See the reviews at > http://www.newegg.com/Product/Product.aspx?Item=N82E16816103099 for > example. Scott Marlowe was griping recently about a similar issue in > some of the LSI models, too. I suspect it's a problem impacting > several of the larger RAID cards that use the big Intel IOP processors > for their RAID computations, given that's the part with the heatsink on > it. > Quick summary: avoid the Adaptec 3405 and 4800. Rest are decent > cards. Just make sure you monitor the temperatures in your case (and > the card too if arcconf lets you, I haven't checked for that yet) if > you end up with a 5405/5805. I can attest to the 5805 and 5805Z cards running a little hot, the ones we're running tend to run in the high 60s and low70s Celsius with fairly good airflow over them. I've been running some 5805s for 3 years now, and 5805Zs for a year and they've been really good, stable, fast cards. Imonitor everything on them (including temperature) with nagios and a simple script that uses the arcconf utility. Glyn
Dan Birken wrote: > - Is the supercap + flash memory considered superior to the BBU in > practice? Is that type of system well tested? The main risk is that it's a pretty new approach. The standard BBU setup has been used for a long time now; this whole flash+supercap thing has only showed up in the last couple of years. Theoretically it's better; the #1 weakness of the old battery setup was only surviving an outage of a few days, and storing to flash doesn't have that issue. It's just got the usual risks of something new. > - Is the linux support of the LSI and Adaptec cards comparable? Seems to be. The latest versions of Adaptec's arcconf utility even provide about the same quality of command-line tools as LSI's megactl, after being behind in that area for a while. Only quirk, and I can't say where this was the manufacturer of the box or not because they installed the base OS, is that the 5405 setup I saw didn't turn off the write caches on the individual drives of the system. That's the standard safe practice and default for the LSI cards. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us "PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books
Am 03.02.2011 00:15, schrieb Dan Birken: > I'm setting up a dedicated linux postgres box with 2x300GB 15k SAS > drive in a RAID 1, though if future load dictates we would like to be > able to upgrade to RAID 10. The hosting provider offers the following > options for a RAID controller (all are the same price): Adaptec at least has good tools for managing the controller, and performance in our RAID-1 (DB) and RAID-5 setups (Files) is very good. I don't think you can do wrong with the Adaptec controllers. Can't say much regarding LSI, but avoid cheap HP controllers.
Am 03.02.2011 07:45, schrieb Magnus Hagander: > On Thu, Feb 3, 2011 at 07:30, Dan Birken<birken@gmail.com> wrote: >> - Is the linux support of the LSI and Adaptec cards comparable? > > Can't comment on that one, sorry. We dropped LSI in favour of Adaptec for exactly this reason. We run hundreds of machines in remote locations, and always again had problems with LSI regarding support of new controller models and kernel versions (esp. on system installation), and because of occasional kernel panics triggers by the LSI driver. The Adaptec-support integrated into the Linux kernel source tree does a flawless job here. On the other hand, Adaptec is not perfect as well: when we attached a 24-drive SAS storage unit to a 5805Z, it failed to properly handle drive failures in this unit. We are using an LSI again there, which so far works well... Joachim
> On Wed, Feb 2, 2011 at 7:00 PM, Craig Ringer > <craig@postnewspapers.com.au> wrote: >> Whatever RAID controller you get, make sure you have a battery backup >> unit (BBU) installed so you can safely enable write-back caching. >> Without that, you might as well use software RAID - it'll generally be >> faster (and cheaper) than HW RAID w/o a BBU. > > Recently we had to pull our RAID controllers and go to plain SAS > cards. While random access dropped a bit, sequential throughput > skyrocketed, saturating the 4 lane cable we use. 4x300Gb/s = > 1200Gb/s or right around 1G of data a second off the array. VERY > impressive. This is really surprising. Software raid generally outperform hardware raid without BBU? Why is that? My company useshardware raid quite a bit without BBU and have never thought to compare with software raid =/ Thanks! --Royce
On Sun, Feb 6, 2011 at 2:39 AM, Royce Ausburn <royce.ml@inomial.com> wrote: > >> On Wed, Feb 2, 2011 at 7:00 PM, Craig Ringer >> <craig@postnewspapers.com.au> wrote: >>> Whatever RAID controller you get, make sure you have a battery backup >>> unit (BBU) installed so you can safely enable write-back caching. >>> Without that, you might as well use software RAID - it'll generally be >>> faster (and cheaper) than HW RAID w/o a BBU. >> >> Recently we had to pull our RAID controllers and go to plain SAS >> cards. While random access dropped a bit, sequential throughput >> skyrocketed, saturating the 4 lane cable we use. 4x300Gb/s = >> 1200Gb/s or right around 1G of data a second off the array. VERY >> impressive. > > > This is really surprising. Software raid generally outperform hardware raid without BBU? Why is that? My company useshardware raid quite a bit without BBU and have never thought to compare with software raid =/ For raw throughtput it's not uncommon to beat a RAID card whether it has a battery backed cache or not. If I'm wiriting a 200G file to the disks, a BBU cache isn't gonna make that any faster, it'll fill up in a second and then it's got to write to disk. BBU Cache are for faster random writes, and will handily beat SW RAID. But for raw large file read and write SW RAID is the fastest thing I've seen.
On Sun, 6 Feb 2011, Scott Marlowe wrote: > On Sun, Feb 6, 2011 at 2:39 AM, Royce Ausburn <royce.ml@inomial.com> wrote: >> >>> On Wed, Feb 2, 2011 at 7:00 PM, Craig Ringer >>> <craig@postnewspapers.com.au> wrote: >>>> Whatever RAID controller you get, make sure you have a battery backup >>>> unit (BBU) installed so you can safely enable write-back caching. >>>> Without that, you might as well use software RAID - it'll generally be >>>> faster (and cheaper) than HW RAID w/o a BBU. >>> >>> Recently we had to pull our RAID controllers and go to plain SAS >>> cards. While random access dropped a bit, sequential throughput >>> skyrocketed, saturating the 4 lane cable we use. 4x300Gb/s = >>> 1200Gb/s or right around 1G of data a second off the array. VERY >>> impressive. >> >> >> This is really surprising. Software raid generally outperform hardware >> raid without BBU? Why is that? My company uses hardware raid quite a >> bit without BBU and have never thought to compare with software raid =/ > > For raw throughtput it's not uncommon to beat a RAID card whether it > has a battery backed cache or not. If I'm wiriting a 200G file to the > disks, a BBU cache isn't gonna make that any faster, it'll fill up in > a second and then it's got to write to disk. BBU Cache are for faster > random writes, and will handily beat SW RAID. But for raw large file > read and write SW RAID is the fastest thing I've seen. > keep in mind that hardware raide with BBU is safer than software raid. since the updates to the drives do not all happen at the same time, there is a chance that a write to software raid may have happened on some drives and not others when the system crashes. with hardware raid and BBU, the controller knows what it was trying to write where, and if it didn't get the scknowledgement, it will complete the write when it comes up again. but with software raid you will have updates some part of the array and not others. this will result in a corrupted stripe in the array. David Lang