Thread: SAN/NAS options
Hello all, It seems that I'm starting to outgrow our current Postgres setup. We've been running a handful of machines as standalone db servers. This is all in a colocation environment, so everything is stuffed into 1U Supermicro boxes. Our standard build looks like this: Supermicro 1U w/SCA backplane and 4 bays 2x2.8 GHz Xeons Adaptec 2015S "zero channel" RAID card 2 or 4 x 73GB Seagate 10K Ultra 320 drives (mirrored+striped) 2GB RAM FreeBSD 4.11 PGSQL data from 5-10GB per box Recently I started studying what we were running up against in our nightly runs that do a ton of updates/inserts to prep things for the tasks the db does during the business day (light mix of selects/inserts/updates). While we have plenty of disk bandwidth (according to bonnie), we are really dying on IOPS. I'm guessing this is a mix of a rather anemic RAID controller (ever notice how adaptec doesn't publish any real performance specs on raid cards?) and having only two or four spindles (effectively 1 or 2 on writes). So that's where we are... I'm new to the whole SAN thing, but did recently pick up a few used NetApp shelves and a Fibre Channel RAID HBA (Mylex ExtremeRAID 3000, also used) to toy with. I started wondering if I could put something together to both get our storage on one set of boxes and allow me to get data striped across more drives. Our budget is not huge and we are not adverse to getting used gear where appropriate. What do you folks recommend? I'm just starting to look at what's out there for SANs and NAS, and from what I've seen, our options are: NetApp Filers - the pluses with these are that if we use NFS, we don't have to worry about either large filesystem support in FreeBSD (2TB practical limit), or limitation on "growing" partitions as the NetApp just deals with that. I also understand these make backups a bit simpler. I have a great, trusted, spare-stocking source for these. Apple X-Serve RAID - well, it's pretty cheap. Honestly, that's all I know about it - they don't talk about IOPS numbers, and I have no idea what lurks in that box as a RAID controller. SAN box w/integrated RAID - it seems like this might not be a good choice since the RAID hardware in the box may be where I hit any limits. I also imagine I'm probably overpaying for some OEM RAID controller integrated into the box. No idea where to look for used gear. SAN box, JBOD - this seems like it might be affordable as well. A few big shelves full of drives a SAN "switch" to plug all the shelves and hosts into and a FC RAID card in each host. No idea where to look for used gear here either. You'll note that I'm being somewhat driven by my OS of choice, FreeBSD. Unlike Solaris or other commercial offerings, there is no nice volume management available. While I'd love to keep managing a dozen or so FreeBSD boxes, I could be persuaded to go to Solaris x86 if the volume management really shines and Postgres performs well on it. Lastly, one thing that I'm not yet finding in trying to educate myself on SANs is a good overview of what's come out in the past few years that's more affordable than the old big-iron stuff. For example I saw some brief info on this list's archives about the Dell/EMC offerings. Anything else in that vein to look at? I hope this isn't too far off topic for this list. Postgres is the main application that I'm looking to accomodate. Anything else I can do with whatever solution we find is just gravy... Thanks! Charles
On Wed, 14 Dec 2005, Charles Sprickman wrote: [big snip] The list server seems to be regurgitating old stuff, and in doing so it reminded me to thank everyone for their input. I was kind of waiting to see if anyone who was very pro-NAS/SAN was going to pipe up, but it looks like most people are content with per-host storage. You've given me a lot to go on... Now I'm going to have to do some research as to real-world RAID controller performance. It's vexing (to say the least) that most vendors don't supply any raw throughput or TPS stats on this stuff... Anyhow, thanks again. You'll probably see me back here in the coming months as I try to shake some mysql info out of my brain as our pgsql DBA gets me up to speed on pgsql and what specifically he's doing to stress things. Charles > I hope this isn't too far off topic for this list. Postgres is the main > application that I'm looking to accomodate. Anything else I can do with > whatever solution we find is just gravy... > > Thanks! > > Charles > > > ---------------------------(end of broadcast)--------------------------- > TIP 2: Don't 'kill -9' the postmaster >
> >> I hope this isn't too far off topic for this list. Postgres is >> the main application that I'm looking to accomodate. Anything >> else I can do with whatever solution we find is just gravy... > You've given me a lot to go on... Now I'm going to have to do some > research as to real-world RAID controller performance. It's vexing > (to say the least) that most vendors don't supply any raw > throughput or TPS stats on this stuff... One word of advice. Stay away from Dell kit. The PERC 4 controllers they use don't implement RAID 10 properly. It's RAID 1 + JBOD array. It also has generally dismal IOPS performance too. You might get away with running software RAID, either in conjunction with, or entirely avoiding the card.
Charles, On 12/20/05 9:58 PM, "Charles Sprickman" <spork@bway.net> wrote: > You've given me a lot to go on... Now I'm going to have to do some > research as to real-world RAID controller performance. It's vexing (to > say the least) that most vendors don't supply any raw throughput or TPS > stats on this stuff... Take a look at this: http://www.wlug.org.nz/HarddiskBenchmarks > Anyhow, thanks again. You'll probably see me back here in the coming > months as I try to shake some mysql info out of my brain as our pgsql DBA > gets me up to speed on pgsql and what specifically he's doing to stress > things. Cool! BTW - based on the above benchmark page, I just immediately ordered 2 x of the Areca 1220 SATA controllers ( http://www.areca.com.tw/products/html/pcie-sata.htm) so that we can compare them to the 3Ware 9550SX that we've been using. The 3Ware controllers have been super fast on sequential access, but I'm concerned about their random IOPs. The Areca's aren't as popular, and there's consequently less volume of them, but people who use them rave about them. - Luke
On Sat, 14 Jan 2006, Luke Lonergan wrote: > Charles, > > On 1/14/06 6:37 PM, "Charles Sprickman" <spork@bway.net> wrote: > >> I'm vaguely considering pairing these two devices: >> >> http://www.areca.us/products/html/products.htm >> >> That's an Areca 16 channel SATA II (I haven't even read up on what's new >> in SATA II) RAID controller with an optional U320 SCSI daughter card to >> connect to the host(s). > > I'm confused - SATA with a SCSI daughter card? Where does the SCSI go? Bad ASCII diagram follows (D=disk, C=controller H=host): SATA ____ D -------| | SCSI ________ D -------| C |--------| H | D -------| | |________| D -------|____| (etc. up to 16 drives) The drives and the controller go in the Chenbro case. U320 SCSI from the RAID controller in the Chenbro case to the 1U server. C
Charles, On 1/14/06 7:23 PM, "Charles Sprickman" <spork@bway.net> wrote: > The drives and the controller go in the Chenbro case. U320 SCSI from the > RAID controller in the Chenbro case to the 1U server. Thanks for the explanation - I didn't click on your Areca link until now, thinking it was a generic link to their products page. Looks great - I think this might do better than the SATA -> FC products because of the use of faster processors, but I'd keep my expectations low until we see some performance data on it. We've had some very poor experiences with Fibre Channel attach SATA disk controllers. A large vendor of same ultimately concluded that they will no longer recommend them for database use because of the terrible performance of their unit. We ended up with a 110MB/s bottleneck on the controller when using 200MB/s FC connections. With the dual U320 attach and 16 drives, you should be able to saturate the SCSI busses at about 600MB/s. It would be great if you could post your I/O results here! - Luke