Thread: SAN/NAS options

SAN/NAS options

From
Charles Sprickman
Date:
Hello all,

It seems that I'm starting to outgrow our current Postgres setup.  We've been
running a handful of machines as standalone db servers.  This is all in a
colocation environment, so everything is stuffed into 1U Supermicro boxes.  Our
standard build looks like this:

Supermicro 1U w/SCA backplane and 4 bays
2x2.8 GHz Xeons
Adaptec 2015S "zero channel" RAID card
2 or 4 x 73GB Seagate 10K Ultra 320 drives (mirrored+striped)
2GB RAM
FreeBSD 4.11
PGSQL data from 5-10GB per box

Recently I started studying what we were running up against in our nightly runs
that do a ton of updates/inserts to prep things for the tasks the db does
during the business day (light mix of selects/inserts/updates). While we have
plenty of disk bandwidth (according to bonnie), we are really dying on IOPS.
I'm guessing this is a mix of a rather anemic RAID controller (ever notice how
adaptec doesn't publish any real performance specs on raid cards?) and having
only two or four spindles (effectively 1 or 2 on writes).

So that's where we are...

I'm new to the whole SAN thing, but did recently pick up a few used NetApp
shelves and a Fibre Channel RAID HBA (Mylex ExtremeRAID 3000, also used) to toy
with.  I started wondering if I could put something together to both get our
storage on one set of boxes and allow me to get data striped across more
drives.  Our budget is not huge and we are not adverse to getting used gear
where appropriate.

What do you folks recommend?  I'm just starting to look at what's out there for
SANs and NAS, and from what I've seen, our options are:

NetApp Filers - the pluses with these are that if we use NFS, we don't have to
worry about either large filesystem support in FreeBSD (2TB practical limit),
or limitation on "growing" partitions as the NetApp just deals with that.  I
also understand these make backups a bit simpler.  I have a great, trusted,
spare-stocking source for these.

Apple X-Serve RAID - well, it's pretty cheap.  Honestly, that's all I know
about it - they don't talk about IOPS numbers, and I have no idea what lurks in
that box as a RAID controller.

SAN box w/integrated RAID - it seems like this might not be a good choice since
the RAID hardware in the box may be where I hit any limits.  I also imagine I'm
probably overpaying for some OEM RAID controller integrated into the box.  No
idea where to look for used gear.

SAN box, JBOD - this seems like it might be affordable as well.  A few big
shelves full of drives a SAN "switch" to plug all the shelves and hosts into
and a FC RAID card in each host.  No idea where to look for used gear here
either.

You'll note that I'm being somewhat driven by my OS of choice, FreeBSD. Unlike
Solaris or other commercial offerings, there is no nice volume management
available.  While I'd love to keep managing a dozen or so FreeBSD boxes, I
could be persuaded to go to Solaris x86 if the volume management really shines
and Postgres performs well on it.

Lastly, one thing that I'm not yet finding in trying to educate myself on SANs
is a good overview of what's come out in the past few years that's more
affordable than the old big-iron stuff.  For example I saw some brief info on
this list's archives about the Dell/EMC offerings.  Anything else in that vein
to look at?

I hope this isn't too far off topic for this list.  Postgres is the main
application that I'm looking to accomodate.  Anything else I can do with
whatever solution we find is just gravy...

Thanks!

Charles


Re: SAN/NAS options

From
"Luke Lonergan"
Date:
Charles,

> Lastly, one thing that I'm not yet finding in trying to
> educate myself on SANs is a good overview of what's come out
> in the past few years that's more affordable than the old
> big-iron stuff.  For example I saw some brief info on this
> list's archives about the Dell/EMC offerings.  Anything else
> in that vein to look at?

My two cents: SAN is a bad investment, go for big internal storage.

The 3Ware or Areca SATA RAID adapters kick butt and if you look in the
newest colos (I was just in ours "365main.net" today), you will see rack
on rack of machines with from 4 to 16 internal SATA drives.  Are they
all DB servers?  Not necessarily, but that's where things are headed.

You can get a 3U server with dual opteron 250s, 16GB RAM and 16x 400GB
SATAII drives with the 3Ware 9550SX controller for $10K - we just
ordered 4 of them.  I don't think you can buy an external disk chassis
and a Fibre channel NIC for that.

Performance?  800MB/s RAID5 reads, 400MB/s RAID5 writes.  Random IOs are
also very high for RAID10, but we don't use it so YMMV - look at Areca
and 3Ware.

Managability? Good web management interfaces with 6+ years of
development from 3Ware, e-mail, online rebuild options, all the goodies.
No "snapshot" or offline backup features like the high-end SANs, but do
you really need it?

Need more power or storage over time? Run a parallel DB like Bizgres
MPP, you can add more servers with internal storage and increase your
I/O, CPU and memory.

- Luke


Re: SAN/NAS options

From
Mark Kirkwood
Date:
Charles Sprickman wrote:
> Hello all,
>
> It seems that I'm starting to outgrow our current Postgres setup.  We've
> been running a handful of machines as standalone db servers.  This is
> all in a colocation environment, so everything is stuffed into 1U
> Supermicro boxes.  Our standard build looks like this:
>
> Supermicro 1U w/SCA backplane and 4 bays
> 2x2.8 GHz Xeons
> Adaptec 2015S "zero channel" RAID card
> 2 or 4 x 73GB Seagate 10K Ultra 320 drives (mirrored+striped)
> 2GB RAM
> FreeBSD 4.11
> PGSQL data from 5-10GB per box
>
> Recently I started studying what we were running up against in our
> nightly runs that do a ton of updates/inserts to prep things for the
> tasks the db does during the business day (light mix of
> selects/inserts/updates). While we have plenty of disk bandwidth
> (according to bonnie), we are really dying on IOPS. I'm guessing this is
> a mix of a rather anemic RAID controller (ever notice how adaptec
> doesn't publish any real performance specs on raid cards?) and having
> only two or four spindles (effectively 1 or 2 on writes).
>
> So that's where we are...
>
> I'm new to the whole SAN thing, but did recently pick up a few used
> NetApp shelves and a Fibre Channel RAID HBA (Mylex ExtremeRAID 3000,
> also used) to toy with.  I started wondering if I could put something
> together to both get our storage on one set of boxes and allow me to get
> data striped across more drives.  Our budget is not huge and we are not
> adverse to getting used gear where appropriate.
>
> What do you folks recommend?  I'm just starting to look at what's out
> there for SANs and NAS, and from what I've seen, our options are:
>

Leaving the whole SAN issue for a moment:

It would be interesting to see if moving to FreeBSD 6.0 would help you -
the vfs layer is no longer throttled by the (SMP) GIANT lock in this
version, and that may make quite a difference (given you have SMP boxes).

Another interesting thing to try is rebuilding the database ufs
filesystem(s) with 32K blocks and 4K frags (as opposed to 8K/1K or
16K/2K - can't recall the default on 4.x). I found this to give a factor
of 2 speedup on random disk access (specifically queries doing indexed
joins).

Is it mainly your 2 disk machines that are IOPS bound? if so, a cheap
option may be to buy 2 more cheetahs for them! If it's the 4's, well how
about a 2U U320 diskpack from whomever supplies you the Supermicro boxes?

I have just noticed Luke's posting - I would second the advice to avoid
SAN - in my experience it's an expensive way to buy storage.

best wishes

Mark


Re: SAN/NAS options

From
Andrew Rawnsley
Date:
The Apple is, as you say, cheap (except, the Apple markup on the disks
fuzzes that a bit). Its easy to set up, and has been quite reliable for me,
but do not expect anything resembling good DB performance out of it (I gave
up running anything but backup DBs on it). From the mouth of Apple guys, it
(and Xsan) are heavily optimized for sequential access. They want to sell
piles of these to the music/film industry, where they have some cred. Oracle
has apparently gotten some performance gains through raw device pixie dust
and voodoo, but even as a (reluctant, kicking-and-screaming) Oracle guy I
wouldn't go there.

Other goofy things about it: it isn't 1 device with 14 disks and redundant
controllers. Its 2 7 disk arrays with non-redundant controllers. It doesn't
do RAID10.

If you want a gob-o-space with no performance requirements, its fine.
Otherwise...

On 12/14/05 1:56 AM, "Charles Sprickman" <spork@bway.net> wrote:

> Hello all,
>
> It seems that I'm starting to outgrow our current Postgres setup.  We've been
> running a handful of machines as standalone db servers.  This is all in a
> colocation environment, so everything is stuffed into 1U Supermicro boxes.
> Our
> standard build looks like this:
>
> Supermicro 1U w/SCA backplane and 4 bays
> 2x2.8 GHz Xeons
> Adaptec 2015S "zero channel" RAID card
> 2 or 4 x 73GB Seagate 10K Ultra 320 drives (mirrored+striped)
> 2GB RAM
> FreeBSD 4.11
> PGSQL data from 5-10GB per box
>
> Recently I started studying what we were running up against in our nightly
> runs
> that do a ton of updates/inserts to prep things for the tasks the db does
> during the business day (light mix of selects/inserts/updates). While we have
> plenty of disk bandwidth (according to bonnie), we are really dying on IOPS.
> I'm guessing this is a mix of a rather anemic RAID controller (ever notice how
> adaptec doesn't publish any real performance specs on raid cards?) and having
> only two or four spindles (effectively 1 or 2 on writes).
>
> So that's where we are...
>
> I'm new to the whole SAN thing, but did recently pick up a few used NetApp
> shelves and a Fibre Channel RAID HBA (Mylex ExtremeRAID 3000, also used) to
> toy
> with.  I started wondering if I could put something together to both get our
> storage on one set of boxes and allow me to get data striped across more
> drives.  Our budget is not huge and we are not adverse to getting used gear
> where appropriate.
>
> What do you folks recommend?  I'm just starting to look at what's out there
> for
> SANs and NAS, and from what I've seen, our options are:
>
> NetApp Filers - the pluses with these are that if we use NFS, we don't have to
> worry about either large filesystem support in FreeBSD (2TB practical limit),
> or limitation on "growing" partitions as the NetApp just deals with that.  I
> also understand these make backups a bit simpler.  I have a great, trusted,
> spare-stocking source for these.
>
> Apple X-Serve RAID - well, it's pretty cheap.  Honestly, that's all I know
> about it - they don't talk about IOPS numbers, and I have no idea what lurks
> in
> that box as a RAID controller.
>
> SAN box w/integrated RAID - it seems like this might not be a good choice
> since
> the RAID hardware in the box may be where I hit any limits.  I also imagine
> I'm
> probably overpaying for some OEM RAID controller integrated into the box.  No
> idea where to look for used gear.
>
> SAN box, JBOD - this seems like it might be affordable as well.  A few big
> shelves full of drives a SAN "switch" to plug all the shelves and hosts into
> and a FC RAID card in each host.  No idea where to look for used gear here
> either.
>
> You'll note that I'm being somewhat driven by my OS of choice, FreeBSD. Unlike
> Solaris or other commercial offerings, there is no nice volume management
> available.  While I'd love to keep managing a dozen or so FreeBSD boxes, I
> could be persuaded to go to Solaris x86 if the volume management really shines
> and Postgres performs well on it.
>
> Lastly, one thing that I'm not yet finding in trying to educate myself on SANs
> is a good overview of what's come out in the past few years that's more
> affordable than the old big-iron stuff.  For example I saw some brief info on
> this list's archives about the Dell/EMC offerings.  Anything else in that vein
> to look at?
>
> I hope this isn't too far off topic for this list.  Postgres is the main
> application that I'm looking to accomodate.  Anything else I can do with
> whatever solution we find is just gravy...
>
> Thanks!
>
> Charles
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings




Re: SAN/NAS options

From
Michael Stone
Date:
On Wed, Dec 14, 2005 at 11:53:52AM -0500, Andrew Rawnsley wrote:
>Other goofy things about it: it isn't 1 device with 14 disks and redundant
>controllers. Its 2 7 disk arrays with non-redundant controllers. It doesn't
>do RAID10.

And if you want hot spares you need *two* per tray (one for each
controller). That definately changes the cost curve. :)

Mike Stone

Re: SAN/NAS options

From
"Anjan Dave"
Date:
Luke,

How did you measure 800MB/sec, is it cached, or physical I/O?

-anjan

-----Original Message-----
From: Luke Lonergan [mailto:LLonergan@greenplum.com]
Sent: Wednesday, December 14, 2005 2:10 AM
To: Charles Sprickman; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] SAN/NAS options

Charles,

> Lastly, one thing that I'm not yet finding in trying to
> educate myself on SANs is a good overview of what's come out
> in the past few years that's more affordable than the old
> big-iron stuff.  For example I saw some brief info on this
> list's archives about the Dell/EMC offerings.  Anything else
> in that vein to look at?

My two cents: SAN is a bad investment, go for big internal storage.

The 3Ware or Areca SATA RAID adapters kick butt and if you look in the
newest colos (I was just in ours "365main.net" today), you will see rack
on rack of machines with from 4 to 16 internal SATA drives.  Are they
all DB servers?  Not necessarily, but that's where things are headed.

You can get a 3U server with dual opteron 250s, 16GB RAM and 16x 400GB
SATAII drives with the 3Ware 9550SX controller for $10K - we just
ordered 4 of them.  I don't think you can buy an external disk chassis
and a Fibre channel NIC for that.

Performance?  800MB/s RAID5 reads, 400MB/s RAID5 writes.  Random IOs are
also very high for RAID10, but we don't use it so YMMV - look at Areca
and 3Ware.

Managability? Good web management interfaces with 6+ years of
development from 3Ware, e-mail, online rebuild options, all the goodies.
No "snapshot" or offline backup features like the high-end SANs, but do
you really need it?

Need more power or storage over time? Run a parallel DB like Bizgres
MPP, you can add more servers with internal storage and increase your
I/O, CPU and memory.

- Luke


---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings


Re: SAN/NAS options

From
"Luke Lonergan"
Date:
Physical using xfs on Linux.
- Luke
--------------------------
Sent from my BlackBerry Wireless Device


-----Original Message-----
From: Anjan Dave <adave@vantage.com>
To: Luke Lonergan <LLonergan@greenplum.com>; Charles Sprickman <spork@bway.net>; pgsql-performance@postgresql.org
<pgsql-performance@postgresql.org>
Sent: Thu Dec 15 16:13:04 2005
Subject: RE: [PERFORM] SAN/NAS options

Luke,

How did you measure 800MB/sec, is it cached, or physical I/O?

-anjan

-----Original Message-----
From: Luke Lonergan [mailto:LLonergan@greenplum.com] 
Sent: Wednesday, December 14, 2005 2:10 AM
To: Charles Sprickman; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] SAN/NAS options

Charles,

> Lastly, one thing that I'm not yet finding in trying to 
> educate myself on SANs is a good overview of what's come out 
> in the past few years that's more affordable than the old 
> big-iron stuff.  For example I saw some brief info on this 
> list's archives about the Dell/EMC offerings.  Anything else 
> in that vein to look at?

My two cents: SAN is a bad investment, go for big internal storage.

The 3Ware or Areca SATA RAID adapters kick butt and if you look in the
newest colos (I was just in ours "365main.net" today), you will see rack
on rack of machines with from 4 to 16 internal SATA drives.  Are they
all DB servers?  Not necessarily, but that's where things are headed.

You can get a 3U server with dual opteron 250s, 16GB RAM and 16x 400GB
SATAII drives with the 3Ware 9550SX controller for $10K - we just
ordered 4 of them.  I don't think you can buy an external disk chassis
and a Fibre channel NIC for that.

Performance?  800MB/s RAID5 reads, 400MB/s RAID5 writes.  Random IOs are
also very high for RAID10, but we don't use it so YMMV - look at Areca
and 3Ware.

Managability? Good web management interfaces with 6+ years of
development from 3Ware, e-mail, online rebuild options, all the goodies.
No "snapshot" or offline backup features like the high-end SANs, but do
you really need it?

Need more power or storage over time? Run a parallel DB like Bizgres
MPP, you can add more servers with internal storage and increase your
I/O, CPU and memory.

- Luke


---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings



Re: SAN/NAS options

From
"Jim C. Nasby"
Date:
On Wed, Dec 14, 2005 at 08:28:56PM +1300, Mark Kirkwood wrote:
> Another interesting thing to try is rebuilding the database ufs
> filesystem(s) with 32K blocks and 4K frags (as opposed to 8K/1K or
> 16K/2K - can't recall the default on 4.x). I found this to give a factor
> of 2 speedup on random disk access (specifically queries doing indexed
> joins).

Even if you're doing a lot of random IO? I would think that random IO
would perform better if you use smaller (8K) blocks, since there's less
data being read in and then just thrown away that way.

> Is it mainly your 2 disk machines that are IOPS bound? if so, a cheap
> option may be to buy 2 more cheetahs for them! If it's the 4's, well how
> about a 2U U320 diskpack from whomever supplies you the Supermicro boxes?

Also, on the 4 drive machines if you can spare the room you might see a
big gain by putting the tables on one mirror and the OS and transaction
logs on the other.
--
Jim C. Nasby, Sr. Engineering Consultant      jnasby@pervasive.com
Pervasive Software      http://pervasive.com    work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf       cell: 512-569-9461

Re: SAN/NAS options

From
"Jim C. Nasby"
Date:
On Wed, Dec 14, 2005 at 01:56:10AM -0500, Charles Sprickman wrote:
 You'll note that I'm being somewhat driven by my OS of choice, FreeBSD.
> Unlike Solaris or other commercial offerings, there is no nice volume
> management available.  While I'd love to keep managing a dozen or so
> FreeBSD boxes, I could be persuaded to go to Solaris x86 if the volume
> management really shines and Postgres performs well on it.

Have you looked at vinum? It might not qualify as a true volume manager,
but it's still pretty handy.
--
Jim C. Nasby, Sr. Engineering Consultant      jnasby@pervasive.com
Pervasive Software      http://pervasive.com    work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf       cell: 512-569-9461

Re: SAN/NAS options

From
Mark Kirkwood
Date:
Jim C. Nasby wrote:
> On Wed, Dec 14, 2005 at 08:28:56PM +1300, Mark Kirkwood wrote:
>
>>Another interesting thing to try is rebuilding the database ufs
>>filesystem(s) with 32K blocks and 4K frags (as opposed to 8K/1K or
>>16K/2K - can't recall the default on 4.x). I found this to give a factor
>>of 2 speedup on random disk access (specifically queries doing indexed
>>joins).
>
>
> Even if you're doing a lot of random IO? I would think that random IO
> would perform better if you use smaller (8K) blocks, since there's less
> data being read in and then just thrown away that way.
>
>

Yeah, that's what I would have expected too! but the particular queries
I tested do a ton of random IO (correlation of 0.013 on the join column
for the big table). I did wonder if the gain has something to do with
the underlying RAID stripe size (64K or 256K in my case), as I have only
tested the 32K vs 8K/16K on RAIDed systems.

I guess for a system where the number of concurrent users give rise to
memory pressure, it will cause more thrashing of the file buffer cache,
much could be a net loss.

Still worth trying out I think, you will know soon enough if it is a win
or lose!

Note that I did *not* alter Postgres page/block size (BLCKSZ) from 8K,
so no dump/reload is required to test this out.

cheers

Mark


Re: SAN/NAS options

From
Michael Stone
Date:
On Fri, Dec 16, 2005 at 04:18:01PM -0600, Jim C. Nasby wrote:
>Even if you're doing a lot of random IO? I would think that random IO
>would perform better if you use smaller (8K) blocks, since there's less
>data being read in and then just thrown away that way.

The overhead of reading an 8k block instead of a 32k block is too small
to measure on modern hardware. The seek is what dominates; leaving the
read head on a little longer and then transmitting a little more over a
200 megabyte channel is statistical fuzz.

Mike Stone

Re: SAN/NAS options

From
"Jim C. Nasby"
Date:
On Fri, Dec 16, 2005 at 05:51:03PM -0500, Michael Stone wrote:
> On Fri, Dec 16, 2005 at 04:18:01PM -0600, Jim C. Nasby wrote:
> >Even if you're doing a lot of random IO? I would think that random IO
> >would perform better if you use smaller (8K) blocks, since there's less
> >data being read in and then just thrown away that way.
>
> The overhead of reading an 8k block instead of a 32k block is too small
> to measure on modern hardware. The seek is what dominates; leaving the
> read head on a little longer and then transmitting a little more over a
> 200 megabyte channel is statistical fuzz.

True, but now you've got 4x the amount of data in your cache that you
probably don't need.

Looks like time to do some benchmarking...
--
Jim C. Nasby, Sr. Engineering Consultant      jnasby@pervasive.com
Pervasive Software      http://pervasive.com    work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf       cell: 512-569-9461

Re: SAN/NAS options

From
Michael Stone
Date:
On Fri, Dec 16, 2005 at 06:25:25PM -0600, Jim C. Nasby wrote:
>True, but now you've got 4x the amount of data in your cache that you
>probably don't need.

Or you might be 4x more likely to have data cached that's needed later.
If you're hitting disk either way, that's probably more likely than the
extra IO pushing something critical out--if *all* the important stuff
were cached you wouldn't be doing the seeks in the first place. This
will obviously be heavily dependent on the amount of ram you've got and
your workload, so (as always) you'll have to benchmark it to get past
the hand-waving stage.

Mike Stone

Re: SAN/NAS options

From
Matthew Schumacher
Date:
Jim C. Nasby wrote:
> On Wed, Dec 14, 2005 at 01:56:10AM -0500, Charles Sprickman wrote:
>  You'll note that I'm being somewhat driven by my OS of choice, FreeBSD.
>
>>Unlike Solaris or other commercial offerings, there is no nice volume
>>management available.  While I'd love to keep managing a dozen or so
>>FreeBSD boxes, I could be persuaded to go to Solaris x86 if the volume
>>management really shines and Postgres performs well on it.
>
>
> Have you looked at vinum? It might not qualify as a true volume manager,
> but it's still pretty handy.

I am looking very closely at purchasing a SANRAD Vswitch 2000, a Nexsan
SATABoy with SATA disks, and the Qlogic iscsi controller cards.

Nexsan claims up to 370MB/s sustained per controller and 44,500 IOPS but
I'm not sure if that is good or bad.  It's certainly faster than the LSI
megaraid controller I'm using now with a raid 1 mirror.

The sanrad box looks like it saves money in that you don't have to by
controller cards for everything, but for I/O intensive servers such as
the database server, I would end up buying an iscsi controller card anyway.

At this point I'm not sure what the best solution is.  I like the idea
of having logical disks available though iscsi because of how flexible
it is, but I really don't want to spend $20k (10 for the nexsan and 10
for the sanrad) and end up with poor performance.

On other advantage to iscsi is that I can go completely diskless on my
servers and boot from iscsi which means that I don't have to have spare
disks for each host, now I just have spare disks for the nexsan chassis.

So the question becomes: has anyone put postgres on an iscsi san, and if
so how did it perform?

schu



Re: SAN/NAS options

From
"Anjan Dave"
Date:
Usually manufacturer's claims are tested in 'ideal' conditions, it may not translate well on bandwidth seen on the host
side.A 2Gbps Fiber Channel connection would (ideally) give you about 250MB/sec per HBA. Not sure how it translates for
GigEconsidering scsi protocol overheads, but you may want to confirm from them how they achieved 370MB/sec (hwo many
iSCSIcontrollers, what file system, how many drives, what RAID type, block size, strip size, cache settings, etc), and
whetherit was physical I/O or cached. In other words, if someone has any benchmark numbers, that would be helpful.
 
 
Regarding diskless iscsi boots for future servers, remember that it's a shared storage, if you have a busy server
attachedto your Nexsan, you may have to think twice on sharing the performance (throughput and IOPS of the storage
controller)without impacting the existing hosts, unless you are zizing it now.
 
 
And you want to have a pretty clean GigE network, more or less dedicated to this block traffic.
 
Large internal storage with more memory and AMD CPUs is an option as Luke had originally suggested. Check out Appro as
well.
 
I'd also be curious to know if someone has been using this (SATA/iSCSI/SAS) solution and what are some I/O numbers
observed.
 
Thanks,
Anjan

    -----Original Message----- 
    From: Matthew Schumacher [mailto:matt.s@aptalaska.net] 
    Sent: Mon 12/19/2005 7:41 PM 
    To: pgsql-performance@postgresql.org 
    Cc: 
    Subject: Re: [PERFORM] SAN/NAS options
    
    

    Jim C. Nasby wrote: 
    > On Wed, Dec 14, 2005 at 01:56:10AM -0500, Charles Sprickman wrote: 
    >  You'll note that I'm being somewhat driven by my OS of choice, FreeBSD. 
    > 
    >>Unlike Solaris or other commercial offerings, there is no nice volume 
    >>management available.  While I'd love to keep managing a dozen or so 
    >>FreeBSD boxes, I could be persuaded to go to Solaris x86 if the volume 
    >>management really shines and Postgres performs well on it. 
    > 
    > 
    > Have you looked at vinum? It might not qualify as a true volume manager, 
    > but it's still pretty handy. 

    I am looking very closely at purchasing a SANRAD Vswitch 2000, a Nexsan 
    SATABoy with SATA disks, and the Qlogic iscsi controller cards. 

    Nexsan claims up to 370MB/s sustained per controller and 44,500 IOPS but 
    I'm not sure if that is good or bad.  It's certainly faster than the LSI 
    megaraid controller I'm using now with a raid 1 mirror. 

    The sanrad box looks like it saves money in that you don't have to by 
    controller cards for everything, but for I/O intensive servers such as 
    the database server, I would end up buying an iscsi controller card anyway. 

    At this point I'm not sure what the best solution is.  I like the idea 
    of having logical disks available though iscsi because of how flexible 
    it is, but I really don't want to spend $20k (10 for the nexsan and 10 
    for the sanrad) and end up with poor performance. 

    On other advantage to iscsi is that I can go completely diskless on my 
    servers and boot from iscsi which means that I don't have to have spare 
    disks for each host, now I just have spare disks for the nexsan chassis. 

    So the question becomes: has anyone put postgres on an iscsi san, and if 
    so how did it perform? 

    schu 



    ---------------------------(end of broadcast)--------------------------- 
    TIP 4: Have you searched our list archives? 

                   http://archives.postgresql.org 


Re: SAN/NAS options

From
Charles Sprickman
Date:
Following up to myself again...

On Wed, 14 Dec 2005, Charles Sprickman wrote:

> Hello all,
>
> Supermicro 1U w/SCA backplane and 4 bays
> 2x2.8 GHz Xeons
> Adaptec 2015S "zero channel" RAID card

I don't want to throw away the four machines like that that we have.  I do
want to throw away the ZCR cards... :)  If I ditch those I still have a 1U
box with a U320 scsi plug on the back.

I'm vaguely considering pairing these two devices:

http://www.areca.us/products/html/products.htm

That's an Areca 16 channel SATA II (I haven't even read up on what's new
in SATA II) RAID controller with an optional U320 SCSI daughter card to
connect to the host(s).

http://www.chenbro.com.tw/Chenbro_Special/RM321.php

How can I turn that box down?  Those people in the picture look very
excited about it?  Seriously though, it looks like an interesting and
economical pairing that gives me most of what I'm looking for:

-a modern RAID engine
-small form factor
-remote management of the array
-ability to reuse my current db hosts that are disk-bound

Disadvantages:

-only 1 or 2 hosts per box
-more difficult to move storage from host to host (compared to a SAN or
NAS system)
-no fancy NetApp features like snapshots
-I have no experience with Areca SATA->SCSI RAID controllers

Any thoughts on this?  The controller looks to be about $1500, the
enclosure about $400, and the drives are no great mystery, cost would
depend on what total capacity I'm looking for.

Our initial plan is to set one up for storage for a mail archive project,
and to also have a host use this storage to host replicated copies of all
Postgres databases.  If things look good, we'd start moving our main PG
hosts to use a similar RAID box.

Thanks,

Charles


Re: SAN/NAS options

From
"Luke Lonergan"
Date:
Charles,

On 1/14/06 6:37 PM, "Charles Sprickman" <spork@bway.net> wrote:

> I'm vaguely considering pairing these two devices:
>
> http://www.areca.us/products/html/products.htm
>
> That's an Areca 16 channel SATA II (I haven't even read up on what's new
> in SATA II) RAID controller with an optional U320 SCSI daughter card to
> connect to the host(s).

I'm confused - SATA with a SCSI daughter card? Where does the SCSI go?

The Areca has a number (8,12,16) of single drive attach SATA ports coming
out of it, each of which will go to a disk drive connection on the
backplane.

> http://www.chenbro.com.tw/Chenbro_Special/RM321.php
>
> How can I turn that box down?  Those people in the picture look very
> excited about it?  Seriously though, it looks like an interesting and
> economical pairing that gives me most of what I'm looking for:

What a picture!  I'm totally enthusiastic all of a sudden!  I'm putting !!!
at the end of every sentence!

We just bought 4 very similar systems that use the chassis from California
Design - our latest favorite source:
  http://www.asacomputers.com/

They did an excellent job of setting the systems up, with proper labeling
and Quality Control.  They also installed Fedora Core 4 and set up the
filesystems, the only mistake they made was that they didn't enable 2TB
clipping so that we had to rebuild the RAIDs (and install CentOS with the
xfs filesystem).

We paid $10.4K each for 16x 400GB WD RE2 SATA II drives, 16GB RAM and two
Opteron 250s.  We also put a single 200GB SATA system drive into each.  RAID
card is the 3Ware 9550SX.

Performance has been stunning - we're getting 800MB/s sustained I/O
throughput using the two 9550SX controllers in parallel.

> Any thoughts on this?  The controller looks to be about $1500, the
> enclosure about $400, and the drives are no great mystery, cost would
> depend on what total capacity I'm looking for.

I'd get ASA to build it for you - use the Tyan 2882 series motherboard for
greatest stablity.  They may try to sell you hard on the SuperMicro boards,
we've had less luck with them.

> Our initial plan is to set one up for storage for a mail archive project,
> and to also have a host use this storage to host replicated copies of all
> Postgres databases.  If things look good, we'd start moving our main PG
> hosts to use a similar RAID box.

Good approach.

I'm personally spending as much time using these machines as I can - they
are the fastest I've been on in a *long* time.

- Luke



Re: SAN/NAS options

From
Christopher Browne
Date:
> Following up to myself again...
>
> On Wed, 14 Dec 2005, Charles Sprickman wrote:
>
>> Hello all,
>>
>> Supermicro 1U w/SCA backplane and 4 bays
>> 2x2.8 GHz Xeons
>> Adaptec 2015S "zero channel" RAID card
>
> I don't want to throw away the four machines like that that we have.
> I do want to throw away the ZCR cards... :)  If I ditch those I still
> have a 1U box with a U320 scsi plug on the back.
>
> I'm vaguely considering pairing these two devices:

> http://www.areca.us/products/html/products.htm
> http://www.chenbro.com.tw/Chenbro_Special/RM321.php

> How can I turn that box down?  Those people in the picture look very
> excited about it?  Seriously though, it looks like an interesting and
> economical pairing that gives me most of what I'm looking for:

The combination definitely looks attractive.  I have only been hearing
positive things about the Areca cards; the overall combination sounds
pretty attractive.

> Disadvantages:
>
> -only 1 or 2 hosts per box
> -more difficult to move storage from host to host (compared to a SAN
> or NAS system)
> -no fancy NetApp features like snapshots
> -I have no experience with Areca SATA->SCSI RAID controllers
>
> Any thoughts on this?  The controller looks to be about $1500, the
> enclosure about $400, and the drives are no great mystery, cost would
> depend on what total capacity I'm looking for.

Another "usage model" that could be appropriate would be
ATA-over-Ethernet...

<http://en.wikipedia.org/wiki/ATA-over-Ethernet>

> Our initial plan is to set one up for storage for a mail archive
> project, and to also have a host use this storage to host replicated
> copies of all Postgres databases.  If things look good, we'd start
> moving our main PG hosts to use a similar RAID box.

We're thinking about some stuff like this to host things that require
bulky amounts of disk that are otherwise "not high TPC" sorts of apps.
This is definitely not a "gold plated" answer, compared to the NetApp
and EMC boxes of the world, but can be useful in contexts where they
are too expensive.
--
let name="cbbrowne" and tld="gmail.com" in String.concat "@" [name;tld];;
http://linuxdatabases.info/info/x.html
It is usually a   good idea to  put  a capacitor of a  few microfarads
across the output, as shown.

Re: SAN/NAS options

From
Michael Adler
Date:
On Sat, Jan 14, 2006 at 09:37:01PM -0500, Charles Sprickman wrote:
> Following up to myself again...
>
> On Wed, 14 Dec 2005, Charles Sprickman wrote:
>
> >Hello all,
> >
> >Supermicro 1U w/SCA backplane and 4 bays
> >2x2.8 GHz Xeons
> >Adaptec 2015S "zero channel" RAID card
>
> I don't want to throw away the four machines like that that we have.  I do
> want to throw away the ZCR cards... :)  If I ditch those I still have a 1U
> box with a U320 scsi plug on the back.
>
> I'm vaguely considering pairing these two devices:
>
> http://www.areca.us/products/html/products.htm
>
> That's an Areca 16 channel SATA II (I haven't even read up on what's new
> in SATA II) RAID controller with an optional U320 SCSI daughter card to
> connect to the host(s).
>
> http://www.chenbro.com.tw/Chenbro_Special/RM321.php

Not sure how significant, but the RM321 backplane claims to support
SATA 150 (aka SATA I) only.

 -Mike