Thread: SAN vs Internal Disks

Hi,

We are currently running our DB on a DualCore, Dual Proc 3.Ghz Xeon, 8GB RAM, 4x SAS 146 GB 15K RPM on RAID 5.

The current data size is about 50GB, but we want to purchase the hardware to scale to about 1TB as we think our business will need to support that much soon.
- Currently we have a 80% read and 20% write perecntages.
- Currently with this configuration the Database is showing signs of over-loading.
- Auto-vaccum, etc run on this database, vaccum full runs nightly.
- Currently CPU loads are about 20%, memory utilization is full (but this is also due to linux caching disk blocks) and IO waits are frequent.
- We have a load of about 400 queries per second

Now we are considering to purchase our own servers and in the process are facing the usual dilemmas. First I'll list out what machine we have decided to use:
2x Quad Xeon 2.4 Ghz (4-way only 2 populated right now)
32 GB RAM
OS Only storage - 2x SCSI 146 GB 15k RPM on RAID-1
(Data Storage mentioned below)

We have already decided to split our database into 3 machines on the basis on disjoint sets of data. So we will be purchasing three of these boxes.

HELP 1: Does something look wrong with above configuration, I know there will be small differences b/w opetron/xeon. But do you think there is something against going for 2.4Ghz Quad Xeons (clovertown i think)?

HELP 2: The main confusion is with regards to Data Storage. We have the option of going for:

A: IBM N-3700 SAN Box, having 12x FC 300GB disks,  Partitioned into 3 disks into RAID-4 for WAL/backup, and 9 disks on RAID-DP for data, 2 hot spare. We are also considering similar solution from EMC - CX310C.

B: Go for Internal of DAS based storage. Here for each server we should be able to have: 2x disks on RAID-1 for logs, 6x disks on RAID-10 for tablespace1 and 6x disks on RAID-10 for tablespace2. Or maybe 12x disks on RAID-10 single table-space.

What do I think? Well..
SAN wins on manageability, replication (say to a DR site), backup, etc...
DAS wins on cost

But for a moment keeping these aside, i wanted to discuss, purely on performance side which one is a winner? It feels like internal-disks will perform better, but need to understand a rough magnitude of difference in performance to see if its worth loosing the manageability features.

Also if we choose to go with DAS, what would be the best tool to do async replication to DR site and maybe even as a extra plus a second read-only DB server to distribute select loads.

Regards,
Azad

On 9/6/07, Harsh Azad <harsh.azad@gmail.com> wrote:
> Hi,
>
> We are currently running our DB on a DualCore, Dual Proc 3.Ghz Xeon, 8GB
> RAM, 4x SAS 146 GB 15K RPM on RAID 5.
>
> The current data size is about 50GB, but we want to purchase the hardware to
> scale to about 1TB as we think our business will need to support that much
> soon.
> - Currently we have a 80% read and 20% write percentages.

For this type load, you should be running on RAID10 not RAID5.  Or, if
you must use RAID 5, use more disks and have a battery backed caching
RAID controller known to perform well with RAID5 and large arrays.

> - Currently with this configuration the Database is showing signs of
> over-loading.

On I/O or CPU?  If you're running out of CPU, then look to increasing
CPU horsepower and tuning postgresql.
If I/O then you need to look into a faster I/O subsystem.

> - Auto-vaccum, etc run on this database, vaccum full runs nightly.

Generally speaking, if you need to run vacuum fulls, you're doing
something wrong.  Is there a reason you're running vacuum full or is
this just precautionary.  vacuum full can bloat your indexes, so you
shouldn't run it regularly.  reindexing might be a better choice if
you do need to regularly shrink your db.  The better option is to
monitor your fsm usage and adjust fsm settings / autovacuum settings
as necessary.

>  - Currently CPU loads are about 20%, memory utilization is full (but this
> is also due to linux caching disk blocks) and IO waits are frequent.
> - We have a load of about 400 queries per second

What does vmstat et. al. say about CPU versus I/O wait?

> Now we are considering to purchase our own servers and in the process are
> facing the usual dilemmas. First I'll list out what machine we have decided
> to use:
> 2x Quad Xeon 2.4 Ghz (4-way only 2 populated right now)
> 32 GB RAM
> OS Only storage - 2x SCSI 146 GB 15k RPM on RAID-1
> (Data Storage mentioned below)
>
> We have already decided to split our database into 3 machines on the basis
> on disjoint sets of data. So we will be purchasing three of these boxes.
>
> HELP 1: Does something look wrong with above configuration, I know there
> will be small differences b/w opetron/xeon. But do you think there is
> something against going for 2.4Ghz Quad Xeons (clovertown i think)?

Look like good machines, plenty fo memory.

> HELP 2: The main confusion is with regards to Data Storage. We have the
> option of going for:
>
> A: IBM N-3700 SAN Box, having 12x FC 300GB disks,  Partitioned into 3 disks
> into RAID-4 for WAL/backup, and 9 disks on RAID-DP for data, 2 hot spare. We
> are also considering similar solution from EMC - CX310C.
>
> B: Go for Internal of DAS based storage. Here for each server we should be
> able to have: 2x disks on RAID-1 for logs, 6x disks on RAID-10 for
> tablespace1 and 6x disks on RAID-10 for tablespace2. Or maybe 12x disks on
> RAID-10 single table-space.
>
> What do I think? Well..
> SAN wins on manageability, replication (say to a DR site), backup, etc...
> DAS wins on cost

The problem with SAN is that it's apparently very easy to build a big
expensive system that performs poorly.  We've seen reports of such
here on the lists a few times.  I would definitely demand an
evaluation period from your supplier to make sure it performs well if
you go SAN.

> But for a moment keeping these aside, i wanted to discuss, purely on
> performance side which one is a winner? It feels like internal-disks will
> perform better, but need to understand a rough magnitude of difference in
> performance to see if its worth loosing the manageability features.

That really really really depends.  The quality of RAID controllers
for either setup is very important, as is the driver support, etc...
All things being even, I'd lean towards the local storage.

> Also if we choose to go with DAS, what would be the best tool to do async
> replication to DR site and maybe even as a extra plus a second read-only DB
> server to distribute select loads.

Look at slony, or PITR with continuous recovery.  Of those two, I've
only used Slony in production, and I was very happy with it's
performance, and it was very easy to write a bash script to monitor
the replication for failures.

Thanks Mark.

If I replicate a snapshot of Data and log files (basically the entire PG data directory) and I maintain same version of postgres on both servers, it should work right?

I am also thinking that having SAN storage will provide me with facility of keeping a warm standby DB. By just shutting one server down and starting the other mounting the same File system I should be able to bing my DB up when the primary inccurs a physical failure.

I'm only considering SAN storage for this feature - has anyone ever used SAN for replication and warm standy-by on Postgres?

Regards,
Harsh




--
Harsh Azad
=======================
Harsh.Azad@gmail.com

Thanks Scott, we have now requested IBM/EMC to provide test machines. Interestingly since you mentioned the importance of Raid controllers and the drivers; we are planning to use Cent OS 5 for hosting the DB.

Firstly, I could only find postgres 8.1.x RPM for CentOS 5, could not find any RPM for 8.2.4. Is there any 8.2.4 RPM for CentOS 5?

Secondly, would investing into Redhat enterprise edition give any performance advantage? I know all the SAN boxes are only certified on RHEL and not CentOS. Or since CentOS is similar to RHEL it would be fine?

Regards,
Harsh




--
Harsh Azad
=======================
Harsh.Azad@gmail.com

On Thu, 2007-09-06 at 18:05 +0530, Harsh Azad wrote:
> Hi,
>
> We are currently running our DB on a DualCore, Dual Proc 3.Ghz Xeon,
> 8GB RAM, 4x SAS 146 GB 15K RPM on RAID 5.
>
> The current data size is about 50GB, but we want to purchase the
> hardware to scale to about 1TB as we think our business will need to
> support that much soon.
> - Currently we have a 80% read and 20% write perecntages.
> - Currently with this configuration the Database is showing signs of
> over-loading.
> - Auto-vaccum, etc run on this database, vaccum full runs nightly.
> - Currently CPU loads are about 20%, memory utilization is full (but
> this is also due to linux caching disk blocks) and IO waits are
> frequent.
> - We have a load of about 400 queries per second
>
> Now we are considering to purchase our own servers and in the process
> are facing the usual dilemmas. First I'll list out what machine we
> have decided to use:
> 2x Quad Xeon 2.4 Ghz (4-way only 2 populated right now)
> 32 GB RAM
> OS Only storage - 2x SCSI 146 GB 15k RPM on RAID-1
> (Data Storage mentioned below)
>
> We have already decided to split our database into 3 machines on the
> basis on disjoint sets of data. So we will be purchasing three of
> these boxes.
>
> HELP 1: Does something look wrong with above configuration, I know
> there will be small differences b/w opetron/xeon. But do you think
> there is something against going for 2.4Ghz Quad Xeons (clovertown i
> think)?
>
> HELP 2: The main confusion is with regards to Data Storage. We have
> the option of going for:
>
> A: IBM N-3700 SAN Box, having 12x FC 300GB disks,  Partitioned into 3
> disks into RAID-4 for WAL/backup, and 9 disks on RAID-DP for data, 2
> hot spare. We are also considering similar solution from EMC -
> CX310C.
>
> B: Go for Internal of DAS based storage. Here for each server we
> should be able to have: 2x disks on RAID-1 for logs, 6x disks on
> RAID-10 for tablespace1 and 6x disks on RAID-10 for tablespace2. Or
> maybe 12x disks on RAID-10 single table-space.
>
> What do I think? Well..
> SAN wins on manageability, replication (say to a DR site), backup,
> etc...
> DAS wins on cost
>
> But for a moment keeping these aside, i wanted to discuss, purely on
> performance side which one is a winner? It feels like internal-disks
> will perform better, but need to understand a rough magnitude of
> difference in performance to see if its worth loosing the
> manageability features.
>
> Also if we choose to go with DAS, what would be the best tool to do
> async replication to DR site and maybe even as a extra plus a second
> read-only DB server to distribute select loads.

Sounds like a good candidate for Slony replication for backups /
read-only slaves.

I haven't seen a SAN yet whose DR / replication facilities are on par
with a good database replication solution.  My impression is that those
facilities are mostly for file servers, mail servers, etc.  It would be
difficult for a SAN to properly replicate a database given the strict
ordering, size and consistency requirements for the data files.  Not
impossible, but in my limited experience I haven't found one that I
trust to do it reliably either, vendor boastings to the contrary
notwithstanding.  (Hint: make sure you know exactly what your vendor's
definition of the term 'snapshot' really means).

So before you invest in a SAN, make sure that you're actually going to
be able to (and want to) use all the nice management features you're
paying for.  We have some SAN's that are basically acting just as
expensive external RAID arrays because we do the database
replication/backup in software anyway.

-- Mark Lewis

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Harsh Azad wrote:
> Thanks Scott, we have now requested IBM/EMC to provide test machines.
> Interestingly since you mentioned the importance of Raid controllers and the
> drivers; we are planning to use Cent OS 5 for hosting the DB.
>
> Firstly, I could only find postgres 8.1.x RPM for CentOS 5, could not find
> any RPM for 8.2.4. Is there any 8.2.4 RPM for CentOS 5?

Look under the RHEL section of ftp.postgresql.org

Joshua D. Drake

>
> Secondly, would investing into Redhat enterprise edition give any
> performance advantage? I know all the SAN boxes are only certified on RHEL
> and not CentOS. Or since CentOS is similar to RHEL it would be fine?
>
> Regards,
> Harsh
>
> On 9/6/07, Scott Marlowe <scott.marlowe@gmail.com> wrote:
>> On 9/6/07, Harsh Azad <harsh.azad@gmail.com> wrote:
>>> Hi,
>>>
>>> We are currently running our DB on a DualCore, Dual Proc 3.Ghz Xeon, 8GB
>>> RAM, 4x SAS 146 GB 15K RPM on RAID 5.
>>>
>>> The current data size is about 50GB, but we want to purchase the
>> hardware to
>>> scale to about 1TB as we think our business will need to support that
>> much
>>> soon.
>>> - Currently we have a 80% read and 20% write percentages.
>> For this type load, you should be running on RAID10 not RAID5.  Or, if
>> you must use RAID 5, use more disks and have a battery backed caching
>> RAID controller known to perform well with RAID5 and large arrays.
>>
>>> - Currently with this configuration the Database is showing signs of
>>> over-loading.
>> On I/O or CPU?  If you're running out of CPU, then look to increasing
>> CPU horsepower and tuning postgresql.
>> If I/O then you need to look into a faster I/O subsystem.
>>
>>> - Auto-vaccum, etc run on this database, vaccum full runs nightly.
>> Generally speaking, if you need to run vacuum fulls, you're doing
>> something wrong.  Is there a reason you're running vacuum full or is
>> this just precautionary.  vacuum full can bloat your indexes, so you
>> shouldn't run it regularly.  reindexing might be a better choice if
>> you do need to regularly shrink your db.  The better option is to
>> monitor your fsm usage and adjust fsm settings / autovacuum settings
>> as necessary.
>>
>>>  - Currently CPU loads are about 20%, memory utilization is full (but
>> this
>>> is also due to linux caching disk blocks) and IO waits are frequent.
>>> - We have a load of about 400 queries per second
>> What does vmstat et. al. say about CPU versus I/O wait?
>>
>>> Now we are considering to purchase our own servers and in the process
>> are
>>> facing the usual dilemmas. First I'll list out what machine we have
>> decided
>>> to use:
>>> 2x Quad Xeon 2.4 Ghz (4-way only 2 populated right now)
>>> 32 GB RAM
>>> OS Only storage - 2x SCSI 146 GB 15k RPM on RAID-1
>>> (Data Storage mentioned below)
>>>
>>> We have already decided to split our database into 3 machines on the
>> basis
>>> on disjoint sets of data. So we will be purchasing three of these boxes.
>>>
>>> HELP 1: Does something look wrong with above configuration, I know there
>>> will be small differences b/w opetron/xeon. But do you think there is
>>> something against going for 2.4Ghz Quad Xeons (clovertown i think)?
>> Look like good machines, plenty fo memory.
>>
>>> HELP 2: The main confusion is with regards to Data Storage. We have the
>>> option of going for:
>>>
>>> A: IBM N-3700 SAN Box, having 12x FC 300GB disks,  Partitioned into 3
>> disks
>>> into RAID-4 for WAL/backup, and 9 disks on RAID-DP for data, 2 hot
>> spare. We
>>> are also considering similar solution from EMC - CX310C.
>>>
>>> B: Go for Internal of DAS based storage. Here for each server we should
>> be
>>> able to have: 2x disks on RAID-1 for logs, 6x disks on RAID-10 for
>>> tablespace1 and 6x disks on RAID-10 for tablespace2. Or maybe 12x disks
>> on
>>> RAID-10 single table-space.
>>>
>>> What do I think? Well..
>>> SAN wins on manageability, replication (say to a DR site), backup,
>> etc...
>>> DAS wins on cost
>> The problem with SAN is that it's apparently very easy to build a big
>> expensive system that performs poorly.  We've seen reports of such
>> here on the lists a few times.  I would definitely demand an
>> evaluation period from your supplier to make sure it performs well if
>> you go SAN.
>>
>>> But for a moment keeping these aside, i wanted to discuss, purely on
>>> performance side which one is a winner? It feels like internal-disks
>> will
>>> perform better, but need to understand a rough magnitude of difference
>> in
>>> performance to see if its worth loosing the manageability features.
>> That really really really depends.  The quality of RAID controllers
>> for either setup is very important, as is the driver support, etc...
>> All things being even, I'd lean towards the local storage.
>>
>>> Also if we choose to go with DAS, what would be the best tool to do
>> async
>>> replication to DR site and maybe even as a extra plus a second read-only
>> DB
>>> server to distribute select loads.
>> Look at slony, or PITR with continuous recovery.  Of those two, I've
>> only used Slony in production, and I was very happy with it's
>> performance, and it was very easy to write a bash script to monitor
>> the replication for failures.
>>
>
>
>


- --

      === The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564   24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997  http://www.commandprompt.com/
            UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG4DT2ATb/zqfZUUQRAoppAJ9Pj+/nDtDd/XhzMdRkjXcGHHuaeACfRTfV
wE8+ErUXuVnXmlchYvCPgu8=
=TihW
-----END PGP SIGNATURE-----

On 6-9-2007 14:35 Harsh Azad wrote:
> 2x Quad Xeon 2.4 Ghz (4-way only 2 populated right now)

I don't understand this sentence. You seem to imply you might be able to
fit more processors in your system?
Currently the only Quad Core's you can buy are dual-processor
processors, unless you already got a quote for a system that yields the
new Intel "Tigerton" processors.
I.e. if they are clovertown's they are indeed Intel Core-architecture
processors, but you won't be able to fit more than 2 in the system and
get 8 cores in a system.
If they are Tigerton, I'm a bit surprised you got a quote for that,
although HP seems to offer a system for those. If they are the old
dual-core MP's (70xx or 71xx), you don't want those...

> 32 GB RAM
> OS Only storage - 2x SCSI 146 GB 15k RPM on RAID-1
> (Data Storage mentioned below)

I doubt you need 15k-rpm drives for OS... But that won't matter much on
the total cost.

> HELP 1: Does something look wrong with above configuration, I know there
> will be small differences b/w opetron/xeon. But do you think there is
> something against going for 2.4Ghz Quad Xeons (clovertown i think)?

Apart from your implication that you may be able to stick more
processors in it: no, not to me. Two Quad Core Xeons were even faster
than 8 dual core opterons in our benchmarks, although that might also
indicate limited OS-, postgres or underlying I/O-scaling.
Obviously the new AMD Barcelona-line of processors (coming next week
orso) and the new Intel Quad Core's DP (Penryn?) and MP (Tigerton) may
be interesting to look at, I don't know how soon systems will be
available with those processors (HP seems to offer a tigerton-server).

> B: Go for Internal of DAS based storage. Here for each server we should
> be able to have: 2x disks on RAID-1 for logs, 6x disks on RAID-10 for
> tablespace1 and 6x disks on RAID-10 for tablespace2. Or maybe 12x disks
> on RAID-10 single table-space.

You don't necessarily need to use internal disks for DAS, since you can
also link an external SAS-enclosure either with or without an integrated
raid-controller (IBM, Sun, Dell, HP and others have options for that),
and those are able to be expanded to either multiple enclosures tied to
eachother or to a controller in the server.
Those may also be usable in a warm-standby-scenario and may be quite a
bit cheaper than FC-hardware.

> But for a moment keeping these aside, i wanted to discuss, purely on
> performance side which one is a winner? It feels like internal-disks
> will perform better, but need to understand a rough magnitude of
> difference in performance to see if its worth loosing the manageability
> features.

As said, you don't necessarily need real internal disks, since SAS can
be used with external enclosures as well, still being DAS. I have no
idea what difference you will or may see between those in terms of
performance. It probably largely depends on the raid-controller
available, afaik the disks will be mostly the same. And it might depend
on your available bandwidth, external SAS offers you a 4port-connection
allowing for a 12Gbit-connection between a disk-enclosure and a
controller. While - as I understand it - even expensive SAN-controllers
only offer dual-ported, 8Gbit connections?
What's more important is probably the amount of disks and raid-cache you
can buy in the SAN vs DAS-scenario. If you can buy 24 disks when going
for DAS vs only 12 whith SAN...

But then again, I'm no real storage expert, we only have two Dell MD1000
DAS-units at our site.

Best regards and good luck,

Arjen

On 9/6/07, Harsh Azad <harsh.azad@gmail.com> wrote:
> Thanks Scott, we have now requested IBM/EMC to provide test machines.
> Interestingly since you mentioned the importance of Raid controllers and the
> drivers; we are planning to use Cent OS 5 for hosting the DB.

What RAID controllers have you looked at.  Seems the two most popular
in terms of performance here have been Areca and 3Ware / Escalade.
LSI seems to come in a pretty close third.  Adaptec is to be avoided
as are cheap RAID controllers (i.e. promise etc...)  battery backed
cache is a must, and the bigger the better.

> Firstly, I could only find postgres 8.1.x RPM for CentOS 5, could not find
> any RPM for 8.2.4. Is there any 8.2.4 RPM for CentOS 5?
>
> Secondly, would investing into Redhat enterprise edition give any
> performance advantage? I know all the SAN boxes are only certified on RHEL
> and not CentOS. Or since CentOS is similar to RHEL it would be fine?

for all intents and purposes, CentOS and RHEL are the same OS, so any
pgsql rpm for one should pretty much work for the other.  At the
worst, you might have to get a srpm and rebuild it for CentOS / White
Box.

Hi,

How about the Dell Perc 5/i card, 512MB battery backed cache or IBM ServeRAID-8k Adapter?

I hope I am sending relevant information here, I am not too well versed with RAID controllers.

Regards,
Harsh




--
Harsh Azad
=======================
Harsh.Azad@gmail.com

On Thu, 2007-09-06 at 22:28 +0530, Harsh Azad wrote:
> Thanks Mark.
>
> If I replicate a snapshot of Data and log files (basically the entire
> PG data directory) and I maintain same version of postgres on both
> servers, it should work right?
>
> I am also thinking that having SAN storage will provide me with
> facility of keeping a warm standby DB. By just shutting one server
> down and starting the other mounting the same File system I should be
> able to bing my DB up when the primary inccurs a physical failure.
>
> I'm only considering SAN storage for this feature - has anyone ever
> used SAN for replication and warm standy-by on Postgres?
>
> Regards,
> Harsh


We used to use a SAN for warm standby of a database, but with Oracle and
not PG.  It worked kinda sorta, except for occasional crashes due to
buggy drivers.

But after going through the exercise, we realized that we hadn't gained
anything over just doing master/slave replication between two servers,
except that it was more expensive, had a tendency to expose buggy
drivers, had a single point of failure in the SAN array, failover took
longer and we couldn't use the warm standby server to perform read-only
queries.  So we reverted back and just used the SAN as expensive DAS and
set up a separate box for DB replication.

So if that's the only reason you're considering a SAN, then I'd advise
you to spend the extra money on more DAS disks.

Maybe I'm jaded by past experiences, but the only real use case I can
see to justify a SAN for a database would be something like Oracle RAC,
but I'm not aware of any PG equivalent to that.

-- Mark Lewis

On 9/6/07, Harsh Azad <harsh.azad@gmail.com> wrote:
> Hi,
>
> How about the Dell Perc 5/i card, 512MB battery backed cache or IBM
> ServeRAID-8k Adapter?

All Dell Percs have so far been based on either adaptec or LSI
controllers, and have ranged from really bad to fairly decent
performers.  There were some recent posts on this list where someone
was benchmarking one, I believe.  searching the list archives might
prove useful.

I am not at all familiar with IBM's ServeRAID controllers.

Do either of these come with or have the option for battery back
module for the cache?

> I hope I am sending relevant information here, I am not too well versed with
> RAID controllers.

Yep.  Def look for a chance to evaluate whichever ones you're
considering.  The Areca's are in the same price range as the IBM
controller you're considering, maybe a few hundred dollars more.  See
if you can get one for review while looking at these other
controllers.

I'd recommend against Dell unless you're at a company that orders
computers by the hundred lot.  My experience with Dell has been that
unless you are a big customer you're just another number (a small one
at that) on a spreadsheet.

Scott Marlowe wrote:
> On 9/6/07, Harsh Azad <harsh.azad@gmail.com> wrote:
>
>> Hi,
>>
>> How about the Dell Perc 5/i card, 512MB battery backed cache or IBM
>> ServeRAID-8k Adapter?
>>
>
> All Dell Percs have so far been based on either adaptec or LSI
> controllers, and have ranged from really bad to fairly decent
> performers.  There were some recent posts on this list where someone
> was benchmarking one, I believe.  searching the list archives might
> prove useful.
>
> I am not at all familiar with IBM's ServeRAID controllers.
>
> Do either of these come with or have the option for battery back
> module for the cache?
>
>
>> I hope I am sending relevant information here, I am not too well versed with
>> RAID controllers.
>>
>
> Yep.  Def look for a chance to evaluate whichever ones you're
> considering.  The Areca's are in the same price range as the IBM
> controller you're considering, maybe a few hundred dollars more.  See
> if you can get one for review while looking at these other
> controllers.
>
> I'd recommend against Dell unless you're at a company that orders
> computers by the hundred lot.  My experience with Dell has been that
> unless you are a big customer you're just another number (a small one
> at that) on a spreadsheet.
>
If you do go with Dell get connected with an account manager instead of
ordering online.  You work with the same people every time you have an
order and in my experience they can noticeably beat the best prices I
can find.  This is definitely the way to go if you don't want to get
lost in the volume.  The group I have worked with for the past ~2 years
is very responsive, remembers me and my company across the 3 - 6 month
gaps between purchases, and the server/storage person in the group is
reasonably knowledgeable and helpful.  This is for small lots of
machines, our first order was just 2 boxes and i've only placed 4 orders
total in the past 2 years.

Just my personal experience, i'd be happy to pass along the account
manager's information if anyone is interested.
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
>        subscribe-nomail command to majordomo@postgresql.org so that your
>        message can get through to the mailing list cleanly
>
Joe Uhl
joeuhl@gmail.com

I am not sure I agree with that evaluation.
I only have 2 dell database servers and they have been 100% reliable.
Maybe he is referring to support which does tend be up to who you get.
When I asked about performance on my new server they were very helpful but I
did have a bad time on my NAS device (but had the really cheap support plan
on it). They did help me get it fixed but I had to RMA all the drives on the
NAS as they were all bad and it was no fun installing the os as it had no
floppy. I got the better support for both the data base servers which are
using jbod from dell for the disk array. The quad proc opteron with duel
cores and 16gig of memory has been extremely fast (like 70%) over my older 4
proc 32 bit single core machine with 8 gig. But both are running postgres
and perform needed functionality. I would like to have redundant backups of
these as they are mission critical, but all in good time.

I'd recommend against Dell unless you're at a company that orders
computers by the hundred lot.  My experience with Dell has been that
unless you are a big customer you're just another number (a small one
at that) on a spreadsheet.

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
       subscribe-nomail command to majordomo@postgresql.org so that your
       message can get through to the mailing list cleanly

On 6-9-2007 20:42 Scott Marlowe wrote:
> On 9/6/07, Harsh Azad <harsh.azad@gmail.com> wrote:
>> Hi,
>>
>> How about the Dell Perc 5/i card, 512MB battery backed cache or IBM
>> ServeRAID-8k Adapter?
>
> All Dell Percs have so far been based on either adaptec or LSI
> controllers, and have ranged from really bad to fairly decent
> performers.  There were some recent posts on this list where someone
> was benchmarking one, I believe.  searching the list archives might
> prove useful.

The Dell PERC5-cards are based on LSI-chips and perform quite well.
Afaik Dell hasn't used adaptecs for a while now, but even recent
(non-cheap ;) ) adaptecs aren't that bad afaik.

The disadvantage of using Areca or 3Ware is obviously the lack of
support in A-brand servers and the lack of support for SAS-disks. Only
recently Areca has stepped in the SAS-market, but I have no idea how
easily those controllers are integrated in standard servers (they tend
to be quite large, which will not fit in 2U and maybe not even in 3U or
4U-servers).

Arjen

On 6-9-2007 20:29 Mark Lewis wrote:
> Maybe I'm jaded by past experiences, but the only real use case I can
> see to justify a SAN for a database would be something like Oracle RAC,
> but I'm not aware of any PG equivalent to that.

PG Cluster II seems to be able to do that, but I don't know whether
that's production quality already...

Arjen

On 9/6/07, Joel Fradkin <jfradkin@wazagua.com> wrote:
> I am not sure I agree with that evaluation.
> I only have 2 dell database servers and they have been 100% reliable.
> Maybe he is referring to support which does tend be up to who you get.
> When I asked about performance on my new server they were very helpful but I
> did have a bad time on my NAS device (but had the really cheap support plan
> on it). They did help me get it fixed but I had to RMA all the drives on the
> NAS as they were all bad and it was no fun installing the os as it had no
> floppy. I got the better support for both the data base servers which are
> using jbod from dell for the disk array. The quad proc opteron with duel
> cores and 16gig of memory has been extremely fast (like 70%) over my older 4
> proc 32 bit single core machine with 8 gig. But both are running postgres
> and perform needed functionality. I would like to have redundant backups of
> these as they are mission critical, but all in good time.

Dell's ok if by support you mean replacing simple broken parts, etc...

I'm talking about issues like the one we had with our 26xx servers
which, thankfully, Dell hasn't made in a while.  We had the adaptec
controllers that locked up once every 1 to 3 months for no reason, and
with 4 servers this meant a lockup of one every 1 to 2 weeks.  Not
acceptable in a production environment.  It took almost 2 years to get
Dell to agree to ship us the LSI based RAID controllers for those 4
machines.  Our account rep told us not to worry about returning the
RAID controllers as they were so old as to be obsolete.  One of the
four replacement controllers they sent us was bad, so they
cross-shipped another one.  Sometimes you get a broken part, it
happens.

One month after we got all our RAID controllers replaced, we started
getting nasty calls from their parts people wanting those parts back,
saying they were gonna charge us for them, etc...  Two thirds of the
parts were still in the server because we hadn't gotten a sheet
identifying them (the RAID key and battery) and had left in not
worrying about it because we'd been told they were needed.

These are production servers, we can't just shut them off for fun to
pull a part we were told we didn't need to return.

On top of that, the firmware updates come as an .exe that has to be
put on a windows floppy.  We don't have a single windows machine with
a floppy at the company I work for (lots of laptops with windows).  We
had to dig out a floppy drive and a windows CD to create a bootable
floppy to install a firmware update for a linux server.

So, my main complaint is about their customer service.  The machines,
when they work, are pretty good.  Normally the hardware gets along
with RH OSes.  But when things go wrong, Dell will not admit to a
design problem that's staring them in the face, and they have wasted
literally hundreds of man hours for us with their stonewalling on this
subject.  So, I see no reason to buy more hardware from them when
there are dealers big and small who have so far treated us much
better.

On Thu, 6 Sep 2007, Harsh Azad wrote:

> Firstly, I could only find postgres 8.1.x RPM for CentOS 5, could not find
> any RPM for 8.2.4. Is there any 8.2.4 RPM for CentOS 5?

You've already been pointed in the right direction. Devrim, the person who
handles this packaging, does a great job of building all the RPMs.  But I
have a small philisophical difference with the suggested instructions for
actually installing them though.  I just finished an alternate
installation guide for RHEL/CentOS 5 that's now posted at
http://www.westnet.com/~gsmith/content/postgresql/pgrpm.htm you may want
to take a look at.

> Secondly, would investing into Redhat enterprise edition give any
> performance advantage? I know all the SAN boxes are only certified on
> RHEL and not CentOS. Or since CentOS is similar to RHEL it would be
> fine?

Wouldn't expect a performance advantage.  The situation you have to ask
consider is this:  your SAN starts having funky problems, and your
database is down because of it.  You call the vendor.  They find out
you're running CentOS instead of RHEL and say that's the cause of your
problem (even though it probably isn't).  How much will such a passing the
buck problem cost your company?  If it's a significant number, you'd be
foolish to run CentOS instead of the real RHEL.  Some SAN vendors can be
very, very picky about what they will support, and for most business
environments the RHEL subscription isn't so expensive that it's worth
wandering into an area where your support situation is fuzzy just to save
that money.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

* Arjen van der Meijden:

> The disadvantage of using Areca or 3Ware is obviously the lack of
> support in A-brand servers and the lack of support for SAS-disks. Only
> recently Areca has stepped in the SAS-market, but I have no idea how
> easily those controllers are integrated in standard servers (they tend
> to be quite large, which will not fit in 2U and maybe not even in 3U
> or 4U-servers).

Recent 3ware controllers are a bit on the hot side, too.  We had to
switch from two 12 port controllers to a single 24 port controller
because of that (combined with an unlucky board layout: the two 8x
PCIe connectors are next to each other).

Unfortunately, read performance maxes out at about 8 disks in a
RAID-10 configuration.  Software RAID-0 across hardware RAID-1 is
significantly faster (factor of 3 to 5 in low-level benchmarks).
However, it seems that something in this stack does not enforce write
barriers properly, so I don't think we will use this in production.

RAID-6 doesn't perform well, either (especially for several processes
reading different files sequentially).

We'll probably split the 24 disks into a couple of RAID-10s, and
distribute tables and indexes manually among the file systems.  This
is a bit disappointing, especially because the system is able to read
at 800+ MB/s, as shown by the software-RAID-on-hardware-RAID
configuration.

I haven't seen 24-disk benchmarks with Areca controllers.  A
comparison might be interesting.

--
Florian Weimer                <fweimer@bfk.de>
BFK edv-consulting GmbH       http://www.bfk.de/
Kriegsstraße 100              tel: +49-721-96201-1
D-76133 Karlsruhe             fax: +49-721-96201-99

On Fri, Sep 07, 2007 at 12:26:23AM -0400, Greg Smith wrote:
>consider is this:  your SAN starts having funky problems, and your
>database is down because of it.  You call the vendor.  They find out
>you're running CentOS instead of RHEL and say that's the cause of your
>problem (even though it probably isn't).  How much will such a passing the
>buck problem cost your company?  If it's a significant number, you'd be
>foolish to run CentOS instead of the real RHEL.  Some SAN vendors can be
>very, very picky about what they will support, and for most business
>environments the RHEL subscription isn't so expensive that it's worth
>wandering into an area where your support situation is fuzzy just to save
>that money.

Correct. Far more sensible to skip the expensive SAN solution, not worry
about having to play games, and save *even more* money.

SANs have their place, but postgres storage generally isn't it; you'll
get more bang/buck with DAS and very likely better absolute performance
as well.  SANs make sense if you're doing a shared filesystem (don't
even think about doing this with postgres), or if you're consolidating
backups & DR (which doesn't work especially well with databases).

Mike Stone

We're also considering to install postgres on SAN - that is, my boss is
convinced this is the right way to go.

Advantages:

 1. Higher I/O (at least the salesman claims so)
 2. Easier to upgrade the disk capacity
 3. Easy to set up "warm standby" functionality.  (Then again, if the
 postgres server fails miserably, it's likely to be due to a disk
 crash).

Also, my boss states that "all big enterprises uses SAN nowadays".

Disadvantages:

 1. Risky?  One gets the impression that there are frequent problems
 with data integrity when reading some of the posts in this thread.

 2. Expensive

 3. "Single point of failure" ... but that you have either it's a SAN or
 a local disk, one will anyway need good backup systems (and eventually
 "warm standby"-servers running from physically separated disks).

 4. More complex setup?

 5. If there are several hosts with write permission towards the same
 disk, I can imagine the risks being higher for data integrity
 breakages.  Particularly, I can imagine that if two postgres instances
 is started up towards the same disk (due to some sysadmin mistake), it
 could be disasterous.

On Fri, Sep 07, 2007 at 12:33:41PM +0200, Tobias Brox wrote:
>Advantages:
>
> 1. Higher I/O (at least the salesman claims so)

Benchmark it. It is extremely unlikely that you'll get I/O *as good as*
DAS at a similar price point.

> 2. Easier to upgrade the disk capacity

Is this an issue? You may find that you can simply get dramatically more
space for the money with DAS and not have to worry about an upgrade.
Also, you can use the postgres tablespace functionality to migrate data
to a new partition fairly transparently.

> 3. Easy to set up "warm standby" functionality.  (Then again, if the
> postgres server fails miserably, it's likely to be due to a disk
> crash).

You may find that using db replication will gain you even more
reliability for less money.

>Also, my boss states that "all big enterprises uses SAN nowadays".

Use SAN *for what*?

Mike Stone

I'm getting a san together to consolidate my disk space usage for my
servers.  It's iscsi based and I'll be pxe booting my servers from it.
The idea is to keep spares on hand for one system (the san) and not have
to worry about spares for each specific storage system on each server.
This also makes growing filesystems and such pretty simple.  Redundancy
is also good since I'll have two iscsi switches plugged into two cisco
ethernet switches and two different raid controllers on the jbod.  I'll
start plugging my servers into each switch for further redundancy.  In
the end I could loose disks, ethernet switches, cables, iscsi switches,
raid controller, whatever, and it keeps on moving.

That said, I'm not putting my postgres data on the san.  The DB server
will boot from the san and use it for is OS, but there are 6 15k SAS
disks in it setup with raid-10 that will be used for the postgres data
mount.  The machine is a dell 2950 and uses an LSI raid card.

The end result is a balance of cost, performance, and reliability.  I'm
using iscsi for the cost, reliability, and ease of use, but where I need
performance I'm sticking to local disks.

schu

We are currently running our database against on SAN share.  It looks like this:

2 x RAID 10 (4 disk SATA 7200 each)

Raid Group 0 contains the tables + indexes
Raid Group 1 contains the log files + backups (pg_dump)

Our database server connects to the san via iSCSI over Gig/E using
jumbo frames.  File system is XFS (noatime).

I believe our raid controller is an ARECA.  Whatever it is, it has the
option of adding a battery to it but I have not yet been able to
convince my boss that we need it.

Maintenance is nice, we can easily mess around with the drive shares,
expand and contract them, snapshot them, yadda yadda yadda.  All
things which we NEVER do to our database anyway. :)

Performance, however, is a mixed bag.  It handles concurrency very
well.  We have a number of shares (production shares, data shares, log
file shares, backup shares, etc. etc.) spread across the two raid
groups and it handles them with aplomb.

Throughput, however, kinda sucks.  I just can't get the kind of
throughput to it I was hoping to get.  When our memory cache is blown,
the database can be downright painful for the next few minutes as
everything gets paged back into the cache.

I'd love to try a single 8 disk RAID 10 with battery wired up directly
to our database, but given the size of our company and limited funds,
it won't be feasible any time soon.

Bryan

On 9/7/07, Matthew Schumacher <matt.s@aptalaska.net> wrote:
> I'm getting a san together to consolidate my disk space usage for my
> servers.  It's iscsi based and I'll be pxe booting my servers from it.
> The idea is to keep spares on hand for one system (the san) and not have
> to worry about spares for each specific storage system on each server.
> This also makes growing filesystems and such pretty simple.  Redundancy
> is also good since I'll have two iscsi switches plugged into two cisco
> ethernet switches and two different raid controllers on the jbod.  I'll
> start plugging my servers into each switch for further redundancy.  In
> the end I could loose disks, ethernet switches, cables, iscsi switches,
> raid controller, whatever, and it keeps on moving.
>
> That said, I'm not putting my postgres data on the san.  The DB server
> will boot from the san and use it for is OS, but there are 6 15k SAS
> disks in it setup with raid-10 that will be used for the postgres data
> mount.  The machine is a dell 2950 and uses an LSI raid card.
>
> The end result is a balance of cost, performance, and reliability.  I'm
> using iscsi for the cost, reliability, and ease of use, but where I need
> performance I'm sticking to local disks.
>
> schu
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
>        subscribe-nomail command to majordomo@postgresql.org so that your
>        message can get through to the mailing list cleanly
>

On Friday 07 September 2007 10:56, "Bryan Murphy"
<bryan.murphy@gmail.com> wrote:
> Our database server connects to the san via iSCSI over Gig/E using
> jumbo frames.  File system is XFS (noatime).
>
> Throughput, however, kinda sucks.  I just can't get the kind of
> throughput to it I was hoping to get.

A single Gig/E couldn't even theoretically do better than 125MB/sec, so
yeah I would expect throughput sucks pretty bad.

--
"A democracy is a sheep and two wolves deciding on what to have for
lunch. Freedom is a well armed sheep contesting the results of the
decision." -- Benjamin Franklin

Bryan Murphy wrote:

>Our database server connects to the san via iSCSI over Gig/E using
>jumbo frames.  File system is XFS (noatime).
>
>
>
>
...

>Throughput, however, kinda sucks.  I just can't get the kind of
>throughput to it I was hoping to get.  When our memory cache is blown,
>the database can be downright painful for the next few minutes as
>everything gets paged back into the cache.
>
>
>

Remember that Gig/E is bandwidth limited to about 100 Mbyte/sec.  Maybe
a little faster than that downhill with a tailwind, but not much.
You're going to get much better bandwidth connecting to a local raid
card talking to local disks simply due to not having the ethernet as a
bottleneck.  iSCSI is easy to set up and manage, but it's slow.  This is
the big advantage Fibre Channel has- serious performance.  You can have
multiple channels on a single fibre channel card- IIRC, QLogic's cards
have a default of 4 channels- each pumping 400 Mbyte/sec.  At which
point the local bus rapidly becomes the bottleneck.  Of course, this
comes at the cost of a signifigant increase in complexity.

Brian

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alan Hodgson wrote:
> On Friday 07 September 2007 10:56, "Bryan Murphy"
> <bryan.murphy@gmail.com> wrote:
>> Our database server connects to the san via iSCSI over Gig/E using
>> jumbo frames.  File system is XFS (noatime).
>>
>> Throughput, however, kinda sucks.  I just can't get the kind of
>> throughput to it I was hoping to get.
>
> A single Gig/E couldn't even theoretically do better than 125MB/sec, so
> yeah I would expect throughput sucks pretty bad.

We have a customer that has a iSCSI SAN that can bond multiple Gig/E
connections that provides them with reasonable performance. You should
see if yours allows it.

Joshua D. Drake


>


- --

      === The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564   24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997  http://www.commandprompt.com/
            UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG4ZbGATb/zqfZUUQRAhtmAKCh/PsmkL/JOPq4++Aci2/XwDDJ7wCfbwJs
5vBg+TG5xQFKoJMdybpjDWo=
=up8R
-----END PGP SIGNATURE-----

On Fri, 7 Sep 2007, Tobias Brox wrote:

> We're also considering to install postgres on SAN - that is, my boss is
> convinced this is the right way to go.
> Advantages:
> 1. Higher I/O (at least the salesman claims so)

Shockingly, the salesman is probably lying to you.  The very concept of
SAN says that you're putting something in between your system and the
disks, and that something therefore must slow things down compared to
connecting directly.  iSCSI, FC, whatever you're using as the
communications channel can't be as fast as a controller card with a good
interface straight into the motherboard.  For example, a PCI-E x16 disk
controller card maxes out at 4GB/s in each direction; good luck bonding
enough iSCSI or FC channels together to reach that transfer rate and
getting something even remotely cost-competative with an internal card.

The cases where a SAN can improve upon performance over direct discs are
when the comparison isn't quite fair; for example:

1) The SAN allows spreading the load over more disks than you can fit
internally in the system
2) The SAN provides a larger memory cache than the internal cards you're
comparing against

If you're in one of those situations, then perhaps the salesman's claim
could have some merit.  There are lots of reasons one might want to use a
SAN, but a higher I/O rate when fairly comparing to connecting disks
directly is unlikely to be on that list.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

On Sep 6, 2007, at 2:42 PM, Scott Marlowe wrote:

> I'd recommend against Dell unless you're at a company that orders
> computers by the hundred lot.  My experience with Dell has been that
> unless you are a big customer you're just another number (a small one
> at that) on a spreadsheet.

I order maybe 5-6 servers per year from dell, and the sales rep knows
me when I call him.  Just set up a business account.

That said, lately I've been buying Sun X4100's for my DB servers.
These machines are built like tanks and extremely fast.  The only
difficulty is hooking up disks to them.  The only sane choice is to
use a fibre channel card to an external array.  The only dual-channel
internal SCSI RAID controller that fits is an Adaptec model, and it
is to be avoided.

On Fri, 7 Sep 2007, Tobias Brox wrote:

> We're also considering to install postgres on SAN - that is, my boss is
> convinced this is the right way to go.
>
> Advantages:
>
> 1. Higher I/O (at least the salesman claims so)

only if you buy better disks for the SAN then for the local system (note
that this includes battery backed ram for write caching. the SAN will
include a bunch becouse it's performance would _suck_ otherwise. if you
don't put any on your stand-alone system you are comparing apples to
oranges)

> 2. Easier to upgrade the disk capacity

only if you buy a SAN with a lot of empty drive slots, but wouldn't buy a
system with empty drive slots.

> 3. Easy to set up "warm standby" functionality.  (Then again, if the
> postgres server fails miserably, it's likely to be due to a disk
> crash).

and if postgres dies for some other reason the image on disk needs repair,
unless you script stopping postgres when the SAN does it's snapshots,
those snapshots are not going to be that good. the problems are useually
repairable, but that makes starting your warm spare harder.

> Also, my boss states that "all big enterprises uses SAN nowadays".

your bos would be very surprised at what the really big shops are doing
(and not doing). yes they have a SAN, they have many SANs, from many
different vendors, and they have many systems that don't use the SAN and
use local disks instead. when you get really large you can find just about
anything _somewhere_ in the company.

> Disadvantages:
>
> 1. Risky?  One gets the impression that there are frequent problems
> with data integrity when reading some of the posts in this thread.

SAN's add more parts and more potential points of failure, then when you
add the SAN replication to the mix things get even more 'interesting'.
doing SAN replication across a significant distance to your DR facility
can be a LOT harder to get working right then the salesman makes it sound.
it's not uncommon to see a san replication decide that it's going to take
a week to catch up after doing a DR test for example.

> 2. Expensive

no, _extremely expensive. price one and then look at how much hardware you
could buy instead. you can probably buy much mroe storage, and a couple
complete spare systems (do replication to a local spare as well as your
remote system) and end up with even more reliability.

> 3. "Single point of failure" ... but that you have either it's a SAN or
> a local disk, one will anyway need good backup systems (and eventually
> "warm standby"-servers running from physically separated disks).

no, with local disks you can afford to have multiple systems so that you
don't have a SPOF

> 4. More complex setup?
>
> 5. If there are several hosts with write permission towards the same
> disk, I can imagine the risks being higher for data integrity
> breakages.  Particularly, I can imagine that if two postgres instances
> is started up towards the same disk (due to some sysadmin mistake), it
> could be disasterous.

when you are useing a SAN for a database the SAN vendor will have you
allocate complete disks to each box, so you don't have multiple boxes
hitting the same drive, but you also don't get a lot of the anvantages the
salesman talks about.

David Lang

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

david@lang.hm wrote:
> On Fri, 7 Sep 2007, Tobias Brox wrote:
>
>> We're also considering to install postgres on SAN - that is, my boss is
>> convinced this is the right way to go.
>>
>> Advantages:
>>
>> 1. Higher I/O (at least the salesman claims so)
>

In general a SAN does not provide more I/O than direct attached storage.
It is all about the BUS, Controller and drive types.

> only if you buy better disks for the SAN then for the local system (note
> that this includes battery backed ram for write caching. the SAN will
> include a bunch becouse it's performance would _suck_ otherwise. if you
> don't put any on your stand-alone system you are comparing apples to
> oranges)
>
>> 2. Easier to upgrade the disk capacity
>
> only if you buy a SAN with a lot of empty drive slots, but wouldn't buy
> a system with empty drive slots.

Well there are SANs that have trays that can be stacked, but then again
you can get the same thing with DAS too.

>
>> 3. Easy to set up "warm standby" functionality.  (Then again, if the
>> postgres server fails miserably, it's likely to be due to a disk
>> crash).
>
>> Also, my boss states that "all big enterprises uses SAN nowadays".
>

Uhmm as someone who consults with many of the big enterprises that are
running PostgreSQL, that is *not* true.

>> 2. Expensive
>
> no, _extremely expensive. price one and then look at how much hardware

Let me just +1 this. The amount of DAS storage you can get for 30k is
amazing compared to the amount of SAN you can get for 30k.

Joshua D. Drake

> you could buy instead. you can probably buy much mroe storage, and a
> couple complete spare systems (do replication to a local spare as well
> as your remote system) and end up with even more reliability.
>
>> 3. "Single point of failure" ... but that you have either it's a SAN or
>> a local disk, one will anyway need good backup systems (and eventually
>> "warm standby"-servers running from physically separated disks).
>
> no, with local disks you can afford to have multiple systems so that you
> don't have a SPOF
>
>> 4. More complex setup?
>>
>> 5. If there are several hosts with write permission towards the same
>> disk, I can imagine the risks being higher for data integrity
>> breakages.  Particularly, I can imagine that if two postgres instances
>> is started up towards the same disk (due to some sysadmin mistake), it
>> could be disasterous.
>
> when you are useing a SAN for a database the SAN vendor will have you
> allocate complete disks to each box, so you don't have multiple boxes
> hitting the same drive, but you also don't get a lot of the anvantages
> the salesman talks about.
>
> David Lang
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings
>


- --

      === The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564   24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997  http://www.commandprompt.com/
            UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG4b9/ATb/zqfZUUQRAnBiAJ4kdOicN3If4scLAVdaU4nS+srGHQCgnkR2
C6RvSyLcAtgQ1bJJEau8s00=
=lqbw
-----END PGP SIGNATURE-----

On Fri, Sep 07, 2007 at 02:10:32PM -0700, david@lang.hm wrote:
> >3. Easy to set up "warm standby" functionality.  (Then again, if the
> >postgres server fails miserably, it's likely to be due to a disk
> >crash).
>
> and if postgres dies for some other reason the image on disk needs repair,
> unless you script stopping postgres when the SAN does it's snapshots,
> those snapshots are not going to be that good. the problems are useually
> repairable, but that makes starting your warm spare harder.

Uh, the "image" you get from a PITR backup "needs repair" too. There's
absolutely nothing wrong with using a SAN or filesystem snapshot as a
backup mechanism, as long as it's a true snapshot, and it includes *all*
PostgreSQL data (everything under $PGDATA as well as all tablespaces).

Also, to reply to someone else's email... there is one big reason to use
a SAN over direct storage: you can do HA that results in 0 data loss.
Good SANs are engineered to be highly redundant, with multiple
controllers, PSUs, etc, so that the odds of losing the SAN itself are
very, very low. The same isn't true with DAS.

But unless you need that kind of HA recovery, I'd tend to stay away from
SANs.

BTW, if you need *serious* bandwidth, look at things like Sun's
"thumper" (I know there's at least one other company that makes
something similar). 40-48 drives in a single 4U chassis == lots of
throughput.

Finally, if you do get a SAN, make sure and benchmark it. I've seen more
than one case of a SAN that wasn't getting anywhere near the performance
it should be, even with a simple dd test.
--
Decibel!, aka Jim Nasby                        decibel@decibel.org
EnterpriseDB      http://enterprisedb.com      512.569.9461 (cell)

On Tue, Sep 11, 2007 at 03:55:51PM -0500, Decibel! wrote:
>Also, to reply to someone else's email... there is one big reason to use
>a SAN over direct storage: you can do HA that results in 0 data loss.
>Good SANs are engineered to be highly redundant, with multiple
>controllers, PSUs, etc, so that the odds of losing the SAN itself are
>very, very low. The same isn't true with DAS.

You can get DAS arrays with multiple controllers, PSUs, etc.  DAS !=
single disk.

Mike Stone

On Tue, Sep 11, 2007 at 05:09:00PM -0400, Michael Stone wrote:
> On Tue, Sep 11, 2007 at 03:55:51PM -0500, Decibel! wrote:
> >Also, to reply to someone else's email... there is one big reason to use
> >a SAN over direct storage: you can do HA that results in 0 data loss.
> >Good SANs are engineered to be highly redundant, with multiple
> >controllers, PSUs, etc, so that the odds of losing the SAN itself are
> >very, very low. The same isn't true with DAS.
>
> You can get DAS arrays with multiple controllers, PSUs, etc.  DAS !=
> single disk.

It's still in the same chassis, though, which means if you lose memory
or mobo you're still screwed. In a SAN setup for redundancy, there's
very little in the way of a single point of failure; generally only the
backplane, and because there's very little that's on there it's
extremely rare for one to fail.
--
Decibel!, aka Jim Nasby                        decibel@decibel.org
EnterpriseDB      http://enterprisedb.com      512.569.9461 (cell)

On Tue, 11 Sep 2007, Decibel! wrote:

> On Tue, Sep 11, 2007 at 05:09:00PM -0400, Michael Stone wrote:
>> On Tue, Sep 11, 2007 at 03:55:51PM -0500, Decibel! wrote:
>>> Also, to reply to someone else's email... there is one big reason to use
>>> a SAN over direct storage: you can do HA that results in 0 data loss.
>>> Good SANs are engineered to be highly redundant, with multiple
>>> controllers, PSUs, etc, so that the odds of losing the SAN itself are
>>> very, very low. The same isn't true with DAS.
>>
>> You can get DAS arrays with multiple controllers, PSUs, etc.  DAS !=
>> single disk.
>
> It's still in the same chassis, though, which means if you lose memory
> or mobo you're still screwed. In a SAN setup for redundancy, there's
> very little in the way of a single point of failure; generally only the
> backplane, and because there's very little that's on there it's
> extremely rare for one to fail.

not nessasarily. direct attached doesn't mean in the same chassis,
external drive shelves attached via SCSI are still DAS

you can even have DAS attached to a pair of machines, with the second box
configured to mount the drives only if the first one dies.

David Lang

Yeah, the DAS we are considering is Dell MD3000, it has redundant hot swappable raid controllers in active-active mode. Provision for hot spare hard-disk. And it can take upto 15 disks in 3U, you can attach two more MD1000 to it, giving a total of 45 disks in total.

-- Harsh




--
Harsh Azad
=======================
Harsh.Azad@gmail.com

[Decibel! - Tue at 06:07:44PM -0500]
> It's still in the same chassis, though, which means if you lose memory
> or mobo you're still screwed. In a SAN setup for redundancy, there's
> very little in the way of a single point of failure; generally only the
> backplane, and because there's very little that's on there it's
> extremely rare for one to fail.

Funny, the only time we lost a database server was due to a backplane
failure ...

On Tue, Sep 11, 2007 at 06:07:44PM -0500, Decibel! wrote:
>On Tue, Sep 11, 2007 at 05:09:00PM -0400, Michael Stone wrote:
>> You can get DAS arrays with multiple controllers, PSUs, etc.  DAS !=
>> single disk.
>
>It's still in the same chassis, though,

I think you're confusing DAS and internal storage.

Mike Stone