Thread: more 10K disks or less 15K disks

I am faced with a  hardware choice for a postgres data warehouse
(extremely high volume inserts..over 200 million records a day) with a
total storage of either

12 x 600G disks (15K)  (the new Dell Poweredge C server)

or

24 x 600G  (10K disks)

ALL direct attached storage.

I am leaning toward the 24 disks as I expect the higher number of
disks to provide overall better performance under high loads

Does anyone have any experience with a mixed 10K / 15K DAS storage
that you can share.

Thank you

Sriram

I forgot to add that the 24 10K disks are 2.5 inch and the 12 15K
disks are 3.5 inch

On Wed, Apr 28, 2010 at 11:27 AM, Anj Adu <fotographs@gmail.com> wrote:
> I am faced with a  hardware choice for a postgres data warehouse
> (extremely high volume inserts..over 200 million records a day) with a
> total storage of either
>
> 12 x 600G disks (15K)  (the new Dell Poweredge C server)
>
> or
>
> 24 x 600G  (10K disks)
>
> ALL direct attached storage.
>
> I am leaning toward the 24 disks as I expect the higher number of
> disks to provide overall better performance under high loads
>
> Does anyone have any experience with a mixed 10K / 15K DAS storage
> that you can share.
>
> Thank you
>
> Sriram
>

Our experience shows that the 2.5 inch 10K disks have about the same
I/O rate as the 3.5 inch 15K disks. Ideal would be 15K 2.5 inch.

To answer you bigger question, it depends on how many threads are
running to perform the inserts. If you have lots (2-3 times as many threads
as disks) then your I/O will spread out over all of your disks, and you
can make use of all drives simultaneously.
If you have fewer threads than you have disks, then latency will be your limiting
factor, so the faster speed drives will be your best bet.

For a single thread, on 10K disks with only small write cache (256M) you will
probably only see approx 1500 random I/O per second. Bear in mind that a single database
transaction in postgres will use some number of threads, but require more than 1 I/O for
each transaction. You can probably tune it to achieve no more than 2,000 inserts per second
as a sustained rate.

If you have very few threads (5-10) then you will probably require some
fancier storage, such as solid state disk or at the very least very large (4-16GB)
write cache systems.

Evan.

Anj Adu wrote:
> I forgot to add that the 24 10K disks are 2.5 inch and the 12 15K
> disks are 3.5 inch
>
> On Wed, Apr 28, 2010 at 11:27 AM, Anj Adu <fotographs@gmail.com> wrote:
>> I am faced with a  hardware choice for a postgres data warehouse
>> (extremely high volume inserts..over 200 million records a day) with a
>> total storage of either
>>
>> 12 x 600G disks (15K)  (the new Dell Poweredge C server)
>>
>> or
>>
>> 24 x 600G  (10K disks)
>>
>> ALL direct attached storage.
>>
>> I am leaning toward the 24 disks as I expect the higher number of
>> disks to provide overall better performance under high loads
>>
>> Does anyone have any experience with a mixed 10K / 15K DAS storage
>> that you can share.
>>
>> Thank you
>>
>> Sriram

I am faced with a  hardware choice for a postgres data warehouse
(extremely high volume inserts..over 200 million records a day) with a
total storage of either

12 x 600G disks (15K)  (the new Dell Poweredge C server)

or

24 x 600G  (10K disks)

ALL direct attached storage.

I am leaning toward the 24 disks as I expect the higher number of
disks to provide overall better performance under high loads

Does anyone have any experience with a mixed 10K / 15K DAS storage
that you can share.

Thank you

Sriram

Thank you all for the comments

We have not benchmarked the new hardware yet..however..we do have
existing hardware that deals with very high volumes and handle pretty
well (Dell 2950 Intel 5430 8-cores with 6x450G 15K disks and 32G RAM -
Perc 6iRaid controllers) with minimal IO wait . there are ususally
about10-15 threads running at peak time with an average of 4-10
threads overall.

We are looking at high density boxes due to increasing storage requirements.

Sriram



On Wed, Apr 28, 2010 at 1:29 PM, Iñigo Martinez Lasala
<imartinez@vectorsf.com> wrote:
> Supposing a 50% performance increase disk-by-disk with 15.000rpm vs
> 10.000rpm you would get better performance (100%) by doubling number of
> disks versus using 15K rpm disk (50%).
> However, you have to check other parameters, for example, if your RAID
> controller can deal with such a high bandwidth or the disk cache size.
> Do you have benchmarks about these hard disk models ?
>
> How about using SSD? ;-)
>
> -----Original Message-----
> From: Anj Adu <fotographs@gmail.com>
> To: pgsql-admin@postgresql.org
> Subject: [ADMIN] more 10K disks or less 15K disks
> Date: Wed, 28 Apr 2010 11:27:26 -0700
>
> I am faced with a  hardware choice for a postgres data warehouse
> (extremely high volume inserts..over 200 million records a day) with a
> total storage of either
>
> 12 x 600G disks (15K)  (the new Dell Poweredge C server)
>
> or
>
> 24 x 600G  (10K disks)
>
> ALL direct attached storage.
>
> I am leaning toward the 24 disks as I expect the higher number of
> disks to provide overall better performance under high loads
>
> Does anyone have any experience with a mixed 10K / 15K DAS storage
> that you can share.
>
> Thank you
>
> Sriram
>
>
>

On Wed, Apr 28, 2010 at 3:18 PM, Anj Adu <fotographs@gmail.com> wrote:
> Thank you all for the comments
>
> We have not benchmarked the new hardware yet..however..we do have
> existing hardware that deals with very high volumes and handle pretty
> well (Dell 2950 Intel 5430 8-cores with 6x450G 15K disks and 32G RAM -
> Perc 6iRaid controllers) with minimal IO wait . there are ususally
> about10-15 threads running at peak time with an average of 4-10
> threads overall.
>
> We are looking at high density boxes due to increasing storage requirements.

I assume the 24 2.5" box is the supermicro 2U machine?  Those are
quite impressive little chassis's.  If you need something bigger, take
a look at this:

http://www.aberdeeninc.com/abcatg/Stirling-X881.htm

You can add up to 8 16 disk JBOD boxes to it if needs be as well.

Anj Adu wrote:
> I am faced with a  hardware choice for a postgres data warehouse
> (extremely high volume inserts..over 200 million records a day)

That's an average of 2314 per second, which certainly isn't easy to pull
off.  You suggested you're already running this app.  Do you have any
idea how high the volume of data going to the WAL is relative to
everything else?

> 12 x 600G disks (15K)  (the new Dell Poweredge C server)
> or
> 24 x 600G  (10K disks)
>

You can expect the 15K disks to be 30% (more sequential work) to 50%
(random/commit work) faster than a similar 10K drive.  So from most
perspectives, twice as many 10K drives should be considerably faster.
The main point of concern here is the commit rate, which you can't
necessarily improve just by throwing drives at it.  That's based on how
fast the drives spin, so there's the potential to discover a 50%
regression there compared to the setup you have right now.  With the
RAID card in there, it should be fine, but it's something to be
concerned about.

Also, you didn't mentioned the RAID card for the new system, and whether
it would be the same in both setups or not.  That can be as important as
the drives when you have larger arrays.  The LSI Megaraid card Dell is
using for the Perc6i is quite fast, and you'll need to make sure you get
something just as good for the new server.

--
Greg Smith  2ndQuadrant US  Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com   www.2ndQuadrant.us

Thanks Greg

We do not archive the WALs. We use application-level replication to
achieve redundancy. WAL archiving was difficult to support with the
earlier hardware we had ( 6x300G 10K disks Dell 2850) given the
volumes we were dealing with. The RAID card should be from the same
manufacturer (LSI in Dell's case).

On our current system (6 x 450G 15K disks), we are able to sustain 100
million records/day but probably not more.

On Wed, Apr 28, 2010 at 6:12 PM, Greg Smith <greg@2ndquadrant.com> wrote:
> Anj Adu wrote:
>>
>> I am faced with a  hardware choice for a postgres data warehouse
>> (extremely high volume inserts..over 200 million records a day)
>
> That's an average of 2314 per second, which certainly isn't easy to pull
> off.  You suggested you're already running this app.  Do you have any idea
> how high the volume of data going to the WAL is relative to everything else?
>
>> 12 x 600G disks (15K)  (the new Dell Poweredge C server)
>> or
>> 24 x 600G  (10K disks)
>>
>
> You can expect the 15K disks to be 30% (more sequential work) to 50%
> (random/commit work) faster than a similar 10K drive.  So from most
> perspectives, twice as many 10K drives should be considerably faster.  The
> main point of concern here is the commit rate, which you can't necessarily
> improve just by throwing drives at it.  That's based on how fast the drives
> spin, so there's the potential to discover a 50% regression there compared
> to the setup you have right now.  With the RAID card in there, it should be
> fine, but it's something to be concerned about.
>
> Also, you didn't mentioned the RAID card for the new system, and whether it
> would be the same in both setups or not.  That can be as important as the
> drives when you have larger arrays.  The LSI Megaraid card Dell is using for
> the Perc6i is quite fast, and you'll need to make sure you get something
> just as good for the new server.
>
> --
> Greg Smith  2ndQuadrant US  Baltimore, MD
> PostgreSQL Training, Services and Support
> greg@2ndQuadrant.com   www.2ndQuadrant.us
>
>

On Wed, Apr 28, 2010 at 10:45 PM, Anj Adu <fotographs@gmail.com> wrote:
> Thanks Greg
>
> We do not archive the WALs. We use application-level replication to
> achieve redundancy. WAL archiving was difficult to support with the
> earlier hardware we had ( 6x300G 10K disks Dell 2850) given the
> volumes we were dealing with. The RAID card should be from the same
> manufacturer (LSI in Dell's case).

I don't think Greg was talking of WAL archiving, but more of where are
the WAL files and how much IO throughput the WAL needs compared to the
regular old /data/base store.

Anj Adu wrote:
> We do not archive the WALs. We use application-level replication to
> achieve redundancy. WAL archiving was difficult to support with the
> earlier hardware we had ( 6x300G 10K disks Dell 2850) given the
> volumes we were dealing with. The RAID card should be from the same
> manufacturer (LSI in Dell's case).
>

The database is generating WAL files that are written to the pg_xlog
directory.  Sometimes this is broken out into a separate drive so that
it's possible to measure how much I/O is being written to there, as
opposed to the main database drive.  That's the WAL writing I was asking
about, not the overhead of archiving WAL files to elsewhere.  The way
that WAL writes go to disk, you can't always speed them up just by
throwing more disks at them--sometimes, you just need the individual
disk involved to be as fast as possible.

You should try to get the same Dell RAID controller you're already
using, that you know delivers good performance running your app.  All
I've heard about the models released after the Perc 6i has been bad
news.  Dell varies how much they tinker with the LSI firmware in their
own version of each card, and they didn't do very much of that in the
Perc 6 series.  They seem to be changing around the newer models more
again, which is always bad news.

--
Greg Smith  2ndQuadrant US  Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com   www.2ndQuadrant.us

All the disks are usually laid out in a single RAID 10 stripe . There
are no dedicated disks for the OS/WAL as storage is a premium

On Wed, Apr 28, 2010 at 10:30 PM, Greg Smith <greg@2ndquadrant.com> wrote:
> Anj Adu wrote:
>>
>> We do not archive the WALs. We use application-level replication to
>> achieve redundancy. WAL archiving was difficult to support with the
>> earlier hardware we had ( 6x300G 10K disks Dell 2850) given the
>> volumes we were dealing with. The RAID card should be from the same
>> manufacturer (LSI in Dell's case).
>>
>
> The database is generating WAL files that are written to the pg_xlog
> directory.  Sometimes this is broken out into a separate drive so that it's
> possible to measure how much I/O is being written to there, as opposed to
> the main database drive.  That's the WAL writing I was asking about, not the
> overhead of archiving WAL files to elsewhere.  The way that WAL writes go to
> disk, you can't always speed them up just by throwing more disks at
> them--sometimes, you just need the individual disk involved to be as fast as
> possible.
>
> You should try to get the same Dell RAID controller you're already using,
> that you know delivers good performance running your app.  All I've heard
> about the models released after the Perc 6i has been bad news.  Dell varies
> how much they tinker with the LSI firmware in their own version of each
> card, and they didn't do very much of that in the Perc 6 series.  They seem
> to be changing around the newer models more again, which is always bad news.
>
> --
> Greg Smith  2ndQuadrant US  Baltimore, MD
> PostgreSQL Training, Services and Support
> greg@2ndQuadrant.com   www.2ndQuadrant.us
>
>

I also want to add that with the perc 6i controllers..we have never
had issues. We have been running postgres nonstop for over 2 years and
sustaining a throughput of over 60-100 million messages a day without
breaking sweat. (postgres 8.1.9 on linux 32 bit )
I have to say I am impressed with the stability/performance of
postgres as well as the commodity hardware we use. We do have rare
disk failures..but the hot swappable drives make is easy to pop one in
and the RAID layer does the rebuilding very well.

On Thu, Apr 29, 2010 at 10:26 AM, Anj Adu <fotographs@gmail.com> wrote:
> All the disks are usually laid out in a single RAID 10 stripe . There
> are no dedicated disks for the OS/WAL as storage is a premium
>
> On Wed, Apr 28, 2010 at 10:30 PM, Greg Smith <greg@2ndquadrant.com> wrote:
>> Anj Adu wrote:
>>>
>>> We do not archive the WALs. We use application-level replication to
>>> achieve redundancy. WAL archiving was difficult to support with the
>>> earlier hardware we had ( 6x300G 10K disks Dell 2850) given the
>>> volumes we were dealing with. The RAID card should be from the same
>>> manufacturer (LSI in Dell's case).
>>>
>>
>> The database is generating WAL files that are written to the pg_xlog
>> directory.  Sometimes this is broken out into a separate drive so that it's
>> possible to measure how much I/O is being written to there, as opposed to
>> the main database drive.  That's the WAL writing I was asking about, not the
>> overhead of archiving WAL files to elsewhere.  The way that WAL writes go to
>> disk, you can't always speed them up just by throwing more disks at
>> them--sometimes, you just need the individual disk involved to be as fast as
>> possible.
>>
>> You should try to get the same Dell RAID controller you're already using,
>> that you know delivers good performance running your app.  All I've heard
>> about the models released after the Perc 6i has been bad news.  Dell varies
>> how much they tinker with the LSI firmware in their own version of each
>> card, and they didn't do very much of that in the Perc 6 series.  They seem
>> to be changing around the newer models more again, which is always bad news.
>>
>> --
>> Greg Smith  2ndQuadrant US  Baltimore, MD
>> PostgreSQL Training, Services and Support
>> greg@2ndQuadrant.com   www.2ndQuadrant.us
>>
>>
>

On Thu, Apr 29, 2010 at 11:26 AM, Anj Adu <fotographs@gmail.com> wrote:
> All the disks are usually laid out in a single RAID 10 stripe . There
> are no dedicated disks for the OS/WAL as storage is a premium

You should at least investigate the performance difference of having a
separate volume for WAL files on your system.    Since WAL files are
mostly sequential, and db access is generally random, the WAL files
can run really quickly on a volume that does nothing else but handle
WAL writes sequentially.  Given the volume you're handling, I would
expect that storage is not any more important than performance.

The fact that you're asking whether to go with 12 or 24 600G disks
shows that you're willing to give up a little storage for performance.
 I would bet that the 24 10k disks with one pair dedicated for OS /
pg_xlog would be noticeably faster than any single large volume config
you'd care to test, especially for lots of active connections.

With the increase in the number of disks that we can afford to have in
1 box..we will definitely plan on having WAL on dedicated disks.
Previously..we were stuck with the chassis limitation of 6 disks per
box.

On Thu, Apr 29, 2010 at 12:25 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote:
> On Thu, Apr 29, 2010 at 11:26 AM, Anj Adu <fotographs@gmail.com> wrote:
>> All the disks are usually laid out in a single RAID 10 stripe . There
>> are no dedicated disks for the OS/WAL as storage is a premium
>
> You should at least investigate the performance difference of having a
> separate volume for WAL files on your system.    Since WAL files are
> mostly sequential, and db access is generally random, the WAL files
> can run really quickly on a volume that does nothing else but handle
> WAL writes sequentially.  Given the volume you're handling, I would
> expect that storage is not any more important than performance.
>
> The fact that you're asking whether to go with 12 or 24 600G disks
> shows that you're willing to give up a little storage for performance.
>  I would bet that the 24 10k disks with one pair dedicated for OS /
> pg_xlog would be noticeably faster than any single large volume config
> you'd care to test, especially for lots of active connections.
>

On Thu, Apr 29, 2010 at 1:35 PM, Anj Adu <fotographs@gmail.com> wrote:
> With the increase in the number of disks that we can afford to have in
> 1 box..we will definitely plan on having WAL on dedicated disks.
> Previously..we were stuck with the chassis limitation of 6 disks per
> box.

Yeah, the smallest machine I use to build a db server now is one that
can hold 8 drives, 4 for a slony clone maybe.  If you go with the
24x2.5 chassis please report back on how it works.  I'm looking at the
16 or 24 x 2.5 cases for increasing density and keeping down costs on
rack space.

During the testing that I did when moving from pg7 to pg8 a few years back, I didn't notice any particular performance
increase on a similarly-configured server.

That is, we've got 14 disks (15k rpm) striped in a single RAID10 array. Moving the logs to an internal RAID
versus leaving them on the "main" storage array didn't impact my performance noticeably either way.

Now, please note:
SUCH A STATEMENT DEPENDS ENTIRELY ON YOUR USE-CASE SCENARIO.

That is, because these tests which I performed showed that for me, if you're using PG in a much different
manner, you may have different results. It's also quite possible that our 50% expansion in the past 3
years has had some effect, but I'm not in a place to retest that at this time.

We specifically chose to put our logs on the fiber SAN in case the underlying machine went down.
Disaster recovery for that box would therefore be:
a) New machine with O/S and pg installed.
b) Mount SAN
c) Start PG. Everything (including logs) is available to you.

It is, in essence, our "externally-stored PG data" in its entirety.

On the 10k vs 15k rpm disks, there's a _lot_ to be said about that. I don't want to start a flame war here,
but 15k versus 10k rpm hard drives does NOT equivocate to a 50% increase in read/write times, to say
the VERY least.

"Average seek time" is the time it takes for the head to move from random place A to
random place B on the drive. The rotational latency of a drive can be easily calculated.
A 15k drive rotates roughly 250 times per second, or 4 msec per rotation versus a 10k
drive which is about 167 rotations per sec or 6 sec per rotation.

This would mean that the rotational latency of a 15k drive adds 2msec and a 10k drive adds 3msec.

So, your true seek time is the "average seek time" of the drive + the rotation listed above.

So, if your average latency is something REALLY good (say 4msec) for each of the drives, your 15k
drive would have 6msec real-world IOPS of around 166, and your 10k drive would have 143. In that
particular case, at a very low level, you'd be getting about a 14% improvement.

HOWEVER, we're not talking about a single drive, here. We're talking about a RAID10 of 12
drives (6 + 6 mirror, I assume) versus 24 drives (12+12 mirror I assume). In that case,
the max IOPS of your first RAID would be around 1000 while the max IOPS of your second RAID
with the "slower" drives would be around 1700.


Hope this helps.

I _really_ don't want to start a war with this. If you're confused how I got these
numbers, please contact me directly.

Cheers!



----- "Scott Marlowe" <scott.marlowe@gmail.com> wrote:
> On Thu, Apr 29, 2010 at 11:26 AM, Anj Adu <fotographs@gmail.com> wrote:
> > All the disks are usually laid out in a single RAID 10 stripe . There
> > are no dedicated disks for the OS/WAL as storage is a premium
>
> You should at least investigate the performance difference of having a
> separate volume for WAL files on your system. Since WAL files are
> mostly sequential, and db access is generally random, the WAL files
> can run really quickly on a volume that does nothing else but handle
> WAL writes sequentially. Given the volume you're handling, I would
> expect that storage is not any more important than performance.
>
> The fact that you're asking whether to go with 12 or 24 600G disks
> shows that you're willing to give up a little storage for performance.
> I would bet that the 24 10k disks with one pair dedicated for OS /
> pg_xlog would be noticeably faster than any single large volume config
> you'd care to test, especially for lots of active connections.
>
> --
> Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-admin
>

On Thu, Apr 29, 2010 at 1:51 PM, Scott Whitney <scott@journyx.com> wrote:
> During the testing that I did when moving from pg7 to pg8 a few years back, I didn't notice any particular
performance
> increase on a similarly-configured server.
>
> That is, we've got 14 disks (15k rpm) striped in a single RAID10 array. Moving the logs to an internal RAID
> versus leaving them on the "main" storage array didn't impact my performance noticeably either way.
>
> Now, please note:
> SUCH A STATEMENT DEPENDS ENTIRELY ON YOUR USE-CASE SCENARIO.
>
> That is, because these tests which I performed showed that for me, if you're using PG in a much different
> manner, you may have different results. It's also quite possible that our 50% expansion in the past 3
> years has had some effect, but I'm not in a place to retest that at this time.
>
> We specifically chose to put our logs on the fiber SAN in case the underlying machine went down.
> Disaster recovery for that box would therefore be:
> a) New machine with O/S and pg installed.
> b) Mount SAN
> c) Start PG. Everything (including logs) is available to you.
>
> It is, in essence, our "externally-stored PG data" in its entirety.
>
> On the 10k vs 15k rpm disks, there's a _lot_ to be said about that. I don't want to start a flame war here,
> but 15k versus 10k rpm hard drives does NOT equivocate to a 50% increase in read/write times, to say
> the VERY least.
>
> "Average seek time" is the time it takes for the head to move from random place A to
> random place B on the drive. The rotational latency of a drive can be easily calculated.
> A 15k drive rotates roughly 250 times per second, or 4 msec per rotation versus a 10k
> drive which is about 167 rotations per sec or 6 sec per rotation.
>
> This would mean that the rotational latency of a 15k drive adds 2msec and a 10k drive adds 3msec.
>
> So, your true seek time is the "average seek time" of the drive + the rotation listed above.
>
> So, if your average latency is something REALLY good (say 4msec) for each of the drives, your 15k
> drive would have 6msec real-world IOPS of around 166, and your 10k drive would have 143. In that
> particular case, at a very low level, you'd be getting about a 14% improvement.
>
> HOWEVER, we're not talking about a single drive, here. We're talking about a RAID10 of 12
> drives (6 + 6 mirror, I assume) versus 24 drives (12+12 mirror I assume). In that case,
> the max IOPS of your first RAID would be around 1000 while the max IOPS of your second RAID
> with the "slower" drives would be around 1700.
>
>
> Hope this helps.
>
> I _really_ don't want to start a war with this. If you're confused how I got these
> numbers, please contact me directly.

No argument from me.  My point was that the RAID-10 is for random
access at x op/s while the WAL is for sequential IO at x mb/s.  If you
put the pg_xlog on the same drives as the data set, then you basically
halve throughput by writing everything randomly twice.  Better RAID
controllers with lots of cache can re-order all that so that you still
get good throughput.  If you've got a RAID controller with less
memory, then it can really help to have a separate pg_xlog.  My
machines about 30% or so faster than with it all on one set with an
areca 1680.

Scott Whitney wrote:
> On the 10k vs 15k rpm disks, there's a _lot_ to be said about that. I don't want to start a flame war here,
> but 15k versus 10k rpm hard drives does NOT equivocate to a 50% increase in read/write times, to say
> the VERY least.
>

Your characterization is correct were there only the drives involved
here, so no flames on your raw data.

Once you've introduced a battery-backed write cache into the mix, this
whole area becomes impossible to compute that way though, and it was
that context I was commenting from at least.  Those are good at turning
random I/O into something more like sequential, as well as reducing the
number of times you pay for rotational latency in several common
database operations.  The effective impact is to significantly narrow
the difference between drives where the seek and rotation time are the
main differences in a database context--even though the worst-case IOPS
doesn't really change.  IOPS is an interesting number to compute, but
real-world database performance isn't linearly correlated with it.
Maybe if your workload consists mainly of random, uncached index scans
system performance will scale just like IOPS, but that's pretty uncommon.

[Rant about making sure not to drink the storage vendor IOPS Kool-Aid
deleted]

--
Greg Smith  2ndQuadrant US  Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com   www.2ndQuadrant.us