Re: sustained update load of 1-2k/sec - Mailing list pgsql-performance

At 03:31 PM 8/19/2005, Alex Turner wrote:
>Don't forget that Ultra 320 is the speed of the bus, not each drive.
>No matter how many honking 15k disks you put on a 320MB bus, you can
>only get 320MB/sec! and have so many outstanding IO/s on the bus.

Of course.  This is exactly why multi-channel SCSI and multichannel
Fibre Channel cards exist; and why external RAID enclosures usually
have multiple such cards in them...

Even moderately acceptable U320 SCSI cards are dual channel at this
point (think Adaptec dual channel AHAxxxx's), and Quad channel ones
are just as common.  The Quads will, of course, saturate a 64b 133MHz
PCI-X bus.  _IF_ the chipset on them can keep up.

The current kings of RAID card performance are all Fibre Channel
based, and all the ones I know of are theoretically capable of
saturating a 64b 133MHz PCI-X bus.  Again, _IF_ the chipset on them
can keep up.

Most commodity RAID card have neither adequate CPU nor enough
buffer.  Regardless of the peripheral IO technology they use.


>Not so with SATA! Each drive is on it's own bus, and you are only
>limited by the speed of your PCI-X Bus, which can be as high as
>800MB/sec at 133Mhz/64bit.

That's the Theory anyway, and latency should be lower as well.  OTOH,
as my wife likes to say "In theory, Theory and Practice are the
same.  In practice, they almost never are."

You are only getting the performance you mention as long as your card
can keep up with multiplexing N IO streams, crunching RAID 5 XORs
(assuming you are using RAID 5), etc, etc.  As I'm sure you know,
"The chain is only as strong as its weakest link.".

Most commodity SATA RAID cards brag about being able to pump 300MB/s
(they were all over LW SF bragging about this!?), which in this
context is woefully unimpressive.  Sigh.

I'm impressed with the Areca cards because they usually have CPUs
that actually can come close to pushing the theoretical IO limit of
the bus they are plugged into; and they can be upgraded to (barely)
acceptable buffer amounts <rant>(come on, manufacturers! 4GB of DDR
PC3200 is only -2- DIMMs, and shortly that will be enough to hold 8GB
of DDR PC3200.  Give us more buffer!)</rant>.


>It's cheap and it's fast - all you have to do is pay for the
>enclosure, which can be a bit pricey, but there are some nice 24bay
>and even 40bay enclosures out there for SATA.

I've even seen 48 bay ones.  However, good enclosures, particularly
for larger numbers of HDs, are examples of non-trivial engineering
and priced accordingly.  Too many times I see people buy "bargain"
enclosures and set themselves and their organizations up for some
_very_ unpleasant times that could easily have been avoided by being
careful to buy quality products.  "Pay when you buy or pay much more later."


>Yes a 15k RPM drive will give you better seek time and better peak
>through put, but put them all on a single U320 bus and you won't see
>much return past a stripe size of 3 or 4

Agreed.  Same holds for 2Gbps FC.  Haven't tested 4Gbps FC personally
yet, but I'm told the limit is higher in the manner you'd expect.


>If it's raw transactions per second data warehouse style, it's all
>about the xlog baby which is sequential writes, and all about large
>block reads, which is sequential reads.
>
>Alex Turner
>NetEconomist
>P.S. Sorry if i'm a bit punchy, I've been up since yestarday with
>server upgrade nightmares that continue ;)

My condolences and sympathies.  I've definitely been there and done that.

Ron Peacetree