Thread: raid10 write performance
Hi folks, is there a general problem with raid10 performance postgresql on it? We see very low performance on writes (2-3x slower than on less performant servers). I wonder if it is solely problem of raid10 configuration, or if it is postgresql's thing. Would moving WAL dir to separate disk help potentially ? We're running centos 5.4, and server config is: x3550 M2, xeon 4c e5530 2.4ghz , 6GB of ram disks: ibm 300gb 2.5 SAS raid: serveRAID M5014 SAS/SATA controller strip size is the default 128k thanks. -- GJ
On 6/22/2010 4:31 AM, Grzegorz Jaśkiewicz wrote: > Hi folks, > > is there a general problem with raid10 performance postgresql on it? > We see very low performance on writes (2-3x slower than on less > performant servers). I wonder if it is solely problem of raid10 > configuration, or if it is postgresql's thing. > RAID 10 is the commonly suggested layout for DB's as its performance to redundancy is good. The question that begs to be ask is what is the IO layout on the other servers your comparing against. > Would moving WAL dir to separate disk help potentially ? > Yes it can have a big impact. http://www.westnet.com/~gsmith/content/postgresql/TuningPGWAL.htm http://wiki.postgresql.org/wiki/Performance_Analysis_Tools http://wiki.postgresql.org/wiki/Performance_Optimization > We're running centos 5.4, and server config is: > > x3550 M2, xeon 4c e5530 2.4ghz , 6GB of ram > disks: ibm 300gb 2.5 SAS > > raid: serveRAID M5014 SAS/SATA controller > > strip size is the default 128k > > > thanks. > > All legitimate Magwerks Corporation quotations are sent in a .PDF file attachment with a unique ID number generated by ourproprietary quotation system. Quotations received via any other form of communication will not be honored. CONFIDENTIALITY NOTICE: This e-mail, including attachments, may contain legally privileged, confidential or other informationproprietary to Magwerks Corporation and is intended solely for the use of the individual to whom it addresses.If the reader of this e-mail is not the intended recipient or authorized agent, the reader is hereby notified thatany unauthorized viewing, dissemination, distribution or copying of this e-mail is strictly prohibited. If you have receivedthis e-mail in error, please notify the sender by replying to this message and destroy all occurrences of this e-mailimmediately. Thank you.
Attachment
Grzegorz Jaśkiewicz wrote: > raid: serveRAID M5014 SAS/SATA controller > Do the "performant servers" have a different RAID card? This one has terrible performance, and could alone be the source of your issue. The ServeRAID cards are slow in general, and certainly slow running RAID10. -- Greg Smith 2ndQuadrant US Baltimore, MD PostgreSQL Training, Services and Support greg@2ndQuadrant.com www.2ndQuadrant.us
Justin Graf wrote:
Strongly recommended.
Be aware that you must pay CLOSE ATTENTION to your backup strategy if WAL is on a different physical disk. Snapshotting the data disk where WAL is on a separate spindle and backing it up **WILL NOT WORK** and **WILL** result in an non-restoreable backup.
The manual discusses this but it's easy to miss.... don't or you'll get a NASTY surprise if something goes wrong.....
-- Karl
WAL on a separate spindle will make a HUGE difference in performance. TPS rates frequently double OR BETTER with WAL on a dedicated spindle.On 6/22/2010 4:31 AM, Grzegorz Jaśkiewicz wrote:Would moving WAL dir to separate disk help potentially ?Yes it can have a big impact.
Strongly recommended.
Be aware that you must pay CLOSE ATTENTION to your backup strategy if WAL is on a different physical disk. Snapshotting the data disk where WAL is on a separate spindle and backing it up **WILL NOT WORK** and **WILL** result in an non-restoreable backup.
The manual discusses this but it's easy to miss.... don't or you'll get a NASTY surprise if something goes wrong.....
-- Karl
Attachment
Of course, no backup strategy is complete without testing a full restore onto bare hardware :-)
On Tue, Jun 22, 2010 at 9:29 AM, Karl Denninger <karl@denninger.net> wrote:
Justin Graf wrote:WAL on a separate spindle will make a HUGE difference in performance. TPS rates frequently double OR BETTER with WAL on a dedicated spindle.On 6/22/2010 4:31 AM, Grzegorz Jaśkiewicz wrote:Would moving WAL dir to separate disk help potentially ?Yes it can have a big impact.
Strongly recommended.
Be aware that you must pay CLOSE ATTENTION to your backup strategy if WAL is on a different physical disk. Snapshotting the data disk where WAL is on a separate spindle and backing it up **WILL NOT WORK** and **WILL** result in an non-restoreable backup.
The manual discusses this but it's easy to miss.... don't or you'll get a NASTY surprise if something goes wrong.....
-- Karl
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
On Jun 22, 2010, at 7:29 AM, Karl Denninger wrote: > Justin Graf wrote: >> >> On 6/22/2010 4:31 AM, Grzegorz Jaśkiewicz wrote: >> >>> Would moving WAL dir to separate disk help potentially ? >>> >>> >> >> Yes it can have a big impact. > WAL on a separate spindle will make a HUGE difference in performance. TPS rates frequently double OR BETTER with WAL ona dedicated spindle. > > Strongly recommended. > Most of the performance increase on Linux, if your RAID card has a data-safe write-back cache (battery or solid state cachepersistence in case of power failure), is a separate file system, not separate spindle. Especially if ext3 is usedwith the default "ordered" journal, it is absolutely caustic to performance to have WAL and data on the same file system. The whole 'separate spindle' thing is important if you don't have a good caching raid card, otherwise it doesn't matter thatmuch. > Be aware that you must pay CLOSE ATTENTION to your backup strategy if WAL is on a different physical disk. Snapshottingthe data disk where WAL is on a separate spindle and backing it up **WILL NOT WORK** and **WILL** result in annon-restoreable backup. > > The manual discusses this but it's easy to miss.... don't or you'll get a NASTY surprise if something goes wrong..... > > -- Karl > <karl.vcf><ATT00001..txt>
On 06/22/10 16:40, Greg Smith wrote: > Grzegorz Jaśkiewicz wrote: >> raid: serveRAID M5014 SAS/SATA controller >> > > Do the "performant servers" have a different RAID card? This one has > terrible performance, and could alone be the source of your issue. The > ServeRAID cards are slow in general, and certainly slow running RAID10. What are some good RAID10 cards nowadays? On the other hand, RAID10 is simple enough that soft-RAID implementations should be more than adequate - any ideas why a dedicated card has it "slow"?
* Ivan Voras: > On the other hand, RAID10 is simple enough that soft-RAID > implementations should be more than adequate - any ideas why a dedicated > card has it "slow"? Barrier support on RAID10 seems to require some smallish amount of non-volatile storage which supports a high number of write operations per second, so a software-only solution might not be available. -- Florian Weimer <fweimer@bfk.de> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99
On 06/23/10 14:00, Florian Weimer wrote: > * Ivan Voras: > >> On the other hand, RAID10 is simple enough that soft-RAID >> implementations should be more than adequate - any ideas why a dedicated >> card has it "slow"? > > Barrier support on RAID10 seems to require some smallish amount of > non-volatile storage which supports a high number of write operations > per second, so a software-only solution might not be available. If I understand you correctly, this can be said in general for all spinning-disk usage and is not specific to RAID10. (And in the case of high, constant TPS, no amount of NVRAM will help you).
On Wed, 23 Jun 2010, Ivan Voras wrote: > On 06/23/10 14:00, Florian Weimer wrote: >> Barrier support on RAID10 seems to require some smallish amount of >> non-volatile storage which supports a high number of write operations >> per second, so a software-only solution might not be available. > > If I understand you correctly, this can be said in general for all > spinning-disk usage and is not specific to RAID10. (And in the case of > high, constant TPS, no amount of NVRAM will help you). No. Write barriers work fine with a single disc, assuming it is set up correctly. The barrier is a command telling the disc to make sure that one piece of data is safe before starting to write another piece of data. However, as soon as you have multiple discs, the individual discs do not have a way of communicating with each other to make sure that the first piece of data is written before the other. That's why you need a little bit of non-volatile storage to mediate that to properly support barriers. Of course, from a performance point of view, yes, you need some NVRAM on any kind of spinning storage to maintain high commit rates. Matthew -- I wouldn't be so paranoid if you weren't all out to get me!!
On Wed, Jun 23, 2010 at 8:25 AM, Ivan Voras <ivoras@freebsd.org> wrote: > On 06/23/10 14:00, Florian Weimer wrote: >> * Ivan Voras: >> >>> On the other hand, RAID10 is simple enough that soft-RAID >>> implementations should be more than adequate - any ideas why a dedicated >>> card has it "slow"? >> >> Barrier support on RAID10 seems to require some smallish amount of >> non-volatile storage which supports a high number of write operations >> per second, so a software-only solution might not be available. > > If I understand you correctly, this can be said in general for all > spinning-disk usage and is not specific to RAID10. (And in the case of > high, constant TPS, no amount of NVRAM will help you). Not entirely true. Let's say you have enough battery backed cache to hold 10,000 transaction writes in memory at once. The RAID controller can now re-order those writes so that they go from one side of the disk to the other, instead of randomly all over the place. That will most certainly help improve your throughput.
On Wed, Jun 23, 2010 at 6:06 AM, Ivan Voras <ivoras@freebsd.org> wrote: > On 06/22/10 16:40, Greg Smith wrote: >> Grzegorz Jaśkiewicz wrote: >>> raid: serveRAID M5014 SAS/SATA controller >>> >> >> Do the "performant servers" have a different RAID card? This one has >> terrible performance, and could alone be the source of your issue. The >> ServeRAID cards are slow in general, and certainly slow running RAID10. > > What are some good RAID10 cards nowadays? LSI, Areca, 3Ware (now LSI I believe) > On the other hand, RAID10 is simple enough that soft-RAID > implementations should be more than adequate - any ideas why a dedicated > card has it "slow"? This is mostly a problem with some older cards that focused on RAID-5 performance, and RAID-10 was an afterthought. On many of these cards (older PERCs for instance) it was faster to either use a bunch of RAID-1 pairs in hardware with RAID-0 in software on top, or put the thing into JBOD mode and do it all in software.