Thread: Tyan Thunder MB for postgres server
Hi All, I've read a fair bit in these lists about server configurations, and have come to some conclusions as to the kind of system that I want to build. However, I would like to hear from anyone with specific experience of this config. before I go ahead and buy it: The system is based on a Tyan Thunder K8S motherboard, dual opterons and 4GB RAM, and the LSI dual channel raid option. OS will be RHEL AS 3 for AMD64 The Tyan/opteron options seems tobe quite popular as a postgres server, but I'm not sure how many people are doing it all 64 bit, so my specific areas of concern at the moment are: 1. 64bit linux driver support - how is it? 2. Is there a postgres 7.4.6 package available for this (ie AMD64) or will I have to compile it myself? Personally, I don't mind compiling it myself - have done it many times, but since I have to write instructions for the system to be completely re-built in the case of a failure, I'd rather go with widely available packages as much as possible. I hope to hear from you, regards Iain
We use that exact configuration right now, except with an Adaptec card and more RAM. We used RHEL 3.0, then switched to Fedora core 2 64Bit as a test, since this server was since placed into standby duties. Well, we needed to use the server when the main server went into maintenance recently, and it worked fantastically well with Fedora 64Bit. You want to get the RPM source package for Postgresql and build the RPM's instead of installing from source. This method builds the binary RPM's for your platform which you can then install normally afterwards. Warmest regards, Ericson Smith Tracking Specialist/DBA +-----------------------+------------------------------+ | http://www.did-it.com | Need help tracking your paid | | eric@did-it.com | search campaigns? | | 516-255-0500 | - Help is on the way! | +-----------------------+------------------------------+ Iain wrote: > Hi All, > > I've read a fair bit in these lists about server configurations, and > have come to some conclusions as to the kind of system that I want to > build. However, I would like to hear from anyone with specific > experience of this config. before I go ahead and buy it: > > The system is based on a Tyan Thunder K8S motherboard, dual opterons > and 4GB RAM, and the LSI dual channel raid option. OS will be RHEL AS > 3 for AMD64 > > The Tyan/opteron options seems tobe quite popular as a postgres > server, but I'm not sure how many people are doing it all 64 bit, so > my specific areas of concern at the moment are: > > 1. 64bit linux driver support - how is it? > > 2. Is there a postgres 7.4.6 package available for this (ie AMD64) or > will I have to compile it myself? > > Personally, I don't mind compiling it myself - have done it many > times, but since I have to write instructions for the system to be > completely re-built in the case of a failure, I'd rather go with > widely available packages as much as possible. > > I hope to hear from you, > regards > Iain > > > > > > > ---------------------------(end of broadcast)--------------------------- > TIP 9: the planner will ignore your desire to choose an index scan if > your > joining column's datatypes do not match >
Attachment
Hi Ericson, I'm planning on using the onboard LSI SCSI controller, and have read alot about poor IO performance on Dells using LSI. Then again, most of the talk about Adaptec hasn't been all that complimentary either as I saw it ;-) It seems that there is more to it than the name of the chip vendor. I havn't heard anything bad about the Tyan opteron based boards in any configuration yet though. Thanks for your feedback, regards Iain ----- Original Message ----- From: "Ericson Smith" <eric@did-it.com> To: <pgsql-admin@postgresql.org> Sent: Tuesday, December 14, 2004 12:11 AM Subject: Re: [ADMIN] Tyan Thunder MB for postgres server > We use that exact configuration right now, except with an Adaptec card > and more RAM. > > We used RHEL 3.0, then switched to Fedora core 2 64Bit as a test, since > this server was since placed into standby duties. Well, we needed to use > the server when the main server went into maintenance recently, and it > worked fantastically well with Fedora 64Bit. > > You want to get the RPM source package for Postgresql and build the > RPM's instead of installing from source. This method builds the binary > RPM's for your platform which you can then install normally afterwards. > > Warmest regards, > Ericson Smith > Tracking Specialist/DBA > +-----------------------+------------------------------+ > | http://www.did-it.com | Need help tracking your paid | > | eric@did-it.com | search campaigns? | > | 516-255-0500 | - Help is on the way! | > +-----------------------+------------------------------+ > > > > Iain wrote: > >> Hi All, >> >> I've read a fair bit in these lists about server configurations, and >> have come to some conclusions as to the kind of system that I want to >> build. However, I would like to hear from anyone with specific >> experience of this config. before I go ahead and buy it: >> >> The system is based on a Tyan Thunder K8S motherboard, dual opterons >> and 4GB RAM, and the LSI dual channel raid option. OS will be RHEL AS >> 3 for AMD64 >> >> The Tyan/opteron options seems tobe quite popular as a postgres >> server, but I'm not sure how many people are doing it all 64 bit, so >> my specific areas of concern at the moment are: >> >> 1. 64bit linux driver support - how is it? >> >> 2. Is there a postgres 7.4.6 package available for this (ie AMD64) or >> will I have to compile it myself? >> >> Personally, I don't mind compiling it myself - have done it many >> times, but since I have to write instructions for the system to be >> completely re-built in the case of a failure, I'd rather go with >> widely available packages as much as possible. >> >> I hope to hear from you, >> regards >> Iain >> >> >> >> >> >> >> ---------------------------(end of broadcast)--------------------------- >> TIP 9: the planner will ignore your desire to choose an index scan if >> your >> joining column's datatypes do not match >> > -------------------------------------------------------------------------------- > > ---------------------------(end of broadcast)--------------------------- > TIP 3: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to majordomo@postgresql.org so that your > message can get through to the mailing list cleanly >
I've used a slew of LSI controllers (22915A, integrated 53C1010, 22320-R, MegaRAID 320-1) and they've all performed w/o issues. Now I have had some hardware die though. One was probably our fault -- during an attempted upgrade, we probably weren't careful enough with the 22320-R (cramped #@#@$! rackmounts) and after we reinstalled it, it just never worked right again. Otherwise, if you leave it alone, they work pretty good. I've got a 4 15K disks (software RAID10) attached to a 22915A in a 2x Opteron 244 server running FC2 64-bit. This machine is deadly fast; every so often, iostat will show peak transfer rates of 160MB/s. (RAID cage+rackmount problems stopped us from upgrading to hardware raid+battery -- we'll tackle this again when we get a 4U case.) Iain wrote: > Hi Ericson, > > I'm planning on using the onboard LSI SCSI controller, and have read > alot about poor IO performance on Dells using LSI. Then again, most of > the talk about Adaptec hasn't been all that complimentary either as I > saw it ;-) It seems that there is more to it than the name of the chip > vendor. I havn't heard anything bad about the Tyan opteron based boards > in any configuration yet though.
I've used LSI/MegaRAID controllers with the 2600 series servers with no problems whatsoever in terms of performance. With battery backed cache the machine never went above 0.05 load factor in production, and could handle pgbench with 300 to 600 tps under various settings. Loading a 10 gig data set with fks and indexes took about 5 or 6 minutes. On Mon, 2004-12-13 at 19:32, Iain wrote: > Hi Ericson, > > I'm planning on using the onboard LSI SCSI controller, and have read alot > about poor IO performance on Dells using LSI. Then again, most of the talk > about Adaptec hasn't been all that complimentary either as I saw it ;-) It > seems that there is more to it than the name of the chip vendor. I havn't > heard anything bad about the Tyan opteron based boards in any configuration > yet though. > > Thanks for your feedback, > regards > Iain > > > ----- Original Message ----- > From: "Ericson Smith" <eric@did-it.com> > To: <pgsql-admin@postgresql.org> > Sent: Tuesday, December 14, 2004 12:11 AM > Subject: Re: [ADMIN] Tyan Thunder MB for postgres server > > > > We use that exact configuration right now, except with an Adaptec card > > and more RAM. > > > > We used RHEL 3.0, then switched to Fedora core 2 64Bit as a test, since > > this server was since placed into standby duties. Well, we needed to use > > the server when the main server went into maintenance recently, and it > > worked fantastically well with Fedora 64Bit. > > > > You want to get the RPM source package for Postgresql and build the > > RPM's instead of installing from source. This method builds the binary > > RPM's for your platform which you can then install normally afterwards. > > > > Warmest regards, > > Ericson Smith > > Tracking Specialist/DBA > > +-----------------------+------------------------------+ > > | http://www.did-it.com | Need help tracking your paid | > > | eric@did-it.com | search campaigns? | > > | 516-255-0500 | - Help is on the way! | > > +-----------------------+------------------------------+ > > > > > > > > Iain wrote: > > > >> Hi All, > >> > >> I've read a fair bit in these lists about server configurations, and > >> have come to some conclusions as to the kind of system that I want to > >> build. However, I would like to hear from anyone with specific > >> experience of this config. before I go ahead and buy it: > >> > >> The system is based on a Tyan Thunder K8S motherboard, dual opterons > >> and 4GB RAM, and the LSI dual channel raid option. OS will be RHEL AS > >> 3 for AMD64 > >> > >> The Tyan/opteron options seems tobe quite popular as a postgres > >> server, but I'm not sure how many people are doing it all 64 bit, so > >> my specific areas of concern at the moment are: > >> > >> 1. 64bit linux driver support - how is it? > >> > >> 2. Is there a postgres 7.4.6 package available for this (ie AMD64) or > >> will I have to compile it myself? > >> > >> Personally, I don't mind compiling it myself - have done it many > >> times, but since I have to write instructions for the system to be > >> completely re-built in the case of a failure, I'd rather go with > >> widely available packages as much as possible. > >> > >> I hope to hear from you, > >> regards > >> Iain > >> > >> > >> > >> > >> > >> > >> ---------------------------(end of broadcast)--------------------------- > >> TIP 9: the planner will ignore your desire to choose an index scan if > >> your > >> joining column's datatypes do not match > >> > > > > > -------------------------------------------------------------------------------- > > > > > > ---------------------------(end of broadcast)--------------------------- > > TIP 3: if posting/reading through Usenet, please send an appropriate > > subscribe-nomail command to majordomo@postgresql.org so that your > > message can get through to the mailing list cleanly > > > > > ---------------------------(end of broadcast)--------------------------- > TIP 2: you can get off all lists at once with the unregister command > (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
Thanks to all for your feedback on this. I'm looking forward to getting my hands on the system. It seems that the battery backed cache is an important factor, though I havn't found any information specifically concerning this on the tyan site. I can't tell if it's an option or standard with the integrated SCSI controller. I don't have much experience with building this kind of server (i'm more on the development side) but I know basically what I want and that is RAID 0+1 and battery backed cache. The target OS is RHEL ES 3 (AMD64 platform) and I guess that it doesn't matter if the RAID is hardware or software (or a combination of both?) but if you think I've missed anything, please let me know. Regards Iain
Iain wrote: > Thanks to all for your feedback on this. > > I'm looking forward to getting my hands on the system. > > It seems that the battery backed cache is an important factor, though I > havn't found any information specifically concerning this on the tyan > site. I can't tell if it's an option or standard with the integrated > SCSI controller. RAID may be an option with your motherboard. LSI Logic has a MegaRAID 320-0 which interfaces with integrated MB SCSI. http://www.lsilogic.com/products/megaraid/scsi_320_0.html They don't list any Tyan's as "certified" so I dunno. Perhaps you should look into a Tyan w/o integrated SCSI and just get the MegaRAID 320-1 or 320-2 to avoid any possible issues. > I don't have much experience with building this kind of server (i'm more > on the development side) but I know basically what I want and that is > RAID 0+1 and battery backed cache. The target OS is RHEL ES 3 (AMD64 > platform) and I guess that it doesn't matter if the RAID is hardware or > software (or a combination of both?) but if you think I've missed > anything, please let me know. Hardware RAID + battery lets you turn write caching on. For a DB with lots of updates, it can make a big difference.
Hi William, > They don't list any Tyan's as "certified" so I dunno. Perhaps you should > look into a Tyan w/o integrated SCSI and just get the MegaRAID 320-1 or > 320-2 to avoid any possible issues. Upon further investigation, I think this the way to go, specificaly I'llbe recommending the 320-2 as I plan to put the WAL and DB on seperate channels and partitions (as opposed to to putting them on the same logical partition split over the two channels). > Hardware RAID + battery lets you turn write caching on. For a DB with > lots of updates, it can make a big difference. And we need all the help we can get, right? ;-) Thanks for pointing me in the right direction there. regards Iain
Iain wrote: > Upon further investigation, I think this the way to go, specificaly > I'llbe recommending the 320-2 as I plan to put the WAL and DB on > seperate channels and partitions (as opposed to to putting them on the > same logical partition split over the two channels). SOmething to think about. Let's suppose a channel/cable completely dies. How would you protect against it? Split a logical mirror device over 2 channels. Another trick I've started doing with my MegaRAID setups is mirroring in hardware but striping in software. To do 100% hardware RAID-10 MegaRAID cards, all drives must have the same geometry. Not a big deal when you're first configuring a server but a few years down the line when a drive dies and you need a replacement, it can drive you up the wall. I had a set of IBM 36GBs that needed a replacement -- couldn't find any old stock so I decided to get the newest IBM (Hitachi) 36GBs -- NOT THE SAME GEOMETRY!@!@!#!$## After learning that lesson, I did some testing with mirroring. LSI is much less stringent for mirroring. I was able to pair a Seagate + IBM and still get a valid mirror drive. I then use Linux software RAID to stripe 2 mirror volumes together.
Hi William, > SOmething to think about. Let's suppose a channel/cable completely dies. > How would you protect against it? Split a logical mirror device over 2 > channels. This effectively implements RAID 0+1, right? RAID 1 (mirroring) over RAID 0 striped volumes. I can certainly see your point regarding the redundancy of the controller channels, but my understanding is that (apart from that) RAID 0+1 is less robust that RAID 10 regarding disk failures. Presuming that the system will continue to operate even in the event of 1 channel failure, it's still not a clear choice. Does that seem like a reasonable assessment? > Another trick I've started doing with my MegaRAID setups is mirroring in > hardware but striping in software. Yeah, that is a good point. I havn't decided either way but I consider that a viable option. If you were building this system now, and want the option of buying the same disks in 3 years time, do you think it would be a bad idea to go for the ~40GB size? Maybe the next size up would be better, though we don't actually need the extra space. If you have any specific recommendations (for or against) specific drives/manufacturers, please let me know. Also, someone asked me what happens if one of the CPUs fails on this system, will the system continue to operate on 1 CPU. I havn't really considered this, and have never read anything either way, so my assumption is "no, it won't". Any comment? Thanks again Iain
Iain wrote: > Hi William, > >> SOmething to think about. Let's suppose a channel/cable completely >> dies. How would you protect against it? Split a logical mirror device >> over 2 channels. > > > This effectively implements RAID 0+1, right? RAID 1 (mirroring) over > RAID 0 striped volumes. I can certainly see your point regarding the > redundancy of the controller channels, but my understanding is that > (apart from that) RAID 0+1 is less robust that RAID 10 regarding disk > failures. Presuming that the system will continue to operate even in the > event of 1 channel failure, it's still not a clear choice. Does that > seem like a reasonable assessment? I'm not sure why people say one is better than the other. Both will survive the loss of 2 drives -- they're just different drives. RAID 0+1: A(1m1) s B(1m1) <-- any drive on A and any drive on B RAID 10: A(1s1) m B(1s1) <-- both drives on A or both drives on B Either way, I think you can do both across 2 channels to be redundant for a channel/cable failing. RAID 0+1: C1-m-C2 s C1-m-C2 RAID 10: C1-s-C1 m C2-s-C2 >> Another trick I've started doing with my MegaRAID setups is mirroring >> in hardware but striping in software. > > > Yeah, that is a good point. I havn't decided either way but I consider > that a viable option. You can probably do the same hardware + software trick for RAID10. Except after OS crashes, Linux+Solaris spends quite a bit of time resyncing mirrors and it looks like the MegaRAID controller is a bit smarter in knowing when to resync. > If you were building this system now, and want the option of buying the > same disks in 3 years time, do you think it would be a bad idea to go > for the ~40GB size? Maybe the next size up would be better, though we > don't actually need the extra space. Always get more space than you need if you can afford it because it's a pain in the ass adding more disk space. Especially on the boot drives. > Also, someone asked me what happens if one of the CPUs fails on this > system, will the system continue to operate on 1 CPU. I havn't really > considered this, and have never read anything either way, so my > assumption is "no, it won't". Any comment? Doing a google search for "hot swap cpu", I see they've added something into Linux but you'd still need hardware support.
On Thu, 16 Dec 2004 11:18:37 +0900, Iain <iain@mst.co.jp> wrote: > Also, someone asked me what happens if one of the CPUs fails on this system, > will the system continue to operate on 1 CPU. I havn't really considered > this, and have never read anything either way, so my assumption is "no, it > won't". Any comment? You are correct. That kind of functionality can't be found on standard x86 gear (sans the $150K NEC redundant behemoth). Regards, aaron.glenn
Thanks again for your valuable input William. > I'm not sure why people say one is better than the other. Both will > survive the loss of 2 drives -- they're just different drives. > RAID 0+1: A(1m1) s B(1m1) <-- any drive on A and any drive on B > RAID 10: A(1s1) m B(1s1) <-- both drives on A or both drives on B I'm wondering if we have a terminology problem here. As I understand it the configs are the reverse of what you said. I only checked www.bytepile.com: so I hope they're not wrong! I quote: "RAID 0+1 is implemented as a mirrored array whose segments are RAID 0" "RAID 10 is implemented as a striped array whose segments are RAID 1" This is the quote that interested me "RAID 0+1 is NOT to be confused with RAID 10. A single drive failure will cause the whole array to become, in essence, a RAID Level 0 array" and "Under certain circumstances, RAID 10 array can sustain multiple simultaneous drive failures". As bytepile has it, failure of 1 disk in 0+1 leaves you with just RAID 0 so one more failure on the other pair and your data is gone. On the other hand, failure of 1 disk in raid 10 leaves you with a working raid 1 that can sustain a second failure. > Either way, I think you can do both across 2 channels to be redundant for > a channel/cable failing. > > RAID 0+1: C1-m-C2 s C1-m-C2 > RAID 10: C1-s-C1 m C2-s-C2 Gaaa! I'll have to ponder that one. C1-m-C2 s C1-m-C2 sounds good in theory. Presumably to be practical it would require hardware and maybe software support, so the machine continues to operate despite the failure (and hopefully tells you about it). It would interesting to test it anyway. > Always get more space than you need if you can afford it because it's a > pain in the ass adding more disk space. Especially on the boot drives. Very true. > Doing a google search for "hot swap cpu", I see they've added something > into Linux but you'd still need hardware support. I'm sure that if the Tyan board had that support they'd be making a fuss of it :-) I'll have a look for "hot swap cpu" too though, just for interests sake. High redundancy is becoming more accessible I think, but it still takes a lot more money than your average PC server motherboard costs to get it ;-) something to look forward to anyway. Cheers, Iain
Hi Aaron, > You are correct. That kind of functionality can't be found on standard > x86 gear (sans the $150K NEC redundant behemoth). I reckon that the day is coming though, and I expect Linux will ready when the hardware arrives :-) cheers Iain
Iain wrote: > As bytepile has it, failure of 1 disk in 0+1 leaves you with just RAID 0 > so one more failure on the other pair and your data is gone. On the > other hand, failure of 1 disk in raid 10 leaves you with a working raid > 1 that can sustain a second failure. What they're saying is in the case of (AsB) m (CsD) -- if A fails, they no longer count B as part of the array and no longer part of the possible drives that can fail. Sorta like the "no one hears a tree fall, did it fall" scenario. I personally disagree with that theory. B is still part of the array. Pop in a new drive and the array is ready to start resync (CsD) --> (AsB). You still have a 1/3 chance in surviving another drive failure as long as B is the one that dies. Although now that I think about it, RAID10 is more resillient because the odds are survival after 1 failure is 2/3. In the case of (AmB) s (CmD), if A fails, you can survive C failing or D failing.
On Wed, Dec 15, 2004 at 11:14:03PM -0800, William Yu wrote: > > I'm not sure why people say one is better than the other. Both will > survive the loss of 2 drives -- they're just different drives. Partly, I think, people who've used both hate 0+1 because of the recovery cost. In most 0+1 arrangements (I'm aware of none which don't do this), you have to re-sync the entire thing in case you lose even one drive. Performance really suffers at that point. A -- Andrew Sullivan | ajs@crankycanuck.ca I remember when computers were frustrating because they *did* exactly what you told them to. That actually seems sort of quaint now. --J.D. Baldwin