Thread: most bang for buck with ~ $20,000
I've asked for some help here a few months ago and got some really helpfull answers regarding RAID controllers and server configuration. Up until recently I've been running PostgreSQL on a two year old Dual Xeon 3.06Ghz machine with a single channel RAID controller (previously Adaptec 2200S, but now changed to LSI MegaRAID). The 2U unit is from a generic vendor using what I believe is a SuperMicro motherboard. In the last week after upgrading the RAID controller, the machine has had disk failure and some other issues. I would like to build a very reliable dedicated postgreSQL server that has the ultimate possible performance and reliabily for around $20,000. The data set size is only currently about 4GB, but is increasing by approximately 50MB daily. The server also requires about 500 connections and I have been monitoring about 100-200 queries per second at the moment. I am planning to run FreeBSD 6.1 if possible, but I am open to any other suggestions if it improves performance. I am considering a setup such as this: - At least dual cpu (possibly with 2 cores each) - 4GB of RAM - 2 disk RAID 1 array for root disk - 4 disk RAID 1+0 array for PGDATA - 2 disk RAID 1 array for pg_xlog Does anyone know a vendor that might be able provide such setup? Any critique in this design? I'm thinking having a 2 channel RAID controller to seperate the PGDATA, root and pg_xlog. Sincerely, Kenji
>
> I am considering a setup such as this:
> - At least dual cpu (possibly with 2 cores each)
> - 4GB of RAM
> - 2 disk RAID 1 array for root disk
> - 4 disk RAID 1+0 array for PGDATA
> - 2 disk RAID 1 array for pg_xlog
>
> Does anyone know a vendor that might be able provide such setup?
I would look at the HP DL 385 or 585. The 385 is going to max a (2) dual
core cpus. The 585 is (4) dual core cpus.
For 20k you should be able to go with a bit more then what you have
above, specifically in the RAM department.
Joshua D. Drake
Any
> critique in this design? I'm thinking having a 2 channel RAID controller to
> seperate the PGDATA, root and pg_xlog.
>
> Sincerely,
> Kenji
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>
--
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/
On Aug 8, 2006, at 4:49 PM, Joshua D. Drake wrote: >> I am considering a setup such as this: >> - At least dual cpu (possibly with 2 cores each) >> - 4GB of RAM >> - 2 disk RAID 1 array for root disk >> - 4 disk RAID 1+0 array for PGDATA >> - 2 disk RAID 1 array for pg_xlog >> Does anyone know a vendor that might be able provide such setup? Wouldn't it be preferable to put WAL on a multi-disk RAID 10 if you had the opportunity? This gives you the redundancy of RAID 1 but approaches the performance of RAID 0, especially as you add disks to the array. In benchmarking, I've seen consistent success with this approach. -- Thomas F. O'Connell Sitening, LLC http://www.sitening.com/ 3004B Poston Avenue Nashville, TN 37203-1314 615-469-5150 x802 615-469-5151 (fax)
The 1+0 on the WAL is better than on PGDATA? I guess I'm confused about the write sequence of the data. I will research more, thank you! -Kenji On Tue, Aug 08, 2006 at 04:59:09PM -0500, Thomas F. O'Connell wrote: > > On Aug 8, 2006, at 4:49 PM, Joshua D. Drake wrote: > > >>I am considering a setup such as this: > >> - At least dual cpu (possibly with 2 cores each) > >> - 4GB of RAM > >> - 2 disk RAID 1 array for root disk > >> - 4 disk RAID 1+0 array for PGDATA > >> - 2 disk RAID 1 array for pg_xlog > >>Does anyone know a vendor that might be able provide such setup? > > Wouldn't it be preferable to put WAL on a multi-disk RAID 10 if you > had the opportunity? This gives you the redundancy of RAID 1 but > approaches the performance of RAID 0, especially as you add disks to > the array. In benchmarking, I've seen consistent success with this > approach. > > -- > Thomas F. O'Connell > Sitening, LLC > > http://www.sitening.com/ > 3004B Poston Avenue > Nashville, TN 37203-1314 > 615-469-5150 x802 > 615-469-5151 (fax)
On Tue, 2006-08-08 at 15:43, Kenji Morishige wrote: > I've asked for some help here a few months ago and got some really helpfull > answers regarding RAID controllers and server configuration. Up until > recently I've been running PostgreSQL on a two year old Dual Xeon 3.06Ghz > machine with a single channel RAID controller (previously Adaptec 2200S, but > now changed to LSI MegaRAID). The 2U unit is from a generic vendor using what > I believe is a SuperMicro motherboard. In the last week after upgrading the > RAID controller, the machine has had disk failure and some other issues. I > would like to build a very reliable dedicated postgreSQL server that has the > ultimate possible performance and reliabily for around $20,000. The data set > size is only currently about 4GB, but is increasing by approximately 50MB > daily. The server also requires about 500 connections and I have been > monitoring about 100-200 queries per second at the moment. I am planning to > run FreeBSD 6.1 if possible, but I am open to any other suggestions if it > improves performance. This really depends on your usage patterns. OLAP or OLTP workloads? Do you need 24/7 reliability and therefore a two machine setup? There's a lot of variety in load. Generally, you spend your money on disks, then memory, then CPU, in that order. Look at the Areca cards, they've come highly recommended here. Look at LOTS of drives. Given the size of your db, you can go with LOTS of smaller drives and get good performance. If you can find a good box to hold 12 to 16 drives and fill it with 37 gig 15k RPM drives, you'll have lots of storage, even in RAID 1+0 config. That's aiming at transactional throughput. Toss as much memory as is reasonably affordable at it. That's normally in the 4 to 8 gig range. After that things start to get expensive fast. Multiple - dual core CPUs are a good idea. Opterons seem to be better "data pumps" with large memory and >2 CPUs than Intels right now. Better to have a 2xdual core opteron with slower processors than a single dual core or dual single core CPU(s) with a faster clock speed. As long as the memory access is equivalent, the more CPUs the better in Opterons, where their interconnect speed increases as you increase the number of CPUs. Intel Xeons are the opposite. Better with fewer faster CPUs / cores. I just ran through a configurator on a site selling quad dual core opteron servers. 8 Seagate cheetah 15k rpm drives, 8 gig ram, and the slowest (1.8 GHz) AMD dual core CPUs (4 of them) for 8 cores, came out to $13,500 or so. I'd take the other $7.5 grand and buy a backup server that can old as much but isn't quite as beefy and set up slony to have a live hot spare sitting ready. Oh, and maybe to buy some spare parts to sit in the desk drawer in case things break.
Great info, which vendor were you looking at for these Opterons? I am goign to be purchasing 2 of these. :) I do need 24/7 reliability. On Tue, Aug 08, 2006 at 05:08:29PM -0500, Scott Marlowe wrote: > On Tue, 2006-08-08 at 15:43, Kenji Morishige wrote: > > I've asked for some help here a few months ago and got some really helpfull > > answers regarding RAID controllers and server configuration. Up until > > recently I've been running PostgreSQL on a two year old Dual Xeon 3.06Ghz > > machine with a single channel RAID controller (previously Adaptec 2200S, but > > now changed to LSI MegaRAID). The 2U unit is from a generic vendor using what > > I believe is a SuperMicro motherboard. In the last week after upgrading the > > RAID controller, the machine has had disk failure and some other issues. I > > would like to build a very reliable dedicated postgreSQL server that has the > > ultimate possible performance and reliabily for around $20,000. The data set > > size is only currently about 4GB, but is increasing by approximately 50MB > > daily. The server also requires about 500 connections and I have been > > monitoring about 100-200 queries per second at the moment. I am planning to > > run FreeBSD 6.1 if possible, but I am open to any other suggestions if it > > improves performance. > > This really depends on your usage patterns. > > OLAP or OLTP workloads? Do you need 24/7 reliability and therefore a > two machine setup? There's a lot of variety in load. > > Generally, you spend your money on disks, then memory, then CPU, in that > order. > > Look at the Areca cards, they've come highly recommended here. Look at > LOTS of drives. Given the size of your db, you can go with LOTS of > smaller drives and get good performance. If you can find a good box to > hold 12 to 16 drives and fill it with 37 gig 15k RPM drives, you'll have > lots of storage, even in RAID 1+0 config. That's aiming at > transactional throughput. > > Toss as much memory as is reasonably affordable at it. That's normally > in the 4 to 8 gig range. After that things start to get expensive fast. > > Multiple - dual core CPUs are a good idea. Opterons seem to be better > "data pumps" with large memory and >2 CPUs than Intels right now. > Better to have a 2xdual core opteron with slower processors than a > single dual core or dual single core CPU(s) with a faster clock speed. > As long as the memory access is equivalent, the more CPUs the better in > Opterons, where their interconnect speed increases as you increase the > number of CPUs. Intel Xeons are the opposite. Better with fewer faster > CPUs / cores. > > I just ran through a configurator on a site selling quad dual core > opteron servers. 8 Seagate cheetah 15k rpm drives, 8 gig ram, and the > slowest (1.8 GHz) AMD dual core CPUs (4 of them) for 8 cores, came out > to $13,500 or so. > > I'd take the other $7.5 grand and buy a backup server that can old as > much but isn't quite as beefy and set up slony to have a live hot spare > sitting ready. Oh, and maybe to buy some spare parts to sit in the desk > drawer in case things break.
On Aug 8, 2006, at 1:43 PM, Kenji Morishige wrote: > > > I am considering a setup such as this: > - At least dual cpu (possibly with 2 cores each) > - 4GB of RAM > - 2 disk RAID 1 array for root disk > - 4 disk RAID 1+0 array for PGDATA > - 2 disk RAID 1 array for pg_xlog > > Does anyone know a vendor that might be able provide such setup? Any > critique in this design? I'm thinking having a 2 channel RAID > controller to > seperate the PGDATA, root and pg_xlog. It's fairly similar to the system I'm using - I have 6 spindles for PGDATA rather than 4. It's a 9550SX SATA based box with 26 bays (16 empty right now) dual Opterons and 4 gigs of RAM. About $7k from asacomputers.com. Performance is good, drastically better than the SCSI Dell server it replaced, but I've not benchmarked it. You mention 500 simultaneous connections. If that's really 500 simultaneous database connections (rather than, say, 500 simultaneous web users feeding into a pool of fewer database connections) then that's a lot. I might go with a bit more RAM than 4GB. You should have a lot left over from $20k, and spending some of it on more RAM would be a good investment (as might a better RAID controller than the 9550SX, some dual core CPUs and maybe some more PGDATA spindles). Cheers, Steve
Thomas F. O'Connell wrote: > > On Aug 8, 2006, at 4:49 PM, Joshua D. Drake wrote: > >>> I am considering a setup such as this: >>> - At least dual cpu (possibly with 2 cores each) >>> - 4GB of RAM >>> - 2 disk RAID 1 array for root disk >>> - 4 disk RAID 1+0 array for PGDATA >>> - 2 disk RAID 1 array for pg_xlog >>> Does anyone know a vendor that might be able provide such setup? > > Wouldn't it be preferable to put WAL on a multi-disk RAID 10 if you had > the opportunity? This gives you the redundancy of RAID 1 but approaches > the performance of RAID 0, especially as you add disks to the array. In > benchmarking, I've seen consistent success with this approach. WALL is written in order so RAID 1 is usually fine. We also don't need journaling for WAL so the speed is even faster. Joshua D. Drake > > -- > Thomas F. O'Connell > Sitening, LLC > > http://www.sitening.com/ > 3004B Poston Avenue > Nashville, TN 37203-1314 > 615-469-5150 x802 > 615-469-5151 (fax) > > ---------------------------(end of broadcast)--------------------------- > TIP 4: Have you searched our list archives? > > http://archives.postgresql.org > -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240 Providing the most comprehensive PostgreSQL solutions since 1997 http://www.commandprompt.com/
On Aug 8, 2006, at 5:28 PM, Joshua D. Drake wrote: > Thomas F. O'Connell wrote: >> On Aug 8, 2006, at 4:49 PM, Joshua D. Drake wrote: >>>> I am considering a setup such as this: >>>> - At least dual cpu (possibly with 2 cores each) >>>> - 4GB of RAM >>>> - 2 disk RAID 1 array for root disk >>>> - 4 disk RAID 1+0 array for PGDATA >>>> - 2 disk RAID 1 array for pg_xlog >>>> Does anyone know a vendor that might be able provide such setup? >> Wouldn't it be preferable to put WAL on a multi-disk RAID 10 if >> you had the opportunity? This gives you the redundancy of RAID 1 >> but approaches the performance of RAID 0, especially as you add >> disks to the array. In benchmarking, I've seen consistent success >> with this approach. > > WALL is written in order so RAID 1 is usually fine. We also don't > need journaling for WAL so the speed is even faster. In which case, which is theoretically better (since I don't have a convenient test bed at the moment) for WAL in a write-heavy environment? More disks in a RAID 10 (which should theoretically improve write throughput in general, to a point) or a 2-disk RAID 1? Does it become a price/performance question, or is there virtually no benefit to throwing more disks at RAID 10 for WAL if you turn off journaling on the filesystem? -- Thomas F. O'Connell Sitening, LLC http://www.sitening.com/ 3004B Poston Avenue Nashville, TN 37203-1314 615-469-5150 x802 615-469-5151 (fax)
>
> In which case, which is theoretically better (since I don't have a
> convenient test bed at the moment) for WAL in a write-heavy environment?
> More disks in a RAID 10 (which should theoretically improve write
> throughput in general, to a point) or a 2-disk RAID 1? Does it become a
> price/performance question, or is there virtually no benefit to throwing
> more disks at RAID 10 for WAL if you turn off journaling on the filesystem?
Over 4 drives, I would gather that RAID 10 wouldn't gain you anything.
Possibly over 6 or 8 however, it may be faster because you are writing
smaller chunks of data, even if two copies of each.
Sincerely,
Joshua D. Drake
--
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/
On Aug 8, 2006, at 6:24 PM, Joshua D. Drake wrote: >> In which case, which is theoretically better (since I don't have a >> convenient test bed at the moment) for WAL in a write-heavy >> environment? More disks in a RAID 10 (which should theoretically >> improve write throughput in general, to a point) or a 2-disk RAID >> 1? Does it become a price/performance question, or is there >> virtually no benefit to throwing more disks at RAID 10 for WAL if >> you turn off journaling on the filesystem? > > Over 4 drives, I would gather that RAID 10 wouldn't gain you > anything. Possibly over 6 or 8 however, it may be faster because > you are writing smaller chunks of data, even if two copies of each. Yeah, where I've seen the benefits in practice, the scenarios have involved the availability of a minimum of 6 drives for a RAID 10 for WAL. I really should do a comparison of a 2-disk RAID 1 with a variety of multi-disk RAID 10 configurations at some point. -- Thomas F. O'Connell Sitening, LLC http://www.sitening.com/ 3004B Poston Avenue Nashville, TN 37203-1314 615-469-5150 x802 615-469-5151 (fax)
With such a budget you should easily be able to get something like: - A 1U high-performance server (for instance the Dell 1950 with 2x Woodcrest 5160, 16GB of FB-Dimm memory, one 5i and one 5e perc raid controller and some disks internally) - An external SAS direct attached disks storage enclosure full with 15k rpm 36GB disks (for instance the MD1000, with 15x 36GB 15k disks) Going for the dell-solution would set you back "only" (including savings) about $13-$14k. HP offers a similar solutions (a HP DL360G5 or a DL380G5/DL385 with two MSA50's for instance) which also fit in your budget afaik. The other players tend to be (a bit) more expensive, force you to go with Fibre Channel or "ancient" SCSI external storage ;) If you'd like to have a product by a generic vendor, have a look at the Adaptec JS50 SAS Jbod enclosure or Promise's Vtrak 300 (both offer 12 sas/sata bays in 2U) for storage. If you're limited to only 2U of rack space, its a bit more difficult to get maximum I/O in your budget (you have basically space for about 8 or 12 3.5" disks (with generic suppliers) or 16 2.5" sff disks (with HP)). But you should still be able to have two top-off-the-line x86 cpu's (amd opteron 285 or intel woorcrest 5160) and 16GB of memory (even FB Dimm, which is pretty expensive). Best regards, Arjen van der Meijden On 8-8-2006 22:43, Kenji Morishige wrote: > I've asked for some help here a few months ago and got some really helpfull > answers regarding RAID controllers and server configuration. Up until > recently I've been running PostgreSQL on a two year old Dual Xeon 3.06Ghz > machine with a single channel RAID controller (previously Adaptec 2200S, but > now changed to LSI MegaRAID). The 2U unit is from a generic vendor using what > I believe is a SuperMicro motherboard. In the last week after upgrading the > RAID controller, the machine has had disk failure and some other issues. I > would like to build a very reliable dedicated postgreSQL server that has the > ultimate possible performance and reliabily for around $20,000. The data set > size is only currently about 4GB, but is increasing by approximately 50MB > daily. The server also requires about 500 connections and I have been > monitoring about 100-200 queries per second at the moment. I am planning to > run FreeBSD 6.1 if possible, but I am open to any other suggestions if it > improves performance. > > I am considering a setup such as this: > - At least dual cpu (possibly with 2 cores each) > - 4GB of RAM > - 2 disk RAID 1 array for root disk > - 4 disk RAID 1+0 array for PGDATA > - 2 disk RAID 1 array for pg_xlog > > Does anyone know a vendor that might be able provide such setup? Any > critique in this design? I'm thinking having a 2 channel RAID controller to > seperate the PGDATA, root and pg_xlog. > > Sincerely, > Kenji > > ---------------------------(end of broadcast)--------------------------- > TIP 2: Don't 'kill -9' the postmaster >
I have unlimited rack space, so 2U is not the issue. The boxes are stored in our lab for internal software tools. I'm going to research those boxes you mention. Regarding the JBOD enclosures, are these generally just 2U or 4U units with SCSI interface connectors? I didn't see these types of boxes availble on Dell website, I'll look again. -Kenji On Wed, Aug 09, 2006 at 07:35:22AM +0200, Arjen van der Meijden wrote: > With such a budget you should easily be able to get something like: > - A 1U high-performance server (for instance the Dell 1950 with 2x > Woodcrest 5160, 16GB of FB-Dimm memory, one 5i and one 5e perc raid > controller and some disks internally) > - An external SAS direct attached disks storage enclosure full with 15k > rpm 36GB disks (for instance the MD1000, with 15x 36GB 15k disks) > > Going for the dell-solution would set you back "only" (including > savings) about $13-$14k. HP offers a similar solutions (a HP DL360G5 or > a DL380G5/DL385 with two MSA50's for instance) which also fit in your > budget afaik. The other players tend to be (a bit) more expensive, force > you to go with Fibre Channel or "ancient" SCSI external storage ;) > > If you'd like to have a product by a generic vendor, have a look at the > Adaptec JS50 SAS Jbod enclosure or Promise's Vtrak 300 (both offer 12 > sas/sata bays in 2U) for storage. > > If you're limited to only 2U of rack space, its a bit more difficult to > get maximum I/O in your budget (you have basically space for about 8 or > 12 3.5" disks (with generic suppliers) or 16 2.5" sff disks (with HP)). > But you should still be able to have two top-off-the-line x86 cpu's (amd > opteron 285 or intel woorcrest 5160) and 16GB of memory (even FB Dimm, > which is pretty expensive). > > Best regards, > > Arjen van der Meijden > > > On 8-8-2006 22:43, Kenji Morishige wrote: > >I've asked for some help here a few months ago and got some really helpfull > >answers regarding RAID controllers and server configuration. Up until > >recently I've been running PostgreSQL on a two year old Dual Xeon 3.06Ghz > >machine with a single channel RAID controller (previously Adaptec 2200S, > >but > >now changed to LSI MegaRAID). The 2U unit is from a generic vendor using > >what > >I believe is a SuperMicro motherboard. In the last week after upgrading > >the > >RAID controller, the machine has had disk failure and some other issues. I > >would like to build a very reliable dedicated postgreSQL server that has > >the > >ultimate possible performance and reliabily for around $20,000. The data > >set > >size is only currently about 4GB, but is increasing by approximately 50MB > >daily. The server also requires about 500 connections and I have been > >monitoring about 100-200 queries per second at the moment. I am planning > >to > >run FreeBSD 6.1 if possible, but I am open to any other suggestions if it > >improves performance. > > > >I am considering a setup such as this: > > - At least dual cpu (possibly with 2 cores each) > > - 4GB of RAM > > - 2 disk RAID 1 array for root disk > > - 4 disk RAID 1+0 array for PGDATA > > - 2 disk RAID 1 array for pg_xlog > > > >Does anyone know a vendor that might be able provide such setup? Any > >critique in this design? I'm thinking having a 2 channel RAID controller to > >seperate the PGDATA, root and pg_xlog. > > > >Sincerely, > >Kenji > > > >---------------------------(end of broadcast)--------------------------- > >TIP 2: Don't 'kill -9' the postmaster > >
We were in a similar situation with a similar budget. But we had two requirements, no "deprecated" scsi while the successor SAS is available and preferrably only 3 or 4U of rack space. And it had to have reasonable amounts of disks (at least 12). The two options we finally choose between where a Dell 1U 1950 with two woodcrests 5160 (I don't think the older dempsey 50x0's are a good idea) and 16GB of memory combined with a PowerVault MD1000 external storage SAS JBOD unit, with 15 36GB 15k rpm disks and from HP a similar configured DL360G5 (also 1U) combined with two MSA50 SFF SAS JBOD enclosures with 20 36GB 10k rpm SFF disks. Both enclosures offer has SAS-connectivity (serial attached scsi), i.e. the "next generation scsi". Which is supposed to be the successor to scsi, but unfortunately its not yet as widely available. The Dell MD1000 is 3U high and can be fitted with 15 3.5" disks, the MSA50 is 1U and can be fitted with 10 2.5" disks. In terms of performance you'll likely need two MSA50's to be up to par with one MD1000. The SFF disks are about as expensive as the 15k 3.5" disks... so its mostly interesting for packing a lot of I/O in a small enclosure. HP is going to offer a 3.5" SAS-enclosure (MSA60) but that one won't be available until Q1 2007 or something like that. As said Promise and Adaptec also offer SAS enclosures, both are 2U and can be fitted with 12 disks. There are more available, but they are generally quite bit hard to find. Good luck with your search. Best regards, Arjen Kenji Morishige wrote: > I have unlimited rack space, so 2U is not the issue. The boxes are stored in > our lab for internal software tools. I'm going to research those boxes you > mention. Regarding the JBOD enclosures, are these generally just 2U or 4U > units with SCSI interface connectors? I didn't see these types of boxes > availble on Dell website, I'll look again. > -Kenji > > On Wed, Aug 09, 2006 at 07:35:22AM +0200, Arjen van der Meijden wrote: >> With such a budget you should easily be able to get something like: >> - A 1U high-performance server (for instance the Dell 1950 with 2x >> Woodcrest 5160, 16GB of FB-Dimm memory, one 5i and one 5e perc raid >> controller and some disks internally) >> - An external SAS direct attached disks storage enclosure full with 15k >> rpm 36GB disks (for instance the MD1000, with 15x 36GB 15k disks) >> >> Going for the dell-solution would set you back "only" (including >> savings) about $13-$14k. HP offers a similar solutions (a HP DL360G5 or >> a DL380G5/DL385 with two MSA50's for instance) which also fit in your >> budget afaik. The other players tend to be (a bit) more expensive, force >> you to go with Fibre Channel or "ancient" SCSI external storage ;) >> >> If you'd like to have a product by a generic vendor, have a look at the >> Adaptec JS50 SAS Jbod enclosure or Promise's Vtrak 300 (both offer 12 >> sas/sata bays in 2U) for storage. >> >> If you're limited to only 2U of rack space, its a bit more difficult to >> get maximum I/O in your budget (you have basically space for about 8 or >> 12 3.5" disks (with generic suppliers) or 16 2.5" sff disks (with HP)). >> But you should still be able to have two top-off-the-line x86 cpu's (amd >> opteron 285 or intel woorcrest 5160) and 16GB of memory (even FB Dimm, >> which is pretty expensive). >> >> Best regards, >> >> Arjen van der Meijden >> >> >> On 8-8-2006 22:43, Kenji Morishige wrote: >>> I've asked for some help here a few months ago and got some really helpfull >>> answers regarding RAID controllers and server configuration. Up until >>> recently I've been running PostgreSQL on a two year old Dual Xeon 3.06Ghz >>> machine with a single channel RAID controller (previously Adaptec 2200S, >>> but >>> now changed to LSI MegaRAID). The 2U unit is from a generic vendor using >>> what >>> I believe is a SuperMicro motherboard. In the last week after upgrading >>> the >>> RAID controller, the machine has had disk failure and some other issues. I >>> would like to build a very reliable dedicated postgreSQL server that has >>> the >>> ultimate possible performance and reliabily for around $20,000. The data >>> set >>> size is only currently about 4GB, but is increasing by approximately 50MB >>> daily. The server also requires about 500 connections and I have been >>> monitoring about 100-200 queries per second at the moment. I am planning >>> to >>> run FreeBSD 6.1 if possible, but I am open to any other suggestions if it >>> improves performance. >>> >>> I am considering a setup such as this: >>> - At least dual cpu (possibly with 2 cores each) >>> - 4GB of RAM >>> - 2 disk RAID 1 array for root disk >>> - 4 disk RAID 1+0 array for PGDATA >>> - 2 disk RAID 1 array for pg_xlog >>> >>> Does anyone know a vendor that might be able provide such setup? Any >>> critique in this design? I'm thinking having a 2 channel RAID controller to >>> seperate the PGDATA, root and pg_xlog. >>> >>> Sincerely, >>> Kenji >>> >>> ---------------------------(end of broadcast)--------------------------- >>> TIP 2: Don't 'kill -9' the postmaster >>> >
Unsubscribe -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.394 / Virus Database: 268.10.8/413 - Release Date: 2006/08/08
On Tue, 2006-08-08 at 17:53, Thomas F. O'Connell wrote: > On Aug 8, 2006, at 5:28 PM, Joshua D. Drake wrote: > > > Thomas F. O'Connell wrote: > >> On Aug 8, 2006, at 4:49 PM, Joshua D. Drake wrote: > >>>> I am considering a setup such as this: > >>>> - At least dual cpu (possibly with 2 cores each) > >>>> - 4GB of RAM > >>>> - 2 disk RAID 1 array for root disk > >>>> - 4 disk RAID 1+0 array for PGDATA > >>>> - 2 disk RAID 1 array for pg_xlog > >>>> Does anyone know a vendor that might be able provide such setup? > >> Wouldn't it be preferable to put WAL on a multi-disk RAID 10 if > >> you had the opportunity? This gives you the redundancy of RAID 1 > >> but approaches the performance of RAID 0, especially as you add > >> disks to the array. In benchmarking, I've seen consistent success > >> with this approach. > > > > WALL is written in order so RAID 1 is usually fine. We also don't > > need journaling for WAL so the speed is even faster. > > In which case, which is theoretically better (since I don't have a > convenient test bed at the moment) for WAL in a write-heavy > environment? More disks in a RAID 10 (which should theoretically > improve write throughput in general, to a point) or a 2-disk RAID 1? > Does it become a price/performance question, or is there virtually no > benefit to throwing more disks at RAID 10 for WAL if you turn off > journaling on the filesystem? Actually, the BIGGEST win comes when you've got battery backed cache on your RAID controller. In fact, I'd spend money on a separate RAID controller for xlog with its own cache hitting a simple mirror set before I'd spring for more drives on pg_xlog. The battery backed cache on the pg_xlog likely wouldn't need to be big, just there and set to write-back. Then put all the rest of your cash into disks on a big RAID 10 config, and as big of a battery backed cache as you can afford for it and memory for the machine.
On 8/9/06, Kenji Morishige <kenjim@juniper.net> wrote: > I have unlimited rack space, so 2U is not the issue. The boxes are stored in > our lab for internal software tools. I'm going to research those boxes you > mention. Regarding the JBOD enclosures, are these generally just 2U or 4U > units with SCSI interface connectors? I didn't see these types of boxes > availble on Dell website, I'll look again. > -Kenji > > On Wed, Aug 09, 2006 at 07:35:22AM +0200, Arjen van der Meijden wrote: > > With such a budget you should easily be able to get something like: > > - A 1U high-performance server (for instance the Dell 1950 with 2x > > Woodcrest 5160, 16GB of FB-Dimm memory, one 5i and one 5e perc raid > > controller and some disks internally) > > - An external SAS direct attached disks storage enclosure full with 15k > > rpm 36GB disks (for instance the MD1000, with 15x 36GB 15k disks) > > > > Going for the dell-solution would set you back "only" (including > > savings) about $13-$14k. HP offers a similar solutions (a HP DL360G5 or > > a DL380G5/DL385 with two MSA50's for instance) which also fit in your > > budget afaik. The other players tend to be (a bit) more expensive, force > > you to go with Fibre Channel or "ancient" SCSI external storage ;) > > > > If you'd like to have a product by a generic vendor, have a look at the > > Adaptec JS50 SAS Jbod enclosure or Promise's Vtrak 300 (both offer 12 > > sas/sata bays in 2U) for storage. I am really curious about the Adaptec SAS product to see what it can do. If you don't know what SAS is, it is Sata Attached SCSI. SAS cables use 4 sata lanes (3gb/sec each) bonded together in a single cable. The raid is handled with via the raid controller or the o/s in a software configuration. SAS is the evolution of SCSI and I think will ultimately replace scsi in enterprise setups becuase it is faster, cheaper, and more flexible. SAS enclosures generally accept sata or sas drives in mix/match configurations. so, you get to choose between cheap, large, 7200 rpm sata drives or small, expensive sas 10k or 15k rpm drives *in the same enclosure*. You also get a compromise drive in the form of the raptor which is 10k rpm sata drive. You could buy a 2u 12 drive SAS encloure (3000$), 12 150g raptors (3000$) and spend another grand on cables/controller and have a hellishly performing raid system for the money assuming sas performs like it does on paper. note that i would not be trying this with my own money unless I was guaranteed a money back rma for a 30 day evaluation period. that would leave you with 12 grand or so to pick up a quad (8 core) opeteron if you bought it right. regards, merlin
On Wed, Aug 09, 2006 at 10:15:27AM -0500, Scott Marlowe wrote: > Actually, the BIGGEST win comes when you've got battery backed cache on > your RAID controller. In fact, I'd spend money on a separate RAID > controller for xlog with its own cache hitting a simple mirror set > before I'd spring for more drives on pg_xlog. The battery backed cache > on the pg_xlog likely wouldn't need to be big, just there and set to > write-back. > > Then put all the rest of your cash into disks on a big RAID 10 config, > and as big of a battery backed cache as you can afford for it and memory > for the machine. Actually, my (limited) testing has show than on a good battery-backed controller, there's no penalty to leaving pg_xlog in with the rest of PGDATA. This means that the OP could pile all 8 drives into a RAID10, which would almost certainly do better than 6+2. Note that some controllers (such as 3ware) need to periodically test the life of the BBU, and they disable write caching when they do so, which would tank performance. -- Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com Pervasive Software http://pervasive.com work: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461
On Wed, 2006-08-09 at 16:35, Jim C. Nasby wrote: > On Wed, Aug 09, 2006 at 10:15:27AM -0500, Scott Marlowe wrote: > > Actually, the BIGGEST win comes when you've got battery backed cache on > > your RAID controller. In fact, I'd spend money on a separate RAID > > controller for xlog with its own cache hitting a simple mirror set > > before I'd spring for more drives on pg_xlog. The battery backed cache > > on the pg_xlog likely wouldn't need to be big, just there and set to > > write-back. > > > > Then put all the rest of your cash into disks on a big RAID 10 config, > > and as big of a battery backed cache as you can afford for it and memory > > for the machine. > > Actually, my (limited) testing has show than on a good battery-backed > controller, there's no penalty to leaving pg_xlog in with the rest of > PGDATA. This means that the OP could pile all 8 drives into a RAID10, > which would almost certainly do better than 6+2. I've seen a few posts that said that before. I wonder if there's a point where the single RAID array / controller would get saturated and a second one would help. I think most of the testing I've seen so far has been multiple RAID arrays under the same controller, hasn't it? > Note that some controllers (such as 3ware) need to periodically test the > life of the BBU, and they disable write caching when they do so, which > would tank performance. ugh, that's a scary thing. Can you at least schedule it?
On Wed, Aug 09, 2006 at 04:50:30PM -0500, Scott Marlowe wrote: > On Wed, 2006-08-09 at 16:35, Jim C. Nasby wrote: > > On Wed, Aug 09, 2006 at 10:15:27AM -0500, Scott Marlowe wrote: > > > Actually, the BIGGEST win comes when you've got battery backed cache on > > > your RAID controller. In fact, I'd spend money on a separate RAID > > > controller for xlog with its own cache hitting a simple mirror set > > > before I'd spring for more drives on pg_xlog. The battery backed cache > > > on the pg_xlog likely wouldn't need to be big, just there and set to > > > write-back. > > > > > > Then put all the rest of your cash into disks on a big RAID 10 config, > > > and as big of a battery backed cache as you can afford for it and memory > > > for the machine. > > > > Actually, my (limited) testing has show than on a good battery-backed > > controller, there's no penalty to leaving pg_xlog in with the rest of > > PGDATA. This means that the OP could pile all 8 drives into a RAID10, > > which would almost certainly do better than 6+2. > > I've seen a few posts that said that before. I wonder if there's a > point where the single RAID array / controller would get saturated and a > second one would help. I think most of the testing I've seen so far has > been multiple RAID arrays under the same controller, hasn't it? Yeah. I've had one client try it so far, but it was a pretty small array (8 drives, IIRC). I suspect that by the time you get to a size where you're saturating a controller, you're looking at enough drives where having two extra (instead of dedicating them to pg_xlog) won't make much difference. > > Note that some controllers (such as 3ware) need to periodically test the > > life of the BBU, and they disable write caching when they do so, which > > would tank performance. > > ugh, that's a scary thing. Can you at least schedule it? Yeah, it's not automatic at all. Which itself is somewhat scarry.... -- Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com Pervasive Software http://pervasive.com work: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461
On Aug 9, 2006, at 5:35 PM, Jim C. Nasby wrote: > Note that some controllers (such as 3ware) need to periodically > test the > life of the BBU, and they disable write caching when they do so, which > would tank performance. Yep. I did the battery capacity test before I went live with our 9550sx controller. The only downside I see by not doing it is its estimated battery lifetime number may be inaccurate, and once a week you get an alarm message about the capacity test being overdue. It does seem like a big design flaw needing to do it, but if you think about it, you don't want to have data in the cache while seeing how long it takes for the battery to drain :) -- Jeff Trout <jeff@jefftrout.com> http://www.dellsmartexitin.com/ http://www.stuarthamm.net/
# jd@commandprompt.com / 2006-08-08 14:49:21 -0700:
> >I am considering a setup such as this:
> > - At least dual cpu (possibly with 2 cores each)
> > - 4GB of RAM
> > - 2 disk RAID 1 array for root disk
> > - 4 disk RAID 1+0 array for PGDATA
> > - 2 disk RAID 1 array for pg_xlog
> >
> >Does anyone know a vendor that might be able provide such setup?
>
> I would look at the HP DL 385 or 585. The 385 is going to max a (2) dual
> core cpus. The 585 is (4) dual core cpus.
I don't know about DL385 or DL585, but DL380 seem to go south within
1 year of heavy hitting; precisely the Smart Array RAID controllers
(4 out of 6 disks suddenly "red"; insert new disks, ooops red as
well).
I've seen this happen several times, and came away with a conclusion
that DL380 is sexy, but you don't want to marry it. Then again,
maybe the DL385 is different, though I seem to remember that both
G3 (Smart Array 5i) and G4 (6i) did this.
--
How many Vietnam vets does it take to screw in a light bulb?
You don't know, man. You don't KNOW.
Cause you weren't THERE. http://bash.org/?255991
Regarding the DL585 etc boxes from HP, they seem to require external JBOD or SCSI/SAS enclosures. Does anyone have any particular preference on how these units should be configured or speced? I'm guessing I'll use the onboard SCSI RAID 1 with the onboard drives for the OS, but will need 2 external channels for the data and xlog. Any recommendations there? Sincerely, Kenji On Sat, Aug 12, 2006 at 04:20:19PM +0000, Roman Neuhauser wrote: > # jd@commandprompt.com / 2006-08-08 14:49:21 -0700: > > >I am considering a setup such as this: > > > - At least dual cpu (possibly with 2 cores each) > > > - 4GB of RAM > > > - 2 disk RAID 1 array for root disk > > > - 4 disk RAID 1+0 array for PGDATA > > > - 2 disk RAID 1 array for pg_xlog > > > > > >Does anyone know a vendor that might be able provide such setup? > > > > I would look at the HP DL 385 or 585. The 385 is going to max a (2) dual > > core cpus. The 585 is (4) dual core cpus. > > I don't know about DL385 or DL585, but DL380 seem to go south within > 1 year of heavy hitting; precisely the Smart Array RAID controllers > (4 out of 6 disks suddenly "red"; insert new disks, ooops red as > well). > > I've seen this happen several times, and came away with a conclusion > that DL380 is sexy, but you don't want to marry it. Then again, > maybe the DL385 is different, though I seem to remember that both > G3 (Smart Array 5i) and G4 (6i) did this. > > -- > How many Vietnam vets does it take to screw in a light bulb? > You don't know, man. You don't KNOW. > Cause you weren't THERE. http://bash.org/?255991
Thanks Arjen, I have unlimited rack space if I really need it. Is serial/SAS really the better route to go than SCSI these days? I'm so used to ordering SCSI that I've been out of the loop with new disk enclosures and disk tech. I been trying to price out a HP DL585, but those are considerably more than the Dells. Is it worth waiting a few more weeks/months for Dell to release something newer? -Kenji On Wed, Aug 09, 2006 at 07:35:22AM +0200, Arjen van der Meijden wrote: > With such a budget you should easily be able to get something like: > - A 1U high-performance server (for instance the Dell 1950 with 2x > Woodcrest 5160, 16GB of FB-Dimm memory, one 5i and one 5e perc raid > controller and some disks internally) > - An external SAS direct attached disks storage enclosure full with 15k > rpm 36GB disks (for instance the MD1000, with 15x 36GB 15k disks) > > Going for the dell-solution would set you back "only" (including > savings) about $13-$14k. HP offers a similar solutions (a HP DL360G5 or > a DL380G5/DL385 with two MSA50's for instance) which also fit in your > budget afaik. The other players tend to be (a bit) more expensive, force > you to go with Fibre Channel or "ancient" SCSI external storage ;) > > If you'd like to have a product by a generic vendor, have a look at the > Adaptec JS50 SAS Jbod enclosure or Promise's Vtrak 300 (both offer 12 > sas/sata bays in 2U) for storage. > > If you're limited to only 2U of rack space, its a bit more difficult to > get maximum I/O in your budget (you have basically space for about 8 or > 12 3.5" disks (with generic suppliers) or 16 2.5" sff disks (with HP)). > But you should still be able to have two top-off-the-line x86 cpu's (amd > opteron 285 or intel woorcrest 5160) and 16GB of memory (even FB Dimm, > which is pretty expensive). > > Best regards, > > Arjen van der Meijden > > > On 8-8-2006 22:43, Kenji Morishige wrote: > >I've asked for some help here a few months ago and got some really helpfull > >answers regarding RAID controllers and server configuration. Up until > >recently I've been running PostgreSQL on a two year old Dual Xeon 3.06Ghz > >machine with a single channel RAID controller (previously Adaptec 2200S, > >but > >now changed to LSI MegaRAID). The 2U unit is from a generic vendor using > >what > >I believe is a SuperMicro motherboard. In the last week after upgrading > >the > >RAID controller, the machine has had disk failure and some other issues. I > >would like to build a very reliable dedicated postgreSQL server that has > >the > >ultimate possible performance and reliabily for around $20,000. The data > >set > >size is only currently about 4GB, but is increasing by approximately 50MB > >daily. The server also requires about 500 connections and I have been > >monitoring about 100-200 queries per second at the moment. I am planning > >to > >run FreeBSD 6.1 if possible, but I am open to any other suggestions if it > >improves performance. > > > >I am considering a setup such as this: > > - At least dual cpu (possibly with 2 cores each) > > - 4GB of RAM > > - 2 disk RAID 1 array for root disk > > - 4 disk RAID 1+0 array for PGDATA > > - 2 disk RAID 1 array for pg_xlog > > > >Does anyone know a vendor that might be able provide such setup? Any > >critique in this design? I'm thinking having a 2 channel RAID controller to > >seperate the PGDATA, root and pg_xlog. > > > >Sincerely, > >Kenji > > > >---------------------------(end of broadcast)--------------------------- > >TIP 2: Don't 'kill -9' the postmaster > >
Hi Kenji, I'm not sure what you mean by 'something newer'? The intel woodcrest-cpu's are brand-new compared to the amd opterons. But if you need a 4-cpu config (I take it you want 8-cores in that case), Dell doesn't offer much. Whether something new will come, I don't know. I'm not sure when (or if?) a MP-Woodcrest will arrive and/or when Dell will start offering Opteron-servers. Sas has been designed as the successor to SCSI. As I see it, SAS has currently one major disadvantage. Lots of new servers are equipped with SAS-drives, a few nice SAS-raidcontrollers exist, but the availability of external enclosures for SAS is not widespread yet. So your options of going beyond (say) 8 disks per system are a bit limited. There are of course advantages as well. The bus is much wider (you can have 4 lanes of 3Gbps each to an enclosure). You can mix sas and sata disks, so you could have two arrays in the same enclosure, one big storage bin and a very fast array or just use only sata disks on a sas controller. The cabling itself is also much simpler/more flexible (although using a hot-plug enclosure of course shields you mostly from that). But whether its the right choice to make now? I'm not sure. We weren't to fond of investing a lot of money in an end-of-life system. And since we're a tech-website, we also had to worry about our "being modern image", of course ;) The main disadvantage I see in this case is, as said, the limited availability of external enclosures in comparison to SCSI and Fibre Channel. HP currently only offers their MSA50 (for the rather expensive SFF disks) while their MSA60 (normal disks) will not be available until somewhere in 2007 and Dell also only offers one enclosure, the MD1000. The other big players offer nothing yet, as far as I know, while they normally offer several SCSI and/or FC-enclosures. There are also some third-party enclosures (adaptec and promise for instance) available of course. Best regards, Arjen On 18-8-2006 21:07, Kenji Morishige wrote: > Thanks Arjen, > I have unlimited rack space if I really need it. Is serial/SAS really the > better route to go than SCSI these days? I'm so used to ordering SCSI that > I've been out of the loop with new disk enclosures and disk tech. I been > trying to price out a HP DL585, but those are considerably more than the > Dells. Is it worth waiting a few more weeks/months for Dell to release > something newer? > > -Kenji > > On Wed, Aug 09, 2006 at 07:35:22AM +0200, Arjen van der Meijden wrote: >> With such a budget you should easily be able to get something like: >> - A 1U high-performance server (for instance the Dell 1950 with 2x >> Woodcrest 5160, 16GB of FB-Dimm memory, one 5i and one 5e perc raid >> controller and some disks internally) >> - An external SAS direct attached disks storage enclosure full with 15k >> rpm 36GB disks (for instance the MD1000, with 15x 36GB 15k disks) >> >> Going for the dell-solution would set you back "only" (including >> savings) about $13-$14k. HP offers a similar solutions (a HP DL360G5 or >> a DL380G5/DL385 with two MSA50's for instance) which also fit in your >> budget afaik. The other players tend to be (a bit) more expensive, force >> you to go with Fibre Channel or "ancient" SCSI external storage ;) >> >> If you'd like to have a product by a generic vendor, have a look at the >> Adaptec JS50 SAS Jbod enclosure or Promise's Vtrak 300 (both offer 12 >> sas/sata bays in 2U) for storage. >> >> If you're limited to only 2U of rack space, its a bit more difficult to >> get maximum I/O in your budget (you have basically space for about 8 or >> 12 3.5" disks (with generic suppliers) or 16 2.5" sff disks (with HP)). >> But you should still be able to have two top-off-the-line x86 cpu's (amd >> opteron 285 or intel woorcrest 5160) and 16GB of memory (even FB Dimm, >> which is pretty expensive). >> >> Best regards, >> >> Arjen van der Meijden >> >> >> On 8-8-2006 22:43, Kenji Morishige wrote: >>> I've asked for some help here a few months ago and got some really helpfull >>> answers regarding RAID controllers and server configuration. Up until >>> recently I've been running PostgreSQL on a two year old Dual Xeon 3.06Ghz >>> machine with a single channel RAID controller (previously Adaptec 2200S, >>> but >>> now changed to LSI MegaRAID). The 2U unit is from a generic vendor using >>> what >>> I believe is a SuperMicro motherboard. In the last week after upgrading >>> the >>> RAID controller, the machine has had disk failure and some other issues. I >>> would like to build a very reliable dedicated postgreSQL server that has >>> the >>> ultimate possible performance and reliabily for around $20,000. The data >>> set >>> size is only currently about 4GB, but is increasing by approximately 50MB >>> daily. The server also requires about 500 connections and I have been >>> monitoring about 100-200 queries per second at the moment. I am planning >>> to >>> run FreeBSD 6.1 if possible, but I am open to any other suggestions if it >>> improves performance. >>> >>> I am considering a setup such as this: >>> - At least dual cpu (possibly with 2 cores each) >>> - 4GB of RAM >>> - 2 disk RAID 1 array for root disk >>> - 4 disk RAID 1+0 array for PGDATA >>> - 2 disk RAID 1 array for pg_xlog >>> >>> Does anyone know a vendor that might be able provide such setup? Any >>> critique in this design? I'm thinking having a 2 channel RAID controller to >>> seperate the PGDATA, root and pg_xlog. >>> >>> Sincerely, >>> Kenji >>> >>> ---------------------------(end of broadcast)--------------------------- >>> TIP 2: Don't 'kill -9' the postmaster >>> >
Thanks Arjen for your reply, this is definitely something to consider. I think in our case, we are not too concerned with the tech image as much as if the machine will allow us to scale the loads we need. I'm not sure if we should worry so much about the IO bandwidth as we are not even close to saturating 320MB/s. I think stability, reliability, and ease-of-use and recovery is our main concern at the moment. I currently am runing a load average of about .5 on a dual Xeon 3.06Ghz P4 setup. How much CPU performance improvement do you think the new woodcrest cpus are over these? -Kenji On Fri, Aug 18, 2006 at 09:41:55PM +0200, Arjen van der Meijden wrote: > Hi Kenji, > > I'm not sure what you mean by 'something newer'? The intel > woodcrest-cpu's are brand-new compared to the amd opterons. But if you > need a 4-cpu config (I take it you want 8-cores in that case), Dell > doesn't offer much. Whether something new will come, I don't know. I'm > not sure when (or if?) a MP-Woodcrest will arrive and/or when Dell will > start offering Opteron-servers. > > Sas has been designed as the successor to SCSI. > > As I see it, SAS has currently one major disadvantage. Lots of new > servers are equipped with SAS-drives, a few nice SAS-raidcontrollers > exist, but the availability of external enclosures for SAS is not > widespread yet. So your options of going beyond (say) 8 disks per system > are a bit limited. > > There are of course advantages as well. The bus is much wider (you can > have 4 lanes of 3Gbps each to an enclosure). You can mix sas and sata > disks, so you could have two arrays in the same enclosure, one big > storage bin and a very fast array or just use only sata disks on a sas > controller. The cabling itself is also much simpler/more flexible > (although using a hot-plug enclosure of course shields you mostly from > that). > But whether its the right choice to make now? I'm not sure. We weren't > to fond of investing a lot of money in an end-of-life system. And since > we're a tech-website, we also had to worry about our "being modern > image", of course ;) > > The main disadvantage I see in this case is, as said, the limited > availability of external enclosures in comparison to SCSI and Fibre > Channel. HP currently only offers their MSA50 (for the rather expensive > SFF disks) while their MSA60 (normal disks) will not be available until > somewhere in 2007 and Dell also only offers one enclosure, the MD1000. > The other big players offer nothing yet, as far as I know, while they > normally offer several SCSI and/or FC-enclosures. > There are also some third-party enclosures (adaptec and promise for > instance) available of course. > > Best regards, > > Arjen > > On 18-8-2006 21:07, Kenji Morishige wrote: > >Thanks Arjen, > >I have unlimited rack space if I really need it. Is serial/SAS really the > >better route to go than SCSI these days? I'm so used to ordering SCSI that > >I've been out of the loop with new disk enclosures and disk tech. I been > >trying to price out a HP DL585, but those are considerably more than the > >Dells. Is it worth waiting a few more weeks/months for Dell to release > >something newer? > > > >-Kenji > > > >On Wed, Aug 09, 2006 at 07:35:22AM +0200, Arjen van der Meijden wrote: > >>With such a budget you should easily be able to get something like: > >>- A 1U high-performance server (for instance the Dell 1950 with 2x > >>Woodcrest 5160, 16GB of FB-Dimm memory, one 5i and one 5e perc raid > >>controller and some disks internally) > >>- An external SAS direct attached disks storage enclosure full with 15k > >>rpm 36GB disks (for instance the MD1000, with 15x 36GB 15k disks) > >> > >>Going for the dell-solution would set you back "only" (including > >>savings) about $13-$14k. HP offers a similar solutions (a HP DL360G5 or > >>a DL380G5/DL385 with two MSA50's for instance) which also fit in your > >>budget afaik. The other players tend to be (a bit) more expensive, force > >>you to go with Fibre Channel or "ancient" SCSI external storage ;) > >> > >>If you'd like to have a product by a generic vendor, have a look at the > >>Adaptec JS50 SAS Jbod enclosure or Promise's Vtrak 300 (both offer 12 > >>sas/sata bays in 2U) for storage. > >> > >>If you're limited to only 2U of rack space, its a bit more difficult to > >>get maximum I/O in your budget (you have basically space for about 8 or > >>12 3.5" disks (with generic suppliers) or 16 2.5" sff disks (with HP)). > >>But you should still be able to have two top-off-the-line x86 cpu's (amd > >>opteron 285 or intel woorcrest 5160) and 16GB of memory (even FB Dimm, > >>which is pretty expensive). > >> > >>Best regards, > >> > >>Arjen van der Meijden > >> > >> > >>On 8-8-2006 22:43, Kenji Morishige wrote: > >>>I've asked for some help here a few months ago and got some really > >>>helpfull > >>>answers regarding RAID controllers and server configuration. Up until > >>>recently I've been running PostgreSQL on a two year old Dual Xeon 3.06Ghz > >>>machine with a single channel RAID controller (previously Adaptec 2200S, > >>>but > >>>now changed to LSI MegaRAID). The 2U unit is from a generic vendor using > >>>what > >>>I believe is a SuperMicro motherboard. In the last week after upgrading > >>>the > >>>RAID controller, the machine has had disk failure and some other issues. > >>>I > >>>would like to build a very reliable dedicated postgreSQL server that has > >>>the > >>>ultimate possible performance and reliabily for around $20,000. The > >>>data set > >>>size is only currently about 4GB, but is increasing by approximately 50MB > >>>daily. The server also requires about 500 connections and I have been > >>>monitoring about 100-200 queries per second at the moment. I am > >>>planning to > >>>run FreeBSD 6.1 if possible, but I am open to any other suggestions if it > >>>improves performance. > >>> > >>>I am considering a setup such as this: > >>> - At least dual cpu (possibly with 2 cores each) > >>> - 4GB of RAM > >>> - 2 disk RAID 1 array for root disk > >>> - 4 disk RAID 1+0 array for PGDATA > >>> - 2 disk RAID 1 array for pg_xlog > >>> > >>>Does anyone know a vendor that might be able provide such setup? Any > >>>critique in this design? I'm thinking having a 2 channel RAID controller > >>>to > >>>seperate the PGDATA, root and pg_xlog. > >>> > >>>Sincerely, > >>>Kenji > >>> > >>>---------------------------(end of broadcast)--------------------------- > >>>TIP 2: Don't 'kill -9' the postmaster > >>> > >
We've been doing some research in this area (the new Woodcrest from Intel, the Opterons from Dell, and SAS). In a nutshell, here's what I'm aware of: Dell does provide a 15 disk external SAS enclosure- the performance numbers they claim look pretty good (of course, go figure) and as far as I can tell, the Perc5/I (the new SAS controller) actually has reasonable performance. I've been playing around with a 2950 with 6x300 GB 10k RPM SAS drives, but no enclosure yet. You can also apparently daisy chain up to 3 enclosures and use multiple perc cards. Dell originally was planning to only support 4 socket opteron boxes, but now they have also apparently decided to support 2 socket ones also. According to this article, they're saying before end of the year. http://www.eweek.com/article2/0,1895,2004886,00.asp Some say that the Woodcrest performs just as well, if not better than the opteron, but I have been unable to do specific tests as of yet. If anyone has a comparable Opteron box (to a PE2950 2x3.0 8 GB RAM Woodcrest), I'd be happy to run some benchmarks. Lastly, Sun just came out with their new X4600. 48 drives, 24 TB storage, 4 U rack space. http://www.sun.com/servers/x64/x4500/getit.jsp That's over your $20k limit though, but looks like it'd be a great DB box. For $20k with dell, you could probably get a 2 CPU 2950, with an external drive cage and 15 SAS drives (just large/med business pricing on their website). I know I would be very curious about the performance of this setup if anyone got their hands on it. We're a Dell shop, so it looks like we'll be settling in on the 2950 Woodcrest for a while, but I have managed to get some people interested in the Sun box and the 4-way opteron from Dell if the need for more performance should arise. HTH, Bucky -----Original Message----- From: pgsql-performance-owner@postgresql.org [mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Arjen van der Meijden Sent: Friday, August 18, 2006 3:42 PM To: Kenji Morishige Cc: pgsql-performance@postgresql.org Subject: Re: [PERFORM] most bang for buck with ~ $20,000 Hi Kenji, I'm not sure what you mean by 'something newer'? The intel woodcrest-cpu's are brand-new compared to the amd opterons. But if you need a 4-cpu config (I take it you want 8-cores in that case), Dell doesn't offer much. Whether something new will come, I don't know. I'm not sure when (or if?) a MP-Woodcrest will arrive and/or when Dell will start offering Opteron-servers. Sas has been designed as the successor to SCSI. As I see it, SAS has currently one major disadvantage. Lots of new servers are equipped with SAS-drives, a few nice SAS-raidcontrollers exist, but the availability of external enclosures for SAS is not widespread yet. So your options of going beyond (say) 8 disks per system are a bit limited. There are of course advantages as well. The bus is much wider (you can have 4 lanes of 3Gbps each to an enclosure). You can mix sas and sata disks, so you could have two arrays in the same enclosure, one big storage bin and a very fast array or just use only sata disks on a sas controller. The cabling itself is also much simpler/more flexible (although using a hot-plug enclosure of course shields you mostly from that). But whether its the right choice to make now? I'm not sure. We weren't to fond of investing a lot of money in an end-of-life system. And since we're a tech-website, we also had to worry about our "being modern image", of course ;) The main disadvantage I see in this case is, as said, the limited availability of external enclosures in comparison to SCSI and Fibre Channel. HP currently only offers their MSA50 (for the rather expensive SFF disks) while their MSA60 (normal disks) will not be available until somewhere in 2007 and Dell also only offers one enclosure, the MD1000. The other big players offer nothing yet, as far as I know, while they normally offer several SCSI and/or FC-enclosures. There are also some third-party enclosures (adaptec and promise for instance) available of course. Best regards, Arjen On 18-8-2006 21:07, Kenji Morishige wrote: > Thanks Arjen, > I have unlimited rack space if I really need it. Is serial/SAS really the > better route to go than SCSI these days? I'm so used to ordering SCSI that > I've been out of the loop with new disk enclosures and disk tech. I been > trying to price out a HP DL585, but those are considerably more than the > Dells. Is it worth waiting a few more weeks/months for Dell to release > something newer? > > -Kenji > > On Wed, Aug 09, 2006 at 07:35:22AM +0200, Arjen van der Meijden wrote: >> With such a budget you should easily be able to get something like: >> - A 1U high-performance server (for instance the Dell 1950 with 2x >> Woodcrest 5160, 16GB of FB-Dimm memory, one 5i and one 5e perc raid >> controller and some disks internally) >> - An external SAS direct attached disks storage enclosure full with 15k >> rpm 36GB disks (for instance the MD1000, with 15x 36GB 15k disks) >> >> Going for the dell-solution would set you back "only" (including >> savings) about $13-$14k. HP offers a similar solutions (a HP DL360G5 or >> a DL380G5/DL385 with two MSA50's for instance) which also fit in your >> budget afaik. The other players tend to be (a bit) more expensive, force >> you to go with Fibre Channel or "ancient" SCSI external storage ;) >> >> If you'd like to have a product by a generic vendor, have a look at the >> Adaptec JS50 SAS Jbod enclosure or Promise's Vtrak 300 (both offer 12 >> sas/sata bays in 2U) for storage. >> >> If you're limited to only 2U of rack space, its a bit more difficult to >> get maximum I/O in your budget (you have basically space for about 8 or >> 12 3.5" disks (with generic suppliers) or 16 2.5" sff disks (with HP)). >> But you should still be able to have two top-off-the-line x86 cpu's (amd >> opteron 285 or intel woorcrest 5160) and 16GB of memory (even FB Dimm, >> which is pretty expensive). >> >> Best regards, >> >> Arjen van der Meijden >> >> >> On 8-8-2006 22:43, Kenji Morishige wrote: >>> I've asked for some help here a few months ago and got some really helpfull >>> answers regarding RAID controllers and server configuration. Up until >>> recently I've been running PostgreSQL on a two year old Dual Xeon 3.06Ghz >>> machine with a single channel RAID controller (previously Adaptec 2200S, >>> but >>> now changed to LSI MegaRAID). The 2U unit is from a generic vendor using >>> what >>> I believe is a SuperMicro motherboard. In the last week after upgrading >>> the >>> RAID controller, the machine has had disk failure and some other issues. I >>> would like to build a very reliable dedicated postgreSQL server that has >>> the >>> ultimate possible performance and reliabily for around $20,000. The data >>> set >>> size is only currently about 4GB, but is increasing by approximately 50MB >>> daily. The server also requires about 500 connections and I have been >>> monitoring about 100-200 queries per second at the moment. I am planning >>> to >>> run FreeBSD 6.1 if possible, but I am open to any other suggestions if it >>> improves performance. >>> >>> I am considering a setup such as this: >>> - At least dual cpu (possibly with 2 cores each) >>> - 4GB of RAM >>> - 2 disk RAID 1 array for root disk >>> - 4 disk RAID 1+0 array for PGDATA >>> - 2 disk RAID 1 array for pg_xlog >>> >>> Does anyone know a vendor that might be able provide such setup? Any >>> critique in this design? I'm thinking having a 2 channel RAID controller to >>> seperate the PGDATA, root and pg_xlog. >>> >>> Sincerely, >>> Kenji >>> >>> ---------------------------(end of broadcast)--------------------------- >>> TIP 2: Don't 'kill -9' the postmaster >>> > ---------------------------(end of broadcast)--------------------------- TIP 5: don't forget to increase your free space map settings
Well, that's of course really hard to tell. From personal experience in a read-mostly environment, the subtop woodcrest 5150 (2.6Ghz) outperforms the top dempsey 5080 (3.7Ghz, in the same system) by quite a nice margin. But that dempsey already has the faster FB-Dimm memory and a much wider FSB compared to your 3.06Ghz Xeons. But if we assume that the 3.7Ghz 5080 is just the extra mhz faster (~ 25%), for a single (dual core) 3Ghz Woodcrest you might already be talking about a 50% improvement in terms of cpu-power over your current set-up. Of course depending on workload and its scalability etc etc. In a perfect world, with linear scalability (note, a read-mostly postgresql can actually do that on a Sun fire T2000 with solaris) that would yield a 200% improvement when going form 2 to 4 cores. A 70-80% scaling is more reasonable and would still imply you'd improve more than 150% over your current set-up. Please note that this is partially based on internal testing and partial on assumptions and would at least require more real-world testing for a app more similar to yours. As soon as we're publishing some numbers on this (and I don't forget), I'll let you know on the list. That will include postgresql and recent x86 cpu's on linux and should be ready soon. Best regards, Arjen On 18-8-2006 21:51, Kenji Morishige wrote: > Thanks Arjen for your reply, this is definitely something to consider. I > think in our case, we are not too concerned with the tech image as much as if > the machine will allow us to scale the loads we need. I'm not sure if we > should worry so much about the IO bandwidth as we are not even close to > saturating 320MB/s. I think stability, reliability, and ease-of-use and > recovery is our main concern at the moment. I currently am runing a load > average of about .5 on a dual Xeon 3.06Ghz P4 setup. How much CPU > performance improvement do you think the new woodcrest cpus are over these? > > -Kenji > > On Fri, Aug 18, 2006 at 09:41:55PM +0200, Arjen van der Meijden wrote: >> Hi Kenji, >> >> I'm not sure what you mean by 'something newer'? The intel >> woodcrest-cpu's are brand-new compared to the amd opterons. But if you >> need a 4-cpu config (I take it you want 8-cores in that case), Dell >> doesn't offer much. Whether something new will come, I don't know. I'm >> not sure when (or if?) a MP-Woodcrest will arrive and/or when Dell will >> start offering Opteron-servers. >> >> Sas has been designed as the successor to SCSI. >> >> As I see it, SAS has currently one major disadvantage. Lots of new >> servers are equipped with SAS-drives, a few nice SAS-raidcontrollers >> exist, but the availability of external enclosures for SAS is not >> widespread yet. So your options of going beyond (say) 8 disks per system >> are a bit limited. >> >> There are of course advantages as well. The bus is much wider (you can >> have 4 lanes of 3Gbps each to an enclosure). You can mix sas and sata >> disks, so you could have two arrays in the same enclosure, one big >> storage bin and a very fast array or just use only sata disks on a sas >> controller. The cabling itself is also much simpler/more flexible >> (although using a hot-plug enclosure of course shields you mostly from >> that). >> But whether its the right choice to make now? I'm not sure. We weren't >> to fond of investing a lot of money in an end-of-life system. And since >> we're a tech-website, we also had to worry about our "being modern >> image", of course ;) >> >> The main disadvantage I see in this case is, as said, the limited >> availability of external enclosures in comparison to SCSI and Fibre >> Channel. HP currently only offers their MSA50 (for the rather expensive >> SFF disks) while their MSA60 (normal disks) will not be available until >> somewhere in 2007 and Dell also only offers one enclosure, the MD1000. >> The other big players offer nothing yet, as far as I know, while they >> normally offer several SCSI and/or FC-enclosures. >> There are also some third-party enclosures (adaptec and promise for >> instance) available of course. >> >> Best regards, >> >> Arjen >> >> On 18-8-2006 21:07, Kenji Morishige wrote: >>> Thanks Arjen, >>> I have unlimited rack space if I really need it. Is serial/SAS really the >>> better route to go than SCSI these days? I'm so used to ordering SCSI that >>> I've been out of the loop with new disk enclosures and disk tech. I been >>> trying to price out a HP DL585, but those are considerably more than the >>> Dells. Is it worth waiting a few more weeks/months for Dell to release >>> something newer? >>> >>> -Kenji >>> >>> On Wed, Aug 09, 2006 at 07:35:22AM +0200, Arjen van der Meijden wrote: >>>> With such a budget you should easily be able to get something like: >>>> - A 1U high-performance server (for instance the Dell 1950 with 2x >>>> Woodcrest 5160, 16GB of FB-Dimm memory, one 5i and one 5e perc raid >>>> controller and some disks internally) >>>> - An external SAS direct attached disks storage enclosure full with 15k >>>> rpm 36GB disks (for instance the MD1000, with 15x 36GB 15k disks) >>>> >>>> Going for the dell-solution would set you back "only" (including >>>> savings) about $13-$14k. HP offers a similar solutions (a HP DL360G5 or >>>> a DL380G5/DL385 with two MSA50's for instance) which also fit in your >>>> budget afaik. The other players tend to be (a bit) more expensive, force >>>> you to go with Fibre Channel or "ancient" SCSI external storage ;) >>>> >>>> If you'd like to have a product by a generic vendor, have a look at the >>>> Adaptec JS50 SAS Jbod enclosure or Promise's Vtrak 300 (both offer 12 >>>> sas/sata bays in 2U) for storage. >>>> >>>> If you're limited to only 2U of rack space, its a bit more difficult to >>>> get maximum I/O in your budget (you have basically space for about 8 or >>>> 12 3.5" disks (with generic suppliers) or 16 2.5" sff disks (with HP)). >>>> But you should still be able to have two top-off-the-line x86 cpu's (amd >>>> opteron 285 or intel woorcrest 5160) and 16GB of memory (even FB Dimm, >>>> which is pretty expensive). >>>> >>>> Best regards, >>>> >>>> Arjen van der Meijden >>>> >>>> >>>> On 8-8-2006 22:43, Kenji Morishige wrote: >>>>> I've asked for some help here a few months ago and got some really >>>>> helpfull >>>>> answers regarding RAID controllers and server configuration. Up until >>>>> recently I've been running PostgreSQL on a two year old Dual Xeon 3.06Ghz >>>>> machine with a single channel RAID controller (previously Adaptec 2200S, >>>>> but >>>>> now changed to LSI MegaRAID). The 2U unit is from a generic vendor using >>>>> what >>>>> I believe is a SuperMicro motherboard. In the last week after upgrading >>>>> the >>>>> RAID controller, the machine has had disk failure and some other issues. >>>>> I >>>>> would like to build a very reliable dedicated postgreSQL server that has >>>>> the >>>>> ultimate possible performance and reliabily for around $20,000. The >>>>> data set >>>>> size is only currently about 4GB, but is increasing by approximately 50MB >>>>> daily. The server also requires about 500 connections and I have been >>>>> monitoring about 100-200 queries per second at the moment. I am >>>>> planning to >>>>> run FreeBSD 6.1 if possible, but I am open to any other suggestions if it >>>>> improves performance. >>>>> >>>>> I am considering a setup such as this: >>>>> - At least dual cpu (possibly with 2 cores each) >>>>> - 4GB of RAM >>>>> - 2 disk RAID 1 array for root disk >>>>> - 4 disk RAID 1+0 array for PGDATA >>>>> - 2 disk RAID 1 array for pg_xlog >>>>> >>>>> Does anyone know a vendor that might be able provide such setup? Any >>>>> critique in this design? I'm thinking having a 2 channel RAID controller >>>>> to >>>>> seperate the PGDATA, root and pg_xlog. >>>>> >>>>> Sincerely, >>>>> Kenji >>>>> >>>>> ---------------------------(end of broadcast)--------------------------- >>>>> TIP 2: Don't 'kill -9' the postmaster >>>>> > > ---------------------------(end of broadcast)--------------------------- > TIP 4: Have you searched our list archives? > > http://archives.postgresql.org >