Thread: Optimizing disk throughput on quad Opteron
Hello, I'm working out specs for a new database server to be purchased for our organization. The applications the server will handle are mainly related to network operations (monitoring, logging, statistical/trend reports, etc.). Disk I/O will be especially high with relation to processing network stats. You can find a diagram of my initial spec here: http://img266.imageshack.us/img266/9171/dbserverdiagramuc3.jpg Server will be a HP ProLiant DL585 G2 with four dual-core 2.6GHz processors and 8GB of RAM. I can always throw in more RAM. I'm trying to find the most effective way to maximize disk throughput, as the list archives suggest that it is the choke point in most cases. I separated the storage into multiple arrays on multiple controllers, and plan to have 512MB RAM on each controller using BBWC. The plan is to utilize multiple tablespaces as well as partitioned tables when necessary. (Note that the StorageWorks 4214R enclosure with Ultra3 disks is used because we already have it lying around.) I heard some say that the transaction log should be on it's own array, others say it doesn't hurt to have it on the same array as the OS. Is it really worthwhile to put it on it's own array? Can you guys see any glaring bottlenecks in my layout? Any other suggestions to offer (throw in more controllers, different RAID layout, etc.)? Our budget limit is $50k. Thanks! P.S. I know there was a very similar thread started by Ben Suffolk recently, I'd still like to have your "eyes of experience" look at my proposed layout :-) __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
> I heard some say that the transaction log should be on > it's own array, others say it doesn't hurt to have it > on the same array as the OS. Is it really worthwhile > to put it on it's own array? > > Can you guys see any glaring bottlenecks in my layout? > Any other suggestions to offer (throw in more > controllers, different RAID layout, etc.)? Our budget > limit is $50k. You should easily be able to fit in 50k since you already have the storage device. I would suggest the following: 1. Throw in as much RAM as you can. 2. Yes put the transaction logs on a separate array. There are a couple of reasons for this: 1. transaction logs are written sequentially so a RAID 1 is enough 2. You don't have to use a journaled fs for the transaction logs so it is really fast. 3. IIRC the MSA 30 can take 14 drives. Make sure you put in all 14 drives and delegate two of them to hot spare duty. I actually wonder if you would be better off putting the indexes on tablespace A and use your core data set on the larger storage works array... Sincerely, Joshua D. Drake Sincerely, Joshua D. Drake > > Thanks! > > P.S. I know there was a very similar thread started by > Ben Suffolk recently, I'd still like to have your > "eyes of experience" look at my proposed layout :-) > > > > > > > > > > > __________________________________________________ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection around > http://mail.yahoo.com > > ---------------------------(end of broadcast)--------------------------- > TIP 2: Don't 'kill -9' the postmaster > -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240 Providing the most comprehensive PostgreSQL solutions since 1997 http://www.commandprompt.com/ Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
> You can find a diagram of my initial > spec here: > http://img266.imageshack.us/img266/9171/dbserverdiagramuc3.jpg > > Can you guys see any glaring bottlenecks in my layout? > Any other suggestions to offer (throw in more > controllers, different RAID layout, etc.)? Our budget > limit is $50k. The thing I would ask is would you not be better with SAS drives? Since the comments on Dell, and the highlighted issues I have been looking at HP and the the Smart Array P600 controller with 512 BBWC. Although I am looking to stick with the 8 internal disks, rather than use external ones. The HP Smart Array 50 is the external array for SAS drives. Not really looked into it much though. Regards Ben
> The thing I would ask is would you not be better > with SAS drives? > > Since the comments on Dell, and the highlighted > issues I have been > looking at HP and the the Smart Array P600 > controller with 512 BBWC. > Although I am looking to stick with the 8 internal > disks, rather than > use external ones. > > The HP Smart Array 50 is the external array for SAS > drives. Not > really looked into it much though. Ben, The Smart Array 50 supports a maximum of 10 disks and has a single I/O module, while the Smart Array 30 supports up to 14 disks and can be configured with a dual I/O module. I was under the assumption that SAS runs at the same speed as Ultra320, in which case the Smart Array 30 is a better bet... Thanks for your feedback. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
On Oct 21, 2006, at 11:43 AM, John Philips wrote: > Can you guys see any glaring bottlenecks in my layout? > Any other suggestions to offer (throw in more > controllers, different RAID layout, etc.)? Our budget > limit is $50k. If I had $50k budget, I'd be buying the SunFire X4500 and running Solaris + ZFS on it. However, you're limited to 2 dual core Opterons, it seems.
Attachment
Vivek Khera wrote:
>
> On Oct 21, 2006, at 11:43 AM, John Philips wrote:
>
>> Can you guys see any glaring bottlenecks in my layout?
>> Any other suggestions to offer (throw in more
>> controllers, different RAID layout, etc.)? Our budget
>> limit is $50k.
>
> If I had $50k budget, I'd be buying the SunFire X4500 and running
> Solaris + ZFS on it. However, you're limited to 2 dual core Opterons,
> it seems.
The HP 585 will give you quad dual core :)
Sincerely,
Joshua D. Drake
>
>
--
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
On Oct 23, 2006, at 4:59 PM, Joshua D. Drake wrote: >> If I had $50k budget, I'd be buying the SunFire X4500 and running >> Solaris + ZFS on it. However, you're limited to 2 dual core >> Opterons, >> it seems. > > The HP 585 will give you quad dual core :) but can you sling the bits to and from the disk as fast as the X4500? the speed numbers on the i/o of the x4500 are mind-numbing.
Attachment
Vivek Khera wrote:
>
> On Oct 23, 2006, at 4:59 PM, Joshua D. Drake wrote:
>
>>> If I had $50k budget, I'd be buying the SunFire X4500 and running
>>> Solaris + ZFS on it. However, you're limited to 2 dual core Opterons,
>>> it seems.
>>
>> The HP 585 will give you quad dual core :)
>
> but can you sling the bits to and from the disk as fast as the X4500?
> the speed numbers on the i/o of the x4500 are mind-numbing.
Honestly, I don't know. I can tell you that they perform VERY well for
us (as do the 385s). I have deployed about several 385s and 585s with
the MSA30s over the last 6 months and have not been disappointed yet.
Sincerely,
Joshua D. Drake
--
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
On Sat, Oct 21, 2006 at 08:43:05AM -0700, John Philips wrote: > I heard some say that the transaction log should be on > it's own array, others say it doesn't hurt to have it > on the same array as the OS. Is it really worthwhile > to put it on it's own array? It all depends on the controller and how much non-WAL workload there is. Theoretically, with a good enough controller, you can leave WAL on the same partition as your data. With a complex setup like you're looking at, you really will want to do some testing to see what makes the most sense. I can also point you at a company that does modeling of stuff like this; they could actually give you some idea of how well that setup would perform before you buy the hardware. BTW, any test results you can provide back to the community would be most appreciated! -- Jim Nasby jim@nasby.net EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
> -----Original Message----- > From: pgsql-performance-owner@postgresql.org [mailto:pgsql-performance- > owner@postgresql.org] On Behalf Of John Philips > Sent: Monday, October 23, 2006 8:17 AM > To: Ben Suffolk > Cc: pgsql-performance@postgresql.org > Subject: Re: [PERFORM] Optimizing disk throughput on quad Opteron > > > The thing I would ask is would you not be better > > with SAS drives? > > > > Since the comments on Dell, and the highlighted > > issues I have been > > looking at HP and the the Smart Array P600 > > controller with 512 BBWC. > > Although I am looking to stick with the 8 internal > > disks, rather than > > use external ones. > > > > The HP Smart Array 50 is the external array for SAS > > drives. Not > > really looked into it much though. > > Ben, > > The Smart Array 50 supports a maximum of 10 disks and > has a single I/O module, while the Smart Array 30 > supports up to 14 disks and can be configured with a > dual I/O module. > > I was under the assumption that SAS runs at the same > speed as Ultra320, in which case the Smart Array 30 is > a better bet... > > Thanks for your feedback. The drives might be about the same speed, but SAS is a completely different bus architecture from SCSI. U320 is a parallel interface limited to 320 MB/s for the total bus (160 MB/s per channel, so be careful here). SAS is a 3.0Gbps direct serial interface to the drive. So, after 5-6 drives, SAS will definitely start to pay off. Take a look at Dell's MD1000 external enclosure vs the previous version. The MD1000 offers much better performance (not saying to go with dell, just giving an example of SCSI vs. SAS from a vendor I'm familiar with). Oh, and if you're not completely against dell, you can daisy chain 3 of the MD1000 enclosures together off one of their new 6850 (Quad Woodcrest) or 6950 (Quad Operton). At the moment, the Woodcrests seem to be outperforming the Opteron in server benchmarks, I have a quad core (dual cpu) 2950 I'd be happy to run some pg_benches (or other preferred benchmark) if someone has a similar opteron so we can get some relevant comparisons on the list. Also, here's a link that was posted a while back on opteron vs. woodcrest: http://tweakers.net/reviews/646 HTH, Bucky