Thread: serveRAID M5014 SAS
Does anyone here have any bad experiences with the RAID card in subject ? This is in an IBM server, with 2.5" 10k drives. But we seem to observe its poor performance in other configurations as well (with different drives, different settings) in comparison with - say, what dell provides. Any experiences ? -- GJ
Grzegorz Jaśkiewicz wrote: > Does anyone here have any bad experiences with the RAID card in subject ? > This is in an IBM server, with 2.5" 10k drives. > > But we seem to observe its poor performance in other configurations as > well (with different drives, different settings) in comparison with - > say, what dell provides. > Older ServeRAID cards have never been reported as very fast. They were an OK controller if you just want to mirror a pair of drives or something simple like that. Their performance on larger RAID arrays is terrible compared to the LSI products that Dell uses. However, the M5014 *is* an LSI derived product, with a proper battery-backed write cache and everything. I'm not sure if older views are even useful now. Few things to check: -Is the battery working, and the write cache set in write-back mode? -Has read-ahead been set usefully? -Did you try to use a more complicated RAID mode than this card can handle? Those are the three easiest ways to trash performance here. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us "PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books
Greg Smith <greg@2ndquadrant.com> wrote: > -Is the battery working, and the write cache set in write-back > mode? My bet is on this point. Our hardware tech says that the difference between an M5014 and an M5015 is that the former takes a maximum of 256MB RAM while the latter takes a maximum of 512MB RAM and that the M5014 ships with *no* RAM by default. He says you have to order the RAM as an extra-cost option on the M5014. -Kevin
On 25/05/11 19:33, Grzegorz Jaśkiewicz wrote: > Does anyone here have any bad experiences with the RAID card in subject ? > This is in an IBM server, with 2.5" 10k drives. > > But we seem to observe its poor performance in other configurations as > well (with different drives, different settings) in comparison with - > say, what dell provides. > > > Interestingly enough, I've been benchmarking a M5015 SAS, with the optional wee cable for enabling the battery backup for the 512MB of cache. With a 6 disk raid 10 + 2 disk raid 1 - with the array settings NORA+DIRECT, and writeback enabled we're seeing quite good pgbench performance (12 cores + 48G ram, Ubuntu 10.04 with xfs): scale 2500 db with 48 clients, 10 minute runs: 2300 tps scale 500 db with 24 clients, 10 minute runs: 6050 tps I did notice that the sequential performance was quite lackluster (using bonnie) - but are not too concerned about that for the use case (could probably fix using blockdev --setra). I'm guessing that even tho your M5014 card comes with less ram (256M I think), if you can enable the battery backup and cache writes it should be quite good. Also I think the amount of ram on the card is upgradable (4G is the max for the M5105 I *think* - can't find the right doc to check this ATM sorry). Cheers Mark
On 26/05/11 10:24, Mark Kirkwood wrote: > > > Also I think the amount of ram on the card is upgradable (4G is the > max for the M5105 I *think* - can't find the right doc to check this > ATM sorry). > > Looking at the (very sparse) product docs, it looks I am mistaken above - and that the cache sizes are 256M for M5014 and 512M for M5015 and are not upgradable beyond that. Looking at Kevin's post, I recommend checking if you ordered the cache and battery with your card. Cheers Mark
The card is configured in 1+0 . with 128k stripe afaik (I'm a developer, we don't have hardware guys here). Are you's sure about the lack of cache by default on the card ? I thought the difference is that 5104 has 256, and 5105 has 512 ram already on it.
On 26/05/11 20:11, Grzegorz Jaśkiewicz wrote: > The card is configured in 1+0 . with 128k stripe afaik (I'm a > developer, we don't have hardware guys here). > Are you's sure about the lack of cache by default on the card ? I > thought the difference is that 5104 has 256, and 5105 has 512 ram > already on it. No, I'm not sure about what the default is for the M5014 - I'd recommend checking this with your supplier (or looking at the invoice if you can get it). My *feeling* is that you may have 256M cache but no battery kit - as this is an optional part - so the the card will not got into writeback mode if that is the case. FWIW - we got our best (pgbench) results with 256K stripe, No (card) readahead and hyperthreading off on the host. You can interrogate the config of the card and the raid 10 array using the megaraid cli package - you need to read the (frankly terrible) manual to discover which switches to use to determine battery and cache status etc. If you email me privately I'll get you a link to the relevant docs! Cheers Mark
Would HT have any impact to the I/O performance (postgresql, and fs in general) ?.
On 26/05/11 20:31, Grzegorz Jaśkiewicz wrote: > Would HT have any impact to the I/O performance (postgresql, and fs in > general) ?. > There have been previous discussions on this list about HT on vs off (I can't recall what the consensus, if any about what the cause of any performance difference was). In our case HT off gave us much better results for what we think the typical number of clients will be - see attached (server learn-db1 is setup with trivial hardware raid and then software raided via md, learn-db2 has its raid all in hardware. We've ended up going with the latter setup). Note that the highest tps on the graph is about 2100 - we got this upto just over 2300 by changing from ext4 to xfs in later tests, and managed to push the tps for 100 clients up a little by setting no read ahead (NORA) for the arrays. Cheers Mark
Attachment
Mark Kirkwood wrote: > You can interrogate the config of the card and the raid 10 array using > the megaraid cli package - you need to read the (frankly terrible) > manual to discover which switches to use to determine battery and > cache status etc. If you email me privately I'll get you a link to the > relevant docs! That's assuming the MegaCli utility will work against IBM's version of the card. They use an LSI chipset for the RAID parts, but I don't know if the card is so similar that it will talk using that utility or not. The main useful site here is http://tools.rapidsoft.de/perc/perc-cheat-sheet.html ; here's how to dump all the main config info from an LSI card: MegaCli64 -LDInfo -Lall -aALL You want to see a line like this: Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU For the arrays. And then check the battery like this: MegaCli64 -AdpBbuCmd -aALL -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us "PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books
On 27/05/11 11:19, Greg Smith wrote: > Mark Kirkwood wrote: >> You can interrogate the config of the card and the raid 10 array >> using the megaraid cli package - you need to read the (frankly >> terrible) manual to discover which switches to use to determine >> battery and cache status etc. If you email me privately I'll get you >> a link to the relevant docs! > > That's assuming the MegaCli utility will work against IBM's version of > the card. They use an LSI chipset for the RAID parts, but I don't > know if the card is so similar that it will talk using that utility or > not. It does seem to. Cheers Mark
On 27/05/11 11:22, Mark Kirkwood wrote: > On 27/05/11 11:19, Greg Smith wrote: >> Mark Kirkwood wrote: >>> You can interrogate the config of the card and the raid 10 array >>> using the megaraid cli package - you need to read the (frankly >>> terrible) manual to discover which switches to use to determine >>> battery and cache status etc. If you email me privately I'll get you >>> a link to the relevant docs! >> >> That's assuming the MegaCli utility will work against IBM's version >> of the card. They use an LSI chipset for the RAID parts, but I don't >> know if the card is so similar that it will talk using that utility >> or not. > > It does seem to. > > Cheers > > Mark > e.g checking battery status: root@learn-db2:~# MegaCli64 -AdpBbuCmd -GetBbuStatus -a0 BBU status for Adapter: 0 BatteryType: iBBU Voltage: 4040 mV Current: 0 mA Temperature: 28 C BBU Firmware Status: Charging Status : None Voltage : OK Temperature : OK Learn Cycle Requested : No Learn Cycle Active : No Learn Cycle Status : OK Learn Cycle Timeout : No I2c Errors Detected : No Battery Pack Missing : No Battery Replacement required : No Remaining Capacity Low : No Periodic Learn Required : No Transparent Learn : No Battery state: GasGuageStatus: Fully Discharged : No Fully Charged : Yes Discharging : Yes Initialized : Yes Remaining Time Alarm : No Remaining Capacity Alarm: No Discharge Terminated : No Over Temperature : No Charging Terminated : No Over Charged : No Relative State of Charge: 99 % Charger System State: 49168 Charger System Ctrl: 0 Charging current: 0 mA Absolute state of charge: 99 % Max Error: 2 % Exit Code: 0x00 Reminds me of out from DB2 diag commands (years ago...am ok now).
Mark Kirkwood <mark.kirkwood@catalyst.net.nz> wrote: > Battery Pack Missing : No > Fully Charged : Yes > Initialized : Yes I'm not familiar with that output (I leave that to the hardware guys), but it sure looks like there's a battery there. The one thing I didn't see is whether it's configured for write-through or write-back. You want write-back for good performance. -Kevin
On 28/05/11 02:38, Kevin Grittner wrote: > Mark Kirkwood<mark.kirkwood@catalyst.net.nz> wrote: > >> Battery Pack Missing : No >> Fully Charged : Yes >> Initialized : Yes > > I'm not familiar with that output (I leave that to the hardware > guys), but it sure looks like there's a battery there. The one > thing I didn't see is whether it's configured for write-through or > write-back. You want write-back for good performance. > > Sorry for the confusion Kevin - that's the output for *our* M5015 with a battery - what we need to see is the output for Grzegorz's M5014. Grzegorz - can you get that? Cheers Mark
Yeah, I got it Mark. Unfortunately my current management decided to suspend that investigation for a while, so I can't get any tests done or anything like that. However we found out that another server we have shows similar issues. The same card, slightly different motherboard and completely different disks. The basic issue is around concurrent reads and writes. The card is ok-ish when one process hammers the disks, but as soon as it is multiple ones - it just blows (and I use it as a technical term ;) ). Still, with only one process reading/writing - the raid card sucks big time performance wise.
On 28/05/11 21:42, Grzegorz Jaśkiewicz wrote: > Yeah, I got it Mark. > > Unfortunately my current management decided to suspend that > investigation for a while, so I can't get any tests done or anything > like that. > > However we found out that another server we have shows similar issues. > The same card, slightly different motherboard and completely different > disks. > The basic issue is around concurrent reads and writes. The card is > ok-ish when one process hammers the disks, but as soon as it is > multiple ones - it just blows (and I use it as a technical term ;) ). > Still, with only one process reading/writing - the raid card sucks big > time performance wise. > Sorry, Grzegorz I didn't mean to suggest you were not listening, I meant to ask if you could run the megaraid cli command to see if it showed up a battery! If/when your management decide to let you look at this again, I'll be happy to help if I can! Best wishes Mark