Thread: [OT] RAID controllers blocking one another?
We have a machine that serves as a fileserver and a database server. Our server hosts a raid array of 40 disk drives, attached to two3-ware cards, one 9640SE-24 and one 9640SE-16. We have noticed that activity on one controller blocks access on the second controller, not only for disk-IO but also the command line tools which become unresponsive for the inactive controller. The controllers are sitting in adjacent PCI-express slots on a machine with dual-dual AMD and 16GB of RAM. Has anyone else noticed issues like this? Throughput for either controller is a pretty respectable 150-200MB/s writing and somewhat faster for reading, but the "blocking" is problematic, as the machine is serving multiple purposes.
I know this is off-topic, but I know lots of folks here deal with very large disk arrays; it is hard to get real-world input on machines such as these.
Thanks,
Sean
I know this is off-topic, but I know lots of folks here deal with very large disk arrays; it is hard to get real-world input on machines such as these.
Thanks,
Sean
Sean Davis wrote: > 150-200MB/s writing and somewhat faster for reading That actually seems dead slow. Whatever RAID level you configured, there is no sane way for it to be that slow. Is this a RAID 5/6 array? Did you forgot to align your file system to stripe boundaries? -- Best regards, Hannes Dorbath
Sean Davis wrote: > I know this is off-topic, but I know lots of folks here deal with very large > disk arrays; it is hard to get real-world input on machines such as these. In my experience the 3ware support staff is competent and replies within one business day. Just sign up at: https://www.3ware.com/userregistration.asp?uType=eUser&boolCompany=1 and place a ticket. Though let us know when you got that solved please. -- Best regards, Hannes Dorbath
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Fri, 18 Jan 2008 16:54:17 +0100 Hannes Dorbath <light@theendofthetunnel.de> wrote: > Sean Davis wrote: > > 150-200MB/s writing and somewhat faster for reading > > That actually seems dead slow. Whatever RAID level you configured, > there is no sane way for it to be that slow. Is this a RAID 5/6 > array? Did you forgot to align your file system to stripe boundaries? That seem that slow for SATA on a write. Assuming 20 (on each controller) disks random write I would guess the best you could pull off would be about 200 Megs a second. That's on a good day. So 200MBs per controllers isn't that out of bounds. I would agree that seems slow for reads though. And to answer the question, no I have not seen the behavior he is experiencing but to be honest I would never run that many sata drives. I would have long pushed to SAS on that scale. Sincerely, Joshua D. Drake - -- The PostgreSQL Company: Since 1997, http://www.commandprompt.com/ Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240 Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate SELECT 'Training', 'Consulting' FROM vendor WHERE name = 'CMD' -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFHkOFvATb/zqfZUUQRAoPLAKCGLze19/Z5VZ/d4TVEQQUC18dTcgCeLVTD Abjvqf686r0eX2K1sxo8giY= =G5t+ -----END PGP SIGNATURE-----
Sean Davis wrote: > We are PURE amateurs, so that is possible. We are running 12-disk RAID6 > (750GB drives--this might be a difference from what you are thinking) > partitions under LVM. We have write-cache enabled and are using ext3 > filesystems. Any suggestions? We use the 9650 controllers in RAID 6 a lot. With 12 discs you should be able to get about 550MB/sec sequential write and 680MB/sec read easily. But it's already pretty clear to me why you can't get anywhere near that: The first thing is that ext3 is a file system designed for small block devices that have a single physical head to seek -- not for large arrays that consist of multiple physical devices that can seek independently. Especially this is true for the anticipatory I/O scheduler that is used in conjunction with ext3. Additionally ext3 does not know any form of stripe alignment which is an absolute requirement for arrays that use a parity based RAID level (RAID 5/6). If you don't stripe align your file system to the stripe width configured in your controller, you will suffer massive I/O trashing. Your controller and discs will be busy with Load-XOR-Store cycles. That means that your discs will do a ton of micro seeks on every write instead of just writing the chunk of data out. Both performance and lifespan of your discs will suffer for no reason. My recommendation is either Linux XFS or Solaris ZFS. Both are designed form the ground up for such setups. If this box is not yet in production and you like to test something I can give you a fitting mkfs.xfs line for your setup -- the defaults won't work well. The last thing to note is that LVM will mess with any kind of stripe alignment, so it's quite useless for parity based arrays. If you need serious volume management look into ZFS. Finally you might need to do tweak some scheduler params, depending if you chose to go with deadline or CFQ. Deadline is recommend by 3ware and usually the best choice for PostgreSQL as well. Speaking of PostgreSQL, put $PGDATA on a RAID 10 array, it will suffer on RAID 5/6. -- Best regards, Hannes Dorbath
2 more things: Get BBUs for the controllers (If you have not already). Upgrade to the latest 3ware firmware (code set 9.4.2). There was data corruption bug with RAID 6. -- Best regards, Hannes Dorbath
Joshua D. Drake wrote: > That seem that slow for SATA on a write. Assuming 20 (on each > controller) disks random write I would guess the best you could pull off > would be about 200 Megs a second. That's on a good day. For random writes/reads that sure is OK. I interpreted those numbers as pure sequential benchmark load. -- Best regards, Hannes Dorbath
Joshua D. Drake wrote: > That seem that slow for SATA on a write. Assuming 20 (on each > controller) disks random write I would guess the best you could pull off > would be about 200 Megs a second. That's on a good day. For random writes/reads that sure is OK. I interpreted those numbers as pure sequential benchmark load. -- Best regards, Hannes Dorbath
Am 2008-01-18 16:54:17, schrieb Hannes Dorbath: > Sean Davis wrote: > >150-200MB/s writing and somewhat faster for reading > > That actually seems dead slow. Whatever RAID level you configured, there > is no sane way for it to be that slow. Is this a RAID 5/6 array? Did you > forgot to align your file system to stripe boundaries? I have had such rates on a 3w85xx with 4+1 WD Raptor WD740 in Raid-5 (the OS was on a Raid-1 with 2+1 WD360) The 3w95xx shoud do more, but with the new Hitachi 1 TByte I do not get more then 140 MBit. It seems, the drives are not so performant. Thanks, Greetings and nice Day Michelle Konzack Systemadministrator Tamay Dogan Network Debian GNU/Linux Consultant -- Linux-User #280138 with the Linux Counter, http://counter.li.org/ ##################### Debian GNU/Linux Consultant ##################### Michelle Konzack Apt. 917 ICQ #328449886 50, rue de Soultz MSN LinuxMichi 0033/6/61925193 67100 Strasbourg/France IRC #Debian (irc.icq.com)