Thread: Postgresql and Software RAID/LVM
I have a small business client that cannot afford high-end/high quality RAID cards for their next server. That's a seperate argument/issue right there for me, but what the client wants is what the client wants. Has anyone ran Postgres with software RAID or LVM on a production box? What have been your experience? I don't forsee more 10-15 concurrent sessions running for an their OLTP application. Thanks. Steve Poe
Steve Poe wrote: > I have a small business client that cannot afford high-end/high quality > RAID cards for their next server. That's a seperate argument/issue right > there for me, but what the client wants is what the client wants. > > Has anyone ran Postgres with software RAID or LVM on a production box? > What have been your experience? I would not run RAID + LVM in a software scenario. Software RAID is fine however. Sincerely, Joshua D. Drake > > I don't forsee more 10-15 concurrent sessions running for an their OLTP > application. > > Thanks. > > Steve Poe > > > ---------------------------(end of broadcast)--------------------------- > TIP 6: Have you searched our list archives? > > http://archives.postgresql.org -- Your PostgreSQL solutions company - Command Prompt, Inc. 1.800.492.2240 PostgreSQL Replication, Consulting, Custom Programming, 24x7 support Managed Services, Shared and Dedicated Hosting Co-Authors: plPHP, plPerlNG - http://www.commandprompt.com/
On Fri, 2005-06-03 at 11:45 -0700, Steve Poe wrote:
> I have a small business client that cannot afford high-end/high quality
> RAID cards for their next server. That's a seperate argument/issue right
> there for me, but what the client wants is what the client wants.
>
> Has anyone ran Postgres with software RAID or LVM on a production box?
> What have been your experience?
Hi,
We regularly run LVM on top of software raid for our PostgreSQL servers
(and our other servers, for that matter).
As far as I can see these systems have not had issues related to either
software RAID or LVM - that's around 30 systems all up, maybe 8 running
PostgreSQL, in production. The database servers are a variety of
dual-Xeon (older) and dual-Opteron (newer) systems.
The Xeons are all running Debian "Woody" with 2.4.xx kernels and the
Opterons are all running "Sarge" with 2.6.x kernels.
Regards,
Andrew.
-------------------------------------------------------------------------
Andrew @ Catalyst .Net .NZ Ltd, PO Box 11-053, Manners St, Wellington
WEB: http://catalyst.net.nz/ PHYS: Level 2, 150-154 Willis St
DDI: +64(4)803-2201 MOB: +64(272)DEBIAN OFFICE: +64(4)499-2267
The secret of being a bore is to say everything -- Voltaire
-------------------------------------------------------------------------
Attachment
> Has anyone ran Postgres with software RAID or LVM on a production box? > What have been your experience? Yes, we have run for a couple years Pg with software LVM (mirroring) against two hardware RAID5 arrays. We host a production Sun box that runs 24/7. My experience: * Software RAID (other than mirroring) is a disaster waiting to happen. If the metadata for the RAID set gives out for any reason (CMOS scrambles, card dies, power spike, etc.) then you are hosed beyond belief. In most cases it is almost impossible to recover. With mirroring, however, you can always boot and operate on a single mirror, pretending that no LVM/RAID is underway. In other words, each mirror is a fully functional copy of the data which will operate your server. * Hardware RAID5 is a terrific way to boost performance via write caching and spreading I/O across multiple spindles. Each of our external arrays operates 14 drives (12 data, 1 parity and 1 hot spare). While RAID5 protects against single spindle failure, it will not hedge against multiple failures in a short time period, SCSI contoller failure, SCSI cable problems or even wholesale failure of the RAID controller. All of these things happen in a 24/7 operation. Using software RAID1 against the hardware RAID5 arrays hedges against any single failure. * Software mirroring gives you tremendous ability to change the system while it is running, by taking offline the mirror you wish to change and then synchronizing it after the change. On a fully operational production server, we have: * restriped the RAID5 array * replaced all RAID5 media with higher capacity drives * upgraded RAID5 controller * moved all data from an old RAID5 array to a newer one * replaced host SCSI controller * uncabled and physically moved storage to a different part of data center Again, all of this has taken place (over the years) while our machine was fully operational.
Marty Scholes wrote: >> Has anyone ran Postgres with software RAID or LVM on a production box? >> What have been your experience? > > Yes, we have run for a couple years Pg with software LVM (mirroring) > against two hardware RAID5 arrays. We host a production Sun box that > runs 24/7. > > My experience: > * Software RAID (other than mirroring) is a disaster waiting to happen. > If the metadata for the RAID set gives out for any reason (CMOS > scrambles, card dies, power spike, etc.) then you are hosed beyond > belief. In most cases it is almost impossible to recover. With > mirroring, however, you can always boot and operate on a single mirror, > pretending that no LVM/RAID is underway. In other words, each mirror is > a fully functional copy of the data which will operate your server. Isn't this actually more of a problem for the meta-data to give out in a hardware situation? I mean, if the card you are using dies, you can't just get another one. With software raid, because the meta-data is on the drives, you can pull it out of that machine, and put it into any machine that has a controller which can read the drives, and a similar kernel, and you are back up and running. > > * Hardware RAID5 is a terrific way to boost performance via write > caching and spreading I/O across multiple spindles. Each of our > external arrays operates 14 drives (12 data, 1 parity and 1 hot spare). > While RAID5 protects against single spindle failure, it will not hedge > against multiple failures in a short time period, SCSI contoller > failure, SCSI cable problems or even wholesale failure of the RAID > controller. All of these things happen in a 24/7 operation. Using > software RAID1 against the hardware RAID5 arrays hedges against any > single failure. No, it hedges against *more* than one failure. But you can also do a RAID1 over a RAID5 in software. But if you are honestly willing to create a full RAID1, just create a RAID1 over RAID0. The performance is much better. And since you have a full RAID1, as long as both drives of a pairing don't give out, you can lose half of your drives. If you want the space, but you feel that RAID5 isn't redundant enough, go to RAID6, which uses 2 parity locations, each with a different method of storing parity, so not only is it more redundant, you have a better chance of finding problems. > > * Software mirroring gives you tremendous ability to change the system > while it is running, by taking offline the mirror you wish to change and > then synchronizing it after the change. > That certainly is a nice ability. But remember that LVM also has the idea of "snapshot"ing a running system. I don't know the exact details, just that there is a way to have some processes see the filesystem as it existed at an exact point in time. Which is also a great way to handle backups. > On a fully operational production server, we have: > * restriped the RAID5 array > * replaced all RAID5 media with higher capacity drives > * upgraded RAID5 controller > * moved all data from an old RAID5 array to a newer one > * replaced host SCSI controller > * uncabled and physically moved storage to a different part of data center > > Again, all of this has taken place (over the years) while our machine > was fully operational. > So you are saying that you were able to replace the RAID controller without turning off the machine? I realize there does exist hot-swappable PCI cards, but I think you are overstating what you mean by "fully operational". For instance, it's not like you can access your data while it is being physically moved. I do think you had some nice hardware. But I know you can do all of this in software as well. It is usually a price/performance tradeoff. You spend quite a bit to get a hardware RAID card that can keep up with a modern CPU. I know we have an FC raid box at work which has a full 512MB of cache on it, but it wasn't that much cheaper than buying a dedicated server. John =:->
Attachment
John A Meinel wrote: > > Isn't this actually more of a problem for the meta-data to give out in a > hardware situation? I mean, if the card you are using dies, you can't > just get another one. > With software raid, because the meta-data is on the drives, you can pull > it out of that machine, and put it into any machine that has a > controller which can read the drives, and a similar kernel, and you are > back up and running. Probably true. If you have a similar kernel and hardware and if you can recover the state information, knowing where the state information is stored. Those are some very big "ifs" during a hectic disaster. > No, it hedges against *more* than one failure. But you can also do a > RAID1 over a RAID5 in software. But if you are honestly willing to > create a full RAID1, just create a RAID1 over RAID0. The performance is > much better. And since you have a full RAID1, as long as both drives of > a pairing don't give out, you can lose half of your drives. True as well. The problem with RAID1 over RAID0 is that, during a drive failure, you are one bad sector from disaster. Further, RAID5 does automatic rebuild, whereas most RAID1 setups do not. RAID5 reduces the amount of time that things are degraded, reducing the time that your data is in danger. > If you want the space, but you feel that RAID5 isn't redundant enough, > go to RAID6, which uses 2 parity locations, each with a different method > of storing parity, so not only is it more redundant, you have a better > chance of finding problems. Agreed, RAID6 is the future, but still won't keep the server running when the RAID controller dies, or the SCSI/FC host adapter goes, or you want to upgrade controller firmware, or you want to replace the media, or... > So you are saying that you were able to replace the RAID controller > without turning off the machine? I realize there does exist > hot-swappable PCI cards, but I think you are overstating what you mean > by "fully operational". For instance, it's not like you can access your > data while it is being physically moved. Detach mirror 1, uncable and move, recable and resync. Detach mirror 2, uncable and move, recable and resync. > > I do think you had some nice hardware. But I know you can do all of this > in software as well. It is usually a price/performance tradeoff. You > spend quite a bit to get a hardware RAID card that can keep up with a > modern CPU. I know we have an FC raid box at work which has a full 512MB > of cache on it, but it wasn't that much cheaper than buying a dedicated > server. We run two Nexsan ATABoy2 arrays. These can be found in 1 TB configurations for about $3,000 each, putting mirrored RAID5 storage at $6 per GB. Is that a lot of money for storage? Maybe. In our case, that's dirt cheap protection against storage-related downtime. Marty