Thread: Recommendations for SSDs in production?
Well, After reading several glowing reviews of the new OCZ Vertex3 SSD last spring, we did some performance testing in dev on RHEL6. (CentOS) The results were nothing short of staggering. Complex query results returned in 1/10th the time as a pessimistic measurement. System loads dropped from 2+ to 0.1 or less. Wow. So after months of using this SSD without any issues at all, we tentatively rolled this out to production, and had blissful, sweet beauty until about 2 weeks ago, now we are running into sudden death scenarios. We have excellent backup system, so the damage is reduced to roadbumps, but are looking for a longer term solution that doesn't compromise performance too much. The config is super-basic, basically no tuning at all was done: # fdisk /dev/NNN; mke2fs -j $partn; mount $partn /var/lib/pgsql; rsync -vaz /var/lib/pgsql.old /var/lib/pgsql; service postgresql start; I don't mind spending some money. Can anybody comment on a recommended drive in real world use? After some review I found: 1) Micron P300 SSD: claims excellent numbers, can't find them for sale anywhere. 2) Intel X25E - good reputation, significantly slower than the Vertex3. We're buying some to reduce downtime. 3) OCZ "Enterprise" - reviews are mixed. 4) Kingston "Enterprise" drives appear to be identical to consumer drives with a different box. 5) STEC drives are astronomically expensive. (EG: "You're kidding, right?") 6) Corsair consumer drives getting excellent reviews, Aberdeen Inc recommended in use with RAID 1. 7) Seagate Pulsar drives, XT.2 drives are expensive SLC but can't find a vendor, Pulsar .2 drives are more available but having trouble finding reviews other than rehashed press releases. Thanks! -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
On Wed, Nov 2, 2011 at 12:01 PM, Benjamin Smith <lists@benjamindsmith.com> wrote: > Well, > > After reading several glowing reviews of the new OCZ Vertex3 SSD last spring, > we did some performance testing in dev on RHEL6. (CentOS) > > The results were nothing short of staggering. Complex query results returned > in 1/10th the time as a pessimistic measurement. System loads dropped from 2+ > to 0.1 or less. > > Wow. > > So after months of using this SSD without any issues at all, we tentatively > rolled this out to production, and had blissful, sweet beauty until about 2 > weeks ago, now we are running into sudden death scenarios. We have excellent > backup system, so the damage is reduced to roadbumps, but are looking for a > longer term solution that doesn't compromise performance too much. > > The config is super-basic, basically no tuning at all was done: > > # fdisk /dev/NNN; > mke2fs -j $partn; > mount $partn /var/lib/pgsql; > rsync -vaz /var/lib/pgsql.old /var/lib/pgsql; > service postgresql start; > > I don't mind spending some money. Can anybody comment on a recommended drive > in real world use? > > After some review I found: > > 1) Micron P300 SSD: claims excellent numbers, can't find them for sale > anywhere. > > 2) Intel X25E - good reputation, significantly slower than the Vertex3. We're > buying some to reduce downtime. > > 3) OCZ "Enterprise" - reviews are mixed. > > 4) Kingston "Enterprise" drives appear to be identical to consumer drives with > a different box. > > 5) STEC drives are astronomically expensive. (EG: "You're kidding, right?") > > 6) Corsair consumer drives getting excellent reviews, Aberdeen Inc recommended > in use with RAID 1. > > 7) Seagate Pulsar drives, XT.2 drives are expensive SLC but can't find a > vendor, Pulsar .2 drives are more available but having trouble finding reviews > other than rehashed press releases. > > Thanks! The X25-E is now essentially being replaced by the 710. I would look there. merlin
On 11/2/2011 11:01 AM, Benjamin Smith wrote: > > 2) Intel X25E - good reputation, significantly slower than the Vertex3. We're > buying some to reduce downtime. > If you don't mind spending money, look at the new 710 Series from Intel. Not SLC like the X25E, but still specified with a very high write endurance.
you really need to watch out for excess write caching on SSDs. only a few are safe against power failures while under heavy database write activity. -- john r pierce N 37, W 122 santa cruz ca mid-left coast
Vertex 3 and ocz in general has a very bad reputation in the "enthusiast scene". Sudden issues, hard locks, data loss and so on. Just go and look in the OCZ forums. I would not dare by on Vertex 3 for my desktop...have 2 Intel ones.
I have no idea what you do but just the fact that you bought ssds to improve performance means it's rather high load and hence important. Using a consumer drive for that IMHO is not the best idea. I know a lot about ssds but just in consumer space. Intel has a good reputation in terms of reliability but they are not the fastest. I guess go Intel route or some other crazy expensive enterprise stuff.
Note that consumers drives can lead to data loss in case of power failure (data in cache, no capacitors that would give enough power to flush to nand).
For database I assume random read and writes are by way the most important thing and any recent ssd is orders of magnitude faster in that are compared to HDD even the "slow" Intel drives.
Regards,
Thomas
I have no idea what you do but just the fact that you bought ssds to improve performance means it's rather high load and hence important. Using a consumer drive for that IMHO is not the best idea. I know a lot about ssds but just in consumer space. Intel has a good reputation in terms of reliability but they are not the fastest. I guess go Intel route or some other crazy expensive enterprise stuff.
Note that consumers drives can lead to data loss in case of power failure (data in cache, no capacitors that would give enough power to flush to nand).
For database I assume random read and writes are by way the most important thing and any recent ssd is orders of magnitude faster in that are compared to HDD even the "slow" Intel drives.
Regards,
Thomas
> Date: Wed, 2 Nov 2011 12:18:10 -0500
> Subject: Re: [GENERAL] Recommendations for SSDs in production?
> From: mmoncure@gmail.com
> To: lists@benjamindsmith.com
> CC: pgsql-general@postgresql.org
>
> On Wed, Nov 2, 2011 at 12:01 PM, Benjamin Smith
> <lists@benjamindsmith.com> wrote:
> > Well,
> >
> > After reading several glowing reviews of the new OCZ Vertex3 SSD last spring,
> > we did some performance testing in dev on RHEL6. (CentOS)
> >
> > The results were nothing short of staggering. Complex query results returned
> > in 1/10th the time as a pessimistic measurement. System loads dropped from 2+
> > to 0.1 or less.
> >
> > Wow.
> >
> > So after months of using this SSD without any issues at all, we tentatively
> > rolled this out to production, and had blissful, sweet beauty until about 2
> > weeks ago, now we are running into sudden death scenarios. We have excellent
> > backup system, so the damage is reduced to roadbumps, but are looking for a
> > longer term solution that doesn't compromise performance too much.
> >
> > The config is super-basic, basically no tuning at all was done:
> >
> > # fdisk /dev/NNN;
> > mke2fs -j $partn;
> > mount $partn /var/lib/pgsql;
> > rsync -vaz /var/lib/pgsql.old /var/lib/pgsql;
> > service postgresql start;
> >
> > I don't mind spending some money. Can anybody comment on a recommended drive
> > in real world use?
> >
> > After some review I found:
> >
> > 1) Micron P300 SSD: claims excellent numbers, can't find them for sale
> > anywhere.
> >
> > 2) Intel X25E - good reputation, significantly slower than the Vertex3. We're
> > buying some to reduce downtime.
> >
> > 3) OCZ "Enterprise" - reviews are mixed.
> >
> > 4) Kingston "Enterprise" drives appear to be identical to consumer drives with
> > a different box.
> >
> > 5) STEC drives are astronomically expensive. (EG: "You're kidding, right?")
> >
> > 6) Corsair consumer drives getting excellent reviews, Aberdeen Inc recommended
> > in use with RAID 1.
> >
> > 7) Seagate Pulsar drives, XT.2 drives are expensive SLC but can't find a
> > vendor, Pulsar .2 drives are more available but having trouble finding reviews
> > other than rehashed press releases.
> >
> > Thanks!
>
> The X25-E is now essentially being replaced by the 710. I would look there.
>
> merlin
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
> Subject: Re: [GENERAL] Recommendations for SSDs in production?
> From: mmoncure@gmail.com
> To: lists@benjamindsmith.com
> CC: pgsql-general@postgresql.org
>
> On Wed, Nov 2, 2011 at 12:01 PM, Benjamin Smith
> <lists@benjamindsmith.com> wrote:
> > Well,
> >
> > After reading several glowing reviews of the new OCZ Vertex3 SSD last spring,
> > we did some performance testing in dev on RHEL6. (CentOS)
> >
> > The results were nothing short of staggering. Complex query results returned
> > in 1/10th the time as a pessimistic measurement. System loads dropped from 2+
> > to 0.1 or less.
> >
> > Wow.
> >
> > So after months of using this SSD without any issues at all, we tentatively
> > rolled this out to production, and had blissful, sweet beauty until about 2
> > weeks ago, now we are running into sudden death scenarios. We have excellent
> > backup system, so the damage is reduced to roadbumps, but are looking for a
> > longer term solution that doesn't compromise performance too much.
> >
> > The config is super-basic, basically no tuning at all was done:
> >
> > # fdisk /dev/NNN;
> > mke2fs -j $partn;
> > mount $partn /var/lib/pgsql;
> > rsync -vaz /var/lib/pgsql.old /var/lib/pgsql;
> > service postgresql start;
> >
> > I don't mind spending some money. Can anybody comment on a recommended drive
> > in real world use?
> >
> > After some review I found:
> >
> > 1) Micron P300 SSD: claims excellent numbers, can't find them for sale
> > anywhere.
> >
> > 2) Intel X25E - good reputation, significantly slower than the Vertex3. We're
> > buying some to reduce downtime.
> >
> > 3) OCZ "Enterprise" - reviews are mixed.
> >
> > 4) Kingston "Enterprise" drives appear to be identical to consumer drives with
> > a different box.
> >
> > 5) STEC drives are astronomically expensive. (EG: "You're kidding, right?")
> >
> > 6) Corsair consumer drives getting excellent reviews, Aberdeen Inc recommended
> > in use with RAID 1.
> >
> > 7) Seagate Pulsar drives, XT.2 drives are expensive SLC but can't find a
> > vendor, Pulsar .2 drives are more available but having trouble finding reviews
> > other than rehashed press releases.
> >
> > Thanks!
>
> The X25-E is now essentially being replaced by the 710. I would look there.
>
> merlin
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
On 11/02/11 11:39 AM, Thomas Strunz wrote: > For database I assume random read and writes are by way the most > important thing and any recent ssd is orders of magnitude faster in > that are compared to HDD even the "slow" Intel drives. actually, SSD's have issues with committed small block (8K) random writes such as databases do a lot of. the SSD has a rather large block size that has to be written all at once, so what they tend to do is accumulate random writes in a buffer, then write them all at once to a contiguous block (remapping the logical LBA sector address to an actual block/offset address). as a test at work, I compared a 2 x 100GB SAS enterprise SSD RAID0 with a 20 x 146GB SAS 15k HD RAID10, both raids using a HP p410 hardware raid controller with 1Gb cache, and both using XFS. Both file systems are approximately the same in sustainable random writes from postgres, up around 12000 wr/sec during heavy sustained pgbench activity (scaling factor of 500, 48 clients, 24 threads, on a 12 core 24 thread dual xeon e5660 48gb server). The HD raid is faster at sustained large block writes from iozone (1.2GB/sec vs 800MB/sec for the SSD). of course, the HD raid10 is 1.3TB of data, while the SSD raid0 is 200GB of data. -- john r pierce N 37, W 122 santa cruz ca mid-left coast
On 2011-11-02 18:01, Benjamin Smith wrote: > So after months of using this SSD without any issues at all, we tentatively > rolled this out to production, and had blissful, sweet beauty until about 2 > weeks ago, now we are running into sudden death scenarios. Could you tell a bit more about the sudden death? Does the drive still respond to queries for smart attributes? What firmware is on the Vertex 3? Anandtech talks about timing issues between certain IO controllers and the SandForce 2281 chipset, which appear to have been resolved in firmware 2.09 (http://www.anandtech.com/show/4341/ocz-vertex-3-max-iops-patriot-wildfire-ssds-reviewed/1). regards, Yeb
On Wednesday, November 02, 2011 01:01:47 PM Yeb Havinga wrote: > Could you tell a bit more about the sudden death? Does the drive still > respond to queries for smart attributes? Just that. It's almost like somebody physically yanked them out of the machine, after months of 24x7 perfect performance. A cold reboot seems to restore order for a while, but the drives die again similarly fairly soon after a failure like this. From what I can tell, SMART is not worth much with SSDs. -Ben -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
On Wednesday, November 02, 2011 11:39:25 AM Thomas Strunz wrote: > I have no idea what you do but just the fact that you bought ssds to > improve performance means it's rather high load and hence important. Important enough that we back everything up hourly. Because of this, we decided to give the SSDs a try. > Using a consumer drive for that IMHO is not the best idea. I know a lot > about ssds but just in consumer space. Intel has a good reputation in > terms of reliability but they are not the fastest. Which is what we're trying next, X25E. 710's apparently have 1/5th the rated write endurance, without much speed increase, so don't seem like such an exciting product. > I guess go Intel > route or some other crazy expensive enterprise stuff. It's advice about some of the "crazy expensive enterprise" stuff that I'm seeking...? I don't mind spending some money if I get to keep up this level of performance, but also am not looking to make somebody's private plane payment, either. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
On 2011-11-03 04:02, Benjamin Smith wrote: > > Which is what we're trying next, X25E. 710's apparently have 1/5th the rated > write endurance, without much speed increase, so don't seem like such an > exciting product. I've tested the 710 with diskchecker.pl and it doesn't lie about it's cache status. I'm note sure about the X25E. that might also be a factor in the equation. According to Greg Smith the X25E is not ok - http://www.2ndquadrant.com/static/2quad/media/pdfs/talks/bottom-up-benchmarking-2011.pdf Also I'm *really* interested to know from one of the bad vertex 3's, what firmware is on it. (hdparm -I /dev/sdX or smartctl -i /dev/sdX). -- Yeb
On Thu, Nov 3, 2011 at 6:02 AM, Benjamin Smith <lists@benjamindsmith.com> wrote: > On Wednesday, November 02, 2011 11:39:25 AM Thomas Strunz wrote: >> I have no idea what you do but just the fact that you bought ssds to >> improve performance means it's rather high load and hence important. > > Important enough that we back everything up hourly. Because of this, we > decided to give the SSDs a try. > >> Using a consumer drive for that IMHO is not the best idea. I know a lot >> about ssds but just in consumer space. Intel has a good reputation in >> terms of reliability but they are not the fastest. > > Which is what we're trying next, X25E. 710's apparently have 1/5th the rated > write endurance, without much speed increase, so don't seem like such an > exciting product. > >> I guess go Intel >> route or some other crazy expensive enterprise stuff. > > It's advice about some of the "crazy expensive enterprise" stuff that I'm > seeking...? I don't mind spending some money if I get to keep up this level of > performance, but also am not looking to make somebody's private plane payment, > either. > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > > > -- > Sent via pgsql-general mailing list (pgsql-general@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-general > How about SSDs on Raid 1+0 (I have no experience on SSD and RAID though) and have replication to another server having the same setup and still do frequent backups. The Crucial m4 SSDs seem to be reasonably priced and perform well. The savings on power and cooling may be used in offsetting some of cost of the warm standby server. Allan.
On Wed, Nov 2, 2011 at 11:02 PM, Benjamin Smith <lists@benjamindsmith.com> wrote: > On Wednesday, November 02, 2011 11:39:25 AM Thomas Strunz wrote: >> I guess go Intel >> route or some other crazy expensive enterprise stuff. > > It's advice about some of the "crazy expensive enterprise" stuff that I'm > seeking...? I don't mind spending some money if I get to keep up this level of > performance, but also am not looking to make somebody's private plane payment, > either. There's a pretty varied mix of speed, durability, and price with any SSD based architecture, but the two that have proven best in our testing and production use (for ourselves and our clients) seem to be Intel (mostly 320 series iirc), and Fusion-IO. I'd start with looking at those. Robert Treat conjecture: xzilla.net consulting: omniti.com
On Thu, Nov 3, 2011 at 4:15 AM, Allan Kamau <kamauallan@gmail.com> wrote:
How about SSDs on Raid 1+0 (I have no experience on SSD and RAID
though) and have replication to another server having the same setup
and still do frequent backups. The Crucial m4 SSDs seem to be
reasonably priced and perform well.
The savings on power and cooling may be used in offsetting some of
cost of the warm standby server.
A question I have wondered about is whether RAID controllers, which were designed
with conventional disk drives in mind, aren't likely to spread the write load out fairly
evenly among the SSDs, and thus lead to the situation where all of the drives are
approaching their rated write cycle capacity at around the same time.
I've asked a few RAID manufacturers whether their controllers can be reconfigured to use SSDs more appropriately, I have yet to get a substantive answer.
Benjamin, have you checked to see if your 'sudden death' problem is heat related?
-
Mike Nolan
with conventional disk drives in mind, aren't likely to spread the write load out fairly
evenly among the SSDs, and thus lead to the situation where all of the drives are
approaching their rated write cycle capacity at around the same time.
I've asked a few RAID manufacturers whether their controllers can be reconfigured to use SSDs more appropriately, I have yet to get a substantive answer.
Benjamin, have you checked to see if your 'sudden death' problem is heat related?
-
Mike Nolan
On Thursday, November 03, 2011 10:59:37 AM you wrote: > There's a pretty varied mix of speed, durability, and price with any > SSD based architecture, but the two that have proven best in our > testing and production use (for ourselves and our clients) seem to be > Intel (mostly 320 series iirc), and Fusion-IO. I'd start with looking > at those. This is *exactly* the type of feedback that I've been looking for - thanks! The Fusion IO looks to be in the "if you have to ask about prices you probably can't afford it" range, although getting a million IOPS is damned impressive. I'm surprised a bit by the Intel 320 referenced, since this is typically touted as a consumer device, but we've consequently decided to give the 710 a shot since it's basically a 320 with some reliability upgrades, and see how it goes. I will post my results here later, time permitting. Thanks again. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
On Thu, Nov 3, 2011 at 16:15, Benjamin Smith <lists@benjamindsmith.com> wrote: > On Thursday, November 03, 2011 10:59:37 AM you wrote: >> There's a pretty varied mix of speed, durability, and price with any >> SSD based architecture, but the two that have proven best in our >> testing and production use (for ourselves and our clients) seem to be >> Intel (mostly 320 series iirc), and Fusion-IO. I'd start with looking >> at those. > > This is *exactly* the type of feedback that I've been looking for - thanks! > > The Fusion IO looks to be in the "if you have to ask about prices you probably > can't afford it" range, although getting a million IOPS is damned impressive. > I'm surprised a bit by the Intel 320 referenced, since this is typically > touted as a consumer device, but we've consequently decided to give the 710 a > shot since it's basically a 320 with some reliability upgrades, and see how it > goes. > > I will post my results here later, time permitting. Oddly enough, Tom's Hardware has a review of the Intel offering today - might be worth your while to take a look at it. Kurt
Am 03.11.2011 18:59, schrieb Robert Treat: > On Wed, Nov 2, 2011 at 11:02 PM, Benjamin Smith > <lists@benjamindsmith.com> wrote: >> On Wednesday, November 02, 2011 11:39:25 AM Thomas Strunz wrote: >>> I guess go Intel >>> route or some other crazy expensive enterprise stuff. >> >> It's advice about some of the "crazy expensive enterprise" stuff that I'm >> seeking...? I don't mind spending some money if I get to keep up this level of Stec (http://stec-inc.com/) or texas memory systems (http://www.ramsan.com/) do the kind of ssds you want for enterprise application. Reading the specs for intel 320, 710 you can calculate how long ssds will live when loaded with maximum random io workload. intel 320 80GB 10TB written 10000 4k IOPS about 3 days to the of end design lifetime intel 710 100GB 500TB written 2700 4k IOPS about 575 days to the of end design lifetime If you are using Linux you can use the values in /proc/iostats to get a rough idea what your system is doing and how many tb get written per day. stec offers a wear resistant ssd which is composed from 8GB RAM, a big capacitor, 8GB Flash and some logic to write the ram contents into flash when the power has gone. see http://www.intel.com/content/dam/doc/product-brief/ssd-320-brief.pdf http://www.intel.com/content/dam/doc/product-specification/ssd-710-series-specification.pdf http://embeddedcomputingsystems.com/wp-content/uploads/2011/05/STEC_AVNET_SSD_Spring2011.pdf >> performance, but also am not looking to make somebody's private plane payment, >> either. > > There's a pretty varied mix of speed, durability, and price with any > SSD based architecture, but the two that have proven best in our > testing and production use (for ourselves and our clients) seem to be > Intel (mostly 320 series iirc), and Fusion-IO. I'd start with looking > at those. > > Robert Treat > conjecture: xzilla.net > consulting: omniti.com >
On 2011-11-04 04:21, Kurt Buff wrote: > Oddly enough, Tom's Hardware has a review of the Intel offering today > - might be worth your while to take a look at it. Kurt Thanks for that link! Seeing media wearout comparisons between 'consumer grade' and 'enterprise' disks was enough for me to stop thinking about the vertex 3 and intel 510 behind hardware raid: I'm going to stick with Intel 710 and Vertex 2 Pro on onboard SATA. Tom's Hardware also showed how to test wearout using the workload indicator, so I thought lets do that with a pgbench workload. First, if your'e interested in doing a test like this yourself, I'm testing on ubuntu 11.10, but even though this is a brand new distribution, the smart database was a few months old. 'update-smart-drivedb' had as effect that the names of the values turned into something useful: instead of #LBA's written, it now shows #32MiB's written. Also there are now three 'workload' related parameters. 225 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always - 108551 226 Workld_Media_Wear_Indic 0x0032 100 100 000 Old_age Always - 17 227 Workld_Host_Reads_Perc 0x0032 100 100 000 Old_age Always - 0 228 Workload_Minutes 0x0032 100 100 000 Old_age Always - 211 232 Available_Reservd_Space 0x0033 100 100 010 Pre-fail Always - 0 233 Media_Wearout_Indicator 0x0032 100 100 000 Old_age Always - 0 241 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always - 108551 242 Host_Reads_32MiB 0x0032 100 100 000 Old_age Always - 21510 Tom's hardware on page http://www.tomshardware.com/reviews/ssd-710-enterprise-x25-e,3038-4.html shows how to turn these numbers into useful values. The numbers above were taken 211 minutes after I cleared the workload values with smartctl -t vendor,0x40 /dev/sda. If you do that, the workload values become 0, then after a few minutes they all become 65535 and not before 60 minutes of testing you'll see some useful values returned. During the test, I did two one hour pgbench runs on a md raid1 with the intel 710 and vertex 2 pro, wal in ram. pgbench -i -s 300 t (fits in ram) pgbench -j 20 -c 20 -M prepared -T 3600 -l t (two times) % mediawear by workload is Workld_Media_Wear_Indic / 1024 17/1024 = .0166015625 % Lets turn this into # days. I take the most pessimistic number of 120 minutes of actual pgbench testing, instead of the total minutes since workload reset of 211 minutes. 120/(17/1024/100)/60/24 = 501.9608599031 days The Host_Reads_32MiB value was 91099 before the test, now it is at 108551. (108551-91099)*32/1024 = 545 GB written during the test. (108551-91099)*32/1024/1024/(17/1024/100) = 3208 TB before media wearout. This number fits between Tom's hardware's calculated wearout numbers, 7268 TB for sequential and 1437 TB for random load. -- Yeb PS: info on test setup Model Number: INTEL SSDSA2BZ100G3 Firmware Revision: 6PB10362 Model Number: OCZ-VERTEX2 PRO Firmware Revision: 1.35 partitions aligned on 512kB boundary. workload on ~20GB software raid mirror (drives are 100GB). Linux client46 3.0.0-12-generic #20-Ubuntu SMP Fri Oct 7 14:56:25 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux PostgreSQL 9.2devel on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu/Linaro 4.6.1-9ubuntu3) 4.6.1, 64-bit /proc/sys/vm/dirty_background_bytes set to 178500000 non standard parameters of pg are: maintenance_work_mem = 1GB # pgtune wizard 2011-10-28 checkpoint_completion_target = 0.9 # pgtune wizard 2011-10-28 effective_cache_size = 16GB # pgtune wizard 2011-10-28 work_mem = 80MB # pgtune wizard 2011-10-28 wal_buffers = 8MB # pgtune wizard 2011-10-28 checkpoint_segments = 96 shared_buffers = 5632MB # pgtune wizard 2011-10-28 max_connections = 300 # pgtune wizard 2011-10-28 Latency and tps graphs of *one* of the 20 clients during the second pgbench test are here: http://imgur.com/a/jjl13 - note that max latency has dropped from ~ 3 seconds from earlier tests to ~ 1 second - this is mainly due to an increase of checkpoint segments from 16 to 96.
On 11/4/2011 8:26 AM, Yeb Havinga wrote: > > First, if your'e interested in doing a test like this yourself, I'm > testing on ubuntu 11.10, but even though this is a brand new > distribution, the smart database was a few months old. > 'update-smart-drivedb' had as effect that the names of the values > turned into something useful: instead of #LBA's written, it now shows > #32MiB's written. Also there are now three 'workload' related parameters. > I submitted the patch for these to smartmontools a few weeks ago and it is now in the current db but not yet in any of the distro update packages. I probably forgot to mention in my post here that you need the latest db for the 710. Also, if you pull the trunk source code and build it yourself it has the ability to decode the drive stats log data (example pasted below). I haven't yet found a use for this myself, but it does seem to have a little more informaiton than the SMART attributes. (Thanks to Christian Franke of the smartmontools project for implementing this feature) Your figures from the workload wear roughly match mine. In production we don't expect to subject the drives to anything close to 100% of the pgbench workload (probably around 1/10 of that on average), so the predicted wear life of the drive is 10+ years in our estimates, under production loads. The big question of course is can the drive's wearout estimate be trusted ? A little more information from Intel about how it is calculated would help allay concerns in this area. # ./smartctl -l devstat,0 /dev/sda smartctl 5.42 2011-10-10 r3434 [x86_64-linux-2.6.32-71.29.1.el6.x86_64] (local build) Copyright (C) 2002-11 by Bruce Allen,http://smartmontools.sourceforge.net Device Statistics (GP Log 0x04) supported pages Page Description 0 List of supported log pages 1 General Statistics 4 General Errors Statistics 5 Temperature Statistics 6 Transport Statistics 7 Solid State Device Statistics # ./smartctl -l devstat /dev/sda smartctl 5.42 2011-10-10 r3434 [x86_64-linux-2.6.32-71.29.1.el6.x86_64] (local build) Copyright (C) 2002-11 by Bruce Allen,http://smartmontools.sourceforge.net Device Statistics (GP Log 0x04) Page Offset Flg Size Value Description 1 ===== == = = == General Statistics (rev 2) == 1 0x008 V- 4 10 Lifetime Power-On Resets 1 0x010 V- 4 200 Power-on Hours 1 0x018 V- 6 3366822529 Logical Sectors Written 1 0x020 V- 6 248189788 Number of Write Commands 1 0x028 V- 6 54653524 Logical Sectors Read 1 0x030 V- 6 2626204 Number of Read Commands 4 ===== == = = == General Errors Statistics (rev 1) == 4 0x008 V- 4 0 Number of Reported Uncorrectable Errors 4 0x010 V- 4 0 Resets Between Cmd Acceptance and Completion 5 ===== == = = == Temperature Statistics (rev 1) == 5 0x008 V- 1 21 Current Temperature 5 0x010 V- 1 20 Average Short Term Temperature 5 0x018 -- 1 20 Average Long Term Temperature 5 0x020 V- 1 30 Highest Temperature 5 0x028 V- 1 17 Lowest Temperature 5 0x030 V- 1 23 Highest Average Short Term Temperature 5 0x038 V- 1 18 Lowest Average Short Term Temperature 5 0x040 -- 1 -128 Highest Average Long Term Temperature 5 0x048 -- 1 -128 Lowest Average Long Term Temperature 5 0x050 V- 4 0 Time in Over-Temperature 5 0x058 V- 1 70 Specified Maximum Operating Temperature 5 0x060 V- 4 0 Time in Under-Temperature 5 0x068 V- 1 0 Specified Minimum Operating Temperature 6 ===== == = = == Transport Statistics (rev 1) == 6 0x008 V- 4 77 Number of hardware resets 6 0x010 V- 4 22 Number of ASR Events 6 0x018 V- 4 0 Number of Interface CRC Errors 7 ===== == = = == Solid State Device Statistics (rev 1) == 7 0x008 V- 1 0 Percentage Used Endurance Indicator ||_ N normalized |__ V valid
On Wed, Nov 2, 2011 at 1:01 PM, Benjamin Smith <lists@benjamindsmith.com> wrote: > I don't mind spending some money. Can anybody comment on a recommended drive > in real world use? We have been using the RamSan-620 from Texas Memory Systems <http://www.ramsan.com/> for over a year now on a heavy write load. I have a pair of them for redundancy on two servers replicated using Slony. Nobody would ever call these things cheap, but the technology behind them is nothing short of phenomenal.
On 2011-11-04 16:24, David Boreham wrote: > On 11/4/2011 8:26 AM, Yeb Havinga wrote: >> >> First, if your'e interested in doing a test like this yourself, I'm >> testing on ubuntu 11.10, but even though this is a brand new >> distribution, the smart database was a few months old. >> 'update-smart-drivedb' had as effect that the names of the values >> turned into something useful: instead of #LBA's written, it now shows >> #32MiB's written. Also there are now three 'workload' related >> parameters. >> > I submitted the patch for these to smartmontools a few weeks ago and > it is now in the current db but not yet in any of the distro update > packages. I probably forgot to mention in my post here that you need > the latest db for the 710. Also, if you pull the trunk source code and > build it yourself it has the ability to decode the drive stats log > data (example pasted below). I haven't yet found a use for this > myself, but it does seem to have a little more informaiton than the > SMART attributes. (Thanks to Christian Franke of the smartmontools > project for implementing this feature) > > Your figures from the workload wear roughly match mine. In production > we don't expect to subject the drives to anything close to 100% of the > pgbench workload (probably around 1/10 of that on average), so the > predicted wear life of the drive is 10+ years in our estimates, under > production loads. > > The big question of course is can the drive's wearout estimate be > trusted ? A little more information from Intel about how it is > calculated would help allay concerns in this area. TLDR: some numbers after three week media wear testing on a software mirror with intel 710 and ocz vertex 2 pro. The last couple of weeks I've been running pgbench for an hour then sleep for 10 minutes in an infinite loop, just to see how values would grow. This is the intel 710 mirror leg: 225 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always - 3020093 226 Workld_Media_Wear_Indic 0x0032 100 100 000 Old_age Always - 2803 227 Workld_Host_Reads_Perc 0x0032 100 100 000 Old_age Always - 0 228 Workload_Minutes 0x0032 100 100 000 Old_age Always - 21444 232 Available_Reservd_Space 0x0033 100 100 010 Pre-fail Always - 0 233 Media_Wearout_Indicator 0x0032 098 098 000 Old_age Always - 0 241 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always - 3020093 242 Host_Reads_32MiB 0x0032 100 100 000 Old_age Always - 22259 Note: raw value of 226 (E2) = 2803. According to http://www.tomshardware.com/reviews/ssd-710-enterprise-x25-e,3038-4.html you have to divide it by 1024 to get a percentage. That would be 2%. This matches with 098 of the (not raw) value at 233 (E9). This is the ocz vertex 2 PRO mirror leg: 5 Retired_Block_Count 0x0033 100 100 003 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 22 100 Gigabytes_Erased 0x0032 000 000 000 Old_age Always - 21120 170 Reserve_Block_Count 0x0032 000 000 000 Old_age Always - 34688 177 Wear_Range_Delta 0x0000 000 000 000 Old_age Offline - 3 230 Life_Curve_Status 0x0013 100 100 000 Pre-fail Always - 100 231 SSD_Life_Left 0x0013 100 100 010 Pre-fail Always - 0 232 Available_Reservd_Space 0x0000 000 000 000 Old_age Offline - 33 233 SandForce_Internal 0x0000 000 000 000 Old_age Offline - 21184 234 SandForce_Internal 0x0032 000 000 000 Old_age Always - 94656 235 SuperCap_Health 0x0033 100 100 002 Pre-fail Always - 0 241 Lifetime_Writes_GiB 0x0032 000 000 000 Old_age Always - 94656 242 Lifetime_Reads_GiB 0x0032 000 000 000 Old_age Always - 960 Here the 177 (B1) wear range delta is on a raw value of 3 - this isn't ssd life left, but Delta between most-worn and least-worn Flash blocks. I really wonder at which point SSD life left will change to 99 on this drive.. regards, Yeb Havinga
On 2011-11-24 14:20, Yeb Havinga wrote: > I really wonder at which point SSD life left will change to 99 on > this drive.. Bingo! On the OCZ Vertex 2 PRO, SSD life left to 99 after just over 100PB written. 230 Life_Curve_Status 0x0013 100 100 000 Pre-fail Always - 100 231 SSD_Life_Left 0x0013 099 099 010 Pre-fail Always - 0 232 Available_Reservd_Space 0x0000 000 000 000 Old_age Offline - 33 233 SandForce_Internal 0x0000 000 000 000 Old_age Offline - 22848 234 SandForce_Internal 0x0032 000 000 000 Old_age Always - 101952 235 SuperCap_Health 0x0033 100 100 002 Pre-fail Always - 0 241 Lifetime_Writes_GiB 0x0032 000 000 000 Old_age Always - 101952 242 Lifetime_Reads_GiB 0x0032 000 000 000 Old_age Always - 960 The Intel 710 now shows media wearout of 97. 226 Workld_Media_Wear_Indic 0x0032 100 100 000 Old_age Always - 3028 227 Workld_Host_Reads_Perc 0x0032 100 100 000 Old_age Always - 0 228 Workload_Minutes 0x0032 100 100 000 Old_age Always - 22966 232 Available_Reservd_Space 0x0033 100 100 010 Pre-fail Always - 0 233 Media_Wearout_Indicator 0x0032 097 097 000 Old_age Always - 0 241 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always - 3255685 242 Host_Reads_32MiB 0x0032 100 100 000 Old_age Always - 22259 regards, Yeb Havinga