Thread: 1 TB of memory
PostgreSQL tuned to the max and still too slow? Database too big to fit into memory? Here's the solution! http://www.superssd.com/ products/tera-ramsan/ Anyone purchasing one will be expected to post benchmarks! :) -- Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com Pervasive Software http://pervasive.com work: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461
Jim Nasby wrote: > PostgreSQL tuned to the max and still too slow? Database too big to > fit into memory? Here's the solution! > http://www.superssd.com/products/tera-ramsan/ > > Anyone purchasing one will be expected to post benchmarks! :) And give us one :) > -- > Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com > Pervasive Software http://pervasive.com work: 512-231-6117 > vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461 > > > > ---------------------------(end of broadcast)--------------------------- > TIP 6: explain analyze is your friend >
The US Dept of Homeland Security has at least two =10=TB SSDs. <begin speculation> Rumor is they are being used for Carnivore or an offshoot/descendent of Carnivore. <end speculation> Good luck getting them to give you benchmark data. You need >deep< pockets to afford >= 1TB of SSD. (...and as the example shows, perhaps more money than sense.) Ron -----Original Message----- >From: Jim Nasby <jnasby@pervasive.com> >Sent: Mar 16, 2006 1:33 PM >To: pgsql-performance@postgresql.org >Subject: [PERFORM] 1 TB of memory > >PostgreSQL tuned to the max and still too slow? Database too big to >fit into memory? Here's the solution! http://www.superssd.com/ >products/tera-ramsan/ > >Anyone purchasing one will be expected to post benchmarks! :) >-- >Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com >Pervasive Software http://pervasive.com work: 512-231-6117 >vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461 > > > >---------------------------(end of broadcast)--------------------------- >TIP 6: explain analyze is your friend
On 3/16/06, Jim Nasby <jnasby@pervasive.com> wrote: > PostgreSQL tuned to the max and still too slow? Database too big to > fit into memory? Here's the solution! http://www.superssd.com/ > products/tera-ramsan/ > > Anyone purchasing one will be expected to post benchmarks! :) Pricing is tight-lipped, but searching shows $1.85 /GB. That's close to $500,000 for 250GB. One report says a person paid $219,000 for 32GB and 1TB costs "well over $1,000,000." But they "guarantee the performance." Too rich for me. -- Matthew Nuzum www.bearfruit.org
Jim, > PostgreSQL tuned to the max and still too slow? Database too big to > fit into memory? Here's the solution! http://www.superssd.com/ > products/tera-ramsan/ With a single 3 Gbyte/second infiniband connection to the device? You'd be better off with 4 x $10K servers that do 800MB/s from disk each and a Bizgres MPP - then you'd do 3.2GB/s (faster than the SSD) at a price 1/10 of the SSD, and you'd have 24TB of RAID5 disk under you. Plus - need more speed? Add 12 more servers, and you'd run at 12.8GB/s and have 96TB of disk to work with, and you'd *still* spend less on HW and SW than the SSD. - Luke
Jim, On 3/16/06 10:44 PM, "Luke Lonergan" <llonergan@greenplum.com> wrote: > Plus - need more speed? Add 12 more servers, and you'd run at 12.8GB/s and > have 96TB of disk to work with, and you'd *still* spend less on HW and SW > than the SSD. And I forgot to mention that with these 16 servers you'd have 64 CPUs and 256GB of RAM working for you in addition to the 96TB of disk. Every query would use all of that RAM and all of those CPUs, all at the same time. By comparison, with the SSD, you'd have 1 CPU trying to saturate 1 connection to the SSD. If you do anything other than just access the data there (order by, group by, join, aggregation, functions), you'll be faced with trying to have 1 CPU do all the work on 1 TB of data. I suggest that it won't be any faster than having the 1 TB on disk for most queries, as you would be CPU bound. By comparison, with the MPP system, all 64 CPUs would be used at one time to process the N TB of data and if you grew from N TB to 2N TB, you could double the machine size and it would take the same amount of time to do 2N as it did to do N. That's what data parallelism and scaling is all about. Without it, you don't have a prayer of using all 1TB of data in queries. - Luke
Luke, > With a single 3 Gbyte/second infiniband connection to the device? Hey, take it easy! Jim's post was tongue-in-cheek. -- Josh Berkus Aglio Database Solutions San Francisco
Josh, On 3/16/06 9:43 PM, "Josh Berkus" <josh@agliodbs.com> wrote: >> With a single 3 Gbyte/second infiniband connection to the device? > > Hey, take it easy! Jim's post was tongue-in-cheek. You're right - I insulted his bandwidth, sorry :-) - Luke
On Thu, Mar 16, 2006 at 10:44:25PM -0800, Luke Lonergan wrote: >You'd be better off with 4 x $10K servers that do 800MB/s from disk each and >a Bizgres MPP - then you'd do 3.2GB/s (faster than the SSD) at a price 1/10 >of the SSD, and you'd have 24TB of RAID5 disk under you. Except, of course, that your solution doesn't have a seek time of zero. That approach is great for applications that are limited by their sequential scan speed, not so good for applications with random access. At 3.2 GB/s it would still take over 5 minutes to seqscan a TB, so you'd probably want some indices--and you're not going to be getting 800MB/s per system doing random index scans from rotating disk (but you might with SSD). Try not to beat your product drum quite so loud... Mike Stone
For God's sake buy a mainframe! =o) On 3/17/06, Michael Stone <mstone+postgres@mathom.us> wrote: > On Thu, Mar 16, 2006 at 10:44:25PM -0800, Luke Lonergan wrote: > >You'd be better off with 4 x $10K servers that do 800MB/s from disk each and > >a Bizgres MPP - then you'd do 3.2GB/s (faster than the SSD) at a price 1/10 > >of the SSD, and you'd have 24TB of RAID5 disk under you. > > Except, of course, that your solution doesn't have a seek time of zero. > That approach is great for applications that are limited by their > sequential scan speed, not so good for applications with random access. > At 3.2 GB/s it would still take over 5 minutes to seqscan a TB, so you'd > probably want some indices--and you're not going to be getting 800MB/s > per system doing random index scans from rotating disk (but you might > with SSD). Try not to beat your product drum quite so loud... > > Mike Stone > > ---------------------------(end of broadcast)--------------------------- > TIP 4: Have you searched our list archives? > > http://archives.postgresql.org >
We got a quote for one of these (entirely for comedy value of course) and it was in the region of £1,500,000 give or take a few thousand. On 16 Mar 2006, at 18:33, Jim Nasby wrote: > PostgreSQL tuned to the max and still too slow? Database too big to > fit into memory? Here's the solution! http://www.superssd.com/ > products/tera-ramsan/ > > Anyone purchasing one will be expected to post benchmarks! :) > -- > Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com > Pervasive Software http://pervasive.com work: 512-231-6117 > vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461 > > > > ---------------------------(end of > broadcast)--------------------------- > TIP 6: explain analyze is your friend
On 3/16/06, Jim Nasby <jnasby@pervasive.com> wrote: > PostgreSQL tuned to the max and still too slow? Database too big to > fit into memory? Here's the solution! http://www.superssd.com/ > products/tera-ramsan/ > > Anyone purchasing one will be expected to post benchmarks! :) I like their approach...ddr ram + raid sanity backup + super reliable power system. Their prices are on jupiter (and i dont mean jupiter, fl) but hopefully there will be some competition and the invetible decline in prices. When prices drop from the current 1-2k$/Gb to a more realistic 250$/Gb there will be no reason not to throw one into a server. You could already make a case for an entry level one to handle the WAL and perhaps a few key tables/indexes, particularly ones that are frequenct vacuum targets. ddr approach is much faster than flash nvram inherintly and has a virtually unlimited duty cycle. My prediction is that by 2010 SSD will be relatively commonplace in the server market, barring some rediculous goverment intervention (patentes, etc). merlin
On 3/17/06, Rodrigo Madera <rodrigo.madera@gmail.com> wrote: > I don't know about you databasers that crunch in some selects, updates > and deletes, but my personal developer workstation is planned to be a > 4x 300GB SATA300 with a dedicated RAID stripping controller (no > checksums, just speedup) and 4x AMD64 CPUs... not to mention 2GB for > each processor... all this in a nice server motherboard... no doubt, that will handle quite a lot of data. in fact, most databases (contrary to popular opinion) are cpu bound, not i/o bound. However, at some point a different set of rules come into play. This point is constantly chaning due to the relentless march of hardware but I'd suggest that at around 1TB you can no longer count on things to run quickly just depending on o/s file caching to bail you out. Or, you may have a single table + indexes thats 50 gb that takes 6 hours to vacuum sucking all your i/o. another useful aspect of SSD is the relative value of using system memory is much less, so you can reduce swappiness and tune postgres to rely more on the filesystem and give all your memory to work_mem and such. merlin
On Thu, Mar 16, 2006 at 10:44:25PM -0800, Luke Lonergan wrote: > Jim, > > > PostgreSQL tuned to the max and still too slow? Database too big to > > fit into memory? Here's the solution! http://www.superssd.com/ > > products/tera-ramsan/ > > With a single 3 Gbyte/second infiniband connection to the device? > > You'd be better off with 4 x $10K servers that do 800MB/s from disk each and > a Bizgres MPP - then you'd do 3.2GB/s (faster than the SSD) at a price 1/10 > of the SSD, and you'd have 24TB of RAID5 disk under you. > > Plus - need more speed? Add 12 more servers, and you'd run at 12.8GB/s and > have 96TB of disk to work with, and you'd *still* spend less on HW and SW > than the SSD. Now what happens as soon as you start doing random I/O? :) -- Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com Pervasive Software http://pervasive.com work: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461
Jim, On 3/17/06 9:36 AM, "Jim C. Nasby" <jnasby@pervasive.com> wrote: > Now what happens as soon as you start doing random I/O? :) Well - given that we've divided the data into 32 separate segments, and that seeking is done in parallel over all 256 disk drives, random I/O rocks hard and scales. Of course, the parallelizing planner is designed to minimize seeking as much as possible, as is the normal Postgres planner, but with more segment and more parallel platters, seeking is faster. The biggest problem with this idea of "put huge amounts of data on your SSD and everything is infinitely fast" is that it ignores several critical scaling factors: - How much bandwidth is available in and out of the device? - Does that bandwidth scale as you grow the data? - As you grow the data, how long does it take to use the data? - Can more than 1 CPU use the data at once? Do they share the path to the data? If you are accessing 3 rows at a time from among billions, the problem you have is mostly access time - so an SSD might be very good for some OLTP applications. However - the idea of putting Terabytes of data into an SSD through a thin straw of a channel is silly. Note that SSDs have been around for a *long* time. I was using them on Cray X/MP and 2 supercomputers back in 1987-92, when we had a 4 Million Word SSD connected over a 2GB/s channel. In fact, some people I worked with built a machine with 4 Cray 2 computers that shared an SSD between them for parallel computing and it was very effective, and also ungodly expensive and special purpose. - Luke
On 3/17/06, Luke Lonergan <llonergan@greenplum.com> wrote: > > Now what happens as soon as you start doing random I/O? :) > If you are accessing 3 rows at a time from among billions, the problem you > have is mostly access time - so an SSD might be very good for some OLTP > applications. However - the idea of putting Terabytes of data into an SSD > through a thin straw of a channel is silly. I'll 'byte' on this..right now the price for gigabyte of ddr ram is hovering around 60$/gigabyte. If you conveniently leave aside the problem of making ddr ram fault tolerant vs making disks tolerant, you are getting 10 orders of magnitude faster seek time and unlimited bandwidth...at least from the physical device. While SANs are getting cheaper they are still fairly expensive at 1-5$/gigabyte depending on various factors. You can do the same tricks on SSD storage as with disks. SSD storage is 1-2k$/gigabyte currently, but I think there is huge room to maneuver price-wise after the major players recoup their investments and market forces kick in. IMO this process is already in play and the next cycle of hardware upgrades in the enterprise will be updating critical servers with SSD storage. Im guessing by as early 2010 a significant percentage of enterpise storage will be SSD of some flavor. merlin
On Fri, 2006-03-17 at 15:28, Merlin Moncure wrote: > On 3/17/06, Luke Lonergan <llonergan@greenplum.com> wrote: > > > Now what happens as soon as you start doing random I/O? :) > > If you are accessing 3 rows at a time from among billions, the problem you > > have is mostly access time - so an SSD might be very good for some OLTP > > applications. However - the idea of putting Terabytes of data into an SSD > > through a thin straw of a channel is silly. > > I'll 'byte' on this..right now the price for gigabyte of ddr ram is > hovering around 60$/gigabyte. If you conveniently leave aside the > problem of making ddr ram fault tolerant vs making disks tolerant, you > are getting 10 orders of magnitude faster seek time and unlimited > bandwidth...at least from the physical device. While SANs are getting > cheaper they are still fairly expensive at 1-5$/gigabyte depending on > various factors. You can do the same tricks on SSD storage as with > disks. > > SSD storage is 1-2k$/gigabyte currently, but I think there is huge > room to maneuver price-wise after the major players recoup their > investments and market forces kick in. IMO this process is already in > play and the next cycle of hardware upgrades in the enterprise will be > updating critical servers with SSD storage. Im guessing by as early > 2010 a significant percentage of enterpise storage will be SSD of some > flavor. Now I'm envisioning building something with commodity 1U servers hold 4 to 16 gigs ram, and interconnected with 1g or 10g ethernet. Open Source SSD via iSCSI with commodity hardware... hmmm. sounds like a useful project.
On Mar 17, 2006, at 8:55 AM, Merlin Moncure wrote: > I like their approach...ddr ram + raid sanity backup + super reliable > power system. Their prices are on jupiter (and i dont mean jupiter, > fl) but hopefully there will be some competition and the invetible Nothing unique to them. I have a 4 year old SSD from a now out-of- business company, Imperial Technology. Initially we bought it for about $20k with 1GB of RAM. Subsequently upgraded to 5GB for another $20k. The speed is wicked fast even with just ultra2 SCSI (4 channels). The unit has the same battery backup to disk stuff (although it only does the backup at power fail). At one time they quoted me about $80k to upgrade it to a full 32MB that the unit supports. I passed. For my use it was worth the price. However, given the speed increase of other components since then, I don't think I'd buy one today. Parallelism (if you can do it like Luke suggested) is the way to go. And no, I have not run a database on one of these... though I am tempted to...
On Mar 17, 2006, at 5:07 PM, Scott Marlowe wrote: > Open Source SSD via iSCSI with commodity hardware... hmmm. sounds > like > a useful project. shhhhh! don't give away our top secret plans!
> > I like their approach...ddr ram + raid sanity backup + super reliable > > power system. Their prices are on jupiter (and i dont mean jupiter, > > fl) but hopefully there will be some competition and the invetible > > Nothing unique to them. I have a 4 year old SSD from a now out-of- > business company, Imperial Technology. Initially we bought it for > about $20k with 1GB of RAM. Subsequently upgraded to 5GB for another > $20k. The speed is wicked fast even with just ultra2 SCSI (4 > channels). The unit has the same battery backup to disk stuff > (although it only does the backup at power fail). you may or may not be intersted to know they are back in business :). > For my use it was worth the price. However, given the speed increase > of other components since then, I don't think I'd buy one today. > Parallelism (if you can do it like Luke suggested) is the way to go. Thats an interesting statement. My personal opionion is that SSD will ultimately take over the database storage market as well as most consumer level devices for primary storage. except perhaps for very large databases (>1tb). Hard disk drives will displace tapes for backup storage. merlin
On Mar 20, 2006, at 2:44 PM, Merlin Moncure wrote: >> For my use it was worth the price. However, given the speed increase >> of other components since then, I don't think I'd buy one today. >> Parallelism (if you can do it like Luke suggested) is the way to go. > > Thats an interesting statement. My personal opionion is that SSD will > ultimately take over the database storage market as well as most > consumer level devices for primary storage. except perhaps for very I tend to agree with you that perhaps one day when the $$ are right, but that day is not today.