Thread: Amazon High I/O instances

Hello,

Le mardi 21 août 2012 à 01:33 -0400, Sébastien Lorion a écrit :

>
>
> Since Amazon has added new high I/O instance types and EBS volumes,
> anyone has done some benchmark of PostgreSQL on them ?
>

I wonder : is there a reason why you have to go through the complexity
of such a setup, rather than simply use bare metal and get good
performance with simplicity?

For instance, the dedibox I use for my app (visible in sig) costs 14,00
euros/month, and sits at .03% load average with 5 active users; you can
admin it like a home pc.


--
Vincent Veyron
http://marica.fr/
Gestion informatique des sinistres d'assurances et des dossiers contentieux pour le service juridique

Le mardi 21 août 2012 à 01:33 -0400, Sébastien Lorion a écrit :

>
>
> Since Amazon has added new high I/O instance types and EBS volumes,
> anyone has done some benchmark of PostgreSQL on them ?
>

I wonder : is there a reason why you have to go through the complexity
of such a setup, rather than simply use bare metal and get good
performance with simplicity?

For instance, the dedibox I use for my app (visible in sig) costs 14,00
euros/month, and sits at .03% load average with 5 active users; you can
admin it like a home pc.


--
Vincent Veyron
http://marica.fr/
Gestion informatique des sinistres d'assurances et des dossiers contentieux pour le service juridique

On 8/21/2012 7:10 AM, Oliver Kohll - Mailing Lists wrote:
>
> This is a general 'cloud or dedicated' question, I won't go into it
> but I believe cloud proponents cite management ease, scalability etc.
> I'm sure there's a place for every type of hosting. However I would be
> interested in hearing some experiences of PostgreSQL on an Amazon high
> I/O instance, given a client has just proposed running on one. If
> there are none forthcoming in the short term I may be in a position to
> provide some results myself in a month or two.
>

Amazon don't say what vendor's SSDs they are using, which is a little
worrying to me -- when we deployed our SSD-based machines last year,
much work was done to address the risk of write endurance problems. Now,
an AWS instance and its ephemeral storage isn't expected to live forever
(keep that in mind when storing data on one!)
so perhaps one can ignore write endurance as a concern in this case
since we'd already be worried about (and have a plan to address) "entire
machine endurance".

For sure performance on these instances for any I/O limited application
is going to be great.
I have a friend who is looking at them for his "big data" analytics
application which spends most of its time sorting Tb sized files.

On 8/21/2012 2:18 AM, Vincent Veyron wrote:
> I wonder : is there a reason why you have to go through the complexity
> of such a setup, rather than simply use bare metal and get good
> performance with simplicity?
In general I agree -- it is much (much!) cheaper to buy tin and deploy
yourself vs any of the current cloud services.

However, there are plenty of counterexample use cases : for example what
if you want one of these machines for a week only?
Another one : what if you are a venture capitalist funding 10 companies
with questionable business models where you expect only one to succeed?
AWS saves you from the headache of selling 500 machines on eBay...

On Tue, Aug 21, 2012 at 12:33 AM, Sébastien Lorion
<sl@thestrangefactory.com> wrote:
> Hello,
>
> Since Amazon has added new high I/O instance types and EBS volumes, anyone
> has done some benchmark of PostgreSQL on them ?
>
> http://perspectives.mvdirona.com/2012/07/20/IOPerformanceNoLongerSucksInTheCloud.aspx
> http://perspectives.mvdirona.com/2012/08/01/EBSProvisionedIOPSOptimizedInstanceTypes.aspx
> http://aws.typepad.com/aws/2012/08/fast-forward-provisioned-iops-ebs.html
>
> I will be testing my app soon, but was curious to know if others have done
> some tests so I can compare / have a rough idea to what to expect. Looking
> on Google, I found an article about MySQL
> (http://palominodb.com/blog/2012/07/24/palomino-evaluates-amazon%E2%80%99s-new-high-io-ssd-instances),
> but nothing about PostgresSQL.

here's a datapoint, stock config:
pgbench -i -s 500
pgbench -c 16 -T 60
number of transactions actually processed: 418012
tps = 6962.607292 (including connections establishing)
tps = 6973.154593 (excluding connections establishing)

not too shabby.  this was run by a friend who is evaluating high i/o
instances for their high load db servers.   we didn't have time to
kick off a high scale read only test unfortunately.

Regarding 'AWS vs bare metal', I think high i/o instances full a huge
niche in their lineup.   Dollar for dollar, I'm coming around to the
point of view that dealing with aws is a cheaper/more effective
solution than renting out space from a data center or (even worse)
running your own data center unless you're very large or have other
special requirements.  Historically the problem with AWS is that you
had no solution for highly transaction bound systems which forced you
to split your environment which ruined most of the benefit, and they
fixed that.

merlin

Le mardi 21 août 2012 à 09:36 -0500, Merlin Moncure a écrit :

> here's a datapoint, stock config:
> pgbench -i -s 500
> pgbench -c 16 -T 60
> number of transactions actually processed: 418012
> tps = 6962.607292 (including connections establishing)
> tps = 6973.154593 (excluding connections establishing)
>
> not too shabby.  this was run by a friend who is evaluating high i/o
> instances for their high load db servers.   we didn't have time to
> kick off a high scale read only test unfortunately.
>
> Regarding 'AWS vs bare metal', I think high i/o instances full a huge
> niche in their lineup.   Dollar for dollar, I'm coming around to the
> point of view that dealing with aws is a cheaper/more effective
> solution than renting out space from a data center or (even worse)
> running your own data center unless you're very large or have other
> special requirements.  Historically the problem with AWS is that you
> had no solution for highly transaction bound systems which forced you
> to split your environment which ruined most of the benefit, and they
> fixed that.
>

Hi Merlin,

I am sure you can get good performance with these.

I simply focused on the part where seb said he was testing his app, and
since you can get some really high data throughput (by my very modest
standards anyway) with a good machine, I wondered why he did it.

Maybe seb is planning for an application that already has hundreds of
users after all, I did oversee that option.

To Sébastien : please use 'reply all' to send your reply to the list

Le mardi 21 août 2012 à 10:29 -0400, Sébastien Lorion a écrit :
Could you elaborate on the complexity you mention ? Setting up a machine
on Amazon, even with a script, is quite simple. As for the pricing you
give, it can be matched on Amazon using Micro or small instances, which
would be adequate given your load average.
>
>

Well, it _has_ to be more complicated to use AWS than a bare machine,
because of the added layer?



--
Vincent Veyron
http://vincentveyron.com/
Gestion informatique des sinistres d'assurances et des dossiers contentieux pour le service juridique

Oops sorry, I thought I did hit reply all.

On 08/21/2012 09:40 PM, David Boreham wrote:
> On 8/21/2012 2:18 AM, Vincent Veyron wrote:
>> I wonder : is there a reason why you have to go through the complexity
>> of such a setup, rather than simply use bare metal and get good
>> performance with simplicity?
> In general I agree -- it is much (much!) cheaper to buy tin and deploy
> yourself vs any of the current cloud services.
>
> However, there are plenty of counterexample use cases : for example what
> if you want one of these machines for a week only?
> Another one : what if you are a venture capitalist funding 10 companies
> with questionable business models where you expect only one to succeed?
> AWS saves you from the headache of selling 500 machines on eBay...

Dedibox appears to be a hosting company that offers dedicated machines.

He appears to be suggesting that buying access to real hardware in a
datacenter (if not buying the hardware yourself) is more cost effective
and easier to manage than using "cloud" style services with more
transient hosts like EC2 offers. At least that's how I understood it.
Vincent?


I wasn't sure what Vincent meant until I did an `mtr` on his host
address, either.

http://dedibox.fr/

redirects to

http://www.online.net/


A look at their product page suggests that they're at least claiming the
machines are dedicated:

http://www.online.net/serveur-dedie/offre-dedibox-sc.xhtml

running Via Nano (Nano U2250) CPUs on Dell VX11-VS8 machines. The VS8
appears to be a blade:


http://en.community.dell.com/dell-blogs/direct2dell/b/direct2dell/archive/2009/05/19/dell-launches-quot-fortuna-quot-via-nano-based-server-for-hyperscale-customers.aspx

http://www.flickr.com/photos/netbooknews/3537912243/


so yeah, a dedicated server for €15/month. That's *AWESOME* when you
mostly need storage and you don't care about performance or storage
reliability; it's a local HDD so you get great gobs of storage w/o
paying per GB.

--
Craig Ringer

Le mercredi 22 août 2012 à 13:15 +0800, Craig Ringer a écrit :

> He appears to be suggesting that buying access to real hardware in a
> datacenter (if not buying the hardware yourself) is more cost effective
> and easier to manage than using "cloud" style services with more
> transient hosts like EC2 offers. At least that's how I understood it.

Hi Craig,

Actually, my comments about costs were misleading : I simply reacted to
the fact that the OP wanted to test his application for high
performance, and thought that it would be easier with bare metal rather
than with AWS, because you have less parameters to control this way.

Also, I'll admit that I jumped the gun without reading about the SSD
offer by Amazon. Still, I would test first with a machine that I
control, but it maybe that Sébastien already did that.

I am curious to know what kind of application requires 10s to 100s of
instances with a PostgreSQL database, because that could get unwieldy
with big data (which I assumed from the high performance specification)

--
Vincent Veyron
http://marica.fr/
Gestion informatique des sinistres d'assurances et des dossiers contentieux pour le service juridique

Vincent, I would appreciate that you stop assuming things based on zero information about what I am doing. I understand that you are trying to be helpful, but I can assure you that going bare-metal only does not make any sense in my context.

On Wednesday, August 22, 2012 04:10:01 PM Andrew Hannon wrote:
> Just looking into High IO instances for a DB deployment. In order to get
> past 1TB, we are looking at RAID-0. I have heard
> (http://hackerne.ws/item?id=4266119) there might be a problem if TRIM isn't
> supported. Does anyone know if it is and has anyone used RAID-0 on these
> instances? (Linux of course…)

Just use LVM striping. If it turns out to be an issue; that seems to be mostly
conjecture.

I note that the SSDs are only instance storage. The data will be gone when the
instance goes away. I have used instance storage in replicated setups but it
always feels rather fragile unless your data really is transient or you can
maintain 2 replicas.

Their other new service, provisioned IOPS for EBS, might be more useful for a
persistent database. Although not nearly SSD speeds, of course.

Le mercredi 22 août 2012 à 13:30 -0400, Sébastien Lorion a écrit :
> Vincent, I would appreciate that you stop assuming things based on
> zero information about what I am doing. I understand that you are
> trying to be helpful, but I can assure you that going bare-metal only
> does not make any sense in my context.


Dude,

I would appreciate you realize that approaching a newsgroup while
providing zero information about what you are doing (in your own words)
is not the best way to get relevant responses to your question.

Ignoring repeated requests for information does not help, castigating
people trying to help for not having said information at least shows a
certain consistency on your part.

Lest we ridicule ourselves publicly, I suggest we leave the discussion
at that and wish you luck in your endeavor.

Vincent Veyron

On Thu, Aug 23, 2012 at 7:39 AM, Vincent Veyron <vv.lists@wanadoo.fr> wrote:
> Lest we ridicule ourselves publicly, I suggest we leave the discussion
> at that and wish you luck in your endeavor.

If anyone has an answer to his question, I'd appreciate hearing it,
despite any faux pas that the OP has committed.

Vincent,

On 08/23/2012 07:39 PM, Vincent Veyron wrote:
> Le mercredi 22 août 2012 à 13:30 -0400, Sébastien Lorion a écrit :
>> Vincent, I would appreciate that you stop assuming things based on
>> zero information about what I am doing. I understand that you are
>> trying to be helpful, but I can assure you that going bare-metal only
>> does not make any sense in my context.
>
>
> Dude,
>
> I would appreciate you realize that approaching a newsgroup while
> providing zero information about what you are doing

In this case, what he's doing is seeking generalized performance
measurements. I don't think details were particularly necessary until it
got pulled off-track.

I'll be interested to hear if you have any results Sébastien, or if
anyone else does. It's good to have data on the increasingly popular
cloud platforms out there.

--
Craig Ringer

On Wed, Aug 22, 2012 at 4:12 PM, Alan Hodgson <ahodgson@simkin.ca> wrote:
> On Wednesday, August 22, 2012 04:10:01 PM Andrew Hannon wrote:
>> Just looking into High IO instances for a DB deployment. In order to get
>> past 1TB, we are looking at RAID-0. I have heard
>> (http://hackerne.ws/item?id=4266119) there might be a problem if TRIM isn't
>> supported. Does anyone know if it is and has anyone used RAID-0 on these
>> instances? (Linux of course…)
>
> Just use LVM striping. If it turns out to be an issue; that seems to be mostly
> conjecture.
>
> I note that the SSDs are only instance storage. The data will be gone when the
> instance goes away. I have used instance storage in replicated setups but it
> always feels rather fragile unless your data really is transient or you can
> maintain 2 replicas.
>
> Their other new service, provisioned IOPS for EBS, might be more useful for a
> persistent database. Although not nearly SSD speeds, of course.

Yeah -- I should have mentioned that: you absolutely must run hs/sr or
some other strategy that maintains your data.  I guess you might as
well turn off fsync, right?

merlin

On 08/23/12 6:49 AM, Craig Ringer wrote:
> In this case, what he's doing is seeking generalized performance
> measurements. I don't think details were particularly necessary until
> it got pulled off-track.


"42"

performance measurements without a very narrow definition of
'performance' are useless.    depending on the nature of the application
workload, postgres can stress completely different aspects of the system
(cpu vs read IO performance vs write IO performance being the big three).



--
john r pierce                            N 37, W 122
santa cruz ca                         mid-left coast

I think both kind of tests (general and app specific) are complementary and useful in their own way. At a minimum, if the general ones fail, why go to the expenses of doing the specific ones ? Setting up a meaningful application test can take a lot of time and it can be hard to pinpoint exactly where in the stack the performance drops occur. The way I see it, synthetic benchmarks allow to isolate somewhat the layers and serve as a base to validate application tests done later on. It surprises me that asking for the general perf behavior of a platform is controversial.

On 08/23/12 11:24 AM, Sébastien Lorion wrote:
> I think both kind of tests (general and app specific) are
> complementary and useful in their own way. At a minimum, if the
> general ones fail, why go to the expenses of doing the specific ones ?
> Setting up a meaningful application test can take a lot of time and it
> can be hard to pinpoint exactly where in the stack the performance
> drops occur. The way I see it, synthetic benchmarks allow to isolate
> somewhat the layers and serve as a base to validate application tests
> done later on. It surprises me that asking for the general perf
> behavior of a platform is controversial.

I don't use AWS at all.   But, it shouldnt take more than a couple hours
to spin up an instance, populate a pgbench database and run a series of
pgbench runs against it, and do the same against any other sort of
system you wish to use as your reference.

I like to test with a database about twice the size of the available
memory if I'm testing IO, and I've found that pgbench -i -s ####, for
####=10000 it generates a 1 billion row table and uses about 150GB (and
a hour or so to initialize on fast IO hardware).  I then run pgbench
with -c of about 2-4X the cpu/thread count, and -j of about -c/16, and a
-t of at least 10000 (so each client connection runs 10000 transactions).

on a modest but decent 2U class 2-socket dedicated server with a decent
raid card and raid10 across enough spindles, I can see numbers as high
as 5000 transactions/second with 15krpm rust, and 7000-8000 with a
couple MLC SSD's striped.   trying to raid10 a bunch of SATA 7200 disks
gives numbers more like 1000.   using host based raid, without a
write-back cache in the raid card, gives numbers about 1/2 the above.
the IOPS during these tests hit around 12000 or 15000 small writes/second.

doing this level of IO on a midsized SAN can often cause the SAN CPU to
run at 80%+ so if there's other activity on the SAN from other hosts,
good luck.

in a heavily virtualized shared-everything environment, I'm guessing
your numbers will be all over the place and difficult to achieve
consistency.

--
john r pierce                            N 37, W 122
santa cruz ca                         mid-left coast

I will be setting up an instance in the coming days and post the results here.

Finally I got time to setup an instance and do some tests.

Le 2012-09-12 à 17:08, Sébastien Lorion a écrit :

> As you can see, I am nowhere near the results John mentioned for a 10,000 scale (about 8000 tps) and I am not sure
why.My instance setup and configuration should be ok, but I am far from an expert (a startup founder has to wear many
hats...),I simply followed advice found in Greg Smith book and what I read on the net. So, if anyone can offer insight
asto why the performance is not as good as expected, please let me know .. 
>
> I did not terminate the AMI yet, so I can do more testing and/or try suggestions to improve the results. I will also
tryto run the benchmarks again on a pure RAID1 configuration with fsync off, which I will use for read-only databases. 

I wonder if work_mem is too high? That's 1 GB per connection (max), but still a lot of RAM. Might want to try with more
reasonablevalues, such as 16MB to 64MB. 

What are the iostat / vmstat numbers during the test?

Bye,
François

On 09/12/12 3:17 PM, François Beausoleil wrote:
> What are the iostat / vmstat numbers during the test?

note you need to run iostat with -x <interval>    and ignore the first
sample as its average since reboot.   I usually use 5, 10, or 30 second
intervals when analyzing IO performance problems.

on a system with multiple storage devices, the device list can get long
and confusing, I'll often specify just the device(s) I'm investigating...

     iostat -x 5 sdc sdd

ditto, vmstat needs a <interval> argument, or it shows average since reboot

--
john r pierce                            N 37, W 122
santa cruz ca                         mid-left coast

I agree 1GB is a lot, I played around with that value, but it hardly makes a difference. Is there a plateau in how that value affects query performance ? On a master DB, I would set it low and raise as necessary, but what would be a good average value on a read-only DB with same spec and max_connections ?

On 09/12/12 4:03 PM, Sébastien Lorion wrote:
> I agree 1GB is a lot, I played around with that value, but it hardly
> makes a difference. Is there a plateau in how that value affects query
> performance ? On a master DB, I would set it low and raise as
> necessary, but what would be a good average value on a read-only DB
> with same spec and max_connections ?

a complex query can require several times work_mem for sorts and hash
merges.     how many queries do you expect to ever be executing
concurrently?   I'll take 25% of my system memory and divide it by
'max_connections' and use that as work_mem for most cases.

on a large memory system doing dedicated transaction processing, I
generally shoot for about 50% of the server memory as disk cache, 1-2GB
as shared_buffers, 512MB-2GB as maintenance_work_mem, and 20-25% as
work_mem (divided by max_connections)



--
john r pierce                            N 37, W 122
santa cruz ca                         mid-left coast

You set shared_buffers way below what is suggested in Greg Smith book (25% or more of RAM) .. what is the rationale behind that rule of thumb ? Other values are more or less what I set, though I could lower the effective_cache_size and vfs.zfs.arc_max and see how it goes.

On 09/12/12 4:49 PM, Sébastien Lorion wrote:
> You set shared_buffers way below what is suggested in Greg Smith book
> (25% or more of RAM) .. what is the rationale behind that rule of
> thumb ? Other values are more or less what I set, though I could lower
> the effective_cache_size and vfs.zfs.arc_max and see how it goes.

I think those 25% rules were typically created when ram was no more than
4-8GB.

for our highly transactional workload, at least, too large of a
shared_buffers seems to slow us down, perhaps due to higher overhead of
managing that many 8k buffers.    I've heard other read-mostly
workloads, such as data warehousing, can take advantage of larger buffer
counts.



--
john r pierce                            N 37, W 122
santa cruz ca                         mid-left coast

The DB back-end of my application has 2 use cases: 

Ok, make sense .. I will update that as well and report back. Thank you for your advice.

Is dedicating 2 drives for WAL too much ? Since my whole raid is comprised of SSD drives, should I just put it in the main pool ?

One more question .. I could not set wal_sync_method to anything else but fsync .. is that expected or should other choices be also available ? I am not sure how the EC2 SSD cache flushing is handled on EC2, but I hope it is flushing the whole cache on every sync .. As a side note, I got corrupted databases (errors about pg_xlog directories not found, etc) at first when running my tests, and I suspect it was because of vfs.zfs.cache_flush_disable=1, though I cannot prove it for sure.

Forgot to say that this is it with new values suggested (see included postgresql.conf) and ARC cache size set to 32GB.

I recreated the DB and WAL pools, and launched pgbench -i -s 10000. Here are the stats during the load (still running):

iostat (xbd13-14 are WAL zpool)

zpool iostat (db pool)
pool        alloc   free   read  write   read  write
db          11.1G   387G      0  6.62K      0  62.9M

vmstat
top


On Wed, Sep 12, 2012 at 9:02 PM, Sébastien Lorion <sl@thestrangefactory.com> wrote:
>
> One more question .. I could not set wal_sync_method to anything else but fsync .. is that expected or should other choices be also available ? I am not sure how the EC2 SSD cache flushing is handled on EC2, but I hope it is flushing the whole cache on every sync .. As a side note, I got corrupted databases (errors about pg_xlog directories not found, etc) at first when running my tests, and I suspect it was because of vfs.zfs.cache_flush_disable=1, though I cannot prove it for sure.
>
> Sébastien
>
>
> On Wed, Sep 12, 2012 at 8:49 PM, Sébastien Lorion <sl@thestrangefactory.com> wrote:
>>
>> Is dedicating 2 drives for WAL too much ? Since my whole raid is comprised of SSD drives, should I just put it in the main pool ?
>>
>> Sébastien
>>
>>
>> On Wed, Sep 12, 2012 at 8:28 PM, Sébastien Lorion <sl@thestrangefactory.com> wrote:
>>>
>>> Ok, make sense .. I will update that as well and report back. Thank you for your advice.
>>>
>>> Sébastien
>>>
>>>
>>> On Wed, Sep 12, 2012 at 8:04 PM, John R Pierce <pierce@hogranch.com> wrote:
>>>>
>>>> On 09/12/12 4:49 PM, Sébastien Lorion wrote:
>>>>>
>>>>> You set shared_buffers way below what is suggested in Greg Smith book (25% or more of RAM) .. what is the rationale behind that rule of thumb ? Other values are more or less what I set, though I could lower the effective_cache_size and vfs.zfs.arc_max and see how it goes.
>>>>
>>>>
>>>> I think those 25% rules were typically created when ram was no more than 4-8GB.
>>>>
>>>> for our highly transactional workload, at least, too large of a shared_buffers seems to slow us down, perhaps due to higher overhead of managing that many 8k buffers.    I've heard other read-mostly workloads, such as data warehousing, can take advantage of larger buffer counts.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> john r pierce                            N 37, W 122
>>>> santa cruz ca                         mid-left coast
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
>>>> To make changes to your subscription:
>>>> http://www.postgresql.org/mailpref/pgsql-general
>>>
>>>
>>
>

On 09/12/12 10:01 PM, Sébastien Lorion wrote:
> pgbench initialization has been going on for almost 5 hours now and
> still stuck before vacuum starts .. something is definitely wrong as I
> don't remember it took so long first time I created the db....

pgbench initialization with a high scale factor, like the -s 10000 I
frequently use, does take quite a few hours.    you need a large
maintenance_work_mem, or the create index phase will take a really long
time to index the 150GB worth of tables its created.



--
john r pierce                            N 37, W 122
santa cruz ca                         mid-left coast

pgbench initialization has been going on for almost 5 hours now and still stuck before vacuum starts .. something is definitely wrong as I don't remember it took so long first time I created the db. Here are the current stats now:

maintenance_work_mem is already 4GB. How large should it be during load then ?

I started db creation over, this time with 16GB maintenance_work_mem and fsync=off and it does not seem to have a great effect. After again 5 hours, during index creation, disk and cpu are barely used: 95% idle and 2-3 MB/s writes (150 reads/s, 90 writes/s).

On 09/13/12 2:08 PM, Sébastien Lorion wrote:
> I started db creation over, this time with 16GB maintenance_work_mem
> and fsync=off and it does not seem to have a great effect. After again
> 5 hours, during index creation, disk and cpu are barely used: 95% idle
> and 2-3 MB/s writes (150 reads/s, 90 writes/s).

I've never had to set maintenance_work_mem any higher than 1gb for
plenty good enough performance.

whats the %busy on the disk ?   if you have a slow disk device (such as
a shared virtual disk), 90 write/sec may be all its good for. MB/s is
fairly meaningless when dealing with random committed writes.





--
john r pierce                            N 37, W 122
santa cruz ca                         mid-left coast

Disks are doing 150 read + 90 write ops/s when they should be able to do a total of 1000 iops each as currently configured (this is the max that can be set). Total bandwidth is 1000mb/s each too. So clearly, either there is something wrong with ZFS/FreeBSD on Amazon (either because of config or something deeper) or PostgreSQL is not fully utilizing the hardware, again because of config or some other issue.

Just saw your email between all the others .. Pinterest, Instagram, Netflix, Shazam, NASDAQ, Cycle Computing (http://arstechnica.com/business/2011/09/30000-core-cluster-built-on-amazon-ec2-cloud/) .. that list could go on and on, see http://aws.amazon.com/solutions/case-studies/ for some more.