Re: Is there a meaningful benchmark? - Mailing list pgsql-general

> -----Original Message-----
> From: pgsql-general-owner@postgresql.org [mailto:pgsql-general-
> owner@postgresql.org] On Behalf Of Will Rutherdale (rutherw)
> Sent: Thursday, March 19, 2009 1:32 PM
> To: pgsql-general@postgresql.org
> Subject: [GENERAL] Is there a meaningful benchmark?
>
> Hi.  I'm writing up a database comparison paper in my department at
> work, with Postgres being a major candidate.  I have been attempting
to
> research various issues and provide a meaningful comparison.
>
> One issue I would like to give some kind of information on is
> comparative performance.  When I look for such things in Google or
> Wikipedia or the pgsql email archives, it's hard to find anything
> reasonably definitive.  I've found isolated claims and anecdotes here
> and there, and a fellow on the list here who attempted to do a
> comparison between Postgres, MySQL, and Oracle but gave it up for now.
>
> Some of the claims I've seen said that in some cases MySQL with MyISAM
> ran 2x faster than Postgres, but that may have been for a special case
> with only read access to the database;  whereas another one claimed
> that
> MySQL with InnoDB was slower than Postgres.  Other people commented
> that
> it depends on how you tune the databases.
>
> Maybe there's nothing definitive out there.  However I'd like to get a
> ballpark idea of how some databases compare, using some kind of
average
> case schema and application, in terms of transactions per second, on a
> common hardware platform.  I would like to be able to point to a
> reasonable reference, rather than engaging in handwaving myself.
>
> Does anyone know where I could look?

The only way to get an answer to a question like this is to actually
benchmark the application you have in mind.
And the answer won't be very good unless you have an expert on each
given system install and tune the application.

There is a regular benchmark that is run against the PostgreSQL
database.  I don't remember where to find the graphs.  Probably, someone
on the list can tell us the location.


Here are some benchmark figures:
http://tweakers.net/reviews/657/6
http://www.spec.org/jAppServer2004/results/res2007q3/jAppServer2004-2007
0606-00065.html
http://www.informationweek.com/news/software/linux/showArticle.jhtml?art
icleID=201001901
http://www.kaltenbrunner.cc/blog/index.php?/archives/21-guid.html
http://benchw.sourceforge.net/benchw_results_open3.html


P.S.
PostgreSQL seems to scale pretty well:
http://www.computerworld.com/action/article.do?command=viewArticleBasic&
taxonomyId=18&articleId=9087918&intsrc=hm_topic


My opinion:
Most benchmarks are run by someone with an axe to grind.  I never
believe them.  The TPC benchmarks are probably the most trustworthy,
because they have to be certified.  But a fast TPC/whatever benchmark is
no guarantee that *your* application will run fast.  So if you want to
evaluation several different technologies do your own benchmark.  Do
your own calculations to find out the total cost of ownership over the
lifetime of the project.  Examine all the features that are available,
and what kind of technical support is possible.  Consider the impact of
product licensing.  What happens if you need to scale up to titanic
volume?  After you have thought all factors over very carefully, make
your choice.

If you rely on someone else to do the work for you, it's really begging
for trouble.  MySQL guys will show you why MySQL is faster.  PostgreSQL
guys will show you why PostgreSQL is faster.  Oracle guys will show you
why Oracle is faster.  SQL*Server guys will show you why SQL*Server is
faster.  DB/2 guys will show you why DB/2 is faster.  Now, none of them
are lying (at least hopefully) but they are experts in their own domain
and not in the domain of the other product and they are also going to
choose those tight little corners where their product has the biggest
advantage.

IMO-YMMV.
P.S.
I'm a PostgreSQL fan and so I am likely to (perhaps unconsciously) favor
PostgreSQL in my remarks.