Re: PostgreSQL TPC-H test result? - Mailing list pgsql-general

On Tuesday 09 September 2008 10:06:01 Amber wrote:
> From: "Andrew Sullivan" <ajs@commandprompt.com>
> Sent: Tuesday, September 09, 2008 8:39 PM
> To: <pgsql-general@postgresql.org>
> Subject: Re: [GENERAL] PostgreSQL TPC-H test result?
>
> > On Tue, Sep 09, 2008 at 07:59:49PM +0800, Amber wrote:
> >> I read something from
> >> http://monetdb.cwi.nl/projects/monetdb/SQL/Benchmark/TPCH/index.html
> >
> > Given that the point of that "study" is to prove something about
> > performance, one should be leery of any claims based on an "out of the
> > box" comparison.  Particularly since the "box" their own product comes
> > out of is "compiled from CVS checkout".  Their argument seems to be
> > that people can learn how to drive CVS and to compile software under
> > active development, but can't read the manual that comes with Postgres
> > (and a release of Postgres well over a year old, at that).
> >
> > I didn't get any further in reading the claims, because it's obviously
> > nothing more than a marketing effort using the principle that deriding
> > everyone else will make them look better.  Whether they have a good
> > product is another question entirely.
> >
> > >Yes, we don't care about the performance results, but we do care
> > >about the
> > > point that PostgreSQL can't give the correct results of TPC-H queries.

Given the point of those benchmarks is to make other systems look bad, I think
you have to take them with a grain of salt. Since we don't know what the
errors/results were, and no information is giving, we are left to wonder if
this is a problem with the software or the tester. The site would have us
believe the former, but I think I would lean toward the latter... case in
point, I did a quick google and turned up this link:
http://www.it.iitb.ac.in/~chetanv/personal/acads/db/report_html/node10.html.
It isn't terribly informative, but it doesindicate one thing, someone else
was able to run query #6 correctly, while the above site claims it returns an
error. Now when I look at query#6 from that site, I notice it shows the
following syntax:

interval '1' year.

when I saw that, it jumped out at me as something that could be an issue, and
it is:

pagila=# select now() - interval '1' year, now() - interval '1 year';
           ?column?            |           ?column?
-------------------------------+-------------------------------
 2008-09-09 11:28:46.938209-04 | 2007-09-09 11:28:46.938209-04
(1 row)

Now, I'm not sure if there is an issue that monet supports the first syntax
and so when they ran thier test on postgres this query produced wrong
results, but that seems possible. In this case I would wonder if the first
syntax is sql compliant, but it doesn't really matter, the tpc-h allows for
changes to queries to support syntax variations between databases; I'm pretty
sure I could make suttle changes to "break" other databases as well.

Incidentally, I poked Mark Wong, who used to work at the OSDL (big linux
kernel hacking shop), and he noted he has successfully run the tpc-h tests
before on postgres.

In the end, I can't speak to what the issues are wrt monet and postgres and
thier tpc-h benchmarks, but personally I don't think they are worth worring
about.

--
Robert Treat
http://www.omniti.com
Database: Scalability: Consulting: