Re: TPC (was Great Bridge benchmark results for Postgres, 4others) - Mailing list pgsql-general

Hi Alex,

Absolutely, as I said, we did this benchmarking for our own internal due diligence in
understanding PostgreSQL's capabilities.  It's not intended to be a formal big-iron TPC
test, like you see at tpc.org.  The software we used was one commercial vendor's
implementation of the published AS3AP and TPC-C specs - it's the same one used by a lot
of trade magazines.

Benchmarking will be a significant part of Great Bridge's ongoing contribution to the
PostgreSQL community - starting with these relatively simple tests, and scaling up to
larger systems over time.

Regards,
Ned

Alex Pilosov wrote:

> A more interesting benchmark would be to compare TPC/C results on same
> kind of hardware other vendors use for THEIR TPC benchmarks, which are
> posted on tpc.org, as well as comparing price/performance of each.
>
> TPC as run by company 'commissioned by GB' cannot be validated and
> accepted into TPC database, they must be run under close supervision by
> TPC-approved monitors. I hope GB actually springs for the price of running
> the REAL TPC benchmark (last I heard it was around 25k$).
>
> To see how postgres performs on low-end (for TPC low-end is <8 processors)
> would be interesting to say the least.
>
> A problem with a real TPC is the strong suggestion to run a transaction
> manager, to improve speed. No transaction manager supports postgres yet.
>
> Another note on TPC is that they require to include as a final price
> support contract, on which GreatBridge should be able to compete.
>
> On Mon, 14 Aug 2000, Ned Lilly wrote:
>
> > Marc's right... we opted for ODBC to ensure as much of an "apples to apples"
> > comparison as possible.  Of the 5 databases we tested, a native driver existed for
> > only the two (ahem) unnamed proprietary products - Postgres, Interbase, and MySQL
> > had to rely on ODBC.  So we used the vendor's own ODBC for each of the other two
> > cases.
> >
> > <disclaimer>
> > As with all benchmarks, your mileage will vary according to hardware, OS, and of
> > course the specific application.  What we attempted to do here was use two
> > industry-standard benchmarks and treat all five products the same.
> > </disclaimer>
> >
> > Presumably, if the vendor had taken the time to write a native driver for
> > Postgres, the results would have seen an even bigger kick.  We don't have any
> > reason to think that the results for all five tests in native driver mode would be
> > out of proportion to the results we got through ODBC.
> >
> > Regards,
> > Ned
> >
> >
> > The Hermit Hacker wrote:
> >
> > > On Mon, 14 Aug 2000, Steve Wolfe wrote:
> > >
> > > > > 1) Using only ODBC drivers.  I don't know how much of an impact a driver
> > > > can
> > > > > make but it would seem that using native drivers would shutdown one source
> > > > > of objections.
> > > >
> > > >   Using ODBC is guaranteed to slow down the benchmark.  I've seen native
> > > > database drivers beat ODBC by anywhere from a factor of two to an order of
> > > > magnitude.
> > >
> > > I haven't had a chance to take a look at the benchmarks yet, having just
> > > seen this, but *if* Great Bridge performed their benchmarks such that all
> > > the databases were access via ODBC, then they are using an
> > > 'apples-to-apples' approach, as each will have similar slowdowns as a
> > > result ...
> >
> >