Great Bridge benchmark results for Postgres, 4 others - Mailing list pgsql-general

Greetings all,

At long last, here are the results of the benchmarking tests that
Great Bridge conducted in its initial exploration of PostgreSQL.  We
held it up so we could test the shipping release of the new
Interbase 6.0.  This is a news release that went out today.

The release is also on our website at
http://www.greatbridge.com/news/p_081420001.html.  Graphics of the
AS3AP and TPC-C test results are at
http:/www.greatbridge.com/img/as3ap.gif and
http://www.greatbridge.com/img/tpc-c.gif respectively.

I'll try and field any questions anyone has, or refer you to someone
who can.

Best regards,

Ned Lilly
VP Hacker Relations
Great Bridge, LLC

--

Open source database routs competition in new benchmark tests

PostgreSQL meets or exceeds speed and scalability of proprietary
database leaders, and significantly surpasses open source
competitors


NORFOLK, Va, August 14, 2000 -PostgreSQL, the world's most advanced
open source database, routed the competition in recent benchmark
testing, topping the proprietary database leaders in
industry-standard transaction-processing tests. PostgreSQL, also
known as "Postgres," is an object-relational database management
system (DBMS) that newly formed Great Bridge LLC will professionally
market, service and support. Postgres also consistently outperformed
open source competitors, including MySQL and Interbase, in the
benchmark tests. Great Bridge will market Postgres-based open source
solutions as a highly reliable and lower cost option for businesses
seeking an alternative to proprietary databases.

On the ANSI SQL Standard Scalable And Portable (AS3AP) benchmark, a
rudimentary information retrieval test that measures raw speed and
scalability, Postgres performed an average of four to five times
faster than every other database tested, including two major
proprietary DBMS packages, the MySQL open source database, and
Interbase, a formerly proprietary product which was recently made
open source by Inprise/Borland. (See Exhibit 1)

In the Transaction Processing Council's TPC-C test, which simulates
a real-world online transaction processing (OLTP) environment,
Postgres consistently matched the performance of the two leading
proprietary database applications. (See Exhibit 2)  The two industry
leaders cannot be mentioned by name because their restrictive
licensing agreements prohibit anyone who buys their closed-source
products from publishing their company names in benchmark testing
results without the companies' prior approval.

"The test results show that Postgres is a robust, well-built product
that must be considered in the same category as enterprise-level
competition," said Robert Gilbert, Great Bridge President and CEO.
"Look at the trendlines in the AS3AP test:  Postgres, like the
proprietary leaders, kept a relatively consistent output level all
the way up to 100 concurrent users - and that output was four to
five times faster than the proprietary products.  Interbase and
MySQL fell apart under heavy usage.  That's a strong affirmation
that Postgres today is a viable alternative to the market-leading
proprietary databases in terms of performance and scalability-and
the clear leader among open source databases."

The tests were conducted by Xperts Inc. of Richmond, Virginia, an
independent technology solutions company, using Quest Software's
Benchmark Factory application.  Both the AS3AP and the TPC-C
benchmarks simulated transactions by one to 100 simultaneous users
in a client-server environment. One hundred concurrent users
approximates the middle range of a traditional database user pool;
many applications never see more than a few users on the system at
any given time, while other more sophisticated enterprise platforms
number concurrent users in the thousands.  In a Web-based
application, where the connection to the database is measured in
milliseconds, 100 simultaneous users would represent a substantial
load-the equivalent of 100 customers hitting the "submit" button on
an order form at exactly the same time.

The AS3AP test measures raw database data retrieval power, showing
an application's scalability, portability and ease of use and
interpretation through the use of simple ANSI standard SQL queries.
The TPC-C test simulates a warehouse distribution system, including
order creation, customer payments, order status checking, delivery,
and inventory management.

"What stood out for us was the consistent performance of Postgres,
which stayed the same or tested better than those of the leading
proprietary applications. Postgres performed consistently whether it
was being used by one or 100 people," said Richard Brosnahan, senior
software developer at Xperts.

Postgres is a standards-based object-relational SQL database
designed for e-business and enterprise applications. The software is
open source and freely owned, continuously augmented by a global
collaborative community of elite programmers who volunteer their
time and expertise to improve the product. In the last two years,
with the introduction of versions 6.5 and 7.0 of the software,
Postgres  has seen rapid enhancement through a series of high-level
refinements.

"Postgres' performance is a powerful affirmation of the open source
method of development," said Gilbert of Great Bridge. "Hundreds,
even thousands, of open source developers work on this software,
demonstrating a rate of innovation and improvement that the
proprietary competition simply can't match.  And it's only going to
get better."

A closer look

Xperts ran the benchmark tests on Compaq Proliant ML350 servers with
512 mb of RAM and two 18.2 Gb hard disks, equipped with Intel
Pentium III processors and Red Hat Linux 6.1 and Windows NT
operating systems.  The company ensured the tests' consistency by
using the same computers for each test, with each product connecting
to the tests through its own preferred ODBC driver.  While Benchmark
Factory does provide native drivers for some commercial databases,
using each product's own ODBC ensured the most valid "apples to
apples" comparison.

In the AS3AP tests, PostgreSQL 7.0 significantly outperformed both
the leading commercial and open source applications in speed and
scalability.  In the tested configuration, Postgres peaked at 1127.8
transactions per second with five users, and still processed at a
steady rate of 1070.4 with 100 users. The proprietary leader also
performed consistently, with a high of 314.15 transactions per
second with eight users, which fell slightly to 288.37 transactions
per second with 100 users. The other leading proprietary database
also demonstrated consistency, running at 200.21 transactions per
second with six users and 197.4 with 100.

The other databases tested against the AS3AP benchmarks, open source
competitors MySQL 3.22 and Interbase 6.0, demonstrated some speed
with a low number of users but a distinct lack of scalability. MySQL
reached a peak of 803.48 with two users, but its performance fell
precipitously under the stress of additional users to a rate of
117.87 transactions per second with 100 users. Similarly, Interbase
reached 424 transactions per second with four users, but its
performance declined steadily with additional users, dropping off to
146.86 transactions per second with 100 users.

"It's just astounding, and unexpected," said Xperts' Brosnahan of
Postgres' performance. "I ran the test twice to make sure it was
running right. Postgres is just a really powerful database."

In the TPC-C tests, Postgres performed neck and neck with the two
leading proprietary databases.  The test simultaneously runs five
different types of simulated transactions; the attached graph of
test results (Exhibit 2) shows steadily ascending intertwined lines
representing all three databases, suggesting the applications scaled
at comparable rates. With all five transactions running with 100
users, the three databases performed at a rate of slightly above
five transactions per second.

"The TPC-C is a challenging test with five transactions running at
once while querying against the database and the stress of a growing
number of users. It showed that all the databases we tested handle
higher loads very well, the way they should," Brosnahan explained.

Neither Interbase nor MySQL could be tested for TPC-C benchmarks.
MySQL could not run the test because the application is not
adequately compliant with minimal ANSI SQL standards set in 1992.
Interbase 6.0, recently released as open source, does not have a
stable ODBC driver yet; while Xperts was able to adapt the version 5
ODBC driver for the AS3AP tests, the TPC-C test would not run.
"With MySQL it's an inherent design issue. Interbase 6 should run
the TPC-C test, and perhaps would with tweaking of the test's code,"
said Brosnahan.

Great Bridge's Gilbert attributes Postgres' high performance to a
quality differential that comes from the open source development
process; the source code for Postgres has been subjected to years of
rigorous peer review by some of the best programmers in the world,
many of whom use the product in their work environments.  "Great
Bridge believes that Postgres is by far the most robust open source
database available.  These tests provide strong affirmation of that
belief," he said.  The company intends to work with hardware vendors
and other interested parties to continue larger-scale testing of
Postgres and other leading open source technologies.

About Great Bridge

Great Bridge LLC provides open source solutions powered by
PostgreSQL, the world's most advanced open source database.  Great
Bridge delivers value-added open source software and support
services based on PostgreSQL, empowering e-business builders with an
enterprise-class database and tools at a fraction of the cost of
closed, proprietary alternatives.

Headquartered in Norfolk, Virginia, Great Bridge is a privately held
company funded by Landmark Communications, Inc., the media company
that also owns The Weather Channel, weather.com, and national and
international interests in newspapers, broadcasting, electronic
publishing, and interactive media.

# # #