Thread: Great Bridge benchmarks?

Great Bridge benchmarks?

From
Andrew Sullivan
Date:
Hi,

I know that Great Bridge did a TPC-C-like benchmark of Postgres.  I
also know that, alas, GB is closed.  Does anyone know if the
benchmark report that GB had online is available anywhere still?
It'd be useful to me in my never-ending battle to fight off the
Oracle-wielding hordes.

Thanks,
A
--
----
Andrew Sullivan                               87 Mowat Avenue
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M6K 3E3
                                         +1 416 646 3304 x110


Re: Great Bridge benchmarks?

From
"Ned Lilly"
Date:
Andrew,

I found this in my email archive.  HTH...

Regards,
Ned



> Open source database routs competition in new benchmark tests
>
> PostgreSQL meets or exceeds speed and scalability of proprietary database leaders,
>
> and significantly surpasses open source competitors
>
>
> NORFOLK, Va, August 14, 2000 -PostgreSQL, the world's most advanced open source
> database, routed the competition in recent benchmark testing, topping the
> proprietary database leaders in industry-standard transaction-processing tests.
> PostgreSQL, also known as "Postgres," is an object-relational database management
> system (DBMS) that newly formed Great Bridge LLC will professionally market,
> service and support. Postgres also consistently outperformed open source
> competitors, including MySQL and Interbase, in the benchmark tests. Great Bridge
> will market Postgres-based open source solutions as a highly reliable and lower
> cost option for businesses seeking an alternative to proprietary databases.
>
> On the ANSI SQL Standard Scalable And Portable (AS3AP) benchmark, a rudimentary
> information retrieval test that measures raw speed and scalability, Postgres
> performed an average of four to five times faster than every other database
> tested, including two major proprietary DBMS packages, the MySQL open source
> database, and Interbase, a formerly proprietary product which was recently made
> open source by Inprise/Borland. (See Exhibit 1)
>
> In the Transaction Processing Council's TPC-C test, which simulates a real-world
> online transaction processing (OLTP) environment, Postgres consistently matched
> the performance of the two leading proprietary database applications. (See Exhibit
> 2)  The two industry leaders cannot be mentioned by name because their restrictive
> licensing agreements prohibit anyone who buys their closed-source products from
> publishing their company names in benchmark testing results without the companies'
> prior approval.
>
> "The test results show that Postgres is a robust, well-built product that must be
> considered in the same category as enterprise-level competition," said Robert
> Gilbert, Great Bridge President and CEO. "Look at the trendlines in the AS3AP
> test:  Postgres, like the proprietary leaders, kept a relatively consistent output
> level all the way up to 100 concurrent users - and that output was four to five
> times faster than the proprietary products.  Interbase and MySQL fell apart under
> heavy usage.  That's a strong affirmation that Postgres today is a viable
> alternative to the market-leading proprietary databases in terms of performance
> and scalability-and the clear leader among open source databases."
>
> The tests were conducted by Xperts Inc. of Richmond, Virginia, an independent
> technology solutions company, using Quest Software's Benchmark Factory
> application.  Both the AS3AP and the TPC-C benchmarks simulated transactions by
> one to 100 simultaneous users in a client-server environment. One hundred
> concurrent users approximates the middle range of a traditional database user
> pool; many applications never see more than a few users on the system at any given
> time, while other more sophisticated enterprise platforms number concurrent users
> in the thousands.  In a Web-based application, where the connection to the
> database is measured in milliseconds, 100 simultaneous users would represent a
> substantial load-the equivalent of 100 customers hitting the "submit" button on an
> order form at exactly the same time.
>
> The AS3AP test measures raw database data retrieval power, showing an
> application's scalability, portability and ease of use and interpretation through
> the use of simple ANSI standard SQL queries.  The TPC-C test simulates a warehouse
> distribution system, including order creation, customer payments, order status
> checking, delivery, and inventory management.
>
> "What stood out for us was the consistent performance of Postgres, which stayed
> the same or tested better than those of the leading proprietary applications.
> Postgres performed consistently whether it was being used by one or 100 people,"
> said Richard Brosnahan, senior software developer at Xperts.
>
> Postgres is a standards-based object-relational SQL database designed for
> e-business and enterprise applications. The software is open source and freely
> owned, continuously augmented by a global collaborative community of elite
> programmers who volunteer their time and expertise to improve the product. In the
> last two years, with the introduction of versions 6.5 and 7.0 of the software,
> Postgres  has seen rapid enhancement through a series of high-level refinements.
>
> "Postgres' performance is a powerful affirmation of the open source method of
> development," said Gilbert of Great Bridge. "Hundreds, even thousands, of open
> source developers work on this software, demonstrating a rate of innovation and
> improvement that the proprietary competition simply can't match.  And it's only
> going to get better."
>
> A closer look
>
> Xperts ran the benchmark tests on Compaq Proliant ML350 servers with 512 mb of RAM
> and two 18.2 Gb hard disks, equipped with Intel Pentium III processors and Red Hat
> Linux 6.1 and Windows NT operating systems.  The company ensured the tests'
> consistency by using the same computers for each test, with each product
> connecting to the tests through its own preferred ODBC driver.  While Benchmark
> Factory does provide native drivers for some commercial databases, using each
> product's own ODBC ensured the most valid "apples to apples" comparison.
>
> In the AS3AP tests, PostgreSQL 7.0 significantly outperformed both the leading
> commercial and open source applications in speed and scalability.  In the tested
> configuration, Postgres peaked at 1127.8 transactions per second with five users,
> and still processed at a steady rate of 1070.4 with 100 users. The proprietary
> leader also performed consistently, with a high of 314.15 transactions per second
> with eight users, which fell slightly to 288.37 transactions per second with 100
> users. The other leading proprietary database also demonstrated consistency,
> running at 200.21 transactions per second with six users and 197.4 with 100.
>
> The other databases tested against the AS3AP benchmarks, open source competitors
> MySQL 3.22 and Interbase 6.0, demonstrated some speed with a low number of users
> but a distinct lack of scalability. MySQL reached a peak of 803.48 with two users,
> but its performance fell precipitously under the stress of additional users to a
> rate of 117.87 transactions per second with 100 users. Similarly, Interbase
> reached 424 transactions per second with four users, but its performance declined
> steadily with additional users, dropping off to 146.86 transactions per second
> with 100 users.
>
> "It's just astounding, and unexpected," said Xperts' Brosnahan of Postgres'
> performance. "I ran the test twice to make sure it was running right. Postgres is
> just a really powerful database."
>
> In the TPC-C tests, Postgres performed neck and neck with the two leading
> proprietary databases.  The test simultaneously runs five different types of
> simulated transactions; the attached graph of test results (Exhibit 2) shows
>
> steadily ascending intertwined lines representing all three databases, suggesting
> the applications scaled at comparable rates. With all five transactions running
> with 100 users, the three databases performed at a rate of slightly above five
> transactions per second.
>
> "The TPC-C is a challenging test with five transactions running at once while
> querying against the database and the stress of a growing number of users. It
> showed that all the databases we tested handle higher loads very well, the way
> they should," Brosnahan explained.
>
> Neither Interbase nor MySQL could be tested for TPC-C benchmarks. MySQL could not
> run the test because the application is not adequately compliant with minimal ANSI
> SQL standards set in 1992. Interbase 6.0, recently released as open source, does
> not have a stable ODBC driver yet; while Xperts was able to adapt the version 5
> ODBC driver for the AS3AP tests, the TPC-C test would not run.  "With MySQL it's
> an inherent design issue. Interbase 6 should run the TPC-C test, and perhaps would
> with tweaking of the test's code," said Brosnahan.
>
> Great Bridge's Gilbert attributes Postgres' high performance to a quality
> differential that comes from the open source development process; the source code
> for Postgres has been subjected to years of rigorous peer review by some of the
> best programmers in the world, many of whom use the product in their work
> environments.  "Great Bridge believes that Postgres is by far the most robust open
> source database available.  These tests provide strong affirmation of that
> belief," he said.  The company intends to work with hardware vendors and other
> interested parties to continue larger-scale testing of Postgres and other leading
> open source technologies.
>
> About Great Bridge
>
> Great Bridge LLC provides open source solutions powered by PostgreSQL, the world's
> most advanced open source database.  Great Bridge delivers value-added open source
> software and support services based on PostgreSQL, empowering e-business builders
> with an enterprise-class database and tools at a fraction of the cost of closed,
> proprietary alternatives.
>
> Headquartered in Norfolk, Virginia, Great Bridge is a privately held company
> funded by Landmark Communications, Inc., the media company that also owns The
> Weather Channel, weather.com, and national and international interests in
> newspapers, broadcasting, electronic publishing, and interactive media.
>
> # # #
>
>
>


--------------------------------------------------------------------------------






--------------------------------------------------------------------------------





Attachment

Why is pgbench suddenly slow?

From
Steve Lane
Date:
Hello all:

I know this is a hard question to answer, but I'm seeing some behavior
that's very odd. I have a postgresql application, with a PHP interface,
running on a single server (PowerMac G4 with 512 meg of RAM and a relatively
old version of LinuxPPC). Withno one on the system, my pgbench results vary
between 70-100 tps.

I just set up a big brand new server, tons of RAM, 15KRPM drive, two
processors, the works. Then I decided to run pgbench to see how much speed
I'd gained. The result? A consistent 30-33 tps, though it scaled very well
and stayed right at that level across loads from 1 to 64 connections.

What on earth could be going on? The new server has 2G of ram and I set
aside about 512 meg for shared memory.

Any ideas?


- steve


Re: Why is pgbench suddenly slow?

From
Martijn van Oosterhout
Date:
Use hdparm to check that your hard disks are going as fast as you expect
them to.

If you are loggin via syslog, check that syslog isn't executing a sync()
after each message. This kills performance majorly.

Use ps to see if all the memory was found and what the processor usage is.
What is the bottleneck (ram,cpu,disk or memory)

hope this helps,

On Fri, Aug 23, 2002 at 11:17:58PM -0500, Steve Lane wrote:
> Hello all:
>
> I know this is a hard question to answer, but I'm seeing some behavior
> that's very odd. I have a postgresql application, with a PHP interface,
> running on a single server (PowerMac G4 with 512 meg of RAM and a relatively
> old version of LinuxPPC). Withno one on the system, my pgbench results vary
> between 70-100 tps.
>
> I just set up a big brand new server, tons of RAM, 15KRPM drive, two
> processors, the works. Then I decided to run pgbench to see how much speed
> I'd gained. The result? A consistent 30-33 tps, though it scaled very well
> and stayed right at that level across loads from 1 to 64 connections.
>
> What on earth could be going on? The new server has 2G of ram and I set
> aside about 512 meg for shared memory.
>
> Any ideas?
>
>
> - steve
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly

--
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> There are 10 kinds of people in the world, those that can do binary
> arithmetic and those that can't.