Thread: Database performance comparison paper.
Some people may find this interesting reading.
http://us.devloop.org.uk/
- Marc
Excerpt from the document:
===================================================
2. What is compared here - "Apples and Oranges"
The setups are as standard as can be. The only principle guiding the installation of all the software is simplicity.
Nooptimization, no tweaks, no editing of configuration files.
===================================================
That doesn't sound like a very useful methodology for benchmarking.
Regards,
Shelby Cain
----- Original Message ----
From: Marc Evans <Marc@SoftwareHackery.Com>
To: pgsql-general@postgresql.org
Sent: Thursday, February 15, 2007 12:21:03 PM
Subject: [GENERAL] Database performance comparison paper.
Some people may find this interesting reading.
http://us.devloop.org.uk/
- Marc
---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
____________________________________________________________________________________
Any questions? Get answers on any topic at www.Answers.yahoo.com. Try it now.
On Thursday 15 February 2007 11:29, Shelby Cain <alyandon@yahoo.com> wrote: > =================================================== > 2. What is compared here - "Apples and Oranges" > The setups are as standard as can be. The only principle guiding the > installation of all the software is simplicity. No optimization, no > tweaks, no editing of configuration files. > =================================================== > > That doesn't sound like a very useful methodology for benchmarking. > In particular, it means they used MyISAM with no fsync for MySQL. They might as well have sent those inserts to /dev/null, it would have been as useful a test. They also didn't use transactions. -- When we vote for taxes, we are voting to steal from our neighbors
Shelby Cain wrote: > Excerpt from the document: > =================================================== > 2. What is compared here - "Apples and Oranges" > The setups are as standard as can be. The only principle guiding the installation of all the software is simplicity. No optimization, no tweaks, no editing of configuration files. > =================================================== > > That doesn't sound like a very useful methodology for benchmarking. Thanks for the excerpt Shelby - just saved me reading the report. -- Richard Huxton Archonet Ltd
In response to Shelby Cain <alyandon@yahoo.com>: > Excerpt from the document: > =================================================== > 2. What is compared here - "Apples and Oranges" > The setups are as standard as can be. The only principle guiding the installation of all the software is simplicity. No optimization, no tweaks, no editing of configuration files. > =================================================== > > That doesn't sound like a very useful methodology for benchmarking. The amazing thing is that PostgreSQL still compared favorably, in _spite_ of this obvious configuration bias. I'm going to have to set up a system and compare a properly tuned MySQL to a properly tuned PostgreSQL to see what happens ... -- Bill Moran Collaborative Fusion Inc.
Am 15.02.2007 um 11:21 schrieb Marc Evans: > http://us.devloop.org.uk/ These *peeeeeeep* [deleted] compared MySQL with MyISAM to ACID compliant databases. So why not compare an F-15 to 747? What? Apples and Oranges? So what? You can compare anything you want, right? Only the result matters. So, my hint to these guys is: learn about the principles of databases (at least read: <http://en.wikipedia.org/wiki/ACID>), then about the principles of optimizing databases, then about the principles of testing (don't compare products or setups that do completely different things) and then do you homework again. Go home. cug
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/15/07 15:29, Guido Neitzer wrote: > Am 15.02.2007 um 11:21 schrieb Marc Evans: > >> http://us.devloop.org.uk/ > > These *peeeeeeep* [deleted] compared MySQL with MyISAM to ACID compliant > databases. So why not compare an F-15 to 747? What? Apples and Oranges? > So what? You can compare anything you want, right? Only the result matters. Bad analogy. Both the F-15 and 747 are high-performance (within their problem domains) and have redundancy out the wazoo. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFF1NVyS9HxQb37XmcRAubPAKDUOQ6n38YnGWhZTIHZM3zyTDFBDQCfYvyn 3Wdim4mnuFXn0hIPEHGu5Vw= =nvPe -----END PGP SIGNATURE-----
Ron Johnson <ron.l.johnson@cox.net> writes:
> -----BEGIN PGP SIGNED MESSAGE-----
>> Am 15.02.2007 um 11:21 schrieb Marc Evans:
>> These *peeeeeeep* [deleted] compared MySQL with MyISAM to ACID compliant
>> databases. So why not compare an F-15 to 747? What? Apples and Oranges?
> Bad analogy. Both the F-15 and 747 are high-performance (within
> their problem domains) and have redundancy out the wazoo.
I think it's a fine analogy, precisely because they're both good in
their respective problem domains. Try to carry 500 people from Los
Angeles to Tokyo in an F-15. No? Try to win a dogfight in a 747. No?
But they both fly, so it must be useful to compare them... especially
on the basis of the most simplistic test case you can think of. For
extra points, use *only one* test case. Perhaps this paper can be
described as "comparing an F-15 to a 747 on the basis of required
runway length".
regards, tom lane
On Friday 16. February 2007 07:10, Tom Lane wrote: > Perhaps this > paper can be described as "comparing an F-15 to a 747 on the basis of > required runway length". There ought to be a proper name for this kind of pseudo-technical Gonzo journalism. The Internet is full of it. -- Leif Biberg Kristensen | Registered Linux User #338009 http://solumslekt.org/ | Cruising with Gentoo/KDE My Jazz Jukebox: http://www.last.fm/user/leifbk/
Marc Evans schrieb: > Some people may find this interesting reading. > > http://us.devloop.org.uk/ Nice, but it would be interesting which storage engine was used for mysql - ok, default is MyIsam. Does mysql (in the latest version) still use a single write-thread for writing? In mysql 3, a badly formed query which writes millions of tuples to a temporary table blocks the database for the whole query execution time. Greetings, Alexander
Richard Huxton schrieb: > Shelby Cain wrote: >> Excerpt from the document: >> =================================================== >> 2. What is compared here - "Apples and Oranges" >> The setups are as standard as can be. The only principle guiding the >> installation of all the software is simplicity. No optimization, no >> tweaks, no editing of configuration files. >> =================================================== >> >> That doesn't sound like a very useful methodology for benchmarking. > > Thanks for the excerpt Shelby - just saved me reading the report. > There is no much to read - just look at the images. ;) Page 24 make the point.
Am 15.02.2007 um 13:05 schrieb Alexander Elgert: > Nice, but it would be interesting which storage engine was used for > mysql - ok, default is MyIsam. They used MyISAM as it is described late in the paper. cug
Leif B. Kristensen wrote: > On Friday 16. February 2007 07:10, Tom Lane wrote: > >> Perhaps this >> paper can be described as "comparing an F-15 to a 747 on the basis of >> required runway length". > > There ought to be a proper name for this kind of pseudo-technical Gonzo > journalism. The Internet is full of it. advertalism?
On Sat, Feb 17, 2007 at 12:02:08AM +0100, Leif B. Kristensen wrote:
>
> There ought to be a proper name for this kind of pseudo-technical Gonzo
> journalism.
There is, but it's not the sort of word one uses in polite company
;-)
A
--
Andrew Sullivan | ajs@crankycanuck.ca
Unfortunately reformatting the Internet is a little more painful
than reformatting your hard drive when it gets out of whack.
--Scott Morris
Tom Allison wrote: > Leif B. Kristensen wrote: >> On Friday 16. February 2007 07:10, Tom Lane wrote: >> >>> Perhaps this >>> paper can be described as "comparing an F-15 to a 747 on the basis of >>> required runway length". >> >> There ought to be a proper name for this kind of pseudo-technical >> Gonzo journalism. The Internet is full of it. > > advertalism? Lies? -- Until later, Geoffrey Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. - Benjamin Franklin
On 2/16/2007 1:10 AM, Tom Lane wrote: > extra points, use *only one* test case. Perhaps this paper can be > described as "comparing an F-15 to a 747 on the basis of required > runway length". Oh, this one wasn't about raw speed of trivial single table statements like all the others? Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
Am 19.02.2007 um 17:49 schrieb Jan Wieck: > Oh, this one wasn't about raw speed of trivial single table > statements like all the others? No, it wasn't. They also tested the insert performance of a system without foreign keys and without transactions (MySQL MyISAM) against systems with foreign key handling and transactions. It would be more or less the same, if you compare copy against insert performance on PostgreSQL and state that insert should be as fast as copy without saying why. Btw: these guys claim to be database consultants. cug
On 2/21/07, Guido Neitzer wrote: > It would be more or less the same, if you compare copy against insert > performance on PostgreSQL and state that insert should be as fast as > copy without saying why. > > Btw: these guys claim to be database consultants. Guess one should consider oneself lucky not to be their customer, then, since they seem to base their decisions on thin air and personal preference... > cug Cheers, Andrej
On 2/20/2007 3:51 PM, Andrej Ricnik-Bay wrote: > On 2/21/07, Guido Neitzer wrote: > >> It would be more or less the same, if you compare copy against insert >> performance on PostgreSQL and state that insert should be as fast as >> copy without saying why. >> >> Btw: these guys claim to be database consultants. > Guess one should consider oneself lucky not to be their > customer, then, since they seem to base their decisions > on thin air and personal preference... As the original author of the PHP TPC-W implementation you can find on pgfoundry, I know pretty good what it takes to make MySQL perform about as good as PostgreSQL under a real benchmarking scenario. I implemented all the database access parts basically two times. Once for PostgreSQL as an experienced DB developer would do it, once turning half the queries upside down in a horribly unintuitive way to give MySQL+InnoDB clues how to do it. Of course did I NOT run any of those tests using MyISAM. In the end, both implementations performed more or less the same, measured at the HTTP interface. What the PHP+PG implementation did more elegantly in SQL, the PHP+My implementation had to do with more PHP code. And that is where all those crappy wannabe-benchmarks just fail to make sense to me. They measure some common denominator SQL statements at an abstracted DB API level. That is just nonsense. It doesn't matter how fast a specific index scan or a specific insert or update operation by itself is. What matters is how many parallel simulated users of a well defined business application the System Under Test (middleware plus database) can support within the responsetime constraints. All that said, what really scares me is that these clowns apparently don't even know the system of their preference. No serious DB consultant would even bother testing anything using MyISAM any more. It is a table handler only considered for "disposable data". Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #