Thread: database contest results

database contest results

From
Lukas Kahwe Smith
Date:
Hi,

I have not studied this contest in any detail. However the performance
differences seem kind of unrealistic:
http://www.mysql.com/news-and-events/press-release/release_2006_35.html

regards,
Lukas


[ignore this and previous email] Re: database contest results

From
Lukas Kahwe Smith
Date:
Lukas Kahwe Smith schrieb:
> Hi,
>
> I have not studied this contest in any detail. However the performance
> differences seem kind of unrealistic:
> http://www.mysql.com/news-and-events/press-release/release_2006_35.html

Ups, sorry for reposting this. I must have overlooked it when I checked
the archives yesterday.

regards,
Lukas

Re: database contest results

From
Hans-Juergen Schoenig
Date:
Lukas Kahwe Smith wrote:
> Hi,
>
> I have not studied this contest in any detail. However the performance
> differences seem kind of unrealistic:
> http://www.mysql.com/news-and-events/press-release/release_2006_35.html
>
> regards,
> Lukas
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
>               http://archives.postgresql.org


this is pure marketing.
i have seen postgres beat mysql in many many cases. same with oracle.
i assume that those tests are all done with ISAM. with ISAM everything
is fast but you cannot reboot the box without facing serious corruption.
in business applications stability is as least as important as speed.

as far as the mysql benchmark is concerned: it is like one of those
studies where professor marlboro says that smoking is good for health ...

    best regards,

       hans


--
Cybertec Geschwinde & Schönig GmbH
Schöngrabern 134; A-2020 Hollabrunn
Tel: +43/1/205 10 35 / 340
www.postgresql.at, www.cybertec.at


Re: database contest results

From
Andreas Pflug
Date:
Hans-Juergen Schoenig wrote:
> Lukas Kahwe Smith wrote:
>> Hi,
>>
>> I have not studied this contest in any detail. However the
>> performance differences seem kind of unrealistic:
>> http://www.mysql.com/news-and-events/press-release/release_2006_35.html
>>
>> regards,
>> Lukas
>>
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 4: Have you searched our list archives?
>>
>>               http://archives.postgresql.org
>
>
> this is pure marketing.
> i have seen postgres beat mysql in many many cases. same with oracle.
> i assume that those tests are all done with ISAM. with ISAM everything
> is fast but you cannot reboot the box without facing serious
> corruption. in business applications stability is as least as
> important as speed.
>
> as far as the mysql benchmark is concerned: it is like one of those
> studies where professor marlboro says that smoking is good for health ...
This was done by c't, not mysql. But the article title is misleading: it
doesn't compare database systems, but application tuning. Unfortunately,
the prerequisites where quite mysql-drawn right from the start. The
MySQL team did a very good job tuning the existent application to access
the database as rare as possible using memcache, and dropped all
constraints (no surprise...). Porting _that_ optimized app to pgsql
would be really interesting and comparable.

Regards,
Andreas


Re: database contest results

From
"Joshua D. Drake"
Date:
> this is pure marketing.
> i have seen postgres beat mysql in many many cases. same with oracle.

I don't care if they beat us. What is ridiculous is what they said they
beat us by.

Joshua D. Drake


> i assume that those tests are all done with ISAM. with ISAM everything
> is fast but you cannot reboot the box without facing serious corruption.
> in business applications stability is as least as important as speed.
>
> as far as the mysql benchmark is concerned: it is like one of those
> studies where professor marlboro says that smoking is good for health ...
>
>    best regards,
>
>       hans
>
>


--

    === The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
    Providing the most comprehensive  PostgreSQL solutions since 1997
              http://www.commandprompt.com/



Re: database contest results

From
Anastasios Hatzis
Date:
Andreas Pflug wrote:
> Hans-Juergen Schoenig wrote:
>
>> Lukas Kahwe Smith wrote:
>>
>>> Hi,
>>>
>>> I have not studied this contest in any detail. However the
>>> performance differences seem kind of unrealistic:
>>> http://www.mysql.com/news-and-events/press-release/release_2006_35.html
>>>
>> this is pure marketing.
>> i have seen postgres beat mysql in many many cases. same with oracle.
>> i assume that those tests are all done with ISAM. with ISAM everything
>> is fast but you cannot reboot the box without facing serious
>> corruption. in business applications stability is as least as
>> important as speed.
>>
> the prerequisites where quite mysql-drawn right from the start. The
> MySQL team did a very good job tuning the existent application to access
> the database as rare as possible using memcache, and dropped all
> constraints (no surprise...). Porting _that_ optimized app to pgsql
> would be really interesting and comparable.
>
Good Point :-)) Does it mean that a faster way working with mysql is by using mysql as little as possible? Probably
thanthe fastest way is  never using mysql? 

SCNR

Anastasios


Re: database contest results

From
Jeff Davis
Date:
On Tue, 2006-08-29 at 12:32 +0200, Hans-Juergen Schoenig wrote:

> this is pure marketing.
> i have seen postgres beat mysql in many many cases. same with oracle.
> i assume that those tests are all done with ISAM. with ISAM everything
> is fast but you cannot reboot the box without facing serious corruption.
> in business applications stability is as least as important as speed.
>

Using MyISAM alone cannot explain those differences. Supposedly it was
3000 opm versus 120 opm.

Clearly, there is a huge difference in the overall application. The
PostgreSQL entry was done quickly, and the author probably didn't
understand the terms of the contest entirely, let alone have the time to
optimize his entry.

Regards,
    Jeff Davis


Re: database contest results

From
Peter Eisentraut
Date:
Jeff Davis wrote:
> Clearly, there is a huge difference in the overall application. The
> PostgreSQL entry was done quickly, and the author probably didn't
> understand the terms of the contest entirely, let alone have the time
> to optimize his entry.

I don't think you should make these kinds of insulting judgements
without research.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Re: database contest results

From
Jeff Davis
Date:
On Tue, 2006-08-29 at 19:20 +0200, Peter Eisentraut wrote:
> Jeff Davis wrote:
> > Clearly, there is a huge difference in the overall application. The
> > PostgreSQL entry was done quickly, and the author probably didn't
> > understand the terms of the contest entirely, let alone have the time
> > to optimize his entry.
>
> I don't think you should make these kinds of insulting judgements
> without research.
>

The author himself said he didn't have time. I didn't mean to be
insulting, and I apologize if I was. 120 versus 3000 seems like the
MySQL entry guys were operating with an entirely separate set of
assumptions, and spent much more time optimizing it and determining the
exact contest requirements.

Regards,
    Jeff Davis



Re: database contest results

From
"Joshua D. Drake"
Date:
Jeff Davis wrote:
> On Tue, 2006-08-29 at 19:20 +0200, Peter Eisentraut wrote:
>> Jeff Davis wrote:
>>> Clearly, there is a huge difference in the overall application. The
>>> PostgreSQL entry was done quickly, and the author probably didn't
>>> understand the terms of the contest entirely, let alone have the time
>>> to optimize his entry.
>> I don't think you should make these kinds of insulting judgements
>> without research.
>>
>
> The author himself said he didn't have time. I didn't mean to be
> insulting, and I apologize if I was. 120 versus 3000 seems like the
> MySQL entry guys were operating with an entirely separate set of
> assumptions, and spent much more time optimizing it and determining the
> exact contest requirements.

I didn't find it insulting.

Sincerely,

Joshua D. Drake



>
> Regards,
>     Jeff Davis
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
>        subscribe-nomail command to majordomo@postgresql.org so that your
>        message can get through to the mailing list cleanly
>


--

    === The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
    Providing the most comprehensive  PostgreSQL solutions since 1997
              http://www.commandprompt.com/



Re: database contest results

From
Jeff Davis
Date:
On Tue, 2006-08-29 at 10:55 -0700, Jeff Davis wrote:
> On Tue, 2006-08-29 at 19:20 +0200, Peter Eisentraut wrote:
> > Jeff Davis wrote:
> > > Clearly, there is a huge difference in the overall application. The
> > > PostgreSQL entry was done quickly, and the author probably didn't
> > > understand the terms of the contest entirely, let alone have the time
> > > to optimize his entry.
> >
> > I don't think you should make these kinds of insulting judgements
> > without research.
> >
>
> The author himself said he didn't have time. I didn't mean to be
> insulting, and I apologize if I was. 120 versus 3000 seems like the
> MySQL entry guys were operating with an entirely separate set of
> assumptions, and spent much more time optimizing it and determining the
> exact contest requirements.
>

After re-reading my original post, it came out differently than I
intended, and I apologize to the author. My point was simply that the
author did not spend as much time as the MySQL entry participants (I'm
not criticizing the author, but just paraphrasing his words). I think
that the time the MySQL entry participants spent preparing, which
included the time they spent understanding the bounds of the contest,
had much more to do with the end result than any technical details.

We can't have a perfect entry for every contest, so I don't think it's a
big deal. If someone sees a contest in the future, they should post it
on -advocacy when they see it, so that PostgreSQL people have more time
to prepare a potential entry. Then, we can have a few extra "wins" among
the crowd that reads those articles.

Regards,
    Jeff Davis





Re: database contest results

From
Andreas Pflug
Date:
Jeff Davis wrote:
> On Tue, 2006-08-29 at 19:20 +0200, Peter Eisentraut wrote:
>
>> Jeff Davis wrote:
>>
>>> Clearly, there is a huge difference in the overall application. The
>>> PostgreSQL entry was done quickly, and the author probably didn't
>>> understand the terms of the contest entirely, let alone have the time
>>> to optimize his entry.
>>>
>> I don't think you should make these kinds of insulting judgements
>> without research.
>>
>>
>
> The author himself said he didn't have time. I didn't mean to be
> insulting, and I apologize if I was. 120 versus 3000 seems like the
> MySQL entry guys were operating with an entirely separate set of
> assumptions, and spent much more time optimizing it and determining the
> exact contest requirements.
>
Maybe you should have had a look at the article before speculating.
Contest requirement was very easy: take the DS sample and make it fast
on a given average PC hardware. The MySQL guys were able to take a
working system, others had to write a new db access or even code a new
one. MySQL had a whole (paid? full-time?) team on that, with half the
work already done. The contest was a wrong labeled app optimization
contest. Declaring it as db comparison and using it as marketing stuff
is dubious and highly misleading.

Regards,
Andreas




Re: database contest results

From
Jeff Davis
Date:
On Wed, 2006-08-30 at 00:10 +0200, Andreas Pflug wrote:
> > The author himself said he didn't have time. I didn't mean to be
> > insulting, and I apologize if I was. 120 versus 3000 seems like the
> > MySQL entry guys were operating with an entirely separate set of
> > assumptions, and spent much more time optimizing it and determining the
> > exact contest requirements.
> >
> Maybe you should have had a look at the article before speculating.
> Contest requirement was very easy: take the DS sample and make it fast
> on a given average PC hardware. The MySQL guys were able to take a

Before I posted, I read the English press release along with the thread
on this list and on pgsql-general, but I don't read German (I only found
the English translation now). I also read your statement in this thread
that the MySQL guys tuned the application "to access the database as
rare [sic] as possible using memcache."

To me, this fact alone means that the author of the PostgreSQL entry
operated under different assumptions than the author of the MySQL entry.
Even "simple" contest requirements can be interpreted differently due to
assumptions. For instance, maybe the author of the PostgreSQL entry made
the wrong assumptions because, as you put it, the contest was "wrong
labeled app optimization"?

I stand by my original statement that it was more about understanding
and adapting to the contest than anything to do with the technical
database details (like storage engines).

Regards,
    Jeff Davis




Re: database contest results

From
Andreas Pflug
Date:
Jeff Davis wrote:
> On Wed, 2006-08-30 at 00:10 +0200, Andreas Pflug wrote:
>
>>> The author himself said he didn't have time. I didn't mean to be
>>> insulting, and I apologize if I was. 120 versus 3000 seems like the
>>> MySQL entry guys were operating with an entirely separate set of
>>> assumptions, and spent much more time optimizing it and determining the
>>> exact contest requirements.
>>>
>>>
>> Maybe you should have had a look at the article before speculating.
>> Contest requirement was very easy: take the DS sample and make it fast
>> on a given average PC hardware. The MySQL guys were able to take a
>>
>
> Before I posted, I read the English press release along with the thread
> on this list and on pgsql-general, but I don't read German (I only found
> the English translation now). I also read your statement in this thread
> that the MySQL guys tuned the application "to access the database as
> rare [sic] as possible using memcache."
>
> To me, this fact alone means that the author of the PostgreSQL entry
> operated under different assumptions than the author of the MySQL entry.
> Even "simple" contest requirements can be interpreted differently due to
> assumptions. For instance, maybe the author of the PostgreSQL entry made
> the wrong assumptions because, as you put it, the contest was "wrong
> labeled app optimization"?
>
> I stand by my original statement that it was more about understanding
> and adapting to the contest than anything to do with the technical
> database details (like storage engines).
>
The MySQL guys could skip the db adaption completely, and procede to
advanced app side cache techniques. This distorted prerequisites, not
assumptions. It was clear that everybody could use any technique, only
the user interface was fixed. I'm not sure c't tested whether data was
made persistent at all, so a pure in-memory fake maybe would have worked
either (one could argue that using a db without constraints is not far
from that)

Regards,
Andreas


Re: database contest results

From
mdean
Date:
Jeff Davis wrote:

>On Wed, 2006-08-30 at 00:10 +0200, Andreas Pflug wrote:
>
>
>
>

Guys, what I need is alternative benchmark data, where postgresql comes
out better on standard assumptions, etc. than i this example  Can this
be done?
Michael



--
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.1.405 / Virus Database: 268.11.7/432 - Release Date: 8/29/2006


Re: database contest results

From
Josh Berkus
Date:
Michael,

> Guys, what I need is alternative benchmark data, where postgresql comes
> out better on standard assumptions, etc. than i this example  Can this
> be done?

Sure, we perform better on DBT2 and other ACID-transaction intensive tests.
Go for it.

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

Re: database contest results

From
"Joshua D. Drake"
Date:
mdean wrote:
> Jeff Davis wrote:
>
>> On Wed, 2006-08-30 at 00:10 +0200, Andreas Pflug wrote:
>>
>>
>>
>>
>
> Guys, what I need is alternative benchmark data, where postgresql comes
> out better on standard assumptions, etc. than i this example  Can this
> be done?

Sure, run dbt2 or dbt3 or dbt4

> Michael
>
>
>


--

    === The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
    Providing the most comprehensive  PostgreSQL solutions since 1997
              http://www.commandprompt.com/



Re: database contest results

From
Chris Browne
Date:
pgsql@j-davis.com (Jeff Davis) writes:

> On Wed, 2006-08-30 at 00:10 +0200, Andreas Pflug wrote:
>> > The author himself said he didn't have time. I didn't mean to be
>> > insulting, and I apologize if I was. 120 versus 3000 seems like the
>> > MySQL entry guys were operating with an entirely separate set of
>> > assumptions, and spent much more time optimizing it and determining the
>> > exact contest requirements.
>> >
>> Maybe you should have had a look at the article before speculating.
>> Contest requirement was very easy: take the DS sample and make it fast
>> on a given average PC hardware. The MySQL guys were able to take a
>
> Before I posted, I read the English press release along with the thread
> on this list and on pgsql-general, but I don't read German (I only found
> the English translation now). I also read your statement in this thread
> that the MySQL guys tuned the application "to access the database as
> rare [sic] as possible using memcache."
>
> To me, this fact alone means that the author of the PostgreSQL entry
> operated under different assumptions than the author of the MySQL entry.
> Even "simple" contest requirements can be interpreted differently due to
> assumptions. For instance, maybe the author of the PostgreSQL entry made
> the wrong assumptions because, as you put it, the contest was "wrong
> labeled app optimization"?
>
> I stand by my original statement that it was more about understanding
> and adapting to the contest than anything to do with the technical
> database details (like storage engines).

I wonder if throwing in pgmemcache could have had some similar effects
on a PostgreSQL-based system...
--
output = reverse("gro.gultn" "@" "enworbbc")
http://www3.sympatico.ca/cbbrowne/x.html
This is Linux country.  On a quiet night, you can hear NT re-boot.

Re: database contest results

From
Andreas Pflug
Date:
Chris Browne wrote:
> pgsql@j-davis.com (Jeff Davis) writes:
>
>
>> On Wed, 2006-08-30 at 00:10 +0200, Andreas Pflug wrote:
>>
>>>> The author himself said he didn't have time. I didn't mean to be
>>>> insulting, and I apologize if I was. 120 versus 3000 seems like the
>>>> MySQL entry guys were operating with an entirely separate set of
>>>> assumptions, and spent much more time optimizing it and determining the
>>>> exact contest requirements.
>>>>
>>>>
>>> Maybe you should have had a look at the article before speculating.
>>> Contest requirement was very easy: take the DS sample and make it fast
>>> on a given average PC hardware. The MySQL guys were able to take a
>>>
>> Before I posted, I read the English press release along with the thread
>> on this list and on pgsql-general, but I don't read German (I only found
>> the English translation now). I also read your statement in this thread
>> that the MySQL guys tuned the application "to access the database as
>> rare [sic] as possible using memcache."
>>
>> To me, this fact alone means that the author of the PostgreSQL entry
>> operated under different assumptions than the author of the MySQL entry.
>> Even "simple" contest requirements can be interpreted differently due to
>> assumptions. For instance, maybe the author of the PostgreSQL entry made
>> the wrong assumptions because, as you put it, the contest was "wrong
>> labeled app optimization"?
>>
>> I stand by my original statement that it was more about understanding
>> and adapting to the contest than anything to do with the technical
>> database details (like storage engines).
>>
>
> I wonder if throwing in pgmemcache could have had some similar effects
> on a PostgreSQL-based system...
>
Certainly. I already proposed to use that very latest tuned app and port
it to pgsql, which is just standard porting using PHP. This would make
the app a truely comparable.

Regards,
Andreas


Re: database contest results

From
Brian Hurt
Date:
mdean wrote:

> Jeff Davis wrote:
>
>> On Wed, 2006-08-30 at 00:10 +0200, Andreas Pflug wrote:
>>
>>
>>
>>
>
> Guys, what I need is alternative benchmark data, where postgresql
> comes out better on standard assumptions, etc. than i this example
> Can this be done?
> Michael
>
>
>
I think it can be done, it's just a lot of work.  At a minimum, I'd say
that:
1) Reliability of final configuration needs to be tested with a "pull
the plug" test, probably repeated a number of times.  If the database
doesn't survive this then it is disqualified as not meeting minimum
requirements.

2) Multi-threaded access to the database is needed- at least dozens of
threads doing asynchronous changes (updates, inserts, deletes, selects)
to the database using transactions.  Non-transactional databases need
not apply.

3) The database needs to be tuned by people who actually know how to
tune the database.  This is the classic "Postgres is slow!" mistake-
they run a default configuration.  This also means the clients run on a
different machine, and the specs of the database machine are a)
reasonable, and b) known to the tuners, so they can actually use the
capabilities of the machine.  At this point, I'd say a reasonable lower
bound would be a 64-bit CPU, at least 4G of memory, and 6-8 SATA drives
in a RAID 1+0 configuration.

Note that in any sort of "real" environment, which includes a small
webserver app that I actually care about, these requirements will
reflect reality.  Sooner or later the plug is going to get pulled on my
database- power outage, the magic smoke being released from some
hardware, something will happen which will make the database die
uncleanly, and need to recover from it.  And sooner or later I'm going
to have more than 1 person accessing the database at a time- at which
point transactions will be a lifesaver (as a side note, this is why I
picked Postgres over Mysql).  Even more so, when performance is most
important is when I have lots of people hitting the DB simultaneously-
my website just got slashdotted or what have you.  And finally, no
matter which database I end up picking, I'm going to put some time into
learning that databases, including how to tune that database.

The problem here is the cost- both in hardware (we're talking ~$3-4K for
the DB server alone), but even more so in time.  Time to set up the
database, time for the knowledgable people to come out of the woodwork
and help you configure the database (and possibly the application), time
to unplug each database multiple times, etc.  Not even a long weekend is
enough time, you're looking at weeks, if not months.  More time than
most journalists are willing to invest just to write a "Database
performance shoot out!  Details inside!" article.

Brian