Thread: Re: [pgsql-advocacy] Benchmarks WAS: Sun Talks about MySQL
(thread crossed over to pgsql-performance, where it belongs, from pgsql-advocacy) Greg, > I think TPC-E will make both of these major improvements much more important. > I suspect it would be hard to get 8.2 to even pass TPC-E due to the checkpoint > dropouts. > You'd be surprised, then. We're still horribly, horribly lock-bound on TPC-E; on anything over 4 cores lock resolution chokes us to death. See Jignesh's and Paul's various posts about attempts to fix this. Without resolving the locking issues, HOT and checkpoint doesn't have much effect on TPCE. --Josh
"Josh Berkus" <josh@agliodbs.com> writes: >> I think TPC-E will make both of these major improvements much more important. >> I suspect it would be hard to get 8.2 to even pass TPC-E due to the checkpoint >> dropouts. > > You'd be surprised, then. We're still horribly, horribly lock-bound on TPC-E; > on anything over 4 cores lock resolution chokes us to death. See Jignesh's and > Paul's various posts about attempts to fix this. Most of those posts have been about scalability issues with extremely large numbers of sessions. Those are interesting too and they may be limiting our results in benchmarks which depend on such a configuration (which I don't think includes TPC-E, but the benchmark Jignesh has been writing about is some Java application benchmark which may be such a beast) but they don't directly relate to whether we're "passing" TPC-E. What I was referring to by "passing" TPC-E was the criteria for a conformant benchmark run. TPC-C has iirc, only two relevant criteria: "95th percentile response time < 5s" and "average response time < 95th percentile response time". You can pass those even if 1 transaction in 20 takes 10-20s which is more than enough to cover checkpoints and other random sources of inconsistent performance. TPC-E has more stringent requirements which explicitly require very consistent response times and I doubt 8.2 would have been able to pass them. So the performance limiting factors whether they be i/o, cpu, lock contention, or whatever don't even come into play. We wouldn't have any conformant results whatsoever, not even low values limited by contention. 8.3 however should be in a better position to pass. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's 24x7 Postgres support!
Gregory Stark wrote: > TPC-E has more stringent requirements which explicitly require very consistent > response times and I doubt 8.2 would have been able to pass them. Sure it would. Just not for a very large scale factor ;-). -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Greg, > What I was referring to by "passing" TPC-E was the criteria for a conformant > benchmark run. TPC-C has iirc, only two relevant criteria: "95th percentile > response time < 5s" and "average response time < 95th percentile response > time". You can pass those even if 1 transaction in 20 takes 10-20s which is > more than enough to cover checkpoints and other random sources of inconsistent > performance. We can do this now. I'm unhappy because we're at about 1/4 of Oracle performance, but we certainly pass -- even with 8.2. --Josh
"Josh Berkus" <josh@agliodbs.com> writes: > Greg, > >> What I was referring to by "passing" TPC-E was the criteria for a conformant >> benchmark run. TPC-C has iirc, only two relevant criteria: "95th percentile >> response time < 5s" and "average response time < 95th percentile response >> time". You can pass those even if 1 transaction in 20 takes 10-20s which is >> more than enough to cover checkpoints and other random sources of inconsistent >> performance. > > We can do this now. I'm unhappy because we're at about 1/4 of Oracle > performance, but we certainly pass -- even with 8.2. We certainly can pass TPC-C. I'm curious what you mean by 1/4 though? On similar hardware? Or the maximum we can scale to is 1/4 as large as Oracle? Can you point me to the actual benchmark runs you're referring to? But I just made an off-hand comment that I doubt 8.2 could pass TPC-E which has much more stringent requirements. It has requirements like: the throughput computed over any period of one hour, sliding over the Steady State by increments of ten minutes, varies from the Reported Throughput by no more than 2% -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's Slony Replication support!
On Mon, 28 Apr 2008 14:40:25 -0400 Gregory Stark <stark@enterprisedb.com> wrote: > We certainly can pass TPC-C. I'm curious what you mean by 1/4 though? > On similar hardware? Or the maximum we can scale to is 1/4 as large > as Oracle? Can you point me to the actual benchmark runs you're > referring to? I would be curious as well considering there has been zero evidence provided to make such a statement. I am not saying it isn't true, it wouldn't be surprising to me if Oracle outperformed PostgreSQL in TPC-C but I would sure like to see in general how wel we do (or don't). Sincerely, Joshua D. Drake -- The PostgreSQL Company since 1997: http://www.commandprompt.com/ PostgreSQL Community Conference: http://www.postgresqlconference.org/ Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL SPI Liaison | SPI Director | PostgreSQL political pundit
Attachment
Joshua D. Drake wrote: > On Mon, 28 Apr 2008 14:40:25 -0400 > Gregory Stark <stark@enterprisedb.com> wrote: > > > >> We certainly can pass TPC-C. I'm curious what you mean by 1/4 though? >> On similar hardware? Or the maximum we can scale to is 1/4 as large >> as Oracle? Can you point me to the actual benchmark runs you're >> referring to? >> > > I would be curious as well considering there has been zero evidence > provided to make such a statement. I am not saying it isn't true, it > wouldn't be surprising to me if Oracle outperformed PostgreSQL in TPC-C > but I would sure like to see in general how wel we do (or don't). > > > Sincerely, > > Joshua D. Drake > I am sorry but I am far from catching my emails: Best thing is to work with TPC-E benchmarks involving the community. (TPC-C requirements is way too high on storage and everybody seems to be getting on the TPC-E bandwagon slowly.) Where can I get the latest DBT5 (TPC-E) kit ? Using the kit should allow me to recreate setups which can then be made available for various PostgreSQL Performance engineers to look at it. Regards, Jignesh