Thread: MonetDB test says that PostgreSQL often has errors or missing results
See: http://monetdb.cwi.nl/SQL/Benchmark/TPCH/ If the result is correct, then the problem queries should be added to the regression test suite. If the result is not correct, then perhaps they could get assistance on proper configuration of PostgreSQL and rerun the tests.
Re: MonetDB test says that PostgreSQL often has errors or missing results
From
Stefan Kaltenbrunner
Date:
Dann Corbit wrote: > See: > http://monetdb.cwi.nl/SQL/Benchmark/TPCH/ > If the result is correct, then the problem queries should be added to > the regression test suite. > If the result is not correct, then perhaps they could get assistance on > proper configuration of PostgreSQL and rerun the tests. this is a known issues and has been discussed in the past - the problem here is that postgresql misparsed the interval syntax in older version (not that they used 8.2 when 8.3 was long released in that test). This has since been fixed in 8.4+ however. Stefan
Dann Corbit wrote: > See: > http://monetdb.cwi.nl/SQL/Benchmark/TPCH/ > If the result is correct, then the problem queries should be added to > the regression test suite. > If the result is not correct, then perhaps they could get assistance on > proper configuration of PostgreSQL and rerun the tests. > We've already gotten a couple of reports that talks about the specific reasons why PostgreSQL 8.2 doesn't do well on this test, such as http://www.mail-archive.com/pgsql-general@postgresql.org/msg120882.html (broken queries) and http://archives.postgresql.org/pgsql-performance/2008-09/msg00157.php (why the default setting are completely ludicrous for this test). One of these days I'm going to re-run the whole suite with 8.4 and see how we're doing. Seriously doubt any queries are still flat out broken, given how many reports I've seen of successful (albeit slow sometimes) runs of the whole thing. That's just MonetDB being negligent in reporting things to favor their own agenda. It's completely permissible to customize queries that won't execute at all in order to fix their compatibility with a particular database, the fact that they didn't look into that or even say what was wrong is just shoddy journalism. I'm hoping to have some crow to serve to them about another benchmark result they've published soon, they're back to really rubbing me the wrong way again lately. -- Greg Smith 2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support greg@2ndQuadrant.com www.2ndQuadrant.com
All, Actually, the report which MonetDB has published I believe is illegal. If they're not running it through the TPC, they can't claim it's a "TPCH" result. --Josh Berkus
Josh Berkus wrote: > Actually, the report which MonetDB has published I believe is illegal. > If they're not running it through the TPC, they can't claim it's a > "TPCH" result. > I just resisted getting into that but now you've set me off again. Presumably they're using the public TPC-H data and query generator distributed by the TPC, and there's certainly plenty of other unofficial reports of results using that floating around. Where I think they really crossed the line here is using that kit to produce unaudited results, and then publishing results that included comparisons against a competitor, which is clearly not what the TPC intends you to do here. I'd rather refute their results than cry to the daddy TPC about it though. Not that I'd be upset if someone else, perhaps someone who already has contacts within the TPC, did so. (looking up into the air and whistling) -- Greg Smith 2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support greg@2ndQuadrant.com www.2ndQuadrant.com
On Tue, Jan 19, 2010 at 10:04 PM, Greg Smith <greg@2ndquadrant.com> wrote: > Josh Berkus wrote: >> >> Actually, the report which MonetDB has published I believe is illegal. >> If they're not running it through the TPC, they can't claim it's a >> "TPCH" result. >> > > I just resisted getting into that but now you've set me off again. > Presumably they're using the public TPC-H data and query generator > distributed by the TPC, and there's certainly plenty of other unofficial > reports of results using that floating around. Where I think they really > crossed the line here is using that kit to produce unaudited results, and > then publishing results that included comparisons against a competitor, > which is clearly not what the TPC intends you to do here. What the TPC provides isn't really a usable kit. It could be entertaining to see how their kit works. Regards, Mark
Mark Wong wrote: > What the TPC provides isn't really a usable kit. It could be > entertaining to see how their kit works. > The one for TPC-H seems to work for a lot of people; the best of the intros I found for how to make it go was http://bhairav.serc.iisc.ernet.in/doc/Installation/tpch.htm , there are others. I'd guess MonetDB followed a procedure just like that, discovered some queries didn't work right, and just called it a day and published rather than investigate. -- Greg Smith 2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support greg@2ndQuadrant.com www.2ndQuadrant.com