Andrew Dunstan <andrew@dunslane.net> writes:
> Hannu Krosing wrote:
>> Can't we make first cut at it by just running with timings on and then
>> compare ratios of running times - maybe with 2-3X tolerance - to catch
>> most obvious regressions ?
> The current regression tests are a series of yes/no answers to this
> question: does the actual output match the expected output. Nothing like
> as fuzzy as what you are suggesting is supported at all.
Quite aside from that, I don't think that's really the framework we
want. The issues that I think would be worth having tests for are
questions like "will the planner push comparisons to constants down
through a full join?" (which was the bug that started this thread).
With a test methodology like the above, it wouldn't be enough to
write a test case that exercised the behavior; you'd have to make
sure that any alternative plan was an order of magnitude worse.
I'm inclined to think that some sort of fuzzy examination of EXPLAIN
output (in this example, "are there constant-comparison conditions in
the relation scans?") might do the job, but I'm not sure how we'd
go about that.
regards, tom lane
