Peter Geoghegan <peter@2ndquadrant.com> writes:
> On 19 March 2012 00:10, Andrew Dunstan <andrew@dunslane.net> wrote:
>> Why exactly does this feature need particularly to have script-driven
>> regression test generation when others don't?
> It's not that it needs it, so much as that it is possible to provide
> coverage for much of the code with black-box testing. In the case of
> most of the hundreds of tests, I can point to a particular piece of
> code that is being tested, that was written *after* the test was.
Well, basically what you're saying is that you did test-driven
development, which is fine. However, that does not mean that those
same tests are ideal for ongoing regression testing. What we want from
a regression test these days is primarily (a) portability testing, ie
does the feature work on platforms other than yours?, and (b) early
warning if someone breaks it down the road. In most cases, fairly
coarse testing is enough to catch drive-by breakage; and when it's not
enough, like as not the breakage is due to something you never thought
about originally and thus never tested for, so you'd not have caught it
anyway.
I am *not* a fan of regression tests that try to microscopically test
every feature in the system. Sure you should do that when initially
developing a feature, but it serves little purpose to do it over again
every time any other developer runs the regression tests for the
foreseeable future. That road leads to a regression suite that's so
voluminous that it takes too long to run and developers start to avoid
running it, which is counterproductive. For an example in our own
problem space look at mysql, whose regression tests take well over an
hour to run on a fast box. So they must be damn near bug-free right?
Uh, not so much, and I think the fact that developers can't easily run
their test suite is not unrelated to that.
So what I'd rather see is a small set of tests that are designed to do a
smoke-test of functionality and then exercise any interfaces to the rest
of the system that seem likely to break. Over time we might augment
that, when we find particular soft spots as a result of previously
undetected bugs. But sheer volume of tests is not a positive IMO.
As for the scripted vs raw-SQL-in-pg_regress question, I'm making the
same point as Andrew: only the pg_regress method is likely to get run
nearly everywhere, which means that the scripted approach is a FAIL
so far as the portability-testing aspect is concerned.
Lastly, even given that we were willing to accept a scripted set of
tests, I'd want to see it in perl not python. Perl is the project
standard; I see no reason to expect developers to learn two different
scripting languages to work on PG. (There might be a case for
accepting python-scripted infrastructure for pl/python, say, but not
for components that are 100% unrelated to python.)
regards, tom lane
