On 19 March 2012 00:10, Andrew Dunstan <andrew@dunslane.net> wrote:
> If your tests are that voluminous then maybe they are not what we're looking
> for anyway. As Tom noted:
>
>
> IMO the objective of a regression test is not to memorialize every single
> case the code author thought about during development. ISTM it would not
> take very many cases to have reasonable code coverage.
Fair enough.
> Why exactly does this feature need particularly to have script-driven
> regression test generation when others don't?
It's not that it needs it, so much as that it is possible to provide
coverage for much of the code with black-box testing. In the case of
most of the hundreds of tests, I can point to a particular piece of
code that is being tested, that was written *after* the test was.
Doing this with pg_regress the old-fashioned way is going to be
incredibly verbose. I'm all for doing script-generation of pg_regress
tests in a well-principled way, and I'm happy to take direction from
others as to what that should look like.
I know that for the most part the tests provide coverage for discrete
units of functionality, and so add value. If they add value, why not
include them? Tests are supposed to be comprehensive. If that
inconveniences you, by slowing down the buildfarm for questionable
benefits, maybe it would be okay to have some tests not run
automatically, even if that did make them "next door to useless" in
Tom's estimation. There could be a more limited set of conventional
pg_regress tests that are run automatically, plus more comprehensive
tests that are run less frequently, typically only as it becomes
necessary to alter pg_stat_statements to take account of those
infrequent changes (typically additions) to the query tree.
We have tests that ensure that header files don't contain C++
keywords, and nominally promise to not do so, and they are not run
automatically. I don't see the sense in requiring that tests should be
easy to run, while also aspiring to have tests that are as useful and
comprehensive as possible. It seems like the code should dictate the
testing infrastructure, and not the other way around.
Part of the reason why I'm resistant to reducing the number of tests
is that it seems to me that excluding some tests but not others would
be quite arbitrary. It is not the case that some tests are clearly
more useful than others (except for the fuzz testing stuff, which
probably isn't all that useful).
--
Peter Geoghegan http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services
