On Sat, Aug 8, 2015 at 8:24 PM, Peter Geoghegan <pg@heroku.com> wrote:
> I think that there needs to be a way of running an extended set of
> regression tests. I could definitely respect the desire for minimalism
The larger expense in having extensive test suites is the cost to
maintain them. With our current test framework tests have to be fairly
carefully written to produce output that isn't very fragile and
sensitive to minor changes in the code. In practice this is what
really drives the push towards minimal tests. If we tried testing
every code path right now -- which I tried to do once for the TOAST
code -- what you'll find is that what you're really testing is not
that the code is correct but that it does exactly what it does today.
At that point the test failures become entirely noise meaning
"something changed" rather than signal meaning "something's broken".
A better test framework can go a long way towards fixing this. It
would be much less of a problem if we had a unit test framework rather
than only black box integration tests. That would allow us to test
that every function in tuplestore.c meets its contract, not just that
some SQL query produces output that's correct and we think happened to
go through some code path we were interested in. There are many code
paths that will be hard to arrange to reach from SQL and impossible to
have any confidence will continue to be reached in the future when the
behaviour is intentionally changed.
That said, I don't think anyone would object to adding some regression
tests that at least test basic correctness of the sorting code. That's
a pretty embarrassing gap and iirc the only reason for it is that it
would make the regression tests sensitive to collation locale
settings.
--
greg
