Re: Testing DDL deparsing support - Mailing list pgsql-hackers

Bruce Momjian wrote:
> On Fri, Dec  5, 2014 at 09:29:59AM -0300, Alvaro Herrera wrote:
> > Bruce Momjian wrote:
> > > On Fri, Nov 28, 2014 at 01:43:36PM +0900, Ian Barwick wrote:
> > > > Standard regression tests are helpful because patch authors include new test
> > > > cases in the patches that stress their new options or commands.  This new test
> > > > framework needs to be something that internally runs the regression tests and
> > > > exercises DDL deparsing as the regression tests execute DDL.  That would mean
> > > > that new commands and options would automatically be deparse-tested by our new
> > > > framework as soon as patch authors add standard regress support.
> > > 
> > > Are you saying every time a new option is added to a command that a new
> > > regression test needs to be added?
> > 
> > Not necessarily -- an existing test could be modified, as well.
> > 
> > > We don't normally do that,
> > 
> > I sure hope we do have all options covered by tests.
> 
> Are you saying that every combination of ALTER options is tested?

Well, ALTER TABLE is special: you can give several subcommands, and each
subcommand can be one of a rather long list of possible subcommands.
Testing every combination would mean a combinatorial explosion, which
would indeed be too large.  But surely we want a small bunch of tests to
prove that having several subcommands works fine, and also at least one
test for every possible subcommand.

> We have rejected simple regression test additions on the basis that
> the syntax works and is unlikely to break once tested once by the
> developer.

This rationale doesn't sound so good to me.  Something might work fine
the minute it is committed, but someone else might break it
inadvertently later; this has actually happened.  Having no tests at all
for a feature isn't good.  

I know we have recently rejected patches that added tests only to
improve the coverage percent, for instance in CREATE DATABASE, because
the runtime of the tests got too large.  Are there other examples of
rejected tests?

> > > and it could easily bloat the regression tests over time.
> > 
> > We had 103 regression tests in 8.2 and we have 145 in 9.4.  Does this
> > qualify as bloat?
> 
> No, that seems fine.  I am worried about having to have a test for every
> syntax change, which we currently don't do?  Was that issue not clear in
> my first email?

Well, if with "every syntax change" you mean "every feature addition",
then I think we should have at least one test for each, yes.  It's not
like we add new syntax every day anyway.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services