Re: TAP test breakage on MacOS X - Mailing list pgsql-hackers

On Tue, Oct 28, 2014 at 9:01 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
> Well, they caught the fact that pg_basebackup can't back up tablespaces
> with names longer than 99 characters, for example.
>
> But it's wrong to expect the primary value of tests to be to detect
> previously unknown bugs.  Yes, that has been the experience in this
> project.  We have software that we think works, and then we send it out
> to test on N obscure platforms, and we find interesting things.
>
> But the other dimension is that tests allow you to make changes with
> confidence.  For example, the tablespace mapping logic in pg_basebackup
> has been whacked around about three times.  I don't have any doubt that
> it still works correctly, because it has extensive test coverage.  If
> someone sends in a patch to support cpio in pg_basebackup (hey, no
> symlink name length limits) and wants to refactor the entire source code
> to achieve that, I'll have no problem with that.
>
> Yes, some of the tests are low in value, perhaps a bit silly.  But it's
> a foundation, and I expect to add more in the future.  The good news is
> that because some of these "silly" tests actually exercise important
> internal functionality of the test mechanisms (e.g., create temporary
> directory, run program, record exit status, capture stdout and stderr
> separately, check output files), I am pretty confident that we can go
> far with the current infrastructure without needing any more external
> modules or something like that.

I'm sympathetic to that line of reasoning, but I really think that if
you want to keep this infrastructure, it needs to be made portable.  I
don't buy the argument that it isn't important to be able to run all
of the tests everywhere all the time.  I grant that, for example, it's
reasonable not to run tests for Linux-style locales on Windows.  But
barring that sort of corner case, I really think they should all run
everywhere.  Now, if it turns out that there are one or two really old
buildfarm critters that can't run all the tests, then I think I can
let that go, but the goal should be for 100% coverage.

Right now the *normal* situation is that the tests, when run on a
typical developer's machine, either (1) don't run, because the
developer's machine has some wart that they realize will break the
tests; or (2) don't pass, because the developer's machine has some
wart that they don't realize will break the tests, but it actually
does.  That means that the tests, as they exist today, are basically
your toy.  Most people aren't running them, and if they do, they fail,
and nobody understands why.  Until that changes, the tests really
don't have any value to the project, and possibly negative value, if
they encourage people to start ignoring test failures.

I also think that the dependencies for the tests to actually run need
to be clearly and explicitly documented.  If the tests don't run on my
machine, and I want them to run, there's nothing in the documentation
I can go and look at to figure out what I need to do to fix that.  In
my mind, that's a big problem.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company