Tom,
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > I'm all for adding tests into pg_dump which do things like drop columns
> > and change column names and other cases which could impact if the
> > pg_dump is correct or not, and there's nothing preventing those tests
> > from being added in the existing structure. Certainly, before we remove
> > the coverage provided by running the serial test suite and then using
> > pg_upgrade, we should analyze what is being tested and ensure that we're
> > providing that same set of testing in the pg_dump TAP tests.
>
> I don't think you grasped my basic point, which is that I'm worried about
> emergent cases that we don't foresee needing to test (and that no amount
> of code coverage checking would have shown up as being overlooked).
> Admittedly, relying on the core regression tests to trigger such cases is
> a pretty haphazard strategy, but it's way better than no strategy at all.
The tests that were added to the core regression suite were done so for
a reason and hopefully we can identify cases where it'd make sense to
also run those tests for pg_upgrade/pg_dump testing. More-or-less
anything that materially changes the catalog should be included, I would
think. Things that are only really only working with the heap/index
files don't really need to be performed because the pg_upgrade process
doesn't change those.
> > We might also consider if there's a way to change the format for those
> > tests to make them a bit less impenetrable for non-Perl folks to work
> > with and to make it simpler to add new tests as new features are added.
>
> TBH, that's part of my allergy to this concept, ie that this test
> mechanism seems pretty write-only. I do not think that people will add
> pg_dump test cases except when required to by project policy, so that
> we will end up with a very skeletal set of tests that won't find any
> unforeseen behaviors.
I certainly agree that the current structure for the tests isn't trivial
to work with and would welcome suggestions as to how to improve it. Now
that we've had this testing structure for a year and have added quite a
bit more to it, it's definitely clear that we need to find a more
developer-friendly approach.
> The TAP tests in general are utterly developer-unfriendly from where
> I sit: not only is the code pretty unreadable, but god help you when
> you need to try to debug a failure. I think that some serious effort
> needs to be spent on improving that situation before we imagine that
> we can throw away other test mechanisms we have today in favor of
> TAP tests.
Agreed.
Thanks!
Stephen
