Stephen Frost <sfrost@snowman.net> writes:
> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
>> Right. But there's a certain amount of serendipity involved in using the
>> core regression tests' final results. For example, I don't know how long
>> it would've taken us to understand the problems around dumping and
>> reloading child tables with inconsistent column orders, had there not been
>> examples of that in the regression tests. I worry that creating a sterile
>> set of objects for testing pg_dump will leave blind spots, because it will
>> mean that we only test cases that we explicitly created test cases for.
> I'm all for adding tests into pg_dump which do things like drop columns
> and change column names and other cases which could impact if the
> pg_dump is correct or not, and there's nothing preventing those tests
> from being added in the existing structure. Certainly, before we remove
> the coverage provided by running the serial test suite and then using
> pg_upgrade, we should analyze what is being tested and ensure that we're
> providing that same set of testing in the pg_dump TAP tests.
I don't think you grasped my basic point, which is that I'm worried about
emergent cases that we don't foresee needing to test (and that no amount
of code coverage checking would have shown up as being overlooked).
Admittedly, relying on the core regression tests to trigger such cases is
a pretty haphazard strategy, but it's way better than no strategy at all.
> We might also consider if there's a way to change the format for those
> tests to make them a bit less impenetrable for non-Perl folks to work
> with and to make it simpler to add new tests as new features are added.
TBH, that's part of my allergy to this concept, ie that this test
mechanism seems pretty write-only. I do not think that people will add
pg_dump test cases except when required to by project policy, so that
we will end up with a very skeletal set of tests that won't find any
unforeseen behaviors.
The TAP tests in general are utterly developer-unfriendly from where
I sit: not only is the code pretty unreadable, but god help you when
you need to try to debug a failure. I think that some serious effort
needs to be spent on improving that situation before we imagine that
we can throw away other test mechanisms we have today in favor of
TAP tests.
regards, tom lane
