Robert Haas <robertmhaas@gmail.com> writes:
> ... That sounds a lot like the definition of a
> regression test suite. Of course, we have that already, but it's
> nowhere near comprehensive. Maybe we should be looking at an expanded
> test suite that runs on a time scale of hours rather than seconds.
mysql's got one of those, and I haven't noticed that it's kept their
defect rate down any. Hour-long regression suites are the sort of
thing developers won't run. Worse, regression suites are necessarily
designed to exercise only 100%-reproducible behavior.
> I don't think that any test suite is going to eliminate the need for
> beta-testing.
Precisely...
What I'd like to see is some sort of test mechanism for WAL recovery.
What I've done sometimes in the past (and recently had to fix the tests
to re-enable) is to kill -9 a backend immediately after running the
regression tests, let the system replay the WAL for the tests, and then
take a pg_dump and compare that to the dump gotten after a conventional
run. However this is quite haphazard since (a) the regression tests
aren't especially designed to exercise all of the WAL logic, and (b)
pg_dump might not show the effects of some problems, particularly not
corruption in non-system indexes. It would be worth the trouble to
create a more specific test methodology.
In short: merely making the tests bigger doesn't impress me in the
least. Focused testing on areas we aren't covering at all could be
worth the trouble.
regards, tom lane
