Robert Haas wrote:
> Jeff Davis <pgsql@j-davis.com> wrote:
>> Or, I could write up a test framework in ruby or python, using
>> the appropriate pg driver, and some not-so-portable shell
>> commands to start and stop the server. Then, I can publish that
>> on this list, and that would at least make it easier to test
>> semi-manually and give greater confidence in pre-commit
>> revisions.
>
> That latter approach is similar to what happened with SSI's
> isolation tester. It started out in Python, and then Heikki
> rewrote it in C.
> If Python/Ruby code is massively simpler to write than the C
> code, that might be a good way to start out. It'll be an aid to
> reviewers even if neither it nor any descendent gets committed.
>
> Frankly, I think some automated testing harness (written in C or
> Perl) that could do fault-injection tests as part of the
> buildfarm would be amazingly awesome. I'm drooling just thinking
> about it. But I guess that's getting ahead of myself.
There may be room for both.
My experience was that the dtester tool from Markus made it
relatively easy for me to hack up new tests which gave detailed
information about which permutations were behaving as desired,
which were known not to be covered, and which had regressions.
That speed of adding new tests and detail about improvements or
regressions allowed faster development than would have been
possible with the isolation tester that Heikki wrote in C.
On the other hand, dtester requires python (in fact, I think it
requries python version 2.x were x is 5 or greater), a requirement
which I don't think we want to add for builds. It wasn't very
compatible with the normal make check environment, either in how it
was run or in its output. And it was much slower than the isolation
test framework -- like by about an order of magnitude.
So for a completed product on which you want to test for
regressions, the isolation tester is much better. For a development
effort on the scale of SSI, I would want to have dtester or
something very like it available.
Neither one quite handles tests for all the types of concurrency
conditions that one might want. I had some idea how to add some
additonal useful cases to dtester, and it didn't look outrageously
hard. I haven't really looked at how to do that in the insolation
tester, so I don't know how hard it would be there.
-Kevin
