On Wed, 22 Sep 2004, Neil Conway wrote:
> On Fri, 2004-09-17 at 01:43, Tom Lane wrote:
> > PS: this points up once again that the regression tests do not do very
> > well at testing concurrent behavior. We need to think about some sort
> > of parallel-test harness that would let us write tests that have better
> > odds of catching bugs like this.
>
> I agree -- something like this would be very useful. I'm a little
> stumped about how to actually write such a thing, though...
>
> One incremental improvement to the existing RT system would be to make
> each test self-contained (i.e. as far as possible it would assume
> nothing about DBMS state before it ran, and not leave the DBMS state
> modified when it finished running). If writing completely self-contained
> tests isn't possible, we could specify a dependency graph for the tests.
> Then we could have the regression test system generate a randomized
> grouping of tests to be executed. In order to have reproducible results,
> we could associate each randomized grouping with a numeric key (there
> should be some way to produce a 1-to-1 mapping). So someone reporting a
> regression test problem would say: "I see the following regression test
> failures, the test key is xyz", and a developer could run the regression
> tests specifying that key and have the RT system run the same
> combination of tests in the same order that produced the problem the
> user saw. This would be a complement to the existing method of running
> the RTs.
If someone puts in the work to make the RTs self contained, is it
worthwhile also including a logic to return the system to the state before
the test took place (self contained tests kind of imply that, to me
anyway).
What this also gives us is the ability to run each test continuously while
we run every single other test. Sure, this would take a while to run but I
think it gives as a lot of concurrent testing with a relatively small
amount of work.
>
> I still think we need a separate way to test parallel behavior though,
> in addition to the above idea. Can anyone think of a good way to do
> this?
I think if the tests are self contained then we have a higher likelihood
of hitting concurrency issues. The thing is, the major concurrency issues
which have come up recently (from memory) are all race conditions. So,
running queries in parallel we cannot pre determine who will win the race
and I cannot see how we can. Best we can do is increase the chances of
finding it by running it more than once.
>
> -Neil
Gavin
