Re: Two weeks to feature freeze - Mailing list pgsql-hackers

On Fri, 20 Jun 2003, Dann Corbit wrote:

> > Hmm... I must have missed the huge corporation paying for in
> > house testing of PostgreSQL.  In the Free Software world the
> > "beta team" is all of those people that need the new features
> > so badly that they are willing to risk their own data and
> > hardware testing it.
>
> I don't see how this model can possibly succeed then.  You can't just
> hope that your end users will:
> 1.  Exhaustively test
> 2.  Accurately report the findings

But it does, and has for 10 years now ...

> Our beta customers do help us to find bugs.  Bugs reported by customers
> for released products are extremely rare.

Check the past archives for the mailing lists ... our "bugs reported by
end users for released products" is extremely rare also, and *generally*
is a result of them doing something that nobody had thought to test for
before ...

> Spoken like a programmer.  Yes, real world data *always* turns up things
> that neither the testers nor the programmers imagined.  But a huge and
> comprehensive conformance testing effort will turn up 99% of the
> problems.

And ours do ... I don't believe I can recall us having a release where
we've had a stream of problem reports come flying in afterwards ... we
might get one or two from ppl that have hit a 'never before seen' bug,
that generally gets fixed very quickly ...

> 100% code coverage is impossible.
> Program proving is impossible.
> 0% defect code delivery is impossible.
>
> But you should try to approach the ideal as closely as can be attained.

And we do ...

> The tests are good tests.  They cover a wide range of features and
> functions and discover if you can cause permanent damage to a database
> by simply performing end-user queries.  The scripts are a bit hokey, but
> it isn't all that difficult to get them to run.

Well, if you would like to volunteer to run them against PostgreSQL, and
let us know what fails, we can let you know why said test is wrong in the
first place ... we've been through crash-me several times before, and
'fixing crash-me' was more work then it was worth ...

> > Basically any time a competitor differed from
> > MySQL an error would be generated (despite the fact that it
> > was very likely that it was MySQL that was wrong).
>
> This is unfair and untrue. (I have no connection whatsoever with the
> MySQL group, BTW).

Been there, done that ... even tried to get changes made to make the tests
more accurate ... it was like trying to move a mountain ...

> PostgreSQL has an excellent programming team.  Why not try to recruit a
> similar testing team?  I think it would strongly differentiate the tool
> set from similar free stuff.

Are you volunteering?  We already have a testing team we're happy with,
but if you would like to extend it with your resources, please feel free
to join in ...