Thread: Status of JDBC test suite?
While working on the executeBatch() patch I noticed that the JDBC 2 test suite with current CVS code generates 9 errors/failures. Some of these errors appear to be caused by tests that do not create their own tables. Before I dive into this and fix the test suite, I just wanted to check if someone else has already looked into this. Any information about the status of the test suite? Regards, René Pijlman
On Sun, 26 Aug 2001, Rene Pijlman wrote: > While working on the executeBatch() patch I noticed that the > JDBC 2 test suite with current CVS code generates 9 > errors/failures. Some of these errors appear to be caused by > tests that do not create their own tables. With the tables added it's still around 6-7 tests that fail. > Before I dive into this and fix the test suite, I just wanted to > check if someone else has already looked into this. Any > information about the status of the test suite? Since it's non-obvious how to initialize it and how to use it, very few have. :-) Most recently some updates in DatabaseMetaData broke some test cases. Ideally we should be able to extract parts of the test suite that could be run without connecting to a backend. It would then be possible to at least run those parts of the tests in "make". /Anders _____________________________________________________________________ A n d e r s B e n g t s s o n ndrsbngtssn@yahoo.se Stockholm, Sweden _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com
On Sun, 26 Aug 2001 21:30:45 +0200 (CEST), Anders Bengtsson wrote: >Since it's non-obvious how to initialize it and how to use it, very few >have. :-) This is probably true, but once you have your environment setup (including a test database, password etc.) its just a matter of typing "make check". Would a brief How-to help? >Most recently some updates in DatabaseMetaData broke some test >cases. :-( >Ideally we should be able to extract parts of the test suite that could be >run without connecting to a backend. It would then be possible to at least >run those parts of the tests in "make". I'm not sure if that would be useful. Its a reasonable requirement for a database driver test suite that there actually is a database to test against :-) Regards, René Pijlman
On Sun, 26 Aug 2001, Rene Pijlman wrote: > This is probably true, but once you have your environment setup > (including a test database, password etc.) its just a matter of > typing "make check". > > Would a brief How-to help? Yes! :) > >Ideally we should be able to extract parts of the test suite that could be > >run without connecting to a backend. It would then be possible to at least > >run those parts of the tests in "make". > > I'm not sure if that would be useful. Its a reasonable > requirement for a database driver test suite that there actually > is a database to test against :-) It's not as crazy as it sounds. :-) The driver as a whole must of course be tested against a real database. But many of the classes that it is made up of could be verified on their own with unit tests. This kind of testing catches most of the small "silly" bugs that seem to be the most common kind of bug. /Anders _____________________________________________________________________ A n d e r s B e n g t s s o n ndrsbngtssn@yahoo.se Stockholm, Sweden _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com
On Sun, Aug 26, 2001 at 09:14:49PM +0200, Rene Pijlman wrote: > Before I dive into this and fix the test suite, I just wanted to > check if someone else has already looked into this. Any > information about the status of the test suite? Not sure if it's the same thing, but I've noticed that often, the first time the test harness runs a particular test bucket, most or all of the tests fail. The second and third passes return work properly (insofar as there's support in the JDBC driver/PostgreSQL). I haven't had time to look into it yet. Liam -- Liam Stewart :: Red Hat Canada, Ltd. :: liams@redhat.com
On Sun, Aug 26, 2001 at 09:56:09PM +0200, Anders Bengtsson wrote: > On Sun, 26 Aug 2001, Rene Pijlman wrote: > > > This is probably true, but once you have your environment setup > > (including a test database, password etc.) its just a matter of > > typing "make check". > > > > Would a brief How-to help? > > Yes! :) I assume that you're wondering about the JDBC 2 test suite from Sun. I've attached a note that has some detailed instructions on getting the suite to work with PostgreSQL. Rene, do you want to post this on your PostgreSQL JDBC page? > > >Ideally we should be able to extract parts of the test suite that could be > > >run without connecting to a backend. It would then be possible to at least > > >run those parts of the tests in "make". I think that most, if not all, of the tests in the JDBC suite need to connect to a database. > It's not as crazy as it sounds. :-) The driver as a whole must of course > be tested against a real database. But many of the classes that it is made > up of could be verified on their own with unit tests. This kind of testing > catches most of the small "silly" bugs that seem to be the most common > kind of bug. Maybe it would be worthwhile to build our own small unit test suite? Can you explain your idea in more detail? Liam -- Liam Stewart :: Red Hat Canada, Ltd. :: liams@redhat.com
Attachment
Yes I have seen this behavior as well (first run has many failures, but later ones succeed). One reason that I noticed had to do with test setup code. Many of the tests do the following: drop table foobar; create table foobar (...); The first time this is run on a clean database the drop table fails because the table does not exist. The next time this is run it succeds because the table now exists. The behavior of the test harness seems to be that it keeps running the same set of tests over until it gets the same number of failures in two consecutive runs, but I could be wrong about that. thanks, --Barry Liam Stewart wrote: > On Sun, Aug 26, 2001 at 09:14:49PM +0200, Rene Pijlman wrote: > >>Before I dive into this and fix the test suite, I just wanted to >>check if someone else has already looked into this. Any >>information about the status of the test suite? >> > > Not sure if it's the same thing, but I've noticed that often, the first > time the test harness runs a particular test bucket, most or all of the > tests fail. The second and third passes return work properly (insofar as > there's support in the JDBC driver/PostgreSQL). I haven't had time to > look into it yet. > > Liam > >
On Wed, 29 Aug 2001, Liam Stewart wrote: > > > Would a brief How-to help? > > > > Yes! :) > > I assume that you're wondering about the JDBC 2 test suite from Sun. > I've attached a note that has some detailed instructions on getting the > suite to work with PostgreSQL. Rene, do you want to post this on your > PostgreSQL JDBC page? Actually, we're not talking about Sun's test suite. :-) But now that you mention it... > I think that most, if not all, of the tests in the JDBC suite need to > connect to a database. Could Sun's tests be used as a kind of regression tests as well? Then their tests could be the only black-box test suite we'll need! > Maybe it would be worthwhile to build our own small unit test suite? Can > you explain your idea in more detail? There is already a small unit test suite in the CVS. But it haven't been kept up to date with recent changes in the drivers, and it's a bit hard to use (thus the need for a how-to). The tests use the tool JUnit (http://www.junit.org), that is originally intended for one-class-at-a-time testing. On their site is a lot of good texts about this kind of unit testing. /Anders _____________________________________________________________________ A n d e r s B e n g t s s o n ndrsbngtssn@yahoo.se Stockholm, Sweden _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com
On Wed, Aug 29, 2001 at 08:08:58PM +0200, Anders Bengtsson wrote: > Actually, we're not talking about Sun's test suite. :-) But now that you > mention it... Ok, I see the one that you're talking about. I never knew it was there. Now to play with it :) > Could Sun's tests be used as a kind of regression tests as well? Then > their tests could be the only black-box test suite we'll need! Ehmm..I've been looking Sun's test suite, but one has to realize that their test suite is testing for JDBC compliance at the J2EE level, which is more stringent than normal JDBC 2. It's useful to have, but it's definately not the only suite we'll need. > There is already a small unit test suite in the CVS. But it haven't been > kept up to date with recent changes in the drivers, and it's a bit hard > to use (thus the need for a how-to). > The tests use the tool JUnit (http://www.junit.org), that is originally > intended for one-class-at-a-time testing. On their site is a lot of good > texts about this kind of unit testing. I'll take a look at these. Thanks for the pointers, Liam -- Liam Stewart :: Red Hat Canada, Ltd. :: liams@redhat.com
On Wed, 29 Aug 2001 13:34:51 -0400, you wrote: >I assume that you're wondering about the JDBC 2 test suite from Sun. >I've attached a note that has some detailed instructions on getting the >suite to work with PostgreSQL. Rene, do you want to post this on your >PostgreSQL JDBC page? Thanks! I've decided to include it in a new "PostgreSQL/JDBC Test Suite Howto" to be put in CVS. I hope that's OK with you. >Maybe it would be worthwhile to build our own small unit test suite? See the other sections of the Howto :-) Regards, René Pijlman <rene@lab.applinet.nl>