Thread: Collection of small fixes to binary transfer code and unit tests
Hi, I just tried to run the unit tests with forcebinary=true and preparethreshold=1 and got 10 new failures. Of them 3 are real bugs (corner cases, but still bugs), rest are just problems in tests. So here are some patches to fix them: binary-force-harder-tests.patch - forces prepare threshold to 1 when forcebinary=true and thus makes the bugs visible binary-fix-some-serverprepared-tests-with-force.patch - the previous patch forcing prepare threshold of 1 breaks 2 tests that test disabling of preparethreshold - just skip part of the tests when forcebinary=true binary-fix-maxfieldlength-test.patch - test used getBytes on integer field which returns different bytes for txt vs bin transfers - fixes the unit test to cope with both bin and txt encoding binary-fix-date-infinity.patch (bug) - the backend sends a smaller infinity value for DATE than TIMESTAMP fields (4 bytes vs 8 bytes) - to ensure compatibility with text protocol convert smaller infinity values to larger infinity values when receiving DATE from backend binary-fix-arrays-with-zero-dimensions.patch (bug) - the new binary code for arrays did not handle arrays with dimension of zero - fix it and add a new test case for array.getResultSet() binary-fix-batch-with-generated-keys.patch (bug) - when executing batch with generated keys returned for each insert the same Query object is shared by the statements, and along with it the field definitions - when the prepare threshold is exceeded during the batch execution the first row of generated keys return text values and latter rows binary values - to fix we detect the situation and force a separate describe round-trip when batch is about to exceed the prepare threshold limit so that all generated in the same result set will be binary binary-fix-timezonetests.patch - based on my patch from 2007 to make the timezonetests 'fair' so that binary transfer has a chance of passing them - some tests in TimezoneTest never actually exercised the correct code paths because they just used getString which just returns the text from the server and does not cause the jdbc driver to try to parse the output - it is not possible in binary transfer mode for the jdbc driver to generate the exactly same getString output for all time values than what the server does - instead it uses the toString() method of the official java.sql classes - fixes the testSetTimestampOnTime, testSetTimestamp, testSetDate, testSetTime to use proper getTime/getTimestamp etc methods when fetching columns instead of relying on the getString - also changes the code verifying that inserts worked correctly to request the server for the actual values in the database casted to text type so that both binary and text transfer get the same results -Mikko
Attachment
Mikko,
I get the following errors after applying your patch.
[junit] Testcase:
testMaxFieldSize(org.postgresql.test.jdbc2.ResultSetTest):
FAILED
[junit] null expected:<[12345]> but was:<[09]>
[junit] junit.framework.ComparisonFailure: null expected:<[12345]>
but was:<[09]>
[junit] at
org.postgresql.test.jdbc2.ResultSetTest.testMaxFieldSize(ResultSetTest.java:174)
[junit]
[junit]
[junit] Testcase:
testGetTimestampWTZ(org.postgresql.test.jdbc2.TimestampTest):
FAILED
[junit] expected:<2000-07-07 11:00:00.123> but was:<2000-07-07
11:00:00.122999>
[junit] junit.framework.AssertionFailedError: expected:<2000-07-07
11:00:00.123> but was:<2000-07-07 11:00:00.122999>
[junit] at
org.postgresql.test.jdbc2.TimestampTest.timestampTestWTZ(TimestampTest.java:436)
[junit] at
org.postgresql.test.jdbc2.TimestampTest.testGetTimestampWTZ(TimestampTest.java:188)
[junit]
[junit]
[junit] Testcase:
testSetTimestampWTZ(org.postgresql.test.jdbc2.TimestampTest):
FAILED
[junit] expected:<2000-07-07 11:00:00.123> but was:<2000-07-07
11:00:00.122999>
[junit] junit.framework.AssertionFailedError: expected:<2000-07-07
11:00:00.123> but was:<2000-07-07 11:00:00.122999>
[junit] at
org.postgresql.test.jdbc2.TimestampTest.timestampTestWTZ(TimestampTest.java:436)
[junit] at
org.postgresql.test.jdbc2.TimestampTest.testSetTimestampWTZ(TimestampTest.java:258)
Dave Cramer
dave.cramer(at)credativ(dot)ca
http://www.credativ.ca
On Wed, Sep 28, 2011 at 2:42 PM, Mikko Tiihonen
<mikko.tiihonen@nitorcreations.com> wrote:
> Hi,
>
> I just tried to run the unit tests with forcebinary=true and
> preparethreshold=1 and got 10 new failures.
> Of them 3 are real bugs (corner cases, but still bugs), rest are just
> problems in tests.
>
> So here are some patches to fix them:
>
> binary-force-harder-tests.patch
> - forces prepare threshold to 1 when forcebinary=true and thus makes the
> bugs visible
>
> binary-fix-some-serverprepared-tests-with-force.patch
> - the previous patch forcing prepare threshold of 1 breaks 2 tests that test
> disabling of preparethreshold
> - just skip part of the tests when forcebinary=true
>
> binary-fix-maxfieldlength-test.patch
> - test used getBytes on integer field which returns different bytes for txt
> vs bin transfers
> - fixes the unit test to cope with both bin and txt encoding
>
> binary-fix-date-infinity.patch (bug)
> - the backend sends a smaller infinity value for DATE than TIMESTAMP fields
> (4 bytes vs 8 bytes)
> - to ensure compatibility with text protocol convert smaller infinity values
> to larger infinity values when receiving DATE from backend
>
> binary-fix-arrays-with-zero-dimensions.patch (bug)
> - the new binary code for arrays did not handle arrays with dimension of
> zero
> - fix it and add a new test case for array.getResultSet()
>
> binary-fix-batch-with-generated-keys.patch (bug)
> - when executing batch with generated keys returned for each insert the same
> Query object is shared by the statements, and along with it the field
> definitions
> - when the prepare threshold is exceeded during the batch execution the
> first
> row of generated keys return text values and latter rows binary values
> - to fix we detect the situation and force a separate describe round-trip
> when
> batch is about to exceed the prepare threshold limit so that all generated
> in the same result set will be binary
>
> binary-fix-timezonetests.patch
> - based on my patch from 2007 to make the timezonetests 'fair' so that
> binary transfer has a chance of passing them
> - some tests in TimezoneTest never actually exercised the correct code paths
> because they just used getString which
> just returns the text from the server and does not cause the jdbc driver to
> try to parse the output
> - it is not possible in binary transfer mode for the jdbc driver to generate
> the exactly same getString output for
> all time values than what the server does - instead it uses the toString()
> method of the official java.sql classes
> - fixes the testSetTimestampOnTime, testSetTimestamp, testSetDate,
> testSetTime to use proper getTime/getTimestamp
> etc methods when fetching columns instead of relying on the getString
> - also changes the code verifying that inserts worked correctly to request
> the server for the actual values in the
> database casted to text type so that both binary and text transfer get the
> same results
>
>
> -Mikko
>
>
> --
> Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-jdbc
>
>
Interestingly I do not see these errors when I run the test inside of eclipse, but I do when I run it from ant. Dave Cramer dave.cramer(at)credativ(dot)ca http://www.credativ.ca On Thu, Sep 29, 2011 at 7:53 AM, Dave Cramer <pg@fastcrypt.com> wrote: > Mikko, > > I get the following errors after applying your patch. > > [junit] Testcase: > testMaxFieldSize(org.postgresql.test.jdbc2.ResultSetTest): > FAILED > [junit] null expected:<[12345]> but was:<[09]> > [junit] junit.framework.ComparisonFailure: null expected:<[12345]> > but was:<[09]> > [junit] at > org.postgresql.test.jdbc2.ResultSetTest.testMaxFieldSize(ResultSetTest.java:174) > [junit] > [junit] > [junit] Testcase: > testGetTimestampWTZ(org.postgresql.test.jdbc2.TimestampTest): > FAILED > [junit] expected:<2000-07-07 11:00:00.123> but was:<2000-07-07 > 11:00:00.122999> > [junit] junit.framework.AssertionFailedError: expected:<2000-07-07 > 11:00:00.123> but was:<2000-07-07 11:00:00.122999> > [junit] at > org.postgresql.test.jdbc2.TimestampTest.timestampTestWTZ(TimestampTest.java:436) > [junit] at > org.postgresql.test.jdbc2.TimestampTest.testGetTimestampWTZ(TimestampTest.java:188) > [junit] > [junit] > [junit] Testcase: > testSetTimestampWTZ(org.postgresql.test.jdbc2.TimestampTest): > FAILED > [junit] expected:<2000-07-07 11:00:00.123> but was:<2000-07-07 > 11:00:00.122999> > [junit] junit.framework.AssertionFailedError: expected:<2000-07-07 > 11:00:00.123> but was:<2000-07-07 11:00:00.122999> > [junit] at > org.postgresql.test.jdbc2.TimestampTest.timestampTestWTZ(TimestampTest.java:436) > [junit] at > org.postgresql.test.jdbc2.TimestampTest.testSetTimestampWTZ(TimestampTest.java:258) > > Dave Cramer > > dave.cramer(at)credativ(dot)ca > http://www.credativ.ca > > > > > On Wed, Sep 28, 2011 at 2:42 PM, Mikko Tiihonen > <mikko.tiihonen@nitorcreations.com> wrote: >> Hi, >> >> I just tried to run the unit tests with forcebinary=true and >> preparethreshold=1 and got 10 new failures. >> Of them 3 are real bugs (corner cases, but still bugs), rest are just >> problems in tests. >> >> So here are some patches to fix them: >> >> binary-force-harder-tests.patch >> - forces prepare threshold to 1 when forcebinary=true and thus makes the >> bugs visible >> >> binary-fix-some-serverprepared-tests-with-force.patch >> - the previous patch forcing prepare threshold of 1 breaks 2 tests that test >> disabling of preparethreshold >> - just skip part of the tests when forcebinary=true >> >> binary-fix-maxfieldlength-test.patch >> - test used getBytes on integer field which returns different bytes for txt >> vs bin transfers >> - fixes the unit test to cope with both bin and txt encoding >> >> binary-fix-date-infinity.patch (bug) >> - the backend sends a smaller infinity value for DATE than TIMESTAMP fields >> (4 bytes vs 8 bytes) >> - to ensure compatibility with text protocol convert smaller infinity values >> to larger infinity values when receiving DATE from backend >> >> binary-fix-arrays-with-zero-dimensions.patch (bug) >> - the new binary code for arrays did not handle arrays with dimension of >> zero >> - fix it and add a new test case for array.getResultSet() >> >> binary-fix-batch-with-generated-keys.patch (bug) >> - when executing batch with generated keys returned for each insert the same >> Query object is shared by the statements, and along with it the field >> definitions >> - when the prepare threshold is exceeded during the batch execution the >> first >> row of generated keys return text values and latter rows binary values >> - to fix we detect the situation and force a separate describe round-trip >> when >> batch is about to exceed the prepare threshold limit so that all generated >> in the same result set will be binary >> >> binary-fix-timezonetests.patch >> - based on my patch from 2007 to make the timezonetests 'fair' so that >> binary transfer has a chance of passing them >> - some tests in TimezoneTest never actually exercised the correct code paths >> because they just used getString which >> just returns the text from the server and does not cause the jdbc driver to >> try to parse the output >> - it is not possible in binary transfer mode for the jdbc driver to generate >> the exactly same getString output for >> all time values than what the server does - instead it uses the toString() >> method of the official java.sql classes >> - fixes the testSetTimestampOnTime, testSetTimestamp, testSetDate, >> testSetTime to use proper getTime/getTimestamp >> etc methods when fetching columns instead of relying on the getString >> - also changes the code verifying that inserts worked correctly to request >> the server for the actual values in the >> database casted to text type so that both binary and text transfer get the >> same results >> >> >> -Mikko >> >> >> -- >> Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org) >> To make changes to your subscription: >> http://www.postgresql.org/mailpref/pgsql-jdbc >> >> >
On 09/29/2011 07:10 PM, Dave Cramer wrote: > Interestingly I do not see these errors when I run the test inside of > eclipse, but I do when I run it from ant. The build.xml forces binary transfers by default. (maybe we should alter it to run all tests twice, once with binary transfer enabled, another with disabled?) > On Thu, Sep 29, 2011 at 7:53 AM, Dave Cramer<pg@fastcrypt.com> wrote: >> Mikko, >> >> I get the following errors after applying your patch. >> >> [junit] Testcase: >> testMaxFieldSize(org.postgresql.test.jdbc2.ResultSetTest): >> FAILED >> [junit] null expected:<[12345]> but was:<[09]> >> [junit] junit.framework.ComparisonFailure: null expected:<[12345]> >> but was:<[09]> >> [junit] at >> org.postgresql.test.jdbc2.ResultSetTest.testMaxFieldSize(ResultSetTest.java:174) Are you sure you applied the binary-fix-maxfieldlength-test.patch - the above failure looks like what it is trying to fix. >> [junit] Testcase: >> testGetTimestampWTZ(org.postgresql.test.jdbc2.TimestampTest): >> FAILED >> [junit] expected:<2000-07-07 11:00:00.123> but was:<2000-07-07 >> 11:00:00.122999> >> [junit] junit.framework.AssertionFailedError: expected:<2000-07-07 >> 11:00:00.123> but was:<2000-07-07 11:00:00.122999> >> [junit] at >> org.postgresql.test.jdbc2.TimestampTest.timestampTestWTZ(TimestampTest.java:436) >> [junit] at >> org.postgresql.test.jdbc2.TimestampTest.testGetTimestampWTZ(TimestampTest.java:188) >> [junit] >> [junit] >> [junit] Testcase: >> testSetTimestampWTZ(org.postgresql.test.jdbc2.TimestampTest): >> FAILED >> [junit] expected:<2000-07-07 11:00:00.123> but was:<2000-07-07 >> 11:00:00.122999> >> [junit] junit.framework.AssertionFailedError: expected:<2000-07-07 >> 11:00:00.123> but was:<2000-07-07 11:00:00.122999> >> [junit] at >> org.postgresql.test.jdbc2.TimestampTest.timestampTestWTZ(TimestampTest.java:436) >> [junit] at >> org.postgresql.test.jdbc2.TimestampTest.testSetTimestampWTZ(TimestampTest.java:258) This is quite interesting: Expected <2000-07-07 11:00:00.123> but was:<2000-07-07 11:00:00.122999> It looks like a rounding error. Do you happen to have integer datetime disabled in backend (which means that it is using float8 internally to store the time). -Mikko