Re: Synchronisation problems in COPY IN - Mailing list pgsql-jdbc

Hi again,

We have corrected to program to wait for the thread to die before closing the db-connection - see attached source.

But we still see problems related to overwritten data buffer. See below for errors using build 800 or 900 driver - the webpage is a bit uncertain ;-)

Any ideas?

Thanks in advance!
Kim

Sep 15, 2010 6:31:40 PM dk.dmi.appl.evejrfeeder.EvejrDB writeIndexedPointData
INFO: Inserting data in temporary_temperature_over_sea_2_m_3 at valid time 2010-09-15 12:00:00 for 38882 gridpoints
Sep 15, 2010 6:31:40 PM dk.dmi.appl.evejrfeeder.EvejrDBCopy run
INFO: ERROR: extra data after last expected column Where: COPY temporary_temperature_over_sea_2_m_3, line 186: "60000187,2010-09-15 12:00:00,288.10498046860000746,2010-09-15 12:00:00,287.8037109375"
Sep 15, 2010 6:31:40 PM dk.dmi.appl.evejrfeeder.EvejrDBCopy run
INFO: ERROR: extra data after last expected column Where: COPY temporary_temperature_over_sea_2_m_3, line 186: "60000187,2010-09-15 12:00:00,288.10498046860000746,2010-09-15 12:00:00,287.8037109375"
org.postgresql.util.PSQLException: ERROR: extra data after last expected column Where: COPY temporary_temperature_over_sea_2_m_3, line 186: "60000187,2010-09-15 12:00:00,288.10498046860000746,2010-09-15 12:00:00,287.8037109375"at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:929)at org.postgresql.core.v3.QueryExecutorImpl.writeToCopy(QueryExecutorImpl.java:838)at org.postgresql.core.v3.CopyInImpl.writeToCopy(CopyInImpl.java:53)at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:179)at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:161)at dk.dmi.appl.evejrfeeder.EvejrDBCopy.run(Unknown Source)at java.lang.Thread.run(Thread.java:619)
Sep 15, 2010 6:31:43 PM dk.dmi.appl.evejrfeeder.EvejrDB writeIndexedPointData



Sep 15, 2010 6:52:21 PM dk.dmi.appl.evejrfeeder.EvejrDB writeIndexedPointData
INFO: Inserting data in temporary_wind_speed_150_m_3 at valid time 2010-09-15 12:00:00 for 38882 gridpoints
Sep 15, 2010 6:52:21 PM dk.dmi.appl.evejrfeeder.EvejrDBCopy run
INFO: ERROR: missing data for column "wind_speed_150_m" Where: COPY temporary_wind_speed_150_m_3, line 173: "60000188,2010-09-15 12:00:00"
Sep 15, 2010 6:52:21 PM dk.dmi.appl.evejrfeeder.EvejrDBCopy run
INFO: ERROR: missing data for column "wind_speed_150_m" Where: COPY temporary_wind_speed_150_m_3, line 173: "60000188,2010-09-15 12:00:00"
org.postgresql.util.PSQLException: ERROR: missing data for column "wind_speed_150_m" Where: COPY temporary_wind_speed_150_m_3, line 173: "60000188,2010-09-15 12:00:00"at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:929)at org.postgresql.core.v3.QueryExecutorImpl.writeToCopy(QueryExecutorImpl.java:838)at org.postgresql.core.v3.CopyInImpl.writeToCopy(CopyInImpl.java:53)at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:179)at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:161)at dk.dmi.appl.evejrfeeder.EvejrDBCopy.run(Unknown Source)at java.lang.Thread.run(Thread.java:619)
Sep 15, 2010 6:52:24 PM dk.dmi.appl.evejrfeeder.EvejrDB writeIndexedPointDataSep 15, 2010 6:52:21 PM dk.dmi.appl.evejrfeeder.EvejrDB writeIndexedPointData
INFO: Inserting data in temporary_wind_speed_150_m_3 at valid time 2010-09-15 12:00:00 for 38882 gridpoints



Sep 15, 2010 6:56:47 PM dk.dmi.appl.evejrfeeder.EvejrDB writeIndexedPointData
INFO: Inserting data in temporary_relative_humidity_3 at valid time 2010-09-15 12:00:00 for 38882 gridpoints
Sep 15, 2010 6:56:47 PM dk.dmi.appl.evejrfeeder.EvejrDBCopy run
INFO: ERROR: invalid input syntax for type timestamp: "2010-09-" Where: COPY temporary_relative_humidity_3, line 173, column valid_at: "2010-09-"
Sep 15, 2010 6:56:47 PM dk.dmi.appl.evejrfeeder.EvejrDBCopy run
INFO: ERROR: invalid input syntax for type timestamp: "2010-09-" Where: COPY temporary_relative_humidity_3, line 173, column valid_at: "2010-09-"
org.postgresql.util.PSQLException: ERROR: invalid input syntax for type timestamp: "2010-09-" Where: COPY temporary_relative_humidity_3, line 173, column valid_at: "2010-09-"at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:929)at org.postgresql.core.v3.QueryExecutorImpl.writeToCopy(QueryExecutorImpl.java:838)at org.postgresql.core.v3.CopyInImpl.writeToCopy(CopyInImpl.java:53)at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:179)at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:161)at dk.dmi.appl.evejrfeeder.EvejrDBCopy.run(Unknown Source)at java.lang.Thread.run(Thread.java:619)
Sep 15, 2010 6:56:51 PM dk.dmi.appl.evejrfeeder.EvejrDB writeIndexedPointData

On 2010-09-14 09:14, Kim Bisgaard wrote:

Thanks everybody!

Yes it was tried with the 701 build (I am almost sure that we also tried with cvs-head as of 2 weeks back - will check up on that)

We will correct the connection.close() problem - thanks for pointing that out!

And probably (if we still get problems) try the new build 800 driver.

Thanks again,
Kim


On 2010-09-13 19:27, Samuel Gendler wrote:

Your error sure looks exactly like the problem that existed when using a Reader instead of InputStream with the older driver (701), though your description implies a different problem.

On Mon, Sep 13, 2010 at 7:17 AM, Kris Jurka <books@ejurka.com> wrote:

On Mon, 13 Sep 2010, Kim Bisgaard wrote:

We are occasionally experiencing problems with our JDBC implementation of COPY IN.

We have cut our program down to the attached example.

Do you get the failure you've shown from this example?  I get another failure from the connection being close before the last copy thread can complete its write operation.  This seems like an expected failure, but I can't reproduce yours.

org.postgresql.util.PSQLException: Database connection failed when canceling copy operation
       at org.postgresql.core.v3.QueryExecutorImpl.cancelCopy(QueryExecutorImpl.java:796)
       at org.postgresql.core.v3.CopyOperationImpl.cancelCopy(CopyOperationImpl.java:32)
       at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:150)
       at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:126)
       at EvejrDBCopy.run(EvejrDBCopy.java:34)

We use PostgreSQL 8.4, and JDBC 8.4-702.

Your stacktrace shows that you are using 8.4-701, not 702.  702 had some fixes to the copy code when using a Reader, but your code shows you are using an InputStream, so those changes shouldn't be relevent.

Kris Jurka

--
Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-jdbc