Thread: Re: Too much data received?
Steve Wampler wrote: > ... > Too much data was received. > at org.postgresql.PG_Stream.ReceiveString(PG_Stream.java:306) > at org.postgresql.Connection.ExecSQL(Connection.java:532) > at org.postgresql.jdbc2.Statement.execute(Statement.java:294) > at org.postgresql.jdbc2.Statement.executeUpdate(Statement.java:78) > at OCS.database.DbServer.execute(DbServer.java:271) > > I seem to remember discussion on a limit in the jdbc driver that might > be causing this, but couldn't find it in the psql-jdbc mail archives. > > Am I likely correct in this being the problem? Is there a fix in the > works? I took a look at the code for the jdbc driver and see how to increase the size of the receive string buffer. (I think that there is no limit on the size of a query/command, just on results - is that correct?) Is any one looking at removing the limit entirely? I can (and have) bumped the limit up, but wonder if it might not be better to just avoid the problem entirely (if the back end can handle sending an absurdly long result, why shouldn't the front end accept it?). Since the data is coming from the backend (presumably a comparitively slow operation), it doesn't seem that any performance hit would be particularly noticable (if well designed) and could likely be restricted to those cases where the results are absurdly long... ---- Steve Wampler- SOLIS Project, National Solar Observatory swampler@noao.edu
Steve Wampler wrote: > > Steve Wampler wrote: > > > ... > > Too much data was received. > > at org.postgresql.PG_Stream.ReceiveString(PG_Stream.java:306) > > at org.postgresql.Connection.ExecSQL(Connection.java:532) > > at org.postgresql.jdbc2.Statement.execute(Statement.java:294) > > at org.postgresql.jdbc2.Statement.executeUpdate(Statement.java:78) > > at OCS.database.DbServer.execute(DbServer.java:271) > > > > I seem to remember discussion on a limit in the jdbc driver that might > > be causing this, but couldn't find it in the psql-jdbc mail archives. > > > > Am I likely correct in this being the problem? Is there a fix in the > > works? > > I took a look at the code for the jdbc driver and see how to increase the > size of the receive string buffer. (I think that there is no limit on > the size of a query/command, just on results - is that correct?) > > Is any one looking at removing the limit entirely? I can (and have) > bumped the limit up, but wonder if it might not be better to just avoid > the problem entirely (if the back end can handle sending an absurdly > long result, why shouldn't the front end accept it?). Since the > data is coming from the backend (presumably a comparitively slow > operation), it doesn't seem that any performance hit would be > particularly noticable (if well designed) and could likely be restricted > to those cases where the results are absurdly long... I'm not sure if anyone else is interested, but I have a version of the PostgreSQL jdbc driver that places no restrictions on the length of a received string (except in places where there was already a low (e.g. 4096) limit). So you can insert/retrieve an arbitrary-length text field to/from the backend. If the length is <= 65535 bytes then performance should be the same as in the current version. There is a performance hit for longer strings (briefly, the byte buffer grows by 50% each time it fills), but that's just the price for using absurdly long text fields (it would be simple to parameterize the driver to tune this performance, but I haven't done so to keep the API the same.) It could use some more rigorous testing, but it works for my purposes just fine. Is anyone interested in looking at this? If so, someone will have to tell me how to generate context diffs against the CVS repository. (There aren't many changes to the code.) Thanks! (The problem originated when storying XMLized java objects into the database - the jdbc driver couldn't handle the long strings produced by the JSX java-to/from-xml package.) -- Steve Wampler- SOLIS Project, National Solar Observatory swampler@noao.edu
Steve, In looking at the source code I couldn't explain why you would be getting the error below. (All of the places that I see a max buffer size specified, seem to be appropriate and shouldn't generate the error you are seeing). Can you send a test case to the list for me to investigate this further? thanks, --Barry Steve Wampler wrote: > Steve Wampler wrote: > > ... > >>Too much data was received. >> at org.postgresql.PG_Stream.ReceiveString(PG_Stream.java:306) >> at org.postgresql.Connection.ExecSQL(Connection.java:532) >> at org.postgresql.jdbc2.Statement.execute(Statement.java:294) >> at org.postgresql.jdbc2.Statement.executeUpdate(Statement.java:78) >> at OCS.database.DbServer.execute(DbServer.java:271) >> >>I seem to remember discussion on a limit in the jdbc driver that might >>be causing this, but couldn't find it in the psql-jdbc mail archives. >> >>Am I likely correct in this being the problem? Is there a fix in the >>works? >> > > I took a look at the code for the jdbc driver and see how to increase the > size of the receive string buffer. (I think that there is no limit on > the size of a query/command, just on results - is that correct?) > > Is any one looking at removing the limit entirely? I can (and have) > bumped the limit up, but wonder if it might not be better to just avoid > the problem entirely (if the back end can handle sending an absurdly > long result, why shouldn't the front end accept it?). Since the > data is coming from the backend (presumably a comparitively slow > operation), it doesn't seem that any performance hit would be > particularly noticable (if well designed) and could likely be restricted > to those cases where the results are absurdly long... > > > ---- > Steve Wampler- SOLIS Project, National Solar Observatory > swampler@noao.edu > > ---------------------------(end of broadcast)--------------------------- > TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org > >
Barry Lind wrote: > > Steve, > > In looking at the source code I couldn't explain why you would be > getting the error below. (All of the places that I see a max buffer > size specified, seem to be appropriate and shouldn't generate the error > you are seeing). Can you send a test case to the list for me to > investigate this further? > > thanks, > --Barry > > Steve Wampler wrote: > > > Steve Wampler wrote: > > > > ... > > > >>Too much data was received. > >> at org.postgresql.PG_Stream.ReceiveString(PG_Stream.java:306) > >> at org.postgresql.Connection.ExecSQL(Connection.java:532) > >> at org.postgresql.jdbc2.Statement.execute(Statement.java:294) > >> at org.postgresql.jdbc2.Statement.executeUpdate(Statement.java:78) > >> at OCS.database.DbServer.execute(DbServer.java:271) Hi Barry, Hmmm, maybe the problem is only with the handling of error responses from the back end? The problem arises because PG_Stream.ReceiveString calls in Connection.ExecSQL class put a bound on the length of strings received from the backend. I think I originally saw the problem when an INSERT that included a *very* long text field entry generated an exception (I had dropped a function used as a trigger on inserts)) - the message that comes back from the back end includes (as near as I can remember) the original insert text, generating the above. The exception above went away when I bumped the size of receive_sbuf to something larger than 8096 [this was with jdbc7.1-1.2.jar] and changed the calls to use the new size where they had been using 8096 before. I bumped it to 262144, incidently - these were large text fields. This enabled me to find the other error and fix it. I'm not sure if ReceiveString is used when dealing with INSERT/SELECT of text fields or not - I had assumed it was, but if you don't see the problem then perhaps it's limited only to similar error cases? If that's the case, then there's really no need for the changes I've made! [And I'll happily drop them!] (but perhaps there should be another way to handle this situation?) It's a little difficult for me to produce a small test case - this arose when running an application that's pretty large and uses a lot of custom classes -as well as needing several other applications (logging service, message service, etc.). I'm reluctant to try and pare it down if this was really only a problem when dealing with error message text. If you'd like to pursue it (that is, this is more than a problem when receiving the text from error messages), let me know and I'll see if I can generate it again. Thanks! -- Steve Wampler- SOLIS Project, National Solar Observatory swampler@noao.edu
Steve Wampler writes: > I think I originally saw the problem when an INSERT that included a > *very* long text field entry generated an exception (I had dropped a > function used as a trigger on inserts)) - the message that > comes back from the back end includes (as near as I can remember) > the original insert text, generating the above. That would explain it perfectly. > The exception above went away when I bumped the size of receive_sbuf > to something larger than 8096 [this was with jdbc7.1-1.2.jar] and > changed the calls to use the new size where they had been using 8096 > before. I bumped it to 262144, incidently - these were large text > fields. This enabled me to find the other error and fix it. Well, the longest piece of data you can store in a column is around 1GB, so we obviously need to find a more flexible solution. -- Peter Eisentraut peter_e@gmx.net http://funkturm.homeip.net/~peter