Thread: Re: Too much data received?

Re: Too much data received?

From
Steve Wampler
Date:
Steve Wampler wrote:
>
...
> Too much data was received.
>         at org.postgresql.PG_Stream.ReceiveString(PG_Stream.java:306)
>         at org.postgresql.Connection.ExecSQL(Connection.java:532)
>         at org.postgresql.jdbc2.Statement.execute(Statement.java:294)
>         at org.postgresql.jdbc2.Statement.executeUpdate(Statement.java:78)
>         at OCS.database.DbServer.execute(DbServer.java:271)
>
> I seem to remember discussion on a limit in the jdbc driver that might
> be causing this, but couldn't find it in the psql-jdbc mail archives.
>
> Am I likely correct in this being the problem?  Is there a fix in the
> works?

I took a look at the code for the jdbc driver and see how to increase the
size of the receive string buffer.  (I think that there is no limit on
the size of a query/command, just on results - is that correct?)

Is any one looking at removing the limit entirely?  I can (and have)
bumped the limit up, but wonder if it might not be better to just avoid
the problem entirely (if the back end can handle sending an absurdly
long result, why shouldn't the front end accept it?).  Since the
data is coming from the backend (presumably a comparitively slow
operation), it doesn't seem that any performance hit would be
particularly noticable (if well designed) and could likely be restricted
to those cases where the results are absurdly long...


----
Steve Wampler-  SOLIS Project, National Solar Observatory
swampler@noao.edu

Re: Re: Too much data received?

From
Steve Wampler
Date:
Steve Wampler wrote:
>
> Steve Wampler wrote:
> >
> ...
> > Too much data was received.
> >         at org.postgresql.PG_Stream.ReceiveString(PG_Stream.java:306)
> >         at org.postgresql.Connection.ExecSQL(Connection.java:532)
> >         at org.postgresql.jdbc2.Statement.execute(Statement.java:294)
> >         at org.postgresql.jdbc2.Statement.executeUpdate(Statement.java:78)
> >         at OCS.database.DbServer.execute(DbServer.java:271)
> >
> > I seem to remember discussion on a limit in the jdbc driver that might
> > be causing this, but couldn't find it in the psql-jdbc mail archives.
> >
> > Am I likely correct in this being the problem?  Is there a fix in the
> > works?
>
> I took a look at the code for the jdbc driver and see how to increase the
> size of the receive string buffer.  (I think that there is no limit on
> the size of a query/command, just on results - is that correct?)
>
> Is any one looking at removing the limit entirely?  I can (and have)
> bumped the limit up, but wonder if it might not be better to just avoid
> the problem entirely (if the back end can handle sending an absurdly
> long result, why shouldn't the front end accept it?).  Since the
> data is coming from the backend (presumably a comparitively slow
> operation), it doesn't seem that any performance hit would be
> particularly noticable (if well designed) and could likely be restricted
> to those cases where the results are absurdly long...

I'm not sure if anyone else is interested, but I have a version of
the PostgreSQL jdbc driver that places no restrictions on the length
of a received string (except in places where there was already
a low (e.g. 4096) limit).  So you can insert/retrieve an arbitrary-length
text field to/from the backend.  If the length is <= 65535 bytes then
performance should be the same as in the current version.  There is
a performance hit for longer strings (briefly, the byte buffer grows by
50% each time it fills), but that's just the price for using absurdly
long text fields (it would be simple to parameterize the driver
to tune this performance, but I haven't done so to keep the API the
same.)

It could use some more rigorous testing, but it works for my purposes
just fine.

Is anyone interested in looking at this?  If so, someone will have to
tell me how to generate context diffs against the CVS repository.
(There aren't many changes to the code.)

Thanks!

(The problem originated when storying XMLized java objects into the
database - the jdbc driver couldn't handle the long strings produced
by the JSX java-to/from-xml package.)

--
Steve Wampler-  SOLIS Project, National Solar Observatory
swampler@noao.edu

Re: Too much data received?

From
Barry Lind
Date:
Steve,

In looking at the source code I couldn't explain why you would be
getting the error below.  (All of the places that I see a max buffer
size specified, seem to be appropriate and shouldn't generate the error
you are seeing).  Can you send a test case to the list for me to
investigate this further?

thanks,
--Barry

Steve Wampler wrote:

> Steve Wampler wrote:
>
> ...
>
>>Too much data was received.
>>        at org.postgresql.PG_Stream.ReceiveString(PG_Stream.java:306)
>>        at org.postgresql.Connection.ExecSQL(Connection.java:532)
>>        at org.postgresql.jdbc2.Statement.execute(Statement.java:294)
>>        at org.postgresql.jdbc2.Statement.executeUpdate(Statement.java:78)
>>        at OCS.database.DbServer.execute(DbServer.java:271)
>>
>>I seem to remember discussion on a limit in the jdbc driver that might
>>be causing this, but couldn't find it in the psql-jdbc mail archives.
>>
>>Am I likely correct in this being the problem?  Is there a fix in the
>>works?
>>
>
> I took a look at the code for the jdbc driver and see how to increase the
> size of the receive string buffer.  (I think that there is no limit on
> the size of a query/command, just on results - is that correct?)
>
> Is any one looking at removing the limit entirely?  I can (and have)
> bumped the limit up, but wonder if it might not be better to just avoid
> the problem entirely (if the back end can handle sending an absurdly
> long result, why shouldn't the front end accept it?).  Since the
> data is coming from the backend (presumably a comparitively slow
> operation), it doesn't seem that any performance hit would be
> particularly noticable (if well designed) and could likely be restricted
> to those cases where the results are absurdly long...
>
>
> ----
> Steve Wampler-  SOLIS Project, National Solar Observatory
> swampler@noao.edu
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
>
>



Re: Too much data received?

From
Steve Wampler
Date:
Barry Lind wrote:
>
> Steve,
>
> In looking at the source code I couldn't explain why you would be
> getting the error below.  (All of the places that I see a max buffer
> size specified, seem to be appropriate and shouldn't generate the error
> you are seeing).  Can you send a test case to the list for me to
> investigate this further?
>
> thanks,
> --Barry
>
> Steve Wampler wrote:
>
> > Steve Wampler wrote:
> >
> > ...
> >
> >>Too much data was received.
> >>        at org.postgresql.PG_Stream.ReceiveString(PG_Stream.java:306)
> >>        at org.postgresql.Connection.ExecSQL(Connection.java:532)
> >>        at org.postgresql.jdbc2.Statement.execute(Statement.java:294)
> >>        at org.postgresql.jdbc2.Statement.executeUpdate(Statement.java:78)
> >>        at OCS.database.DbServer.execute(DbServer.java:271)

Hi Barry,

Hmmm, maybe the problem is only with the handling of error responses from
the back end?

The problem arises because PG_Stream.ReceiveString calls in Connection.ExecSQL
class put a bound on the length of strings received from the backend.

I think I originally saw the problem when an INSERT that included a
*very* long text field entry generated an exception (I had dropped a
function used as a trigger on inserts)) - the message that
comes back from the back end includes (as near as I can remember)
the original insert text, generating the above.  The exception above
went away when I bumped the size of receive_sbuf to something larger
than 8096 [this was with jdbc7.1-1.2.jar] and changed the calls
to use the new size where they had been using 8096 before.  I bumped it to
262144, incidently - these were large text fields.  This enabled me to
find the other error and fix it.

I'm not sure if ReceiveString is used when dealing with INSERT/SELECT of
text fields or not - I had assumed it was, but if you don't see the problem
then perhaps it's limited only to similar error cases?  If that's the case, then
there's really no need for the changes I've made!  [And I'll happily
drop them!] (but perhaps there should be another way to handle this situation?)

It's a little difficult for me to produce a small test case - this arose
when running an application that's pretty large and uses a lot of custom
classes -as well as needing several other applications (logging service,
message service, etc.).  I'm reluctant to try and pare it down if this was
really only a problem when dealing with error message text.

If you'd like to pursue it (that is, this is more than a problem when
receiving the text from error messages), let me know and I'll see if I
can generate it again.

Thanks!
--
Steve Wampler-  SOLIS Project, National Solar Observatory
swampler@noao.edu

Re: Re: Too much data received?

From
Peter Eisentraut
Date:
Steve Wampler writes:

> I think I originally saw the problem when an INSERT that included a
> *very* long text field entry generated an exception (I had dropped a
> function used as a trigger on inserts)) - the message that
> comes back from the back end includes (as near as I can remember)
> the original insert text, generating the above.

That would explain it perfectly.

> The exception above went away when I bumped the size of receive_sbuf
> to something larger than 8096 [this was with jdbc7.1-1.2.jar] and
> changed the calls to use the new size where they had been using 8096
> before.  I bumped it to 262144, incidently - these were large text
> fields.  This enabled me to find the other error and fix it.

Well, the longest piece of data you can store in a column is around 1GB,
so we obviously need to find a more flexible solution.

--
Peter Eisentraut   peter_e@gmx.net   http://funkturm.homeip.net/~peter