Thread: JDBC drivers and streaming content

JDBC drivers and streaming content

From
"Chris Smith"
Date:
I'm about to start implemention streaming of queries to the server in the
pgsql jdbc drivers when PreparedStatement is used with setBinaryStream...
but before I get started, since I've never contributed before, I wanted to
run it by everyone.

I'm planning on making the following changes:

1. In QueryExecutor.sendQuery, if an object in m_binds is a java.io.Reader,
then instead of calling toString and sending that, I'll read and send the
contents in 16K increments.

2. In AbstractJdbc1Statement.setBinaryStream, instead of delegating to
setBytes, I'll call bind directly, but send a custom java.io.Reader subclass
(which adds the quotes and properly escapes the bytes and such) as the
object.

A couple questions:

- There are a few constants that could probably be tuned for performance
(ie, the size of the buffer used for streaming, and a threshold to avoid the
streaming overhead for very short streams).  Is there a fairly standard way
to handle this stuff in the JDBC drivers?  Should it be made configurable to
the user?  Read from a properties file?  Stored in any specific class as a
constant?

- It seems to be quite a pain that org.postgresql.core.Encoding works only
with String, and can't copy into a provided char[] -- this will mean
creating a large number of String objects during the streaming.  I could fix
this easily with java.nio.CharsetEncoder, but it would make the code
dependent on JDK 1.4.  Not desired?

- If there's a general dislike for runtime testing of object classes among
the developer community, then I could, instead of special-casing Reader in
sendQuery, wrap *all* PreparedStatement parameters in an interface that has
a getReader method, and provide a default non-streaming implementation that
uses StringReader.  This is more intrusive, but I'll go whichever way makes
it more likely for the patch to be committed.

- Am I missing anything?  Is this harder than it seems?  Seems like someone
would have done it already...

--
www.designacourse.com
The Easiest Way to Train Anyone... Anywhere.

Chris Smith - Lead Software Developer/Technical Trainer
MindIQ Corporation



Re: JDBC drivers and streaming content

From
Dave Cramer
Date:
Chris,

It's already being done, you should post this to the jdbc list.

Dave
On Fri, 2003-01-24 at 11:07, Chris Smith wrote:
> I'm about to start implemention streaming of queries to the server in the
> pgsql jdbc drivers when PreparedStatement is used with setBinaryStream...
> but before I get started, since I've never contributed before, I wanted to
> run it by everyone.
> 
> I'm planning on making the following changes:
> 
> 1. In QueryExecutor.sendQuery, if an object in m_binds is a java.io.Reader,
> then instead of calling toString and sending that, I'll read and send the
> contents in 16K increments.
> 
> 2. In AbstractJdbc1Statement.setBinaryStream, instead of delegating to
> setBytes, I'll call bind directly, but send a custom java.io.Reader subclass
> (which adds the quotes and properly escapes the bytes and such) as the
> object.
> 
> A couple questions:
> 
> - There are a few constants that could probably be tuned for performance
> (ie, the size of the buffer used for streaming, and a threshold to avoid the
> streaming overhead for very short streams).  Is there a fairly standard way
> to handle this stuff in the JDBC drivers?  Should it be made configurable to
> the user?  Read from a properties file?  Stored in any specific class as a
> constant?
> 
> - It seems to be quite a pain that org.postgresql.core.Encoding works only
> with String, and can't copy into a provided char[] -- this will mean
> creating a large number of String objects during the streaming.  I could fix
> this easily with java.nio.CharsetEncoder, but it would make the code
> dependent on JDK 1.4.  Not desired?
> 
> - If there's a general dislike for runtime testing of object classes among
> the developer community, then I could, instead of special-casing Reader in
> sendQuery, wrap *all* PreparedStatement parameters in an interface that has
> a getReader method, and provide a default non-streaming implementation that
> uses StringReader.  This is more intrusive, but I'll go whichever way makes
> it more likely for the patch to be committed.
> 
> - Am I missing anything?  Is this harder than it seems?  Seems like someone
> would have done it already...
> 
> --
> www.designacourse.com
> The Easiest Way to Train Anyone... Anywhere.
> 
> Chris Smith - Lead Software Developer/Technical Trainer
> MindIQ Corporation
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
-- 
Dave Cramer <dave@fastcrypt.com>
Cramer Consulting



Re: JDBC drivers and streaming content

From
"Chris Smith"
Date:
Actually, I originally sent that message a week ago, and it got posted then
and again now.  Not sure exactly why it was repeated... in any case, the
work I was planning on doing is pretty much done; I want to do some more
testing and refinement to handle performance issues, and I can post a patch.
Then again, if you've already got it done (but, it seems, not released) then
I won't do so.

--
www.designacourse.com
The Easiest Way to Train Anyone... Anywhere.

Chris Smith - Lead Software Developer/Technical Trainer
MindIQ Corporation