Thread: JDBC drivers and streaming content
I'm about to start implemention streaming of queries to the server in the pgsql jdbc drivers when PreparedStatement is used with setBinaryStream... but before I get started, since I've never contributed before, I wanted to run it by everyone. I'm planning on making the following changes: 1. In QueryExecutor.sendQuery, if an object in m_binds is a java.io.Reader, then instead of calling toString and sending that, I'll read and send the contents in 16K increments. 2. In AbstractJdbc1Statement.setBinaryStream, instead of delegating to setBytes, I'll call bind directly, but send a custom java.io.Reader subclass (which adds the quotes and properly escapes the bytes and such) as the object. A couple questions: - There are a few constants that could probably be tuned for performance (ie, the size of the buffer used for streaming, and a threshold to avoid the streaming overhead for very short streams). Is there a fairly standard way to handle this stuff in the JDBC drivers? Should it be made configurable to the user? Read from a properties file? Stored in any specific class as a constant? - It seems to be quite a pain that org.postgresql.core.Encoding works only with String, and can't copy into a provided char[] -- this will mean creating a large number of String objects during the streaming. I could fix this easily with java.nio.CharsetEncoder, but it would make the code dependent on JDK 1.4. Not desired? - If there's a general dislike for runtime testing of object classes among the developer community, then I could, instead of special-casing Reader in sendQuery, wrap *all* PreparedStatement parameters in an interface that has a getReader method, and provide a default non-streaming implementation that uses StringReader. This is more intrusive, but I'll go whichever way makes it more likely for the patch to be committed. - Am I missing anything? Is this harder than it seems? Seems like someone would have done it already... -- www.designacourse.com The Easiest Way to Train Anyone... Anywhere. Chris Smith - Lead Software Developer/Technical Trainer MindIQ Corporation
Chris, It's already being done, you should post this to the jdbc list. Dave On Fri, 2003-01-24 at 11:07, Chris Smith wrote: > I'm about to start implemention streaming of queries to the server in the > pgsql jdbc drivers when PreparedStatement is used with setBinaryStream... > but before I get started, since I've never contributed before, I wanted to > run it by everyone. > > I'm planning on making the following changes: > > 1. In QueryExecutor.sendQuery, if an object in m_binds is a java.io.Reader, > then instead of calling toString and sending that, I'll read and send the > contents in 16K increments. > > 2. In AbstractJdbc1Statement.setBinaryStream, instead of delegating to > setBytes, I'll call bind directly, but send a custom java.io.Reader subclass > (which adds the quotes and properly escapes the bytes and such) as the > object. > > A couple questions: > > - There are a few constants that could probably be tuned for performance > (ie, the size of the buffer used for streaming, and a threshold to avoid the > streaming overhead for very short streams). Is there a fairly standard way > to handle this stuff in the JDBC drivers? Should it be made configurable to > the user? Read from a properties file? Stored in any specific class as a > constant? > > - It seems to be quite a pain that org.postgresql.core.Encoding works only > with String, and can't copy into a provided char[] -- this will mean > creating a large number of String objects during the streaming. I could fix > this easily with java.nio.CharsetEncoder, but it would make the code > dependent on JDK 1.4. Not desired? > > - If there's a general dislike for runtime testing of object classes among > the developer community, then I could, instead of special-casing Reader in > sendQuery, wrap *all* PreparedStatement parameters in an interface that has > a getReader method, and provide a default non-streaming implementation that > uses StringReader. This is more intrusive, but I'll go whichever way makes > it more likely for the patch to be committed. > > - Am I missing anything? Is this harder than it seems? Seems like someone > would have done it already... > > -- > www.designacourse.com > The Easiest Way to Train Anyone... Anywhere. > > Chris Smith - Lead Software Developer/Technical Trainer > MindIQ Corporation > > > ---------------------------(end of broadcast)--------------------------- > TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org -- Dave Cramer <dave@fastcrypt.com> Cramer Consulting
Actually, I originally sent that message a week ago, and it got posted then and again now. Not sure exactly why it was repeated... in any case, the work I was planning on doing is pretty much done; I want to do some more testing and refinement to handle performance issues, and I can post a patch. Then again, if you've already got it done (but, it seems, not released) then I won't do so. -- www.designacourse.com The Easiest Way to Train Anyone... Anywhere. Chris Smith - Lead Software Developer/Technical Trainer MindIQ Corporation