Thread: experimental Netty-based protocol implementation

Possibly of interest to driver developers and those with too much spare time..

I spent some time over the weekend playing with a Netty-based
reimplementation of the v3 protocol. It seems somewhat promising, the
code is a bit more manageable than the current v3 code so far and
using Netty to run the I/O in a separate thread could help with the
various deadlock / timeout / memory consumption issues that affect the
current driver.

For example, I could see batch execution working roughly like this:

 * on the read side (upstream handler), buffer result rows as they
arrive, but turn off read interest when we have more than X amount of
data buffered
 * on the write side, if we notice that write interest was turned off
(too much buffered write data) then:
   * try to process some buffered results from the read side
   * wait for write interest to be reasserted

which is hopefully deadlock-free regardless of the amount of result data.

It might help with the suggestions for processing result data as it
arrives rather than buffering it all first:

 * upstream handler buffers result rows and turns off read interest
when more than X amount of data buffered
 * ResultSet handler pulls data when needed from the buffer, which may
make us reassert read interest
 * execution of a subsequent query forces the upstream handler to go
into "buffer everything" mode for any outstanding unconsumed execution
results

We can also do connection timeouts fairly cheaply without the current
horrible helper thread hack.

Patch is here: http://s3test.randomly.org/jdbc/0001-Skeleton-of-a-Netty-based-v3-protocol-implementation.patch
This gets as far as doing (MD5) authentication and processing startup
messages, then blows up because there's no query executor yet.
It also may have some odd things in there, as I'm using this as an
exercise to learn Netty & git at the same time, so YMMV.

Hopefully I'll have some more time to spend on this next weekend.

Oliver

For what it's worth, I've been working on an ActionScript PostgreSQL
driver [1] off and on for a year and change, and I've found the
event-driven I/O is a godsend when writing this sort of library. It's
much easier to just react to protocol-level messages as they happen
rather than trying to follow one of several specific sequences of
steps. I don't know anything about Netty, but from glancing through
the patch, Oliver, this actually looks rather similar to what I did.

Of course, in ActionScript, I had the luxury of defining an
asynchronous interface for the driver. In fact, due to Flash Player's
non-blocking nature, this was a requirement. With Java, we still need
to serve the blocking interface of methods like execute(), and that
might get a little tricky.

However, my overall point is that I'm very excited for this work. It
could get greatly simplify a lot of driver guts and help solve a
number of open "back-burner" issues.

[1]: https://github.com/deafbybeheading/pegasus (N.B.: although trust
and basic password authentication and most major aspects of both the
simple and extended query protocols work, this still hasn't had a
public release--caveat emptor)
---
Maciek Sakrejda | System Architect | Truviso

1065 E. Hillsdale Blvd., Suite 215
Foster City, CA 94404
(650) 242-3500 Main
www.truviso.com