Thread: driver initialization and connection separation

Hello All,

I've got a Java based application that started life way back in Java 1.0.
It supports about 6 RDBMS engines, presently, including Postgres, of
course... When it needs to have multiple RDBMS connections, as it
occasionally does, it merely creates an instance of itself and calls
itself. This lets it connect to as many database instances as it likes and
never confuse or otherwise conflate them as the instances connect to
various RDBMSes. (For the curious, the API the application is coded
against does sql dialect translation on-the-fly, sophisticated error
handling, such as losing a network connection to an RDBMS, client-side,
RDBMS independent journaling, etc. and this is why new instances, though
seemingly a bit 'heavy-weight' is in fact a good, clean solution.)

However, I haven't ever previously asked it to use the embeded SSL
features before and this seems to present a problem - maybe more than one
problem!

I plan on eventually connecting a Postgres instance to the wilds of the
internet. There is no need to validate users via certificates, though
that's a cool thing to do that we'll eventually get to, and in fact
forcing client authentication would be a real show-stopper at this point.
So I was pleased to see the NonValidating Factory feature. I got the
connectivity working just fine, thank you, but when I then asked the
application to switch between encrypted and non-encrypted connections, it
"broke." A modest amount of testing showed that when the app. attempted to
reload the driver, the Postgres JDBC driver wasn't initializing as
outlined here:

http://java.sun.com/j2se/1.4.2/docs/api/java/lang/Class.html#forName%28java.lang.String%29

I would have taken "initialization" to mean removing components (like
NonValidatingFactory and WrappedFactory). Please note that the applciation
did not call deregisterDriver as outlined here:

http://java.sun.com/j2se/1.4.2/docs/api/java/sql/DriverManager.html#deregisterDriver%28java.sql.Driver%29

Rather, the app. just calls Class.forName() as it typically would in
making a connection. ...Driver initialization has never before been an
issue, so I'd never even considered that anything else was necessary.

My supposition is that because the driver already found itself loaded, it
just left itself in place.

I'm considering what to do about this; On the one hand, to my mind this
would be considered a bug, but on the other, it might be deemed that the
proper thing to do to deregister the driver - deregistering wouldn't be a
work-around, per se. (I'm assuming that de-registering the driver works to
solve this problem!) Still, what, then, would "initialize" mean, as
describe above, if not to clear out issues like this?

I was also thinking about the same issue for the "WrappedFactory", which
I'm not presently using but well might.

Because the application creates multiple instances for multiple
connections, I presume existing connections are safe from disruption by
loading an incompatible driver setup - if anyone knows otherwise, please
comment! I'm thinking that the driver instances will be unique to each
instance of my class that calls them, but at least I know I am making an
assumption.

Meanwhile, I'm off to see if in fact deregistering the driver works...

Thanks for your thoughts.

Regards,
Richard


--
Richard Troy, Chief Scientist
Science Tools Corporation
510-717-6942
rtroy@ScienceTools.com, http://ScienceTools.com/

Richard Troy wrote:

> I got the
> connectivity working just fine, thank you, but when I then asked the
> application to switch between encrypted and non-encrypted connections, it
> "broke."

What were you trying to do exactly, and how did it break?

> A modest amount of testing showed that when the app. attempted to
> reload the driver, the Postgres JDBC driver wasn't initializing as
> outlined here:
>
> http://java.sun.com/j2se/1.4.2/docs/api/java/lang/Class.html#forName%28java.lang.String%29

Well, classes only ever get initialized once.

-O

On Sun, 31 Jan 2010, Oliver Jowett wrote:
>
> Richard Troy wrote:
>
> > I got the
> > connectivity working just fine, thank you, but when I then asked the
> > application to switch between encrypted and non-encrypted connections, it
> > "broke."
>
> What were you trying to do exactly, and how did it break?

Sorry Oliver, I meant to be clear.

The application can connect to any RDBMS, Postgres, Oracle, Ingres,
whatever, and reconnect to any other, whenever the user wants. But once it
has connected to Postgres using the SSL feature, all future connections
using Postgres will use SSL. The only solution thus far is to kill the
application and then start it again - there is no switching between using
SSL and not using SSL, even though the driver is officially reloaded using
Class.forName(driverClassName).

Note that the official Java documentation from Sun says that doing so is
equivalent to Class.forName(className, true, currentLoader) which is using
the method for.Name(name, initialize, ClassLoader) - I cited a URL to
their site to support this.

To me this says that subsequent executions of Class.forName("driver")
should yield identical results as with the first time it's executed, but
that's demonstrably not true with Postgres - it "remembers" that it was
asked to use SSL in the past and it continues to want to do so in the
future.

It seems to me that there's something missing from the initialization
code, namely to remove the SSL features if the driver was already loaded.
It should only be loaded when a URL asking for it comes along with the
option ssl=true (for example). However, one could easily argue that it's
not the initialization code that needs help but the code that constructs a
new connection; it makes just as much sense that the request for a new
connection give the appropriate type of connection, with or without SSL
depending on whether the option in the URL was specified.

...I readily admit I haven't tried making multiple, simultaneous
connections from the same instance of the driver; I presume that's
possible but I've never needed. It seems to me this is
much more a question of each individual connection than the driver itself.

...Your follow-on comment (below) about classes being initialized only
once isn't helpful here because it isn't clear to me whether this is a
case of initializing an object or loading _code_ that instantiates
objects. In other words, even though I've been using Java since it's first
release, I've never before had to worry about the code that instantiates
objects changing while my code is running, so I haven't thought about it
much. To my mind, each instance of my objects which then instantiate other
objects - like JDBC connection objects - should get clean, separate and
distinct objects, so in effect the driver code is loaded multiple times.
But as I said, I could be wrong - I don't know much about how the JVM
manages code that's loaded, but my suspicion is that it'll manage the
individual instances of loading the same driver separately.

Meanwhile, I just added this bit of code prior to the call to
Class.forName():

  for (Enumeration fu = DriverManager.getDrivers(); fu.hasMoreElements();)
  {
     DriverManager.deregisterDriver((java.sql.Driver)fu.nextElement());
  }

As I said, I don't have to worry about multiple connections within a
single instance of this class, so deregistering all the drivers is fine -
there should only ever be one connection and we're about to replace it. If
there's an unacceptable performance hit for unloading a driver when it
isn't needed, then I'll think about adding to this. But if it works, it's
staying! -smile- I'll probably test it in the next few minutes...


Regards,
Richard

>
> > A modest amount of testing showed that when the app. attempted to
> > reload the driver, the Postgres JDBC driver wasn't initializing as
> > outlined here:
> >
> > http://java.sun.com/j2se/1.4.2/docs/api/java/lang/Class.html#forName%28java.lang.String%29
>
> Well, classes only ever get initialized once.
>
> -O
>

--
Richard Troy, Chief Scientist
Science Tools Corporation
510-717-6942
rtroy@ScienceTools.com, http://ScienceTools.com/

Richard Troy wrote:

> It seems to me that there's something missing from the initialization
> code, namely to remove the SSL features if the driver was already loaded.
> It should only be loaded when a URL asking for it comes along with the
> option ssl=true (for example). However, one could easily argue that it's
> not the initialization code that needs help but the code that constructs a
> new connection; it makes just as much sense that the request for a new
> connection give the appropriate type of connection, with or without SSL
> depending on whether the option in the URL was specified.

Driver initialization has nothing to do with this; you are making
assumptions about Class.forName() that are not correct. It does not
"initialize" the driver in any way other than what the JVM requires for
any other class (see below)

If you get a different sort of SSL-vs-not-SSL connection for the same
connection URL depending on what connections you got in the past - that
*is* a bug. Do you have a testcase showing the problem?

> ...Your follow-on comment (below) about classes being initialized only
> once isn't helpful here because it isn't clear to me whether this is a
> case of initializing an object or loading _code_ that instantiates
> objects. In other words, even though I've been using Java since it's first
> release, I've never before had to worry about the code that instantiates
> objects changing while my code is running, so I haven't thought about it
> much. To my mind, each instance of my objects which then instantiate other
> objects - like JDBC connection objects - should get clean, separate and
> distinct objects, so in effect the driver code is loaded multiple times.
> But as I said, I could be wrong - I don't know much about how the JVM
> manages code that's loaded, but my suspicion is that it'll manage the
> individual instances of loading the same driver separately.

Class.forName()'s comment about initialization is talking about class
initialization as described in the JVM spec - it involves (for example)
running static initializer blocks in the class. This is guaranteed to
happen before certain uses of the class (e.g. calling any method), and
can be forced early via Class.forName(). But it is only ever done once
per class.

If you load the same class in a different classloader, it's effectively
a different class, so will be separately initialized. (But I don't
recommend doing that with the JDBC driver, because it'll mess with
driver registration)

-O

On Sun, 31 Jan 2010, Oliver Jowett wrote:
> > It seems to me that there's something missing from the initialization
> > code, namely to remove the SSL features if the driver was already loaded.
> > It should only be loaded when a URL asking for it comes along with the
> > option ssl=true (for example). However, one could easily argue that it's
> > not the initialization code that needs help but the code that constructs a
> > new connection; it makes just as much sense that the request for a new
> > connection give the appropriate type of connection, with or without SSL
> > depending on whether the option in the URL was specified.
>
> Driver initialization has nothing to do with this; you are making
> assumptions about Class.forName() that are not correct. It does not
> "initialize" the driver in any way other than what the JVM requires for
> any other class (see below)

I'm happy to conceed to your points on "initialization."

> If you get a different sort of SSL-vs-not-SSL connection for the same
> connection URL depending on what connections you got in the past - that
> *is* a bug. Do you have a testcase showing the problem?

Well, that's basically it:  Connect with any Postgres JDBC valid URL you
like as many times as you like and it always works perfectly so long as
that URL doesn't ask for SSL, but once you have connected using SSL, the
driver will thereafter ONLY connect with SSL whether the URL asks for it
or not. IF the URL happens to want an SSL connection, it'll work, even if
it's to a different server. Or, at least, so showed my testing last night.

However, I think I just figured it out: Even though it's been in
production for literally a dozen years, I think I just found a bug with
our URL processing code that no one had reported. We're using URL
templates so we can handle everybody's URL format with ease. The ones for
Postgres look like this:

jdbc:postgresql://%h:%p/%d
jdbc:postgresql://%h:%p/%d?%o
jdbc:postgresql://%h:%p/%d?user=%u&password=%pw
jdbc:postgresql://%h:%p/%d?user=%u&password=%pw&%o

where %h is the host
      %p is the port
      %u is the username
      %pw is the password
 and  %o is the optional keyword=value[&keyword=value] string.

...I think what's happening is that the value that gets substituted for %o
isn't being cleared when it evaluates the new connection data; if there
was no "optional" information to add to the URL for the new connection,
it's retaining its old value. This would explain what I saw and not be an
issue with Postgres at all, obviously!

I've got some work to do to prove the point, and I've got the flu, so I'll
probably leave it until tomorrow. However, it fits the observed
behavior... I should have proved the point before opening my big yap! I'll
blame that social gaff on the flu! -wink-

Thanks for your patience,
Richard

On Sat, 30 Jan 2010, Richard Troy wrote:
>
> jdbc:postgresql://%h:%p/%d
> jdbc:postgresql://%h:%p/%d?%o
> jdbc:postgresql://%h:%p/%d?user=%u&password=%pw
> jdbc:postgresql://%h:%p/%d?user=%u&password=%pw&%o
>
> where %h is the host
>       %p is the port
>       %u is the username
>       %pw is the password
>  and  %o is the optional keyword=value[&keyword=value] string.
>
> ...I think what's happening is that the value that gets substituted for %o
> isn't being cleared when it evaluates the new connection data; if there
> was no "optional" information to add to the URL for the new connection,
> it's retaining its old value. This would explain what I saw and not be an
> issue with Postgres at all, obviously!
>
> I've got some work to do to prove the point, and I've got the flu, so I'll
> probably leave it until tomorrow. However, it fits the observed
> behavior... I should have proved the point before opening my big yap! I'll
> blame that social gaff on the flu! -wink-
>

...That was it alright: our code wasn't clearing the options - which in
this case included:

ssl=true&sslfactory=org.postgresql.ssl.NonValidatingFactory

Sorry to have troubled anyone,
Richard

Richard Troy wrote:
> The application can connect to any RDBMS, Postgres, Oracle, Ingres,
> whatever, and reconnect to any other, whenever the user wants. But once it
> has connected to Postgres using the SSL feature, all future connections
> using Postgres will use SSL. The only solution thus far is to kill the
> application and then start it again - there is no switching between using
> SSL and not using SSL, even though the driver is officially reloaded using
> Class.forName(driverClassName).

As Oliver pointed out, that does not reload the driver.

> Note that the official Java documentation from Sun says that doing so is
> equivalent to Class.forName(className, true, currentLoader) which is using
> the method for.Name(name, initialize, ClassLoader) - I cited a URL to
> their site to support this.

Once a class loader has loaded the class, it does not reload it again but
determines that it already has done so.

> To me this says that subsequent executions of Class.forName("driver")
> should yield identical results as with the first time it's executed, but

To you, maybe, but not in reality.  Your perception will match reality once
you've read the documentation.
<http://java.sun.com/javase/6/docs/api/java/lang/ClassLoader.html>

If you use a different class loader you can load the class again, but that
poses other gotchas.

> that's demonstrably not true with Postgres - it "remembers" that it was
> asked to use SSL in the past and it continues to want to do so in the
> future.

Remembering that it used SSL is not curable by failing to reload the class.

> It seems to me that there's something missing from the initialization
> code, namely to remove the SSL features if the driver was already loaded.

The class is not re-initialized since you're using the same class loader and
it was already initialized once.  Ergo, nothing is missing from the
initialization.

> It should only be loaded when a URL asking for it comes along with the
> option ssl=true (for example). However, one could easily argue that it's
> not the initialization code that needs help but the code that constructs a
> new connection; it makes just as much sense that the request for a new
> connection give the appropriate type of connection, with or without SSL
> depending on whether the option in the URL was specified.

According to how I read
<http://jdbc.postgresql.org/documentation/84/connect.html>
you should be able to get a non-SSL connection by not specifying the "ssl"
property in the version of 'DriverManager.getConnection()' that takes a
'Properties' argument.

> ...I readily admit I haven't tried making multiple, simultaneous
> connections from the same instance of the driver; I presume that's
> possible but I've never needed. It seems to me this is
> much more a question of each individual connection than the driver itself.

You use the driver with static methods, such as
'DriverManager.getConnection()'.  AFAICS there is no access directly to driver
instances.

> ...Your follow-on comment (below) about classes being initialized only
> once isn't helpful here because it isn't clear to me whether this is a
> case of initializing an object or loading _code_ that instantiates

He wasn't talking about initializing objects.  He was talking about
initialization of the driver *class*, and it's entirely relevant.  Whether
it's helpful to you or not depends on whether you assimilate and act upon the
information.

> objects. In other words, even though I've been using Java since it's first
> release, I've never before had to worry about the code that instantiates
> objects changing while my code is running, so I haven't thought about it

Again, this is not about instantiation.  You really should think about it much.

> much. To my mind, each instance of my objects which then instantiate other
> objects - like JDBC connection objects - should get clean, separate and
> distinct objects, so in effect the driver code is loaded multiple times.

Not how it works.  Your mind needs to change on this.

> But as I said, I could be wrong - I don't know much about how the JVM
> manages code that's loaded, but my suspicion is that it'll manage the
> individual instances of loading the same driver separately.

Nope.  It's not about instances.

> Meanwhile, I just added this bit of code prior to the call to
> Class.forName():
>
>   for (Enumeration fu = DriverManager.getDrivers(); fu.hasMoreElements();)
>   {
>      DriverManager.deregisterDriver((java.sql.Driver)fu.nextElement());
>   }
>
> As I said, I don't have to worry about multiple connections within a
> single instance of this class, so deregistering all the drivers is fine -

I am not so sure about that.  It seems like an awful lot of work, and how will
it reregister?  Once loaded and initialized, a class will not reinitialize
unless it's actually garbage collected first.  Since registration is part of
intialization, and you don't get to reinitialize, I don't think you'll get the
class to re-register.  Deregistration is not idiomatic and I expect it will
cause heartache.  Better go with a conventional approach.

> there should only ever be one connection and we're about to replace it. If
> there's an unacceptable performance hit for unloading a driver when it

Deregistration != unloading.

> isn't needed, then I'll think about adding to this. But if it works, it's
> staying! -smile- I'll probably test it in the next few minutes...

Better would be to use the correct approach, which might be the
'DriverManager.getConnection()' call to which I just alluded.  I haven't
tested it yet - why don't you and let us know how it works?

--
Lew