Thread: Pre-processing during build

I'm working on a change proposal to switch the build of the driver
from Ant to Maven [1].

Working on this change I found the use of pre-processing of template
files to create valid Java classes. In my experience this isn't very
common in Java libraries, other than to inject version numbers. To me,
it does not seem to be a very elegant solution, although I can confirm
that it works fine and has been present in the driver since the
beginning.

The use of these templates is scattered around the codebase and
usually occurs when there is JDBC version-specific functionality to
use, e.g. Driver.java.in and PGConnectionPoolDataSource.java.in. [2]

The use of these templates means the code will not successfully
compile in an IDE, without running the build script first. I believe
the historical reason for part of this is to retain the
org.postgresql.Driver class to be used in a Class.forName method call
without the developer needing to worry about the JDBC spec version.
Please correct me if I'm wrong in this assumption.

Within the current build there is also the filtering of source
packages to only include the version of the JDBC spec you are
compiling for and any previous versions.

This can all be implemented in Maven, so won't prevent the project
using that for a build system. However it will make it possibly more
complicated than it needs to be, like the Ant build is now. The
negative of this is that it could put off new people building and
committing fixes and functionality to the codebase.

So, do we want to change this? It would be good to discuss this to
arrive at a solution that would best serve the users as well as
contributors (I'm aware that is rarely possible!)

[1] https://github.com/pgjdbc/pgjdbc/pull/322
[2] https://github.com/pgjdbc/pgjdbc/pull/322/files#r32378395

Marcus,

>Has anybody tried whether it is possible to simply load a JRE8-JDBC42.jar on JRE6?

As you are a fan of JEPs, you might know of
http://openjdk.java.net/jeps/238 : Multi-Version JAR Files
It does not "just work" yet.

Suppose you want implement PreparedStatement(...java.sql.SQLType)
features of JDBC 4.2 (see [1]).
Even if you compile that with target 1.7, JRE 7 might fail to load the
class as it won't be able to validate what that SQLType is. It just
does not exist in JRE7.

As far as I understand, the only manageable way of using "new
features" in "JDK6-7 jars" is to isolate JDK8-using methods to
JDK8-only-loaded classes. For instance, PreparedStatement41.java and
PreparedStatement42.java

[1]:
https://docs.oracle.com/javase/8/docs/api/java/sql/PreparedStatement.html#setObject-int-java.lang.Object-java.sql.SQLType-

Vladimir

On Mon, 15 Jun 2015 23:59:45 +0200, "Markus KARG" <markus@headcrashing.eu>
wrote:
> If it would be up to me, I would try to get rid of pre-processing if any
> possible, since it is a real p.i.t.a., as long as we can find a
different
> solution to provide the same number of supported JDKs and JDBC versions.
>
> The question is: How? Possibly by simply using "JRE8-JDBC42.jar" ALWAYS?

For Java 6 and 7 it would be possible to use the same codebase (if you
ignore certain types iirc), but for Java 8 you need a separate library due
to the introduction of the SQLType interface. And if you fully support JDBC
4.2, you also need the classes from java.time.

> Has anybody tried whether it is possible to simply load a
JRE8-JDBC42.jar
> on
> JRE6? I mean, not to actually invoke the new JDBC42 APIs, just to load
the
> JAR and invoke the JDBC3 APIs only for example. The APIs themselved are
> backwards compatible, and as long as we don't invoke the new APIs, no
> ClassNotFound should happen (AFAIK the JRE loads classes only at first
> actual instantiation, but not simply because it is contained in a loaded
> .class file as a parameter). I mean, as long as we do not use JRE8-only
> APIs
> inside the Driver, and as long as we don't write the .class files in
JRE8
> byte code, certainly.
>
> Or did I miss something in this theoretical approach?

JDBC APIs themselves are not backwards compatible (ie: they introduce new
types in the signature, or require you to handle new types in existing
methods), the API is only backwards compatible from the perspective of the
user.

Mark

On Mon, 15 Jun 2015 18:42:13 -0400, Sehrope Sarkuni <sehrope@jackdb.com>
wrote:
> I think a purely Maven based build with separate targets should be
possible
> via a generated-sources plugin. Off the top of my head I'm not sure
which
> plugins it would use though. It's been a while since I wrote anything
like
> that though I do remember it being a bit of a pain to get right. On the
> plus side we only have to figure it out once right? :)

My experience is that Ant gives you a lot more flexibility. For Jaybird I
considered moving to a Maven based build, but I finally decided against it
because it was too much hassle. The difference with the PostgreSQL JDBC is
that Jaybird uses JDBC version specific sources-folders with common classes
(and abstract classes for common implementation).

> I'm a big fan of Mavenizing the build process. A lot of the value of it
> will come from how it will simplify things like adding tests. It
eliminates
> a lot of the double and sometimes triple entry (i.e. add the test class,
> add it to a suite, add the suite to build.xml).

You don't need Maven to achieve that. For one you don't need testsuites,
with JUnit 3 it is a bit harder, but you could use a consistent naming
convention and filter tests in the Ant plugin, with JUnit 4 you could use
(class or instance) rules, or filtering based on annotations.

Mark

Dave> Apparently https://github.com/brettwooldridge/HikariCP uses this method;

As far as I understand, HikariCP is using dynamic bytecode generation,
so they do not have to include all the signatures in their sources.
On contrary, we need to implement 4.2 interfaces, so at some point we
would end up with SQLType in pgjdbc's signatures.

Mark> My experience is that Ant gives you a lot more flexibility

What that flexibility is good for?

The drawbacks of Ant are:
1) No "easy way to configure IDE". This includes "download javadoc and
source", adding dependencies to the classpath, etc, etc.

2) No easy way to run tests. With maven you just hit `mvn install` or
`mvn test` and it just works. With Ant you have to read instructions.

3) No easy way to test different versions. With maven, I can depend on
"snapshot" versions in my client application, and I easily can
debug&step-into.
Unfortunately, due to #1, debugging dependencies is not that easy.

4) No easy integration with other systems. For instance, if using
maven you can just add findbugs, sonar, etc, etc.
If you want to try recent Facebook's infer, you just hit `infer mvn install`

Well, gradle might be even better approach, however I have not used it
yet, so I have nothing to say here.

Vladimir

Vladimir,

you misunderstood my proposal. I do not want to have multiple versions of the same class. I want ONLY JDBC42 API
compileddown to Java 6 byte code in one single JAR. JDBC42 is backwards compatible to all older JDBC levels, and Java 7
and8 can load Java 6 byte code. 

Also the idea is NOT to allow Java 6 and 7 clients to INVOKE JDBC42 but just to load the JAR and invoke e. g. JDBC4
methodsthat are part of JDBC42 still (again, it's backwards compatible). 

Regarding your example: A Java 7 client will never try to call that methods as Java 7 clients do not even know that
sucha method exist. Only Java 8 clients will know. That method is more than clearly marked as "since 1.8" so a Java 7
applicationprogrammer does not even KNOW about it pure existence! 

So it boils down to verification and I doubt that the bytecode verifier will try to actually load java.sql.Type class.
Haveyou really tried this out or do you have another link where it is written that the byte code verifier will CHECK
theexistence of a parameter class when it verifies the loaded class? 

Regards
Markus

-----Original Message-----
From: Vladimir Sitnikov [mailto:sitnikov.vladimir@gmail.com]
Sent: Dienstag, 16. Juni 2015 00:48
To: Dave Cramer
Cc: Markus KARG; List
Subject: Re: [JDBC] Pre-processing during build

Marcus,

>Has anybody tried whether it is possible to simply load a JRE8-JDBC42.jar on JRE6?

As you are a fan of JEPs, you might know of
http://openjdk.java.net/jeps/238 : Multi-Version JAR Files
It does not "just work" yet.

Suppose you want implement PreparedStatement(...java.sql.SQLType)
features of JDBC 4.2 (see [1]).
Even if you compile that with target 1.7, JRE 7 might fail to load the
class as it won't be able to validate what that SQLType is. It just
does not exist in JRE7.

As far as I understand, the only manageable way of using "new
features" in "JDK6-7 jars" is to isolate JDK8-using methods to
JDK8-only-loaded classes. For instance, PreparedStatement41.java and
PreparedStatement42.java

[1]:
https://docs.oracle.com/javase/8/docs/api/java/sql/PreparedStatement.html#setObject-int-java.lang.Object-java.sql.SQLType-

Vladimir

Well, we actually opened this discussion because we WANT particularly Maven and we DO NOT want the flexibility of ANT
butfollow CoC, hence make PGJDBC at-most a simple, "standard" project with as less as possible customizations... The
lessflexibility you have the more standards you follow the easier new programmer's can get at speed with the project.
Moreflexiblity = more complexity = more customization = longer time to get about = less new programmers will join.
That'sthe idea behind this thread, actually. 

-----Original Message-----
From: pgsql-jdbc-owner@postgresql.org [mailto:pgsql-jdbc-owner@postgresql.org] On Behalf Of Mark Rotteveel
Sent: Dienstag, 16. Juni 2015 09:09
To: pgsql-jdbc@postgresql.org
Subject: Re: [JDBC] Pre-processing during build

On Mon, 15 Jun 2015 18:42:13 -0400, Sehrope Sarkuni <sehrope@jackdb.com>
wrote:
> I think a purely Maven based build with separate targets should be
possible
> via a generated-sources plugin. Off the top of my head I'm not sure
which
> plugins it would use though. It's been a while since I wrote anything
like
> that though I do remember it being a bit of a pain to get right. On the
> plus side we only have to figure it out once right? :)

My experience is that Ant gives you a lot more flexibility. For Jaybird I
considered moving to a Maven based build, but I finally decided against it
because it was too much hassle. The difference with the PostgreSQL JDBC is
that Jaybird uses JDBC version specific sources-folders with common classes
(and abstract classes for common implementation).

> I'm a big fan of Mavenizing the build process. A lot of the value of it
> will come from how it will simplify things like adding tests. It
eliminates
> a lot of the double and sometimes triple entry (i.e. add the test class,
> add it to a suite, add the suite to build.xml).

You don't need Maven to achieve that. For one you don't need testsuites,
with JUnit 3 it is a bit harder, but you could use a consistent naming
convention and filter tests in the Ant plugin, with JUnit 4 you could use
(class or instance) rules, or filtering based on annotations.

Mark


--
Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-jdbc

2015-06-16 22:30 GMT+03:00 Markus KARG <markus@headcrashing.eu>:
> The answer is pretty simple: Try it out. :-)
>
> Just compile a JRE 8 class down to byte code level 6 and load it on Java
> level 7. That's what I proposed. Nothing else. It really bet will work
> unless you try to INSTANTIATE JRE-only classes, but it should LOAD. And
> nothing more we need.

Markus, can you please be more explicit in your suggestion?

I did try a simple "Hello, world" and it does not run in stock JDKs of
MacOS: https://gist.github.com/vlsi/aeeb4a61d9c2b67ad213
Even if you manage to make that fly, that would be built on sand.

If you read Spring java support blog post
(https://spring.io/blog/2015/04/03/how-spring-achieves-compatibility-with-java-6-7-and-8),
you'll spot that they use separate _classes_ to isolate JDK8 features.

Vladimir

On Tue, 16 Jun 2015 17:19:27 +0300, Vladimir Sitnikov
<sitnikov.vladimir@gmail.com> wrote:
> Mark> My experience is that Ant gives you a lot more flexibility
>
> What that flexibility is good for?

I am not here to start a "my favorite build system"-war, I am just saying
that for a project like a JDBC driver, which needs to target different
versions of Java, the flexibility of Ant may be necessary, or getting it
working and working correctly on Maven may be more complex, especially as
Ant is already in place and working.

> The drawbacks of Ant are:
> 1) No "easy way to configure IDE". This includes "download javadoc and
> source", adding dependencies to the classpath, etc, etc.

Good point.

> 2) No easy way to run tests. With maven you just hit `mvn install` or
> `mvn test` and it just works. With Ant you have to read instructions.

That is a matter of using targetnames that follow a convention, correct
dependencies and a sane default target (in which case just running ant
could be sufficient to build and test).

> 3) No easy way to test different versions. With maven, I can depend on
> "snapshot" versions in my client application, and I easily can
> debug&step-into.
> Unfortunately, due to #1, debugging dependencies is not that easy.

Good point, you can achieve that with Ant+Ivy, though that will complicate
IDE integration further.

> 4) No easy integration with other systems. For instance, if using
> maven you can just add findbugs, sonar, etc, etc.
> If you want to try recent Facebook's infer, you just hit `infer mvn
> install`

SonarQube, findbugs and Infer support ant. It might require more explicit
config than in Maven, but that should be setup once and forget, for infer:
infer -- ant <your build target>

> Well, gradle might be even better approach, however I have not used it
> yet, so I have nothing to say here.

I use Gradle as well, but I don't know it too well yet. It is a bit of a
maven/ant hybrid, and I am not yet sure if I'd prefer it over Ant (or
Maven).

Mark

On Tue, 16 Jun 2015 21:18:17 +0200, "Markus KARG" <markus@headcrashing.eu>
wrote:
> To make things a bit more clear: It is NOT about whether or not an
> interface method has a parameter of an unknown type. It is solely about
> whether that method is actually INVOKED at runtime. As a JDBC42 is
unknown
> to a JDBC3 client, that client CANNOT invoke that method, hence NEVER a
> ClassNotFoundException can ever happen. Actually the cause of the
exception
> is not even the method invocation but it is solely the instantiation of
the
> parameter value which happens IN THE CALLER. We're pretty safe I think.
So
> still I do not see any need for separate JARs for the purpose of serving
> multiple JDBC levels, as long as the byte code is Java 6 level.

When I tried it in the past (for Jaybird) I ran into problems, it might
have been an IncompatibleClassChangeError (or subclass) on classloading
time, but I can't remember the exact details. I tried reading the JLS and
JVM spec on this point, but I find it hard to come to a conclusion, so I
will try a small experiment this weekend.

However that is half of your problem. The other half is that you risk
using classes, methods or maybe even features that are not available in a
lower Java version and that will not be detected compile time, but only at
runtime (and not just in newly added JDBC 4.2 methods).

On top of that, for a compliant JDBC 4.2 implementation, you must support
java.time objects in setObject, getObject, updateObject, etc. If you want
to do this in a unified source, you will need to load a delegated
implementation that is appropriate for the Java version, otherwise you will
get ClassNotFoundException.

For example in Jaybird we did this:
https://github.com/FirebirdSQL/jaybird/blob/master/src/main/org/firebirdsql/jdbc/field/FBField.java#L739

Which uses (for JDBC 4.2):
https://github.com/FirebirdSQL/jaybird/blob/master/src/jdbc_42/org/firebirdsql/jdbc/field/JDBC42ObjectConverter.java

And for JDBC 4.1 (and 4.0 in Jaybird 2.2):
https://github.com/FirebirdSQL/jaybird/blob/master/src/main/org/firebirdsql/jdbc/field/DefaultObjectConverter.java

Mark

On Tue, 16 Jun 2015 21:26:12 +0200, "Markus KARG" <markus@headcrashing.eu>
wrote:
> So it boils down to verification and I doubt that the bytecode verifier
> will try to actually load java.sql.Type class. Have you really tried
this
> out or do you have another link where it is written that the byte code
> verifier will CHECK the existence of a parameter class when it verifies
the
> loaded class?

It looks like the JLS and JVM specification allow for resolution of
symbolic references at classloading or "first use" and a JVM implementation
is free to choose: see
http://docs.oracle.com/javase/specs/jls/se8/html/jls-12.html#jls-12.1.2 and
http://docs.oracle.com/javase/specs/jls/se8/html/jls-12.html#jls-12.3

Although I am not entirely sure about this, I interpret this that loading
a class that has a method whose signature includes a type that is not
available (like SQLType on Java 7 or lower) could work on one JVM
implementation (late resolution), but not on others (early resolution).

Mark

On Wed, 17 Jun 2015 00:02:40 +0300, Vladimir Sitnikov
<sitnikov.vladimir@gmail.com> wrote:
> 2015-06-16 22:30 GMT+03:00 Markus KARG <markus@headcrashing.eu>:
>> The answer is pretty simple: Try it out. :-)
>>
>> Just compile a JRE 8 class down to byte code level 6 and load it on
Java
>> level 7. That's what I proposed. Nothing else. It really bet will work
>> unless you try to INSTANTIATE JRE-only classes, but it should LOAD. And
>> nothing more we need.
>
> Markus, can you please be more explicit in your suggestion?
>
> I did try a simple "Hello, world" and it does not run in stock JDKs of
> MacOS: https://gist.github.com/vlsi/aeeb4a61d9c2b67ad213
> Even if you manage to make that fly, that would be built on sand.

Good example: it demonstrates at least that using reflection (eg
getDeclaredMethods; or in this case privateGetDeclaredMethods) will lead to
a NoClassDefFoundError.

Mark

On Wed, 17 Jun 2015 06:15:47 -0400, Dave Cramer <pg@fastcrypt.com> wrote:
> On 17 June 2015 at 03:07, Mark Rotteveel <mark@lawinegevaar.nl> wrote:
>
>> On Wed, 17 Jun 2015 00:02:40 +0300, Vladimir Sitnikov
>> <sitnikov.vladimir@gmail.com> wrote:
>> > 2015-06-16 22:30 GMT+03:00 Markus KARG <markus@headcrashing.eu>:
>> >> The answer is pretty simple: Try it out. :-)
>> >>
>> >> Just compile a JRE 8 class down to byte code level 6 and load it on
>> Java
>> >> level 7. That's what I proposed. Nothing else. It really bet will
work
>> >> unless you try to INSTANTIATE JRE-only classes, but it should LOAD.
>> >> And
>> >> nothing more we need.
>> >
>> > Markus, can you please be more explicit in your suggestion?
>> >
>> > I did try a simple "Hello, world" and it does not run in stock JDKs
of
>> > MacOS: https://gist.github.com/vlsi/aeeb4a61d9c2b67ad213
>> > Even if you manage to make that fly, that would be built on sand.
>>
>> Good example: it demonstrates at least that using reflection (eg
>> getDeclaredMethods; or in this case privateGetDeclaredMethods) will
lead
>> to
>> a NoClassDefFoundError.
>>
>>
> I'm not sure this is a great example as Optional itself is a java 8
> construct.

Yes, and so is java.sql.SQLType. So if this doesn't work for Optional, it
also won't work for SQLType.

> Either way Spring is able to do this, as are others?

Spring uses a lot of reflection, proxies, byte code generation, etc to get
things done. I am not sure if you want to go that way.

Mark

>We're going to need some kind of preprocessing step to handle things like this.

Sehrope, Marr, sorry, I do not follow you.
I do not see how you prove preprocessing is required.

For instance,

org.postgresql.jdbc42.PreparedStatement extends
org.postgresql.jdbc4.PreparedStatement {
  public setObject(int parameterIndex, Object x, SQLType targetSqlType) {
     ...
  }
}

It does not require "reflection, proxies, byte code generation", etc, etc.
It is just a simple "extends".

Vladimir

On Wed, 17 Jun 2015 14:17:51 +0300, Vladimir Sitnikov
<sitnikov.vladimir@gmail.com> wrote:
>>We're going to need some kind of preprocessing step to handle things
like
>>this.
>
> Sehrope, Marr, sorry, I do not follow you.
> I do not see how you prove preprocessing is required.

I was talking about the situation where you have a single implementation
of the interfaces, but even the way of working below has its problems.

> For instance,
>
> org.postgresql.jdbc42.PreparedStatement extends
> org.postgresql.jdbc4.PreparedStatement {
>   public setObject(int parameterIndex, Object x, SQLType targetSqlType)
{
>      ...
>   }
> }
>
> It does not require "reflection, proxies, byte code generation", etc,
etc.
> It is just a simple "extends".

That is almost what PostgreSQL uses now, but this is not going to work if
you compile with Java 8 and assume that
org.postgresql.jdbc4.PreparedStatement would then also be new-able when the
same jar is used under Java 6 or 7, because
org.postgresql.jdbc4.PreparedStatement would need to be abstract at compile
time as it doesn't contain the methods required by the Java 8 (JDBC 4.2)
API during compilation.

So getting this to work would need some form of reflection (to get the
right type at runtime based on the Java version), some preprocessing (as
done currently) to get around the compilation problem or some byte code
generation/modification to "unabstract"
org.postgresql.jdbc4.PreparedStatement after compilation, or some form of
tiered compilation (where the org.postgresql.jdbc4.PreparedStatement is
compiled with Java 7, and org.postgresql.jdbc42.PreparedStatement with Java
8; this might be more complex than the existing solution.

Mark

On Wed, 17 Jun 2015 13:52:52 +0200, Mark Rotteveel <mark@lawinegevaar.nl>
wrote:
> That is almost what PostgreSQL uses now, but this is not going to work
if
> you compile with Java 8 and assume that
> org.postgresql.jdbc4.PreparedStatement would then also be new-able when
the
> same jar is used under Java 6 or 7, because
> org.postgresql.jdbc4.PreparedStatement would need to be abstract at
compile
> time as it doesn't contain the methods required by the Java 8 (JDBC 4.2)
> API during compilation.
>
> So getting this to work would need some form of reflection (to get the
> right type at runtime based on the Java version), some preprocessing (as
> done currently) to get around the compilation problem or some byte code
> generation/modification to "unabstract"
> org.postgresql.jdbc4.PreparedStatement after compilation, or some form
of
> tiered compilation (where the org.postgresql.jdbc4.PreparedStatement is
> compiled with Java 7, and org.postgresql.jdbc42.PreparedStatement with
Java
> 8; this might be more complex than the existing solution.

I just realized it might actually work: some (maybe all) methods added in
the JDBC API for Java 8 were added as default interface methods (with an
implementation that throws UnsupportedOperationException), so compilation
would succeed for org.postgresql.jdbc4.PreparedStatement without having an
implementation for the new methods.

You'd still need reflection or an other trick to decide based on the Java
version which classes (Statement, PreparedStatement, ResultSet, etc) to
instantiate.

Mark

>So getting this to work would need some form of reflection (to get the
right type at runtime based on the Java version)

Reflection is sufficient, isn't it?

I think reflection would be rather clean solution here.
It is more IDE-friendly and developer-friendly than pre-processing.
Reflection is much easier to debug than byte-code generation.

So, do you see drawbacks with using reflection to select the specific
implementation?

From implementation point of view it just replaces "preprocessing"
with some "Class.forName" and that is it.
It even allows to ship the same jar file and select implementation on the fly.

Vladimir

Christopher BROWN wrote:

> Then, merge all JARs into a single JAR.  Clients could then refer to the
> specific driver version they require in code, or use a generic Driver class that
> (in the constructor) detects the appropriate JDBC version and fixes a "final"
> int or Enum field, used thereafter in "switch" blocks to call the appropriate
> driver version, acting as a lightweight proxy when the specific driver version
> can't be referred to (for backwards compatibility).  More adventurous developers
 > ..........................................

Somehow as someone who manages a generic database access tool I don't like the
sounds of this requirement. Why as a client developer should I have to detect
the appropriate Java Version then somehow figure out the user's requirement
for the Driver class to call in your JDBC? I don't have to do this for any other
database so why for PostgreSQL's JDBC.

It may be of no concern really, but that is going to require me to change
the coding in my client for instantiating your Driver class, which is the
same for all databases so far, all so that you can change your build process,
which does not appear to be broken.

How about backup and state the one, two, three pros, and cons for initiating
the change in the build process again. Then highlight what additional work
would be required in the code, etc. to accomplish the new build process. Then
the list could input on the proposal. Maybe that has already taken place and
I missed it?

danap.

> ~
 > ~
 > ~
 > ~
 > ~
> Hope that helps ; hope it's not redundant with regards to messages sent since I
> started typing away my 2 cents...  In any case, I regularly use these techniques
> in production code with no accidents.
>
> --
> Christopher

Christopher BROWN wrote:
> The idea, for administrators and client developers, is that you wouldn't need to
> change anything, and you wouldn't need to pick a driver version.
>
> The idea being, that instead of their being different versions of the driver
> (for different levels of JDBC compatibility) is that there could be one single
> binary.  For type safety and to avoid creating source files from template
> pre-processing, the binary package could contain one driver implementation for
> each implemented JDBC version, each more recent version extending the previous
> version.  Managing this as a client would obviously be a mess, so my suggestion,
> if this approach seemed like an appropriate solution (I'm not pushing for it
> either, but I don't maintain the project...) is that "org.postgresql.Driver"
> could be reimplemented to delegate to the most recent driver implementation
> without changing client code.
>
> This could be done (in the driver code, not in client code) using an approach
> like this (hope the indentation doesn't disappear when sending the e-mail...):
>
 > ~
 > ~

Thank you for the clarification.
danap

danap wrote:
>     Somehow as someone who manages a generic database access tool I don't like the
>     sounds of this requirement. Why as a client developer should I have to detect
>     the appropriate Java Version then somehow figure out the user's requirement
>     for the Driver class to call in your JDBC? I don't have to do this for any other
>     database so why for PostgreSQL's JDBC.
>
>     It may be of no concern really, but that is going to require me to change
>     the coding in my client for instantiating your Driver class, which is the
>     same for all databases so far, all so that you can change your build process,
>     which does not appear to be broken.
>
>     How about backup and state the one, two, three pros, and cons for initiating
>     the change in the build process again. Then highlight what additional work
>     would be required in the code, etc. to accomplish the new build process. Then
>     the list could input on the proposal. Maybe that has already taken place and
>     I missed it?
>
>     danap.

>When I tried it in the past (for Jaybird) I ran into problems, it might
have been an IncompatibleClassChangeError (or subclass) on classloading
time, but I can't remember the exact details. I tried reading the JLS and
JVM spec on this point, but I find it hard to come to a conclusion, so I
will try a small experiment this weekend.

Let's base further discussions on your weekend's test results aka hard facts. :-)

>However that is half of your problem. The other half is that you risk
using classes, methods or maybe even features that are not available in a
lower Java version and that will not be detected compile time, but only at
runtime (and not just in newly added JDBC 4.2 methods).

This can most simply be detected by a "jre6 compliance" test drive with explicit JRE6 runtime lib in bootclasspath.

-Markus

Mark,

the chapters you describe list explicitly the exceptions that can happen, and the causes. None of them covers "a class
isreferenced that has a non-referenced method that has a non-existent class". 

Even with early resolution, the JLS does not cover the case we have, which is containing non-existent classes in
non-referencedmethods of referenced classes! 

The devil's in the details, right? ;-)

Hence we're still safe.

If that won't work, nobody could write a Java EE program and compile and test it agains the official javaee.jar, as
thatone not even contains ANY byte code but SOLELY declarations. And that one's an official JAR from the makers of
Java!

Regards
-Markus

-----Original Message-----
From: pgsql-jdbc-owner@postgresql.org [mailto:pgsql-jdbc-owner@postgresql.org] On Behalf Of Mark Rotteveel
Sent: Mittwoch, 17. Juni 2015 09:03
To: List
Subject: Re: [JDBC] Pre-processing during build

On Tue, 16 Jun 2015 21:26:12 +0200, "Markus KARG" <markus@headcrashing.eu>
wrote:
> So it boils down to verification and I doubt that the bytecode
> verifier will try to actually load java.sql.Type class. Have you
> really tried
this
> out or do you have another link where it is written that the byte code
> verifier will CHECK the existence of a parameter class when it
> verifies
the
> loaded class?

It looks like the JLS and JVM specification allow for resolution of symbolic references at classloading or "first use"
anda JVM implementation is free to choose: see 
http://docs.oracle.com/javase/specs/jls/se8/html/jls-12.html#jls-12.1.2 and
http://docs.oracle.com/javase/specs/jls/se8/html/jls-12.html#jls-12.3

Although I am not entirely sure about this, I interpret this that loading a class that has a method whose signature
includesa type that is not available (like SQLType on Java 7 or lower) could work on one JVM implementation (late
resolution),but not on others (early resolution). 

Mark



--
Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org) To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-jdbc

You got fooled by the fact that there is a difference between loading the very initial main class at JVM bootstrap, and
loadingANY OTHER subsequent class. This is difference is clearly documented in chapters JLS 12.1 (initial class) and
JLS12.2 (other classes). 

After correcting your example it works pretty well, see: https://gist.github.com/mkarg/88a89ae0dbffcfb7543e


-----Original Message-----
From: pgsql-jdbc-owner@postgresql.org [mailto:pgsql-jdbc-owner@postgresql.org] On Behalf Of Mark Rotteveel
Sent: Mittwoch, 17. Juni 2015 09:07
To: List
Subject: Re: [JDBC] Pre-processing during build

On Wed, 17 Jun 2015 00:02:40 +0300, Vladimir Sitnikov
<sitnikov.vladimir@gmail.com> wrote:
> 2015-06-16 22:30 GMT+03:00 Markus KARG <markus@headcrashing.eu>:
>> The answer is pretty simple: Try it out. :-)
>>
>> Just compile a JRE 8 class down to byte code level 6 and load it on
Java
>> level 7. That's what I proposed. Nothing else. It really bet will work
>> unless you try to INSTANTIATE JRE-only classes, but it should LOAD. And
>> nothing more we need.
>
> Markus, can you please be more explicit in your suggestion?
>
> I did try a simple "Hello, world" and it does not run in stock JDKs of
> MacOS: https://gist.github.com/vlsi/aeeb4a61d9c2b67ad213
> Even if you manage to make that fly, that would be built on sand.

Good example: it demonstrates at least that using reflection (eg
getDeclaredMethods; or in this case privateGetDeclaredMethods) will lead to
a NoClassDefFoundError.

Mark


--
Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-jdbc

The discussion has veered off topic somewhat with the discussion of build systems and their respective qualities.

My pull request is proposing to change build system from Ant to Maven. All existing functionality in the Ant build has been/will be replicated in the Maven build - including the pre-processing step. There are many advantages to using Maven as opposed to Ant. However, this is orthogonal to the discussion I'd like to have about the pre-processing used during the build process.

Focusing on the aim of the current pre-processing step - which is to select the appropriate spec implementation at compile time. The options seem quite complicated to me as a relative novice in the finer points of the language spec.

Taking a step back, could this not be achieved using annotations and a custom annotation processor to emit some code to select the correct implementation of the spec? This seems more Java-like and not a massive change from the existing code.

Sorry but you're wrong here.

Vladimir's example was invalid. See https://gist.github.com/mkarg/88a89ae0dbffcfb7543e instead.

Resolution will not fail. Even early static resolution won't. Check again the chapter of JLS about the critera to throw
thelisted exceptions. None of them is met with the corrected example
https://gist.github.com/mkarg/88a89ae0dbffcfb7543e.

The direct example will NOT fail on Java 7 as the method is not changed, but added. The original method is still in
place.Both methods exist at the same time and differ by signature, hence the linker of the old program finds the old
methodin the new class. No problem at all. 

The indirect example will NOT fail on Java 7 as a JRE 7 client will never pass in an instance of a JRE 8 type (how
shouldeven know of its existence?), and the Java 6 machine executing the invoked method will never INSTANTIATE that
typeso it will not fail. No problem at all. 

Still should work. You'd possibly like to set up a proof using the corrected example
https://gist.github.com/mkarg/88a89ae0dbffcfb7543e.:-) 

Regards
-Markus

-----Original Message-----
From: Mark Rotteveel [mailto:mark@lawinegevaar.nl]
Sent: Mittwoch, 17. Juni 2015 09:18
To: Markus KARG
Subject: RE: [JDBC] Pre-processing during build

On Tue, 16 Jun 2015 21:35:33 +0200, "Markus KARG" <markus@headcrashing.eu>
wrote:
> On Mon, 15 Jun 2015 23:59:45 +0200, "Markus KARG"
<markus@headcrashing.eu>
> wrote:
>> For Java 6 and 7 it would be possible to use the same codebase (if you
> ignore certain types iirc), but for Java 8 you need a separate library
due
> to the introduction of the SQLType interface. And if you fully support
JDBC
> 4.2, you also need the classes from java.time.
>
> Actually I doubt that truely. The idea is NOT to support JDBC42 on Java
6.
> The idea is that Java 6 loads our JAR and calls JDBC4 methods. The fact
> that the same JAR also contains Java 8 types like java.time or the new
> SQLType is invisible to the JRE 6 client application, as that one is
> written against JDBC4, hence does neither INSTANTIATE that types NOR
> invokes the methods using them as parameters!

Vladimir already demonstrated one problem, at least when using reflection
(which is not atypical with a lot of tools that use JDBC and with
connections pools). And I already pointed at the resolution phase as
described in the JLS that might happen at classloading time.

>> JDBC APIs themselves are not backwards compatible (ie: they introduce
new
> types in the signature, or require you to handle new types in existing
> methods), the API is only backwards compatible from the perspective of
the
> user.
>
> I think that's pretty enough. Nobody forces is to use Oracle's JARs. The
> driver just has to fulil the API towards the client application. That's
why
> it's called an API (not SPI). Can you please make an example what API is
> incompatible between JDBC4 and JDBC42?

An example of direct API incompatibility, use of a type new in Java 8
(SQLType):
ResultSet.updateObject(int columnIndex, Object x, SQLType targetSqlType)
http://docs.oracle.com/javase/8/docs/api/java/sql/ResultSet.html#updateObject-int-java.lang.Object-java.sql.SQLType-

An example of indirect API incompatibility: requirement to support
java.time types (added in Java 8) in get/set/updateObject, although that
can be worked around.

Mark

On Wed, 17 Jun 2015 13:52:52 +0200, Mark Rotteveel <mark@lawinegevaar.nl>
wrote:
> That is almost what PostgreSQL uses now, but this is not going to work
if
> you compile with Java 8 and assume that
> org.postgresql.jdbc4.PreparedStatement would then also be new-able when
the
> same jar is used under Java 6 or 7, because
> org.postgresql.jdbc4.PreparedStatement would need to be abstract at
compile
> time as it doesn't contain the methods required by the Java 8 (JDBC 4.2)
> API during compilation.
>
> So getting this to work would need some form of reflection (to get the
> right type at runtime based on the Java version), some preprocessing (as
> done currently) to get around the compilation problem or some byte code
> generation/modification to "unabstract"
> org.postgresql.jdbc4.PreparedStatement after compilation, or some form
of
> tiered compilation (where the org.postgresql.jdbc4.PreparedStatement is
> compiled with Java 7, and org.postgresql.jdbc42.PreparedStatement with
Java
> 8; this might be more complex than the existing solution.

>I just realized it might actually work: some (maybe all) methods added in
the JDBC API for Java 8 were added as default interface methods (with an
implementation that throws UnsupportedOperationException), so compilation
would succeed for org.postgresql.jdbc4.PreparedStatement without having an
implementation for the new methods.

Yay, seems you finally noticed we guys at the Java EE EGs are not so dumb as people might think! Honestly, what do you
thinkwhat we actually invented default methods for if not for this case...? ;-) 

>You'd still need reflection or an other trick to decide based on the Java
version which classes (Statement, PreparedStatement, ResultSet, etc) to
instantiate.

Only in case of one single driver name and not wanting to provide a property telling the JDBC version. Also not in case
ofseparate driver names like "Jdbc3Driver". In fact I do not see any difference in providing a list of separate
downloadscompared to a list of separate driver names. 

I think you not even need reflection, but could simply rely on proxies. Should be faster and do the job faster.
http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/reflect/Proxy.html:-) 

An official Java SE standard since v1.3, BTW, to prevent further discussions about versions. ;-)

-Markus


-----Original Message-----
From: pgsql-jdbc-owner@postgresql.org [mailto:pgsql-jdbc-owner@postgresql.org] On Behalf Of Vladimir Sitnikov
Sent: Mittwoch, 17. Juni 2015 14:04
To: Mark Rotteveel
Cc: List
Subject: Re: [JDBC] Pre-processing during build

>So getting this to work would need some form of reflection (to get the
right type at runtime based on the Java version)

Reflection is sufficient, isn't it?

I think reflection would be rather clean solution here.
It is more IDE-friendly and developer-friendly than pre-processing.
Reflection is much easier to debug than byte-code generation.

So, do you see drawbacks with using reflection to select the specific
implementation?

From implementation point of view it just replaces "preprocessing"
with some "Class.forName" and that is it.
It even allows to ship the same jar file and select implementation on the fly.

Vladimir


--
Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-jdbc

>This is difference is clearly documented in chapters JLS 12.1 (initial class) and JLS 12.2 (other classes).

There is no difference.

12.1.1 clearly reads: "This loading process is described further in §12.2."

Just in case, I've updated my gist:
https://gist.github.com/vlsi/aeeb4a61d9c2b67ad213, so you can see it
is reflection that makes the thing fail.

I thought it was obvious from the start: reflection has to represent
method arguments as Class somehow, so it fails as anyone would expect.

Vladimir

Stephen>However, this is orthogonal to the discussion I'd like to have
about the pre-processing used during the build process.

If I understand well, we all agree that pre-processing should be
dodged in the future.

I think it might be easier (from development and testing points of
view) to do the following:
1) remove source pre-processing in the current Ant build. That should
be rather trivial.
2) mavenize

That would eliminate "complex preprocessing" in the maven
configuration. Even if that processing is "easy" it would still take
time to review and test, and we would drop that pre-processing anyway.

Any objections?

PS. I could file a PR for removing pre-processing, however I am afraid
you'll kill me for that many PRs to review.
PPS. I've added my review comments to 3 of 4 non-mine PRs.

Vladimir

Wrong. 12.1.1 contains more restriction on the initial class than 12.2 which describes the general process for ANY
class.Hence it is a difference. That is what you see in the difference of you initial example to mine on Gist. :-) 

Sorry I was not aware that you only posted the example to demonstrate REFLECTION to fail. Maybe I missed that. I always
talkedabout normal instantiation, not about reflection. Also AFAIK JDBC spec does not say reflection MUST be possible,
doesit? 

-----Original Message-----
From: Vladimir Sitnikov [mailto:sitnikov.vladimir@gmail.com]
Sent: Mittwoch, 17. Juni 2015 23:15
To: Markus KARG
Cc: List
Subject: Re: [JDBC] Pre-processing during build

>This is difference is clearly documented in chapters JLS 12.1 (initial class) and JLS 12.2 (other classes).

There is no difference.

12.1.1 clearly reads: "This loading process is described further in §12.2."

Just in case, I've updated my gist:
https://gist.github.com/vlsi/aeeb4a61d9c2b67ad213, so you can see it is reflection that makes the thing fail.

I thought it was obvious from the start: reflection has to represent method arguments as Class somehow, so it fails as
anyonewould expect. 

Vladimir

Well, there IS a difference: We can pack our own copy of java.sql.Type with our Driver, but we should not dare to do
thatwith non-java.sql.* types. Hence, if that is the only package, we're safe. No problem. 

This is a typical pattern used by several Java EE specifications. For example, JPA. If the JRE finds the class on the
bootstrapclasspath, it uses it. If not, it's using the one on the application classpath. Finely documented by the JVM
specification,BTW, and performed daily by Java EE specs. 

-----Original Message-----
From: pgsql-jdbc-owner@postgresql.org [mailto:pgsql-jdbc-owner@postgresql.org] On Behalf Of Mark Rotteveel
Sent: Mittwoch, 17. Juni 2015 13:13
To: List
Subject: Re: [JDBC] Pre-processing during build

On Wed, 17 Jun 2015 06:15:47 -0400, Dave Cramer <pg@fastcrypt.com> wrote:
> On 17 June 2015 at 03:07, Mark Rotteveel <mark@lawinegevaar.nl> wrote:
>
>> On Wed, 17 Jun 2015 00:02:40 +0300, Vladimir Sitnikov
>> <sitnikov.vladimir@gmail.com> wrote:
>> > 2015-06-16 22:30 GMT+03:00 Markus KARG <markus@headcrashing.eu>:
>> >> The answer is pretty simple: Try it out. :-)
>> >>
>> >> Just compile a JRE 8 class down to byte code level 6 and load it on
>> Java
>> >> level 7. That's what I proposed. Nothing else. It really bet will
work
>> >> unless you try to INSTANTIATE JRE-only classes, but it should LOAD.
>> >> And
>> >> nothing more we need.
>> >
>> > Markus, can you please be more explicit in your suggestion?
>> >
>> > I did try a simple "Hello, world" and it does not run in stock JDKs
of
>> > MacOS: https://gist.github.com/vlsi/aeeb4a61d9c2b67ad213
>> > Even if you manage to make that fly, that would be built on sand.
>>
>> Good example: it demonstrates at least that using reflection (eg
>> getDeclaredMethods; or in this case privateGetDeclaredMethods) will
lead
>> to
>> a NoClassDefFoundError.
>>
>>
> I'm not sure this is a great example as Optional itself is a java 8
> construct.

Yes, and so is java.sql.SQLType. So if this doesn't work for Optional, it
also won't work for SQLType.

> Either way Spring is able to do this, as are others?

Spring uses a lot of reflection, proxies, byte code generation, etc to get
things done. I am not sure if you want to go that way.

Mark


--
Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-jdbc

> Also AFAIK JDBC spec does not say reflection MUST be possible, does it?

Does the spec say anything on reflection might not being available?

I can easily imagine application that uses reflection, cglib, etc to
call/manage/pool connections, statements.
Can you list a couple of libraries that forbid reflection usage?

I think it is uncommon in java when a class that implements some
public API fails to "getMethods".

> Wrong. 12.1.1 contains more restriction on the initial class

Can you please cite that "restriction"?

> That is what you see in the difference of you initial example to mine on Gist. :-)

False. It was obvious from the start (see stacktrace) that it was
java.lang.Class.privateGetDeclaredMethods that failed.
If the restriction exists indeed, another example is required to
highlight that. My examples are doomed to die of reflection.

Vladimir

On Wed, 17 Jun 2015 22:09:09 +0200, "Markus KARG" <markus@headcrashing.eu>
wrote:
> Mark,
>
> the chapters you describe list explicitly the exceptions that can
happen,
> and the causes. None of them covers "a class is referenced that has a
> non-referenced method that has a non-existent class".

I think you fail to see the meaning of "resolution of symbolic references"
and the implications of JLS 12.3. A referenced class (be it in a method
signature or body) in the java code, has a symbolic reference (ie: a String
with type information, for simplicity: the fully qualified name) in the
bytecode. A JVM implementation can choose to resolve those to actual
classes at classloading time, or when it really needs it (eg if the method
is called). Depending on the specific JVM implementation, this means that a
class with a method that has a type in its signature that is not available
at runtime might either fail at classloading time or at runtime when - as
demonstrated by Vladimir - you enumerate the methods of the class using
reflection.

> Even with early resolution, the JLS does not cover the case we have,
which
> is containing non-existent classes in non-referenced methods of
referenced
> classes!

Yes it does.

> The devil's in the details, right? ;-)
>
> Hence we're still safe.
>
> If that won't work, nobody could write a Java EE program and compile and
> test it agains the official javaee.jar, as that one not even contains
ANY
> byte code but SOLELY declarations. And that one's an official JAR from
the
> makers of Java!

Sorry, but that doesn't make any sense at all. The "official" JavaEE jar
does contain bytecode: interfaces and some supporting classes like
exceptions.

Mark

On Wed, 17 Jun 2015 22:47:23 +0200, "Markus KARG" <markus@headcrashing.eu>
wrote:
> Sorry but you're wrong here.
>
> Vladimir's example was invalid. See
> https://gist.github.com/mkarg/88a89ae0dbffcfb7543e instead.
>
> Resolution will not fail. Even early static resolution won't. Check
again
> the chapter of JLS about the critera to throw the listed exceptions.
None
> of them is met with the corrected example
> https://gist.github.com/mkarg/88a89ae0dbffcfb7543e.
>
> The direct example will NOT fail on Java 7 as the method is not changed,
> but added. The original method is still in place. Both methods exist at
the
> same time and differ by signature, hence the linker of the old program
> finds the old method in the new class. No problem at all.
>
> The indirect example will NOT fail on Java 7 as a JRE 7 client will
never
> pass in an instance of a JRE 8 type (how should even know of its
> existence?), and the Java 6 machine executing the invoked method will
never
> INSTANTIATE that type so it will not fail. No problem at all.
>
> Still should work. You'd possibly like to set up a proof using the
> corrected example https://gist.github.com/mkarg/88a89ae0dbffcfb7543e.
:-)

Sorry, but you're the one that is wrong, it is not only about actually
calling methods with types, it is about the presence or absence of those
types when the JVM does decide to resolve the symbolic reference (eg when
you reflect the declared methods). I am about done with this discussion. I
think the onus is on you to prove this scheme will work, not for us to
prove it won't work (which we already did). Your prove should not only
include simple direct instance access, but also when using reflection
**which is very common with JDBC drivers** (eg connection pools,
tools/libraries that bridge differences in JDBC implementations, etc).

It sounds like you want to trade minor complexity in the build/IDE process
for a world of hurt for the users of your driver. I don't think that is a
good way forward.

Mark

On Wed, 17 Jun 2015 22:59:18 +0200, "Markus KARG" <markus@headcrashing.eu>
wrote:
>>I just realized it might actually work: some (maybe all) methods added
in
> the JDBC API for Java 8 were added as default interface methods (with an
> implementation that throws UnsupportedOperationException), so
compilation
> would succeed for org.postgresql.jdbc4.PreparedStatement without having
an
> implementation for the new methods.
>
> Yay, seems you finally noticed we guys at the Java EE EGs are not so
dumb
> as people might think! Honestly, what do you think what we actually
> invented default methods for if not for this case...? ;-)

It still doesn't solve the problem if you actually have an implementation
of that method in your class, which was the starting point of the
discussion as I saw it.

>>You'd still need reflection or an other trick to decide based on the
Java
> version which classes (Statement, PreparedStatement, ResultSet, etc) to
> instantiate.
>
> Only in case of one single driver name and not wanting to provide a
> property telling the JDBC version. Also not in case of separate driver
> names like "Jdbc3Driver". In fact I do not see any difference in
providing
> a list of separate downloads compared to a list of separate driver
names.

It is a problem (although solvable), because JDBC 4 driver loading
requires you to provide a META-INF/services/java.sql.Driver file that
declares the driver(s) provided by the jar. This means that the driver it
declares must be loadable in all JVMs. This means that you either need to
provide a single driver that handles the differences, or you need to have
multiple drivers and ensure that the JDBC4 driver refuses to create
connections if the JDBC 4.1 or 4.2 driver is also loaded.

It will also make things harder for tools that want to load your driver
directly (instead of going through DriverManager).

Mark

On Wed, 17 Jun 2015 23:02:44 +0200, "Markus KARG" <markus@headcrashing.eu>
wrote:
> I think you not even need reflection, but could simply rely on proxies.
> Should be faster and do the job faster.
>
http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/reflect/Proxy.html
> :-)
>
> An official Java SE standard since v1.3, BTW, to prevent further
> discussions about versions. ;-)

Proxies use reflection (which is why they are part of the
java.lang.reflect package.

Mark

On Wed, 17 Jun 2015 23:51:26 +0200, "Markus KARG" <markus@headcrashing.eu>
wrote:
> Wrong. 12.1.1 contains more restriction on the initial class than 12.2
> which describes the general process for ANY class. Hence it is a
> difference. That is what you see in the difference of you initial
example
> to mine on Gist. :-)
>
> Sorry I was not aware that you only posted the example to demonstrate
> REFLECTION to fail. Maybe I missed that. I always talked about normal
> instantiation, not about reflection. Also AFAIK JDBC spec does not say
> reflection MUST be possible, does it?

Of course you need to support reflection, in some cases the JVM itself
will use reflection and as I said in an earlier email: a lot of tools and
libraries depend on it. Not supporting reflection will simply make the
driver unusable in a lot of situations.

I can't believe I actually need to argue this point.

Mark

Hi Mark,

>>
>> An official Java SE standard since v1.3, BTW, to prevent further
>> discussions about versions. ;-)
>
> Proxies use reflection (which is why they are part of the
> java.lang.reflect package.
>

Just because the class is in the package java.lang.reflect, it does
not mean it uses the reflection. Proxies are on the fly generated
classes and use generic byte code to invoke methods. No reflection is
involved. Just take a look at the class sun.misc.ProxyClassFactory.

Best regards,
Andrej Golovnin

On Thu, 18 Jun 2015 08:58:14 +0200, Andrej Golovnin
<andrej.golovnin@gmail.com> wrote:
> Hi Mark,
>
>>>
>>> An official Java SE standard since v1.3, BTW, to prevent further
>>> discussions about versions. ;-)
>>
>> Proxies use reflection (which is why they are part of the
>> java.lang.reflect package.
>>
>
> Just because the class is in the package java.lang.reflect, it does
> not mean it uses the reflection. Proxies are on the fly generated
> classes and use generic byte code to invoke methods. No reflection is
> involved. Just take a look at the class sun.misc.ProxyClassFactory.

Instead I looked at java.lang.reflect.Proxy.ProxyClassFactory and it
certainly does use reflection (although on interfaces), and even if the
proxy generation itself wouldn't use reflection, most InvocationHandler
implementations I have seen do use reflection one way or the other.

Mark

>I see nothing wrong with implementing Driver using java.reflect.Proxy,

+1.

>PGDriver6 / PGDriver7 / PGDriver8 which could be compiled in steps

We can just compile all the drivers using JDK8 and -source 1.6 -target
1.6, can't we?
Well, step-by-step might be more robust, however it would require
careful setup of development environment (developers would have to
install different JDK versions and that is a high entry bar).

>I've not yet encountered any JVM that attempts to fully-resolve all signatures as soon as it loads a class.  Does such
aJVM exist? 

JLS allows JVM to load classes at any point in time, however if JVM
decides to do early loading, it should not throw exceptions before
application code indeed tries to touch the "bad method".

http://www.excelsiorjet.com/ is a JVM that compiles java down to
native code. I guess they do a lot of resolutions at compile phase
(much more than OpenJDK does).

Here's the relevant JLS section:
https://docs.oracle.com/javase/specs/jls/se7/html/jls-12.html#jls-12.2.1
"... however, to reflect loading errors only at points in the program
where they could have arisen without prefetching or group loading"

Vladimir

> even with source/target set to 1.6, "javac" will complain that (for your pre-Java-8 drivers) you haven't implemented
everything

I see.
This indeed makes "step-by-step" compilation a good solution.

We might want to drop JDK6 support to make it easier for us and for
the contributors (1 JDK less to install).

Vladimir

>Removing JDK 1.6 is not being contemplated at the moment

When do you think it is safe to remove JDK 1.6 support?

Vladimir

If you ever had a look at the Java EE specs for example, they always EXPLICITLY list the REQUIRED the APIs (not the
NON-REQUIREDones). It's never the case that the specs are listing dependencies in a negative way, but always in a
positiveway. Just like Maven would do, too, and just like there is no "excludes" clause restricting "import java.*" in
theJava language. :-) 

Anyways, it seems to be common sense among us that reflection SHOULD work, so my solution is not the right way to go.

Regarding your questions and just for couriosity, even being irrelevant due to the reflection topic meanwhile: My
examplealso fails even without (!) reflection once you remove the inner class and simply have the Java 8 method
implementeddirectly by the main class. Hence it proofs the special restriction for main classes. That restriction is
thedifference between 12.1 (= how an initial class is loaded) and 12.2 (= how other classes are loaded). It is an
optionalimplementation choice of the Oracle JVM to do so: It demands that ALL references from main class to other
classesMUST have existing byte code. This demand does not exists for classes NOT directly references but loaded lazily
ata later time (following 12.2 rules ONLY but not 12.1 rules). Just change my example and you'll see the effect - it
failsimmediately as soon as there is no wrapping inner class anymore decoupling the missing classes from the main
class.

-----Original Message-----
From: Vladimir Sitnikov [mailto:sitnikov.vladimir@gmail.com]
Sent: Donnerstag, 18. Juni 2015 00:07
To: Markus KARG
Cc: List
Subject: Re: [JDBC] Pre-processing during build

> Also AFAIK JDBC spec does not say reflection MUST be possible, does it?

Does the spec say anything on reflection might not being available?

I can easily imagine application that uses reflection, cglib, etc to
call/manage/pool connections, statements.
Can you list a couple of libraries that forbid reflection usage?

I think it is uncommon in java when a class that implements some
public API fails to "getMethods".

> Wrong. 12.1.1 contains more restriction on the initial class

Can you please cite that "restriction"?

> That is what you see in the difference of you initial example to mine on Gist. :-)

False. It was obvious from the start (see stacktrace) that it was
java.lang.Class.privateGetDeclaredMethods that failed.
If the restriction exists indeed, another example is required to
highlight that. My examples are doomed to die of reflection.

Vladimir

> The difficult part about this is that we could put up a survey and ask "what version of JVM do you use" I suspect
that1.8 will be overwhelmingly popular 

Can we get some access logs from https://jdbc.postgresql.org/download.html page?
It can give us an insight on how often people download "old" drivers.

Vladimir

On 6/18/2015 11:59 AM, Vladimir Sitnikov wrote:
>> The difficult part about this is that we could put up a survey and ask "what version of JVM do you use" I suspect
that1.8 will be overwhelmingly popular 
> Can we get some access logs fromhttps://jdbc.postgresql.org/download.html  page?
> It can give us an insight on how often people download "old" drivers.

except for organizations like mine, where things like this were
downloaded a long time ago and bundled with internal releases.



--
john r pierce, recycling bits in santa cruz

> except for organizations like mine, where things like this were downloaded a long time ago and bundled with internal
releases.

I do understand, the numbers would not be perfect. However that would
be at least something.

By the way, can you share if you are still using pgjdbc + java6? =)

Vladimir

John> and that stuff NEVER gets updated until the equipment is replaced.

That is cool. However, that means you don't care if pgjdbc drops
updates for JDK6, do you?

My theory is as follows: if there is a nice "decay of download rate",
then we might have a bit more educated guess on "the number of
supported JDKs"

Dave> I provide professional services to companies that do

Ok, let's try supporting JDK6.


So, the next step is to start from
https://github.com/pgjdbc/pgjdbc/pull/322 and try to split it into
several maven submodules.

The interesting question is if we want to those submodules (jdbc4,
jdbc41, jdbc42) to be public (in other words, allow users depend on
them) or if we consider them a pure implementation detail.
In the latter case, we might want to have some fixed name of the
module that includes the latest driver.

I would prefer to have a single artifact in the "public API" that
includes all the jdbc versions.

Vladimir

> If that won't work, nobody could write a Java EE program and compile
> and test it agains the official javaee.jar, as that one not even
> contains
ANY
> byte code but SOLELY declarations. And that one's an official JAR from
the
> makers of Java!

>Sorry, but that doesn't make any sense at all. The "official" JavaEE jar does contain bytecode: interfaces and some
supportingclasses like exceptions. 

Wrong. There are lots of classes in that JAR that actually are "broken" from the view of the JLS. When you try to
instantiatethem you'll simply get a weird exception telling you that for some of them the byte code is missing. This is
donewith several APIs of Java EE as due to the way it is specified in the particular specifications, it is up to the
vendorto provide them. So you can compile against that explicitly bytecode-dropped classes without any problem, but you
canneither test nor otherwise instantiate them. 

Regards
-Markus

>Sorry, but you're the one that is wrong, it is not only about actually calling methods with types, it is about the
presenceor absence of those types when the JVM does decide to resolve the symbolic reference (eg when you reflect the
declaredmethods). I am about done with this discussion. I think the onus is on you to prove this scheme will work, not
forus to prove it won't work (which we already did). Your prove should not only include simple direct instance access,
butalso when using reflection **which is very common with JDBC drivers** (eg connection pools, tools/libraries that
bridgedifferences in JDBC implementations, etc). 

I agree that this thread is done, because I already provided a proof that my hypothesis works (the link was published
yesterday),and my hypothesis never said that reflection would work. Whether or not reflection is MANDATORY for JDBC
alsois a fruitless discussion, as all of you WANT it to be supported. 

>It sounds like you want to trade minor complexity in the build/IDE process for a world of hurt for the users of your
driver.I don't think that is a good way forward. 

I do not. You fear risks that do not exist if you go 100% with the JDBC specifications words, but I accept that you
liketo be safe from several uncertainties, so it is OK if we skip my idea and go with a different approach -- even when
Iam still convinced that it would be correct and working (but not for things you just WANT to support like reflection). 

-Markus

On 6/18/2015 12:33 PM, Vladimir Sitnikov wrote:
> John> and that stuff NEVER gets updated until the equipment is replaced.
>
> That is cool. However, that means you don't care if pgjdbc drops
> updates for JDK6, do you?

as long as a legacy version remains available, sure, I dunno why not.
the older stuff is /very/ stable anyways, most of the work is towards
supporting new features of the new versions, is it not?



--
john r pierce, recycling bits in santa cruz

>It will also make things harder for tools that want to load your driver directly (instead of going through
DriverManager).

JDBC specification says that an application has to go through DriverManager or DataSource. Going a third way is not
coveredby the specification, hence is not JDBC. I would say these tools are not JDBC compliant, but it would be up to
Lanceto judge, not me. 

In hikaricp they have a java6 version for 1.6 and 1.7 and the main version has for 1.8. This has worked really well for our osgi installations.

>Do we have any tools to summarize them ? Preferably command line

I usually use r + data.table + ggplot for that kind of analysis.

If you mean "summarize on postgresql.org server", then "extract jar
name somehow" | sort | uniq -c

Vladimir

>not insignificant number of /download/pgjdbc2.jar

Let's pretend they are either crawlers, or they are just not aware of
the fact that jbdc41 works for 8.1 as fine.
Vladimir

Look at this https://gist.github.com/mkarg/0da7f7dce8d9025511bb please.


-Markus

-----Original Message-----
From: pgsql-jdbc-owner@postgresql.org [mailto:pgsql-jdbc-owner@postgresql.org] On Behalf Of Mark Rotteveel
Sent: Donnerstag, 18. Juni 2015 08:38
To: List
Subject: Re: [JDBC] Pre-processing during build

On Wed, 17 Jun 2015 23:02:44 +0200, "Markus KARG" <markus@headcrashing.eu>
wrote:
> I think you not even need reflection, but could simply rely on proxies.
> Should be faster and do the job faster.
>
http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/reflect/Proxy.html
> :-)
>
> An official Java SE standard since v1.3, BTW, to prevent further
> discussions about versions. ;-)

Proxies use reflection (which is why they are part of the
java.lang.reflect package.

Mark


--
Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-jdbc

> Look at this https://gist.github.com/mkarg/0da7f7dce8d9025511bb please.

Markus,

Your example here does not apply for JDBC driver for the reason
highlighted by Christopher: <<"javac" will complain that (for your
pre-Java-8 drivers) you haven't implemented everything>>.

The problem is _both_ I7 and I8 should be the same interface (they
both should be `java.sql.PreparedStatement`).

We do want to have "PS7 implements java.sql.PreparedStatement" _and_
"PS8 implements java.sql.PreparedStatement".
Can you please show how are you going to cover that?

If we leave "PS7 implements java.sql.PreparedStatement", then javac
would blame us for "not implementing enough methods".

Are you suggesting to use Proxy for all the JDBC interfaces
(Statement, ResultSet, etc, etc)?
I'm afraid that would hit the performance wall.

Well, I thought we have already settled that `java.lang.reflect.Proxy`
is good enough for `Driver`.
However, I do not like the idea of using j.l.r for ResultSet.

Vladimir

>JDBC applications CANNOT migrate easily to JRE 7 at least! Do you have any statistics available? :-)

Dave has already shown the statistics.
The number of "jdbc4.jar" downloads is impressive.

We still have to support JRE7 anyway, so supporting JRE6 as well is
not that big deal. It is a bit painful for developers/contributors,
but if we support multiple JREs, then two or three does not make much
difference.

I suggest we just stop spending our time on that discussion.

We do not depend on other libraries that plan to drop JRE6 support, so
it looks like there is no hurry other than "simplification of a build
system a bit".

Vladimir

If you downloaded long time ago then you never will notice us dropping JDK6 support, obviously, as we solely talk about
futurefeature releases of PJDBC not about your "old and already downloaded" software. 

-----Original Message-----
From: pgsql-jdbc-owner@postgresql.org [mailto:pgsql-jdbc-owner@postgresql.org] On Behalf Of John R Pierce
Sent: Donnerstag, 18. Juni 2015 21:06
To: pgsql-jdbc@postgresql.org
Subject: Re: [JDBC] Pre-processing during build

On 6/18/2015 11:59 AM, Vladimir Sitnikov wrote:
>> The difficult part about this is that we could put up a survey and ask "what version of JVM do you use" I suspect
that1.8 will be overwhelmingly popular 
> Can we get some access logs fromhttps://jdbc.postgresql.org/download.html  page?
> It can give us an insight on how often people download "old" drivers.

except for organizations like mine, where things like this were
downloaded a long time ago and bundled with internal releases.



--
john r pierce, recycling bits in santa cruz



--
Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-jdbc

+1 ok for me

-----Original Message-----
From: pgsql-jdbc-owner@postgresql.org [mailto:pgsql-jdbc-owner@postgresql.org] On Behalf Of Vladimir Sitnikov
Sent: Donnerstag, 18. Juni 2015 23:42
To: Markus KARG
Cc: List
Subject: Re: [JDBC] Pre-processing during build

>JDBC applications CANNOT migrate easily to JRE 7 at least! Do you have any statistics available? :-)

Dave has already shown the statistics.
The number of "jdbc4.jar" downloads is impressive.

We still have to support JRE7 anyway, so supporting JRE6 as well is
not that big deal. It is a bit painful for developers/contributors,
but if we support multiple JREs, then two or three does not make much
difference.

I suggest we just stop spending our time on that discussion.

We do not depend on other libraries that plan to drop JRE6 support, so
it looks like there is no hurry other than "simplification of a build
system a bit".

Vladimir


--
Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-jdbc

On Thu, 18 Jun 2015 14:32:51 -0400, Dave Cramer <pg@fastcrypt.com> wrote:
> Vladimir,
>
> To be honest I don't really know. The difficult part about this is that
we
> could put up a survey and ask "what version of JVM do you use" I suspect
> that 1.8 will be overwhelmingly popular. The problem is the subset of
> people responding may not be representative of the population that is
using
> it. From my experience working with bigger companies they view JDBC as a
> simple tool. The metaphor that comes to mind is a car with aftermarket
> wheels. You can't get any more for the car because you spend 2000 on the
> wheels. Cars come with wheels; similarly PostgreSQL comes with a JDBC
> driver. It occurs to me that maven actually exacerbates this as you
don't
> even have to come to the site to get the driver now that you can just
add
> the dependency to your pom and it will be automagically downloaded for
you.

If you use Sonatype OSS repository to upload the binaries to Maven, then
you can query the Maven central statistics on the dashboard, see
http://blog.sonatype.com/2010/12/now-available-central-download-statistics-for-oss-projects/

Mark

On Thu, 18 Jun 2015 18:53:49 +0200, "Markus KARG" <markus@headcrashing.eu>
wrote:
> Regarding your questions and just for couriosity, even being irrelevant
> due to the reflection topic meanwhile: My example also fails even
without
> (!) reflection once you remove the inner class and simply have the Java
8
> method implemented directly by the main class. Hence it proofs the
special
> restriction for main classes. That restriction is the difference between
> 12.1 (= how an initial class is loaded) and 12.2 (= how other classes
are
> loaded). It is an optional implementation choice of the Oracle JVM to do
> so: It demands that ALL references from main class to other classes MUST
> have existing byte code. This demand does not exists for classes NOT
> directly references but loaded lazily at a later time (following 12.2
rules
> ONLY but not 12.1 rules). Just change my example and you'll see the
effect
> - it fails immediately as soon as there is no wrapping inner class
anymore
> decoupling the missing classes from the main class.

JLS 12.1 is defined in terms of the normal class loading and
initialization that is described in later sections. They even say "We now
outline the steps the Java Virtual Machine may take to execute Test, as an
example of the loading, linking, and initialization processes that are
described further in later sections.". The only difference I see is section
12.1.4: invoking the main method. If you see other difference I'd like you
to point them out.

If you look at the stacktrace when method is directly in the main class,
you'll notice that the JVM itself uses reflection to obtain and execute the
main method. And that use of reflection is what causes it to fail
immediately as opposed to the case with a separate class.

Mark

I have one more question: what if we use true "pre-processing" during
build? (e.g. https://github.com/raydac/java-comment-preprocessor)

The suggestion is as follows:
1) "AbstractStatement implements java.sql.Statement"
2) JDK8 is used for development
3) During build, AbstractStatement code is pre-processed as per
"current jdk" (e.g. if compiling under java6, then java7 and java8
would be skipped out)


The only problem was we could not spell literally "AbstractStatement
implements java.sql.Statement" as different JDK versions have
different number of methods. Using a pre-procesor solves that out.

It would eliminate current jdbc3, jdbc3g, jdbc4, ... stuff.
It will make sane class hierarchies possible as well.
Currently we cannot have AbstractPreparedStatement extends
AbstractStatement since we have multiple different
AbstractJdbc2Statement and AbstractJdbc3Statement.

From my point of view "jdbcXX" packages serve no good except allowing
multiple-jre support. I think having a single "AbstractStatement
implements java.sql.Statement" would be a huge win in terms of ease of
development.

Vladimir