Thread: Caching driver on pgFoundry?

Caching driver on pgFoundry?

From
Josh Berkus
Date:
JDBC team,

I may have missed some stuff over the holiday weekend, but why is the new
caching wrapper on pgfoundry instead of on jdbc.postgresql.org?

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

Re: Caching driver on pgFoundry?

From
"Heikki Linnakangas"
Date:
Josh Berkus wrote:
> I may have missed some stuff over the holiday weekend, but why is the new
> caching wrapper on pgfoundry instead of on jdbc.postgresql.org?

See the discussion here:
http://archives.postgresql.org/pgsql-jdbc/2007-07/msg00137.php
continuing in August here:
http://archives.postgresql.org/pgsql-jdbc/2007-08/msg00006.php

--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

Re: Caching driver on pgFoundry?

From
Josh Berkus
Date:
Heikki,

> See the discussion here:
> http://archives.postgresql.org/pgsql-jdbc/2007-07/msg00137.php
> continuing in August here:
> http://archives.postgresql.org/pgsql-jdbc/2007-08/msg00006.php

I understand why it's a wrapper.  I don't understand why the wrapper isn't at
jdbc.postgresql.org.  Putting it on pgfoundry, completely separate from all
the JDBC drivers, is pretty much a guarentee that nobody will ever download
it.

Or is it moving to jdbc.postgresql.org once it's tested?

Just so everyone is clear on why this is important & urgent ... we published a
benchmark[1] using the caching driver, which is the only published benchmark
PostgreSQL has.  This benchmark has generated a huge amount of interest in
PostgreSQL as an alternative to Oracle[2], and is very important to driving
the adoption of PostgreSQL *especially* amoung J2EE developers.  So it would
be nice to see the caching wrapper represented as "official" unless there's
something technically wrong with it.

[1]http://www.spec.org/jAppServer2004/results/res2007q3/jAppServer2004-20070703-00073.html
[2]http://www.informationweek.com/showArticle.jhtml?articleID=201001901

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

Re: Caching driver on pgFoundry?

From
"Heikki Linnakangas"
Date:
Josh Berkus wrote:
> I understand why it's a wrapper.  I don't understand why the wrapper isn't at
> jdbc.postgresql.org.  Putting it on pgfoundry, completely separate from all
> the JDBC drivers, is pretty much a guarentee that nobody will ever download
> it.

I don't have a problem with linking from jdbc.postgresql.org to the
wrapper. And other pooling and caching implementations as well while
you're at it.

> Just so everyone is clear on why this is important & urgent ... we published a
> benchmark[1] using the caching driver, which is the only published benchmark
> PostgreSQL has.  This benchmark has generated a huge amount of interest in
> PostgreSQL as an alternative to Oracle[2], and is very important to driving
> the adoption of PostgreSQL *especially* amoung J2EE developers.  So it would
> be nice to see the caching wrapper represented as "official" unless there's
> something technically wrong with it.

Can't you use DBCP or some other open source statement cache
implementation that's in a more mature state? Or actually, why don't you
have a statement cache in Sun Application Server like the competitors? ;-)

--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

Re: Caching driver on pgFoundry?

From
Josh Berkus
Date:
Heikki,

> I don't have a problem with linking from jdbc.postgresql.org to the
> wrapper. And other pooling and caching implementations as well while
> you're at it.

Only one I can think of is Sequoia.

> Can't you use DBCP or some other open source statement cache
> implementation that's in a more mature state?

Unfortunately, no.  The benchmark is already out.

> Or actually, why don't you
> have a statement cache in Sun Application Server like the competitors? ;-)

Don't even get me started.

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

Re: Caching driver on pgFoundry?

From
Kris Jurka
Date:

On Wed, 5 Sep 2007, Josh Berkus wrote:

>> Can't you use DBCP or some other open source statement cache
>> implementation that's in a more mature state?
>
> Unfortunately, no.  The benchmark is already out.
>

But that benchmark was run with a different caching implementation than
the wrapper version, so I'm not sure how that's relevant.  When Sun chose
to use unpublished and unreviewed code for the benchmark they got
themselves in a little bind and I'm not sure it's our job to bail them out
by publishing and advertising code that we're not confident in.  Heikki,
Oliver, and myself did not believe the code used by Sun in the benchmark
was correct in the general case so it was rejected for inclusion in the
core driver.  Dave/Lazlo have since started a new implemention on
pgfoundry, but that was never discussed with the JDBC list or submitted
for inclusion.

To satisfy the benchmark requirements, Sun should publish the code/driver
actually used in the benchmark somewhere on Sun's website and, if honest,
should recommend that people don't use it.  From there we should try to
gather some consensus on whether PG needs its own statement cache
implementation and then rerun the benchmarks with it or some other
implementation.

Kris Jurka

Re: Caching driver on pgFoundry?

From
Dave Cramer
Date:
On 5-Sep-07, at 12:21 PM, Heikki Linnakangas wrote:

> Josh Berkus wrote:
>> I understand why it's a wrapper.  I don't understand why the
>> wrapper isn't at
>> jdbc.postgresql.org.  Putting it on pgfoundry, completely separate
>> from all
>> the JDBC drivers, is pretty much a guarentee that nobody will ever
>> download
>> it.
>
The drivers will end up there eventually anyway.
And we can link it to the jdbc home page.

Dave
> I don't have a problem with linking from jdbc.postgresql.org to the
> wrapper. And other pooling and caching implementations as well while
> you're at it.
>
>> Just so everyone is clear on why this is important & urgent ... we
>> published a
>> benchmark[1] using the caching driver, which is the only published
>> benchmark
>> PostgreSQL has.  This benchmark has generated a huge amount of
>> interest in
>> PostgreSQL as an alternative to Oracle[2], and is very important
>> to driving
>> the adoption of PostgreSQL *especially* amoung J2EE developers.
>> So it would
>> be nice to see the caching wrapper represented as "official"
>> unless there's
>> something technically wrong with it.
>
> Can't you use DBCP or some other open source statement cache
> implementation that's in a more mature state? Or actually, why
> don't you
> have a statement cache in Sun Application Server like the
> competitors? ;-)
>
> --
>   Heikki Linnakangas
>   EnterpriseDB   http://www.enterprisedb.com
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
>                http://archives.postgresql.org


Re: Caching driver on pgFoundry?

From
Dave Cramer
Date:
On 5-Sep-07, at 1:27 PM, Kris Jurka wrote:

>
>
> On Wed, 5 Sep 2007, Josh Berkus wrote:
>
>>> Can't you use DBCP or some other open source statement cache
>>> implementation that's in a more mature state?
>>
>> Unfortunately, no.  The benchmark is already out.
>>
>
> But that benchmark was run with a different caching implementation
> than the wrapper version, so I'm not sure how that's relevant.
> When Sun chose to use unpublished and unreviewed code for the
> benchmark they got themselves in a little bind and I'm not sure
> it's our job to bail them out by publishing and advertising code
> that we're not confident in.  Heikki, Oliver, and myself did not
> believe the code used by Sun in the benchmark was correct in the
> general case so it was rejected for inclusion in the core driver.

So as I understand it the objection to the caching implementation is
that  statement caching belongs in the app server ?

If this is the case then I would argue that having caching in the
appserver is a great idea for everyone using an appserver it does not
help the rest of the world that doesn't use an app server.

Is there a technical argument here ?

Dave

> Dave/Lazlo have since started a new implemention on pgfoundry, but
> that was never discussed with the JDBC list or submitted for
> inclusion.
>
> To satisfy the benchmark requirements, Sun should publish the code/
> driver actually used in the benchmark somewhere on Sun's website
> and, if honest, should recommend that people don't use it.  From
> there we should try to gather some consensus on whether PG needs
> its own statement cache implementation and then rerun the
> benchmarks with it or some other implementation.
>
> Kris Jurka
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
>               http://archives.postgresql.org


Re: Caching driver on pgFoundry?

From
Kris Jurka
Date:

On Wed, 5 Sep 2007, Dave Cramer wrote:

>
> On 5-Sep-07, at 1:27 PM, Kris Jurka wrote:
>
>> But that benchmark was run with a different caching implementation than the
>> wrapper version, so I'm not sure how that's relevant.  When Sun chose to
>> use unpublished and unreviewed code for the benchmark they got themselves
>> in a little bind and I'm not sure it's our job to bail them out by
>> publishing and advertising code that we're not confident in.  Heikki,
>> Oliver, and myself did not believe the code used by Sun in the benchmark
>> was correct in the general case so it was rejected for inclusion in the
>> core driver.
>
> So as I understand it the objection to the caching implementation is that
> statement caching belongs in the app server ?

No, the objection above is that the code you originally submitted was not
good enough to be included in the driver regardless of whether it should
be in the driver or app server.  If we aren't using the code that you
originally submitted then the argument that we need to publish something
for benchmark compliance is silly because the requirement is you publish
the code/driver used, not a somewhat similar implementation.

> If this is the case then I would argue that having caching in the appserver
> is a great idea for everyone using an appserver it does not help the rest of
> the world that doesn't use an app server.

This is the consensus I'd like to see built before we spend a lot of time
on something.  Heikki doesn't believe we should do this at all as it is
covered by DBCP and app server pools.  You clearly believe we need this
and I'm sort of on the fence (I recall that's where Oliver is as well, but
I don't claim to speak for him).

> Is there a technical argument here ?
>

No, this is just a recap of the previous events from my perspective.

Kris Jurka

Re: Caching driver on pgFoundry?

From
Dave Cramer
Date:
On 5-Sep-07, at 2:12 PM, Kris Jurka wrote:

>
>
> On Wed, 5 Sep 2007, Dave Cramer wrote:
>
>>
>> On 5-Sep-07, at 1:27 PM, Kris Jurka wrote:
>>
>>> But that benchmark was run with a different caching
>>> implementation than the wrapper version, so I'm not sure how
>>> that's relevant.  When Sun chose to use unpublished and
>>> unreviewed code for the benchmark they got themselves in a little
>>> bind and I'm not sure it's our job to bail them out by publishing
>>> and advertising code that we're not confident in.  Heikki,
>>> Oliver, and myself did not believe the code used by Sun in the
>>> benchmark was correct in the general case so it was rejected for
>>> inclusion in the core driver.
>>
>> So as I understand it the objection to the caching implementation
>> is that statement caching belongs in the app server ?
>
> No, the objection above is that the code you originally submitted
> was not good enough to be included in the driver regardless of
> whether it should be in the driver or app server.  If we aren't
> using the code that you originally submitted then the argument that
> we need to publish something for benchmark compliance is silly
> because the requirement is you publish the code/driver used, not a
> somewhat similar implementation.

As I recall the events. The only objection to code that I submitted
for inclusion was Heikki's objection as  to where caching belonged.
The proof of concept code was clearly not production and was not
meant for inclusion it was meant to generate this argument. As nobody
argued vehemently at the time I went ahead with a production version.
>
>> If this is the case then I would argue that having caching in the
>> appserver is a great idea for everyone using an appserver it does
>> not help the rest of the world that doesn't use an app server.
>
> This is the consensus I'd like to see built before we spend a lot
> of time on something.  Heikki doesn't believe we should do this at
> all as it is covered by DBCP and app server pools.
Well DBCP has it's issues. It is very large and not the fastest piece
of code in the world.

> You clearly believe we need this and I'm sort of on the fence (I
> recall that's where Oliver is as well, but I don't claim to speak
> for him).



So we basically have one nay vote holding this up ?

Dave
>
>> Is there a technical argument here ?
>>
>
> No, this is just a recap of the previous events from my perspective.
>
> Kris Jurka


Re: Caching driver on pgFoundry?

From
Kris Jurka
Date:

On Wed, 5 Sep 2007, Dave Cramer wrote:

> On 5-Sep-07, at 2:12 PM, Kris Jurka wrote:
>
>> You clearly believe we need this and I'm sort of on the fence (I recall
>> that's where Oliver is as well, but I don't claim to speak for him).
>
> So we basically have one nay vote holding this up ?
>

But we only have one yes vote pushing it forward.  At least of the four of
us, I haven't gone back to the archives to see if anyone else weighed in
on the discussion.

Kris Jurka

Re: Caching driver on pgFoundry?

From
Dave Cramer
Date:
On 5-Sep-07, at 2:25 PM, Kris Jurka wrote:

>
>
> On Wed, 5 Sep 2007, Dave Cramer wrote:
>
>> On 5-Sep-07, at 2:12 PM, Kris Jurka wrote:
>>
>>> You clearly believe we need this and I'm sort of on the fence (I
>>> recall that's where Oliver is as well, but I don't claim to speak
>>> for him).
>>
>> So we basically have one nay vote holding this up ?
>>
>
> But we only have one yes vote pushing it forward.  At least of the
> four of us, I haven't gone back to the archives to see if anyone
> else weighed in on the discussion.
>
I think you can consider Josh Berkus, Laszlo, Paul Vandenbogaard to
have  voted in favour of it.

> Kris Jurka


Re: Caching driver on pgFoundry?

From
Paul van den Bogaard
Date:
posted my ideas on august 9. seen no reaction. I think that's a yes
vote. Not sure if it counts though.
To restate:

apps servers have statement caching build in because at the time it
was created there were (almost) no drivers out there implementing it.
Guess Suns driver is one of the few that does not have an
implementation.

 From an architectural point of view I feel a statement is part of a
connection. Therefore it belongs in a connections. However I do admit
there are many ways to implement this view.

I know that adding an extra jar file will reduce its acceptance in
the ISV community. Been working with different ISVs for almost 10
years now and every element they include in their suite will in
general be included in their Q&A tests. One more element of the suite
they support means many more tests. It makes it more difficult (this
to some extent might be perception, but it is something that does
matter to them) to do it so they will be more reluctant to support it.

The wrapper is an extra layer that needs to be crossed while
"creating" a statement from the pool. However I doubt that it will
make a significant (negative) impact when compared to the all-in-one
approach. The real net effect of using statement caching is the
reduction of work on the database side.

Finally it is not just appsservers out there. Lots if J2SE
application exist in at least the Telco segment. Would love to see
these guys to adopt a free database instead of Oracle that is all
over that place.


--Paul


On 5-sep-2007, at 20:25, Kris Jurka wrote:

>
>
> On Wed, 5 Sep 2007, Dave Cramer wrote:
>
>> On 5-Sep-07, at 2:12 PM, Kris Jurka wrote:
>>
>>> You clearly believe we need this and I'm sort of on the fence (I
>>> recall that's where Oliver is as well, but I don't claim to speak
>>> for him).
>>
>> So we basically have one nay vote holding this up ?
>>
>
> But we only have one yes vote pushing it forward.  At least of the
> four of us, I haven't gone back to the archives to see if anyone
> else weighed in on the discussion.
>
> Kris Jurka
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 9: In versions below 8.0, the planner will ignore your desire to
>       choose an index scan if your joining column's datatypes do not
>       match

------------------------------------------------------------------------
---------------------
Paul van den Bogaard
Paul.vandenBogaard@sun.com
CIE -- Collaboration and ISV Engineering, Opensource Engineering group

Sun Microsystems, Inc                              phone:        +31
334 515 918
Saturnus 1
extentsion: x (70)15918
3824 ME Amersfoort                                 mobile:       +31
651 913 354
The Netherlands
fax:            +31 334 515 001


Re: Caching driver on pgFoundry?

From
Josh Berkus
Date:
Kris,

> But that benchmark was run with a different caching implementation than
> the wrapper version, so I'm not sure how that's relevant.  When Sun
> chose to use unpublished and unreviewed code for the benchmark they got
> themselves in a little bind and I'm not sure it's our job to bail them
> out by publishing and advertising code that we're not confident in.

This isn't for *Sun*, it's for PostgreSQL.  Do you have any idea how many
times a month I field the question "Is there a benchmark for PostgreSQL?"

--
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

Re: Caching driver on pgFoundry?

From
Josh Berkus
Date:
Kris,

> But we only have one yes vote pushing it forward.  At least of the four
> of us, I haven't gone back to the archives to see if anyone else weighed
> in on the discussion.

Assuming I have a vote, I vote to push it forward *unless* there is a
technical objection to code quality, for the following reasons:

1) The benchmark publication strongly benefits the whole PostgreSQL project
and will hopefully lead to expanding our JDBC contributor pool;

2) Since most other DBMSes (including DB2, MySQL and Oracle) have caching
in their drivers this makes PostgreSQL more competitive for Java projects;

3) Not all Java users are using appservers, so it's a useful feature for
those still using JSP and other non-appserver implementations.

--
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

Re: Caching driver on pgFoundry?

From
"Heikki Linnakangas"
Date:
Kris Jurka wrote:
> On Wed, 5 Sep 2007, Dave Cramer wrote:
>> If this is the case then I would argue that having caching in the
>> appserver is a great idea for everyone using an appserver it does not
>> help the rest of the world that doesn't use an app server.
>
> This is the consensus I'd like to see built before we spend a lot of
> time on something.  Heikki doesn't believe we should do this at all as
> it is covered by DBCP and app server pools.

To be clear, I think it's nice to have one more statement cache
implementation around, with friendly BSD license. I encourage you to
keep working on it, regardless of how this discussion ends.

But let's not bundle it with the PostgreSQL JDBC driver, it's clearly a
separate component. For applications that already have another
connection pool / statement cache implementation, it would be just extra
baggage. And as a separate project, you can use it with other DBMSs as
well. Or bundle it with an app server :).

Another point of view is to think about the release cycles of the JDBC
driver and the caching wrapper. If a bug is found in the wrapper, and
it's bundled with the JDBC driver, we would have to release a new
version of the bundle, forcing an upgrade of both components even if you
only use one. If we add a new feature to the wrapper, is that going to
be backpatched to say the 8.1 branch? What if you're running 8.1 server
and driver, and you want to take advantage of a new wrapper feature?

The wrapper has enough merit to live a happy and successful life as a
project of its own. I don't care if it's a separate pgfoundry project,
or just a separate CVS directory within jdbc project and separate
builds; the main point is that it should be a separate jar file with a
separate release cycle. If we add a link to the latest version of the
wrapper jar from jdbc.postgresql.org download page, people will find it
regardless of where the development happens.

--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

Re: Caching driver on pgFoundry?

From
Oliver Jowett
Date:
Dave Cramer wrote:

> As I recall the events. The only objection to code that I submitted for
> inclusion was Heikki's objection as  to where caching belonged.

I objected to not having a wrapper at all and using the driver's
statements directly. That produces semantically incorrect behaviour,
that's unacceptable for the main driver.

If you're fixing that great but it's quite a different beast to the
original implementation and I haven't seen a new version of it yet.

The "is it part of the driver or not" question is more of a maintenance
issue than anything. If it is packaged separately it can be maintained
more easily by whoever cares about it (you, Josh, whoever). If it's part
of the driver you're going to run into the barrier that whenever it
needs an update you will need to build consensus amongst the JDBC
maintainers, which so far you don't have a very good record of doing.

I still have not seen an argument as to why it needs to be part of the
driver at all, and "not part of the driver" certainly seems simpler and
more flexible to me. Did you finish those benchmarks of in-driver versus
out-of-driver implementations yet?

-O

Re: Caching driver on pgFoundry?

From
Oliver Jowett
Date:
Kris Jurka wrote:

> This is the consensus I'd like to see built before we spend a lot of
> time on something.  Heikki doesn't believe we should do this at all as
> it is covered by DBCP and app server pools.  You clearly believe we need
> this and I'm sort of on the fence (I recall that's where Oliver is as
> well, but I don't claim to speak for him).

I think it belongs outside the driver, but I could be convinced
otherwise if Dave etc. presents a good technical argument as to why it
is easier to implement within the driver. Haven't seen that yet.

-O

Re: Caching driver on pgFoundry?

From
Till Toenges
Date:
Oliver Jowett wrote:
> The "is it part of the driver or not" question is more of a
> maintenance issue than anything. If it is packaged separately it can
> be maintained more easily by whoever cares about it (you, Josh,
> whoever). If it's part of the driver you're going to run into the
> barrier that whenever it needs an update you will need to build
> consensus amongst the JDBC maintainers, which so far you don't have a
> very good record of doing.

How about making it a separate package, but offering a combined download
or jar with both packages on the drivers download page. And a README
including instructions on how to use / activate caching.

Like:

postgresql-8.2.506.jdbc4.jar

and

caching-123.jar

combined as:

postgresql-8.2.506.jdbc4.caching-123.jar


The only consensus needed would be who has to build and upload the
combined jar when one of the packages gets updated.

That would combine the benefits of having it obviously available for
anyone who downloads the driver (unlike a completly separate project on
an otherwise unrelated site), and keeping it separated for easy maintenance.


Till


Re: Caching driver on pgFoundry?

From
Josh Berkus
Date:
> How about making it a separate package, but offering a combined download
> or jar with both packages on the drivers download page. And a README
> including instructions on how to use / activate caching.

Hey, if we can get the same performance, I'm all for this.  Thanks for
suggestion!

--
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

Re: Caching driver on pgFoundry?

From
"Peter Kovacs"
Date:
> If this is the case then I would argue that having caching in the
> appserver is a great idea for everyone using an appserver it does not
> help the rest of the world that doesn't use an app server.

We have a product that doesn't use app server, still we need a cached
connection pool. In this case this is an Oracle backend, so we started
using the cached pool implementation found in the Oracle jdbc driver
-- assuming that Oracle will do caching/pooling best for its
Connections. The fact is that we've been having some issues with the
Oracle cached pool implementation, so I started to extensively test
the Apache DBCP and the issues seems to have gone away.

Apart from parabolic example above, I think that from a "global"
perspective, it would be much more reasonable to help the Apache
people solve any issue which someone using the Commons DBCP with
PostgreSQL might have than doing your own little house work excercise
of implementing yet-another-ccp.

Thanks
Peter

On 9/5/07, Dave Cramer <pg@fastcrypt.com> wrote:
>
> On 5-Sep-07, at 1:27 PM, Kris Jurka wrote:
>
> >
> >
> > On Wed, 5 Sep 2007, Josh Berkus wrote:
> >
> >>> Can't you use DBCP or some other open source statement cache
> >>> implementation that's in a more mature state?
> >>
> >> Unfortunately, no.  The benchmark is already out.
> >>
> >
> > But that benchmark was run with a different caching implementation
> > than the wrapper version, so I'm not sure how that's relevant.
> > When Sun chose to use unpublished and unreviewed code for the
> > benchmark they got themselves in a little bind and I'm not sure
> > it's our job to bail them out by publishing and advertising code
> > that we're not confident in.  Heikki, Oliver, and myself did not
> > believe the code used by Sun in the benchmark was correct in the
> > general case so it was rejected for inclusion in the core driver.
>
> So as I understand it the objection to the caching implementation is
> that  statement caching belongs in the app server ?
>
> If this is the case then I would argue that having caching in the
> appserver is a great idea for everyone using an appserver it does not
> help the rest of the world that doesn't use an app server.
>
> Is there a technical argument here ?
>
> Dave
>
> > Dave/Lazlo have since started a new implemention on pgfoundry, but
> > that was never discussed with the JDBC list or submitted for
> > inclusion.
> >
> > To satisfy the benchmark requirements, Sun should publish the code/
> > driver actually used in the benchmark somewhere on Sun's website
> > and, if honest, should recommend that people don't use it.  From
> > there we should try to gather some consensus on whether PG needs
> > its own statement cache implementation and then rerun the
> > benchmarks with it or some other implementation.
> >
> > Kris Jurka
> >
> > ---------------------------(end of
> > broadcast)---------------------------
> > TIP 4: Have you searched our list archives?
> >
> >               http://archives.postgresql.org
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>

Re: Caching driver on pgFoundry?

From
"Peter Kovacs"
Date:
> Been working with different ISVs for almost 10
> years now and every element they include in their suite will in
> general be included in their Q&A tests. One more element of the suite
> they support means many more tests. It makes it more difficult (this
> to some extent might be perception, but it is something that does
> matter to them) to do it so they will be more reluctant to support it.

ISV will not extra-test the driver(s) which comes with the database,
will they? (I mean, they don't test driver's separately by default.)
If you, for example, bundle the Commons DBCP with the JDBC driver,
this argument becomes pretty much empty.

Thanks
Peter

On 9/5/07, Paul van den Bogaard <Paul.Vandenbogaard@sun.com> wrote:
> posted my ideas on august 9. seen no reaction. I think that's a yes
> vote. Not sure if it counts though.
> To restate:
>
> apps servers have statement caching build in because at the time it
> was created there were (almost) no drivers out there implementing it.
> Guess Suns driver is one of the few that does not have an
> implementation.
>
>  From an architectural point of view I feel a statement is part of a
> connection. Therefore it belongs in a connections. However I do admit
> there are many ways to implement this view.
>
> I know that adding an extra jar file will reduce its acceptance in
> the ISV community. Been working with different ISVs for almost 10
> years now and every element they include in their suite will in
> general be included in their Q&A tests. One more element of the suite
> they support means many more tests. It makes it more difficult (this
> to some extent might be perception, but it is something that does
> matter to them) to do it so they will be more reluctant to support it.
>
> The wrapper is an extra layer that needs to be crossed while
> "creating" a statement from the pool. However I doubt that it will
> make a significant (negative) impact when compared to the all-in-one
> approach. The real net effect of using statement caching is the
> reduction of work on the database side.
>
> Finally it is not just appsservers out there. Lots if J2SE
> application exist in at least the Telco segment. Would love to see
> these guys to adopt a free database instead of Oracle that is all
> over that place.
>
>
> --Paul
>
>
> On 5-sep-2007, at 20:25, Kris Jurka wrote:
>
> >
> >
> > On Wed, 5 Sep 2007, Dave Cramer wrote:
> >
> >> On 5-Sep-07, at 2:12 PM, Kris Jurka wrote:
> >>
> >>> You clearly believe we need this and I'm sort of on the fence (I
> >>> recall that's where Oliver is as well, but I don't claim to speak
> >>> for him).
> >>
> >> So we basically have one nay vote holding this up ?
> >>
> >
> > But we only have one yes vote pushing it forward.  At least of the
> > four of us, I haven't gone back to the archives to see if anyone
> > else weighed in on the discussion.
> >
> > Kris Jurka
> >
> > ---------------------------(end of
> > broadcast)---------------------------
> > TIP 9: In versions below 8.0, the planner will ignore your desire to
> >       choose an index scan if your joining column's datatypes do not
> >       match
>
> ------------------------------------------------------------------------
> ---------------------
> Paul van den Bogaard
> Paul.vandenBogaard@sun.com
> CIE -- Collaboration and ISV Engineering, Opensource Engineering group
>
> Sun Microsystems, Inc                              phone:        +31
> 334 515 918
> Saturnus 1
> extentsion: x (70)15918
> 3824 ME Amersfoort                                 mobile:       +31
> 651 913 354
> The Netherlands
> fax:            +31 334 515 001
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 9: In versions below 8.0, the planner will ignore your desire to
>        choose an index scan if your joining column's datatypes do not
>        match
>

Fwd: Caching driver on pgFoundry?

From
"Peter Kovacs"
Date:
The mail below didn't make onto the list. Reason: I sent with the
wrong FROM address. Why do I have several e-mail addresses? Because
the e-mail address "peter.kovacs.1.0rc@gmail.com" was not
accepted/handled/seen/whatever by the postgres mail list server.
(Sorry for my having a troubled life.)

---------- Forwarded message ----------
From: Peter Kovacs <peter.kovacs.1.0rc@gmail.com>
Date: Sep 6, 2007 9:38 AM
Subject: Re: [JDBC] Caching driver on pgFoundry?
To: Dave Cramer <pg@fastcrypt.com>
Cc: Kris Jurka <books@ejurka.com>, Josh Berkus <josh@agliodbs.com>,
Heikki Linnakangas <heikki@enterprisedb.com>,
pgsql-jdbc@postgresql.org


> If this is the case then I would argue that having caching in the
> appserver is a great idea for everyone using an appserver it does not
> help the rest of the world that doesn't use an app server.

We have a product that doesn't use app server, still we need a cached
connection pool. In this case this is an Oracle backend, so we started
using the cached pool implementation found in the Oracle jdbc driver
-- assuming that Oracle will do caching/pooling best for its
Connections. The fact is that we've been having some issues with the
Oracle cached pool implementation, so I started to extensively test
the Apache DBCP and the issues seems to have gone away.

Apart from parabolic example above, I think that from a "global"
perspective, it would be much more reasonable to help the Apache
people solve any issue which someone using the Commons DBCP with
PostgreSQL might have than doing your own little house work excercise
of implementing yet-another-ccp.

Thanks
Peter

On 9/5/07, Dave Cramer <pg@fastcrypt.com> wrote:
>
> On 5-Sep-07, at 1:27 PM, Kris Jurka wrote:
>
> >
> >
> > On Wed, 5 Sep 2007, Josh Berkus wrote:
> >
> >>> Can't you use DBCP or some other open source statement cache
> >>> implementation that's in a more mature state?
> >>
> >> Unfortunately, no.  The benchmark is already out.
> >>
> >
> > But that benchmark was run with a different caching implementation
> > than the wrapper version, so I'm not sure how that's relevant.
> > When Sun chose to use unpublished and unreviewed code for the
> > benchmark they got themselves in a little bind and I'm not sure
> > it's our job to bail them out by publishing and advertising code
> > that we're not confident in.  Heikki, Oliver, and myself did not
> > believe the code used by Sun in the benchmark was correct in the
> > general case so it was rejected for inclusion in the core driver.
>
> So as I understand it the objection to the caching implementation is
> that  statement caching belongs in the app server ?
>
> If this is the case then I would argue that having caching in the
> appserver is a great idea for everyone using an appserver it does not
> help the rest of the world that doesn't use an app server.
>
> Is there a technical argument here ?
>
> Dave
>
> > Dave/Lazlo have since started a new implemention on pgfoundry, but
> > that was never discussed with the JDBC list or submitted for
> > inclusion.
> >
> > To satisfy the benchmark requirements, Sun should publish the code/
> > driver actually used in the benchmark somewhere on Sun's website
> > and, if honest, should recommend that people don't use it.  From
> > there we should try to gather some consensus on whether PG needs
> > its own statement cache implementation and then rerun the
> > benchmarks with it or some other implementation.
> >
> > Kris Jurka
> >
> > ---------------------------(end of
> > broadcast)---------------------------
> > TIP 4: Have you searched our list archives?
> >
> >               http://archives.postgresql.org
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>

Fwd: Caching driver on pgFoundry?

From
"Peter Kovacs"
Date:
Due to painful remnants of my troubled childhood, the message below
haven't made it to the list.

---------- Forwarded message ----------
From: Peter Kovacs <peter.kovacs.1.0rc@gmail.com>
Date: Sep 6, 2007 9:51 AM
Subject: Re: [JDBC] Caching driver on pgFoundry?
To: Paul van den Bogaard <Paul.Vandenbogaard@sun.com>
Cc: Kris Jurka <books@ejurka.com>, Dave Cramer <pg@fastcrypt.com>,
Josh Berkus <josh@agliodbs.com>, Heikki Linnakangas
<heikki@enterprisedb.com>, pgsql-jdbc@postgresql.org


> Been working with different ISVs for almost 10
> years now and every element they include in their suite will in
> general be included in their Q&A tests. One more element of the suite
> they support means many more tests. It makes it more difficult (this
> to some extent might be perception, but it is something that does
> matter to them) to do it so they will be more reluctant to support it.

ISV will not extra-test the driver(s) which comes with the database,
will they? (I mean, they don't test driver's separately by default.)
If you, for example, bundle the Commons DBCP with the JDBC driver,
this argument becomes pretty much empty.

Thanks
Peter

On 9/5/07, Paul van den Bogaard <Paul.Vandenbogaard@sun.com> wrote:
> posted my ideas on august 9. seen no reaction. I think that's a yes
> vote. Not sure if it counts though.
> To restate:
>
> apps servers have statement caching build in because at the time it
> was created there were (almost) no drivers out there implementing it.
> Guess Suns driver is one of the few that does not have an
> implementation.
>
>  From an architectural point of view I feel a statement is part of a
> connection. Therefore it belongs in a connections. However I do admit
> there are many ways to implement this view.
>
> I know that adding an extra jar file will reduce its acceptance in
> the ISV community. Been working with different ISVs for almost 10
> years now and every element they include in their suite will in
> general be included in their Q&A tests. One more element of the suite
> they support means many more tests. It makes it more difficult (this
> to some extent might be perception, but it is something that does
> matter to them) to do it so they will be more reluctant to support it.
>
> The wrapper is an extra layer that needs to be crossed while
> "creating" a statement from the pool. However I doubt that it will
> make a significant (negative) impact when compared to the all-in-one
> approach. The real net effect of using statement caching is the
> reduction of work on the database side.
>
> Finally it is not just appsservers out there. Lots if J2SE
> application exist in at least the Telco segment. Would love to see
> these guys to adopt a free database instead of Oracle that is all
> over that place.
>
>
> --Paul
>
>
> On 5-sep-2007, at 20:25, Kris Jurka wrote:
>
> >
> >
> > On Wed, 5 Sep 2007, Dave Cramer wrote:
> >
> >> On 5-Sep-07, at 2:12 PM, Kris Jurka wrote:
> >>
> >>> You clearly believe we need this and I'm sort of on the fence (I
> >>> recall that's where Oliver is as well, but I don't claim to speak
> >>> for him).
> >>
> >> So we basically have one nay vote holding this up ?
> >>
> >
> > But we only have one yes vote pushing it forward.  At least of the
> > four of us, I haven't gone back to the archives to see if anyone
> > else weighed in on the discussion.
> >
> > Kris Jurka
> >
> > ---------------------------(end of
> > broadcast)---------------------------
> > TIP 9: In versions below 8.0, the planner will ignore your desire to
> >       choose an index scan if your joining column's datatypes do not
> >       match
>
> ------------------------------------------------------------------------
> ---------------------
> Paul van den Bogaard
> Paul.vandenBogaard@sun.com
> CIE -- Collaboration and ISV Engineering, Opensource Engineering group
>
> Sun Microsystems, Inc                              phone:        +31
> 334 515 918
> Saturnus 1
> extentsion: x (70)15918
> 3824 ME Amersfoort                                 mobile:       +31
> 651 913 354
> The Netherlands
> fax:            +31 334 515 001
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 9: In versions below 8.0, the planner will ignore your desire to
>        choose an index scan if your joining column's datatypes do not
>        match
>

Re: Caching driver on pgFoundry?

From
Oliver Jowett
Date:
Paul van den Bogaard wrote:

> Finally it is not just appsservers out there. Lots if J2SE application
> exist in at least the Telco segment. Would love to see these guys to
> adopt a free database instead of Oracle that is all over that place.

Going off at a bit of a tangent here.. OpenCloud does telco
infrastructure software and we package postgresql with our product to do
persistence of long-lived state (deployed services, cluster
configuration, customer provisioning data etc). The statement caching
stuff is not really an issue either way for what we do, since we already
intelligently reuse statements in our persistance layer.

One thing we've found is that our customers don't really care about
database setup for this sort of use, they want it to "just work". In
fact we package Derby instead of PostgreSQL with our development
versions just so there's one less thing for the developer to set up. So
even if we cared about statement caching it wouldn't really matter if it
was together with the driver or a separate package as we'd still end up
doing all the packaging together with the application and testing of the
combined product.

While what we do is essentially an appserver (JAIN SLEE) I suspect the
same applies to standalone java telco apps, you're going to be doing DB
integration as part of your app not as something that gets plugged in
afterwards, so I'm not sure how relevant having it in the standard
driver is.

-O

Re: Caching driver on pgFoundry?

From
Paul van den Bogaard
Date:
if the caching & pooling stuff is in the same jar file as the rest of
the JDBC driver than my point is empty. If it is (can be seen as) an
extra element that it is very likely it will mean extra work for the
Q&A department of the ISV.
Test suite is likely to become

on platform A, test with and test without
on platform B, test with and test without

continues till all platforms are covered.

In general one more element translates in more permutations.

--Paul

On 6-sep-2007, at 9:51, Peter Kovacs wrote:

>> Been working with different ISVs for almost 10
>> years now and every element they include in their suite will in
>> general be included in their Q&A tests. One more element of the suite
>> they support means many more tests. It makes it more difficult (this
>> to some extent might be perception, but it is something that does
>> matter to them) to do it so they will be more reluctant to support
>> it.
>
> ISV will not extra-test the driver(s) which comes with the database,
> will they? (I mean, they don't test driver's separately by default.)
> If you, for example, bundle the Commons DBCP with the JDBC driver,
> this argument becomes pretty much empty.
>
> Thanks
> Peter
>
> On 9/5/07, Paul van den Bogaard <Paul.Vandenbogaard@sun.com> wrote:
>> posted my ideas on august 9. seen no reaction. I think that's a yes
>> vote. Not sure if it counts though.
>> To restate:
>>
>> apps servers have statement caching build in because at the time it
>> was created there were (almost) no drivers out there implementing it.
>> Guess Suns driver is one of the few that does not have an
>> implementation.
>>
>>  From an architectural point of view I feel a statement is part of a
>> connection. Therefore it belongs in a connections. However I do admit
>> there are many ways to implement this view.
>>
>> I know that adding an extra jar file will reduce its acceptance in
>> the ISV community. Been working with different ISVs for almost 10
>> years now and every element they include in their suite will in
>> general be included in their Q&A tests. One more element of the suite
>> they support means many more tests. It makes it more difficult (this
>> to some extent might be perception, but it is something that does
>> matter to them) to do it so they will be more reluctant to support
>> it.
>>
>> The wrapper is an extra layer that needs to be crossed while
>> "creating" a statement from the pool. However I doubt that it will
>> make a significant (negative) impact when compared to the all-in-one
>> approach. The real net effect of using statement caching is the
>> reduction of work on the database side.
>>
>> Finally it is not just appsservers out there. Lots if J2SE
>> application exist in at least the Telco segment. Would love to see
>> these guys to adopt a free database instead of Oracle that is all
>> over that place.
>>
>>
>> --Paul
>>
>>
>> On 5-sep-2007, at 20:25, Kris Jurka wrote:
>>
>>>
>>>
>>> On Wed, 5 Sep 2007, Dave Cramer wrote:
>>>
>>>> On 5-Sep-07, at 2:12 PM, Kris Jurka wrote:
>>>>
>>>>> You clearly believe we need this and I'm sort of on the fence (I
>>>>> recall that's where Oliver is as well, but I don't claim to speak
>>>>> for him).
>>>>
>>>> So we basically have one nay vote holding this up ?
>>>>
>>>
>>> But we only have one yes vote pushing it forward.  At least of the
>>> four of us, I haven't gone back to the archives to see if anyone
>>> else weighed in on the discussion.
>>>
>>> Kris Jurka
>>>
>>> ---------------------------(end of
>>> broadcast)---------------------------
>>> TIP 9: In versions below 8.0, the planner will ignore your desire to
>>>       choose an index scan if your joining column's datatypes do not
>>>       match
>>
>> ---------------------------------------------------------------------
>> ---
>> ---------------------
>> Paul van den Bogaard
>> Paul.vandenBogaard@sun.com
>> CIE -- Collaboration and ISV Engineering, Opensource Engineering
>> group
>>
>> Sun Microsystems, Inc                              phone:        +31
>> 334 515 918
>> Saturnus 1
>> extentsion: x (70)15918
>> 3824 ME Amersfoort                                 mobile:       +31
>> 651 913 354
>> The Netherlands
>> fax:            +31 334 515 001
>>
>>
>> ---------------------------(end of
>> broadcast)---------------------------
>> TIP 9: In versions below 8.0, the planner will ignore your desire to
>>        choose an index scan if your joining column's datatypes do not
>>        match
>>

------------------------------------------------------------------------
---------------------
Paul van den Bogaard
Paul.vandenBogaard@sun.com
CIE -- Collaboration and ISV Engineering, Opensource Engineering group

Sun Microsystems, Inc                              phone:        +31
334 515 918
Saturnus 1
extentsion: x (70)15918
3824 ME Amersfoort                                 mobile:       +31
651 913 354
The Netherlands
fax:            +31 334 515 001


Re: Caching driver on pgFoundry?

From
"Heikki Linnakangas"
Date:
Paul van den Bogaard wrote:
> if the caching & pooling stuff is in the same jar file as the rest of
> the JDBC driver than my point is empty. If it is (can be seen as) an
> extra element that it is very likely it will mean extra work for the Q&A
> department of the ISV.
> Test suite is likely to become
>
> on platform A, test with and test without
> on platform B, test with and test without
>
> continues till all platforms are covered.
>
> In general one more element translates in more permutations.

I don't get that. If you need the caching, you're going distribute the
caching component with your software, and you're going to test it *with*
 the caching. Why would you test it without it?

On the other hand, if you want to support your software with and without
the caching, you should test it in both modes, regardless of the packaging.

--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

Re: Caching driver on pgFoundry?

From
Paul van den Bogaard
Date:
Seen this too. However I did spend a significant amount of time
tuning these applications since they have to deliver a certain
throughput.  Statement caching is definitely helpful in those cases.
For the ones you describe could it be that the dbase layer is just
storage and not part of the (critical) performance path?
If the database related work is critical in performance/throughput
than caching is a welcome technology.
I found this matters for the lower level apps to establish  call
setups, check balance on pre paid, and these basic thing were every
tick translates into a tiny bit of money. Unfortunately there are an
awful lot of these tiny little bits. This is what makes it so
interesting from a performance tuning perspective that is.
And yes some of the telco ISVs use Times Ten to do it all in memory.
In that case everything is cached. Mmmm could be a nice extension for
PG...

--Paul


On 6-sep-2007, at 11:14, Oliver Jowett wrote:

> Paul van den Bogaard wrote:
>
>> Finally it is not just appsservers out there. Lots if J2SE
>> application exist in at least the Telco segment. Would love to see
>> these guys to adopt a free database instead of Oracle that is all
>> over that place.
>
> Going off at a bit of a tangent here.. OpenCloud does telco
> infrastructure software and we package postgresql with our product
> to do persistence of long-lived state (deployed services, cluster
> configuration, customer provisioning data etc). The statement
> caching stuff is not really an issue either way for what we do,
> since we already intelligently reuse statements in our persistance
> layer.
>
> One thing we've found is that our customers don't really care about
> database setup for this sort of use, they want it to "just work".
> In fact we package Derby instead of PostgreSQL with our development
> versions just so there's one less thing for the developer to set
> up. So even if we cared about statement caching it wouldn't really
> matter if it was together with the driver or a separate package as
> we'd still end up doing all the packaging together with the
> application and testing of the combined product.
>
> While what we do is essentially an appserver (JAIN SLEE) I suspect
> the same applies to standalone java telco apps, you're going to be
> doing DB integration as part of your app not as something that gets
> plugged in afterwards, so I'm not sure how relevant having it in
> the standard driver is.
>
> -O

------------------------------------------------------------------------
---------------------
Paul van den Bogaard
Paul.vandenBogaard@sun.com
CIE -- Collaboration and ISV Engineering, Opensource Engineering group

Sun Microsystems, Inc                              phone:        +31
334 515 918
Saturnus 1
extentsion: x (70)15918
3824 ME Amersfoort                                 mobile:       +31
651 913 354
The Netherlands
fax:            +31 334 515 001


Re: Caching driver on pgFoundry?

From
Oliver Jowett
Date:
Paul van den Bogaard wrote:

> For the ones you
> describe could it be that the dbase layer is just storage and not part
> of the (critical) performance path?

Right, in our case we keep all state in memory as the primary
representation (and maybe replicated across a cluster) and some specific
state is also asynchronously written to disk via a RDBMS so that it
survives a cluster restart. Performance of the write-back step is still
important as it affects things like our MTTR from a complete
database+cluster failure, but it does not directly affect the call
throughput / latency.

> If the database related work is critical in performance/throughput than
> caching is a welcome technology.

Well no not necessarily because *statement caching does nothing that you
can't do directly in your application anyway*. For example in our case
we would get no additional benefit from statement caching because we (as
a JDBC client) already hold on to statement objects that we know we're
going to reuse, i.e. we're already doing our own caching, and smarter
than a generic layer can do because we have better knowledge of how we
use our queries.

Statement caching is really a "go faster" switch when you don't want to
(or cannot) optimize the application code, or where the exact query
strings are unpredictable. But again it's not something that *has* to be
implemented in the driver, as far as I can tell there's nothing there
that can't be done by a layer between the client and the real driver.

-O

Re: Caching driver on pgFoundry?

From
Paul van den Bogaard
Date:
On 6-sep-2007, at 11:40, Oliver Jowett wrote:

> Paul van den Bogaard wrote:
>
>> For the ones you describe could it be that the dbase layer is just
>> storage and not part of the (critical) performance path?
>
> Right, in our case we keep all state in memory as the primary
> representation (and maybe replicated across a cluster) and some
> specific state is also asynchronously written to disk via a RDBMS
> so that it survives a cluster restart. Performance of the write-
> back step is still important as it affects things like our MTTR
> from a complete database+cluster failure, but it does not directly
> affect the call throughput / latency.
>
>> If the database related work is critical in performance/throughput
>> than caching is a welcome technology.
>
> Well no not necessarily because *statement caching does nothing
> that you can't do directly in your application anyway*. For example
> in our case we would get no additional benefit from statement
> caching because we (as a JDBC client) already hold on to statement
> objects that we know we're going to reuse, i.e. we're already doing
> our own caching, and smarter than a generic layer can do because we
> have better knowledge of how we use our queries.

If all ISVs would do it intelligently than I would have been out of
a  job :-). I definitely agree. There is only one who can (should)
know best and that is the application writer. However I found this is
a rare situation. On the other hand if you do not need this most
optimal situation than indeed it would be very helpfull if this layer
in between would do it for you.
>
> Statement caching is really a "go faster" switch when you don't
> want to (or cannot) optimize the application code, or where the
> exact query strings are unpredictable. But again it's not something
> that *has* to be implemented in the driver, as far as I can tell
> there's nothing there that can't be done by a layer between the
> client and the real driver.

I think from an implementation point of view you can build in and/or
directly on top the JDBC layer. From a performance point of view I do
not expect a significant advantage for one or the other either.

--Paul

>
> -O

------------------------------------------------------------------------
---------------------
Paul van den Bogaard
Paul.vandenBogaard@sun.com
CIE -- Collaboration and ISV Engineering, Opensource Engineering group

Sun Microsystems, Inc                              phone:        +31
334 515 918
Saturnus 1
extentsion: x (70)15918
3824 ME Amersfoort                                 mobile:       +31
651 913 354
The Netherlands
fax:            +31 334 515 001


Re: Caching driver on pgFoundry?

From
Paul van den Bogaard
Date:
You have a point here.

--Paul

On 6-sep-2007, at 11:26, Heikki Linnakangas wrote:

> Paul van den Bogaard wrote:
>> if the caching & pooling stuff is in the same jar file as the rest of
>> the JDBC driver than my point is empty. If it is (can be seen as) an
>> extra element that it is very likely it will mean extra work for
>> the Q&A
>> department of the ISV.
>> Test suite is likely to become
>>
>> on platform A, test with and test without
>> on platform B, test with and test without
>>
>> continues till all platforms are covered.
>>
>> In general one more element translates in more permutations.
>
> I don't get that. If you need the caching, you're going distribute the
> caching component with your software, and you're going to test it
> *with*
>  the caching. Why would you test it without it?
>
> On the other hand, if you want to support your software with and
> without
> the caching, you should test it in both modes, regardless of the
> packaging.
>
> --
>   Heikki Linnakangas
>   EnterpriseDB   http://www.enterprisedb.com

------------------------------------------------------------------------
---------------------
Paul van den Bogaard
Paul.vandenBogaard@sun.com
CIE -- Collaboration and ISV Engineering, Opensource Engineering group

Sun Microsystems, Inc                              phone:        +31
334 515 918
Saturnus 1
extentsion: x (70)15918
3824 ME Amersfoort                                 mobile:       +31
651 913 354
The Netherlands
fax:            +31 334 515 001


Re: Caching driver on pgFoundry?

From
"Heikki Linnakangas"
Date:
Simon Riggs wrote:
> Sun chose to use Blowfish, but could have chosen others for the
> benchmark. We shouldn't go down a route with the software that is driven
> because of an external, unrelated decision.

Blowfish? Is that a nickname for Sun Java System Application Server?

--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

Re: Caching driver on pgFoundry?

From
Simon Riggs
Date:
On Wed, 2007-09-05 at 21:20 +0100, Heikki Linnakangas wrote:
> Kris Jurka wrote:
> > On Wed, 5 Sep 2007, Dave Cramer wrote:
> >> If this is the case then I would argue that having caching in the
> >> appserver is a great idea for everyone using an appserver it does not
> >> help the rest of the world that doesn't use an app server.
> >
> > This is the consensus I'd like to see built before we spend a lot of
> > time on something.  Heikki doesn't believe we should do this at all as
> > it is covered by DBCP and app server pools.
>
> To be clear, I think it's nice to have one more statement cache
> implementation around, with friendly BSD license. I encourage you to
> keep working on it, regardless of how this discussion ends.
>
> But let's not bundle it with the PostgreSQL JDBC driver, it's clearly a
> separate component. For applications that already have another
> connection pool / statement cache implementation, it would be just extra
> baggage. And as a separate project, you can use it with other DBMSs as
> well. Or bundle it with an app server :).
>
> Another point of view is to think about the release cycles of the JDBC
> driver and the caching wrapper. If a bug is found in the wrapper, and
> it's bundled with the JDBC driver, we would have to release a new
> version of the bundle, forcing an upgrade of both components even if you
> only use one. If we add a new feature to the wrapper, is that going to
> be backpatched to say the 8.1 branch? What if you're running 8.1 server
> and driver, and you want to take advantage of a new wrapper feature?
>
> The wrapper has enough merit to live a happy and successful life as a
> project of its own. I don't care if it's a separate pgfoundry project,
> or just a separate CVS directory within jdbc project and separate
> builds; the main point is that it should be a separate jar file with a
> separate release cycle. If we add a link to the latest version of the
> wrapper jar from jdbc.postgresql.org download page, people will find it
> regardless of where the development happens.

This sounds like the right approach to me.

We need to ensure Postgres works with everything and everybody. We do
want to help Blowfish, but not at an expense for other app servers.

Sun chose to use Blowfish, but could have chosen others for the
benchmark. We shouldn't go down a route with the software that is driven
because of an external, unrelated decision.

--
  Simon Riggs
  2ndQuadrant  http://www.2ndQuadrant.com


Re: Caching driver on pgFoundry?

From
Simon Riggs
Date:
On Fri, 2007-09-07 at 10:20 +0100, Heikki Linnakangas wrote:
> Simon Riggs wrote:
> > Sun chose to use Blowfish, but could have chosen others for the
> > benchmark. We shouldn't go down a route with the software that is driven
> > because of an external, unrelated decision.
>
> Blowfish? Is that a nickname for Sun Java System Application Server?

Sorry, Glassfish.

--
  Simon Riggs
  2ndQuadrant  http://www.2ndQuadrant.com


Re: Caching driver on pgFoundry?

From
Dave Cramer
Date:
On 7-Sep-07, at 5:17 AM, Simon Riggs wrote:

> On Wed, 2007-09-05 at 21:20 +0100, Heikki Linnakangas wrote:
>> Kris Jurka wrote:
>>> On Wed, 5 Sep 2007, Dave Cramer wrote:
>>>> If this is the case then I would argue that having caching in the
>>>> appserver is a great idea for everyone using an appserver it
>>>> does not
>>>> help the rest of the world that doesn't use an app server.
>>>
>>> This is the consensus I'd like to see built before we spend a lot of
>>> time on something.  Heikki doesn't believe we should do this at
>>> all as
>>> it is covered by DBCP and app server pools.
>>
>> To be clear, I think it's nice to have one more statement cache
>> implementation around, with friendly BSD license. I encourage you to
>> keep working on it, regardless of how this discussion ends.
>>
>> But let's not bundle it with the PostgreSQL JDBC driver, it's
>> clearly a
>> separate component. For applications that already have another
>> connection pool / statement cache implementation, it would be just
>> extra
>> baggage. And as a separate project, you can use it with other
>> DBMSs as
>> well. Or bundle it with an app server :).
>>
>> Another point of view is to think about the release cycles of the
>> JDBC
>> driver and the caching wrapper. If a bug is found in the wrapper, and
>> it's bundled with the JDBC driver, we would have to release a new
>> version of the bundle, forcing an upgrade of both components even
>> if you
>> only use one. If we add a new feature to the wrapper, is that
>> going to
>> be backpatched to say the 8.1 branch? What if you're running 8.1
>> server
>> and driver, and you want to take advantage of a new wrapper feature?
>>
>> The wrapper has enough merit to live a happy and successful life as a
>> project of its own. I don't care if it's a separate pgfoundry
>> project,
>> or just a separate CVS directory within jdbc project and separate
>> builds; the main point is that it should be a separate jar file
>> with a
>> separate release cycle. If we add a link to the latest version of the
>> wrapper jar from jdbc.postgresql.org download page, people will
>> find it
>> regardless of where the development happens.
>
> This sounds like the right approach to me.
>
> We need to ensure Postgres works with everything and everybody. We do
> want to help Blowfish, but not at an expense for other app servers.

How does the architecture of the driver hinder other app servers ?
>
> Sun chose to use Blowfish, but could have chosen others for the
> benchmark. We shouldn't go down a route with the software that is
> driven
> because of an external, unrelated decision.

After a quick survey I couldn't find another non-GPL open source app
server.

Dave


Re: Caching driver on pgFoundry?

From
"Heikki Linnakangas"
Date:
Dave Cramer wrote:
> After a quick survey I couldn't find another non-GPL open source app
> server.

JOnAS and Apache Geronimo spring to mind.

--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

Re: Caching driver on pgFoundry?

From
Dave Cramer
Date:
On 7-Sep-07, at 7:33 AM, Heikki Linnakangas wrote:

> Dave Cramer wrote:
>> After a quick survey I couldn't find another non-GPL open source app
>> server.
>
> JOnAS and Apache Geronimo spring to mind.

JOnAS is LGPL

>
> --
>   Heikki Linnakangas
>   EnterpriseDB   http://www.enterprisedb.com


Re: Caching driver on pgFoundry?

From
Oliver Jowett
Date:
Dave Cramer wrote:

> After a quick survey I couldn't find another non-GPL open source app
> server.

Isn't that a fairly arbitary categorization to make? How about open
source app servers that don't begin with 'B'? ;-)

Seriously, not sure exactly what point you're trying to make here. Why
exactly should the existance and licensing of 3rd party software affect
technical decisions about the postgresql JDBC driver?

I think we're drifting away from the main point which is, as I see it,
fairly simple:

What is the implementation advantage of making statement pooling part of
the main driver? There are maintenance issues which count *against* it
being part of the driver so you need to provide a good reason to include it.

Packaging issues are IMO fairly irrelevant because if you must give a
single package to your users, you can always take the base driver and a
separate pooling wrapper, package them together, and say "Hey look here
is the statement-pooling postgresql JDBC driver".

-O

Re: Caching driver on pgFoundry?

From
Dave Cramer
Date:
On 7-Sep-07, at 9:13 AM, Oliver Jowett wrote:

> Dave Cramer wrote:
>
>> After a quick survey I couldn't find another non-GPL open source
>> app server.
>
> Isn't that a fairly arbitary categorization to make? How about open
> source app servers that don't begin with 'B'? ;-)

It's not arbitrary at all. see below
>
> Seriously, not sure exactly what point you're trying to make here.
> Why exactly should the existance and licensing of 3rd party
> software affect technical decisions about the postgresql JDBC driver?
>
Companies are fairly careful when choosing components to put in their
stack. GPL licensed products are considered to be business non-
friendly. I was responding to Simon's suggestion that they could have
used any other app server which did implement statement caching.

> I think we're drifting away from the main point which is, as I see
> it, fairly simple:
>
> What is the implementation advantage of making statement pooling
> part of the main driver? There are maintenance issues which count
> *against* it being part of the driver so you need to provide a good
> reason to include it.
>
Well, it has to be maintained regardless of where it is. So how does
that make any difference ?

> Packaging issues are IMO fairly irrelevant because if you must give
> a single package to your users, you can always take the base driver
> and a separate pooling wrapper, package them together, and say "Hey
> look here is the statement-pooling postgresql JDBC driver".

Dave
>
> -O


Fwd: Caching driver on pgFoundry?

From
"László Hornyák"
Date:
Sorry, I wanted to send this to the list...

---------- Forwarded message ----------
From: László Hornyák <kocka.public@gmail.com >
Date: 07-Sep-2007 15:55
Subject: Re: [JDBC] Caching driver on pgFoundry?
To: Oliver Jowett <oliver@opencloud.com>

Hi!

On 07/09/07, Oliver Jowett <oliver@opencloud.com > wrote:


What is the implementation advantage of making statement pooling part of
the main driver? There are maintenance issues which count *against* it
being part of the driver so you need to provide a good reason to include it.

There are two main reasons I can mention:
- JDBC 4 allows prepared statement pooling (Which is optional, the driver can ignore the programs request to cache the prepared statement, and I believe it enables some kind of early optimization for those who like to do early optimizations. But that is just my opinion, but JDBC 4 specification expects drivers to deal with it. Time will tell, maybe data access layers like hibernate or JPA implementations in the future will use it) see http://java.sun.com/javase/6/docs/api/java/sql/Statement.html#setPoolable%28boolean%29
- Clients that do not use DataSource, but only DriverManager can gain significant speed improvement. Now, you get prepapred statement caching if the wrapper ( e.g. DBCP) gives it, not if you use a pg connection directly.

Packaging issues are IMO fairly irrelevant because if you must give a
single package to your users, you can always take the base driver and a
separate pooling wrapper, package them together, and say "Hey look here
is the statement-pooling postgresql JDBC driver".

Yes, any developer can do that, but it would probably save some time for developers if they wouldn't have to.


-O

---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
       choose an index scan if your joining column's datatypes do not
       match



--
László Hornyák

--
László Hornyák

Re: Fwd: Caching driver on pgFoundry?

From
"Heikki Linnakangas"
Date:
László Hornyák wrote:
> On 07/09/07, Oliver Jowett <oliver@opencloud.com> wrote:
>> What is the implementation advantage of making statement pooling part of
>> the main driver? There are maintenance issues which count *against* it
>> being part of the driver so you need to provide a good reason to include
>> it.
>
> There are two main reasons I can mention:
> - JDBC 4 allows prepared statement pooling (Which is optional, the driver
> can ignore the programs request to cache the prepared statement, and I
> believe it enables some kind of early optimization for those who like to do
> early optimizations. But that is just my opinion, but JDBC 4 specification
> expects drivers to deal with it. Time will tell, maybe data access layers
> like hibernate or JPA implementations in the future will use it) see
> http://java.sun.com/javase/6/docs/api/java/sql/Statement.html#setPoolable%28boolean%29

As far as I can see, there's nothing in the spec that you can't
implement in a generic wrapper. The methods added in JDBC4 allow the
application to explicitly ask the connection pool/statement cache to not
cache a statement. Implementation of that feature is exactly the same,
whether the statement cache comes with the driver, the application
server, or as a separate module.

> - Clients that do not use DataSource, but only DriverManager can gain
> significant speed improvement. Now, you get prepapred statement caching if
> the wrapper ( e.g. DBCP) gives it, not if you use a pg connection directly.

Nothing stops you from using the caching wrapper with the DriverManager
interface. I believe DBCP supports that as well.

> Packaging issues are IMO fairly irrelevant because if you must give a
>> single package to your users, you can always take the base driver and a
>> separate pooling wrapper, package them together, and say "Hey look here
>> is the statement-pooling postgresql JDBC driver".
>
> Yes, any developer can do that, but it would probably save some time for
> developers if they wouldn't have to.

Frankly I don't understand what the benefit of a combined jar is. Surely
if you can add one jar to your classpath, you can add two. But I won't
object if someone wants to maintain such a combined jar; it's not going
to harm me or anyone else that such a thing exists, as long as you're
not forced to use it.

--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

Re: Caching driver on pgFoundry?

From
Josh Berkus
Date:
People,

> What is the implementation advantage of making statement pooling part of
> the main driver? There are maintenance issues which count *against* it
> being part of the driver so you need to provide a good reason to include
> it.

How about if we wait for the performance numbers before settling this?

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

Re: Caching driver on pgFoundry?

From
Oliver Jowett
Date:
Dave Cramer wrote:
>
> On 7-Sep-07, at 9:13 AM, Oliver Jowett wrote:
>>
>> What is the implementation advantage of making statement pooling part
>> of the main driver? There are maintenance issues which count *against*
>> it being part of the driver so you need to provide a good reason to
>> include it.
>>
> Well, it has to be maintained regardless of where it is. So how does
> that make any difference ?

Err if it is part of the main driver it needs to be maintained as part
of the main driver and will follow the main driver's release cycle.
Others have commented on the same thing. It also increases the
complexity of the main driver, making it harder for a particular person
to maintain the whole thing.

I'm going to ask again: What is the implementation advantage of making
statement pooling part of the main driver?

-O

Re: Caching driver on pgFoundry?

From
Oliver Jowett
Date:
Josh Berkus wrote:
> People,
>
>> What is the implementation advantage of making statement pooling part of
>> the main driver? There are maintenance issues which count *against* it
>> being part of the driver so you need to provide a good reason to include
>> it.
>
> How about if we wait for the performance numbers before settling this?

Sure, if they're on their way that's fine, Dave hasn't responded to my
earier email asking about it.

-O

Re: Caching driver on pgFoundry?

From
Dave Cramer
Date:
On 8-Sep-07, at 5:38 PM, Oliver Jowett wrote:

> Dave Cramer wrote:
>> On 7-Sep-07, at 9:13 AM, Oliver Jowett wrote:
>>>
>>> What is the implementation advantage of making statement pooling
>>> part of the main driver? There are maintenance issues which count
>>> *against* it being part of the driver so you need to provide a
>>> good reason to include it.
>>>
>> Well, it has to be maintained regardless of where it is. So how
>> does that make any difference ?
>
> Err if it is part of the main driver it needs to be maintained as
> part of the main driver and will follow the main driver's release
> cycle. Others have commented on the same thing. It also increases
> the complexity of the main driver, making it harder for a
> particular person to maintain the whole thing.
>
> I'm going to ask again: What is the implementation advantage of
> making statement pooling part of the main driver?
>
It's simpler, there's about half a dozen classes and one if statement
as opposed to implementing the Connection and Statement interfaces
for 3 versions of  jdbc and yet another build system that does the same.

Dave
> -O


Re: Caching driver on pgFoundry?

From
Oliver Jowett
Date:
Dave Cramer wrote:

> It's simpler, there's about half a dozen classes and one if statement as
> opposed to implementing the Connection and Statement interfaces for 3
> versions of  jdbc and yet another build system that does the same.

How do you avoid wrapping Connection/Statement/etc and still avoid the
problems we discussed with Statement.close()? Can you send me a patch
for review?

-O

Re: Caching driver on pgFoundry?

From
Dave Cramer
Date:
Hi Oliver,

I'm working on the other wrapper driver now. I want to finish
*something* and get it working before I start looking at the issues
that you raised. As you are eluding to it might just be as
complicated to get it right in the driver, but at this point I have
concentrate on the wrapper implementation.

Dave
On 8-Sep-07, at 6:57 PM, Oliver Jowett wrote:

> Dave Cramer wrote:
>
>> It's simpler, there's about half a dozen classes and one if
>> statement as opposed to implementing the Connection and Statement
>> interfaces for 3 versions of  jdbc and yet another build system
>> that does the same.
>
> How do you avoid wrapping Connection/Statement/etc and still avoid
> the problems we discussed with Statement.close()? Can you send me a
> patch for review?
>
> -O


Re: Caching driver on pgFoundry?

From
Oliver Jowett
Date:
Dave Cramer wrote:
> Hi Oliver,
>
> I'm working on the other wrapper driver now. I want to finish
> *something* and get it working before I start looking at the issues that
> you raised. As you are eluding to it might just be as complicated to get
> it right in the driver, but at this point I have concentrate on the
> wrapper implementation.

Ok, how about we shelve this discussion until you're done with the
wrapper, and looked at how you'd do something equivalent as part of the
driver, and got some rough comparisons between the two?

-O

Re: Caching driver on pgFoundry?

From
"Heikki Linnakangas"
Date:
Dave Cramer wrote:
> It's simpler, there's about half a dozen classes and one if statement as
> opposed to implementing the Connection and Statement interfaces for 3
> versions of  jdbc and yet another build system that does the same.

I think it's reasonable to assume that the cache will only be used in
new applications, so you don't need to support as many JDBC versions as
the driver itself.

Actually, do you really need to have a separate version for each JDBC
version? Wouldn't a wrapper written and compiled with JDBC4 work with
older JDKs as well, as long as the application doesn't use the JDBC4
features? I might be missing something, I don't fully understand how it
works in the core driver either.

Have you considered using java.lang.reflect.Proxy? It has some overhead,
but is it significant in modern JVMs?

--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

Re: Caching driver on pgFoundry?

From
"Andy Redhead"
Date:
>
> On 7-Sep-07, at 9:13 AM, Oliver Jowett wrote:
>
> > Dave Cramer wrote:
> >
> >> After a quick survey I couldn't find another non-GPL open
> source app
> >> server.
> >
> > Isn't that a fairly arbitary categorization to make? How about open
> > source app servers that don't begin with 'B'? ;-)
>
> It's not arbitrary at all. see below
> >
> > Seriously, not sure exactly what point you're trying to make here.
> > Why exactly should the existance and licensing of 3rd party
> software
> > affect technical decisions about the postgresql JDBC driver?
> >
> Companies are fairly careful when choosing components to put
> in their stack. GPL licensed products are considered to be
> business non- friendly. I was responding to Simon's
> suggestion that they could have used any other app server
> which did implement statement caching.
>

As a "satisfied" postgresql jdbc user and someone who works with a couple of
open source appservers I think its fair to point out that while GPL is
considered by many to be "non business friendly", many businesses are happy
with the LGPL licence.

Jboss is released under the LGPL and the jboss connection pool does include
statement caching (ie, the pool remembers which statements have been
prepared against which connections - which is my understanding of the
purpose of this thread).

So, if you are looking for an open source java/jee with postgresql option
that includes connection pool level caching of prepared statements then
there is already at least one option out there.

My own view is that I'd be suspicious of including connection level caching
in the same package as the standard JDBC driver if:

1) the pooling code slows down the standard driver when the pooling isn't
used

2) the release cycle of the pooling code "hobbles" the release of the main
driver

3) the pooling code breaks existing external pooling/caching options

Anyway, that’s my £0.02 - thanks for all the good work :)

Cheers,

Andy

> > I think we're drifting away from the main point which is,
> as I see it,
> > fairly simple:
> >
> > What is the implementation advantage of making statement
> pooling part
> > of the main driver? There are maintenance issues which count
> > *against* it being part of the driver so you need to provide a good
> > reason to include it.
> >
> Well, it has to be maintained regardless of where it is. So
> how does that make any difference ?
>
> > Packaging issues are IMO fairly irrelevant because if you
> must give a
> > single package to your users, you can always take the base
> driver and
> > a separate pooling wrapper, package them together, and say
> "Hey look
> > here is the statement-pooling postgresql JDBC driver".
>
> Dave
> >
> > -O
>
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings
>