Thread: Re: [HACKERS] Packaging of postgresql-jdbc

Hi Matteo,

On Wednesday 17 of February 2016 10:39:43 Matteo Melli wrote:
> I am in the process of creating package torodb (
> https://github.com/torodb/torodb) for Fedora (
> https://bugzilla.redhat.com/show_bug.cgi?id=1302156). postgresql-jdbc is a
> torodb's dependency and, if possible, our goal is to include it in Fedora
> 22 and above. Actually we are trying to add the postgresql jdbc driver
> latest version as a bundle in our package but, as Pavel Raiskup told us,
> the right way is to have it stand-alone and depend on it. Since we have
> same goal to build the postgresql jdbc driver I decided to kick in and
> share my progresses with the community.

this is certainly an issue and we have already written similar patches
you've proposed.  Thanks for your work!  As this issue is not "just" about
Fedora, but it rather hurts all GNU/Linux distributions - let's discuss
how to do it a consistent way...

> For now I just removed waffle dependency patching code (attached) and
> removing a couple classes from build:
>
> * org/postgresql/sspi/NTDSAPI.java
> * org/postgresql/sspi/NTDSAPIWrapper.java

Yeah, that is something which should be optional.  As this is not easy, we
should patch it out.

> Actually I am looking into the org.osgi.enterprise artifact dependency that
> is missing, as you mentioned, due to incompatible license with Fedora.
> After investigating a bit I found out that org.osgi.enterprise library is
> shipped with an Apache License (Version 2.0, January 2004) so it is not
> clear to me if there are license issues or not. In any case postgresql-jdbc
> package use a single interface (org.osgi.service.jdbc.DataSourceFactory)
> from artifact org.osgi.enterprise. It would be really trivial to generate a
> clean room implementation of the interface and put it in a separate package
> or just use for build without installing.

AFAIU, osgi.enterprise refuses you to modify it's code.  That requirement
makes it incompatible with Fedora and all open source re-distributors
(vendor-lock-in).

I'm not sure what are the issues with GPL license compatibility .. IANAL,
but IMO you can use MIT/BSD libraries from GPL licensed project, but then
you have to distribute your whole work as GPL code -- which is not a
problem per se, but GPL *enforces* you to keep the code modifiable, while
osgi.enterprise refuses this;  this sounds like a clear collision.
However, we agreed this is not an issue for pgjdbc project and thus no
need to complicate upstream maintenance.  There was long discussion .. and
probably nothing more to add.

So -- patching out osgi.enterprise support would work for us similarly to
waffle.  And possibly in future; make the 'osgi.enterprise' support in
separate jar file with clear license clarification would work too..

---

People mostly (including pgjdbc upstream maintainers) suggest us to patch
the unclear stuff out from pgjdbc code .. which is pain to do over and
over again in all distributions.  We are considering a "close" fork of
pgjdbc (say 'pgjdbc-foss').  That fork should allow us all to package the
jdbc plugin "consistent" FOSS way.  There is no point in duplicating the
maintenance effort -- pgjdbc upstream does a good job here -- so we would
be in sync with pgjdbc and concentrating just on the distribution POV;
doing just the necessary changes.

I'm going to write a separate mail into new trhead to this list (with CC
to package maintainers in other distros) with ideas, requirements, etc.
But any idea from anyone here is welcome.

Thanks, Pavel

     Hi list!

On 17/02/16 12:43, Pavel Raiskup wrote:
> Hi Matteo,
>
> On Wednesday 17 of February 2016 10:39:43 Matteo Melli wrote:
>> I am in the process of creating package torodb (
>> https://github.com/torodb/torodb) for Fedora (
>> https://bugzilla.redhat.com/show_bug.cgi?id=1302156). postgresql-jdbc is a
>> torodb's dependency and, if possible, our goal is to include it in Fedora
>> 22 and above. Actually we are trying to add the postgresql jdbc driver
>> latest version as a bundle in our package but, as Pavel Raiskup told us,
>> the right way is to have it stand-alone and depend on it. Since we have
>> same goal to build the postgresql jdbc driver I decided to kick in and
>> share my progresses with the community.
> this is certainly an issue and we have already written similar patches
> you've proposed.  Thanks for your work!  As this issue is not "just" about
> Fedora, but it rather hurts all GNU/Linux distributions - let's discuss
> how to do it a consistent way...

     Absolutely.

>
>> For now I just removed waffle dependency patching code (attached) and
>> removing a couple classes from build:
>>
>> * org/postgresql/sspi/NTDSAPI.java
>> * org/postgresql/sspi/NTDSAPIWrapper.java
> Yeah, that is something which should be optional.  As this is not easy, we
> should patch it out.

     So I believe this is a simple, good solution. The problem with
waffle is how to "remove" it without asking upstream to change it. Well,
what Matteo did is simply patch it but as part of the spec to build the
src.rpm. This patch is very simple and makes it buildable as an RPM
without asking upstream to make any changes to the code. Thus I think
this fixes this problem.

>
>> Actually I am looking into the org.osgi.enterprise artifact dependency that
>> is missing, as you mentioned, due to incompatible license with Fedora.
>> After investigating a bit I found out that org.osgi.enterprise library is
>> shipped with an Apache License (Version 2.0, January 2004) so it is not
>> clear to me if there are license issues or not. In any case postgresql-jdbc
>> package use a single interface (org.osgi.service.jdbc.DataSourceFactory)
>> from artifact org.osgi.enterprise. It would be really trivial to generate a
>> clean room implementation of the interface and put it in a separate package
>> or just use for build without installing.
> AFAIU, osgi.enterprise refuses you to modify it's code.  That requirement
> makes it incompatible with Fedora and all open source re-distributors
> (vendor-lock-in).
>
> I'm not sure what are the issues with GPL license compatibility .. IANAL,
> but IMO you can use MIT/BSD libraries from GPL licensed project, but then
> you have to distribute your whole work as GPL code -- which is not a
> problem per se, but GPL *enforces* you to keep the code modifiable, while
> osgi.enterprise refuses this;  this sounds like a clear collision.
> However, we agreed this is not an issue for pgjdbc project and thus no
> need to complicate upstream maintenance.  There was long discussion .. and
> probably nothing more to add.
>
> So -- patching out osgi.enterprise support would work for us similarly to
> waffle.  And possibly in future; make the 'osgi.enterprise' support in
> separate jar file with clear license clarification would work too..
>
> ---
>
> People mostly (including pgjdbc upstream maintainers) suggest us to patch
> the unclear stuff out from pgjdbc code .. which is pain to do over and
> over again in all distributions.  We are considering a "close" fork of
> pgjdbc (say 'pgjdbc-foss').  That fork should allow us all to package the
> jdbc plugin "consistent" FOSS way.  There is no point in duplicating the
> maintenance effort -- pgjdbc upstream does a good job here -- so we would
> be in sync with pgjdbc and concentrating just on the distribution POV;
> doing just the necessary changes.
>
> I'm going to write a separate mail into new trhead to this list (with CC
> to package maintainers in other distros) with ideas, requirements, etc.
> But any idea from anyone here is welcome.
>
> Thanks, Pavel

     I think that creating and maintaining a pgjdbc-foss is both time
consuming and potentially confusing for end users. I believe what Matteo
was suggesting is a probably better solution and it's definitely KISS.

     Summarizing Matteo's suggestion:

- osgi.enterprise is not allowed in Fedora (and it seems futile to me to
discuss whether this is right or wrong).

- Apache Felix, which is already packaged for Fedora, provides almost
all the code needed for OSGI support in pgjdbc

- Only 1 interface is not provided by Felix. Rather than looking for
someone else to provide this interface, we should just re-implement it
(it's just a few lines of code!). It could be done as a clean-room
implementation to avoid actually copying the code. This interface could
either be added to pgjdbc or to a side repo (something like
pgjdbc-osgi-compatibility or any other similar name) and setup a
dependency on this trivial project.


     With the above recipe, pgjdbc should also be buildable as an RPM,
without modifying upstream code. I believe no other problems would be
left for packaging it. What do you think?


     Cheers,

     Álvaro


--
Álvaro Hernández Tortosa


-----------
8Kdata

On Wednesday 17 of February 2016 16:55:24 Álvaro Hernández Tortosa wrote:
> > Yeah, that is something which should be optional.  As this is not easy, we
> > should patch it out.
>
>      So I believe this is a simple, good solution. The problem with
> waffle is how to "remove" it without asking upstream to change it. Well,
> what Matteo did is simply patch it but as part of the spec to build the
> src.rpm. This patch is very simple and makes it buildable as an RPM
> without asking upstream to make any changes to the code. Thus I think
> this fixes this problem.

Right, but Vladimir called this "time-bomb", and I mostly agree.  And it
fixes RPM packaging only -> not DEB packaging, archlinux, gentoo, etc., ..
This deserves central place.

>      I think that creating and maintaining a pgjdbc-foss is both time
> consuming and potentially confusing for end users. I believe what Matteo
> was suggesting is a probably better solution and it's definitely KISS.

It is not better solution, just easier -- but as all distros need to do
this, it should be done on one place.  It wouldn't be confusing, there
would definitely be documentation for it.

>      Summarizing Matteo's suggestion:
>
> - osgi.enterprise is not allowed in Fedora (and it seems futile to me to
> discuss whether this is right or wrong).

Right, the Fedora's POV is off-topic.  I'm saying it "might not" be safe
for FOSS distribution model to depend on this software.  That is why the
-foss suffix.

> - Apache Felix, which is already packaged for Fedora, provides almost
> all the code needed for OSGI support in pgjdbc
>
> - Only 1 interface is not provided by Felix. Rather than looking for
> someone else to provide this interface, we should just re-implement it
> (it's just a few lines of code!). It could be done as a clean-room
> implementation to avoid actually copying the code. This interface could
> either be added to pgjdbc or to a side repo (something like
> pgjdbc-osgi-compatibility or any other similar name) and setup a
> dependency on this trivial project.

It does not seem to be worth the trouble at all, because nobody wants
this.  Distributions probably don't care about osgi.enterprise too much
(== obsoleted packages provided, probably unused, patching out the osgi
support to allow build, etc.), and upstream does not care (for
understandable reasons) to make this soft-dependancy.  Who would be this
additional work for to make the trouble?

>      With the above recipe, pgjdbc should also be buildable as an RPM,
> without modifying upstream code. I believe no other problems would be
> left for packaging it. What do you think?

There are other problems (and we probably don't have all):

- There is the '*-parent-poms' *separate* project, this makes us to make
  the package build done in two phases.  I still don't understand this
  artificial split.

- non-traditional release practices -- we need 'pgdjbc-X.Y.Z.tar.gz'
  file, which clearly describes how to build from source.

- I haven't been able to run the new tests.., which seem to be Travis
  (ubuntu && maven?) oriented

I agree that this could look like expensive effort, but as I said - we do
not want diverge.  Just fix the distribution issues ;).  Nothing more than
what all distributions do anyway with new release, but a consistent way.

Pavel

On 17/02/16 17:47, Pavel Raiskup wrote:
> On Wednesday 17 of February 2016 16:55:24 Álvaro Hernández Tortosa wrote:
>>> Yeah, that is something which should be optional.  As this is not easy, we
>>> should patch it out.
>>       So I believe this is a simple, good solution. The problem with
>> waffle is how to "remove" it without asking upstream to change it. Well,
>> what Matteo did is simply patch it but as part of the spec to build the
>> src.rpm. This patch is very simple and makes it buildable as an RPM
>> without asking upstream to make any changes to the code. Thus I think
>> this fixes this problem.
> Right, but Vladimir called this "time-bomb", and I mostly agree.  And it
> fixes RPM packaging only -> not DEB packaging, archlinux, gentoo, etc., ..
> This deserves central place.

     Why would it be a time-bomb?

     While I'd agree a better solution is to work around pgjdbc's code
to circumvent waffle on non-Windows systems, I see this could take long,
it's non trivial and I don't see anyone working on it (please correct me
if I'm wrong). So why not, rather than searching for the perfect
solution, take a simpler one and then iterate?

     Other distros will surely benefit anyway, as the process of
patching a source rpm/deb/whatever only differs in implementation
details (how you do it), not on the concept of doing it.


>
>>       I think that creating and maintaining a pgjdbc-foss is both time
>> consuming and potentially confusing for end users. I believe what Matteo
>> was suggesting is a probably better solution and it's definitely KISS.
> It is not better solution, just easier

     Well, that's already probably better: KISS! :)


>   -- but as all distros need to do
> this, it should be done on one place.  It wouldn't be confusing, there
> would definitely be documentation for it.

     I'd take this one step at a time. Let's package for Fedora, next
when others ask, we will provide the answer: do this patching and blah
blah blah, done :)

>
>>       Summarizing Matteo's suggestion:
>>
>> - osgi.enterprise is not allowed in Fedora (and it seems futile to me to
>> discuss whether this is right or wrong).
> Right, the Fedora's POV is off-topic.  I'm saying it "might not" be safe
> for FOSS distribution model to depend on this software.  That is why the
> -foss suffix.
>
>> - Apache Felix, which is already packaged for Fedora, provides almost
>> all the code needed for OSGI support in pgjdbc
>>
>> - Only 1 interface is not provided by Felix. Rather than looking for
>> someone else to provide this interface, we should just re-implement it
>> (it's just a few lines of code!). It could be done as a clean-room
>> implementation to avoid actually copying the code. This interface could
>> either be added to pgjdbc or to a side repo (something like
>> pgjdbc-osgi-compatibility or any other similar name) and setup a
>> dependency on this trivial project.
> It does not seem to be worth the trouble at all, because nobody wants
> this.

     The trouble is just some hours of work, and we're (8Kdata) willing
to contribute those, no problem.

>   Distributions probably don't care about osgi.enterprise too much
> (== obsoleted packages provided, probably unused, patching out the osgi
> support to allow build, etc.), and upstream does not care (for
> understandable reasons) to make this soft-dependancy.  Who would be this
> additional work for to make the trouble?

     I think we should let the users choose whether they care or not.
This is a functionality of the driver, and removing it is removing
functionality from the driver. And upstream seems not to be very happy
with this.

     We definitely don't know how much or less users use this, but
removing something that works and might be used is what may cause
trouble. Specially, if it is just a few hours of work away to fix the
licensing issue.

>
>>       With the above recipe, pgjdbc should also be buildable as an RPM,
>> without modifying upstream code. I believe no other problems would be
>> left for packaging it. What do you think?
> There are other problems (and we probably don't have all):
>
> - There is the '*-parent-poms' *separate* project, this makes us to make
>    the package build done in two phases.  I still don't understand this
>    artificial split.

     It is done to help make versions of the driver for different
versions of the JRE without duplicating a lot of code. Whatever it is,
it is not a big problem, and I believe this is nowadays non-negotiable
for upstream. Again, at 8Kdata we have been able to work around this
easily and can provide a spec that handles this :)

>
> - non-traditional release practices -- we need 'pgdjbc-X.Y.Z.tar.gz'
>    file, which clearly describes how to build from source.

     Again this can be easily fixed in the spec. There are of course
versioned tar.gz on Github: https://github.com/pgjdbc/pgjdbc/releases

>
> - I haven't been able to run the new tests.., which seem to be Travis
>    (ubuntu && maven?) oriented

     I think running tests is cool for the packaging process, but is
this needed for packaging? I mean, it's upstream who should run them
before releasing versions, right? :)


>
> I agree that this could look like expensive effort, but as I said - we do
> not want diverge.  Just fix the distribution issues ;).  Nothing more than
> what all distributions do anyway with new release, but a consistent way.

     It's not easy to package Maven-based software, as there is some
overlapping in the concept but with different point of view. I won't
further look into expensive efforts. There are few resources and we'd
better direct them to other areas.

     If the proposed solution works, and it's simple, why not just
simply go for it?

     Cheers,

     Álvaro



--
Álvaro Hernández Tortosa


-----------
8Kdata

Álvaro>    I think running tests is cool for the packaging process,
but is this needed for packaging? I mean, it's upstream who should run
them before releasing versions, right? :)

If you pick "upstream build artifact", you know it is tested.
If you use custom-made-kludge-patched build process, you'd better test
that new artifact before use.
It can easily fail in a unexpected way. For instance: not all ${...}
being replaced, not all the classes present, missing translations,
etc, etc.

Nothing personal. Just facts.

Vladimir

On 17/02/16 18:43, Vladimir Sitnikov wrote:
> Álvaro>    I think running tests is cool for the packaging process,
> but is this needed for packaging? I mean, it's upstream who should run
> them before releasing versions, right? :)
>
> If you pick "upstream build artifact", you know it is tested.
     Yeah, that's what I meant :) No need to test it if taken from upstream.

> If you use custom-made-kludge-patched build process, you'd better test
> that new artifact before use.
> It can easily fail in a unexpected way. For instance: not all ${...}
> being replaced, not all the classes present, missing translations,
> etc, etc.

     Absolutely. But the two proposed changes here are:

- Remove waffle dependency from pom file (it's obviously not needed in
Fedora). I don't think this requires any testing (if it compiles, of course)

- Replace osgi.enterprise by something else. This something else will be
present in upstream, and hence tests.

     :)


     Álvaro

--
Álvaro Hernández Tortosa


-----------
8Kdata

On Wednesday 17 of February 2016 18:25:30 Álvaro Hernández Tortosa wrote:
> > Right, but Vladimir called this "time-bomb", and I mostly agree.  And it
> > fixes RPM packaging only -> not DEB packaging, archlinux, gentoo, etc., ..
> > This deserves central place.
>
>      Why would it be a time-bomb?

Because the patch might start to not apply in future, and all the
distributions need to start re-patching...

>      While I'd agree a better solution is to work around pgjdbc's code
> to circumvent waffle on non-Windows systems, I see this could take long,
> it's non trivial and I don't see anyone working on it (please correct me
> if I'm wrong).

I believe this has been discussed before, and Pavel Kajaba already worked
on it .. but this is not acceptable by upstream.

> So why not, rather than searching for the perfect solution, take a
> simpler one and then iterate?

Don't take me wrong -- it is easier to do the patching your way, but on
one place!  Then take the tarball and you are done.

>      Other distros will surely benefit anyway, as the process of
> patching a source rpm/deb/whatever only differs in implementation
> details (how you do it), not on the concept of doing it.

Well, how exactly you expect this patch to be applied?  Because deb
maintainers usually do not watch RPM packaging and vice versa.

> >>       I think that creating and maintaining a pgjdbc-foss is both time
> >> consuming and potentially confusing for end users. I believe what Matteo
> >> was suggesting is a probably better solution and it's definitely KISS.
> > It is not better solution, just easier
>
>      Well, that's already probably better: KISS! :)

Again, KISS and DRY principles.

> >   -- but as all distros need to do
> > this, it should be done on one place.  It wouldn't be confusing, there
> > would definitely be documentation for it.
>
>      I'd take this one step at a time. Let's package for Fedora, next
> when others ask, we will provide the answer: do this patching and blah
> blah blah, done :)

Agreed here.

> > It does not seem to be worth the trouble at all, because nobody wants
> > this.
>
>      The trouble is just some hours of work, and we're (8Kdata) willing
> to contribute those, no problem.

Right, but you need to do the contribution somewhere.  This is not about
RPMs only.

> >   Distributions probably don't care about osgi.enterprise too much
> > (== obsoleted packages provided, probably unused, patching out the osgi
> > support to allow build, etc.), and upstream does not care (for
> > understandable reasons) to make this soft-dependancy.  Who would be this
> > additional work for to make the trouble?
>
>      I think we should let the users choose whether they care or not.

Exactly, but not possible ATM if we pay attention to foss principles.

> This is a functionality of the driver, and removing it is removing
> functionality from the driver. And upstream seems not to be very happy
> with this.

End users would be able to use 'pgdjbc-foss' or 'pgjdbc-foss-osgi' (KISS).

>      We definitely don't know how much or less users use this, but
> removing something that works and might be used is what may cause
> trouble. Specially, if it is just a few hours of work away to fix the
> licensing issue.

Oh, I see what you mean :) now -- but those licensing issues should be
fixed upstream..

> >>       With the above recipe, pgjdbc should also be buildable as an RPM,
> >> without modifying upstream code. I believe no other problems would be
> >> left for packaging it. What do you think?
> > There are other problems (and we probably don't have all):
> >
> > - There is the '*-parent-poms' *separate* project, this makes us to make
> >    the package build done in two phases.  I still don't understand this
> >    artificial split.
>
>      It is done to help make versions of the driver for different
> versions of the JRE without duplicating a lot of code. Whatever it is,
> it is not a big problem,

I still don't get the reason why there can't be single one tarball.  It is
a problem, every distro is going to build the package in two-steps.
Gentoo guys do build their packages themselves!

> and I believe this is nowadays non-negotiable for upstream. Again, at
> 8Kdata we have been able to work around this easily and can provide a
> spec that handles this :)

I was able to do this too.  But having a fork will make this easy for
everyone (DRY).

> > - I haven't been able to run the new tests.., which seem to be Travis
> >    (ubuntu && maven?) oriented
>
>      I think running tests is cool for the packaging process, but is
> this needed for packaging? I mean, it's upstream who should run them
> before releasing versions, right? :)

No-no-no :).  Ideally¹ we should be able to (a) run the testsuite during
package build (no root access, no network access) in cloud, and (b) we
should be able to package the testsuite for user -- so anyone is able to
run the testsuite on end-user box.  This sounds like sci-fi now, but it is
really important to run the testsuite at least during package build.

> > I agree that this could look like expensive effort, but as I said - we do
> > not want diverge.  Just fix the distribution issues ;).  Nothing more than
> > what all distributions do anyway with new release, but a consistent way.
>
>      It's not easy to package Maven-based software, as there is some
> overlapping in the concept but with different point of view. I won't
> further look into expensive efforts. There are few resources and we'd
> better direct them to other areas.

Right, thats why I do not plant to do anything expensive!  Just
accumulate the things at one place -- the things which upstream forces us
to do on per-distribution level.

>      If the proposed solution works, and it's simple, why not just
> simply go for it?

It works.

¹ This has been don for postgresql.spec (server).  But with pgjdbc we are
  even unable to *build* from source :).

Pavel

On Wednesday 17 of February 2016 20:43:50 Vladimir Sitnikov wrote:
> Álvaro>    I think running tests is cool for the packaging process,
> but is this needed for packaging? I mean, it's upstream who should run
> them before releasing versions, right? :)
>
> If you pick "upstream build artifact", you know it is tested.

Right, test in some defined CI environment, which is perfect -- but you
are still not able to run the tests on Gentoo/Arch/Fedora/RHEL
environments, without maven, etc.

We do care about testing in Fedora -- that is why we enable testsuites
after package builds automatically (before it is provided to end users)
and we have real people doing the QA manually.  So you *still know* that
the software is tested.

It still does not mean that your tests are not beneficial, just saying
that those should be probably more adaptable to allow us to use those.

> If you use custom-made-kludge-patched build process, you'd better test
> that new artifact before use.

This is truth for every software, maven is not an exception.  The negative
thing in maven is that you might easily end-up crossing laws (if you e.g.
distribute to your customer GPL based software dependant on osgi).

> It can easily fail in a unexpected way. For instance: not all ${...}
> being replaced, not all the classes present, missing translations, etc,
> etc.
>
> Nothing personal. Just facts.

The missing classes are not missed on Linux.  And nobody else is going to
use those jar files, so no problem there.  But yes.  What I propose here
is not perfect, but is the safer, legally safer -- and at first it is the
less painful :( if we do sum(downstream maintainers work).

Pavel

On 17/02/16 19:22, Pavel Raiskup wrote:
> On Wednesday 17 of February 2016 18:25:30 Álvaro Hernández Tortosa wrote:
>>> Right, but Vladimir called this "time-bomb", and I mostly agree.  And it
>>> fixes RPM packaging only -> not DEB packaging, archlinux, gentoo, etc., ..
>>> This deserves central place.
>>       Why would it be a time-bomb?
> Because the patch might start to not apply in future, and all the
> distributions need to start re-patching...

     The patch is a very very simple one. It's not much more than "under
this XML Path, remove this XML node (the maven dependency)". While it
may break at some point (well, that's what happens when you patch
upstream), but it's unlikely.

     The only other real alternative is change the code to work around
waffle, and after a long time debating I have seen nobody to come up
with a fix for this.

     So I'd rather move forward even if the solution is not ideal...

     or is there any other *already working, not theoretically*
alternative? My main point is: please, GTD! :)


>
>>       While I'd agree a better solution is to work around pgjdbc's code
>> to circumvent waffle on non-Windows systems, I see this could take long,
>> it's non trivial and I don't see anyone working on it (please correct me
>> if I'm wrong).
> I believe this has been discussed before, and Pavel Kajaba already worked
> on it .. but this is not acceptable by upstream.

     Yeah, so we have to work around it. That's why we are suggesting
this alternative.

>
>> So why not, rather than searching for the perfect solution, take a
>> simpler one and then iterate?
> Don't take me wrong -- it is easier to do the patching your way, but on
> one place!  Then take the tarball and you are done.

     If I understood it correctly, the RPM specs specifically allow (and
I believe is common place) to patch upstream software, to adapt it to
the specificities of RPM packaging. I believe this patch perfectly
belongs to this category.
Good news is that to do this, we don't need any upstream approval :)

>
>>       Other distros will surely benefit anyway, as the process of
>> patching a source rpm/deb/whatever only differs in implementation
>> details (how you do it), not on the concept of doing it.
> Well, how exactly you expect this patch to be applied?  Because deb
> maintainers usually do not watch RPM packaging and vice versa.

     Well, when Debian maintainers would ask pgjdcb, the same way as you
did, we will point them to the solution applied to the RPM, which will
be already working, so that they could apply the same technique
--shouldn't be hard.

>
>>>>        I think that creating and maintaining a pgjdbc-foss is both time
>>>> consuming and potentially confusing for end users. I believe what Matteo
>>>> was suggesting is a probably better solution and it's definitely KISS.
>>> It is not better solution, just easier
>>       Well, that's already probably better: KISS! :)
> Again, KISS and DRY principles.

     We both agree :) But forking pgjdbc to create pgjdbc-foss is, to
me, much more DRY than a tiny patch that might be distribution-dependant.

>
>>>    -- but as all distros need to do
>>> this, it should be done on one place.  It wouldn't be confusing, there
>>> would definitely be documentation for it.
>>       I'd take this one step at a time. Let's package for Fedora, next
>> when others ask, we will provide the answer: do this patching and blah
>> blah blah, done :)
> Agreed here.
>
>>> It does not seem to be worth the trouble at all, because nobody wants
>>> this.
>>       The trouble is just some hours of work, and we're (8Kdata) willing
>> to contribute those, no problem.
> Right, but you need to do the contribution somewhere.  This is not about
> RPMs only.

     We can do this contribution on our own repos, and send a PR to
pgjdbc. This is not hard, we can do it in <24h.

>
>>>    Distributions probably don't care about osgi.enterprise too much
>>> (== obsoleted packages provided, probably unused, patching out the osgi
>>> support to allow build, etc.), and upstream does not care (for
>>> understandable reasons) to make this soft-dependancy.  Who would be this
>>> additional work for to make the trouble?
>>       I think we should let the users choose whether they care or not.
> Exactly, but not possible ATM if we pay attention to foss principles.

     I care a lot about FOSS principles, and that's the main reason if
we rewrite the interfaces, they will be completely FOSS.

>
>> This is a functionality of the driver, and removing it is removing
>> functionality from the driver. And upstream seems not to be very happy
>> with this.
> End users would be able to use 'pgdjbc-foss' or 'pgjdbc-foss-osgi' (KISS).
     I think this will make users nuts. There will be three versions of
the driver:

- One in the official website (pgjdbc), not in Fedora
- One called pgjdbc-foss, only in Fedora, not mentioned anywhere else
- One called pgjdbc-foss-osgi, only in Fedora, not mentioned anywhere
else, which will create even confusion against the non-osgi one

     Plus the huge burden of maintaining two! forks of pgjdbc.

     I believe it's more KISS to just re-write the OSGI stuff used in
pgjdbc, license that PostgreSQL, and you're good to go, with a pure FOSS
package.

>
>>       We definitely don't know how much or less users use this, but
>> removing something that works and might be used is what may cause
>> trouble. Specially, if it is just a few hours of work away to fix the
>> licensing issue.
> Oh, I see what you mean :) now -- but those licensing issues should be
> fixed upstream..

     Right. We can work on fixing them and submitting a PR.

>
>>>>        With the above recipe, pgjdbc should also be buildable as an RPM,
>>>> without modifying upstream code. I believe no other problems would be
>>>> left for packaging it. What do you think?
>>> There are other problems (and we probably don't have all):
>>>
>>> - There is the '*-parent-poms' *separate* project, this makes us to make
>>>     the package build done in two phases.  I still don't understand this
>>>     artificial split.
>>       It is done to help make versions of the driver for different
>> versions of the JRE without duplicating a lot of code. Whatever it is,
>> it is not a big problem,
> I still don't get the reason why there can't be single one tarball.  It is
> a problem, every distro is going to build the package in two-steps.
> Gentoo guys do build their packages themselves!

     I think this is a matter of taste. I believe having a separate
package for the osgi interfaces that you are "copying" to provide a
compatibility layer belong to a separate project, and hence two
tarballs. But if that would be a showstopper, it could be integrated
into pgjdbc itself. But I'd wait for other's (pgjdbc core) opinion here
before doing that.
>
>> and I believe this is nowadays non-negotiable for upstream. Again, at
>> 8Kdata we have been able to work around this easily and can provide a
>> spec that handles this :)
> I was able to do this too.  But having a fork will make this easy for
> everyone (DRY).
>
>>> - I haven't been able to run the new tests.., which seem to be Travis
>>>     (ubuntu && maven?) oriented
>>       I think running tests is cool for the packaging process, but is
>> this needed for packaging? I mean, it's upstream who should run them
>> before releasing versions, right? :)
> No-no-no :).  Ideally¹ we should be able to (a) run the testsuite during
> package build (no root access, no network access) in cloud, and (b) we
> should be able to package the testsuite for user -- so anyone is able to
> run the testsuite on end-user box.  This sounds like sci-fi now, but it is
> really important to run the testsuite at least during package build.

     OK, nothing against that :)

>
>>> I agree that this could look like expensive effort, but as I said - we do
>>> not want diverge.  Just fix the distribution issues ;).  Nothing more than
>>> what all distributions do anyway with new release, but a consistent way.
>>       It's not easy to package Maven-based software, as there is some
>> overlapping in the concept but with different point of view. I won't
>> further look into expensive efforts. There are few resources and we'd
>> better direct them to other areas.
> Right, thats why I do not plant to do anything expensive!  Just
> accumulate the things at one place -- the things which upstream forces us
> to do on per-distribution level.
>
>>       If the proposed solution works, and it's simple, why not just
>> simply go for it?
> It works.
>
> ¹ This has been don for postgresql.spec (server).  But with pgjdbc we are
>    even unable to *build* from source :).

     I'm sorry I'm not that familiar with pgjdbc test suite, but post a
separate message to this list with the errors you encountered to check
them :)

     Thanks for all your input!

     Álvaro


--
Álvaro Hernández Tortosa


-----------
8Kdata

Thanks a lot, I appreciate your concerns and I mostly agree with you.  Let
me answer just quickly to the important points which need to be
underlined.  I could more, but I would repeat myself:

  I do agree that the fork is not optimal, but that is the
  only way to go to keep the distribution sane, consistent.
  At least as long as upstream considers downstream packaging
  as something which is not important.

On Wednesday 17 of February 2016 19:41:52 Álvaro Hernández Tortosa wrote:
>      The only other real alternative is change the code to work around
> waffle, and after a long time debating I have seen nobody to come up
> with a fix for this.

Please 's/work-around/make soft-dependency/' and specify that the work
must be done upstream.

> > I believe this has been discussed before, and Pavel Kajaba already
> > worked on it .. but this is not acceptable by upstream.
>
>      Yeah, so we have to work around it. That's why we are suggesting
> this alternative.

This _is good alternative_ which needs, however, to be done on one place.
More eyes see more bugs -- so the "hacks" will be done properly if
distributions concentrate on one place.  At least those upstreams who are
interested.

> > Don't take me wrong -- it is easier to do the patching your way, but on
> > one place!  Then take the tarball and you are done.
>
>      If I understood it correctly, the RPM specs specifically allow (and
> I believe is common place) to patch upstream software, to adapt it to
> the specificities of RPM packaging. I believe this patch perfectly
> belongs to this category.

No, this is not possible.  If you fix our spec file to apply some
downstream patch, you fix only RPM.

> Good news is that to do this, we don't need any upstream approval :)

That is what the fork is meant for.  From that forked code we can generate
(the only one - self-standing) tarball -- which anybody can take and use
to build from source easily (fixed build scripts).

> >>       Other distros will surely benefit anyway, as the process of
> >> patching a source rpm/deb/whatever only differs in implementation
> >> details (how you do it), not on the concept of doing it.
> > Well, how exactly you expect this patch to be applied?  Because deb
> > maintainers usually do not watch RPM packaging and vice versa.
>
>      Well, when Debian maintainers would ask pgjdcb, the same way as you
> did, we will point them to the solution applied to the RPM, which will
> be already working, so that they could apply the same technique
> --shouldn't be hard.

Yeah, I'll ping (not only) Debian guys soon.  We should do the hacking
technique together and the constant way.

>      We both agree :) But forking pgjdbc to create pgjdbc-foss is, to
> me, much more DRY than a tiny patch that might be distribution-dependant.

The patches we plan to support in fork are distribution-independent.
Otherwise it would make no sense to concentrate the work.

>      I care a lot about FOSS principles, and that's the main reason if
> we rewrite the interfaces, they will be completely FOSS.

Thanks that you do care here..  foss alternative is acceptable, but see
below it is still not perfect.  The dependancy hell -- you should be able
to build/test/deploy your software without software you actually don't
need.

> > End users would be able to use 'pgdjbc-foss' or 'pgjdbc-foss-osgi' (KISS).
>
>      I think this will make users nuts. There will be three versions of
> the driver:
>
> - One in the official website (pgjdbc), not in Fedora
> - One called pgjdbc-foss, only in Fedora, not mentioned anywhere else
> - One called pgjdbc-foss-osgi, only in Fedora, not mentioned anywhere
> else, which will create even confusion against the non-osgi one

This is all unfortunate, I agree with you.  I would still prefer to have
fixed upstream so it is re-distributable correctly.

On the other hand, that is what using hard-to-package-correctly
licenses/software result in, and anybody should be aware of.

>      Plus the huge burden of maintaining two! forks of pgjdbc.

This is not a problem.  Patching out the 'osgi.enterprise' and 'waffle' is
easy enough to take care of.  And all distros will be sure that it is done
the right way (and if not, somebody needs to fix it or everybody will
loose -- thats why we need to be together).

>      I believe it's more KISS to just re-write the OSGI stuff used in
> pgjdbc, license that PostgreSQL, and you're good to go, with a pure FOSS
> package.

Right, it is the right - foss way.  But I still claim this is useless
effort.  If the osgi was an issue, people would care to have up-2-date
alternatives in GNU/Linux ecosystem already.  So ... repeating myself ...
to make it super perfect, :) users should be able to build the software
without osgi.  You can look how postgresql server is built! (tons of
options available during build, nobody tries to pretend that all the
functionality needs to be built in, _everywhere_ OR _nowhere_).  Even if
you switch to free alternative to osgi.enterprise, 90% of the packagers
will still patch it out .. and there you go -- something is wrong and the
work-around needs to be done on one place.

> >>       We definitely don't know how much or less users use this, but
> >> removing something that works and might be used is what may cause
> >> trouble. Specially, if it is just a few hours of work away to fix the
> >> licensing issue.
> > Oh, I see what you mean :) now -- but those licensing issues should be
> > fixed upstream..
>
>      Right. We can work on fixing them and submitting a PR.

Let's wait for upstream.

> > I still don't get the reason why there can't be single one tarball.  It is
> > a problem, every distro is going to build the package in two-steps.
> > Gentoo guys do build their packages themselves!
>
>      I think this is a matter of taste.

Disagree :( it is a lot of thinking to make the packaging right;  and
nobody convinced me that this is actually necessary.  U just agree that
this issue can be worked around...

>      I believe having a separate
> package for the osgi interfaces that you are "copying" to provide a
> compatibility layer belong to a separate project, and hence two
> tarballs.

I must disagree here.  The osgi as opt-out can't exist, so the osgi in
separate "layer above" pgjdbc (not talking about copying the code) sounds
like natural solution.

> But if that would be a showstopper, it could be integrated
> into pgjdbc itself. But I'd wait for other's (pgjdbc core) opinion here
> before doing that.

That is what we need to do! :)

Pavel

On Wednesday 17 of February 2016 22:47:04 Pavel Raiskup wrote:
> On Wednesday 17 of February 2016 19:41:52 Álvaro Hernández Tortosa wrote:
> >      The only other real alternative is change the code to work around
> > waffle, and after a long time debating I have seen nobody to come up
> > with a fix for this.
>
> Please 's/work-around/make soft-dependency/' and specify that the work
> must be done upstream.

And also note that we can't start to work on patches if we are told that
this is something which is not desired for upstream.  So the fact that
nobody (we in particular :)) come up with fix is due to this.

Pavel

On Wednesday 17 of February 2016 17:13:29 Dave Cramer wrote:
> I think I already made my position clear. The time to chime in on how we
> organized the code is not after we do a major re-organization.

ACK.

> FWIW, I've had conversations with other people in redhat projects that
> consume the driver and they use maven

At least this discussion is with Fedora hat on so if some Red Hat people
use jar from maven, it is not that important.  I'm taking Fedora as
self-standing distribution, which provides Java ; provides PostgreSQL ;
but needs to provide up-2-date 'pgjdbc' (important part of system!).
Sounds like trivial task.

> This is the defacto standard for java dependency resolution. At least
> one other programming language use a similar mechanism. I can think of
> three off the top of my head. Python, go, ruby.

Right, we must support 'maven', 'pip', 'gem', 'npm', ... but it does not
mean that we can make "system package" dependant on software provided by
repository managed by those tools.  We can't.  Our users are free to use
this software, and it is their's responsibility.

> What project in the distribution requires this dependency? Probably JBoss
> as it appears to be the last major product that is still built using ant?

(a) it is not as important as you think - and I thus do not care too much
who is using it within Red Hat.  Within the set of packages I
(co-)maintain [1] there are pieces of system which need to be available on
in native repositories (including pgjdbc).  This is similar for other
distros.
(b) How is this relevant to 'ant' build system?  We can use maven build.

[1] https://admin.fedoraproject.org/pkgdb/packager/praiskup/

Pavel