Thread: Link requirements creep

I was displeased to discover just now that in a standard RPM build of
PG 8.3, psql and the other basic client programs pull in libxml2 and
libxslt; this creates a package dependency that should not be there
by any stretch of the imagination.

The reason of course is that configure puts -lxml2 -lxslt into LIBS
and psql's Makefile unquestioningly links with that list.  Back in
the days of static linking, this didn't hurt because unreferenced
libraries didn't get pulled into the executable.  But it seems that
at least with Linux' linker, you get a reference to every shared
library mentioned in the link command.

One possible answer is to put these two libraries into a special
make variable, comparable to the way that libossp-uuid is being
handled, and use that variable only where wanted.  However this
seems to be a band-aid solution; we'll inevitably have the same
kind of problem again in future.

Another approach we could take is to allow configure to dump
everything into LIBS, and institute a policy that no executable
includes LIBS verbatim.  Instead every link command must do something
like libpq already does:$(filter -lcrypt -ldes -lcom_err -lcrypto -lk5crypto -lkrb5  -lgssapi_krb5 -lgss -lssl -lsocket
-lnsl-lresolv -lintl, $(LIBS))
 
to explicitly list the libraries this executable might need.
This seems a lot more fail-safe, but it'd probably take awhile
to get the filter lists right; and this doesn't seem like a route
to a readily back-patchable solution.

Thoughts, better ideas?
        regards, tom lane

On 5/18/08, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I was displeased to discover just now that in a standard RPM build of
>  PG 8.3, psql and the other basic client programs pull in libxml2 and
>  libxslt; this creates a package dependency that should not be there
>  by any stretch of the imagination.

>  Thoughts, better ideas?

1. Use  -Wl,--as-needed as linker flag.  Portability unknown...  Can be autoconfed.

2. Lets have few top-level library dependencies as make variables:
  LIBPQ_LDFLAGS (LIBPQ_LIBS?)  LIBPQ_CFLAGS
  Same for READLINE, LIBXML2, etc
  Maybe also BACKEND_*

Then each component can pick groups as it needs.

Basically the default autoconf style of putting all into LIBS
work well only when you have single program to build.  As in
Postgres build-tree we have lots of modules that want different
selection of libraries to link to, this style does not fit.

-- 
marko

"Marko Kreen" <markokr@gmail.com> writes:
> On 5/18/08, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I was displeased to discover just now that in a standard RPM build of
>> PG 8.3, psql and the other basic client programs pull in libxml2 and
>> libxslt; this creates a package dependency that should not be there
>> by any stretch of the imagination.

>> Thoughts, better ideas?

> 1. Use  -Wl,--as-needed as linker flag.  Portability unknown...
>    Can be autoconfed.

This might actually be the best solution.  OS X has a similar disease
but some trolling of the ld man page suggests that -dead_strip_dylibs
might work there.  Taking this path would amount to assuming that all
linkers we care about either have an equivalent switch or don't link
unrequired libraries in the first place.

I'll do some experimenting...
        regards, tom lane

Tom Lane wrote:
>
>> 1. Use  -Wl,--as-needed as linker flag.  Portability unknown...
>>    Can be autoconfed.
>>     
>
> This might actually be the best solution.  OS X has a similar disease
> but some trolling of the ld man page suggests that -dead_strip_dylibs
> might work there.  Taking this path would amount to assuming that all
> linkers we care about either have an equivalent switch or don't link
> unrequired libraries in the first place.
>
>
>   

Didn't we run into problems with this before?

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:
>>> 1. Use  -Wl,--as-needed as linker flag.  Portability unknown...

> Didn't we run into problems with this before?

Hm, yeah, thanks for reminding me to check the archives.  We tried to do
exactly this three years ago, and it crashed and burned.  See

http://archives.postgresql.org/pgsql-hackers/2005-10/msg01364.php

The followup discussion to that was along the lines of "it's too late
for 8.1, but we should try it for 8.2".  Seems to have slipped through
the cracks though.

I've been testing a new patch and do not see the problem on Fedora 8,
so at least that platform's readline seems to be fixed.  I find
the hack Martijn proposes in the above message to be pretty ugly,
so what I'm inclined to do is leave that out for now and see what
failures we get.  The availability of the buildfarm makes experimenting
with this kind of thing a lot safer ...
        regards, tom lane

On Sat, 17 May 2008, Tom Lane wrote:

> I was displeased to discover just now that in a standard RPM build of
> PG 8.3, psql and the other basic client programs pull in libxml2 and
> libxslt; this creates a package dependency that should not be there
> by any stretch of the imagination.

When we noticed this recently, my digging suggested you'll be hard pressed 
to have a RedHat system now without those two installed.  The libxslt RPM 
provides necessary components for KDE, GNOME, and Sun's Java RPM. 
libxml2 is far more intertwined even than that.  These dependencies are 
unpleasant technically, but I don't think the introduce any real 
functional creep.  It would be difficult to even strip a system down to 
the point where these packages weren't available.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

Greg Smith <gsmith@gregsmith.com> writes:
> On Sat, 17 May 2008, Tom Lane wrote:
>> I was displeased to discover just now that in a standard RPM build of
>> PG 8.3, psql and the other basic client programs pull in libxml2 and
>> libxslt; this creates a package dependency that should not be there
>> by any stretch of the imagination.

> When we noticed this recently, my digging suggested you'll be hard pressed 
> to have a RedHat system now without those two installed.

Indeed, I've not heard any squawks from the field yet.  It's still
wrong though ...
        regards, tom lane

Tom Lane wrote:
> I've been testing a new patch and do not see the problem on Fedora 8,
> so at least that platform's readline seems to be fixed.  I find
> the hack Martijn proposes in the above message to be pretty ugly,
> so what I'm inclined to do is leave that out for now and see what
> failures we get.  The availability of the buildfarm makes experimenting
> with this kind of thing a lot safer ...
>
>             
>   

It broke my FC6 box :-(

The box is due to be upgraded when I return from pgcon, but we have a 
few oldish boxes on the buildfarm.

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:
> Tom Lane wrote:
>> I've been testing a new patch and do not see the problem on Fedora 8,
>> so at least that platform's readline seems to be fixed.

> It broke my FC6 box :-(

Yeah, I saw.  I'm inclined to wait a day to get a handle on the scope
of the problem before trying to choose an appropriate fix.
        regards, tom lane

On 18/05 00.59, Tom Lane wrote:
> Greg Smith <gsmith@gregsmith.com> writes:
> > On Sat, 17 May 2008, Tom Lane wrote:
> >> I was displeased to discover just now that in a standard RPM build of
> >> PG 8.3, psql and the other basic client programs pull in libxml2 and
> >> libxslt; this creates a package dependency that should not be there
> >> by any stretch of the imagination.
> 
> > When we noticed this recently, my digging suggested you'll be hard pressed 
> > to have a RedHat system now without those two installed.
> 
> Indeed, I've not heard any squawks from the field yet.  It's still
> wrong though ...

I agree it's wrong, but I don't this is likely to be any problem in
practice on Solaris either.

-- 
Bjorn Munch
PostgreSQL Release Engineering
Database Group, Sun Microsystems

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Greg Smith <gsmith@gregsmith.com> writes:
> > When we noticed this recently, my digging suggested you'll be hard pressed
> > to have a RedHat system now without those two installed.
>
> Indeed, I've not heard any squawks from the field yet.  It's still
> wrong though ...

Unsuprisingly, half the world in Debian also depends on libxml2, but I
agree 110% w/ Tom- it's wrong, and I feel it really ought to be fixed
regardless.  It's entirely likely that there will come a time when it's
a less used library getting pulled in, too.  I also personally hate
useless clutter in dependencies as it can cause package management
headaches..

After poking around a bit I did find a box that only pulled in libxml2
for subversion, and we've been talking about moving to a different SCM
(which don't appear to depend on libxml2), so it might eventually only
be pulled in by psql for us.  Not a show-stopper, but it's also not
completely out of the question that it'll get pulled in unnecessairly.
Thanks,
    Stephen

I wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
>> It broke my FC6 box :-(

> Yeah, I saw.  I'm inclined to wait a day to get a handle on the scope
> of the problem before trying to choose an appropriate fix.

So the returns are in, and buildfarm says: this is only broken on
Red Hat-based systems.  How embarrassing :-(

It's evidently fixed in Fedora 7 and up, which means that only EOL'd
Fedora distributions are affected, but since RHEL 5 is affected I'm
still on the hook to fix it.

What I'm inclined to do is move the test for -Wl,--as-needed down till
after we've determined the readline dependent-libraries situation, and
then enable it only if we can still link readline with it on.  So a
system with broken readline won't get the benefit.  But the available
evidence says that there are too few such systems to justify a more
complicated solution.

Comments?
        regards, tom lane

Tom Lane wrote:
> I wrote:
>   
>> Andrew Dunstan <andrew@dunslane.net> writes:
>>     
>>> It broke my FC6 box :-(
>>>       
>
>   
>> Yeah, I saw.  I'm inclined to wait a day to get a handle on the scope
>> of the problem before trying to choose an appropriate fix.
>>     
>
> So the returns are in, and buildfarm says: this is only broken on
> Red Hat-based systems.  How embarrassing :-(
>
> It's evidently fixed in Fedora 7 and up, which means that only EOL'd
> Fedora distributions are affected, but since RHEL 5 is affected I'm
> still on the hook to fix it.
>
> What I'm inclined to do is move the test for -Wl,--as-needed down till
> after we've determined the readline dependent-libraries situation, and
> then enable it only if we can still link readline with it on.  So a
> system with broken readline won't get the benefit.  But the available
> evidence says that there are too few such systems to justify a more
> complicated solution.
>
>
>   

Works for me. I don't think we need anything heroic either.

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:
> Tom Lane wrote:
>> What I'm inclined to do is move the test for -Wl,--as-needed down till
>> after we've determined the readline dependent-libraries situation, and
>> then enable it only if we can still link readline with it on.

> Works for me. I don't think we need anything heroic either.

Applied, we'll see what happens ...
        regards, tom lane