Thread: NetBSD vs libxml2

In a moment of idle curiosity, I tried to build PG --with-libxml
on NetBSD-current (well, mostly current, from May or so).
The configure script blew up, complaining that it couldn't execute
a test program.  Investigation showed that xml2-config reports this:

$ xml2-config --libs
-Wl,-R/usr/pkg/lib -L/usr/pkg/lib -lxml2 -L/usr/lib -lz -L/usr/lib -llzma -L/usr/lib -lm

and we're only paying attention to the -L switches out of that.
So we successfully link to /usr/pkg/lib/libxml2.so, but then
execution fails for lack of rpath pointing at /usr/pkg/lib.

We could fix this by teaching configure to absorb -Wl,-R... switches
into LDFLAGS from xml2-config's output, and that seems to make things
work, but I wonder whether we should or not.  This seems like a new height
of unfriendliness to non-default packages on NetBSD's part, and it's a bit
hard to believe the behavior will make it to a formal release.  I don't
see any comparable behavior on FreeBSD for instance --- it puts packages'
libraries into /usr/local/lib, but that seems to be searched automatically
without additional switches beyond -L.  Don't have an easy way to check
things on OpenBSD.

Thoughts?

            regards, tom lane

On Sat, Aug 11, 2018 at 01:18:26PM -0400, Tom Lane wrote:
> In a moment of idle curiosity, I tried to build PG --with-libxml
> on NetBSD-current (well, mostly current, from May or so).
> The configure script blew up, complaining that it couldn't execute
> a test program.  Investigation showed that xml2-config reports this:
> 
> $ xml2-config --libs
> -Wl,-R/usr/pkg/lib -L/usr/pkg/lib -lxml2 -L/usr/lib -lz -L/usr/lib -llzma -L/usr/lib -lm
> 
> and we're only paying attention to the -L switches out of that.
> So we successfully link to /usr/pkg/lib/libxml2.so, but then
> execution fails for lack of rpath pointing at /usr/pkg/lib.
> 
> We could fix this by teaching configure to absorb -Wl,-R... switches
> into LDFLAGS from xml2-config's output, and that seems to make things
> work, but I wonder whether we should or not.  This seems like a new height
> of unfriendliness to non-default packages on NetBSD's part, and it's a bit
> hard to believe the behavior will make it to a formal release.  I don't
> see any comparable behavior on FreeBSD for instance --- it puts packages'
> libraries into /usr/local/lib, but that seems to be searched automatically
> without additional switches beyond -L.  Don't have an easy way to check
> things on OpenBSD.
> 
> Thoughts?

-Wl,-R (and friends, like -Wl,-rpath) is part and parcel of dynamic
linking, and are precious.  All software should honor these.

That non-default packages don't end up in /usr is a perfectly legitimate
thing for a distribution to do.

More importantly, a site-build that uses a non-system location for e.g.
locally-patched open source packages, is a perfectly legitimate thing
for _users_ to do.  It isn't nice to force them to hack a project's
./configure file or work out precious envvar settings to make that
project support non-system locations for dependencies.

I guess the problem here is that xml2-config is (like so many *-config
programs) broken by not having a way to get ld flags and libs
separately...  Which would be why ./configure is parsing the output of
xml2-config --libs.

The better thing to do would be to take all the words from xml2-config
--libs that match -l* and put them in LIBS while all others go into
LDFLAGS.  It's safer to assume that -l* means "link this in" than that
there won't be new linker options other than -L or -l.

I'll note too that -lxml2 is hardcoded in ./configure.in.  That's not
right either.

IMO this is just a minor bug in PG.

I'm willing to write a patch after lunch.  In ./configure.in this:

    for pgac_option in `$XML2_CONFIG --libs`; do
      case $pgac_option in
        -L*) LDFLAGS="$LDFLAGS $pgac_option";;
      esac
    done

should change to:

    for pgac_option in `$XML2_CONFIG --libs`; do
      case $pgac_option in
        -l*) LDLIBS="$LDLIBS $pgac_option";;
        *) LDFLAGS="$LDFLAGS $pgac_option";;
      esac
    done

More changes are needed to stop hard-coding -lxml2.

I can write a patch if you like.

Nico
-- 

On Sat, Aug 11, 2018 at 01:12:09PM -0500, Nico Williams wrote:
> I'm willing to write a patch after lunch.  In ./configure.in this:
> 
>     for pgac_option in `$XML2_CONFIG --libs`; do
>       case $pgac_option in
>         -L*) LDFLAGS="$LDFLAGS $pgac_option";;
>       esac
>     done
> 
> should change to:
> 
>     for pgac_option in `$XML2_CONFIG --libs`; do
>       case $pgac_option in
>         -l*) LDLIBS="$LDLIBS $pgac_option";;
>         *) LDFLAGS="$LDFLAGS $pgac_option";;
>       esac
>     done
> 
> More changes are needed to stop hard-coding -lxml2.

Actually, no, no more changes are needed.  The -lxml2 comes from:

1193 if test "$with_libxml" = yes ; then
1194   AC_CHECK_LIB(xml2, xmlSaveToBuffer, [], [AC_MSG_ERROR([library 'xml2' (version >= 2.6.23) is required for XML
support])])
1195 fi

in configure.in, and I think contrib/xml2/Makefile needs to hardcode it.

So the above quoted change is all that is needed, plus re-run autoconf.

See attached.  (I'm not including unrelated changes made to configure by
autoconf.)

Nico
--

On 08/11/2018 01:18 PM, Tom Lane wrote:
> In a moment of idle curiosity, I tried to build PG --with-libxml
> on NetBSD-current (well, mostly current, from May or so).
> The configure script blew up, complaining that it couldn't execute
> a test program.  Investigation showed that xml2-config reports this:
>
> $ xml2-config --libs
> -Wl,-R/usr/pkg/lib -L/usr/pkg/lib -lxml2 -L/usr/lib -lz -L/usr/lib -llzma -L/usr/lib -lm
>
> and we're only paying attention to the -L switches out of that.
> So we successfully link to /usr/pkg/lib/libxml2.so, but then
> execution fails for lack of rpath pointing at /usr/pkg/lib.
>
> We could fix this by teaching configure to absorb -Wl,-R... switches
> into LDFLAGS from xml2-config's output, and that seems to make things
> work, but I wonder whether we should or not.  This seems like a new height
> of unfriendliness to non-default packages on NetBSD's part, and it's a bit
> hard to believe the behavior will make it to a formal release.  I don't
> see any comparable behavior on FreeBSD for instance --- it puts packages'
> libraries into /usr/local/lib, but that seems to be searched automatically
> without additional switches beyond -L.  Don't have an easy way to check
> things on OpenBSD.
>



OpenBSD-6.3's "xml2-config --libs" doesn't contain any rpath settings, 
it's pretty much like the one on my (fairly old) FBSD machine, as you 
describ4ed above.


cheers

andrew

-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 2018-08-12 01:32, Andrew Dunstan wrote:

> OpenBSD-6.3's "xml2-config --libs" doesn't contain any rpath settings, 
> it's pretty much like the one on my (fairly old) FBSD machine, as you 
> describ4ed above.

Just FYI!

My bf-animal curculio that runs OpenBSD 5.9 shows the following:

$ xml2-config --libs
-L/usr/local/lib -lxml2 -lz -L/usr/local/lib -liconv -lm

/Mikael

On Sat, Aug 11, 2018 at 1:18 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> We could fix this by teaching configure to absorb -Wl,-R... switches
> into LDFLAGS from xml2-config's output, and that seems to make things
> work, but I wonder whether we should or not.  This seems like a new height
> of unfriendliness to non-default packages on NetBSD's part, and it's a bit
> hard to believe the behavior will make it to a formal release.

I kind of agree with Nico: why do we think we get to tell operating
system distributions which switches they're allowed to need to make
things work?  The point of things like pg_config and xmlconfig is to
reveal what is needed.  If we editorialize on that, we do so at our
own risk.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas@gmail.com> writes:
> On Sat, Aug 11, 2018 at 1:18 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> We could fix this by teaching configure to absorb -Wl,-R... switches
>> into LDFLAGS from xml2-config's output, and that seems to make things
>> work, but I wonder whether we should or not.  This seems like a new height
>> of unfriendliness to non-default packages on NetBSD's part, and it's a bit
>> hard to believe the behavior will make it to a formal release.

> I kind of agree with Nico: why do we think we get to tell operating
> system distributions which switches they're allowed to need to make
> things work?  The point of things like pg_config and xmlconfig is to
> reveal what is needed.  If we editorialize on that, we do so at our
> own risk.

Well, the issue is that new kinds of switches introduce new potential
for bugs.  In the case of -Wl,-R..., I'm not even sure that you can
write that more than once per link, so absorbing one from xml2-config
might well break things on some platforms.  Or, if you can write more
than one, we'd need to make sure they end up in a desirable order.
(cf commit dddfc4cb2 which fixed similar issues for -L switches;
it looks to me like the current coding would in fact fail to order
our $(rpath) correctly against one absorbed from xml2, and it would
not be trivial to fix that.)

I'm not, personally, eager to do that work for a requirement which
somehow hasn't surfaced on any other platform, nor on any previous
NetBSD release.  I think NetBSD is way out in left field here.

            regards, tom lane

[Quoting out of order.]

On Mon, Aug 13, 2018 at 11:16:23AM -0400, Tom Lane wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>
> I'm not, personally, eager to do that work for a requirement which
> somehow hasn't surfaced on any other platform, nor on any previous
> NetBSD release.  I think NetBSD is way out in left field here.

It's not just about the distros.  It's also about sites that build and
install alternate versions of things that PG might depend on.  I've seen
that many times.  It is PG that is being hostile to them, not NetBSD
that is being hostile to PG.

Building PG outside a distro, with libxml2 outside a distro, ought to be
possible, but currently isn't without hacking on PG.  That's not good.

In any case, the patch I posted is trivial and small and should do the
job.

> > I kind of agree with Nico: why do we think we get to tell operating
> > system distributions which switches they're allowed to need to make
> > things work?  The point of things like pg_config and xmlconfig is to
> > reveal what is needed.  If we editorialize on that, we do so at our
> > own risk.
> 
> Well, the issue is that new kinds of switches introduce new potential
> for bugs.  In the case of -Wl,-R..., I'm not even sure that you can
> write that more than once per link, so absorbing one from xml2-config
> might well break things on some platforms.  Or, if you can write more
> than one, we'd need to make sure they end up in a desirable order.
> (cf commit dddfc4cb2 which fixed similar issues for -L switches;
> it looks to me like the current coding would in fact fail to order
> our $(rpath) correctly against one absorbed from xml2, and it would
> not be trivial to fix that.)

I believe no new options have _ever_ been added to linker-editors for
specifying dependencies, but lots of options for other purposes have
been added over the last several decades.

The link-editors are free to add new linker options.  There is nothing
we can do about that.

The fundamental problem here is that *-config programs should have had
separate --libs and --ldflags options, but don't, obligating us (and
others too) to parse the output of the --libs option.

What we need is an approach to parsing the output of the --libs option
that is likely to be more stable.  I believe looking for words that
start with -l is better than looking for words that start with -L,
because there haven't been new ways to specify dependencies ever, but
there have been tons of new linker-editor options for other things.  It
follows that this approach should be more stable.

And yes, I understand that if the linker-editors ever add new options
for specifying dependencies then we'll be here again.  It just seems a)
unlikely, b) not that burdensome if it does ever happen.

Aside:  It is truly shocking that in a world where, for better or worse,
        autoconf is so prevalent, *-config programs still fail to
        separate --libs and --ldflags.  After all, the separation of
        LIBS (not precious) and LDFLAGS (precious), is very much a
        tradition in autoconf'ed projects, and autoconf very much
        supports it.

Nico
-- 

On Mon, Aug 13, 2018 at 11:16:23AM -0400, Tom Lane wrote:
> Well, the issue is that new kinds of switches introduce new potential
> for bugs.  In the case of -Wl,-R..., I'm not even sure that you can
> write that more than once per link, so absorbing one from xml2-config
> might well break things on some platforms.  Or, if you can write more
> than one, we'd need to make sure they end up in a desirable order.

I forgot to address this.  You're absolutely right that -Wl,-R options
could break things.  E.g., if you have dependency -lA in one location
and -lB in another, but the second location also has a libA.so that
doesn't match the one used at link-edit time.

The problem here is that ELF, the linker-editor, and the RTLD, don't
really have a way of keeping things together that they should keep
together.

In practice this usually doesn't cause serious problems that users
cannot workaround...  For NetBSD in particular this should not cause
problems because NetBSD (presumably!) would not put a libA.so in
/usr/lib and in their ports.

Question: why even separate -lxml2 and the -L and -Wl,-R options into
LIBS and LDFLAGS?  Why not put all of them into LIBS and not LDFLAGS?
What would break (aside from the runpath confusion issue)?

After all, why should a user's use of LDFLAGS on the ./configure step
all over xml2-config's --libs?

With ./configure just doing:

  LIBS="$LIBS `$XML2_CONFIG --libs`"

and a libxml2 built and installed into /tmp/xpg/, with xml2-config
modified to include -Wl,-rpath=/tmp/xpg, this works correctly:

$ ldd ./src/backend/postgres|grep xml2
        libxml2.so.2 => /tmp/xpg/lib/libxml2.so.2 (0x00007fc9ebfd1000)
$ readelf -a ./src/backend/postgres|grep xpg
 0x000000000000001d (RUNPATH)            Library runpath: [/opt/pg/lib:/tmp/xpg/lib]
$ 

And while we're here, why is configure parsing the output of xml2-config
--cflags??  That too seems to assume that there will never be any C
compiler options to absorb other than -I or -D.  That at least seems
like a safer assumption.

The -l/-L/-R mess is... a mess.  It's a shortcoming of ELF, and of the
linker-editors, and the run-time linker-loaders.

Solaris/Illumos has something called "direct linking", whereby ELF
objects record for each symbol which dependency should provide it, and
this speeds up run-time linking and loading significantly.  But that
seems insufficiently advanced: ELF should also record for each
dependency where to look for it, not just a one-size-fits-all rpath.
And the CLI for the linker-editor should reflect such a state of affairs
somewhow (e.g., ld ... -lxml2:L/opt/foo/lib:R/opt/foo/lib).  The linker-
editor projects move awfully slow.

Nico
--