Thread: Use of zlib
Problem: PL/Java use a JVM. On some platforms and with some JVM's (Sun's in particular) a libzip.so is bundled that contains a 1.1.3 version of functions also provided in zlib (why they do this is beyond me, but they do so I'll have to live with it). PostgreSQL is linked with zlib by default. This resuls in a conflict which resuls in a JVM crash. I can circumvein this crash by using LD_PRELOAD to force a load of the JVM bundled libzip.so but I suspect that might result in a crash as soon as PostgreSQL calls on zlib to do something. It's of course possible to configure postgresql with --without-zlib also provided you have accesst o the source. Question:From what I can understand from the documentation, the only utility in PostgreSQL that actually uses zlib is pg_dump? If so, why is the postgres process linked with -lz? Regards, Thomas Hallgren
Am Donnerstag, 23. September 2004 13:02 schrieb Thomas Hallgren: > From what I can understand from the documentation, the only utility in > PostgreSQL that actually uses zlib is pg_dump? If so, why is the > postgres process linked with -lz? Because we are too lazy to fine-tune the build system for cases like this. The best solution would be to build zlib with symbol versioning. -- Peter Eisentraut http://developer.postgresql.org/~petere/
Peter Eisentraut wrote: > Because we are too lazy to fine-tune the build system for cases like > this. > >The best solution would be to build zlib with symbol versioning. > > I'm not so sure. I think zlib is a commodity on most systems. You don't want to build it at all. Perhaps if I submit a patch that takes care of the fine-tuning? - thomas
Peter Eisentraut wrote: >Am Donnerstag, 23. September 2004 13:02 schrieb Thomas Hallgren: > > >> From what I can understand from the documentation, the only utility in >>PostgreSQL that actually uses zlib is pg_dump? If so, why is the >>postgres process linked with -lz? >> >> > >Because we are too lazy to fine-tune the build system for cases like this. >The best solution would be to build zlib with symbol versioning. > > > Or to be less lazy ;-) Is it such a huge task? There have been complaints before about our maximal linking. cheers andrew
Peter Eisentraut <peter_e@gmx.net> writes:
> Am Donnerstag, 23. September 2004 13:02 schrieb Thomas Hallgren:
>> From what I can understand from the documentation, the only utility in
>> PostgreSQL that actually uses zlib is pg_dump? If so, why is the
>> postgres process linked with -lz?
> Because we are too lazy to fine-tune the build system for cases like this.
I was just reading about a linker option (--as-needed or something like
that) that exists in more recent versions of the GNU linker. It means
"only link the libraries actually referenced". Applying this or local
equivalent where available would seem like a fairly painless way to
solve the problem, at least on some platforms.
I agree that trying to keep track of the set of libraries really needed
for each executable would be a hopeless task, but if we can make the
linker fix it for us ...
regards, tom lane
Tom Lane wrote: >Peter Eisentraut <peter_e@gmx.net> writes: > > >>Am Donnerstag, 23. September 2004 13:02 schrieb Thomas Hallgren: >> >> >>>From what I can understand from the documentation, the only utility in >>>PostgreSQL that actually uses zlib is pg_dump? If so, why is the >>>postgres process linked with -lz? >>> >>> > > > >>Because we are too lazy to fine-tune the build system for cases like this. >> >> > >I was just reading about a linker option (--as-needed or something like >that) that exists in more recent versions of the GNU linker. It means >"only link the libraries actually referenced". Applying this or local >equivalent where available would seem like a fairly painless way to >solve the problem, at least on some platforms. > >I agree that trying to keep track of the set of libraries really needed >for each executable would be a hopeless task, but if we can make the >linker fix it for us ... > > Only in version 2.15, which as you say is fairly new. And people who don't use GNU ld won't get it anyway. The docs ( http://sources.redhat.com/binutils/docs-2.15/ld/Options.html#Options ) say: |--as-needed| |--no-as-needed| This option affects ELF DT_NEEDED tags for dynamic libraries mentioned on the command line after the|--as-needed| option. Normally, the linker will add a DT_NEEDED tag for each dynamic library mentioned on the commandline, regardless of whether the library is actually needed. |--as-needed| causes DT_NEEDED tags to only be emittedfor libraries that satisfy some reference from regular objects. |--no-as-needed| restores the default behaviour. I'm not sure I understand why keeping track of what we need for each executable is such a difficult task, though. I count 23 executables and a handful of libraries. Is this such a herculean task? cheers andrew
Andrew Dunstan <andrew@dunslane.net> writes:
> I'm not sure I understand why keeping track of what we need for each
> executable is such a difficult task, though. I count 23 executables and
> a handful of libraries. Is this such a herculean task?
"Handful"? I count 32 AC_CHECK_LIB and AC_SEARCH_LIBS calls in
configure.in.
But that's only part of the problem. Consider that many of these
libraries are interdependent to some extent. The raw results from
Autoconf tell you that libfoo is available and we could link with it,
but not whether libfoo requires the previously-checked libbar to link.
We'd need to redo the whole library detection process for each
executable in order to be perfectly accurate about this. Since the
Autoconf macros only cope with one LIBS list, that means throwing away
the Autoconf macros and writing our own.
Multiply that by N supported platforms, and you start to see why this is
a mess. It's certainly work far out of proportion to the actual value
of not linking some unneeded libraries.
Adding "--as-needed" on platforms that support it seems like a
reasonable amount of work to expend in comparison to the value received.
Trying to do it manually does not.
regards, tom lane
Thomas Hallgren wrote: > Problem: > PL/Java use a JVM. On some platforms and with some JVM's (Sun's in > particular) a libzip.so is bundled that contains a 1.1.3 version of > functions also provided in zlib (why they do this is beyond me, but > they do so I'll have to live with it). PostgreSQL is linked with zlib > by default. This resuls in a conflict which resuls in a JVM crash. > > I can circumvein this crash by using LD_PRELOAD to force a load of the > JVM bundled libzip.so but I suspect that might result in a crash as > soon as PostgreSQL calls on zlib to do something. It's of course > possible to configure postgresql with --without-zlib also provided you > have accesst o the source. > > Question: > From what I can understand from the documentation, the only utility in > PostgreSQL that actually uses zlib is pg_dump? If so, why is the > postgres process linked with -lz? > > I did a small experiment by installing binutils 2.15 and adding -Wl,--as-needed to the LDFLAGS, as Tom had suggested might be useful. This seemed to work quite well and trimmed back the needed libs quite a bit. However, when you configure --with-openssl, libz is again linked in. Not sure where that leaves us. cheers andrew
Andrew Dunstan wrote: > > > Thomas Hallgren wrote: > >> Problem: >> PL/Java use a JVM. On some platforms and with some JVM's (Sun's in >> particular) a libzip.so is bundled that contains a 1.1.3 version of >> functions also provided in zlib (why they do this is beyond me, but >> they do so I'll have to live with it). PostgreSQL is linked with zlib >> by default. This resuls in a conflict which resuls in a JVM crash. >> >> I can circumvein this crash by using LD_PRELOAD to force a load of >> the JVM bundled libzip.so but I suspect that might result in a crash >> as soon as PostgreSQL calls on zlib to do something. It's of course >> possible to configure postgresql with --without-zlib also provided >> you have accesst o the source. >> >> Question: >> From what I can understand from the documentation, the only utility >> in PostgreSQL that actually uses zlib is pg_dump? If so, why is the >> postgres process linked with -lz? >> >> > > I did a small experiment by installing binutils 2.15 and adding > -Wl,--as-needed to the LDFLAGS, as Tom had suggested might be useful. > > This seemed to work quite well and trimmed back the needed libs quite > a bit. However, when you configure --with-openssl, libz is again > linked in. From PL/Javas standpoint, I think it would be great if this change could be incorporated into the 8.0 release. The openssl issue is something I'll have to investigate. Perhaps it works using the libzip from the JRE if I use LD_PRELOAD. I'm happy to see that Sun has an open bug on the subject in their Bug Database (bug 4953050 if anyone is interested). Regards, Thomas Hallgren
I am confused. I just checked my backend binary and I don't see any unusual libs required: $ ldd postgres libz.so => /usr/lib/libz.so (0x2833f000) libncurses.so.5 => /shlib/libncurses.so.5 (0x2834e000) libdl.so => /shlib/libdl.so (0x2838c000) libm.so => /shlib/libm.so.0.0 (0x2838f000) libgcc.so.1=> /shlib/libgcc.so.1 (0x283a0000) libc.so.2 => /shlib/libc.so.2 (0x283ac000) The libncurses is a little unusual but I don't see libreadline in there even though it is on the link line. Are some OS's adding libs that aren't required by the binary? --------------------------------------------------------------------------- Andrew Dunstan wrote: > > > Thomas Hallgren wrote: > > > Problem: > > PL/Java use a JVM. On some platforms and with some JVM's (Sun's in > > particular) a libzip.so is bundled that contains a 1.1.3 version of > > functions also provided in zlib (why they do this is beyond me, but > > they do so I'll have to live with it). PostgreSQL is linked with zlib > > by default. This resuls in a conflict which resuls in a JVM crash. > > > > I can circumvein this crash by using LD_PRELOAD to force a load of the > > JVM bundled libzip.so but I suspect that might result in a crash as > > soon as PostgreSQL calls on zlib to do something. It's of course > > possible to configure postgresql with --without-zlib also provided you > > have accesst o the source. > > > > Question: > > From what I can understand from the documentation, the only utility in > > PostgreSQL that actually uses zlib is pg_dump? If so, why is the > > postgres process linked with -lz? > > > > > > I did a small experiment by installing binutils 2.15 and adding > -Wl,--as-needed to the LDFLAGS, as Tom had suggested might be useful. > > This seemed to work quite well and trimmed back the needed libs quite a > bit. However, when you configure --with-openssl, libz is again linked in. > > Not sure where that leaves us. > > cheers > > andrew > > ---------------------------(end of broadcast)--------------------------- > TIP 2: you can get off all lists at once with the unregister command > (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Sat, 25 Sep 2004, Bruce Momjian wrote: > I am confused. I just checked my backend binary and I don't see any > unusual libs required: > > $ ldd postgres > libz.so => /usr/lib/libz.so (0x2833f000) > libncurses.so.5 => /shlib/libncurses.so.5 (0x2834e000) > libdl.so => /shlib/libdl.so (0x2838c000) > libm.so => /shlib/libm.so.0.0 (0x2838f000) > libgcc.so.1 => /shlib/libgcc.so.1 (0x283a0000) > libc.so.2 => /shlib/libc.so.2 (0x283ac000) > > The libncurses is a little unusual but I don't see libreadline in there > even though it is on the link line. > > Are some OS's adding libs that aren't required by the binary? This is the output in Fedora Core 2, and it includes readline: $ ldd /usr/bin/postgres linux-gate.so.1 => (0x00240000) libpam.so.0 => /lib/libpam.so.0 (0x00cf1000) libssl.so.4=> /lib/libssl.so.4 (0x0014e000) libcrypto.so.4 => /lib/libcrypto.so.4 (0x00241000) libkrb5.so.3 =>/usr/lib/libkrb5.so.3 (0x0052a000) libcom_err.so.2 => /lib/libcom_err.so.2 (0x00e56000) libz.so.1 => /usr/lib/libz.so.1(0x00aca000) libreadline.so.4 => /usr/lib/libreadline.so.4 (0x00111000) libtermcap.so.2 =>/lib/libtermcap.so.2 (0x00f25000) libcrypt.so.1 => /lib/libcrypt.so.1 (0x00406000) libresolv.so.2 => /lib/libresolv.so.2(0x00f8a000) libnsl.so.1 => /lib/libnsl.so.1 (0x00215000) libdl.so.2 => /lib/libdl.so.2 (0x00908000) libm.so.6 => /lib/tls/libm.so.6 (0x008a5000) libc.so.6 => /lib/tls/libc.so.6 (0x00592000) libgssapi_krb5.so.2=> /usr/lib/libgssapi_krb5.so.2 (0x00a57000) libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x00ccd000) /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x004ca000) -- /Dennis Björklund
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I am confused. I just checked my backend binary and I don't see any
> unusual libs required:
> $ ldd postgres
> libz.so => /usr/lib/libz.so (0x2833f000)
> libncurses.so.5 => /shlib/libncurses.so.5 (0x2834e000)
I do not believe either of those are actually *used* by the backend.
regards, tom lane
Dennis Bjorklund wrote: >This is the output in Fedora Core 2, and it includes readline: > >$ ldd /usr/bin/postgres > linux-gate.so.1 => (0x00240000) > libpam.so.0 => /lib/libpam.so.0 (0x00cf1000) > libssl.so.4 => /lib/libssl.so.4 (0x0014e000) > libcrypto.so.4 => /lib/libcrypto.so.4 (0x00241000) > libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x0052a000) > libcom_err.so.2 => /lib/libcom_err.so.2 (0x00e56000) > libz.so.1 => /usr/lib/libz.so.1 (0x00aca000) > libreadline.so.4 => /usr/lib/libreadline.so.4 (0x00111000) > libtermcap.so.2 => /lib/libtermcap.so.2 (0x00f25000) > libcrypt.so.1 => /lib/libcrypt.so.1 (0x00406000) > libresolv.so.2 => /lib/libresolv.so.2 (0x00f8a000) > libnsl.so.1 => /lib/libnsl.so.1 (0x00215000) > libdl.so.2 => /lib/libdl.so.2 (0x00908000) > libm.so.6 => /lib/tls/libm.so.6 (0x008a5000) > libc.so.6 => /lib/tls/libc.so.6 (0x00592000) > libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0x00a57000) > libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x00ccd000) > /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x004ca000) > > > > Now, to see the things postgres really needs, as opposed to things other libraries need or that we just throw in for good measure in case somebody needs them, run this: LDFLAGS=-Wl,--as-necessary ./configure without any third-party options (like pam, openssl, kerberos ...), Alternatively, you can set the flag in src/Makefile.global This should work on FC2, which I believe has binutils 2.15. Almost every line above will disappear. cheers andrew
of course, I meant --as-needed rather than --as-necessary Andrew Dunstan wrote: > > > Dennis Bjorklund wrote: > >> This is the output in Fedora Core 2, and it includes readline: >> >> $ ldd /usr/bin/postgres >> linux-gate.so.1 => (0x00240000) >> libpam.so.0 => /lib/libpam.so.0 (0x00cf1000) >> libssl.so.4 => /lib/libssl.so.4 (0x0014e000) >> libcrypto.so.4 => /lib/libcrypto.so.4 (0x00241000) >> libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x0052a000) >> libcom_err.so.2 => /lib/libcom_err.so.2 (0x00e56000) >> libz.so.1 => /usr/lib/libz.so.1 (0x00aca000) >> libreadline.so.4 => /usr/lib/libreadline.so.4 (0x00111000) >> libtermcap.so.2 => /lib/libtermcap.so.2 (0x00f25000) >> libcrypt.so.1 => /lib/libcrypt.so.1 (0x00406000) >> libresolv.so.2 => /lib/libresolv.so.2 (0x00f8a000) >> libnsl.so.1 => /lib/libnsl.so.1 (0x00215000) >> libdl.so.2 => /lib/libdl.so.2 (0x00908000) >> libm.so.6 => /lib/tls/libm.so.6 (0x008a5000) >> libc.so.6 => /lib/tls/libc.so.6 (0x00592000) >> libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0x00a57000) >> libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x00ccd000) >> /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x004ca000) >> >> >> >> > > Now, to see the things postgres really needs, as opposed to things > other libraries need or that we just throw in for good measure in case > somebody needs them, run this: > > LDFLAGS=-Wl,--as-necessary ./configure > > without any third-party options (like pam, openssl, kerberos ...), > Alternatively, you can set the flag in src/Makefile.global > > This should work on FC2, which I believe has binutils 2.15. > > Almost every line above will disappear. > > cheers > > andrew >