Thread: static libpq (and other libraries) overwritten on aix
Hi,
I was hacking in making aix work with the meson patchset last night when I
noticed this delightful bit:
gmake -C src/interfaces/libpq
...
rm -f libpq.a
ar crs libpq.a fe-auth-scram.o fe-connect.o fe-exec.o fe-lobj.o fe-misc.o fe-print.o fe-protocol3.o fe-secure.o
fe-trace.olegacy-pqsignal.o libpq-events.o pqexpbuffer.o fe-auth.o
touch libpq.a
( echo '#! libpq.so.5'; gawk '/^[^#]/ {printf "%s\n",$1}'
/home/andres/src/postgres/build-ac/../src/interfaces/libpq/exports.txt) >libpq.exp
gcc -maix64 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels
-Wmissing-format-attribute-Wimplicit-fallthrough=3 -Wformat-security -fno-strict-aliasing -fwrapv
-fexcess-precision=standard-Wno-format-truncation -O2 -pthread -D_REENTRANT -D_THREAD_SAFE -o libpq.so.5 libpq.a
-Wl,-bE:libpq.exp-L../../../src/port -L../../../src/common -lpgcommon_shlib -lpgport_shlib -Wl,-bbigtoc
-Wl,-blibpath:'/usr/local/pgsql/lib:/usr/lib:/lib' -Wl,-bnoentry -Wl,-H512 -Wl,-bM:SRE -lm
rm -f libpq.a
ar crs libpq.a libpq.so.5
we first create a static library libpq.a as normal, but then we overwrite it
with the special aix way of packing up shared libraries, by packing them up in
a static library. That part is correct, it's apparently the easiest way of
getting applications to link to shared libraries on AIX (I think the
-Wl,-bM:SRE is relevant for ensuring it'll be a dynamic link, rather than a
static one).
This likely has been going on for approximately forever.
Two questions:
1) Do we continue building static libraries for libpq etc?
2) Do we care about static libraries not suriving on AIX? There could also be
a race in the buildrules leading to sometimes static libs sometimes shared
libs winning, I think.
Greetings,
Andres Freund
On Wed, Aug 17, 2022 at 3:02 PM Andres Freund <andres@anarazel.de> wrote: > 2) Do we care about static libraries not suriving on AIX? There could also be > a race in the buildrules leading to sometimes static libs sometimes shared > libs winning, I think. Instead of overwriting the same file, can we not use different filenames for different things? -- Robert Haas EDB: http://www.enterprisedb.com
Hi,
On 2022-08-17 15:28:18 -0400, Robert Haas wrote:
> On Wed, Aug 17, 2022 at 3:02 PM Andres Freund <andres@anarazel.de> wrote:
> > 2) Do we care about static libraries not suriving on AIX? There could also be
> > a race in the buildrules leading to sometimes static libs sometimes shared
> > libs winning, I think.
>
> Instead of overwriting the same file, can we not use different
> filenames for different things?
Not easily, as far as I understand. The way one customarily links to shared
libraries on aix is to have an .a archive containing the shared library. That
way the -lpq picks up libpq.a, which then triggers the shared library to be
referenced.
E.g.
andres@gcc119:[/home/andres/src/postgres/build-ac]$ LIBPATH=$(pwd)/src/interfaces/libpq ldd src/bin/scripts/clusterdb
src/bin/scripts/clusterdb needs:
/usr/lib/libc.a(shr_64.o)
/usr/lib/libpthread.a(shr_xpg5_64.o)
/usr/lib/libreadline.a(libreadline.so.6)
/home/andres/src/postgres/build-ac/src/interfaces/libpq/libpq.a(libpq.so.5)
/unix
/usr/lib/libcrypt.a(shr_64.o)
/usr/lib/libcurses.a(shr42_64.o)
/usr/lib/libpthreads.a(shr_xpg5_64.o)
Note the .a(libpq.so.5) bit.
Unfortunately that's exactly how one links to a static library as well.
So we'd have to change the name used as -l$this between linking to a shared
libpq and a static libpq.
Greetings,
Andres Freund
The AIX sections of Makefile.shlib misuse the terms "static" and "shared". Imagine s/static library/library ending in .a/ and s/shared library/library ending in .so/. That yields an accurate description of the makefile rules. On Wed, Aug 17, 2022 at 12:01:54PM -0700, Andres Freund wrote: > Two questions: > 1) Do we continue building static libraries for libpq etc? Essentially, we don't build static libpq today, and we should continue not building it. (The first-built libpq.a is static, but that file is an implementation detail of the makefile rules. The surviving libpq.a is a normal AIX shared library.) > 2) Do we care about static libraries not suriving on AIX? No. > There could also be > a race in the buildrules leading to sometimes static libs sometimes shared > libs winning, I think. Not since commit e8564ef, to my knowledge. Along the lines of Robert's comment, it could be a nice code beautification to use a different suffix for the short-lived .a file. Perhaps _so_inputs.a. I found this useful years ago: https://web.archive.org/web/20151003130212/http://seriousbirder.com/blogs/aix-shared-and-static-libraries-explained/
On Thu, Aug 18, 2022 at 12:59 AM Noah Misch <noah@leadboat.com> wrote: > Along the lines of Robert's comment, it could be a nice code beautification to > use a different suffix for the short-lived .a file. Perhaps _so_inputs.a. Yeah, this is the kind of thing I was thinking about. -- Robert Haas EDB: http://www.enterprisedb.com
On 2022-08-18 Th 10:10, Robert Haas wrote: > On Thu, Aug 18, 2022 at 12:59 AM Noah Misch <noah@leadboat.com> wrote: >> Along the lines of Robert's comment, it could be a nice code beautification to >> use a different suffix for the short-lived .a file. Perhaps _so_inputs.a. > Yeah, this is the kind of thing I was thinking about. +1 for that and clarifying Makefile.shlib. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
Hi, On 2022-08-17 21:59:29 -0700, Noah Misch wrote: > The AIX sections of Makefile.shlib misuse the terms "static" and "shared". > > Imagine s/static library/library ending in .a/ and s/shared library/library > ending in .so/. That yields an accurate description of the makefile rules. I realize that aspect. My point is that we currently, across most of the other platforms, support building a "proper" static library, and install it too. But on AIX (and I think mingw), we don't, but without an explicit comment about not doing so. In fact, the all-static-lib target on those platforms will build a non-static library, which seems not great. > > There could also be > > a race in the buildrules leading to sometimes static libs sometimes shared > > libs winning, I think. > > Not since commit e8564ef, to my knowledge. I'd missed that the $(stlib): ... bit is not defined due to haslibarule being defined... > Along the lines of Robert's comment, it could be a nice code beautification to > use a different suffix for the short-lived .a file. Perhaps _so_inputs.a. Agreed, it'd be an improvement. Afaict we could just stop building the intermediary static lib. Afaict the MKLDEXPORT path isn't needed for libraries without an exports.txt because the linker defaults to exporting "most" symbols, and for symbols with an exports.txt we don't need it either. The only path that really needs MKLDEXPORT is postgres. Not really for the export side either, but for the import side. Greetings, Andres Freund
On Thu, Aug 18, 2022 at 09:03:57AM -0700, Andres Freund wrote: > My point is that we currently, across most of the other platforms, support > building a "proper" static library, and install it too. But on AIX (and I > think mingw), we don't, but without an explicit comment about not doing so. In > fact, the all-static-lib target on those platforms will build a non-static > library, which seems not great. Yep. If someone had just pushed a correct patch to make AIX match our GNU/Linux static linking assistance, I wouldn't be arguing to revert that patch. At the same time, if someone asks me to choose high-value projects for 20 people, doing more for static linking on AIX won't be on the list. > On 2022-08-17 21:59:29 -0700, Noah Misch wrote: > > Along the lines of Robert's comment, it could be a nice code beautification to > > use a different suffix for the short-lived .a file. Perhaps _so_inputs.a. > > Agreed, it'd be an improvement. > > Afaict we could just stop building the intermediary static lib. Afaict the > MKLDEXPORT path isn't needed for libraries without an exports.txt because the > linker defaults to exporting "most" symbols If that works, great.
Hi,
On 2022-08-18 22:56:43 -0700, Noah Misch wrote:
> > On 2022-08-17 21:59:29 -0700, Noah Misch wrote:
> > > Along the lines of Robert's comment, it could be a nice code beautification to
> > > use a different suffix for the short-lived .a file. Perhaps _so_inputs.a.
> >
> > Agreed, it'd be an improvement.
> >
> > Afaict we could just stop building the intermediary static lib. Afaict the
> > MKLDEXPORT path isn't needed for libraries without an exports.txt because the
> > linker defaults to exporting "most" symbols
>
> If that works, great.
I looked at that. It's not too hard to make it work. But while doing so I
encountered some funny bits.
As far as I can tell the way we build shared libraries on aix with gcc isn't
correct:
Without -shared gcc won't know that it's building a shared library, which
afaict will prevent gcc from generating correct unwind info and we end up with
a statically linked copy of libgcc each time.
The naive thing of just adding -shared fails, but that's our fault:
ldd pgoutput.so
pgoutput.so needs:
Cannot find libgcc_s.a(shr.o)
/usr/lib/libc.a(shr_64.o)
/unix
/usr/lib/libcrypt.a(shr_64.o)
Makefile.aix has:
# -blibpath must contain ALL directories where we should look for libraries
libpath := $(shell echo $(subst -L,:,$(filter -L/%,$(LDFLAGS))) | sed -e's/ //g'):/usr/lib:/lib
but that's insufficient for gcc, because it won't find gcc's runtime lib. We
could force a build of the statically linked libgcc, but once it knows it's
generating with a shared library, a static libgcc unfortunately blows up the
size of the output considerably.
So I think we need something like
ifeq ($(GCC), yes)
libpath := $(libpath):$(dir $(shell gcc -print-libgcc-file-name))
endif
although deferring the computation of that would be nicer, but would require
some cleanup before.
With that libraries do shrink a bit. E.g. cube.so goes from 140k to 96k.
Afaict there's no reason to generate lib<name>.a for extension .so's, right?
We have plenty of detritus that's vaguely AIX related. The common.mk rule to
generate SUBSYS.o isn't used (mea culpa), and backend/Makefile's postgres.o
rule hasn't been used for well over 20 years.
I'll send in a patch series tomorrow, too tired for today.
Greetings,
Andres Freund
On Sat, Aug 20, 2022 at 01:35:22AM -0700, Andres Freund wrote: > Afaict there's no reason to generate lib<name>.a for extension .so's, right? Right. We install cube.so, not any *cube.a file.
Hi, On 2022-08-20 01:35:22 -0700, Andres Freund wrote: > I'll send in a patch series tomorrow, too tired for today. Here it goes. 0001 aix: Fix SHLIB_EXPORTS reference in VPATH builds That's mostly so I could even build. It's not quite right in the sense that we don't depend on the file, but that's a preexisting issue. Could be folded in with 0005, which fixes that aspect. Or it could be backpatched as the minimal fix. 0002 Remove SUBSYS.o rule in common.mk, hasn't been used in a long time 0003 Remove rule to generate postgres.o, not needed for 20+ years Both obvious, I think. 0004 aix: when building with gcc, tell gcc we're building a shared library That's the gcc -shared issue I explained in the email I'm replying to. We should probably consider building executables with -shared-libgcc too, that shrinks them a decent amount (e.g. 1371684 -> 1126765 for psql). But I've not done that here. 0005 aix: No need to use mkldexport when we want to export all symbols This makes the building of shared libraries a lot more similar to other platforms. Export files are only used when an exports.txt is present and there's no more intermediary static libraries. 0006 configure: Expand -fvisibility checks to more compilers, add -qvisibility This isn't strictly speaking part of the same "thread" of work, but I don't want to touch aix more often than I have too... I'll post it in the other thread too. I did just test that this passes at least some tests on aix with xlc and solaris with sunpro. Greetings, Andres
Attachment
- v1-0001-aix-Fix-SHLIB_EXPORTS-reference-in-VPATH-builds.patch
- v1-0002-Remove-SUBSYS.o-rule-in-common.mk-hasn-t-been-use.patch
- v1-0003-Remove-rule-to-generate-postgres.o-not-needed-for.patch
- v1-0004-aix-when-building-with-gcc-tell-gcc-we-re-buildin.patch
- v1-0005-aix-No-need-to-use-mkldexport-when-we-want-to-exp.patch
- v1-0006-configure-Expand-fvisibility-checks-to-more-compi.patch
Hi, On 2022-08-20 10:42:13 -0700, Andres Freund wrote: > On 2022-08-20 01:35:22 -0700, Andres Freund wrote: > > I'll send in a patch series tomorrow, too tired for today. > > Here it goes. > 0001 aix: Fix SHLIB_EXPORTS reference in VPATH builds > > That's mostly so I could even build. It's not quite right in the sense that > we don't depend on the file, but that's a preexisting issue. Could be folded > in with 0005, which fixes that aspect. Or it could be backpatched as the > minimal fix. > > > 0002 Remove SUBSYS.o rule in common.mk, hasn't been used in a long time > 0003 Remove rule to generate postgres.o, not needed for 20+ years > > Both obvious, I think. Pushed these, given that they're all pretty trivial. > 0004 aix: when building with gcc, tell gcc we're building a shared library > > That's the gcc -shared issue I explained in the email I'm replying to. > > We should probably consider building executables with -shared-libgcc too, > that shrinks them a decent amount (e.g. 1371684 -> 1126765 for psql). But > I've not done that here. > > > 0005 aix: No need to use mkldexport when we want to export all symbols > > This makes the building of shared libraries a lot more similar to other > platforms. Export files are only used when an exports.txt is present and > there's no more intermediary static libraries. > > > 0006 configure: Expand -fvisibility checks to more compilers, add -qvisibility > > This isn't strictly speaking part of the same "thread" of work, but I don't > want to touch aix more often than I have too... I'll post it in the other > thread too. > > I did just test that this passes at least some tests on aix with xlc and > solaris with sunpro. Any comments here? Greetings, Andres Freund
On Wed, Aug 24, 2022 at 08:43:04PM -0700, Andres Freund wrote: > On 2022-08-20 10:42:13 -0700, Andres Freund wrote: > > 0004 aix: when building with gcc, tell gcc we're building a shared library > > > > That's the gcc -shared issue I explained in the email I'm replying to. > > > > We should probably consider building executables with -shared-libgcc too, > > that shrinks them a decent amount (e.g. 1371684 -> 1126765 for psql). But > > I've not done that here. > > > > > > 0005 aix: No need to use mkldexport when we want to export all symbols > > > > This makes the building of shared libraries a lot more similar to other > > platforms. Export files are only used when an exports.txt is present and > > there's no more intermediary static libraries. > > > > > > 0006 configure: Expand -fvisibility checks to more compilers, add -qvisibility > > > > This isn't strictly speaking part of the same "thread" of work, but I don't > > want to touch aix more often than I have too... I'll post it in the other > > thread too. > > > > I did just test that this passes at least some tests on aix with xlc and > > solaris with sunpro. > > Any comments here? I don't know much about them, but they sound like the sort of thing that can't cause subtle bugs. If they build and test the first time, they're probably valid. You may as well push them.