Thread: AIX: Symbols are missing in libpq.a

On Thu, Aug 26, 2021 at 12:49:01PM +0000, REIX, Tony wrote:
> While porting postgresql-odbc v13 to AIX, we have found that (at least) 2 symbols are missing in libpq.a provided by
theport of PostgreSQL v13.1 to AIX 7.1 by the BullFreeware project :
 
> 
> pg_char_to_encoding
> pg_encoding_to_char
> 
> Looking at details, it appears that these symbols are present in version 12.8 .
> They were still missing in 13.4 .
> Something has changed between v12 and v13.
> 
> Looking at more details, the way libpq.a is built on AIX is different from the way libpq.so is built on Linux.
> On Linux, the file "exports.txt" is used for building the list of symbols to be exported.
> On AIX, the tool  mkldexport.sh is used for dynamically generating the symbols to be exported.
> And it appears that 5 symbols (including the 2 above) are missing on AIX. Don't know why.

Would you study why it changed?  If $(MKLDEXPORT) is no longer able to find
all symbols, then we're likely to have problems in more libraries than libpq,
including libraries that don't use a $(SHLIB_EXPORTS) file.

On Mon, Aug 30, 2021 at 02:33:32PM +0000, REIX, Tony wrote:
> It appears that the Makefile in src/interfaces/libpq has been modified between v12 and v13, removing encnames.o (and
wchar.o)from the object files being used for creating the static libpq.a file which is used for creating the libpq.so
andlibpq.a AIX shared-library files.
 
> And, since pg_encoding_to_char() is defined in encnames.o , it has disappeared from the libpq.exp file .
> 
> # diff postgresql-12.8/32bit/src/interfaces/libpq/Makefile postgresql-13.1/32bit/src/interfaces/libpq/Makefile | grep
encnames
> < OBJS += encnames.o wchar.o
> < encnames.c wchar.c: % : $(backend_src)/utils/mb/%
> <       rm -f encnames.c wchar.c
> 
> Remember how the final libpq.a is built on AIX:
> 
>  rm -f libpq.a
>  /usr/bin/ar crs libpq.a fe-auth-scram.o ...
>  touch libpq.a
>  ../../../src/backend/port/aix/mkldexport.sh libpq.a libpq.so.5 >libpq.exp
>  /opt/freeware/bin/gcc  -maix64    -O3 .....  -o libpq.so.5 libpq.a -Wl,-bE:libpq.exp .......
>  rm -f libpq.a
>  /usr/bin/ar crs libpq.a libpq.so.5
> 
>  12.8 : /usr/bin/ar crs libpq.a fe-auth.o fe-auth-scram.o fe-connect.o fe-exec.o fe-lobj.o fe-misc.o fe-print.o
fe-protocol2.ofe-protocol3.o
 
>                                 pqexpbuffer.o fe-secure.o legacy-pqsignal.o libpq-events.o encnames.o wchar.o
fe-secure-openssl.ofe-secure-common.o
 
> 
>  13.1 : /usr/bin/ar crs libpq.a fe-auth.o fe-auth-scram.o fe-connect.o fe-exec.o fe-lobj.o fe-misc.o fe-print.o
fe-protocol2.ofe-protocol3.o
 
>                                 pqexpbuffer.o fe-secure.o legacy-pqsignal.o libpq-events.o
fe-secure-common.ofe-secure-openssl.o
 
> 
> 
> Maybe you can discover why these changes were made in v13 for src/interfaces/libpq/Makefile .
> And  mkldexport.sh  , unchanged between v12 and v13, works perfectly.

Thanks; that makes sense.  Those files moved to libpgcommon_shlib.a.  The
$(MKLDEXPORT) call sees symbols in the .o files, but it doesn't see symbols
pulled in via libpg*.a.  Let's fix this by having Makefile.shlib skip
$(MKLDEXPORT) when a $(SHLIB_EXPORTS) file is available and have it instead
create lib$(NAME).exp from $(SHLIB_EXPORTS).  That's more correct than
merging, and it's simpler.  Would you like to try that?

On Mon, Aug 30, 2021 at 03:35:23PM +0000, REIX, Tony wrote:
> Yes, trying to use the create lib$(NAME).exp from $(SHLIB_EXPORTS) when it exists was my first idea, too.
> However, I do not master (or I forgot) this kind of "if...." in a Makefile and I was unable to find a solution by
readingMakefile manuals or by searching for a similar example. So, I did it in an easier (to me!) and quicker way:
mergewith a new command line in the Makefile rule.
 
> Now that we have a clear understanding of what is happenning, I may have a deeper look at a clean Makefile solution.
However,if you know how to manage this, I would really appreciate some example. I'm asking my colleague too if he can
helpme here.
 

Here's an example from elsewhere in Makefile.shlib:

# If SHLIB_EXPORTS is set, the rules below will build a .def file from that.
# Else we just use --export-all-symbols.
ifeq (,$(SHLIB_EXPORTS))
$(shlib): $(OBJS) | $(SHLIB_PREREQS)
    $(CC) $(CFLAGS)  -shared -static-libgcc -o $@  $(OBJS) $(LDFLAGS) $(LDFLAGS_SL) $(SHLIB_LINK) $(LIBS)
-Wl,--export-all-symbols-Wl,--out-implib=$(stlib)
 
else
DLL_DEFFILE = lib$(NAME)dll.def

$(shlib): $(OBJS) $(DLL_DEFFILE) | $(SHLIB_PREREQS)
    $(CC) $(CFLAGS)  -shared -static-libgcc -o $@  $(OBJS) $(DLL_DEFFILE) $(LDFLAGS) $(LDFLAGS_SL) $(SHLIB_LINK)
$(LIBS)-Wl,--out-implib=$(stlib)
 

UC_NAME = $(shell echo $(NAME) | tr 'abcdefghijklmnopqrstuvwxyz' 'ABCDEFGHIJKLMNOPQRSTUVWXYZ')

$(DLL_DEFFILE): $(SHLIB_EXPORTS)
    echo 'LIBRARY LIB$(UC_NAME).dll' >$@
    echo 'EXPORTS' >>$@
    sed -e '/^#/d' -e 's/^\(.*[     ]\)\([0-9][0-9]*\)/    \1@ \2/' $< >>$@
endif

A new patch, using exports.txt file instead of building the list of symbols to be exported and merging the 2 files, has
beenprovided. This seems much better. 

On Wed, Sep 01, 2021 at 08:59:57AM +0000, REIX, Tony wrote:
> Here is a new patch, using the export.txt whenever it does exist.
> I have tested it with v13.4 : it's OK.
> Patch for 14beta3 should be the same since there was no change for src/Makefile.shlib between v13 and v14.

Thanks.  This looks good.  I'm attaching what I intend to push, which just
adds a log message and some cosmetic changes compared to your version.  Here
are the missing symbols restored by the patch:

pg_encoding_to_char
pg_utf_mblen
pg_char_to_encoding
pg_valid_server_encoding
pg_valid_server_encoding_id

I was ambivalent about whether to back-patch to v13 or to stop at v14, but I
decided that v13 should have this change.  We should expect sad users when
libpq lacks a documented symbol.  Complaints about loss of undocumented
symbols (e.g. pqParseInput3) are unlikely, and we're even less likely to have
users opposing reintroduction of long-documented symbols.  An alternative
would be to have v13 merge the symbol lists, like your original proposal, so
we're not removing even undocumented symbols.  I doubt applications have
accrued dependencies on libpq-internal symbols in the year since v13 appeared,
particularly since those symbols are inaccessible on Linux.  Our AIX export
lists never included libpgport or libpgcommon symbols.