Thread: [HACKERS] PostgreSQL10 beta2 with ICU - initdb fails on MacOS

Sandeep Thakkar <sandeep.thakkar@enterprisedb.com> writes:
> On MacOS, I configured PG10 beta2 sources with '--with-icu" option along
> with ICU_LIBS and ICU_CFLAGS env variables to define the location of the
> ICU libs that I built. Once the staging is ready, I executed initdb but it
> fails with the following error:
> [53618] FATAL:  could not open collator for locale "und":
> U_FILE_ACCESS_ERROR*
> *2017-07-12 11:18:21.187 BST [53618] STATEMENT:  SELECT
> pg_import_system_collations('pg_catalog');*

Ugh.  Please provide details about the hand-built ICU --- what version,
what configure options did you use for it, etc.
        regards, tom lane

Sandeep Thakkar <sandeep.thakkar@enterprisedb.com> writes:
> On Wed, Jul 12, 2017 at 8:13 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Ugh.  Please provide details about the hand-built ICU --- what version,
>> what configure options did you use for it, etc.

> I tried with ICU 53.1 and 57.1 and the results is same.
> There was no configure options other than --prefix.

Well, when I try that on macOS, I get an ICU installation that's basically
nonfunctional, apparently because they forgot to specify -install_name
while linking the dylibs.  The symptom looks a bit different:

$ initdb
dyld: Library not loaded: libicui18n.57.dylib Referenced from: /Users/tgl/testversion/bin/postgres Reason: image not
found

but it's clear from otool that the dynamic linker doesn't know where
to find the ICU libraries:

$ otool -L ~/testversion/bin/postgres
/Users/tgl/testversion/bin/postgres:       /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
1238.60.2)      libicui18n.57.dylib (compatibility version 57.0.0, current version 57.1.0)       libicuuc.57.dylib
(compatibilityversion 57.0.0, current version 57.1.0) 

and this is evidently because the libraries themselves don't know where
they live:

$ otool -D /usr/local/icu-57.1/lib/libicui18n.57.dylib
/usr/local/icu-57.1/lib/libicui18n.57.dylib:
libicui18n.57.dylib

Compare to what you get for a properly built dylib:

$ otool -D /usr/local/ssl-1.1.0e/lib/libssl.1.1.dylib
/usr/local/ssl-1.1.0e/lib/libssl.1.1.dylib:
/usr/local/ssl-1.1.0e/lib/libssl.1.1.dylib

I can make it work by setting DYLD_LIBRARY_PATH:

$ DYLD_LIBRARY_PATH=/usr/local/icu-57.1/lib initdb
... goes through normally ...

but that hardly qualifies as a production-grade solution.

I've not dug into the ICU sources to see what it would take to get the
correct link option applied.  But the short answer seems to be that ICU
is not up to speed for installation in non-default locations on macOS.

For the record's sake, I configured icu4c-57_1-src.tgz like this:

$ ./runConfigureICU MacOSX --prefix=/usr/local/icu-57.1

and since there's no pkg-config on this machine, I did this to
configure Postgres:

$ ICU_CFLAGS=" " ICU_LIBS="-licui18n -licuuc -licudata" ./configure [usual options] --with-icu
--with-includes=/usr/local/icu-57.1/include--with-libraries=/usr/local/icu-57.1/lib 

It's possible that if I did have pkg-config installed, it would have
produced some different value for ICU_LIBS, but I rather doubt it.
I'm not sure that it's possible to fix this on the consumer executable's
link line anyway.
        regards, tom lane

Sandeep Thakkar <sandeep.thakkar@enterprisedb.com> writes:
> On Thu, Jul 13, 2017 at 12:44 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> and this is evidently because the libraries themselves don't know where
>> they live:
>> $ otool -D /usr/local/icu-57.1/lib/libicui18n.57.dylib
>> /usr/local/icu-57.1/lib/libicui18n.57.dylib:
>> libicui18n.57.dylib

> Right. I got that and I fixed the names and loader_paths for ICU libs and
> postgres and that is why initdb in my case got going and didn't complain
> about library not found.

Uh, so what did you do *exactly*?

>> I can make it work by setting DYLD_LIBRARY_PATH:
>> $ DYLD_LIBRARY_PATH=/usr/local/icu-57.1/lib initdb
>> ... goes through normally ...

> You mean you are able to initialize cluster after this? Or you just
> executed initdb and found that it doesn't complain about ICU lib location?

initdb completed successfully.  I didn't try running any tests beyond
that; I'm not sure that we have any regression tests that will exercise
ICU locales.

> As mentioned above instead of using DYLD_LIBRARY_PATH, I fixed the
> loader_paths and am able to execute initdb and postgres binaries:
> But, initdb -D data fails with error code "U_FILE_ACCESS_ERROR".

Yeah, but notice that only two of the three interesting ICU libraries
are actually linked into the postgres executable so far as otool and
the dynamic linker are concerned.  I suspect that the other one,
libicudata, is dynamically loaded by the ICU code --- and in your
configuration it fails to find that library.  The error message is
not definitive that that's what's happening, but it's suggestive.
If that's the right interpretation, it means that setting
DYLD_LIBRARY_PATH allows that third library to be found, but whatever
you did doesn't.

I still think that this represents under-engineering by the ICU crew
and not anything we're doing wrong.

BTW, when I skimmed the "readme.html" docs in the ICU sources this
morning, I noted that there were multiple ways you could configure
it to find the ICU data.  I did not read in detail, figuring that
their default configuration would be sane, but maybe that was an
overly charitable assumption.
        regards, tom lane

I wrote:
> I still think that this represents under-engineering by the ICU crew
> and not anything we're doing wrong.

After some more examination of their readme.html, I tried this:

$ ./runConfigureICU MacOSX --prefix=/usr/local/icu-57.1 --enable-rpath

and that gave me sane-looking libraries:

$ otool -L /usr/local/icu-57.1/lib/libicui18n.57.1.dylib
/usr/local/icu-57.1/lib/libicui18n.57.1.dylib:       /usr/local/icu-57.1/lib/libicui18n.57.dylib (compatibility version
57.0.0,current version 57.1.0)       /usr/local/icu-57.1/lib/libicuuc.57.dylib (compatibility version 57.0.0, current
version57.1.0)       /usr/local/icu-57.1/lib/libicudata.57.dylib (compatibility version 57.0.0, current version 57.1.0)
     /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)       /usr/lib/libc++.1.dylib
(compatibilityversion 1.0.0, current version 307.5.0) 

--- note that now libicudata is shown as a dependency.  And then
configuring Postgres as I showed before leads to executables that
build, and initdb, and pass regression tests, and have a bunch
of ICU entries in pg_collation, without needing to fool with
DYLD_LIBRARY_PATH.

Their configure --help claims
 --enable-rpath          use rpath when linking default is only if necessary

but apparently that's for small values of "necessary" :-(
        regards, tom lane