Thread: [HACKERS] PostgreSQL10 beta2 with ICU - initdb fails on MacOS

[HACKERS] PostgreSQL10 beta2 with ICU - initdb fails on MacOS

From
Sandeep Thakkar
Date:
Hi

On MacOS, I configured PG10 beta2 sources with '--with-icu" option along with ICU_LIBS and ICU_CFLAGS env variables to define the location of the ICU libs that I built. Once the staging is ready, I executed initdb but it fails with the following error:
--
$./initdb -D /tmp/data
The files belonging to this database system will be owned by user "buildfarm".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

creating directory /tmp/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... 2017-07-12 11:18:21.187 BST [53618] FATAL:  could not open collator for locale "und": U_FILE_ACCESS_ERROR
2017-07-12 11:18:21.187 BST [53618] STATEMENT:  SELECT pg_import_system_collations('pg_catalog');
child process exited with exit code 1
initdb: removing data directory "/tmp/data"
--

Has anyone came across this? This error is not seen on Linux and Windows though.

--
Sandeep Thakkar
EDB


Re: [HACKERS] PostgreSQL10 beta2 with ICU - initdb fails on MacOS

From
Tom Lane
Date:
Sandeep Thakkar <> writes:
> On MacOS, I configured PG10 beta2 sources with '--with-icu" option along
> with ICU_LIBS and ICU_CFLAGS env variables to define the location of the
> ICU libs that I built. Once the staging is ready, I executed initdb but it
> fails with the following error:
> [53618] FATAL:  could not open collator for locale "und":
> U_FILE_ACCESS_ERROR*
> *2017-07-12 11:18:21.187 BST [53618] STATEMENT:  SELECT
> pg_import_system_collations('pg_catalog');*

Ugh.  Please provide details about the hand-built ICU --- what version,
what configure options did you use for it, etc.
        regards, tom lane



Re: [HACKERS] PostgreSQL10 beta2 with ICU - initdb fails on MacOS

From
Sandeep Thakkar
Date:


On Wed, Jul 12, 2017 at 8:13 PM, Tom Lane <> wrote:
Sandeep Thakkar <sandeep.thakkar@enterprisedb.com> writes:
> On MacOS, I configured PG10 beta2 sources with '--with-icu" option along
> with ICU_LIBS and ICU_CFLAGS env variables to define the location of the
> ICU libs that I built. Once the staging is ready, I executed initdb but it
> fails with the following error:
> [53618] FATAL:  could not open collator for locale "und":
> U_FILE_ACCESS_ERROR*
> *2017-07-12 11:18:21.187 BST [53618] STATEMENT:  SELECT
> pg_import_system_collations('pg_catalog');*

Ugh.  Please provide details about the hand-built ICU --- what version,
what configure options did you use for it, etc.

I tried with ICU 53.1 and 57.1 and the results is same.
There was no configure options other than --prefix. In environment, CXXFLAGS was set to "-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk -mmacosx-version-min=10.7 -arch i386 -arch x86_64"


                        regards, tom lane



--
Sandeep Thakkar
EDB


Re: [HACKERS] PostgreSQL10 beta2 with ICU - initdb fails on MacOS

From
Tom Lane
Date:
Sandeep Thakkar <> writes:
> On Wed, Jul 12, 2017 at 8:13 PM, Tom Lane <> wrote:
>> Ugh.  Please provide details about the hand-built ICU --- what version,
>> what configure options did you use for it, etc.

> I tried with ICU 53.1 and 57.1 and the results is same.
> There was no configure options other than --prefix.

Well, when I try that on macOS, I get an ICU installation that's basically
nonfunctional, apparently because they forgot to specify -install_name
while linking the dylibs.  The symptom looks a bit different:

$ initdb
dyld: Library not loaded: libicui18n.57.dylib Referenced from: /Users/tgl/testversion/bin/postgres Reason: image not
found

but it's clear from otool that the dynamic linker doesn't know where
to find the ICU libraries:

$ otool -L ~/testversion/bin/postgres
/Users/tgl/testversion/bin/postgres:       /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
1238.60.2)      libicui18n.57.dylib (compatibility version 57.0.0, current version 57.1.0)       libicuuc.57.dylib
(compatibilityversion 57.0.0, current version 57.1.0) 

and this is evidently because the libraries themselves don't know where
they live:

$ otool -D /usr/local/icu-57.1/lib/libicui18n.57.dylib
/usr/local/icu-57.1/lib/libicui18n.57.dylib:
libicui18n.57.dylib

Compare to what you get for a properly built dylib:

$ otool -D /usr/local/ssl-1.1.0e/lib/libssl.1.1.dylib
/usr/local/ssl-1.1.0e/lib/libssl.1.1.dylib:
/usr/local/ssl-1.1.0e/lib/libssl.1.1.dylib

I can make it work by setting DYLD_LIBRARY_PATH:

$ DYLD_LIBRARY_PATH=/usr/local/icu-57.1/lib initdb
... goes through normally ...

but that hardly qualifies as a production-grade solution.

I've not dug into the ICU sources to see what it would take to get the
correct link option applied.  But the short answer seems to be that ICU
is not up to speed for installation in non-default locations on macOS.

For the record's sake, I configured icu4c-57_1-src.tgz like this:

$ ./runConfigureICU MacOSX --prefix=/usr/local/icu-57.1

and since there's no pkg-config on this machine, I did this to
configure Postgres:

$ ICU_CFLAGS=" " ICU_LIBS="-licui18n -licuuc -licudata" ./configure [usual options] --with-icu
--with-includes=/usr/local/icu-57.1/include--with-libraries=/usr/local/icu-57.1/lib 

It's possible that if I did have pkg-config installed, it would have
produced some different value for ICU_LIBS, but I rather doubt it.
I'm not sure that it's possible to fix this on the consumer executable's
link line anyway.
        regards, tom lane



Re: [HACKERS] PostgreSQL10 beta2 with ICU - initdb fails on MacOS

From
Sandeep Thakkar
Date:


On Thu, Jul 13, 2017 at 12:44 AM, Tom Lane <> wrote:
Sandeep Thakkar <sandeep.thakkar@enterprisedb.com> writes:
> On Wed, Jul 12, 2017 at 8:13 PM, Tom Lane <> wrote:
>> Ugh.  Please provide details about the hand-built ICU --- what version,
>> what configure options did you use for it, etc.

> I tried with ICU 53.1 and 57.1 and the results is same.
> There was no configure options other than --prefix.

Well, when I try that on macOS, I get an ICU installation that's basically
nonfunctional, apparently because they forgot to specify -install_name
while linking the dylibs.  The symptom looks a bit different:

$ initdb
dyld: Library not loaded: libicui18n.57.dylib
  Referenced from: /Users/tgl/testversion/bin/postgres
  Reason: image not found

but it's clear from otool that the dynamic linker doesn't know where
to find the ICU libraries:

$ otool -L ~/testversion/bin/postgres
/Users/tgl/testversion/bin/postgres:
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
        libicui18n.57.dylib (compatibility version 57.0.0, current version 57.1.0)
        libicuuc.57.dylib (compatibility version 57.0.0, current version 57.1.0)

and this is evidently because the libraries themselves don't know where
they live:

$ otool -D /usr/local/icu-57.1/lib/libicui18n.57.dylib
/usr/local/icu-57.1/lib/libicui18n.57.dylib:
libicui18n.57.dylib

Right. I got that and I fixed the names and loader_paths for ICU libs and postgres and that is why initdb in my case got going and didn't complain about library not found.
 
Compare to what you get for a properly built dylib:

$ otool -D /usr/local/ssl-1.1.0e/lib/libssl.1.1.dylib
/usr/local/ssl-1.1.0e/lib/libssl.1.1.dylib:
/usr/local/ssl-1.1.0e/lib/libssl.1.1.dylib

I can make it work by setting DYLD_LIBRARY_PATH:

$ DYLD_LIBRARY_PATH=/usr/local/icu-57.1/lib initdb
... goes through normally ...

You mean you are able to initialize cluster after this? Or you just executed initdb and found that it doesn't complain about ICU lib location? As mentioned above instead of using DYLD_LIBRARY_PATH, I fixed the loader_paths and am able to execute initdb and postgres binaries:
$ ./initdb -V
initdb (PostgreSQL) 10beta2

$ ./postgres -V
postgres (PostgreSQL) 10beta2

But, initdb -D data fails with error code "U_FILE_ACCESS_ERROR".
 
but that hardly qualifies as a production-grade solution.

I've not dug into the ICU sources to see what it would take to get the
correct link option applied.  But the short answer seems to be that ICU
is not up to speed for installation in non-default locations on macOS.

For the record's sake, I configured icu4c-57_1-src.tgz like this:

$ ./runConfigureICU MacOSX --prefix=/usr/local/icu-57.1

and since there's no pkg-config on this machine, I did this to
configure Postgres:

$ ICU_CFLAGS=" " ICU_LIBS="-licui18n -licuuc -licudata" ./configure [usual options] --with-icu --with-includes=/usr/local/icu-57.1/include --with-libraries=/usr/local/icu-57.1/lib

It's possible that if I did have pkg-config installed, it would have
produced some different value for ICU_LIBS, but I rather doubt it.
I'm not sure that it's possible to fix this on the consumer executable's
link line anyway.

yes, we use the same flags during configure.
 
                        regards, tom lane



--
Sandeep Thakkar
EDB


Re: [HACKERS] PostgreSQL10 beta2 with ICU - initdb fails on MacOS

From
Tom Lane
Date:
Sandeep Thakkar <> writes:
> On Thu, Jul 13, 2017 at 12:44 AM, Tom Lane <> wrote:
>> and this is evidently because the libraries themselves don't know where
>> they live:
>> $ otool -D /usr/local/icu-57.1/lib/libicui18n.57.dylib
>> /usr/local/icu-57.1/lib/libicui18n.57.dylib:
>> libicui18n.57.dylib

> Right. I got that and I fixed the names and loader_paths for ICU libs and
> postgres and that is why initdb in my case got going and didn't complain
> about library not found.

Uh, so what did you do *exactly*?

>> I can make it work by setting DYLD_LIBRARY_PATH:
>> $ DYLD_LIBRARY_PATH=/usr/local/icu-57.1/lib initdb
>> ... goes through normally ...

> You mean you are able to initialize cluster after this? Or you just
> executed initdb and found that it doesn't complain about ICU lib location?

initdb completed successfully.  I didn't try running any tests beyond
that; I'm not sure that we have any regression tests that will exercise
ICU locales.

> As mentioned above instead of using DYLD_LIBRARY_PATH, I fixed the
> loader_paths and am able to execute initdb and postgres binaries:
> But, initdb -D data fails with error code "U_FILE_ACCESS_ERROR".

Yeah, but notice that only two of the three interesting ICU libraries
are actually linked into the postgres executable so far as otool and
the dynamic linker are concerned.  I suspect that the other one,
libicudata, is dynamically loaded by the ICU code --- and in your
configuration it fails to find that library.  The error message is
not definitive that that's what's happening, but it's suggestive.
If that's the right interpretation, it means that setting
DYLD_LIBRARY_PATH allows that third library to be found, but whatever
you did doesn't.

I still think that this represents under-engineering by the ICU crew
and not anything we're doing wrong.

BTW, when I skimmed the "readme.html" docs in the ICU sources this
morning, I noted that there were multiple ways you could configure
it to find the ICU data.  I did not read in detail, figuring that
their default configuration would be sane, but maybe that was an
overly charitable assumption.
        regards, tom lane



Re: [HACKERS] PostgreSQL10 beta2 with ICU - initdb fails on MacOS

From
Sandeep Thakkar
Date:


On Thu, Jul 13, 2017 at 8:23 AM, Tom Lane <> wrote:
Sandeep Thakkar <sandeep.thakkar@enterprisedb.com> writes:
> On Thu, Jul 13, 2017 at 12:44 AM, Tom Lane <> wrote:
>> and this is evidently because the libraries themselves don't know where
>> they live:
>> $ otool -D /usr/local/icu-57.1/lib/libicui18n.57.dylib
>> /usr/local/icu-57.1/lib/libicui18n.57.dylib:
>> libicui18n.57.dylib

> Right. I got that and I fixed the names and loader_paths for ICU libs and
> postgres and that is why initdb in my case got going and didn't complain
> about library not found.

Uh, so what did you do *exactly*?

I changed the id and the loader_paths for ICU libs and also changed the install name of ICU libs on postgres binary
$ otool -L postgres  | grep icu
@loader_path/../lib/libicuuc.57.dylib (compatibility version 57.0.0, current version 57.1.0)
@loader_path/../lib/libicui18n.57.dylib (compatibility version 57.0.0, current version 57.1.0)
 
>> I can make it work by setting DYLD_LIBRARY_PATH:
>> $ DYLD_LIBRARY_PATH=/usr/local/icu-57.1/lib initdb
>> ... goes through normally ...

> You mean you are able to initialize cluster after this? Or you just
> executed initdb and found that it doesn't complain about ICU lib location?

initdb completed successfully.  I didn't try running any tests beyond
that; I'm not sure that we have any regression tests that will exercise
ICU locales.

OK. I'll check at my end then.
 
> As mentioned above instead of using DYLD_LIBRARY_PATH, I fixed the
> loader_paths and am able to execute initdb and postgres binaries:
> But, initdb -D data fails with error code "U_FILE_ACCESS_ERROR".

Yeah, but notice that only two of the three interesting ICU libraries
are actually linked into the postgres executable so far as otool and
the dynamic linker are concerned.  I suspect that the other one,
libicudata, is dynamically loaded by the ICU code --- and in your
configuration it fails to find that library.  The error message is
not definitive that that's what's happening, but it's suggestive.
If that's the right interpretation, it means that setting
DYLD_LIBRARY_PATH allows that third library to be found, but whatever
you did doesn't.

Yeah, I observed that otool -L lists only two of three ICU libs. But not sure why it doesn't load the third one in my case. Here is the otool -D and otool -L output on ICU libs:
$ otool -D ../lib/libicuuc.dylib 
../lib/libicuuc.dylib:
libicuuc.57.dylib

$ otool -L ../lib/libicuuc.dylib 
../lib/libicuuc.dylib:
libicuuc.57.dylib (compatibility version 57.0.0, current version 57.1.0)
@loader_path/../lib/libicudata.57.dylib (compatibility version 57.0.0, current version 57.1.0)

$ otool -D ../lib/libicui18n.dylib 
../lib/libicui18n.dylib:
libicui18n.57.dylib

$ otool -L ../lib/libicui18n.dylib 
../lib/libicui18n.dylib:
libicui18n.57.dylib (compatibility version 57.0.0, current version 57.1.0)
@loader_path/../lib/libicuuc.57.dylib (compatibility version 57.0.0, current version 57.1.0)
@loader_path/../lib/libicudata.57.dylib (compatibility version 57.0.0, current version 57.1.0)

$ otool -D ../lib/libicudata.dylib 
../lib/libicudata.dylib:
libicudata.57.dylib

$ otool -L ../lib/libicudata.dylib 
../lib/libicudata.dylib:
libicudata.57.dylib (compatibility version 57.0.0, current version 57.1.0)
 
I still think that this represents under-engineering by the ICU crew
and not anything we're doing wrong.

BTW, when I skimmed the "readme.html" docs in the ICU sources this
morning, I noted that there were multiple ways you could configure
it to find the ICU data.  I did not read in detail, figuring that
their default configuration would be sane, but maybe that was an
overly charitable assumption.

Sure, thanks for checking this out. It seems the issue is with ICU and something I did which is why the libicudata.dylib is not able to load. I'll dig deeper. Thanks for your help.
 
                        regards, tom lane



--
Sandeep Thakkar
EDB


Re: [HACKERS] PostgreSQL10 beta2 with ICU - initdb fails on MacOS

From
Tom Lane
Date:
I wrote:
> I still think that this represents under-engineering by the ICU crew
> and not anything we're doing wrong.

After some more examination of their readme.html, I tried this:

$ ./runConfigureICU MacOSX --prefix=/usr/local/icu-57.1 --enable-rpath

and that gave me sane-looking libraries:

$ otool -L /usr/local/icu-57.1/lib/libicui18n.57.1.dylib
/usr/local/icu-57.1/lib/libicui18n.57.1.dylib:       /usr/local/icu-57.1/lib/libicui18n.57.dylib (compatibility version
57.0.0,current version 57.1.0)       /usr/local/icu-57.1/lib/libicuuc.57.dylib (compatibility version 57.0.0, current
version57.1.0)       /usr/local/icu-57.1/lib/libicudata.57.dylib (compatibility version 57.0.0, current version 57.1.0)
     /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)       /usr/lib/libc++.1.dylib
(compatibilityversion 1.0.0, current version 307.5.0) 

--- note that now libicudata is shown as a dependency.  And then
configuring Postgres as I showed before leads to executables that
build, and initdb, and pass regression tests, and have a bunch
of ICU entries in pg_collation, without needing to fool with
DYLD_LIBRARY_PATH.

Their configure --help claims
 --enable-rpath          use rpath when linking default is only if necessary

but apparently that's for small values of "necessary" :-(
        regards, tom lane