Thread: Debian PostgreSQL 15 pg_createcluster - could not open collator for locale

Debian PostgreSQL 15 pg_createcluster - could not open collator for locale

From

Matt Gibbins

Date:

25 March 2023, 03:55:35

I am having trouble with a fresh installation of PostgreSQL 15 from the apt repository 'http://apt.postgresql.org/pub/repos/apt bullseye-pgdg main'.

After installing the packages the following locations were empty:

/etc/postgresql/15/main/
/var/lib/postgresql/

Searching for information suggested running 'sudo pg_createcluster 15 main --start' to initialise the database.

The process failed with the message 'could not open collator for locale "yue-Hans"'.

Following is the stdout for the process. Any suggestions welcome

Matt.

Creating new PostgreSQL cluster 15/main ...
/usr/lib/postgresql/15/bin/initdb -D /var/lib/postgresql/15/main --auth-local peer --auth-host scram-sha-256 --no-instructions
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_AU.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/15/main ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Australia/Canberra
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... 2023-03-25 11:50:31.124 AEDT [12104] FATAL: could not open collator for locale "yue-Hans": U_INVALID_FORMAT_ERROR
2023-03-25 11:50:31.124 AEDT [12104] STATEMENT: SELECT pg_import_system_collations('pg_catalog');

child process exited with exit code 1
initdb: removing contents of data directory "/var/lib/postgresql/15/main"
Error: initdb failed

Re: Debian PostgreSQL 15 pg_createcluster - could not open collator for locale

From

Erik Wienhold

Date:

25 March 2023, 19:39:18

> On 25/03/2023 01:55 CET Matt Gibbins <matt_gibbins@fastmail.com.au> wrote:
>
> I am having trouble with a fresh installation of PostgreSQL 15 from the apt
> repository 'http://apt.postgresql.org/pub/repos/apt bullseye-pgdg main'.
> After installing the packages the following locations were empty:
>   * /etc/postgresql/15/main/
>   * /var/lib/postgresql/
> Searching for information suggested running 'sudo pg_createcluster 15 main --start'
> to initialise the database.
> The process failed with the message 'could not open collator for locale "yue-Hans"'.
>
> Following is the stdout for the process. Any suggestions welcome
> Matt.
>
> ------------------------------
> Creating new PostgreSQL cluster 15/main ...
>  /usr/lib/postgresql/15/bin/initdb -D /var/lib/postgresql/15/main --auth-local peer --auth-host scram-sha-256
--no-instructions
>  The files belonging to this database system will be owned by user "postgres".
>  This user must also own the server process.
>
>  The database cluster will be initialized with locale "en_AU.UTF-8".
>  The default database encoding has accordingly been set to "UTF8".
>  The default text search configuration will be set to "english".
>
>  Data page checksums are disabled.
>
>  fixing permissions on existing directory /var/lib/postgresql/15/main ... ok
>  creating subdirectories ... ok
>  selecting dynamic shared memory implementation ... posix
>  selecting default max_connections ... 100
>  selecting default shared_buffers ... 128MB
>  selecting default time zone ... Australia/Canberra
>  creating configuration files ... ok
>  running bootstrap script ... ok
>  performing post-bootstrap initialization ... 2023-03-25 11:50:31.124 AEDT [12104] FATAL: could not open collator for
locale"yue-Hans": U_INVALID_FORMAT_ERROR
 
>  2023-03-25 11:50:31.124 AEDT [12104] STATEMENT: SELECT pg_import_system_collations('pg_catalog');
>
>  child process exited with exit code 1
>  initdb: removing contents of data directory "/var/lib/postgresql/15/main"
>  Error: initdb failed

Looks like an issue with libicu.  What version is installed?

    dpkg -l 'libicu*'
    ls -l /usr/lib/x86_64-linux-gnu/libicu*

--
Erik

Re: Debian PostgreSQL 15 pg_createcluster - could not open collator for locale

From

Matt Gibbins

Date:

26 March 2023, 02:06:03

Erik

Thanks.

The version currently installed as identified from dpkg -l is.

Desired=Unknown/Install/Remove/Purge/Hold
| 
Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description

+++-==============-============-============-====================================
ii  libicu67:amd64 67.1-7       amd64        International Components 
for Unicode

The ls command returns.

lrwxrwxrwx 1 root root       18 Jul  1  2021 
/usr/lib/x86_64-linux-gnu/libicudata.so.67 -> libicudata.so.67.1
-rw-r--r-- 1 root root 28407344 Jul  1  2021 
/usr/lib/x86_64-linux-gnu/libicudata.so.67.1
lrwxrwxrwx 1 root root       18 Jul  1  2021 
/usr/lib/x86_64-linux-gnu/libicui18n.so.67 -> libicui18n.so.67.1
-rw-r--r-- 1 root root  3160560 Jul  1  2021 
/usr/lib/x86_64-linux-gnu/libicui18n.so.67.1
lrwxrwxrwx 1 root root       16 Jul  1  2021 
/usr/lib/x86_64-linux-gnu/libicuio.so.67 -> libicuio.so.67.1
-rw-r--r-- 1 root root    55456 Jul  1  2021 
/usr/lib/x86_64-linux-gnu/libicuio.so.67.1
lrwxrwxrwx 1 root root       18 Jul  1  2021 
/usr/lib/x86_64-linux-gnu/libicutest.so.67 -> libicutest.so.67.1
-rw-r--r-- 1 root root    81400 Jul  1  2021 
/usr/lib/x86_64-linux-gnu/libicutest.so.67.1
lrwxrwxrwx 1 root root       16 Jul  1  2021 
/usr/lib/x86_64-linux-gnu/libicutu.so.67 -> libicutu.so.67.1
-rw-r--r-- 1 root root   211880 Jul  1  2021 
/usr/lib/x86_64-linux-gnu/libicutu.so.67.1
lrwxrwxrwx 1 root root       16 Jul  1  2021 
/usr/lib/x86_64-linux-gnu/libicuuc.so.67 -> libicuuc.so.67.1
-rw-r--r-- 1 root root  1988784 Jul  1  2021 
/usr/lib/x86_64-linux-gnu/libicuuc.so.67.1

Information obtained from 'apt-cache policy libicu67'.

libicu67:
   Installed: 67.1-7
   Candidate: 67.1-7
   Version table:
  *** 67.1-7 500
         500 http://mirror.linux.org.au/debian bullseye/main amd64 Packages
         100 /var/lib/dpkg/status

Matt

On 26/3/23 03:39, Erik Wienhold wrote:
>> On 25/03/2023 01:55 CET Matt Gibbins <matt_gibbins@fastmail.com.au> wrote:
>>
>> I am having trouble with a fresh installation of PostgreSQL 15 from the apt
>> repository 'http://apt.postgresql.org/pub/repos/apt bullseye-pgdg main'.
>> After installing the packages the following locations were empty:
>>    * /etc/postgresql/15/main/
>>    * /var/lib/postgresql/
>> Searching for information suggested running 'sudo pg_createcluster 15 main --start'
>> to initialise the database.
>> The process failed with the message 'could not open collator for locale "yue-Hans"'.
>>
>> Following is the stdout for the process. Any suggestions welcome
>> Matt.
>>
>> ------------------------------
>> Creating new PostgreSQL cluster 15/main ...
>>   /usr/lib/postgresql/15/bin/initdb -D /var/lib/postgresql/15/main --auth-local peer --auth-host scram-sha-256
--no-instructions
>>   The files belonging to this database system will be owned by user "postgres".
>>   This user must also own the server process.
>>
>>   The database cluster will be initialized with locale "en_AU.UTF-8".
>>   The default database encoding has accordingly been set to "UTF8".
>>   The default text search configuration will be set to "english".
>>
>>   Data page checksums are disabled.
>>
>>   fixing permissions on existing directory /var/lib/postgresql/15/main ... ok
>>   creating subdirectories ... ok
>>   selecting dynamic shared memory implementation ... posix
>>   selecting default max_connections ... 100
>>   selecting default shared_buffers ... 128MB
>>   selecting default time zone ... Australia/Canberra
>>   creating configuration files ... ok
>>   running bootstrap script ... ok
>>   performing post-bootstrap initialization ... 2023-03-25 11:50:31.124 AEDT [12104] FATAL: could not open collator
forlocale "yue-Hans": U_INVALID_FORMAT_ERROR
 
>>   2023-03-25 11:50:31.124 AEDT [12104] STATEMENT: SELECT pg_import_system_collations('pg_catalog');
>>
>>   child process exited with exit code 1
>>   initdb: removing contents of data directory "/var/lib/postgresql/15/main"
>>   Error: initdb failed
> Looks like an issue with libicu.  What version is installed?
>
>     dpkg -l 'libicu*'
>     ls -l /usr/lib/x86_64-linux-gnu/libicu*
>
> --
> Erik

Re: Debian PostgreSQL 15 pg_createcluster - could not open collator for locale

From

Erik Wienhold

Date:

26 March 2023, 07:30:06

> On 26/03/2023 00:06 CET Matt Gibbins <matt_gibbins@fastmail.com.au> wrote:
>
> The version currently installed as identified from dpkg -l is.
>
> Desired=Unknown/Install/Remove/Purge/Hold
> |
> Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> ||/ Name           Version      Architecture Description
>
> +++-==============-============-============-====================================
> ii  libicu67:amd64 67.1-7       amd64        International Components
> for Unicode
>
> The ls command returns.
>
> lrwxrwxrwx 1 root root       18 Jul  1  2021
> /usr/lib/x86_64-linux-gnu/libicudata.so.67 -> libicudata.so.67.1
> -rw-r--r-- 1 root root 28407344 Jul  1  2021
> /usr/lib/x86_64-linux-gnu/libicudata.so.67.1
> lrwxrwxrwx 1 root root       18 Jul  1  2021
> /usr/lib/x86_64-linux-gnu/libicui18n.so.67 -> libicui18n.so.67.1
> -rw-r--r-- 1 root root  3160560 Jul  1  2021
> /usr/lib/x86_64-linux-gnu/libicui18n.so.67.1
> lrwxrwxrwx 1 root root       16 Jul  1  2021
> /usr/lib/x86_64-linux-gnu/libicuio.so.67 -> libicuio.so.67.1
> -rw-r--r-- 1 root root    55456 Jul  1  2021
> /usr/lib/x86_64-linux-gnu/libicuio.so.67.1
> lrwxrwxrwx 1 root root       18 Jul  1  2021
> /usr/lib/x86_64-linux-gnu/libicutest.so.67 -> libicutest.so.67.1
> -rw-r--r-- 1 root root    81400 Jul  1  2021
> /usr/lib/x86_64-linux-gnu/libicutest.so.67.1
> lrwxrwxrwx 1 root root       16 Jul  1  2021
> /usr/lib/x86_64-linux-gnu/libicutu.so.67 -> libicutu.so.67.1
> -rw-r--r-- 1 root root   211880 Jul  1  2021
> /usr/lib/x86_64-linux-gnu/libicutu.so.67.1
> lrwxrwxrwx 1 root root       16 Jul  1  2021
> /usr/lib/x86_64-linux-gnu/libicuuc.so.67 -> libicuuc.so.67.1
> -rw-r--r-- 1 root root  1988784 Jul  1  2021
> /usr/lib/x86_64-linux-gnu/libicuuc.so.67.1
>
> Information obtained from 'apt-cache policy libicu67'.
>
> libicu67:
>    Installed: 67.1-7
>    Candidate: 67.1-7
>    Version table:
>   *** 67.1-7 500
>          500 http://mirror.linux.org.au/debian bullseye/main amd64 Packages
>          100 /var/lib/dpkg/status

Same output as for me.

Would you please build icu-test from the following source and run it with
locale "yue-Hans".  Let's see if ucol_open fails with U_INVALID_FORMAT_ERROR
in this case.  There are only two functions in Postgres that call ucol_open
and emit error message "could not open collator for locale" for ICU >=54:

    initdb.c:check_icu_locale
    pg_locale.c:pg_ucol_open (called by pg_import_system_collations)

File icu-test.c:

    #include <stdio.h>

    #include <unicode/ucol.h>
    #include <unicode/uloc.h>

    int main(int argc, char** argv) {
      if (argc < 2) {
        fprintf(stderr, "usage: %s LOCALE\n", argv[0]);
        return 1;
      }

      const char *test_locale = argv[1];

      printf("version = %s\n", U_ICU_VERSION);

      UErrorCode status = U_ZERO_ERROR;
      UCollator *coll = ucol_open(test_locale, &status);
      printf("ucol_open\n  status = %s\n", u_errorName(status));

      if (U_SUCCESS(status)) {
        const char *valid_locale = ucol_getLocaleByType(coll, ULOC_VALID_LOCALE, &status);
        printf("ucol_getLocaleByType\n  status = %s\n  locale = %s\n",
               u_errorName(status), valid_locale);
      }

      ucol_close(coll);
    }

Build and test:

    apt-get install build-essential pkg-config libicu-dev
    gcc -o icu-test icu-test.c $(pkg-config --cflags --libs icu-i18n)
    ./icu-test yue-Hans

I get the following on Debian Bullseye:

    version = 67.1
    ucol_open
      status = U_USING_FALLBACK_WARNING
    ucol_getLocaleByType
      status = U_USING_FALLBACK_WARNING
      locale = zh_Hans

--
Erik

Re: Debian PostgreSQL 15 pg_createcluster - could not open collator for locale

From

Matt Gibbins

Date:

26 March 2023, 11:26:42

On 26/3/23 15:30, Erik Wienhold wrote:
> ./icu-test yue-Hans

Erik

Thanks for the test code.

I compiled and executed as instructed with the following result

Matt

__

version = 67.1
ucol_open
   status = U_INVALID_FORMAT_ERROR

Re: Debian PostgreSQL 15 pg_createcluster - could not open collator for locale

From

Erik Wienhold

Date:

26 March 2023, 12:37:43

> On 26/03/2023 10:26 CEST Matt Gibbins <matt_gibbins@fastmail.com.au> wrote:
>
> On 26/3/23 15:30, Erik Wienhold wrote:
> > ./icu-test yue-Hans
>
> Erik
>
> Thanks for the test code.
>
> I compiled and executed as instructed with the following result
>
> Matt
>
> __
>
> version = 67.1
> ucol_open
>    status = U_INVALID_FORMAT_ERROR

Okay, at least not a problem with Postgres.  I wonder if libicudata is corrupt
or simply not compatible even though it's the same version according to the
filenames.  Whats the hash of your libicu files?

    find /usr/lib/x86_64-linux-gnu -name 'libicu*.so*' | xargs sha256sum | sort -k2

I get:

    daecc24ce07ac15b77833042bd07fa5c62e8b82f5d2c38361b45fe76658b0e45  /usr/lib/x86_64-linux-gnu/libicudata.so
    daecc24ce07ac15b77833042bd07fa5c62e8b82f5d2c38361b45fe76658b0e45  /usr/lib/x86_64-linux-gnu/libicudata.so.67
    daecc24ce07ac15b77833042bd07fa5c62e8b82f5d2c38361b45fe76658b0e45  /usr/lib/x86_64-linux-gnu/libicudata.so.67.1
    0dd9fd815ec02f5052cb666433d380cb372f6016213c83b13bbf4a402b09f198  /usr/lib/x86_64-linux-gnu/libicui18n.so
    0dd9fd815ec02f5052cb666433d380cb372f6016213c83b13bbf4a402b09f198  /usr/lib/x86_64-linux-gnu/libicui18n.so.67
    0dd9fd815ec02f5052cb666433d380cb372f6016213c83b13bbf4a402b09f198  /usr/lib/x86_64-linux-gnu/libicui18n.so.67.1
    1702a12144088442b222fcbef1de64f319656c7b58001fdfd639b8689d6721c5  /usr/lib/x86_64-linux-gnu/libicuio.so
    1702a12144088442b222fcbef1de64f319656c7b58001fdfd639b8689d6721c5  /usr/lib/x86_64-linux-gnu/libicuio.so.67
    1702a12144088442b222fcbef1de64f319656c7b58001fdfd639b8689d6721c5  /usr/lib/x86_64-linux-gnu/libicuio.so.67.1
    fc098d6df6825c1e46ac31d984a0c1839a514eef018b1f835b7610b00e008b92  /usr/lib/x86_64-linux-gnu/libicutest.so
    fc098d6df6825c1e46ac31d984a0c1839a514eef018b1f835b7610b00e008b92  /usr/lib/x86_64-linux-gnu/libicutest.so.67
    fc098d6df6825c1e46ac31d984a0c1839a514eef018b1f835b7610b00e008b92  /usr/lib/x86_64-linux-gnu/libicutest.so.67.1
    742eedc54a2b6b12003fbb5a71105fae1ed3062b5c2765083d5310738fc2b2b6  /usr/lib/x86_64-linux-gnu/libicutu.so
    742eedc54a2b6b12003fbb5a71105fae1ed3062b5c2765083d5310738fc2b2b6  /usr/lib/x86_64-linux-gnu/libicutu.so.67
    742eedc54a2b6b12003fbb5a71105fae1ed3062b5c2765083d5310738fc2b2b6  /usr/lib/x86_64-linux-gnu/libicutu.so.67.1
    650d19e2164b13dbcac1f1cd493dbc01f0e3d873086e3a718d9ca09ddd59f2fd  /usr/lib/x86_64-linux-gnu/libicuuc.so
    650d19e2164b13dbcac1f1cd493dbc01f0e3d873086e3a718d9ca09ddd59f2fd  /usr/lib/x86_64-linux-gnu/libicuuc.so.67
    650d19e2164b13dbcac1f1cd493dbc01f0e3d873086e3a718d9ca09ddd59f2fd  /usr/lib/x86_64-linux-gnu/libicuuc.so.67.1

Maybe reinstalling libicu67 solves your issue.

I'm out of ideas at this point.  I recommend addressing your issue to Debian
or the ICU mailing list:

https://bugs.debian.org/cgi-bin/pkgreport.cgi?package=libicu67
https://sourceforge.net/p/icu/mailman/

--
Erik

Re: Debian PostgreSQL 15 pg_createcluster - could not open collator for locale

From

Matt Gibbins

Date:

26 March 2023, 14:41:24

On 26/3/23 20:37, Erik Wienhold wrote:

On 26/03/2023 10:26 CEST Matt Gibbins <matt_gibbins@fastmail.com.au> wrote:

On 26/3/23 15:30, Erik Wienhold wrote:

./icu-test yue-Hans

Erik

Thanks for the test code.

I compiled and executed as instructed with the following result

Matt

__

version = 67.1
ucol_open   status = U_INVALID_FORMAT_ERROR

Okay, at least not a problem with Postgres.  I wonder if libicudata is corrupt
or simply not compatible even though it's the same version according to the
filenames.  Whats the hash of your libicu files?
	find /usr/lib/x86_64-linux-gnu -name 'libicu*.so*' | xargs sha256sum | sort -k2

I get:
	daecc24ce07ac15b77833042bd07fa5c62e8b82f5d2c38361b45fe76658b0e45  /usr/lib/x86_64-linux-gnu/libicudata.so	daecc24ce07ac15b77833042bd07fa5c62e8b82f5d2c38361b45fe76658b0e45  /usr/lib/x86_64-linux-gnu/libicudata.so.67	daecc24ce07ac15b77833042bd07fa5c62e8b82f5d2c38361b45fe76658b0e45  /usr/lib/x86_64-linux-gnu/libicudata.so.67.1	0dd9fd815ec02f5052cb666433d380cb372f6016213c83b13bbf4a402b09f198  /usr/lib/x86_64-linux-gnu/libicui18n.so	0dd9fd815ec02f5052cb666433d380cb372f6016213c83b13bbf4a402b09f198  /usr/lib/x86_64-linux-gnu/libicui18n.so.67	0dd9fd815ec02f5052cb666433d380cb372f6016213c83b13bbf4a402b09f198  /usr/lib/x86_64-linux-gnu/libicui18n.so.67.1	1702a12144088442b222fcbef1de64f319656c7b58001fdfd639b8689d6721c5  /usr/lib/x86_64-linux-gnu/libicuio.so	1702a12144088442b222fcbef1de64f319656c7b58001fdfd639b8689d6721c5  /usr/lib/x86_64-linux-gnu/libicuio.so.67	1702a12144088442b222fcbef1de64f319656c7b58001fdfd639b8689d6721c5  /usr/lib/x86_64-linux-gnu/libicuio.so.67.1	fc098d6df6825c1e46ac31d984a0c1839a514eef018b1f835b7610b00e008b92  /usr/lib/x86_64-linux-gnu/libicutest.so	fc098d6df6825c1e46ac31d984a0c1839a514eef018b1f835b7610b00e008b92  /usr/lib/x86_64-linux-gnu/libicutest.so.67	fc098d6df6825c1e46ac31d984a0c1839a514eef018b1f835b7610b00e008b92  /usr/lib/x86_64-linux-gnu/libicutest.so.67.1	742eedc54a2b6b12003fbb5a71105fae1ed3062b5c2765083d5310738fc2b2b6  /usr/lib/x86_64-linux-gnu/libicutu.so	742eedc54a2b6b12003fbb5a71105fae1ed3062b5c2765083d5310738fc2b2b6  /usr/lib/x86_64-linux-gnu/libicutu.so.67	742eedc54a2b6b12003fbb5a71105fae1ed3062b5c2765083d5310738fc2b2b6  /usr/lib/x86_64-linux-gnu/libicutu.so.67.1	650d19e2164b13dbcac1f1cd493dbc01f0e3d873086e3a718d9ca09ddd59f2fd  /usr/lib/x86_64-linux-gnu/libicuuc.so	650d19e2164b13dbcac1f1cd493dbc01f0e3d873086e3a718d9ca09ddd59f2fd  /usr/lib/x86_64-linux-gnu/libicuuc.so.67	650d19e2164b13dbcac1f1cd493dbc01f0e3d873086e3a718d9ca09ddd59f2fd  /usr/lib/x86_64-linux-gnu/libicuuc.so.67.1

Maybe reinstalling libicu67 solves your issue.

I'm out of ideas at this point.  I recommend addressing your issue to Debian
or the ICU mailing list:

https://bugs.debian.org/cgi-bin/pkgreport.cgi?package=libicu67
https://sourceforge.net/p/icu/mailman/

--
Erik

Erik

Thank you for your assistance.

The offending library was libicudata.so.67.1.

Reinstalling libicu67 has resolved the issue.

Not sure why it wasn't upgraded as I have been doing several upgrades on the system.

Matt.

body,div,table,thead,tbody,tfoot,tr,th,td,p { font-family:"Liberation Sans"; font-size:x-small }a.comment-indicator:hover + comment { background:#ffd; position:absolute; display:block; border:1px solid black; padding:0.5em; }a.comment-indicator { background:red; display:inline-block; border:1px solid black; width:0.5em; height:0.5em; }comment { display:none; }

Re: Debian PostgreSQL 15 pg_createcluster - could not open collator for locale

From

Erik Wienhold

Date:

26 March 2023, 20:39:54

> On 26/03/2023 13:41 CEST Matt Gibbins <matt_gibbins@fastmail.com.au> wrote:
>
> Thank you for your assistance.
> The offending library was libicudata.so.67.1.
> Reinstalling libicu67 has resolved the issue.
> Not sure why it wasn't upgraded as I have been doing several upgrades on the
> system.

You're welcome.  It's weird and I have never seen this before.  Maybe disk
failure or accidentally modified the files after installing?

I just found out about debsums to resolve this very issue.

https://manpages.debian.org/bullseye/debsums/debsums.1.en.html#apt_get

--
Erik