Peter Eisentraut wrote:
> There is a PG18 open item to document this possible upgrade incompatibility.
>
> I think the following text could be added to the release notes:
>
> """
> The locale implementation underlying full-text search was improved. It
> now observes the locale provider configured for the database. It was
> previously hardcoded to use the configured libc LC_CTYPE setting
> [...]
That sounds misleading because LC_CTYPE is still used in 18.
To illustrate in an ICU database, the parser will classify "Em Dash"
as a separator or not depending on LC_CTYPE.
with LC_CTYPE=C
=> select alias, token,lexemes from ts_debug('simple', U&'ABCD\2014EFGH');
alias | token | lexemes
-------+-----------+-------------
word | ABCD—EFGH | {abcd—efgh}
with LC_CTYPE=en_US.utf8 (glibc 2.35):
=> select alias, token,lexemes from ts_debug('simple', U&'ABCD\2014EFGH');
alias | token | lexemes
-----------+-------+---------
asciiword | ABCD | {abcd}
blank | — |
asciiword | EFGH | {efgh}
OTOH lower casing uses LC_CTYPE in 17, but not in 18, leading
to better lexemes.
pg17, ICU locale, LC_TYPE=C
=> select alias, token,lexemes from ts_debug('simple', 'ÉTÉ');
alias | token | lexemes
-------+-------+---------
word | ÉTÉ | {ÉtÉ}
pg18, ICU locale, LC_TYPE=C
select alias, token,lexemes from ts_debug('simple', 'ÉTÉ');
alias | token | lexemes
-------+-------+---------
word | ÉTÉ | {été}
So maybe the release notes should say
"now observes the locale provider configured for the database to
convert strings to lower case".
Best regards,
--
Daniel Vérité
https://postgresql.verite.pro/
