Re: Concerning about Unicode-aware string handling - Mailing list pgsql-general

From Tom Lane
Subject Re: Concerning about Unicode-aware string handling
Date
Msg-id 14801.1337609055@sss.pgh.pa.us
Whole thread Raw
In response to Re: Concerning about Unicode-aware string handling  (Vincas Dargis <vindrg@gmail.com>)
List pgsql-general
Vincas Dargis <vindrg@gmail.com> writes:
> Database created using:
> initdb -D ../data -E utf-8 -U postgres

That looks fairly dangerous, as it will absorb the database's locale
settings (particularly LC_CTYPE, which is what you care about for these
operations) from your shell environment.  If the environment locale is
not for UTF8 encoding then it won't work at all.  Best to specify a
--locale switch as well.  See
http://www.postgresql.org/docs/9.1/static/charset.html

> But regexp_replace issue is still there. Regexp "\w" understands only
> as "ascii word character" ?

Locale-specific character classes in regexps are not terribly bright
about UTF8, because historically that code has not considered any
character codes above 255 :-(.  So in UTF8 you only got correct behavior
for ASCII and LATIN1 characters.  9.2 will be better though not perfect:
http://git.postgresql.org/gitweb/?p=postgresql.git&a=commitdiff&h=e00f68e49

            regards, tom lane

pgsql-general by date:

Previous
From: Vincas Dargis
Date:
Subject: Re: Concerning about Unicode-aware string handling
Next
From: Merlin Moncure
Date:
Subject: Re: Global Named Prepared Statements