Re: Corruption of multibyte identifiers on UTF-8 locale - Mailing list pgsql-bugs

From Victor Snezhko
Subject Re: Corruption of multibyte identifiers on UTF-8 locale
Date
Msg-id uhcyyl9dh.fsf@indorsoft.ru
Whole thread Raw
In response to Re: Corruption of multibyte identifiers on UTF-8 locale  (Tom Lane <tgl@sss.pgh.pa.us>)
List pgsql-bugs
Tom Lane <tgl@sss.pgh.pa.us> writes:

>> Agreed, but such corruption indicates that there is non-multibyte-safe
>> (octet-wise) case conversion somewhere, at best (with fully working
>> locale) it will cause case conversion to do nothing instead of actual
>> conversion.
>
> Yours is the first installation I've heard of that fails to get this
> right, and the code in question (downcase_truncate_identifier) has
> been like that since PG 7.4.something ...

This code from downcase_truncate_identifier():

    else if (ch >= 0x80 && isupper(ch))
        ch = tolower(ch);

just can't work on multibyte encodings unless tolower can magically
guess what unicode symbol it operates on (having only one octet of
it). On my (ok, broken) locale definition it corrupts multibyte
characters, on working locale defs it must fail to downcase
identifiers. Unless I'm again missing something obvious...

But, from the comment above:
 * SQL99 specifies Unicode-aware case normalization, which we don't yet
 * have the infrastructure for.

OK, a lot of work is required to fix it, I see. Are there any plans to
either switch to wide-char strings or do a per-character (unlike
per-octet) processing?

--
WBR, Victor V. Snezhko
E-mail: snezhko@indorsoft.ru

pgsql-bugs by date:

Previous
From: Tom Lane
Date:
Subject: Re: Corruption of multibyte identifiers on UTF-8 locale
Next
From: Tom Lane
Date:
Subject: Re: Out of memory error during large hashagg