Re: UTF-8 encoding problem w/ libpq - Mailing list pgsql-hackers

From Tom Lane
Subject Re: UTF-8 encoding problem w/ libpq
Date
Msg-id 4852.1370284110@sss.pgh.pa.us
Whole thread Raw
In response to Re: UTF-8 encoding problem w/ libpq  (Heikki Linnakangas <hlinnakangas@vmware.com>)
Responses Re: UTF-8 encoding problem w/ libpq  (Heikki Linnakangas <hlinnakangas@vmware.com>)
Re: UTF-8 encoding problem w/ libpq  (Andrew Dunstan <andrew@dunslane.net>)
List pgsql-hackers
Heikki Linnakangas <hlinnakangas@vmware.com> writes:
> He *is* using UTF-8. Or trying to, anyway :-). The downcasing in the 
> backend is supposed to leave bytes with the high-bit set alone, ie. in 
> UTF-8 encoding, it's supposed to leave ä and ß alone.

Well, actually, downcase_truncate_identifier() is doing this:
    unsigned char ch = (unsigned char) ident[i];
    if (ch >= 'A' && ch <= 'Z')        ch += 'a' - 'A';    else if (IS_HIGHBIT_SET(ch) && isupper(ch))        ch =
tolower(ch);

There's basically no way that that second case can give pleasant results
in a multibyte encoding, other than by not doing anything.  I suspect
that Windows' libc has fewer defenses than other implementations and
performs some transformation that we don't get elsewhere.  This may also
explain the gripe yesterday in -general about funny results in OS X.

We talked about this before and went off into the weeds about whether
it was sensible to try to use towlower() and whether that wouldn't
create undesirably platform-sensitive results.  I wonder though if we
couldn't just fix this code to not do anything to high-bit-set bytes
in multibyte encodings.
        regards, tom lane



pgsql-hackers by date:

Previous
From: Andres Freund
Date:
Subject: Re: Vacuum, Freeze and Analyze: the big picture
Next
From: Kevin Grittner
Date:
Subject: Re: Implicit rule created for materialized views