On Dec 11, 2006, at 02:47 , Daniel van Ham Colchete wrote:
> I never understood what's the matter between the ASCII/ISO-8859-1/UTF8
> charsets to a database. They're all simple C strings that doesn't have
> the zero-byte in the midlle (like UTF16 would) and that doesn't
> require any different processing unless you are doing case insensitive
> search (them you would have a problem).
That's not the whole story. UTF-8 and other variable-width encodings
don't provide a 1:1 mapping of logical characters to single bytes; in
particular, combination characters opens the possibility of multiple
different byte sequences mapping to the same code point; therefore,
string comparison in such encodings generally cannot be done at the
byte level (unless, of course, you first acertain that the strings
involved are all normalized to an unambiguous subset of your encoding).
PostgreSQL's use of strings is not limited to string comparison.
Substring extraction, concatenation, regular expression matching, up/
downcasing, tokenization and so on are all part of PostgreSQL's small
library of text manipulation functions, and all deal with logical
characters, meaning they must be Unicode-aware.
Alexander.
