Re: How to find freak UTF-8 character? - Mailing list pgsql-general

From Andrew Sullivan
Subject Re: How to find freak UTF-8 character?
Date
Msg-id 20111001192706.GA44962@shinkuro.com
Whole thread Raw
In response to How to find freak UTF-8 character?  (Leif Biberg Kristensen <leif@solumslekt.org>)
Responses Re: How to find freak UTF-8 character?  (Leif Biberg Kristensen <leif@solumslekt.org>)
List pgsql-general
On Sat, Oct 01, 2011 at 07:55:01AM +0200, Leif Biberg Kristensen wrote:
> I've somehow introduced a spurious UTF-8 character in my database. When I try
> to export to an application that requires LATIN1 encoding, my export script
> bombs out with this message:
>
> psycopg2.DataError: character 0xe2808e of encoding "UTF8" has no equivalent in
> "LATIN1"

I see you found it, but note that it's _not_ a spurious UTF-8
character: it's a right-to-left mark, ans is a perfectly ok UTF-8 code
point.

If you need a subset of the UTF-8 character set, you want to make sure
you have some sort of constraint in your application or your database
that prevents insertion of anything at all in UTF-8.  This is a need
people often forget when working in an internationalized setting,
because there's a lot of crap that comes from the client side in a
UTF-8 setting that might not come in other settings (like LATIN1).

Best,

A

--
Andrew Sullivan
ajs@crankycanuck.ca

pgsql-general by date:

Previous
From: Devrim GÜNDÜZ
Date:
Subject: Re: how to install 9.1 on non-graphical server?
Next
From: Merlin Moncure
Date:
Subject: Re: bytea columns and large values