Thread: ERROR: invalid byte sequence for encoding "UTF8": 0xc35c

I am getting error in Postgresql 9.0.1.

update import_details_test
set data_row = '["4","1 Monor JoÃ\u083ão S. AntÃ\u0083ão (Schools 21,22,76)( Ru)","http://www.asdas.aa.nj.us","","908 436 4861","","","--None--","",","","--None--","","","0","","--None--","0","2008-12-29 17:53:08","","2010-08-23 14:58:56","0","Vantage Learning-OTS","hsmith@vantage.com","","","","","","","","","","","","","","0","","","","","","","0","","300 S BROAD ST","","","","ELIZABETH","","NJ","","07208","","","","","UNION","0","School","0","Tier II","","0","","","","","ELIZABETH CITY","Elementary School","Public","500","0","0","0","Suburban","","--None--","","","","0","","","","","","","","","","","","","--None--","0","","","","","","","","","","","","","","","","0","0","","0","0","0","--None--","--None--","","","","","0","","","0","","0","","","","0","","","","","","","","","","","","","","","","","","","","","0","","","","","","","","",""]'
WHERE import_queue_id = '35' AND record_num = 181;
ERROR:  invalid byte sequence for encoding "UTF8": 0xc35c

Any idea please.

On Feb 27, 2011, at 5:47 AM, AI Rumman wrote:

> Any idea please.

Don't do that ;-)

Seriously, the error means exactly what it says, so you have to figure out why your app is trying to insert invalid
UTF-8.

--
Scott Ribe
scott_ribe@elevated-dev.com
http://www.elevated-dev.com/
(303) 722-0567 voice

On 27/02/11 20:47, AI Rumman wrote:
> I am getting error in Postgresql 9.0.1.
>
> update import_details_test
> set data_row = '["4","1 Monor JoÃ\u083ão S. AntÃ\u0083ão
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Because your email client may have transformed the text encoding, I
can't make any certain conclusions about what you're actually sending to
the database, but it's highly likely that you're sending latin-1 encoded
text to the database while your client_encoding is set to 'utf8'.

The marked text is most likely the problem... but I think there's more
wrong with it than just being latin-1 encoded. That kind of mangling
often comes about when utf-8 text has been incorrectly interpreted as
latin-1 and modified, or when something has incorrectly tried to do
utf8<->latin-1 conversions more than once. You really need to figure out
what encoding your input is in, convert it to a known encoding like
utf-8 *once*, and keep it that way.

If you're using Python, which I suspect you might be, the "".decode()
function is useful. For example, I can convert a latin-1 encoded byte
string to a python Unicode string with:

   "somelatin1string".decode("latin-1")

Sometimes you can get away with just "SET client_encoding=latin-1" but
in this case your string data looks like it's been mangled by more than
just a single encoding mis-interpretation, so you'll probably just
silently insert corrupt data by doing that. Don't. Fix your code so it
knows what the text encoding of the input is.

If you are, in fact, using Python, it's a really good idea to always
"".decode() all your inputs so your internal processing is done in
Unicode (UTF-16, in fact). Similarly, Qt programmers should convert
everything to unicode QString as soon as possible and use that for all
internal manipulation. It'll save a lot of pain.

@ALL: Isn't it possible and wise to include an (optional) encoder in pgsql?

On 03/03/11 09:18, Jasmin Dizdarevic wrote:
> @ALL: Isn't it possible and wise to include an (optional) encoder in pgsql?
>
> we're importing a lot of data from textfiles, which are not utf-8. we
> always have to change the encoding in another tool before using COPY.

SET client_encoding

--
Craig Ringer