Re: Invalid byte sequence for encoding "UTF8": 0xedbebf - Mailing list pgsql-general

From Albe Laurenz
Subject Re: Invalid byte sequence for encoding "UTF8": 0xedbebf
Date
Msg-id D960CB61B694CF459DCFB4B0128514C2068E598C@exadv11.host.magwien.gv.at
Whole thread Raw
In response to Re: Invalid byte sequence for encoding "UTF8": 0xedbebf  (BRUSSER Michael <Michael.BRUSSER@3ds.com>)
List pgsql-general
BRUSSER Michael wrote:
>>> Is there a way to find the records with the text field containing
Unicode bytes "0xedbebf"?
>>> Unfortunately this is a very old version 7.3.10
>>
>> This should work on 7.3 (according to the documentation):
>> SELECT id FROM nlsdata WHERE position('\360\235\204\236'::bytea IN
val::bytea) = 1;

> Albe, thanks for pointing this out!

> I made a minor change, added decode since text cannot be cast to bytea
and tried something like this:
>  SELECT id FROM myTable WHERE position('\360\235\204\236'::bytea IN
decode(myTextField, 'escape')) !=0
>  ERROR:  decode: Bad input string for type bytea

Hrm. I didn't know that there was no cast from text to bytea in 7.3.

> Maybe this explains why?
> testdb=# select decode('\360\235\204\236'::text, 'escape');
> ERROR:  Unicode >= 0x10000 is not supported

No, that is an error on my side. I gave you the wrong byte sequence.

For "0xedbebf" you should actually write '\355\276\277'. But that's no
valid UTF-8 sequence.

> but I'm not ready to give up yet...

If you know the byte sequence that causes trouble, you could also use
something like "sed"
to search and replace it in the dump file.

Or (if there are not too many) you could search for the pattern and
identify the rows
in the database. Then you know which database rows to update.

Yours,
Laurenz Albe

pgsql-general by date:

Previous
From: Alban Hertroys
Date:
Subject: Re: PostgreSQL 8.4.8 bringing my website down every evening
Next
From: "Albe Laurenz"
Date:
Subject: Re: Postgres performance and the Linux scheduler