Thread: invalid byte sequence for encoding "UTF8": 0xff

On Mon, Sep 03, 2007 at 01:36:58PM +0530, Ashish Karalkar wrote:
> Hello All,
>
> I have a data script which runs fine from PgAdmin SQL Editor,but when I  run this  from command prompt I get
followingerror: 
> test=# \i /usr/local/pgsql/qsweb1/QSWEB_100_4_Default_Data.sql
>
> psql:/usr/local/pgsql/qsweb1/QSWEB_100_4_Default_Data.sql:1: ERROR:  invalid byt
> e sequence for encoding "UTF8": 0xff
> HINT:  This error can also happen if the byte sequence does not match the encodi
> ng expected by the server, which is controlled by "client_encoding".

Well, the error is correct, that's not a valid UTF-8 character. I seem
to remember someone saying that ooasionally windows puts BOMs in UTF-8
files (which is completely bogus). Check the file using a simple text
editor a check if there are some odd characters at the beginning of the
file.

Have a ncie day,
--
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> From each according to his ability. To each according to his ability to litigate.

Ashish Karalkar wrote:
> I have a data script which runs fine from PgAdmin SQL
> Editor,but when I  run this  from command prompt I get
> following error:
>
>
> test=# \i /usr/local/pgsql/qsweb1/QSWEB_100_4_Default_Data.sql
>
> psql:/usr/local/pgsql/qsweb1/QSWEB_100_4_Default_Data.sql:1:
> ERROR:  invalid byte sequence for encoding "UTF8": 0xff
> HINT:  This error can also happen if the byte sequence does
> not match the encoding expected by the server, which is
> controlled by "client_encoding".
>
> can anybody suggest me what is going wrong.
> database  encoding :UTF8
>
> PostgreSQL details:
>
>  version
> --------------------------------------------------------------
>  PostgreSQL 8.2.0 on i686-pc-linux-gnu, compiled by GCC gcc
> (GCC) 3.4.3 20041212 (Red Hat 3.4.3-9.EL4)

Can you tell us the following:

- What is the client operating system (where you run psql and
  PgAdmin III)?
- What is the value of the environment variable PGCLIENTENCODING
  set to on the client?
- What does the SQL command "show client_encoding;" return
  when you issue it in
  a) PgAdmin III
  b) psql
- Please create a file that contains only the first line
  of QSWEB_100_4_Default_Data.sql (I call it "l" in the following
  commands), run the following two (Linux) commands on it:
  a) od -t c l
  b) od -t x1 l
  and show us the output of both commands.

Yours,
Laurenz Albe

----- Original Message -----
From: "Ashish Karalkar" <ashish.karalkar@info-spectrum.com>
To: "Albe Laurenz" <all@adv.magwien.gv.at>
Sent: Monday, September 03, 2007 4:09 PM
Subject: Re: [GENERAL] invalid byte sequence for encoding "UTF8": 0xff


>
> ----- Original Message -----
> From: "Albe Laurenz" <all@adv.magwien.gv.at>
> To: "Ashish Karalkar *EXTERN*" <ashish.karalkar@info-spectrum.com>;
> <pgsql-general@postgresql.org>
> Sent: Monday, September 03, 2007 2:12 PM
> Subject: Re: [GENERAL] invalid byte sequence for encoding "UTF8": 0xff
>
>
> Ashish Karalkar wrote:
>> I have a data script which runs fine from PgAdmin SQL
>> Editor,but when I  run this  from command prompt I get
>> following error:
>>
>>
>> test=# \i /usr/local/pgsql/qsweb1/QSWEB_100_4_Default_Data.sql
>>
>> psql:/usr/local/pgsql/qsweb1/QSWEB_100_4_Default_Data.sql:1:
>> ERROR:  invalid byte sequence for encoding "UTF8": 0xff
>> HINT:  This error can also happen if the byte sequence does
>> not match the encoding expected by the server, which is
>> controlled by "client_encoding".
>>
>> can anybody suggest me what is going wrong.
>> database  encoding :UTF8
>>
>> PostgreSQL details:
>>
>>  version
>> --------------------------------------------------------------
>>  PostgreSQL 8.2.0 on i686-pc-linux-gnu, compiled by GCC gcc
>> (GCC) 3.4.3 20041212 (Red Hat 3.4.3-9.EL4)
>
> Can you tell us the following:
>
> Please find my answer below
>
> - What is the client operating system (where you run psql and
>  PgAdmin III)?
>
> Its Windows XP - PgAdmin III
> RHEL 3.4.3-9.EL4-psql (Server Machine)
>
>
> - What is the value of the environment variable PGCLIENTENCODING
>  set to on the client?
> PGCLIENTENCODING is not set and as per documantation I think by default it
> takes value of database i.e. UTF8
>
> - What does the SQL command "show client_encoding;" return
>  when you issue it in
>  a) PgAdmin III
> UNICODE
>  b) psql
> UTF8
>
> - Please create a file that contains only the first line
>  of QSWEB_100_4_Default_Data.sql (I call it "l" in the following
>  commands), run the following two (Linux) commands on it:
>  a) od -t c l
>  b) od -t x1 l
>  and show us the output of both commands.
>
>
> [root@localhost qsweb]# od -t c test.sql
> 0000000   \   s   e   t       O   N   _   E   R   R   O   R   _   S   T
> 0000020   O   P
> 0000022
> [root@localhost qsweb]# od -t x1 test.sql
> 0000000 5c 73 65 74 20 4f 4e 5f 45 52 52 4f 52 5f 53 54
> 0000020 4f 50
> 0000022
> [root@localhost qsweb]#
>
> Thanks Albe for your replay.
> here is the data you wanted
>
>
> Yours,
> Laurenz Albe
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
>               http://archives.postgresql.org/

Ashish Karalkar wrote:
>>> I have a data script which runs fine from PgAdmin SQL
>>> Editor,but when I  run this  from command prompt I get
>>> following error:
>>>
>>> test=# \i /usr/local/pgsql/qsweb1/QSWEB_100_4_Default_Data.sql
>>>
>>> psql:/usr/local/pgsql/qsweb1/QSWEB_100_4_Default_Data.sql:1:
>>> ERROR:  invalid byte sequence for encoding "UTF8": 0xff
>>>
>>>  version
>>> --------------------------------------------------------------
>>>  PostgreSQL 8.2.0 on i686-pc-linux-gnu, compiled by GCC gcc
>>> (GCC) 3.4.3 20041212 (Red Hat 3.4.3-9.EL4)
>>
>> - What is the client operating system (where you run psql and
>>  PgAdmin III)?
>>
> Its Windows XP - PgAdmin III
> RHEL 3.4.3-9.EL4-psql (Server Machine)

So I get it that you run "psql" on Windows XP, right?

> PGCLIENTENCODING is not set and as per documantation I
> think by default it takes value of database i.e. UTF8
>
>> - What does the SQL command "show client_encoding;" return
>>  when you issue it in
>>  a) PgAdmin III
> UNICODE
>>  b) psql
> UTF8

Ok, I suspect that's your problem.
You created QSWEB_100_4_Default_Data.sql by using the "Save" dialog
in PgAdmin III on the Windows machine, right?

Then the file will probably be encoded in Windows-1252.

If your client_encoding is set to UTF8, psql will expect UTF-8
data in the SQL script and complain if it meets wrong ones.

Does the script work as expected when you change the client
encoding to WIN1252?

>> - Please create a file that contains only the first line
>>  of QSWEB_100_4_Default_Data.sql (I call it "l" in the following
>>  commands), run the following two (Linux) commands on it:
>>  a) od -t c l
>>  b) od -t x1 l
>>  and show us the output of both commands.
>
> [root@localhost qsweb]# od -t c test.sql
> 0000000   \   s   e   t       O   N   _   E   R   R   O   R   _   S
T
> 0000020   O   P
> 0000022
> [root@localhost qsweb]# od -t x1 test.sql
> 0000000 5c 73 65 74 20 4f 4e 5f 45 52 52 4f 52 5f 53 54
> 0000020 4f 50
> 0000022

That's weird, because psql complained about line 1.

Maybe you messed something up by extracting the first line.

Try the following:

- Use binary file transfer and transfer the SQL script to a Linux
machine.

- Run "od -t c -t x1" on the file

- Find the 0xff that psql complains about.

Maybe that helps to locate the problem.
0xff is an unusual Windows-1252 character as well...

Yours,
Laurenz Albe

----- Original Message -----
From: "Albe Laurenz" <all@adv.magwien.gv.at>
To: "Ashish Karalkar *EXTERN*" <ashish.karalkar@info-spectrum.com>
Cc: <pgsql-general@postgresql.org>
Sent: Monday, September 03, 2007 4:54 PM
Subject: RE: [GENERAL] invalid byte sequence for encoding "UTF8": 0xff


Ashish Karalkar wrote:
>>> I have a data script which runs fine from PgAdmin SQL
>>> Editor,but when I  run this  from command prompt I get
>>> following error:
>>>
>>> test=# \i /usr/local/pgsql/qsweb1/QSWEB_100_4_Default_Data.sql
>>>
>>> psql:/usr/local/pgsql/qsweb1/QSWEB_100_4_Default_Data.sql:1:
>>> ERROR:  invalid byte sequence for encoding "UTF8": 0xff
>>>
>>>  version
>>> --------------------------------------------------------------
>>>  PostgreSQL 8.2.0 on i686-pc-linux-gnu, compiled by GCC gcc
>>> (GCC) 3.4.3 20041212 (Red Hat 3.4.3-9.EL4)
>>
>> - What is the client operating system (where you run psql and
>>  PgAdmin III)?
>>
> Its Windows XP - PgAdmin III
> RHEL 3.4.3-9.EL4-psql (Server Machine)

So I get it that you run "psql" on Windows XP, right?

> PGCLIENTENCODING is not set and as per documantation I
> think by default it takes value of database i.e. UTF8
>
>> - What does the SQL command "show client_encoding;" return
>>  when you issue it in
>>  a) PgAdmin III
> UNICODE
>>  b) psql
> UTF8

Ok, I suspect that's your problem.
You created QSWEB_100_4_Default_Data.sql by using the "Save" dialog
in PgAdmin III on the Windows machine, right?

Then the file will probably be encoded in Windows-1252.

If your client_encoding is set to UTF8, psql will expect UTF-8
data in the SQL script and complain if it meets wrong ones.

Does the script work as expected when you change the client
encoding to WIN1252?

>> - Please create a file that contains only the first line
>>  of QSWEB_100_4_Default_Data.sql (I call it "l" in the following
>>  commands), run the following two (Linux) commands on it:
>>  a) od -t c l
>>  b) od -t x1 l
>>  and show us the output of both commands.
>
> [root@localhost qsweb]# od -t c test.sql
> 0000000   \   s   e   t       O   N   _   E   R   R   O   R   _   S
T
> 0000020   O   P
> 0000022
> [root@localhost qsweb]# od -t x1 test.sql
> 0000000 5c 73 65 74 20 4f 4e 5f 45 52 52 4f 52 5f 53 54
> 0000020 4f 50
> 0000022

That's weird, because psql complained about line 1.

Maybe you messed something up by extracting the first line.

Try the following:

- Use binary file transfer and transfer the SQL script to a Linux
machine.

- Run "od -t c -t x1" on the file

- Find the 0xff that psql complains about.

Maybe that helps to locate the problem.
0xff is an unusual Windows-1252 character as well...

Hey Thanks Albe it worked.


Yours,
Laurenz Albe