Thread: ODBC & character encoding

ODBC & character encoding

From
"Layet Benjamin"
Date:
Hi,
I am trying to run a query from .Net C# using ODBC on a Postgres 7.1 Database.
 
The database encoding is EUC_JP.
And as I use .Net the default encoding is UTF8.
 
My problem is that the 7.1 version does not support to set the client encoding to UTF8 :
Here is the ODBC error log:
<code>
[ Client encoding = 'EUC_JP' (code = 1) ]
conn=105462528, query='set client_encoding to 'UTF8''
ERROR from backend during send_query: 'ERROR:  Client encoding UTF8 is not supported'
</code>
 
So when I insert Data, the japanese characters don't appear properly....
The database is running for a production server and I cannot stop the service at all... Not to mention upgrading...
 
 
So my question is :
 - Can the ODBC driver convert the query string from UTF8 to EUC_JP before sending it to the database server ?
and how ?
 
I hope you understand my problem and my question...
Any help would be extremely appreciated,
Benjamin
 

Re: ODBC & character encoding

From
"Iain"
Date:
Hi,
 
I havn't dne much with unicode on a PG database yet, so this is just a guess (albeit an educated one based on what i read in the docs)
 
Maybe you need to use "set client_encoding to 'UNICODE'" as there didn't appear to be a UTF8 encoding as such.
 
refer to section 20.2.1 of the 7.4 manual: http://www.postgresql.org/docs/7.4/interactive/multibyte.html - don't think this has changed since 7.1, but you should check it.
 
regards
Iain 
----- Original Message -----
Sent: Monday, February 07, 2005 4:04 PM
Subject: [ODBC] ODBC & character encoding

Hi,
I am trying to run a query from .Net C# using ODBC on a Postgres 7.1 Database.
 
The database encoding is EUC_JP.
And as I use .Net the default encoding is UTF8.
 
My problem is that the 7.1 version does not support to set the client encoding to UTF8 :
Here is the ODBC error log:
<code>
[ Client encoding = 'EUC_JP' (code = 1) ]
conn=105462528, query='set client_encoding to 'UTF8''
ERROR from backend during send_query: 'ERROR:  Client encoding UTF8 is not supported'
</code>
 
So when I insert Data, the japanese characters don't appear properly....
The database is running for a production server and I cannot stop the service at all... Not to mention upgrading...
 
 
So my question is :
 - Can the ODBC driver convert the query string from UTF8 to EUC_JP before sending it to the database server ?
and how ?
 
I hope you understand my problem and my question...
Any help would be extremely appreciated,
Benjamin
 

Re: ODBC & character encoding

From
"Layet Benjamin"
Date:
Hi,
Thank you Iain for your quick reply.
But unfortunately this did not solve the problem, 7.1 does not support this conversion neither yet. (7.3 and 7.4 do):
 
SET client_encoding='UNICODE';
ERROR:  Conversion between UNICODE and EUC_JP is not supported
 
In theory, I have 3 possible places to convert UTF8 to EUC_JP:
- 1 inside postgres
- 2 inside ODBC
- 3 inside C#
 
 
1
 
        C#                                    ODBC                 Postgres 7.1
Utf8                  l =============>  l                l =====>l UTF8->EUC_JP
(possible->Unicode)   l     UTF8        l                l UTF8  l
______________________l  (or Unicode)   l________________l       l_______________
 
 
2
        C#                                    ODBC                 Postgres 7.1
Utf8                  l =============>  l UTF8->EUC_JP   l =====>l
(possible->Unicode)   l     UTF8        l                l EUC_JPl
______________________l  (or Unicode)   l________________l       l_______________
 
 
3

        C#                                    ODBC                 Postgres 7.1
UTF8->EUC_JP          l =============>  l                l =====>l
(possible->Unicode)   l     EUC_JP      l                l EUC_JPl
______________________l                 l________________l       l_______________
 
 
Number 1 seem to be the normal solution, but it does not work with 7.1.
I would like to know if number 2 or 3 are possible:
so,
 - Can the ODBC driver convert the query string from UTF8 to EUC_JP before sending it to the database server ?
or
- Can the ODBC driver take EUC_JP string as input ? (then I would 'just' have to figure out how to send EUC_JP stings from C#...) 
 
Help extremely needed...
Thank you !
 
Benjamin
 

Re: ODBC & character encoding

From
"Iain"
Date:
Hi Benjamin,
 
I don't have direct experience of your exact situation, but I use a 7.1 database via ODBC. The difference is that the client is not unicode, it's plain old windows shift-JIS and that works fine iff you use the unicode support odbc driver. When you install the postgres ODBC driver it actually contains 3 drivers, which one are you using?
 
> Can the ODBC driver convert the query string from UTF8 to EUC_JP before sending it to the database server ?
 
I'm not sure...
 
> Can the ODBC driver take EUC_JP string as input ? (then I would 'just' have to figure out how to send EUC_JP stings from C#...) 
 
I dunno about that either, but it certainly accepts shift-JIS which is the native encoding for windows.
 
So far I've never had to pay much attention to client encoding (with the notable exception of getting input data from a web application) as the clients and drivers seem to work everything out by themselves.
 
Regards
Iain
 
 
----- Original Message -----
Sent: Monday, February 07, 2005 5:06 PM
Subject: Re: [ODBC] ODBC & character encoding

Hi,
Thank you Iain for your quick reply.
But unfortunately this did not solve the problem, 7.1 does not support this conversion neither yet. (7.3 and 7.4 do):
 
SET client_encoding='UNICODE';
ERROR:  Conversion between UNICODE and EUC_JP is not supported
 
In theory, I have 3 possible places to convert UTF8 to EUC_JP:
- 1 inside postgres
- 2 inside ODBC
- 3 inside C#
 
 
1
 
        C#                                    ODBC                 Postgres 7.1
Utf8                  l =============>  l                l =====>l UTF8->EUC_JP
(possible->Unicode)   l     UTF8        l                l UTF8  l
______________________l  (or Unicode)   l________________l       l_______________
 
 
2
        C#                                    ODBC                 Postgres 7.1
Utf8                  l =============>  l UTF8->EUC_JP   l =====>l
(possible->Unicode)   l     UTF8        l                l EUC_JPl
______________________l  (or Unicode)   l________________l       l_______________
 
 
3

        C#                                    ODBC                 Postgres 7.1
UTF8->EUC_JP          l =============>  l                l =====>l
(possible->Unicode)   l     EUC_JP      l                l EUC_JPl
______________________l                 l________________l       l_______________
 
 
Number 1 seem to be the normal solution, but it does not work with 7.1.
I would like to know if number 2 or 3 are possible:
so,
 - Can the ODBC driver convert the query string from UTF8 to EUC_JP before sending it to the database server ?
or
- Can the ODBC driver take EUC_JP string as input ? (then I would 'just' have to figure out how to send EUC_JP stings from C#...) 
 
Help extremely needed...
Thank you !
 
Benjamin
 

Re: ODBC & character encoding

From
"Layet Benjamin"
Date:
Hi Iain,
Thank you again for your reply.
It's good to know that things works for people almost in my situation :-).
 
I am not surprise that there is no problem with Shift-Jis, since "SET client_encoding= 'SJIS';" works fine on a EUC_JP database.
 
But "SET client_encoding= 'UNICODE';" does not work with 7.1.... and I feel really stuck because of that...
 
>When you install the postgres ODBC driver it actually contains 3 drivers, which one are you using?
I tried the PostgreSQL 8.00. and PostgreSQL Unicode beta but both of them try to do "SET client_encoding= 'UNICODE'"...
 
In my case, I really need to convert the encoding before sending it to Postgres... but I don't know if it is possible.
 
>So far I've never had to pay much attention to client encoding...
Lucky man :-)
 
Cheers,
Ben
----- Original Message -----
From: Iain
Sent: Monday, February 07, 2005 5:31 PM
Subject: Re: [ODBC] ODBC & character encoding

Hi Benjamin,
 
I don't have direct experience of your exact situation, but I use a 7.1 database via ODBC. The difference is that the client is not unicode, it's plain old windows shift-JIS and that works fine iff you use the unicode support odbc driver. When you install the postgres ODBC driver it actually contains 3 drivers, which one are you using?
 
> Can the ODBC driver convert the query string from UTF8 to EUC_JP before sending it to the database server ?
 
I'm not sure...
 
> Can the ODBC driver take EUC_JP string as input ? (then I would 'just' have to figure out how to send EUC_JP stings from C#...) 
 
I dunno about that either, but it certainly accepts shift-JIS which is the native encoding for windows.
 
So far I've never had to pay much attention to client encoding (with the notable exception of getting input data from a web application) as the clients and drivers seem to work everything out by themselves.
 
Regards
Iain
 
 
----- Original Message -----
Sent: Monday, February 07, 2005 5:06 PM
Subject: Re: [ODBC] ODBC & character encoding

Hi,
Thank you Iain for your quick reply.
But unfortunately this did not solve the problem, 7.1 does not support this conversion neither yet. (7.3 and 7.4 do):
 
SET client_encoding='UNICODE';
ERROR:  Conversion between UNICODE and EUC_JP is not supported
 
In theory, I have 3 possible places to convert UTF8 to EUC_JP:
- 1 inside postgres
- 2 inside ODBC
- 3 inside C#
 
 
1
 
        C#                                    ODBC                 Postgres 7.1
Utf8                  l =============>  l                l =====>l UTF8->EUC_JP
(possible->Unicode)   l     UTF8        l                l UTF8  l
______________________l  (or Unicode)   l________________l       l_______________
 
 
2
        C#                                    ODBC                 Postgres 7.1
Utf8                  l =============>  l UTF8->EUC_JP   l =====>l
(possible->Unicode)   l     UTF8        l                l EUC_JPl
______________________l  (or Unicode)   l________________l       l_______________
 
 
3

        C#                                    ODBC                 Postgres 7.1
UTF8->EUC_JP          l =============>  l                l =====>l
(possible->Unicode)   l     EUC_JP      l                l EUC_JPl
______________________l                 l________________l       l_______________
 
 
Number 1 seem to be the normal solution, but it does not work with 7.1.
I would like to know if number 2 or 3 are possible:
so,
 - Can the ODBC driver convert the query string from UTF8 to EUC_JP before sending it to the database server ?
or
- Can the ODBC driver take EUC_JP string as input ? (then I would 'just' have to figure out how to send EUC_JP stings from C#...) 
 
Help extremely needed...
Thank you !
 
Benjamin