Re: JSON and unicode surrogate pairs - Mailing list pgsql-hackers

From Andrew Dunstan
Subject Re: JSON and unicode surrogate pairs
Date
Msg-id 51B7292E.3070904@dunslane.net
Whole thread Raw
In response to Re: JSON and unicode surrogate pairs  (Hannu Krosing <hannu@2ndQuadrant.com>)
Responses Re: JSON and unicode surrogate pairs
List pgsql-hackers
On 06/11/2013 09:16 AM, Hannu Krosing wrote:


>>
>> It's a pity that we don't have a non-error producing conversion function
>> (or if we do that I haven't found it). Then we might adopt a rule for
>> processing
>> unicode escapes that said "convert unicode escapes to the database
>> encoding
> only when extracting JSON keys or values to text makes it sense to unescape
> to database encoding.

That's exactly the scenario we are talking about. When emitting JSON the
functions have always emitted unicode escapes as they are in the text,
and will continue to do so.

>
> strings inside JSON itself are by definition utf8


We have deliberately extended that to allow JSON strings to be in any
database server encoding. That was argued back in the 9.2 timeframe and
I am not interested in re-litigating it.

The only issue at hand is how to handle unicode escapes (which in their
string form are pure ASCII) when emitting text strings.

>> if possible, and if not then emit them unchanged." which might be a
>> reasonable
>> compromise.
> I'd opt for "... and if not then emit them quoted". The default should
> be not loosing
> any data.
>
>
>


I don't know what this means at all. Quoted how? Let's say I have a
Latin1 database and have the following JSON string: "\u20AC2.00". In a
UTF8 database the text representation of this is €2.00 - what are you
saying it should be in the Latin1 database?

cheers

andrew



pgsql-hackers by date:

Previous
From: Hannu Krosing
Date:
Subject: Re: JSON and unicode surrogate pairs
Next
From: Tom Lane
Date:
Subject: Re: how to find out whether a view is updatable