Re: invalidly encoded strings - Mailing list pgsql-hackers

From Tatsuo Ishii
Subject Re: invalidly encoded strings
Date
Msg-id 20070911.161706.26986487.t-ishii@sraoss.co.jp
Whole thread Raw
In response to Re: invalidly encoded strings  (Jeff Davis <pgsql@j-davis.com>)
Responses Re: invalidly encoded strings
List pgsql-hackers
> On Tue, 2007-09-11 at 14:50 +0900, Tatsuo Ishii wrote:
> > 
> > > On Tue, 2007-09-11 at 12:29 +0900, Tatsuo Ishii wrote:
> > > > Please show me concrete examples how I could introduce a
> > vulnerability
> > > > using this kind of convert() usage.
> > > 
> > > Try the sequence below. Then, try to dump and then reload the
> > database.
> > > When you try to reload it, you will get an error:
> > > 
> > > ERROR:  invalid byte sequence for encoding "UTF8": 0xbd
> > 
> > I know this could be a problem (like chr() with invalid byte pattern).
> > What I really want to know is, read query something like this:
> > 
> > SELECT * FROM japanese_table ORDER BY convert(japanese_text using
> > utf8_to_euc_jp);
> 
> I guess I don't quite understand the question.
> 
> I agree that ORDER BY convert() must be safe in the C locale, because it
> just passes the strings to strcmp(). 
> 
> Are you saying that we should not remove convert() until we can support
> multiple locales in one database?
>
> If we make convert() operate on bytea and return bytea, as Tom
> suggested, would that solve your use case?

The problem is, the above use case is just one of what I can think of.

Another use case is, something like this:

SELECT sum(octet_length(convert(text_column using utf8_to_euc_jp))) FROM mytable;

to know the total byte length of text column if it's encoded in
EUC_JP. 

So I'm not sure we could change convert() returning bytea without
complaing from users...
--
Tatsuo Ishii
SRA OSS, Inc. Japan


pgsql-hackers by date:

Previous
From: db@zigo.dhs.org
Date:
Subject: Re: invalidly encoded strings
Next
From: Oleg Bartunov
Date:
Subject: Re: Ts_rank internals