Re: Small patch to improve safety of utf8_to_unicode(). - Mailing list pgsql-hackers

From Tom Lane
Subject Re: Small patch to improve safety of utf8_to_unicode().
Date
Msg-id 3228110.1783963143@sss.pgh.pa.us
Whole thread
In response to Re: Small patch to improve safety of utf8_to_unicode().  (Jeff Davis <pgsql@j-davis.com>)
Responses Re: Small patch to improve safety of utf8_to_unicode().
List pgsql-hackers
Jeff Davis <pgsql@j-davis.com> writes:
> On Mon, 2026-07-13 at 09:42 -0400, Tom Lane wrote:
>> What I think this is pointing out is that even if the input string's
>> length fits in size_t, the length of the case-converted equivalent
>> string might not.

> From Unicode 17.0 section 5.18.2: "the maximum string expansion as a
> result of case mapping in the Unicode Standard is three". Along with a
> potential for 4X difference in byte length for a single code point in
> UTF8, that makes a maximum of 12X expansion.

Well, there aren't going to be any ASCII characters that produce 12
bytes, and probably not any LATIN1 characters either, so I think we
could assume that the shortest input character that could produce
such a long output would be 3 bytes.  So it might be safe to assume
that the expansion rate is not more than 4X, which would make this
okay (since as you say we can assume the input is < 1GB).  Probably
needs a little bit of inspection of the LATIN1 cases though, ie
could any of those 2-byte characters produce more than 8 bytes?

            regards, tom lane



pgsql-hackers by date:

Previous
From: Jeff Davis
Date:
Subject: Re: Small patch to improve safety of utf8_to_unicode().
Next
From: Tom Lane
Date:
Subject: Re: Small patch to improve safety of utf8_to_unicode().