Re: Proposal: tighten validation for legacy EUC encodings or document that accepted byte sequences may be unconvertible to UTF8 - Mailing list pgsql-hackers

From David G. Johnston
Subject Re: Proposal: tighten validation for legacy EUC encodings or document that accepted byte sequences may be unconvertible to UTF8
Date
Msg-id CAKFQuwZuEZFYK9Arp_qFsoJ5o2EDDDCfsTwBYvoxzhBiXRJHQg@mail.gmail.com
Whole thread
In response to Re: Proposal: tighten validation for legacy EUC encodings or document that accepted byte sequences may be unconvertible to UTF8  (Zhongpu Chen <chenloveit@gmail.com>)
Responses Re: Proposal: tighten validation for legacy EUC encodings or document that accepted byte sequences may be unconvertible to UTF8
List pgsql-hackers
On Friday, May 1, 2026, Zhongpu Chen <chenloveit@gmail.com> wrote:
The issue is not specific to E'\\x..' literals. A normal COPY FROM data file with ENCODING 'EUC_CN' can create text rows that later cannot be retrieved with SELECT.

 This suggests that input validation for EUC_CN is only structural, while the EUC_CN-to-UTF8 conversion table is stricter.

I suspect a lack of desire to maintain and ensure that specific values are verified; or accepting the runtime cost to do so.  It is indeed structural.  This point should probably be documented better.  But it’s hard to feel too bad if the input claims it is providing verifiable EUC_CN data then proceeds to supply data that lacks meaning in reality.  We are happy to just store and return your data to you - but it’s unreasonable to ask for it to be converted.  It would be nice for the database to provide an extra layer of protection, so I’m not against the idea.  Either automatically or or at least providing a function that could, say, be called in a trigger for opt-in.  But definitely feels like a problematic benefit-to-cost proposition.

David J.

pgsql-hackers by date:

Previous
From: Zhongpu Chen
Date:
Subject: Re: Proposal: tighten validation for legacy EUC encodings or document that accepted byte sequences may be unconvertible to UTF8
Next
From: Tatsuo Ishii
Date:
Subject: Re: Row pattern recognition