Implementing full UTF-8 support (aka supporting 0x00) - Mailing list pgsql-hackers

From Álvaro Hernández Tortosa
Subject Implementing full UTF-8 support (aka supporting 0x00)
Date
Msg-id 5aa1df8a-96f5-1d14-46fd-032e32846c71@8kdata.com
Whole thread Raw
Responses Re: Implementing full UTF-8 support (aka supporting 0x00)  (Tom Lane <tgl@sss.pgh.pa.us>)
Re: Implementing full UTF-8 support (aka supporting 0x00)  (Kevin Grittner <kgrittn@gmail.com>)
Re: Implementing full UTF-8 support (aka supporting 0x00)  (Geoff Winkless <pgsqladmin@geoff.dj>)
Re: Implementing full UTF-8 support (aka supporting 0x00)  (Craig Ringer <craig@2ndquadrant.com>)
List pgsql-hackers
    Hi list.
    As has been previously discussed (see 
https://www.postgresql.org/message-id/BAY7-F17FFE0E324AB3B642C547E96890%40phx.gbl 
for instance) varlena fields cannot accept the literal 0x00 value. Sure, 
you can use bytea, but this hardly a good solution. The problem seems to 
be hitting some use cases, like:

- People migrating data from other databases (apart from PostgreSQL, I 
don't know of any other database which suffers the same problem).
- People using drivers which use UTF-8 or equivalent encodings by 
default (Java for example)
    Given that 0x00 is a perfectly legal UTF-8 character, I conclude 
we're strictly non-compliant. And given the general Postgres policy 
regarding standards compliance and the people being hit by this, I think 
it should be addressed. Specially since all the usual fixes are a real 
PITA (re-parsing, re-generating strings, which is very expensive, or 
dropping data).
    What would it take to support it? Isn't the varlena header 
propagated everywhere, which could help infer the real length of the 
string? Any pointers or suggestions would be welcome.
    Thanks,
    Álvaro


-- 

Álvaro Hernández Tortosa


-----------
8Kdata




pgsql-hackers by date:

Previous
From: Tom Lane
Date:
Subject: Re: regression test for extended query protocol
Next
From: Tom Lane
Date:
Subject: Re: Wanting to learn about pgsql design decision