- People migrating data from other databases (apart from PostgreSQL, I don't know of any other database which suffers the same problem). - People using drivers which use UTF-8 or equivalent encodings by default (Java for example)
Given that 0x00 is a perfectly legal UTF-8 character, I conclude we're strictly non-compliant. And given the general Postgres policy regarding standards compliance and the people being hit by this, I think it should be addressed. Specially since all the usual fixes are a real PITA (re-parsing, re-generating strings, which is very expensive, or dropping data).
What would it take to support it? Isn't the varlena header propagated everywhere, which could help infer the real length of the string? Any pointers or suggestions would be welcome.
One of the bigger pain points is that our interaction with C library collation routines for sorting uses NULL-terminated C strings. strcoll, strxfrm, etc.
