Re: Fw:Re: Fw: gbt_var_consistent in contrib/btree_gist/btree_utils_var.c has internal-node type confusion on the <> strategy, bypassing exclusion constraints - Mailing list pgsql-bugs

Ayush Tiwari <ayushtiwari.slg01@gmail.com> writes:
> On Thu, 2 Jul 2026 at 20:08, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> None of this passes the smell test for me.  If it's unsafe to
>> call the type's f_eq function on a truncated key, how is it
>> any safer to call the f_cmp function?

> I think the difference is that for bit/varbit f_eq and f_cmp aren't the
> same kind of function.  f_eq is biteq (reads bit_len, does VARSIZE-8),
> while f_cmp is byteacmp (plain byte-wise, only needs the varlena
> header).

Oh!  You are right, the internal keys are *not* valid bit/varbit
values, just bytea.  The level of non-documentation in this code
is staggering.

So the actual API contract here is that f_eq etc. are to be used only
on leaf keys, while f_cmp is to be used only on internal keys.  It's
not too surprising that whoever added NotEqual support didn't know
that, it being documented nowhere.

It kind of looks actually like there's an expectation that all
internal keys are effectively bytea's; note the naming of
gbt_bytea_pf_match, despite the fact that it's applied to all
types under the purview of btree_utils_var.c.

Anyway, I think part of our work here has got to be to raise the
level of documentation of this code.  I've had it with having
to reverse-engineer details as fundamental as these.