Re: Fw:Re: Fw: gbt_var_consistent in contrib/btree_gist/btree_utils_var.c has internal-node type confusion on the <> strategy, bypassing exclusion constraints - Mailing list pgsql-bugs

Thanks for the report!

In gbt_var_consistent() (contrib/btree_gist/btree_utils_var.c), the <>
(BtreeGistNotEqual) branch is the only strategy that does not distinguish
leaf from internal pages. It applies the type's equality function directly
to the stored key on both bounds:

  retval = !(tinfo->f_eq(query, key->lower, ...) &&
           tinfo->f_eq(query, key->upper, ...));

That seems to be the problem: on internal pages those keys are
truncated (and for bit/varbit the length header is gone too), so
they aren't really valid values to hand to the type's equality
function.  Hence the negative length reaching memcmp().

Would it make sense to just have <> follow the same leaf-vs-internal
pattern as the other strategies; negate equality on a leaf, and
otherwise recurse?  An internal page can always contain some
non-equal value, so I don't think we lose anything by descending.
I put together a small patch along those lines (plus a bit(8) test
that goes through internal pages); it's attached if it's useful.
(PS. The test exercises the path but passes on a plain build; the OOB
read shows up under ASan (negative-size-param).)

I didn't touch the fixed-size path in btree_utils_num.c, since its
internal keys look like they're exact lower/upper values, but
please correct me if I'm missing something there.

Regards,
Ayush