Fix type of 'reduction' variable in _bt_singleval_fillfactor() - Mailing list pgsql-hackers

Hi,

While reviewing nbtdedup.c I noticed a minor type mismatch in
_bt_singleval_fillfactor(). The local variable 'reduction' is declared
as 'int', but it holds the result of multiplying 'leftfree' (which is
type Size, i.e. size_t / unsigned long) by a double factor, and is
then subtracted from state->maxpostingsize, which is also Size.

Using a signed int here is semantically incorrect: Size is the
appropriate type for any variable representing a byte count, and
it matches the type of every other variable involved in this
calculation.

While no overflow occurs with current BLCKSZ limits (the product is
at most ~30KB on a standard build, well within INT_MAX), the type
mismatch could silently produce incorrect behaviour on non-standard
builds compiled with a very large BLCKSZ. In that case, if the
product exceeded INT_MAX, 'reduction' would wrap to a large negative
number. The subsequent check:

    if (state->maxpostingsize > reduction)
        state->maxpostingsize -= reduction;

would then subtract a negative value, i.e. increase maxpostingsize
instead of reducing it, silently defeating the single-value fill
strategy entirely.

The fix is a one-line change: declare 'reduction' as Size instead
of int.

A patch file is attached.