Re: intarray: fix an edge case int32 overflow bug - Mailing list pgsql-hackers

From Chao Li
Subject Re: intarray: fix an edge case int32 overflow bug
Date
Msg-id EEAA5E92-AFB9-4A84-86BA-E8A5D1977739@gmail.com
Whole thread Raw
In response to Re: intarray: fix an edge case int32 overflow bug  (David Rowley <dgrowleyml@gmail.com>)
List pgsql-hackers

> On Jan 4, 2026, at 14:28, David Rowley <dgrowleyml@gmail.com> wrote:
>
> On Sun, 4 Jan 2026 at 16:20, Chao Li <li.evan.chao@gmail.com> wrote:
>> I noticed an int32 overflow problem in intarray’s compare_val_int4():
>> ```
>> /*
>> * Comparison function for binary search in mcelem array.
>> */
>> static int
>> compare_val_int4(const void *a, const void *b)
>> {
>>    int32       key = *(int32 *) a;
>>    const Datum *t = (const Datum *) b;
>>
>>    return key - DatumGetInt32(*t);
>> }
>> ```
>>
>> As this function is a bsearch comparator, it is supposed to return >0, =0 or <0. However this function uses
subtractionwith two int32 and returns an int, which may result in an overflow. Say, key is INT32_MAX and *t is -1, the
returnvalue will be negative due to overflow. 
>
> Nice find. Was that found by a static analyser or by eye?
>
> I can take care of the overflow issue. I feel the test is a step too
> far as it seems unlikely ever to be rebroken, but thanks for the
> SQL-based test case to demonstrate the issue.
>
> David

Hi David,

It was spotted by eye. As a newcomer, I’m trying to get more familiar with the codebase, so while reviewing other
patchesI’ve been in the habit of poking around related files. In this case, the comparison function looked error-prone,
soI verified the overflow scenario with the small program. I didn’t post this one too quickly because I spent time
creatingthe test. :) 

I added the test to demonstrate the issue and to prove the fix. If you think including the test is unnecessary and
preferto just take the fix, that’s absolutely fine with me. 

Thanks again for taking care of this.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/







pgsql-hackers by date:

Previous
From: David Rowley
Date:
Subject: Re: intarray: fix an edge case int32 overflow bug
Next
From: David Rowley
Date:
Subject: Re: Correction of RowMark Removal During Sel-Join Elimination