Re: [sqlsmith] Failed assertion in brin_minmax_multi_distance_float4 on REL_14_STABLE - Mailing list pgsql-hackers

Hi,

On 11/4/21 17:53, Justin Pryzby wrote:
> On Thu, Nov 04, 2021 at 09:46:49AM +0100, Andreas Seltenreich wrote:
>> sqlsmith triggers the following assertion when testing REL_14_STABLE:
>>
>>      TRAP: FailedAssertion("a1 <= a2", File: "brin_minmax_multi.c", Line: 1879, PID: 631814)
>>
>> I can reproduce it with the following query on a fresh regression
>> database:
>>
>>      insert into public.brintest_multi (float4col) values (real 'nan');
>>
>> The branch was at f6162c020c while testing, backtrace below.
> 
> I couldn't reproduce this, but it reminds me of this one, which we also had
> trouble reproducing.
> 

I can reproduce that just fine - all I had to do was 'make installcheck' 
and then connect to the regression db and run the insert.

It seems to be a simple case of confusion in handling of NaN values. We 
do sort them correctly (by calling float4_lt), but given two values

   arg1 = nan (0x400000)
   arg2 = 0.0909090936

then a simple comparison does not give the expected result

   (arg1 < arg2)
   (arg1 == arg2)
   (arg1 > arg2)

all evaluate to false, which is why the assert fails. So I guess the 
distance function for float4 (and probably float8 too) need a couple 
more lines checking NaN.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company