Re: [PATCH] Fix overflow and underflow in regr_r2() - Mailing list pgsql-hackers

From Chengpeng Yan
Subject Re: [PATCH] Fix overflow and underflow in regr_r2()
Date
Msg-id D7866E05-B2B7-4475-94CB-A5C404413F67@outlook.com
Whole thread
In response to Re: [PATCH] Fix overflow and underflow in regr_r2()  (Dean Rasheed <dean.a.rasheed@gmail.com>)
Responses Re: [PATCH] Fix overflow and underflow in regr_r2()
List pgsql-hackers
Hi,

> On May 28, 2026, at 20:37, Dean Rasheed <dean.a.rasheed@gmail.com> wrote:
>
> [Shrug] I think dy is common enough to denote a difference in
> y-values, and it seems clear enough, given the large comment above it.

Makes sense, thanks for the clarification.

>> 3. Do we need to add tests for the underflow path, and perhaps for the
>> Inf/NaN guard?
>
> Yeah, I think it makes sense to include a test with underflow, since
> that really can lead to a large relative error. I don't think it's
> worth testing the Inf/NaN guard, since that's more about avoiding
> operating on technically uninitialised variables, and I don't believe
> that it actually affects the results.
>
> I've add this test case:
>
> SELECT regr_intercept(y, x) FROM (VALUES (-1e-131, 0), (2e-131,
> 3e-131)) v(x, y);
>
> Here, directly computing Sx * Sxy / Sxx causes an underflow to zero,
> while the correct result should be 1e-131. Since Sy is 3e-131, this
> makes a noticeable difference to the final result (without the patch,
> it returns an intercept of 1.5e-131, whereas with the patch, it
> correctly returns 1e-131).
>
> If there are no objections from the RMT, I'll push both of these (to
> HEAD only) in a couple of days or so.
>
> Regards,
> Dean
>
<v2-0001-Improve-overflow-underflow-handling-in-regr_r2.patch><v2-0002-Improve-overflow-underflow-handling-in-regr_inter.patch>

v2 LGTM.

I ran the regression tests locally on Apple Silicon as well, and all
tests passed.

--
Best regards,
Chengpeng Yan



pgsql-hackers by date:

Previous
From: Fujii Masao
Date:
Subject: Re: [PATCH] Release replication slot on error in SQL-callable slot functions
Next
From: solai v
Date:
Subject: Re: Beautify read stream "per buffer data" APIs