[Moving this to -hackers]
> On 13/10/2022 18:20 CEST Adrian Klaver <adrian.klaver@aklaver.com> wrote:
> > select power(10, -18::numeric);
> > power
> > --------------------
> > 0.0000000000000000
> >
On Thu, 13 Oct 2022 at 18:17, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> An inexact result isn't surprising, but it shouldn't be *that* inexact.
>
> I'm inclined to think that we should push the responsibility for choosing
> its rscale into power_var_int(), because internally that already does
> estimate the result weight, so with a little code re-ordering we won't
> need duplicative estimates. Don't have time to work on that right now
> though ... Dean, are you interested in fixing this?
>
Here's a patch along those lines, bringing power_var_int() more in
line with neighbouring functions by having it choose its own result
scale.
It was necessary to also move the overflow/underflow tests up, in
order to avoid a potential integer overflow when deciding the rscale.
Looking more closely at the upper limit of the overflow test, it turns
out it was far too large. I'm not sure where the "3 * SHRT_MAX" came
from, but I suspect it was just a thinko on my part, back in
7d9a4737c2. I've replaced that with SHRT_MAX + 1, which kicks in much
sooner without changing the actual maximum result allowed, which is <
10^131072 (the absolute upper limit of the numeric type).
The first half the the underflow test condition "f + 1 < -rscale" goes
away, since this is now being done before rscale is computed, and the
choice of rscale makes that condition false. In fact, the new choice
of rscale now ensures that when sig_digits is computed further down,
it is guaranteed to be strictly greater than 0, rather than merely
being >= 0 as before, which is good.
As expected, various regression test results change, since the number
of significant digits computed is now different, but I think the new
results look a lot better, and more consistent. I regenerated the
numeric_big test results by re-running the bc script and rounding to
the new output precisions, and the results from power_var_int()
exactly match in every case. This already included a number of cases
that used to round to zero, and now produce much more reasonable
results.
The test cases where the result actually does round to zero now output
1000 zeros after the decimal point. That looks a little messy, but I
think it's the right thing to do in fixed-point arithmetic -- it's
consistent with the fractional power case, and with exp(numeric),
reflecting the fact that the result is zero to 1000 decimal places,
whilst not being exactly zero.
Overall, I'm quite happy with these results. The question is, should
this be back-patched?
In the past, I think I've only back-patched numeric bug-fixes where
the digits output by the old code were incorrect or an error was
thrown, not changes that resulted in a different number of digits
being output, changing the precision of already-correct results.
However, having 10.0^(-18) produce zero seems pretty bad, so my
inclination is to back-patch, unless anyone objects.
Regards,
Dean
