Re: Optimize mul_var() for var1ndigits >= 8 - Mailing list pgsql-hackers

On Mon, Aug 12, 2024, at 17:14, Joel Jacobson wrote:
> The case found with the smallest rscale adjustment was this one:
> -[ RECORD 1 ]------+--------------------------------
> var1               | 0.0000000000009873307197037692
> var2               | 0.426697279270850
> rscale_adjustment  | -15
> expected           | 0.0000000000004212913318381285
> numeric_mul_rscale | 0.0000000000004212913318381284
> diff               | -0.0000000000000000000000000001

To avoid confusion, correction: I mean "largest", since rscale_adjustment is less than or equal to zero.

Here is a group by rscale_adjustment to get a better picture:

SELECT
    rscale_adjustment,
    COUNT(*)
FROM
    test_numeric_mul_rscale,
    numeric_mul_rscale(var1, var2, rscale_adjustment)
WHERE numeric_mul_rscale IS DISTINCT FROM expected
GROUP BY rscale_adjustment
ORDER BY rscale_adjustment;

 rscale_adjustment | count
-------------------+-------
              -237 |     2
              -235 |     1
              -232 |     3
              -229 |     2
              -228 |     8
              -218 |     1
              -108 |     4
               -77 |     1
               -67 |     1
               -51 |     2
               -38 |     3
               -36 |     1
               -28 |     5
               -22 |    42
               -17 |     7
               -16 | 14959
               -15 |   574
(17 rows)

Regards,
Joel