Re: Optimize numeric multiplication for one and two base-NBASE digit multiplicands. - Mailing list pgsql-hackers

On Tue, Jul 2, 2024, at 18:20, Joel Jacobson wrote:
> * v3-optimize-numeric-mul_var-small-var1-arbitrary-var2.patch

Hmm, v3 contains a bug which I haven't been able to solve yet.
Reporting now to avoid time waste reviewing it since it's buggy.

The attached patch is how I tested and found the bug.
It contains a file test-mul-var.sql, which tests mul_var()
with varying rscale, using the SQL-callable numeric_mul_patched(),
which third argument is the rscale_adjustment.

Out of 2481600 random tests, all passed except 4:

SELECT
     result = numeric_mul_patched(var1,var2,rscale_adjustment),
     COUNT(*)
FROM test_numeric_mul_patched
GROUP BY 1
ORDER BY 1;
 ?column? |  count
----------+---------
 f        |       4
 t        | 2481596
(2 rows)

SELECT
    var1,
    var2,
    var1*var2 AS full_resolution,
    rscale_adjustment,
    result AS expected,
    numeric_mul_patched(var1,var2,rscale_adjustment) AS actual
FROM test_numeric_mul_patched
WHERE result <> numeric_mul_patched(var1,var2,rscale_adjustment);
       var1        |      var2      |      full_resolution      | rscale_adjustment | expected | actual
-------------------+----------------+---------------------------+-------------------+----------+--------
     676.797214075 | 0.308068877759 | 208.500158210502929257925 |               -21 |      209 |    208
         555.07029 | 0.381033094735 |     211.50015039415392315 |               -17 |      212 |    211
    0.476637718921 |      66.088276 |     31.500165120061470196 |               -18 |       32 |     31
 0.060569165063082 |      998.85933 |   60.50007563356949425506 |               -20 |       61 |     60
(4 rows)

Trying to wrap my head around what could cause this.

It's rounding down instead of up, and these cases all end with decimal .500XXXX.

Regards,
Joel