On Mon, Aug 12, 2024, at 12:47, Joel Jacobson wrote:
>> 2). Attempt to fix the formulae incorporating maxdigits mentioned
>> above. This part really made my brain hurt, and I'm still not quite
>> sure that I've got it right. In particular, it needs double-checking
>> to ensure that it's not losing accuracy in the reduced-rscale case.
>
> To test if there are any differences that actually matter in the result,
> I patched mul_var to log what combinations that occur when running
> the test suite:
I expanded the test to generate 10k different random numerics
for each of the reduced rscale cases.
This actually found some differences,
where the last decimal digit differs by one,
except in one case where the last decimal digit differs by two.
Not sure if this is a real problem though,
since these differences might not affect the result of the SQL-callable functions.
The case found with the smallest rscale adjustment was this one:
-[ RECORD 1 ]------+--------------------------------
var1 | 0.0000000000009873307197037692
var2 | 0.426697279270850
rscale_adjustment | -15
expected | 0.0000000000004212913318381285
numeric_mul_rscale | 0.0000000000004212913318381284
diff | -0.0000000000000000000000000001
Here is a count grouped by diff:
diff | count
--------------+----------
0.000e+00 | 14114384
1.000e-108 | 1
1.000e-211 | 1
1.000e-220 | 2
1.000e-228 | 6
1.000e-232 | 2
1.000e-235 | 1
1.000e-28 | 13
1.000e-36 | 1
1.000e-51 | 2
1.000e-67 | 1
1.000e-68 | 1
1.000e-80 | 1
-1.000e-1024 | 2485
-1.000e-108 | 3
-1.000e-144 | 2520
-1.000e-16 | 2514
-1.000e-228 | 4
-1.000e-232 | 1
-1.000e-27 | 36
-1.000e-28 | 538
-1.000e-32 | 2513
-1.000e-48 | 2473
-1.000e-68 | 1
-1.000e-80 | 2494
-2.000e-16 | 2
(26 rows)
Should I investigate where each reduced rscale case originates from,
and then try to test the actual SQL-callable functions with values
that cause the same inputs to mul_var as the cases found,
or do we feel confident these differences are not problematic?
Regards,
Joel
