Re: [HACKERS] 123.45 - 123 = 0.45 - Mailing list pgsql-hackers

From Theo Kramer
Subject Re: [HACKERS] 123.45 - 123 = 0.45
Date
Msg-id 199812160522.HAA06771@flame.flame.co.za
Whole thread Raw
In response to Re: [HACKERS] 123.45 - 123 = 0.45  (jwieck@debis.com (Jan Wieck))
List pgsql-hackers
>     We  need  to define what the precision of a result should be,
>     if it is not assigned to a column (where the precision can be
>     the  atttypmod).  Is  there any standard defined for? If not,
>     what about this:
>
>     Internal representation holds different precision for DISPLAY
>     and CALC.
>
>     On  any operation, the DISPLAY precision is set to the higher
>     of the two operands.
>
>     On add/subtract, the CALC precision becomes the higher of the
>     two.
>
>     On multiply, the CALC precision is adjusted to hold the exact
>     result up to a (variable settable?) maximum.
>
>     On divide, the CALC precision is set to max and after  it  to
>     the number of used digits.
>
>     If  the  result get's assigned to an attribute, it is rounded
>     to it's atttypmod and both precisions set to that.
>
>     The types output function rounds it to the DISPLAY precision.
>
>     The  input  function  sets  both  precisions to the number of
>     digits present after decimal point.
>
>     Needless to say that there will be special functions to round
>     explicitly and set the precisions.

I have a routine that does the necessary rounding on 8 byte floating points to a
precision up to 8 decimal places. Not exactly based on higher math but it
does the job in many financial applications on all assignments, comparisons
and when displayed.

I never use it on arithmetic operations.

The interface is as follows

double RoundDouble(double value, int decimals);

I am happy to submit it for use in the postgres code.

Regards
Theo

pgsql-hackers by date:

Previous
From: Bruce Momjian
Date:
Subject: Re: [HACKERS] Old parser feature
Next
From: Vadim Mikheev
Date:
Subject: CURRENT: crash in select_view regression test...