Re: Arbitrary precision modulo operation - Mailing list pgsql-general

From Tom Lane
Subject Re: Arbitrary precision modulo operation
Date
Msg-id 23910.1083132060@sss.pgh.pa.us
Whole thread Raw
In response to Re: Arbitrary precision modulo operation  (Paul Tillotson <pntil@shentel.net>)
Responses Re: Arbitrary precision modulo operation
List pgsql-general
Paul Tillotson <pntil@shentel.net> writes:
> mod(x, y) is computed as x - trunc(x / y) * y in the mod_var() function

> However, it appears that the division operator itself is rounding up,

This is because div_var rounds its output to the number of fractional
digits specified by select_div_scale, which saith

    /*
     * The result scale of a division isn't specified in any SQL standard.
     * For PostgreSQL we select a result scale that will give at least
     * NUMERIC_MIN_SIG_DIGITS significant digits, so that numeric gives a
     * result no less accurate than float8; but use a scale not less than
     * either input's display scale.
     */

You get the "correct" answer if you force a couple more digits to be
carried by increasing either input's dscale, eg

regression=# select 123456789012345678901234567890.00 % 123;
 ?column?
----------
   117.00
(1 row)

It could be that select_div_scale needs to allow some more slop, or that
that routine is okay but mod_var shouldn't depend on it to select the
intermediate result scale for its internal division.  Comments?

            regards, tom lane

pgsql-general by date:

Previous
From:
Date:
Subject: Re: shadowing (like IB/Firebird)
Next
From: Bruno Wolff III
Date:
Subject: Re: Arbitrary precision modulo operation