Re: Modulus operator returns negative values / numeric division rounds up sometimes - Mailing list pgsql-general

From Tom Lane
Subject Re: Modulus operator returns negative values / numeric division rounds up sometimes
Date
Msg-id 23606.1117084680@sss.pgh.pa.us
Whole thread Raw
In response to Modulus operator returns negative values / numeric division rounds up sometimes  (Paul Tillotson <pntil@shentel.net>)
Responses Re: Modulus operator returns negative values / numeric
List pgsql-general
Paul Tillotson <pntil@shentel.net> writes:
> I don't think anyone wants to defend the negative modulus as such, but to fix it, we have to do one of these:

> (1) Keep rounding division, but rewrite the numeric modulus operator to use a form of division that always rounds
towardszero. 

> or

> (2) Give up rounding division in favor of truncating towards zero.

or (3) increase the calculation precision (rscale), as suggested by
Alvaro's message.

Possibly that cannot work, but I haven't seen a proof.

> It looks like the "bug" can be easily fixed by changing the end of div_var where it says
>     round_var(result, rscale);
> to
>     trunc_var(result, scale);

I cannot believe that that won't create problems at least as bad as it
solves.  Have you even tried the regression tests on this?

            regards, tom lane

pgsql-general by date:

Previous
From: Paul Tillotson
Date:
Subject: Modulus operator returns negative values / numeric division rounds up sometimes
Next
From: Dave E Martin
Date:
Subject: enable_sort optimization problem