Re: WIP: rewrite numeric division - Mailing list pgsql-patches

From Tom Lane
Subject Re: WIP: rewrite numeric division
Date
Msg-id 3829.1184696113@sss.pgh.pa.us
Whole thread Raw
In response to Re: WIP: rewrite numeric division  (Gregory Stark <stark@enterprisedb.com>)
Responses Re: WIP: rewrite numeric division
List pgsql-patches
Gregory Stark <stark@enterprisedb.com> writes:
> "Tom Lane" <tgl@sss.pgh.pa.us> writes:
>> Well, this doesn't take a lot of numerical methods background: the
>> fundamental problem is that the existing code generates an *approximate*
>> answer, whereas people who are doing div and mod on large integers tend
>> to expect an *exact* answer.  Approximate doesn't cut it --- there will
>> always be cases where an off-by-one-in-the-last-internal-place error can
>> carry far enough to the left to be visible to the user.

> So does your code behave differently for this or does it round off to the
> arbitrary division precision before hitting trunc?

> postgres=# select trunc(99999999999999999999::numeric / 1000000000::numeric);
>     trunc
> --------------
>  100000000000
> (1 row)

No, my proposed patch doesn't change that.  It might be that we should
provide an "integer division" operator for NUMERIC, so that you can get
at the exact result of trunc(x/y).

            regards, tom lane

pgsql-patches by date:

Previous
From: Magnus Hagander
Date:
Subject: Re: [HACKERS] msvc, build and install with cygwin in the PATH
Next
From: "Simon Riggs"
Date:
Subject: Re: Async Commit, v21 (now: v22)