Re: Floating point comparison inconsistencies of the geometric types - Mailing list pgsql-hackers

From Robert Haas
Subject Re: Floating point comparison inconsistencies of the geometric types
Date
Msg-id CA+Tgmoa0uSeM5AR1OTjAZELvV+KKatc51SfSMm18GXMc_013Xg@mail.gmail.com
Whole thread Raw
In response to Floating point comparison inconsistencies of the geometric types  (Emre Hasegeli <emre@hasegeli.com>)
Responses Re: Floating point comparison inconsistencies of the geometric types
List pgsql-hackers
On Fri, May 27, 2016 at 6:43 AM, Emre Hasegeli <emre@hasegeli.com> wrote:
> There are those macros defined for the built-in geometric types:
>
>> #define EPSILON                 1.0E-06
>
>> #define FPzero(A)               (fabs(A) <= EPSILON)
>> #define FPeq(A,B)               (fabs((A) - (B)) <= EPSILON)
>> #define FPne(A,B)               (fabs((A) - (B)) > EPSILON)
>> #define FPlt(A,B)               ((B) - (A) > EPSILON)
>> #define FPle(A,B)               ((A) - (B) <= EPSILON)
>> #define FPgt(A,B)               ((A) - (B) > EPSILON)
>> #define FPge(A,B)               ((B) - (A) <= EPSILON)
>
> with this warning:
>
>>  *    XXX These routines were not written by a numerical analyst.

I agree that those macros looks like a pile of suck.  It's unclear to
me what purpose they're trying to accomplish, but regardless of what
it is, it's hard for me to believe that they are accomplishing it.
Whether 1.0E-06 is a correct fuzz factor presumably depends greatly on
the scale of the number; e.g. if the values are all like "3" it's
probably fine but if they are like "37142816124856" it's probably not
enough fuzz and if they are all like ".00000004" it's probably way too
much fuzz.

Figuring out what to do about it is harder.  Your proposal seems to be
to remove them except where we need the fuzzy behavior, which doesn't
sound unreasonable, but I don't personally understand why we need it
in some places and not others.  It would be good if some of the people
who are more numerically inclined than I am (and hate floats less, but
then that's everyone) could jump in here.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



pgsql-hackers by date:

Previous
From: Robert Haas
Date:
Subject: Re: PostmasterPid not marked with PGDLLIMPORT
Next
From: "David G. Johnston"
Date:
Subject: Re: PostmasterPid not marked with PGDLLIMPORT