Re: my float function returns NaN - Mailing list pgsql-novice

From Tom Lane
Subject Re: my float function returns NaN
Date
Msg-id 2542.1333810462@sss.pgh.pa.us
Whole thread Raw
In response to Re: my float function returns NaN  (Nathaniel Trellice <naptrel@yahoo.co.uk>)
Responses Re: my float function returns NaN  (Frank Bax <fbax@sympatico.ca>)
List pgsql-novice
Nathaniel Trellice <naptrel@yahoo.co.uk> writes:
> My guess would be that due to rounding errors, the argument passed to
> acos() is outwith the interval [-1,1] and acos() is only defined
> within this interval (for reals).

Yeah.  It's not helping any that you've defined geo_deg2rad() to do its
arithmetic in float (that is float4) precision.  Making it work in
float8 would probably improve matters.  Still, I agree that switching to
a more numerically stable distance calculation would be a good idea.

            regards, tom lane

pgsql-novice by date:

Previous
From: Nathaniel Trellice
Date:
Subject: Re: my float function returns NaN
Next
From: Frank Bax
Date:
Subject: Re: my float function returns NaN