Re: Possible rounding error of large float values? - Mailing list pgsql-sql

From Oliver Elphick
Subject Re: Possible rounding error of large float values?
Date
Msg-id 1093378875.15248.39.camel@linda
Whole thread Raw
In response to Re: Possible rounding error of large float values?  (Josh Berkus <josh@agliodbs.com>)
Responses Re: Possible rounding error of large float values?
List pgsql-sql
On Tue, 2004-08-24 at 20:52, Josh Berkus wrote:
> Adam,
> 
> > 9223372036854775807
> >
> > It gets selected out as:
> >
> > 9.22337203685478E18
> 
> This is a property of FLOAT data types.  They round.
> 
> > Which appears to be rounded. When we cast it to numeric type we get:
> >
> > 9223372036854780000
> >
> > Which also is rounded. It is still possible to find the row using the
> > original value
> 
> Hmmm ... is 15 digits the limit of NUMERIC?   It may be.

It must be the limit of float.  Numeric can't produce any more than it
was given and it was cast from float.

bray=# select 653596708775675750507850507570708696432 ::numeric;                numeric
-----------------------------------------653596708775675750507850507570708696432
(1 row)

-- 
Oliver Elphick                                          olly@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
GPG: 1024D/A54310EA  92C8 39E7 280E 3631 3F0E  1EC0 5664 7A2F A543 10EA
========================================   "I saw in the night visions, and, behold, one like the      Son of man came
withthe clouds of heaven, and came to     the Ancient of days, and they brought him near before      him. And there was
givenhim dominion, and glory, and      a kingdom, that all people, nations, and languages,      should serve him; his
dominionis an everlasting      dominion, which shall not pass away, and his kingdom      that which shall not be
destroyed."                                       Daniel 7:13,14
 



pgsql-sql by date:

Previous
From: Josh Berkus
Date:
Subject: Re: Possible rounding error of large float values?
Next
From: Jan Wieck
Date:
Subject: Re: Possible rounding error of large float values?