Re: [OT] somebody could explain this? - Mailing list pgsql-hackers

From Kevin Grittner
Subject Re: [OT] somebody could explain this?
Date
Msg-id 436B6DB7020000250000048F@gwmta.wicourts.gov
Whole thread Raw
List pgsql-hackers
No, the IEEE formats can not store .1 exactly.  How close it
comes depends on the rest of the number.  For single and
double precision, respectively, the IEEE representations fall
at about:

0.100000001490116119384765625
0.1000000000000000055511151231257827021181583404541015625

Libraries must do some rounding of these approximations when
formatting for display, so the error is rarely seen, except when
conversions to other data types are done via truncation.  The
problem values from the previous example were actually stored
as something close to the following.

0.11999999999999999555910790149937383830547332763671875
0.1499999999999999944488848768742172978818416595458984375

Whole numbers, on the other hand, are always accurate unless
they exceed the number of significant digits for the format.

-Kevin


>>> "Otto Hirr" <otto.hirr@olabinc.com>  >>>
Most notably, the IEEE rep, either single or double, most certainly
has the ability to store the EXACT value for 0.1.



pgsql-hackers by date:

Previous
From: Tom Lane
Date:
Subject: Re: Reducing the overhead of NUMERIC data
Next
From: "Jim C. Nasby"
Date:
Subject: Re: Crash during elog.c...