Re: rounding problems - Mailing list pgsql-general

From Sam Mason
Subject Re: rounding problems
Date
Msg-id 20080514162630.GG1657@frubble.xen.chris-lamb.co.uk
Whole thread Raw
In response to Re: rounding problems  (Justin <justin@emproshunts.com>)
Responses Re: rounding problems
List pgsql-general
On Wed, May 14, 2008 at 11:47:52AM -0400, Justin wrote:
> I have forgotten how much i hate C++

What we're talking about doesn't have much to do with C++, it's floating
point maths in general.

> Its not doing what you say it would but it did do other odd ball
> things.  I miss my foxpro :-(.

What does foxpro use for storing numbers? or is it just that you never
pushed it hard enough for the abstractions to show through.

> Plus its not holding 15 precision points

after changing the output to be:

  printf("%.10f %.10f\n", d, d-c);

I get:

  100000000.0999999940 0.0999999940
  100000000.0100000054 0.0100000054
  100000000.0010000020 0.0010000020
  100000000.0001000017 0.0001000017
  100000000.0000099987 0.0000099987
  100000000.0000009984 0.0000009984
  100000000.0000001043 0.0000001043
  100000000.0000000149 0.0000000149
  100000000.0000000000 0.0000000000

Which looks reasonable.  Remember that floating point numbers store
their state in base two, not base ten.  All of those numbers look good
to 15 decimal digits.


  Sam

pgsql-general by date:

Previous
From: Decibel!
Date:
Subject: Re: Request for Materialized Views
Next
From: Glyn Astill
Date:
Subject: Re: postgres crash when select a record