Re: rounding problems - Mailing list pgsql-general

From Justin
Subject Re: rounding problems
Date
Msg-id 482B2AAF.1050003@emproshunts.com
Whole thread Raw
In response to Re: rounding problems  (Sam Mason <sam@samason.me.uk>)
Responses Re: rounding problems  (Sam Mason <sam@samason.me.uk>)
List pgsql-general


Sam Mason wrote:
On Wed, May 14, 2008 at 11:47:52AM -0400, Justin wrote: 
I have forgotten how much i hate C++   
What we're talking about doesn't have much to do with C++, it's floating
point maths in general.
 
Its not doing what you say it would but it did do other odd ball 
things.  I miss my foxpro :-(.    
What does foxpro use for storing numbers? or is it just that you never
pushed it hard enough for the abstractions to show through. 
I know i pushed it.  Foxpro  for the most has only  4 basic data types   Numeric (similar to Posgresql numeric), Boolean, Date, Text aka (string)  The foxpro tables supported far more data types but when every it was dumped to variable it acted like one of the 4. 

Foxpro did not suffer floating point math errors.  I normally used 8 to 10 points precision.  Foxpro was limited to 15 points of precision  period.   No more and no less, once you hit that was it.


 
Plus its not holding 15 precision points   
after changing the output to be:
 printf("%.10f %.10f\n", d, d-c);

I get:
 100000000.0999999940 0.0999999940 100000000.0100000054 0.0100000054 100000000.0010000020 0.0010000020 100000000.0001000017 0.0001000017 100000000.0000099987 0.0000099987 100000000.0000009984 0.0000009984 100000000.0000001043 0.0000001043 100000000.0000000149 0.0000000149 100000000.0000000000 0.0000000000

Which looks reasonable.  Remember that floating point numbers store
their state in base two, not base ten.  All of those numbers look good
to 15 decimal digits. 

My problem is we calculate resistance of parts in a Foxpro app that we want to move because we want to bring all the custom apps into one framework and single database.

Take this calculation  (0.05/30000* 1.0025) which is used to calculate parts resistance and Tolerance. (its Ohms Law)  The value returned  from C++ = .0000016708 which is wrong
it should be .00000167418.  We just shrank the tolerance on the part we make

Take the other side  (0.05/30000* .9975) = .0000016625 from C++  this way wrong and the tolerance just grew .00000166583.  Guess what this could result in shipping a $12,000+ out to a customer wrong.

The Documentation from MS says 15 points of precision but the result say otherwise.  I'm glad You and others are taking the time to explain to me the odd results before i get into redoing that application.

Why oh Why did MS kill Foxpro. :'(   I understood it, knew its quirks and it worked very well with Postgresql

pgsql-general by date:

Previous
From: Glyn Astill
Date:
Subject: Re: postgres crash when select a record
Next
From: mailtolouis2020-postgres@yahoo.com
Date:
Subject: Re: postgres crash when select a record