Re: Floating point error - Mailing list pgsql-general

From James Cloos
Subject Re: Floating point error
Date
Msg-id m3r4k07wu8.fsf@carbon.jhcloos.org
Whole thread Raw
In response to Floating point error  (Tom Duffey <tduffey@trillitech.com>)
Responses Re: Floating point error
List pgsql-general
>>>>> "TD" == Tom Duffey <tduffey@trillitech.com> writes:

TD> Riddle me this. I have a database column of type "real" that gets
TD> mapped to a Java field of type double via JDBC. ...

TD> - Selecting values from both test and production DBs using psql
TD>   shows "10.3885" as the value

TD> - The Java app on production shows "10.3884573" while the test app
TD>   shows "10.3885"

I suspect the issue is that psql(1) and whatever java method you use to
convert the floats to text choose different rounding.

By default, it seems that psql(1) uses something like printf("%.4f",...)
whereas your java app calls a routing which works more like "%.7f".

(The wire format for floats is the same as they are stored, not a text
representation thereof.)

-JimC
--
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

pgsql-general by date:

Previous
From: Chris Hanks
Date:
Subject: Re: Poor performance when using a window function in a view
Next
From: Tom Duffey
Date:
Subject: Re: Floating point error