Thread: gmpy adapter
Hello, all,
I have a client app that reads in many values from a numeric column,
does some heavy computations, then writes the results back to another
numeric column.
Python's decimal.Decimal is SLOOOOoooow. I'm trying to use gmpy.mpq
instead. I have the adapter for reading the values from the database
working fine:
numeric2mpq = lambda d,c: None if d is None else gmpy.mpq(d)
MPQ = psycopg2.extensions.new_type((1700,), "MPQ", numeric2mpq)
psycopg2.extensions.register_type(MPQ)
This is the adapter I'm using for the reverse (converting the mpq to a
string suitable for casting to numeric):
def mpq2numeric(mpq):
s = '%d::numeric/%d::numeric' % (mpq.numer(), mpq.denom())
return psycopg2.extensions.AsIs(s)
psycopg2.extensions.register_adapter(gmpy.mpq(0).__class__,
mpq2numeric)
While the adapter works, it seems less than optimal as it creates an
expression for the server to process, e.g:
print psycopg2.extensions.adapt(gmpy.mpq('.333'))
333::numeric/1000::numeric
Questions:
1) Is there something I'm overlooking with gmpy that could make this
much simpler?
2) What other solutions do folk use for working around pythons slow,
slow, slow Decimal?
Thanks,
Dan
On Mon, Feb 28, 2011 at 4:56 PM, Daniel Popowich <danielpopowich@gmail.com> wrote: > 1) Is there something I'm overlooking with gmpy that could make this > much simpler? Why don't you use mpf instead? Arbitrary precision float may map better than rationals to the database decimals. > 2) What other solutions do folk use for working around pythons slow, > slow, slow Decimal? There are around C implementations compatible with Python decimals: http://www.bytereef.org/mpdecimal/index.html for example. Also notice that float64 has 53 bits of precision, 15 full decimal digits. If you don't need more precision when you write data in the database (even if you have used more during calculations) you may just use the repr(float()) of your mpq to write into the database. -- Daniele
On Mon, Feb 28, 2011 at 11:07 PM, Daniele Varrazzo
<daniele.varrazzo@gmail.com> wrote:
> Also notice that float64 has 53 bits of precision, 15 full decimal
> digits. If you don't need more precision when you write data in the
> database (even if you have used more during calculations) you may just
> use the repr(float()) of your mpq to write into the database.
Or else, in the adapter you may use mpf to approximate the rational to
the precision you need.
>>> gmpy.mpq(2,3)
mpq(2,3)
>>> gmpy.mpf(_, 100)
mpf('6.666666666666666666666666666666666666667e-1',100)
>>> str(_)
'0.6666666666666666666666666666666666666667'
-- Daniele
On Tue, Mar 1, 2011 at 5:03 PM, Daniel Popowich <danielpopowich@gmail.com> wrote: > Thanks for the ideas. One thing I didn't specify in my original post > is that the numeric columnss in my database are for monetary values, > so I can't use binary floating point representations in my python > code, else I risk inexactness and accumulated error in long > computations. Yes, of course: this is the normal reason to avoid floating point numbers and go for fixed point. > I need to use decimal floating point, e.g., decimal.Decimal (or > cdecimal for a C implementation) or a rational object, like > fractions.Fraction (or gmpy.mpq for a C impl). > > I could use mpq for internal computations then use mpf for the final > conversion to the string I need for my adapter, but I want a single > type I can use throughout my code without such concerns. > > I'm going to try cdecimal for now. If cdecimal is good for you (i.e. if the decimal semantics is fine), you may use mpf throughout your program instead of mpq. It is an exact type (as exact as decimal of course: you can't represent all the rationals) and likely to be more efficient than mpq. Plus, it maps directly with the postgres decimal on I/O. cdecimal may make your life easier instead if you have already working code using the Python Decimal and you want to speed it up. -- Daniele
Daniele Varrazzo writes:
> On Mon, Feb 28, 2011 at 11:07 PM, Daniele Varrazzo
> <daniele.varrazzo@gmail.com> wrote:
>
> > Also notice that float64 has 53 bits of precision, 15 full decimal
> > digits. If you don't need more precision when you write data in the
> > database (even if you have used more during calculations) you may just
> > use the repr(float()) of your mpq to write into the database.
>
> Or else, in the adapter you may use mpf to approximate the rational to
> the precision you need.
>
> >>> gmpy.mpq(2,3)
> mpq(2,3)
> >>> gmpy.mpf(_, 100)
> mpf('6.666666666666666666666666666666666666667e-1',100)
> >>> str(_)
> '0.6666666666666666666666666666666666666667'
Daniele,
Thanks for the ideas. One thing I didn't specify in my original post
is that the numeric columnss in my database are for monetary values,
so I can't use binary floating point representations in my python
code, else I risk inexactness and accumulated error in long
computations.
I need to use decimal floating point, e.g., decimal.Decimal (or
cdecimal for a C implementation) or a rational object, like
fractions.Fraction (or gmpy.mpq for a C impl).
I could use mpq for internal computations then use mpf for the final
conversion to the string I need for my adapter, but I want a single
type I can use throughout my code without such concerns.
I'm going to try cdecimal for now.
Thanks, again!
Cheers,
Dan
Daniele Varrazzo writes: > If cdecimal is good for you (i.e. if the decimal semantics is fine), > you may use mpf throughout your program instead of mpq. It is an > exact type (as exact as decimal of course: you can't represent all > the rationals) and likely to be more efficient than mpq. Plus, it > maps directly with the postgres decimal on I/O. > Really? But it's binary floating point. From the GMP manual, http://gmplib.org/manual-4.3.2/Floating_002dpoint-Functions.html: The mantissa in stored in binary, as might be imagined from the fact precisions are expressed in bits. One consequence of this is that decimal fractions like 0.1 cannot be represented exactly. The same is true of plain IEEE double floats. This makes both highly unsuitable for calculations involving money or other values that should be exact decimal fractions. (Suitably scaled integers, or perhaps rationals, are better choices.) > cdecimal may make your life easier instead if you have already > working code using the Python Decimal and you want to speed it up. First in importance is performance, but correctness cannot be compromised. Python Decimal semantics are fine for my application, but performance is too slow. A drop in replacement like cdecimal makes life easier, but if a faster solution is available, I'll exchange ease of use for performance. Dan
On Tue, Mar 1, 2011 at 5:39 PM, Daniel Popowich <danielpopowich@gmail.com> wrote: > > Daniele Varrazzo writes: >> If cdecimal is good for you (i.e. if the decimal semantics is fine), >> you may use mpf throughout your program instead of mpq. It is an >> exact type (as exact as decimal of course: you can't represent all >> the rationals) and likely to be more efficient than mpq. Plus, it >> maps directly with the postgres decimal on I/O. >> > > Really? But it's binary floating point. From the GMP manual, > http://gmplib.org/manual-4.3.2/Floating_002dpoint-Functions.html: Uhm... I think you are right. I thought mpf was more similar to a scaled integer. -- Daniele