Re: suggestions to improve postgresql suitability for data-mining - Mailing list pgsql-hackers

From Darren King
Subject Re: suggestions to improve postgresql suitability for data-mining
Date
Msg-id 26751392596DDD4D84FE1806D1F819B794A963@exchange.insight
Whole thread Raw
In response to suggestions to improve postgresql suitability for data-mining  (Fabien COELHO <coelho@cri.ensmp.fr>)
List pgsql-hackers
> You want to process all invoices to count them
> and to sum up the amounts on a per month/area/type
> basis. The initial data size is in GB, but the
> size of the expected result is in KB (namely 2 data
> for each 100 areas * 12 months * 4 types).

The key to handling large datasets for data mining is pre-aggregation based on the smallest time frame needed for
details.

I'd suggest running these large queries and storing the results in other tables, and then writing a set of functions to
workwith those aggregate tables. 

No sense in summing up the same set of static data more than once if you can help it.

Darren


pgsql-hackers by date:

Previous
From: Bruno Wolff III
Date:
Subject: Re: suggestions to improve postgresql suitability for data-mining
Next
From: Rod Taylor
Date:
Subject: Re: php with postgres