On 9/28/06, Carlo Stonebanks <stonec.register@sympatico.ca> wrote:
> The deduplication process requires so many programmed procedures that it
> runs on the client. Most of the de-dupe lookups are not "straight" lookups,
> but calculated ones emplying fuzzy logic. This is because we cannot dictate
> the format of our input data and must deduplicate with what we get.
>
> This was one of the reasons why I went with PostgreSQL in the first place,
> because of the server-side programming options. However, I saw incredible
> performance hits when running processes on the server and I partially
> abandoned the idea (some custom-buiilt name-comparison functions still run
> on the server).
imo, the key to high performance big data movements in postgresql is
mastering sql and pl/pgsql, especially the latter. once you get good
at it, your net time of copy+plpgsql is going to be less than
insert+tcl.
merlin
