Re: optimizing import of large CSV file into partitioned table? - Mailing list pgsql-general

From Filip Rembiałkowski
Subject Re: optimizing import of large CSV file into partitioned table?
Date
Msg-id 92869e661003281637w6f23b1b7gc279a40ef14a884a@mail.gmail.com
Whole thread Raw
In response to Re: optimizing import of large CSV file into partitioned table?  (Thom Brown <thombrown@gmail.com>)
List pgsql-general
2010/3/28 Thom Brown <thombrown@gmail.com>:

> The problem here is that you appear to require an index update, trigger
> firing and constraint check for every single row.  First thing I'd suggest
> is remove the indexes.  Apply that after your import, otherwise it'll have
> to update the index for every single entry.
+1

> And the trigger won't help
> either.  Import into a single table and split it out into further tables
> after if required.
note: partitioning could help if there were multiple physical volumes
/ spindles for data directory.
for maximizing performance, I would rather split the CSV input (with
awk/perl/whatever) before loading, to have one backend for each
partition loader.

> And finally the constraint should probably be applied
> after too, so cull any violating rows after importing.
+1



--
Filip Rembiałkowski
JID,mailto:filip.rembialkowski@gmail.com
http://filip.rembialkowski.net/

pgsql-general by date:

Previous
From: Stefan Keller
Date:
Subject: hstore equality-index performance question
Next
From: Tim Landscheidt
Date:
Subject: Re: Using readline for frequently used queries