Re: COPY from question - Mailing list pgsql-hackers

From Tom Lane
Subject Re: COPY from question
Date
Msg-id 17867.1075821029@sss.pgh.pa.us
Whole thread Raw
In response to Re: COPY from question  (Kevin Brown <kevin@sysexperts.com>)
List pgsql-hackers
Kevin Brown <kevin@sysexperts.com> writes:
> Slavisa Garic wrote:
>> Using pg module in python I am trying to run the COPY command to populate
>> the large table. I am using this to replace the INSERT which takes about
>> few hours to add 70000 entries where copy takes minute and a half. 

> That difference in speed seems quite large.  Too large.  Are you batching
> your INSERTs into transactions (you should be in order to get good
> performance)?  Do you have a ton of indexes on the table?  Does it have
> triggers on it or some other thing (if so then COPY may well wind up doing
> the wrong thing since the triggers won't fire for the rows it
> inserts)?

COPY *does* fire triggers, and has done so for quite a few releases.

My bet is that the issue is failing to batch individual INSERTs into
transactions.  On a properly-set-up machine you can't get more than one
transaction commit per client per disk revolution, so the penalty for
trivial transactions like single inserts is pretty steep.
        regards, tom lane


pgsql-hackers by date:

Previous
From: Tom Lane
Date:
Subject: Re: [pgsql-hackers-win32] Sync vs. fsync during checkpoint
Next
From: "Andrew Dunstan"
Date:
Subject: Re: [PATCHES] log session end - again