Re: COPY enhancements - Mailing list pgsql-hackers

From Simon Riggs
Subject Re: COPY enhancements
Date
Msg-id 1255353195.15590.93.camel@ebony
Whole thread Raw
In response to Re: COPY enhancements  (Tom Lane <tgl@sss.pgh.pa.us>)
List pgsql-hackers
On Thu, 2009-10-08 at 11:01 -0400, Tom Lane wrote:

> So as far as I can see, the only form of COPY error handling that
> wouldn't be a cruel joke is to run a separate subtransaction for each
> row, and roll back the subtransaction on error.  Of course the
> problems
> with that are (a) speed, (b) the 2^32 limit on command counter IDs
> would mean a max of 2^32 rows per COPY, which is uncomfortably small
> these days.  Previous discussions of the problem have mentioned trying
> to batch multiple rows per subtransaction to alleviate both issues.
> Not easy of course, but that's why it's not been done yet.  With a
> patch like this you'd also have (c) how to avoid rolling back the
> insertions into the logging table.

(d) using too many xids will force the system to begin immediate
wraparound-avoidance vacuuming to freeze rows. 

Dimitri's pgloader is looking even more attractive, not least because it
exists and it works. (And is the reason I personally stopped considering
the COPY-error-logging feature as important).

-- Simon Riggs           www.2ndQuadrant.com



pgsql-hackers by date:

Previous
From: Simon Riggs
Date:
Subject: Hot Standby and 64+ subxids (was COPY enhancements)
Next
From: Teodor Sigaev
Date:
Subject: Re: Re: [GENERAL] contrib/plantuner - enable PostgreSQL planner hints