Re: Bulk Insert tuning - Mailing list pgsql-patches

From Tom Lane
Subject Re: Bulk Insert tuning
Date
Msg-id 7279.1204056778@sss.pgh.pa.us
Whole thread Raw
In response to Bulk Insert tuning  (Simon Riggs <simon@2ndquadrant.com>)
Responses Re: Bulk Insert tuning
Re: Bulk Insert tuning
List pgsql-patches
Simon Riggs <simon@2ndquadrant.com> writes:
> Following patch implements a simple mechanism to keep a buffer pinned
> while we are bulk loading.

This will fail to clean up nicely after a subtransaction abort, no?
(For that matter I don't think it's right even for a top-level abort.)
And I'm pretty sure it will trash your table entirely if someone
inserts into another relation while a bulk insert is happening.
(Not at all impossible, think of triggers for instance.)

From a code structural point of view, we are already well past the
number of distinct options that heap_insert ought to have.  I was
thinking the other day that bulk inserts ought to use a ring-buffer
strategy to avoid having COPY IN trash the whole buffer arena, just
as we've taught COPY OUT not to.  So maybe a better idea is to
generalize BufferAccessStrategy to be able to handle write as well
as read concerns; or have two versions of it, one for writing and one
for reading.  In any case the point being to encapsulate all these
random little options in a struct, which could also carry along
state that needs to be saved across a series of inserts, such as
the last pinned buffer.

            regards, tom lane

pgsql-patches by date:

Previous
From: Neil Conway
Date:
Subject: Re: SRF memory leaks
Next
From: Simon Riggs
Date:
Subject: Re: Bulk Insert tuning