Re: Memory exhaustion during bulk insert - Mailing list pgsql-hackers

From Tom Lane
Subject Re: Memory exhaustion during bulk insert
Date
Msg-id 9306.1239804274@sss.pgh.pa.us
Whole thread Raw
In response to Memory exhaustion during bulk insert  (Xin Wang <andywx@gmail.com>)
List pgsql-hackers
Xin Wang <andywx@gmail.com> writes:
> I searched the mailinglist archive and noticed that a patch to improve
> bulk insert performance is committed in Nov 2008. The log message said
> "(the patch) keeps the current target buffer pinned and make it work
> in a small ring of buffers to avoid having bulk inserts trash the whole
> buffer arena."
> However, I do not know much about the code below the heapam layer. Can that
> patch solve my problem (the version I use is 8.3.5)?

No.  You have a memory leak to fix.  I suspect you need to be paying
attention to evaluating the successive tuple values in a short-term
memory context that you can reset on each cycle.  There are other
possibilities though --- looking at the memory map produced on an
out-of-memory error would help narrow down the problem.  (If the thing
"hangs up" without producing such an error, that's the *first* problem
to solve.  It could be that it's not so much hanging up as going into
swap hell; in which case I'd suggest running the postmaster under a more
restrictive ulimit, so that it fails before starting to swap.)
        regards, tom lane


pgsql-hackers by date:

Previous
From: Robert Haas
Date:
Subject: Re: Replacing plpgsql's lexer
Next
From: "Hiroshi Saito"
Date:
Subject: Re: Patch for server-side encoding issues