Re: DO with a large amount of statements get stuck with high memory consumption - Mailing list pgsql-hackers

From Merlin Moncure
Subject Re: DO with a large amount of statements get stuck with high memory consumption
Date
Msg-id CAHyXU0yzRxAbQTe7QmXanh0_U2PodSWECPri+nKAC+TuWqZdfQ@mail.gmail.com
Whole thread Raw
In response to Re: DO with a large amount of statements get stuck with high memory consumption  (Jan Wieck <jan@wi3ck.info>)
Responses Re: DO with a large amount of statements get stuck with high memory consumption
List pgsql-hackers
On Mon, Jul 18, 2016 at 8:59 AM, Jan Wieck <jan@wi3ck.info> wrote:
>
>
> On Mon, Jul 18, 2016 at 9:43 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>
>> Merlin Moncure <mmoncure@gmail.com> writes:
>> > BTW, while the fix does address the cleanup performance issue, it's
>> > still the case that anonymous code blocks burn up lots of resident
>> > memory (my 315k example I tested with ate around 8gb IIRC) when run
>> > like this.  My question is, if the pl/pgsql code block is anonymous
>> > and not in some kind of a loop, why bother caching the plan at all?
>>
>> Nobody got around to it.  Also, as you note, it's not as simple as
>> "don't cache if in a DO block".  You'd need to track whether you were
>> inside any sort of looping construct.  Depending on how difficult
>> that turned out to be, it might add overhead to regular functions
>> that we don't want.
>
> Agreed. And from the structures themselves it is not really easy to detect
> if inside of a loop, the toplevel, while, for and if all use the same
> statement
> block and call exec_stmts(), which in turn calls exec_stmt() for  each
> element in that list. It is not impossible to add a flag, set at PL compile
> time, to that element and check it every time, the statement is executed.
> But such a change definitely needs more testing and probably won't
> qualify for backpatching.

Right. Note, not arguing for backpatch here, just some open
speculation and some evidence that we still have a problem (although
nearly as nasty of one -- the pre-patch behavior of not responding to
cancel is very dangerous and solved).

Hm, maybe, instead of trying to figure out if in a loop, set a
'called' flag  with each statement and only cache when touched the
second time.  (If that's easier, dunno).

merlin



pgsql-hackers by date:

Previous
From: Greg Stark
Date:
Subject: Re: One process per session lack of sharing
Next
From: Greg Stark
Date:
Subject: Re: rethinking dense_alloc (HashJoin) as a memory context