Re: Excessive memory usage in multi-statement queries w/ partitioning - Mailing list pgsql-hackers

From Amit Langote
Subject Re: Excessive memory usage in multi-statement queries w/ partitioning
Date
Msg-id CA+HiwqEwK-rZMt3TY_FL4jPKOSFJhH_FzAdW8XXNZx4Er753sQ@mail.gmail.com
Whole thread Raw
In response to Re: Excessive memory usage in multi-statement queries w/ partitioning  (Julien Rouhaud <rjuju123@gmail.com>)
Responses Re: Excessive memory usage in multi-statement queries w/ partitioning  (Julien Rouhaud <rjuju123@gmail.com>)
Re: Excessive memory usage in multi-statement queries w/ partitioning  (Tom Lane <tgl@sss.pgh.pa.us>)
List pgsql-hackers
Hi Julien,

Thanks for taking a look at this.

On Thu, Jul 4, 2019 at 6:52 PM Julien Rouhaud <rjuju123@gmail.com> wrote:
> On Tue, May 28, 2019 at 6:57 AM Amit Langote wrote:
> > >> Maybe like the attached?  I'm not sure if we need to likewise be concerned
> > >> about exec_sql_string() being handed multi-query strings.
>
> the whole extension sql script is passed to execute_sql_string(), so I
> think that it's a good thing to have similar workaround there.

That makes sense, although it is perhaps much less likely for memory
usage explosion to occur in execute_sql_strings(), because the scripts
passed to execute_sql_strings() mostly contain utility statements and
rarely anything whose planning will explode in memory usage.

Anyway, I've added similar handling in execute_sql_strings() for consistency.

Now I wonder if we'll need to consider another path which calls
pg_plan_queries() on a possibly multi-statement query --
BuildCachedPlan()...

> About the patch:
>
>  -        * Switch to appropriate context for constructing querytrees (again,
> -        * these must outlive the execution context).
> +        * Switch to appropriate context for constructing querytrees.
> +        * Memory allocated during this construction is released before
> +        * the generated plan is executed.
>
> The comment should mention query and plan trees, everything else seems ok to me.

Okay, fixed.

Attached updated patch.  Thanks again.

Regards,
Amit

Attachment

pgsql-hackers by date:

Previous
From: "kuroda.hayato@fujitsu.com"
Date:
Subject: RE: [PATCH] memory leak in ecpglib
Next
From: Masahiko Sawada
Date:
Subject: Re: [Proposal] Table-level Transparent Data Encryption (TDE) and KeyManagement Service (KMS)