Home > mailing lists

Re: [PoC] Reducing planning time when tables have many partitions - Mailing list pgsql-hackers

From	Ashutosh Bapat
Subject	Re: [PoC] Reducing planning time when tables have many partitions
Date	March 27 07:42:46
Msg-id	CAExHW5vdBMvNMnFsiFhL7-mvY3ESxmqsZX-9+BYOHOOARcdv2A@mail.gmail.com Whole thread Raw
In response to	Re: [PoC] Reducing planning time when tables have many partitions (David Rowley <dgrowleyml@gmail.com>)
Responses	Re: [PoC] Reducing planning time when tables have many partitions Re: [PoC] Reducing planning time when tables have many partitions
List	pgsql-hackers

Tree view

On Mon, Mar 24, 2025 at 11:08 AM David Rowley <dgrowleyml@gmail.com> wrote:

2. You should use list_free() in the following:

/*
* XXX Should we use list_free()? I decided to use this style to take
* advantage of speculative execution.
*/
if (unlikely(it->list_is_copy))
pfree(it->ec_members);

The reason is that you're wrongly assuming that calling pfree on the
List pointer is enough to get rid of all memory used by the list. The
List may have a separately allocated elements[] array (this happens
when there's > 5 elements) which you're leaking with the current code.

I assume the speculative execution comment is there because you want
to omit the "list == NULL" check in list_free_private. Is this
measurable, performance-wise?

Here are memory consumption numbers using list_free() instead of pfree(), using the same method as [1], using a binary without asserts and debug info. PFA the patchset where all the patches are the same as v35 but with an extra patch fixing memory leak. The memory leak is visible with a higher number of joins. At a lower number of joins, I expect that the memory saved is less than a KB or the leaked memory fits within 1 chunk of memory context and hence not visible.

rows by number of partitions

columns by number of joins

each cell is a triplet, s = memory saving in %, mm - memory consumed without fix, pm = memory consumed with fix

with PWJ = off

without PWJ = on

But overall the patches consume more memory than before as seen from measurements below

Each cell is a triplet (s, mm, pm) where s = memory saving in % (-ve indicates that memory consumption has increased), mm = memory consumption with no patches applied, pm = memory consumption with all patches applied

PWJ=off

PWJ=on

In the case of PWJ = on, the % wise memory consumption is less because memory consumption without fixes is huge and the patch adds on top of it. But without PWJ, the memory consumption is high, especially at higher number of joins and higher number of partitions.

[1] https://www.postgresql.org/message-id/CAExHW5vNk4f8VSTnizL-avREYhPgocEtH3E7MCEJgp=R33ChyQ@mail.gmail.com

Best Wishes,

Ashutosh Bapat

pgsql-hackers by date:

From: "Hayato Kuroda (Fujitsu)"
Date: 27 March, 07:26:53
Subject: RE: Enhance 'pg_createsubscriber' to retrieve databases automatically when no database is provided.

From: Amul Sul
Date: 27 March, 07:48:46
Subject: Re: NOT ENFORCED constraint feature

Re: [PoC] Reducing planning time when tables have many partitions - Mailing list pgsql-hackers

Previous

Next