Re: Memory consumed by paths during partitionwise join planning - Mailing list pgsql-hackers

From Andrei Lepikhov
Subject Re: Memory consumed by paths during partitionwise join planning
Date
Msg-id 3bcf826a-6d69-4232-b43b-d5894be8bd6f@gmail.com
Whole thread Raw
In response to Memory consumed by paths during partitionwise join planning  (Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>)
List pgsql-hackers
On 27/6/2025 12:01, Andrei Lepikhov wrote:
> On 6/2/2024 13:51, Ashutosh Bapat wrote:
>> On Fri, Dec 15, 2023 at 5:22 AM Ashutosh Bapat
>> <ashutosh.bapat.oss@gmail.com> wrote:
>> First patch is no longer required. Here's rebased set
>>
>> The patches are raw. make check has some crashes that I need to fix. I
>> am waiting to hear whether this is useful and whether the design is on
>> the right track.
> I think this project is quite important to move forward and discover how 
> far we can go this way. In the attachment, the rebased patch set with 
> fixes allows it to pass the regression tests.
> 
> This idea of a refcounter resolves the problem with blind usage of 
> add_path. It is not only about extensions, which sometimes want to add 
> paths on different levels of the join tree and don't care about dangling 
> pointers. It is also about possible hidden bugs (a freed path staying in 
> the path list, just not employed) that aren't yet detected due to 
> costings and low test coverage.
After further consideration, I believe the main issue lies in managing 
increments and decrements of a path refcounter, especially in light of 
ongoing code changes. Additionally, recursion presents another challenge.

To address this complexity, I propose the following solutions:
1. Localise reference management within the add_path function.
2. Transition from a 'strict' approach, which removes all unused paths, 
to a more straightforward 'pinning' method. In this approach, we would 
simply mark a path as 'used' when someone who was added to the upper 
path list references it. Removing less memory, we leave the code much 
simpler.

This strategy would help us avoid recursion and dereferencing, which 
significantly complicate the code.

-- 
regards, Andrei Lepikhov




pgsql-hackers by date:

Previous
From: Peter Eisentraut
Date:
Subject: Re: C11 / VS 2019
Next
From: Jacob Champion
Date:
Subject: Re: [PATCH] Fix hostaddr crash during non-blocking cancellation