Re: O(n) tasks cause lengthy startups and checkpoints - Mailing list pgsql-hackers

From Bossart, Nathan
Subject Re: O(n) tasks cause lengthy startups and checkpoints
Date
Msg-id AED00E15-7BB1-4514-95A3-9D6658043B61@amazon.com
Whole thread Raw
In response to O(n) tasks cause lengthy startups and checkpoints  ("Bossart, Nathan" <bossartn@amazon.com>)
List pgsql-hackers
On 12/13/21, 9:20 AM, "Justin Pryzby" <pryzby@telsasoft.com> wrote:
> On Mon, Dec 13, 2021 at 08:53:37AM -0500, Robert Haas wrote:
>> Another issue is that we don't want to increase the number of
>> processes without bound. Processes use memory and CPU resources and if
>> we run too many of them it becomes a burden on the system. Low-end
>> systems may not have too many resources in total, and high-end systems
>> can struggle to fit demanding workloads within the resources that they
>> have. Maybe it would be cheaper to do more things at once if we were
>> using threads rather than processes, but that day still seems fairly
>> far off.
>
> Maybe that's an argument that this should be a dynamic background worker
> instead of an auxilliary process.  Then maybe it would be controlled by
> max_parallel_maintenance_workers (or something similar).  The checkpointer
> would need to do these tasks itself if parallel workers were disabled or
> couldn't be launched.

I think this is an interesting idea.  I dislike the prospect of having
two code paths for all this stuff, but if it addresses the concerns
about resource usage, maybe it's worth it.

Nathan


pgsql-hackers by date:

Previous
From: Tomas Vondra
Date:
Subject: Re: using extended statistics to improve join estimates
Next
From: Andres Freund
Date:
Subject: Re: isolationtester: add session name to application name