Re: bg worker: patch 1 of 6 - permanent process - Mailing list pgsql-hackers

From Markus Wanner
Subject Re: bg worker: patch 1 of 6 - permanent process
Date
Msg-id 4C8FBC67.5030809@bluegap.ch
Whole thread Raw
In response to Re: bg worker: patch 1 of 6 - permanent process  (Tom Lane <tgl@sss.pgh.pa.us>)
List pgsql-hackers
Hi,

I'm glad discussion on this begins.

On 09/14/2010 07:55 PM, Tom Lane wrote:
> So there is a minimum of one avworker per database?

Nope, you can set that to 0. You don't *need* to keep idle workers around.

> That's a guaranteed
> nonstarter.  There are many people with thousands of databases, but no
> need for thousands of avworkers.

Well, yeah, bgworkers are primarily designed to be used for Postgres-R, 
where you easily get more background workers than normal backends. And 
having idle backends around waiting for a next job is certainly 
preferable over having to re-connect every time.

I've advertised the bgworker infrastructure for use for parallel 
querying as well. Again, that use case easily leads to having more 
background workers than normal backends. And you don't want to wait for 
them all to re-connect for every query they need to help with.

> I'm also pretty unclear why you speak of min and max numbers of workers
> when the proposal (AIUI) is to have the workers there always, rather
> than have them come and go.

This step 1 of the bgworker set of patches turns the av*launcher* 
(coordinator) into a permanent process (even if autovacuum is off).

The background workers can still come and go. However, they don't 
necessarily *need* to terminate after having done their job. The 
coordinator controls them and requests new workers or commands idle ones 
to terminate *as required*.

I don't think there's that much different to the current implementation. 
Setting both, the min and max number of idle bgworkers to 0 should in 
fact give you the exact same behavior as we currently have: namely to 
terminate each av/bgworker after its job is done, never having idle 
workers around. Which might or might not be the optimal configuration 
for users with lots of databases, that's hard to predict. And it depends 
a lot on the load distribution over the databases and on how clever the 
coordinator manages the bgworkers.

Regards

Markus Wanner


pgsql-hackers by date:

Previous
From: Simon Riggs
Date:
Subject: Sync Replication with transaction-controlled durability
Next
From: Markus Wanner
Date:
Subject: Re: bg worker: patch 1 of 6 - permanent process