Re: background workers, round three - Mailing list pgsql-hackers

On Tue, Oct 15, 2013 at 4:02 PM, Kohei KaiGai <kaigai@kaigai.gr.jp> wrote:
> Hmm. It probably allows to clean-up smaller fraction of data structure
> constructed on dynamic shared memory segment, if we map / unmap
> for each transactions.

I think the primary use of dynamic shared memory will be for segments
that get created for a single transaction, used, and then destroyed.
Perhaps people will find other uses for them that involve keeping them
mapped for longer periods of time, and that is fine.  But whether or
long or short, it seems clear to me that shared memory data structures
will need cleanup on detach, just as backends must clean up the main
shared memory segment before they detach.

> I assumed smaller chunks allocated on static or dynamic shared
> memory segment to be used for communicate between main
> process and worker processes because of my motivation.
> When we move a chunk of data to co-processor using asynchronous
> DMA transfer, API requires the source buffer is mlock()'ed to avoid
> unintentional swap out during DMA transfer. On the other hand,
> cost of mlock() operation is not ignorable, so it may be a reasonable
> design to lock a shared memory segment on start-up time then
> continue to use it, without unmapping.
> So, I wondered how to handle the situation when extension tries
> to manage a resource with smaller granularity than the one
> managed by PostgreSQL core.

I'm still not sure exactly what your concern is here, but I agree
there are some thorny issues around resource management here.  I've
spent a lot of the last couple months trying to sort through them.  As
it seems to me, the problem is basically that we're introducing major
new types of resources (background workers, dynamic shared memory
segments, and perhaps other things) and they all require management -
just as our existing types of resources (locks, buffer pins, etc.)
require management.  But the code to manage existing resources has
been around for so long that we just rely on it without thinking about
it, so when we suddenly DO need to think about it, it's an unpleasant
surprise.

As far as shared memory resources specifically, one consideration is
that some systems (e.g. some versions of HP-UX) have fairly small
limits (< 15) on the number of shared memory segments that can be
mapped, and all 32-bit systems are prone to running out of usable
address space.  So portable code is going to have to use these
resources judiciously.  On the other hand, I think that people wanting
to write code that only needs to run on 64-bit Linux will be able to
pretty much go nuts.

Unless there are further objections to the terminate patch as written,
I'm going to go ahead and commit that, with the additional of
documentation (as pointed out by Michael) and a change to the lock
mode (as pointed out by KaiGai).  The other patches, at least for the
time being, are withdrawn.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company