Thread: Allocating shared memory in Postgres
Hi,
I need to allocate shared memory in Postgres 11.0 outside the initialization phase. In order to achieve this I have done the following:
- increased the amount of shared memory by increasing the value of size in CreateSharedMemoryAndSemaphores (int port) in ipci.c. I have made sure that the amount of memory that I need, say m << M, where M is the amount of additional shared memory that I have allocated.
- during a particular query, where I need to allocate shared memory (which is a function of the sizes of the tables in the query), I invoke ShmemInitStruct() in shmem.c
This seems to work in my case, although I haven't tested it extensively.
My concern here is that when I go through the NOTES in shmem.c, I find the following line:
Fixed-size structures contain things like global variables for a module and should never be allocated after the shared memory initialization phase.
I'm allocating a shared array data structure through ShmemInitStruct and I'm not sure if the lines above apply to my case, since I'm doing the allocation during a query.
Any help/clarifications in this regard would be appreciated.
Best,
-SB
Souvik Bhattacherjee <kivuosb@gmail.com> writes:
> I need to allocate shared memory in Postgres 11.0 outside the
> initialization phase. In order to achieve this I have done the following:
> - during a particular query, where I need to allocate shared memory (which
> is a function of the sizes of the tables in the query), I invoke
> ShmemInitStruct() in shmem.c
This seems like a pretty horrid idea. For starters, what happens if two
backends do this concurrently? Even with only one backend, if you do a
query that requires X space, and then you do another query that requires
X+1 space, what's going to happen?
Recent PG releases have a "DSM" mechanism for short-lived (query lifespan,
typically) shared memory that's separate from the core shmem pool. That
might suit your needs better. The system design is really not friendly
to demanding more core shmem after postmaster start.
regards, tom lane
For starters, what happens if two backends do this concurrently?
> I'm assuming here that a single backend process handles all queries from a given client. In case of parallel queries, the master process will be responsible for allocating the shared memory and not the workers. Please let me know if this is not something that you implied by two backends.
Even with only one backend, if you do a query that requires X space, and then you do another query that requires X+1 space, what's going to happen?
> In my application, every query that allocates shared memory is wrapped in a separate txn, and the life of the shared memory is for the duration of the query. The only purpose of allocating shared memory here is to make the memory segment visible to the worker processes for that particular query. No other txns/query actually accesses this shared memory. Also, when I allocate shared memory, the txn id is used as a key to the ShmemIndex to differentiate between two concurrent shared memory allocation requests.
Recent PG releases have a "DSM" mechanism for short-lived (query lifespan,
typically) shared memory that's separate from the core shmem pool. That
might suit your needs better. The system design is really not friendly
to demanding more core shmem after postmaster start.
typically) shared memory that's separate from the core shmem pool. That
might suit your needs better. The system design is really not friendly
to demanding more core shmem after postmaster start.
> Yes, I understand that "DSM" mechanisms exist. But I wanted to know if the approach that I had outlined will work even if there are certain drawbacks to it such as
1. overestimating the initial shared memory size that needs to be allocated,
2. not able to free the shared memory after use
Best,
-SB
On Wed, Jul 3, 2019 at 4:17 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Souvik Bhattacherjee <kivuosb@gmail.com> writes:
> I need to allocate shared memory in Postgres 11.0 outside the
> initialization phase. In order to achieve this I have done the following:
> - during a particular query, where I need to allocate shared memory (which
> is a function of the sizes of the tables in the query), I invoke
> ShmemInitStruct() in shmem.c
This seems like a pretty horrid idea. For starters, what happens if two
backends do this concurrently? Even with only one backend, if you do a
query that requires X space, and then you do another query that requires
X+1 space, what's going to happen?
Recent PG releases have a "DSM" mechanism for short-lived (query lifespan,
typically) shared memory that's separate from the core shmem pool. That
might suit your needs better. The system design is really not friendly
to demanding more core shmem after postmaster start.
regards, tom lane