Re: Allocating shared memory in Postgres - Mailing list pgsql-general
| From | Souvik Bhattacherjee |
|---|---|
| Subject | Re: Allocating shared memory in Postgres |
| Date | |
| Msg-id | CAANrPSd5ViPrOm5t_k+6YZp7=R6RxAq4OTsyFkcMoRwF_9tc3Q@mail.gmail.com Whole thread Raw |
| In response to | Re: Allocating shared memory in Postgres (Tom Lane <tgl@sss.pgh.pa.us>) |
| List | pgsql-general |
For starters, what happens if two backends do this concurrently?
> I'm assuming here that a single backend process handles all queries from a given client. In case of parallel queries, the master process will be responsible for allocating the shared memory and not the workers. Please let me know if this is not something that you implied by two backends.
Even with only one backend, if you do a query that requires X space, and then you do another query that requires X+1 space, what's going to happen?
> In my application, every query that allocates shared memory is wrapped in a separate txn, and the life of the shared memory is for the duration of the query. The only purpose of allocating shared memory here is to make the memory segment visible to the worker processes for that particular query. No other txns/query actually accesses this shared memory. Also, when I allocate shared memory, the txn id is used as a key to the ShmemIndex to differentiate between two concurrent shared memory allocation requests.
Recent PG releases have a "DSM" mechanism for short-lived (query lifespan,
typically) shared memory that's separate from the core shmem pool. That
might suit your needs better. The system design is really not friendly
to demanding more core shmem after postmaster start.
typically) shared memory that's separate from the core shmem pool. That
might suit your needs better. The system design is really not friendly
to demanding more core shmem after postmaster start.
> Yes, I understand that "DSM" mechanisms exist. But I wanted to know if the approach that I had outlined will work even if there are certain drawbacks to it such as
1. overestimating the initial shared memory size that needs to be allocated,
2. not able to free the shared memory after use
Best,
-SB
On Wed, Jul 3, 2019 at 4:17 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Souvik Bhattacherjee <kivuosb@gmail.com> writes:
> I need to allocate shared memory in Postgres 11.0 outside the
> initialization phase. In order to achieve this I have done the following:
> - during a particular query, where I need to allocate shared memory (which
> is a function of the sizes of the tables in the query), I invoke
> ShmemInitStruct() in shmem.c
This seems like a pretty horrid idea. For starters, what happens if two
backends do this concurrently? Even with only one backend, if you do a
query that requires X space, and then you do another query that requires
X+1 space, what's going to happen?
Recent PG releases have a "DSM" mechanism for short-lived (query lifespan,
typically) shared memory that's separate from the core shmem pool. That
might suit your needs better. The system design is really not friendly
to demanding more core shmem after postmaster start.
regards, tom lane
pgsql-general by date: