Thread: Static memory, shared memory
I am writing a Postgres extension, and thought that I had memory corruption, (thanks for the --enable-cassert lead). I might, but It now looks like I need to understand the use of shared memory and locking in Postgres. So I have two questions.
1) I am now guessing that my original problem is caused by relying on static memory in my extension (i.e., in the source declaring PG_MODULE_MAGIC). This static memory is almost but not quite constant -- it is initialized from _PG_init, and then never modified. I suspect that this cannot work in general (since Postgres is multi-process), but I thought it would be adequate for early development. However, I am seeing this static memory get corrupted even when there is only a single process executing the extension code (verified by examining getpid()). So the question is this: Is the use of non-constant static memory ill-advised, even assuming there is just one process relying on it? Or is it more likely that I still have run-of-the-mill memory corruption. (--enable-cassert hasn't notified me of any problems).
2) Assuming that I should be using shared memory instead of static, I am reading https://www.postgresql.org/docs/12/xfunc-c.html#id-1.8.3.13.14, and examining contrib/pg_prewarm. The xfunc-c documentation mentions RequestNamedLWLockTranche to get an array of LWLocks. But the sample code that follows calls GetNamedLWLockTranche. And the pg_prewarm code doesn't rely on an array of locks, it initializes a single lock, (which I think would be adequate for my needs). I understand the purpose of locks for obtaining and manipulating shared memory but I am confused about the correct way to proceed. I'm guessing it is safe to assume that pg_prewarm works and is a good model to follow, and that the doc may be buggy. Can someone clarify my confusion, and perhaps point me at a tutorial on correct usage of the interfaces for LWLocks and shared memory, (I haven't been able to find one).
Thank you.
Jack Orenstein
1) I am now guessing that my original problem is caused by relying on static memory in my extension (i.e., in the source declaring PG_MODULE_MAGIC). This static memory is almost but not quite constant -- it is initialized from _PG_init, and then never modified. I suspect that this cannot work in general (since Postgres is multi-process), but I thought it would be adequate for early development. However, I am seeing this static memory get corrupted even when there is only a single process executing the extension code (verified by examining getpid()). So the question is this: Is the use of non-constant static memory ill-advised, even assuming there is just one process relying on it? Or is it more likely that I still have run-of-the-mill memory corruption. (--enable-cassert hasn't notified me of any problems).
2) Assuming that I should be using shared memory instead of static, I am reading https://www.postgresql.org/docs/12/xfunc-c.html#id-1.8.3.13.14, and examining contrib/pg_prewarm. The xfunc-c documentation mentions RequestNamedLWLockTranche to get an array of LWLocks. But the sample code that follows calls GetNamedLWLockTranche. And the pg_prewarm code doesn't rely on an array of locks, it initializes a single lock, (which I think would be adequate for my needs). I understand the purpose of locks for obtaining and manipulating shared memory but I am confused about the correct way to proceed. I'm guessing it is safe to assume that pg_prewarm works and is a good model to follow, and that the doc may be buggy. Can someone clarify my confusion, and perhaps point me at a tutorial on correct usage of the interfaces for LWLocks and shared memory, (I haven't been able to find one).
Thank you.
Jack Orenstein
Jack Orenstein <jao@geophile.com> writes:
> I am writing a Postgres extension, and thought that I had memory
> corruption, (thanks for the --enable-cassert lead). I might, but It now
> looks like I need to understand the use of shared memory and locking in
> Postgres. So I have two questions.
> 1) I am now guessing that my original problem is caused by relying on
> static memory in my extension (i.e., in the source declaring
> PG_MODULE_MAGIC). This static memory is almost but not quite constant -- it
> is initialized from _PG_init, and then never modified. I suspect that this
> cannot work in general (since Postgres is multi-process), but I thought it
> would be adequate for early development. However, I am seeing this static
> memory get corrupted even when there is only a single process executing the
> extension code (verified by examining getpid()).
Define what you mean by "corrupted". It seems highly unlikely that any
code but your own is touching this memory.
Really the big-picture question here is what are you hoping to accomplish
and why do you think this memory might need to be shared?
regards, tom lane
On Sat, Jan 9, 2021 at 12:18 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Jack Orenstein <jao@geophile.com> writes:
> I am writing a Postgres extension, and thought that I had memory
> corruption, (thanks for the --enable-cassert lead). I might, but It now
> looks like I need to understand the use of shared memory and locking in
> Postgres. So I have two questions.
> 1) I am now guessing that my original problem is caused by relying on
> static memory in my extension (i.e., in the source declaring
> PG_MODULE_MAGIC). This static memory is almost but not quite constant -- it
> is initialized from _PG_init, and then never modified. I suspect that this
> cannot work in general (since Postgres is multi-process), but I thought it
> would be adequate for early development. However, I am seeing this static
> memory get corrupted even when there is only a single process executing the
> extension code (verified by examining getpid()).
Define what you mean by "corrupted". It seems highly unlikely that any
code but your own is touching this memory.
Some fields have expected values, others do not.
I think I have just figured out this problem, and it was indeed my own gun shooting my own foot.
Really the big-picture question here is what are you hoping to accomplish
and why do you think this memory might need to be shared?
The type I am implementing depends on looking up data in an array whose size is approximately 64k. This array needs to be computed once and for all early on, and is then consulted as the type is used. For now, I have hardwired the parameters that determine the array's contents, and the array is constructed during _PG_init. I will eventually remove this scaffolding, and compute the array contents when required, which should be a rare event.
Jack Orenstein