Thread: Shared memory and processes
<div class="WordSection1"><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">DoesPostgres provide aconvenient way for one process to pass data to another using shared memory?</span><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"> </span><p class="MsoNormal"><spanstyle="font-size:10.0pt;font-family:"Georgia",serif;color:#1F497D">Regards</span><span style="font-size:11.0pt;font-family:"Georgia",serif;color:#1F497D"></span><pclass="MsoNormal" style="mso-margin-top-alt:6.0pt;margin-right:0cm;margin-bottom:6.0pt;margin-left:0cm"><span style="font-size:14.0pt;font-family:"Georgia",serif;color:#1F497D">DavidM Bennett </span><span style="font-size:8.0pt;font-family:"Georgia",serif;color:#1F497D">FACS</span><span style="font-size:14.0pt;font-family:"Georgia",serif;color:#1F497D"></span><divalign="center" class="MsoNormal" style="text-align:center"><i><spanstyle="font-size:8.0pt;font-family:"Georgia",serif;color:#1F497D"><hr align="center" size="2"width="100%" /></span></i></div><p class="MsoNormal"><i><span style="font-size:8.0pt;font-family:"Georgia",serif;color:#1F497D">Andl- A New Database Language - <a href="andl.org">andl.org</a></span></i><pclass="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span></div>
On Thu, Apr 28, 2016 at 12:56 PM, <david@andl.org> wrote: > Does Postgres provide a convenient way for one process to pass data to > another using shared memory? 1. Look at ShmemInitStruct, ShmemInitHash (as in hash table), ShmemInitQueue in storage/shmem.h. These use memory that is mapped at the same address in every backend (process) which is convenient for sharing data structures with internal pointers. The amount of memory available is fixed at cluster startup however. 2. Look at dsm_XXX in storage/dsm.h. This subsystem provides segments of memory that is "dynamic" in the sense that it is limited only by your system's available virtual memory, but you have to explicitly attach these segment in any backend that wants to access them by passing a handle around and the memory may be mapped at any address in each backend, so you need to work harder to build data structures that reference each other (using offsets rather than pointers, that kind of thing). DSM segments won't work well if you try to create large numbers of them, so you'll need to provide a way to manage the space inside a smallish number of large chunks of DSM memory yourself (this is a problem I'm working to fix by providing a general purpose allocator backed by a bunch of DSM segments -- watch this space). LWLocks (our usual lock primitive for cases where spinlocks are inappropriate) currently don't work correctly inside DSM segments (this too will be fixed). -- Thomas Munro http://www.enterprisedb.com
On Thu, Apr 28, 2016 at 9:56 AM, <david@andl.org> wrote: > Does Postgres provide a convenient way for one process to pass data to > another using shared memory? If you are talking about enabling the use of shared memory by 3rd-part plugins or modules, there is shmem_startup_hook for this purpose. Another thing that you may want to look at is DSM (dynamic shared memory), with its interface in dsm.h. An example of use is in src/test/modules/test_shm_mq. -- Michael
> From: Thomas Munro [mailto:thomas.munro@enterprisedb.com] > > Does Postgres provide a convenient way for one process to pass data to > > another using shared memory? > > 1. Look at ShmemInitStruct, ShmemInitHash (as in hash table), ShmemInitQueue > in storage/shmem.h. These use memory that is mapped at the same address in > every backend (process) which is convenient for sharing data structures with > internal pointers. The amount of memory available is fixed at cluster > startup however. Thanks. That limit could be an issue. The notes in shmem.c are helpful. > 2. Look at dsm_XXX in storage/dsm.h. This subsystem provides segments of > memory that is "dynamic" in the sense that it is limited only by your > system's available virtual memory, but you have to explicitly attach these > segment in any backend that wants to access them by passing a handle around > and the memory may be mapped at any address in each backend, so you need to > work harder to build data structures that reference each other (using offsets > rather than pointers, that kind of thing). DSM segments won't work well if > you try to create large numbers of them, so you'll need to provide a way to > manage the space inside a smallish number of large chunks of DSM memory > yourself (this is a problem I'm working to fix by providing a general purpose > allocator backed by a bunch of DSM segments -- watch this space). LWLocks > (our usual lock primitive for cases where spinlocks are inappropriate) > currently don't work correctly inside DSM segments (this too will be fixed). I've now found this through the test_shm_mq sample. Looks like an answer, if quite a bit of machinery needed. Thanks for the pointers. Regards David M Bennett FACS Andl - A New Database Language - andl.org
> From: Michael Paquier [mailto:michael.paquier@gmail.com] > > Does Postgres provide a convenient way for one process to pass data to > > another using shared memory? > > If you are talking about enabling the use of shared memory by 3rd-part > plugins or modules, there is shmem_startup_hook for this purpose. Thanks -- looks interesting. > Another thing that you may want to look at is DSM (dynamic shared memory), > with its interface in dsm.h. An example of use is in src/test/modules/test_shm_mq. Yes, that is useful. I'm curious why the documentation (F.41) was removed between 9.4 and 9.5 -- is there a problem? Regards David M Bennett FACS Andl - A New Database Language - andl.org
On Thu, Apr 28, 2016 at 12:24 PM, <david@andl.org> wrote: >> From: Michael Paquier [mailto:michael.paquier@gmail.com] >> Another thing that you may want to look at is DSM (dynamic shared memory), >> with its interface in dsm.h. An example of use is in src/test/modules/test_shm_mq. > > Yes, that is useful. I'm curious why the documentation (F.41) was removed between 9.4 and 9.5 -- is there a problem? No, some of the contrib/ modules have been moved in the code tree and are now considered as only test modules, which do not get installed by default. As a result, their documentation has been removed. -- Michael