Re: Copy data to DSA area - Mailing list pgsql-hackers
From | Kyotaro HORIGUCHI |
---|---|
Subject | Re: Copy data to DSA area |
Date | |
Msg-id | 20181128.170846.92281550.horiguchi.kyotaro@lab.ntt.co.jp Whole thread Raw |
In response to | RE: Copy data to DSA area ("Ideriha, Takeshi" <ideriha.takeshi@jp.fujitsu.com>) |
Responses |
RE: Copy data to DSA area
|
List | pgsql-hackers |
Hello. At Wed, 28 Nov 2018 05:13:26 +0000, "Ideriha, Takeshi" <ideriha.takeshi@jp.fujitsu.com> wrote in <4E72940DA2BF16479384A86D54D0988A6F3BD73A@G01JPEXMBKW04> > Hi > > >From: Thomas Munro [mailto:thomas.munro@enterprisedb.com] > >Here we need something better, because once this new kind of memory fills up with > >leaked objects, you have to restart the whole cluster to recover. > > Thank you for the clarification. I agree that leaving memory leaking in shared memory > without freeing it ends up cluster-wide problem. If some of the "leaked" objects are liked each other with other "live" objects including those on local memory, we must tell whether each "maybe-leaked" object is actually a leak or not. That looks apprently stupid. Maybe we shouldn't manipulate objects on shared memory at such a low-level interface. We would need a kind of resource manager for such structures holding shared objects and should (perhaps explicitly) register the objects that shuold be free'd on error-exiting from a scope. This seems a bit different from reset callbacks of the current MemoryContext. > >If the solution involves significantly different control flows (like PG_TRY(), > >PG_FINALLY() manual cleanup all over the place), then lots of existing palloc-based > >code will need to be reviewed and rewritten very carefully for use in this new kind of > >memory context. If it can be done automagically, then more of our existing code will > >be able to participate in the construction of stuff that we want to put in shared > >memory. > > About resetting X->a I was thinking about how to treat dangling pointer inside a new memory > context machinery instead of using PG_TRY() stuff. That's because putting PG_TRY() all over the > place becomes responsible for a developer trying to shared-memory-version MemoryContext and > it needs a lot of efforts as you noted. But in my mind even if PG_TRY() is used, it only affects new > code uses shared-memory version MemoryContext and doesn't affect current code. > I may be missing something here... Thomas mentioned exiting upper-layer library code using palloc, like pg_list. As I wrote above, pg_list is one of such feature that would need to be carefully adjusted. > >I first thought about this in the context of shared plan caches, a topic that appears on > >this mailing list from time to time. There are some other fun problems, like cache > >invalidation for shared caches, space recycling (LRUs etc), and of course > >locking/concurrency > >(dsa_allocate() and dsa_free() are concurrency safe, but whatever data structures > >you build with the memory of course need their own plan for dealing with concurrency). > >But a strategy for clean-up on errors seems to be a major subproblem to deal with > >early in the project. I will be very happy if we can come up with something :-) > > Yeah, I hope we can do it somehow. In addition to recycling issue within a fixed area, I suppose that we will need to "expand" the fixed-address shared memory. PROT_NONE could work for that but it must break some platforms.. > >On Wed, Nov 14, 2018 at 12:40 AM Ideriha, Takeshi <ideriha.takeshi@jp.fujitsu.com> > >wrote: > >> From: Thomas Munro [mailto:thomas.munro@enterprisedb.com] > >> >postgres=# call hoge_list(3); > >> >NOTICE: Contents of list: > >> >NOTICE: 1 > >> >NOTICE: 2 > >> >NOTICE: 3 > >> >CALL > >> > > >> >You can call that procedure from various different backends to add > >> >and remove integers from a List. The point is that it could just as > >> >easily be a query plan or any other palloc-based stuff in our tree, and the pointers > >work in any backend. > >> > >> Thanks for the patch. I know it's a rough sketch but basically that's what I want to > >do. > >> I tried it and it works well. > > > >After your message I couldn't resist a small experiment. But it seems there are > >plenty more problems to be solved to make it useful... > > Thank you for kind remark. > By the way I just thought meta variable "hoge" is used only in Japan :) I've seen it many-many times including mine in this list, thus why don't you know it is already a world-wide jargon? :p regards. -- Kyotaro Horiguchi NTT Open Source Software Center
pgsql-hackers by date: