RE: Copy data to DSA area - Mailing list pgsql-hackers

Thank you for the comment.

From: Thomas Munro [mailto:thomas.munro@enterprisedb.com]
>> I'm thinking to go with plan 1. No need to think about address
>> translation seems tempting. Plan 2 (as well as plan 3) looks a big project.
>
>The existing function dsa_create_in_place() interface was intended to support that,
>but has never been used in that way so I'm not sure what extra problems will come up.
>Here are some assorted thoughts:
>
>* You can prevent a DSA area from creating extra DSM segments, so that it is
>constrained to stay entirely in the space you give it, by calling dsa_set_size_limit(area,
>size) using the same size that you gave to dsa_create_in_place(); now you have a
>DSA area that manages a single fixed-sized chunk of memory that you gave it, in your
>case inside the traditional shared memory segment (but it could be anywhere,
>including inside a DSM segment or another DSA area!)
Yeah, I will use it.

>* You can probably write a MemoryContext wrapper for it, if it has only one segment
>that is in the traditional shared memory segment.
>You would need to do very simple kind of address translation: the result from palloc()
>needs to be base + dsa_allocate()'s result, and the argument to pfree() needs to be
>subtracted from base when
>dsa_free() is called.  That is a version of your idea C that should work AFAIK.

I didn't notice that if only one segment is used dsa_get_address() is not needed and
simple math is enough.

>* Once you have that working, you now have a new kind of resource management
>problem on your hands: memory leaks will be cluster-wide and cluster-life-time!
>That's hard, because the goal is to be able to use arbitrary code in the tree that deals
>with plans etc, but that code all assumes that it can "throw" (elog()) on errors.
>PostgreSQL C is generally "garbage collected" (in a way), but in this sketch, that
>doesn't work anymore: this area *never* goes out of scope and gets cleaned up.
>Generally, languages with exceptions either need garbage collection or scoped
>destructors to clean up the mess, but in this sketch we don't have that anymore...
>much like allocating stuff in TopMemoryContext, except worse because it doesn't go
>away when one backend exits.
>
>* I had some ideas about some kind of "allocation rollback" interface:
>you begin an "allocation transaction", allocate a bunch of stuff (perhaps indirectly, by
>calling some API that makes query plans or whatever and is totally unaware of this
>stuff).  Then if there is an error, whatever was allocated so far is freed in the usual
>cleanup paths by a rollback that happens via the resource manager machinery.
>If you commit, then the allocation becomes permanent.  Then you only commit stuff
>that you promise not to leak (perhaps stuff that has been added to a very carefully
>managed cluster-wide plan cache).  I am not sure of the details, and this might be
>crazy...


Can I check my understanding?
The situation you are talking about is the following:
Data structure A and B will be allocated on DSA. Data A has a pointer to B.
Another data X is already allocated on some area (might be on local heap) and has a pointer variable X->a,
which will be linked to A.

 old_context = MemoryContextSwitchTo(dsa_memory_context);
 A = palloc();
 B = palloc();
 A->b = B; 
 X->a = A;
 MemoryContextSwitchTo(old_context);
 
If error happens in the way of this flow, palloc'd data (A & B) should be freed 
and pointer to freed data (X->a) should be back to its original one. 
So handling these cases introduces an "transaction" API like begin_allocate() and end_allocate().

I'm thinking begin_allocate() starts to keeping a record of palloc'd data 
until end_allocate() is done. If error occurs, just pfree() these data. However, to rollback pointers we need to 
take notes of old value of the pointer. This would introduce a new API like 
"points_to_pallocd_data(pointer, new_data)" to remember old_value and do `X->a = A`. 
To implement them I still need further consideration about how to fit or extend existing MemoryContext machinery.

Regards,
Takeshi Ideriha