Postgres Pro
Downloads
Home   >   mailing lists

Thread: [HACKERS] CurTransactionContext freed before transaction COMMIT ???

[HACKERS] CurTransactionContext freed before transaction COMMIT ???

From
Gaddam Sai Ram
Date:

Hello people,

            We are implementing in-memory index. As a part of that, during index callbacks, under CurTransactionContext, we cache all the DMLs of a transaction in dlist(postgres's doubly linked list).
            We registered transaction callback, and in transaction pre-commit event(XACT_EVENT_PRE_COMMIT), we iterate through all cached DMLs(dlist) and populate in my in-memory data structure.

           --> For detailed explanation of our implementation, please look into below thread.

           --> It was working fine until I was greeted with a segmentation fault while accessing dlist!
           
           --> On further examining I found that dlist head is not NULL and it is pointing to some address, but the container I extracted is pointing to 0x7f7f7f7f7f7f7f5f, and I was not able to access any members in my container.

            From the above wiki, we came to a conclusion that the memory context in which that dlist was present was freed.
            And yes CLOBBER_FREED_MEMORY is enabled by passing --enable-cassert to enable asserts in my code.

           --> Normally the memory context inside XACT_EVENT_PRE_COMMIT is TopTransactionContext but in this particular case, the memory context was 'MessageContext'. Little unusual! Not sure why!

           --> So basically CurTransactionContext shouldn't get freed before transaction COMMIT.
           --> So what has actually happened here??? Kindly help us with your insights!

For your reference, a code snippet of our implementation is pasted below:

Sample code snippet:

/*** Code snippet starts ***/

dlist_head DMLInfo = {{NULL, NULL}};

Insert_callback()
{
      old_context = MemoryContextSwitchTo(CurTransactionContext);
      
      GraphIndex_InsertInfo *current;
      current = (GraphIndex_InsertInfo *)palloc(sizeof(GraphIndex_InsertInfo));
      current->tableOid = tableOid;
      current->subTransactionID = GetCurrentSubTransactionId();
      current->colValue = (long)values[0]; // Varies with column type
      current->ctid = ctid;

      dlist_push_head(&DMLInfo, &current->list_node);
      
      MemoryContextSwitchTo(old_context);
      return TRUE;
}

static void GraphIndex_xactCallback(XactEvent event, void *arg) 
{
      GraphIndex_Table *tableInfo;
      GraphIndex_InsertInfo *current;
      dlist_iter iter;

      switch (event) 
      {
              case XACT_EVENT_PRE_PREPARE:
              case XACT_EVENT_PRE_COMMIT:
              PG_TRY();
              {
                    if(DMLInfo.head.next)
                    {
                          dlist_reverse_foreach(iter, &DMLInfo)
                          {
                                current = dlist_container(GraphIndex_InsertInfo, list_node, iter.cur);
                                DMLProcessingFunction(current); // This is giving me the ERROR (while accessing members of current)
                          }
                          DMLInfo.head.next = NULL;
                    }
             }
             PG_CATCH();
             {
                   /* Do cleanup */
             }
             PG_END_TRY();

             break;

             case XACT_EVENT_ABORT:
             case XACT_EVENT_PARALLEL_ABORT:
                  /* No cleanup Necessary(No structure created) */
                  break;

             default:
                  return;
      }
}

/*** Code snippet ends ***/

Regards
G. Sai Ram


Re: [HACKERS] CurTransactionContext freed before transaction COMMIT ???

From
Tom Lane
Date:
Gaddam Sai Ram <gaddamsairam.n@zohocorp.com> writes:
>             We are implementing in-memory index. As a part of that, during index callbacks, under
CurTransactionContext,we cache all the DMLs of a transaction in dlist(postgres's doubly linked list). 
>             We registered transaction callback, and in transaction pre-commit event(XACT_EVENT_PRE_COMMIT), we
iteratethrough all cached DMLs(dlist) and populate in my in-memory data structure. 

This sounds broken on its face --- if you want stuff to survive to
top-level commit, you need to keep it in TopTransactionContext.
CurTransactionContext might be a subtransaction's context that will
go away at subtransaction commit/abort.
        regards, tom lane


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] CurTransactionContext freed before transaction COMMIT???

From
Gaddam Sai Ram
Date:


This sounds broken on its face --- if you want stuff to survive to
top-level commit, you need to keep it in TopTransactionContext.
CurTransactionContext might be a subtransaction's context that will
go away at subtransaction commit/abort.



From the above README:

CurTransactionContext --- this holds data that has to survive until the end
of the current transaction, and in particular will be needed at top-level
transaction commit.  When we are in a top-level transaction this is the same
as TopTransactionContext, but in subtransactions it points to a child context.
It is important to understand that if a subtransaction aborts, its
CurTransactionContext is thrown away after finishing the abort processing;
but a committed subtransaction's CurTransactionContext is kept until top-level
commit (unless of course one of the intermediate levels of subtransaction
aborts).  This ensures that we do not keep data from a failed subtransaction
longer than necessary.  Because of this behavior, you must be careful to clean
up properly during subtransaction abort --- the subtransaction's state must be
delinked from any pointers or lists kept in upper transactions, or you will
have dangling pointers leading to a crash at top-level commit.  An example of
data kept here is pending NOTIFY messages, which are sent at top-level commit,
but only if the generating subtransaction did not abort.

--> So even if sub-transaction is committed, subtransaction's CurTransactionContext is kept until top-level commit.
--> If sub-transaction is aborted, we handled(clearing the data) it via RegisterSubXactCallback().

And in our particular use case(which gave segmentation fault), we didn't issue any sub-transaction. It was a single insert transaction.
Even then it resulted in some kind of context free.


Regards
G. Sai Ram

Re: [HACKERS] CurTransactionContext freed before transaction COMMIT ???

From
Amit Kapila
Date:

On Tue, Oct 24, 2017 at 7:42 PM, Gaddam Sai Ram <gaddamsairam.n@zohocorp.com> wrote:

Hello people,

            We are implementing in-memory index. As a part of that, during index callbacks, under CurTransactionContext, we cache all the DMLs of a transaction in dlist(postgres's doubly linked list).
            We registered transaction callback, and in transaction pre-commit event(XACT_EVENT_PRE_COMMIT), we iterate through all cached DMLs(dlist) and populate in my in-memory data structure.

           --> For detailed explanation of our implementation, please look into below thread.

           --> It was working fine until I was greeted with a segmentation fault while accessing dlist!
           
           --> On further examining I found that dlist head is not NULL and it is pointing to some address, but the container I extracted is pointing to 0x7f7f7f7f7f7f7f5f, and I was not able to access any members in my container.

            From the above wiki, we came to a conclusion that the memory context in which that dlist was present was freed.
            And yes CLOBBER_FREED_MEMORY is enabled by passing --enable-cassert to enable asserts in my code.

           --> Normally the memory context inside XACT_EVENT_PRE_COMMIT is TopTransactionContext but in this particular case, the memory context was 'MessageContext'. Little unusual! Not sure why!

           --> So basically CurTransactionContext shouldn't get freed before transaction COMMIT.
           --> So what has actually happened here??? Kindly help us with your insights!


Can you check if CurTransactionContext is valid at that point?  To see, if this problem is related to CurTransactionContext, can you try to populate the list in TopMemoryContext and see if that works.


--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] CurTransactionContext freed before transaction COMMIT???

From
Gaddam Sai Ram
Date:
Thanks for the response,

Can you check if CurTransactionContext is valid at that point? 

Any Postgres function to check if CurTransactionContext is valid or not?

To see, if this problem is related to CurTransactionContext, can you try to populate the list in TopMemoryContext and see if that works.

Did you mean TopTransactionContext? 
As of now, we don't free our dlist. We solely depend on Postgres to free our dlist while it frees the TopTransactionContext. But if we do allocate in TopMemoryContext, we need to take care of freeing our allocations.

And one more issue is, we found this bug once in all the testing we did. So trying to replay this bug seems very difficult.

Regards,
G. Sai Ram

Re: [HACKERS] CurTransactionContext freed before transaction COMMIT ???

From
Michael Paquier
Date:
On Wed, Oct 25, 2017 at 7:34 AM, Gaddam Sai Ram
<gaddamsairam.n@zohocorp.com> wrote:
> Thanks for the response,
>
> Can you check if CurTransactionContext is valid at that point?
>
>
> Any Postgres function to check if CurTransactionContext is valid or not?

Assert(MemoryContextIsValid(CurTransactionContext));
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] CurTransactionContext freed before transaction COMMIT ???

From
Amit Kapila
Date:
On Wed, Oct 25, 2017 at 8:04 PM, Gaddam Sai Ram
<gaddamsairam.n@zohocorp.com> wrote:
> Thanks for the response,
>
> Can you check if CurTransactionContext is valid at that point?
>
>
> Any Postgres function to check if CurTransactionContext is valid or not?
>
> To see, if this problem is related to CurTransactionContext, can you try to
> populate the list in TopMemoryContext and see if that works.
>
>
> Did you mean TopTransactionContext?
>

No, I mean what I have written.  I suspect in your case
TopTransactionContext will be same as CurTransactionContext because
you don't have any subtransaction.

> As of now, we don't free our dlist. We solely depend on Postgres to free our
> dlist while it frees the TopTransactionContext. But if we do allocate in
> TopMemoryContext, we need to take care of freeing our allocations.
>

Can't we do it temporarily to test?  I am not suggesting to make this
a permanent change rather a way to see the reason of the problem.

> And one more issue is, we found this bug once in all the testing we did. So
> trying to replay this bug seems very difficult.
>

Oh, then it is tricky.  I think there is a good chance that this is
some of your application issues where you probably haven't used memory
context as required, so probably you need to figure out a way to
reproduce this issue, otherwise, it might be difficult to track down
the actual cause.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] CurTransactionContext freed before transaction COMMIT???

From
Gaddam Sai Ram
Date:
Thanks for your responses Michael and Amit Kapila,

            Yes, we have included your suggestions in our code and started testing to reproduce the same issue. In case we encounter this issue again we will get back here.

Regards
G. Sai Ram