Thread: I think that my data is saved correctly, but when printing again,other data appears

I think that my data is saved correctly, but when printing again,other data appears

From
Yessica Brinkmann
Date:
Good day,
I am doing a thesis from the University. My thesis is the modification of the Gurjeet Index Adviser, to add some features.  
I have a function called get_columnnames, which runs on a Postgresql server and returns the names of the columns of a table in a Postgresql database. In the first foreach, in the statement: idxcd-> varattnames [cont] = data; column names are saved. Those column names are saved correctly. I could verify this by printing some values. 
But later, in the following foreach, when I reprint the values of the names of the columns that had been saved in the idxcd-> varattnames [cont] = data statement; They appear to me with errors. 
That is, for example: if a and b were saved, then print me b and b. This occurs only when foreach is entered more than once, that is, when there is more than one candidate index. The truth is that I thought a lot why this can happen but I can't find an answer. I will greatly appreciate an answer please. 
The line with emphasis (* *) is the one that produces the wrong result. 
I clarify that the foreach cycle for the c language is defined in #include "nodes / pg_list.h"
I am using Postgresql 8.3.23.
I clarify that I use that version because it is compatible with the Index Adviser, which I am modifying for my University thesis.  
The Linux distribution I use is Debian 9.x 64-bit.
And the version of C (gcc) that I use is 6.3.0  
I attach my source code. 
Best regards,
Yessica Brinkmann
Attachment
Yessica Brinkmann <yessica.brinkmann@gmail.com> writes:
> I have a function called get_columnnames, which runs on a Postgresql server
> and returns the names of the columns of a table in a Postgresql database.
> In the first foreach, in the statement: idxcd-> varattnames [cont] = data;
> column names are saved. Those column names are saved correctly. I could
> verify this by printing some values.
> But later, in the following foreach, when I reprint the values of the names
> of the columns that had been saved in the idxcd-> varattnames [cont] = data
> statement; They appear to me with errors.

Yeah, because the "data" value is just a pointer into the tupdesc
associated with the SPI result, and that disappears the moment you
do SPI_finish().  You'd need to do something to copy the strings
into a longer-lived context.  A plain pstrdup() won't suffice
because you're in a short-lived SPI context already inside that
loop; but you could save CurrentMemoryContext before starting up
SPI and then use MemoryContextStrdup.

            regards, tom lane



Re: I think that my data is saved correctly, but when printing again,other data appears

From
Yessica Brinkmann
Date:
Good evening, sorry for the delay in answering. I have a part-time job and I was at it.
I understand what you tell me about the "data" value is just a pointer into the tupdesc associated with the SPI result, and that disappears the moment I do SPI_finish ().
What I do not understand well is how to use CurrentMemoryContext and MemoryContextStrdup, since there are not many examples of using them on the Internet (most are only definitions) and it is the first time I have heard of this, although I already understand this part now of contexts, because I was reading on the subject.
Could you please give me an example of use?
Best Regards,
Yessica Brinkmann

El vie., 25 oct. 2019 a las 12:24, Yessica Brinkmann (<yessica.brinkmann@gmail.com>) escribió:
Thank you so much for your answer. I will be testing the indicated and then I give you return.
Best regards,

Yessica Brinkmann


-------- Mensaje original --------
Asunto: Re: I think that my data is saved correctly, but when printing again, other data appears
De: Tom Lane
Para: Yessica Brinkmann
CC: pgsql-general@lists.postgresql.org


Yessica Brinkmann writes:
> I have a function called get_columnnames, which runs on a Postgresql server
> and returns the names of the columns of a table in a Postgresql database.
> In the first foreach, in the statement: idxcd-> varattnames [cont] = data;
> column names are saved. Those column names are saved correctly. I could
> verify this by printing some values.
> But later, in the following foreach, when I reprint the values of the names
> of the columns that had been saved in the idxcd-> varattnames [cont] = data
> statement; They appear to me with errors.

Yeah, because the "data" value is just a pointer into the tupdesc
associated with the SPI result, and that disappears the moment you
do SPI_finish(). You'd need to do something to copy the strings
into a longer-lived context. A plain pstrdup() won't suffice
because you're in a short-lived SPI context already inside that
loop; but you could save CurrentMemoryContext before starting up
SPI and then use MemoryContextStrdup.

regards, tom lane

Re: I think that my data is saved correctly, but when printing again,other data appears

From
Yessica Brinkmann
Date:
Or if you know any link to a website where to find an example of this, I will thank you very much, please.
Best regards,
Yessica Brinkmann

El vie., 25 oct. 2019 a las 22:06, Yessica Brinkmann (<yessica.brinkmann@gmail.com>) escribió:
Good evening, sorry for the delay in answering. I have a part-time job and I was at it.
I understand what you tell me about the "data" value is just a pointer into the tupdesc associated with the SPI result, and that disappears the moment I do SPI_finish ().
What I do not understand well is how to use CurrentMemoryContext and MemoryContextStrdup, since there are not many examples of using them on the Internet (most are only definitions) and it is the first time I have heard of this, although I already understand this part now of contexts, because I was reading on the subject.
Could you please give me an example of use?
Best Regards,
Yessica Brinkmann

El vie., 25 oct. 2019 a las 12:24, Yessica Brinkmann (<yessica.brinkmann@gmail.com>) escribió:
Thank you so much for your answer. I will be testing the indicated and then I give you return.
Best regards,

Yessica Brinkmann


-------- Mensaje original --------
Asunto: Re: I think that my data is saved correctly, but when printing again, other data appears
De: Tom Lane
Para: Yessica Brinkmann
CC: pgsql-general@lists.postgresql.org


Yessica Brinkmann writes:
> I have a function called get_columnnames, which runs on a Postgresql server
> and returns the names of the columns of a table in a Postgresql database.
> In the first foreach, in the statement: idxcd-> varattnames [cont] = data;
> column names are saved. Those column names are saved correctly. I could
> verify this by printing some values.
> But later, in the following foreach, when I reprint the values of the names
> of the columns that had been saved in the idxcd-> varattnames [cont] = data
> statement; They appear to me with errors.

Yeah, because the "data" value is just a pointer into the tupdesc
associated with the SPI result, and that disappears the moment you
do SPI_finish(). You'd need to do something to copy the strings
into a longer-lived context. A plain pstrdup() won't suffice
because you're in a short-lived SPI context already inside that
loop; but you could save CurrentMemoryContext before starting up
SPI and then use MemoryContextStrdup.

regards, tom lane
On Fri, 2019-10-25 at 22:06 -0300, Yessica Brinkmann wrote:
> What I do not understand well is how to use CurrentMemoryContext and MemoryContextStrdup

For examples using memotry contexts, search the PostgreSQL source code.

To make memory management easier and safer, PostgreSQL has implemented
its own memory management using "memory contexts".

If you allocate memoty with "palloc", "pstrdup" and similar, that memory
is allocated in the current memory context.  If the memory context is
reset or dropped, the memory is gone.

MemoryContextStrdup enables you to create a copy of a string in an
explicitly specified memory context.

Yours,
Laurenz Albe
-- 
Cybertec | https://www.cybertec-postgresql.com




Hi,
Worked on something similar a few years back, have a look - it might give you a few pointers :)
It's on similar lines to what you are looking at (I kept it updated up to PG version 9.4/9.5)

My main focus was to add support for more index types: partial, functional, CTE, composites etc...
(I also had to keep the original context to restore it - so I think you will find a sample to what you are looking for... )

there's also https://github.com/HypoPG/hypopg which is actively maintained.
I think you will find code samples even up to the latest versions here...

Regards,
 - Jony

On Sat, Oct 26, 2019 at 4:06 AM Yessica Brinkmann <yessica.brinkmann@gmail.com> wrote:
Good evening, sorry for the delay in answering. I have a part-time job and I was at it.
I understand what you tell me about the "data" value is just a pointer into the tupdesc associated with the SPI result, and that disappears the moment I do SPI_finish ().
What I do not understand well is how to use CurrentMemoryContext and MemoryContextStrdup, since there are not many examples of using them on the Internet (most are only definitions) and it is the first time I have heard of this, although I already understand this part now of contexts, because I was reading on the subject.
Could you please give me an example of use?
Best Regards,
Yessica Brinkmann

El vie., 25 oct. 2019 a las 12:24, Yessica Brinkmann (<yessica.brinkmann@gmail.com>) escribió:
Thank you so much for your answer. I will be testing the indicated and then I give you return.
Best regards,

Yessica Brinkmann


-------- Mensaje original --------
Asunto: Re: I think that my data is saved correctly, but when printing again, other data appears
De: Tom Lane
Para: Yessica Brinkmann
CC: pgsql-general@lists.postgresql.org


Yessica Brinkmann writes:
> I have a function called get_columnnames, which runs on a Postgresql server
> and returns the names of the columns of a table in a Postgresql database.
> In the first foreach, in the statement: idxcd-> varattnames [cont] = data;
> column names are saved. Those column names are saved correctly. I could
> verify this by printing some values.
> But later, in the following foreach, when I reprint the values of the names
> of the columns that had been saved in the idxcd-> varattnames [cont] = data
> statement; They appear to me with errors.

Yeah, because the "data" value is just a pointer into the tupdesc
associated with the SPI result, and that disappears the moment you
do SPI_finish(). You'd need to do something to copy the strings
into a longer-lived context. A plain pstrdup() won't suffice
because you're in a short-lived SPI context already inside that
loop; but you could save CurrentMemoryContext before starting up
SPI and then use MemoryContextStrdup.

regards, tom lane

Re: I think that my data is saved correctly, but when printing again,other data appears

From
Yessica Brinkmann
Date:
Thank you so much for the answers.
By telling me this: "MemoryContextStrdup enables you to create a copy of a string in an explicitly specified memory context." I better understood the function of MemoryContextStrdup.
And thank you very much to Mr. Jony Cohen for giving me the reference of his work. I really was already researching about his work, and I included it in the state of the art of my thesis, but I didn't look at the source code.
I will be looking at the source code for a help, and especially in this case to see for the moment, the maintenance of the context for its subsequent restoration.
Regards, 
Yessica Brinkmann 

El dom., 27 oct. 2019 a las 19:42, Jony Cohen (<jony.cohenjo@gmail.com>) escribió:
Hi,
Worked on something similar a few years back, have a look - it might give you a few pointers :)
It's on similar lines to what you are looking at (I kept it updated up to PG version 9.4/9.5)

My main focus was to add support for more index types: partial, functional, CTE, composites etc...
(I also had to keep the original context to restore it - so I think you will find a sample to what you are looking for... )

there's also https://github.com/HypoPG/hypopg which is actively maintained.
I think you will find code samples even up to the latest versions here...

Regards,
 - Jony

On Sat, Oct 26, 2019 at 4:06 AM Yessica Brinkmann <yessica.brinkmann@gmail.com> wrote:
Good evening, sorry for the delay in answering. I have a part-time job and I was at it.
I understand what you tell me about the "data" value is just a pointer into the tupdesc associated with the SPI result, and that disappears the moment I do SPI_finish ().
What I do not understand well is how to use CurrentMemoryContext and MemoryContextStrdup, since there are not many examples of using them on the Internet (most are only definitions) and it is the first time I have heard of this, although I already understand this part now of contexts, because I was reading on the subject.
Could you please give me an example of use?
Best Regards,
Yessica Brinkmann

El vie., 25 oct. 2019 a las 12:24, Yessica Brinkmann (<yessica.brinkmann@gmail.com>) escribió:
Thank you so much for your answer. I will be testing the indicated and then I give you return.
Best regards,

Yessica Brinkmann


-------- Mensaje original --------
Asunto: Re: I think that my data is saved correctly, but when printing again, other data appears
De: Tom Lane
Para: Yessica Brinkmann
CC: pgsql-general@lists.postgresql.org


Yessica Brinkmann writes:
> I have a function called get_columnnames, which runs on a Postgresql server
> and returns the names of the columns of a table in a Postgresql database.
> In the first foreach, in the statement: idxcd-> varattnames [cont] = data;
> column names are saved. Those column names are saved correctly. I could
> verify this by printing some values.
> But later, in the following foreach, when I reprint the values of the names
> of the columns that had been saved in the idxcd-> varattnames [cont] = data
> statement; They appear to me with errors.

Yeah, because the "data" value is just a pointer into the tupdesc
associated with the SPI result, and that disappears the moment you
do SPI_finish(). You'd need to do something to copy the strings
into a longer-lived context. A plain pstrdup() won't suffice
because you're in a short-lived SPI context already inside that
loop; but you could save CurrentMemoryContext before starting up
SPI and then use MemoryContextStrdup.

regards, tom lane
On 10/27/19 8:01 PM, Yessica Brinkmann wrote:
> Thank you so much for the answers.
> By telling me this: "MemoryContextStrdup enables you to create a copy of
> a string in an explicitly specified memory context." I better understood
> the function of MemoryContextStrdup.
> And thank you very much to Mr. Jony Cohen for giving me the reference of
> his work. I really was already researching about his work, and I
> included it in the state of the art of my thesis, but I didn't look at
> the source code.
> I will be looking at the source code for a help, and especially in this
> case to see for the moment, the maintenance of the context for its
> subsequent restoration.

For better understanding of how Postgres manages memory, you might want
to also read this:

https://github.com/postgres/postgres/blob/master/src/backend/utils/mmgr/README

and possibly browse through this:

https://github.com/postgres/postgres/blob/master/src/backend/utils/mmgr/mcxt.c

HTH,

Joe
--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development


Attachment

Re: I think that my data is saved correctly, but when printing again,other data appears

From
Yessica Brinkmann
Date:
Thank you very much for the answer.
Best regards,
Yessica Brinkmann

El lun., 28 oct. 2019 a las 8:03, Joe Conway (<mail@joeconway.com>) escribió:
On 10/27/19 8:01 PM, Yessica Brinkmann wrote:
> Thank you so much for the answers.
> By telling me this: "MemoryContextStrdup enables you to create a copy of
> a string in an explicitly specified memory context." I better understood
> the function of MemoryContextStrdup.
> And thank you very much to Mr. Jony Cohen for giving me the reference of
> his work. I really was already researching about his work, and I
> included it in the state of the art of my thesis, but I didn't look at
> the source code.
> I will be looking at the source code for a help, and especially in this
> case to see for the moment, the maintenance of the context for its
> subsequent restoration.

For better understanding of how Postgres manages memory, you might want
to also read this:

https://github.com/postgres/postgres/blob/master/src/backend/utils/mmgr/README

and possibly browse through this:

https://github.com/postgres/postgres/blob/master/src/backend/utils/mmgr/mcxt.c

HTH,

Joe
--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

Re: I think that my data is saved correctly, but when printing again,other data appears

From
Yessica Brinkmann
Date:
Good afternoon,
Well, I spent some time without implementing my thesis for a topic of mental fatigue, I had to go for a while.
A few days ago I came back and was trying to implement the context change.
But I really believe that, although I read the explanations and source codes that you indicated, and others that I found, I didn't quite understand what I should do.
Well, as I was told that I should save the CurrentMemoryContext before starting the SPI,
Let's say it would be before doing SPI_connect (). Is this correct?
And then I must use MemoryContextStrdup. As you told me the MemoryContextStrdup It is used to create a copy of a string in a specific memory context.
Well, where in the source code should I use MemoryContextStrdup? After doing the SPI_connect () or where? I would use it from MemoryContextStrdup to copy the data variable as I understand it, But in what context would I have to create the copy? In a new context or what would the theme be like?
Should I use AllocSetContextCreate to create the new context or what would the theme be like?
And if I have to create the new context with AllocSetContextCreate, where in the source code will I have to create it? After doing SPI_connect () or where?
The truth is that I also read the source code of https://github.com/cohenjo/pg_idx_advisor but I don't see that The MemoryContextStrdup that they told me to use has been used there.
Sorry for the inconvenience and see the same thing again. But as I indicated, I also read the explanations they told me to read (which were explanations about the memory contexts in Postgresql mainly) but there were no examples of source code.
And since there is nothing on the Internet of examples that will help me to use, I am asking again.
And the truth is that I didn't find examples of this in the Postgres source code, just definitions, That is the source code where MemoryContextStrdup is defined.
It may be very easy for you and you will see it very clearly, but for me it really is not, and there are no examples of use on the Internet.
I really searched a lot and found nothing.
I would greatly appreciate a help please.
Regards,
Yessica Brinkmann

El lun., 28 oct. 2019 a las 12:39, Yessica Brinkmann (<yessica.brinkmann@gmail.com>) escribió:
Thank you very much for the answer.
Best regards,
Yessica Brinkmann

El lun., 28 oct. 2019 a las 8:03, Joe Conway (<mail@joeconway.com>) escribió:
On 10/27/19 8:01 PM, Yessica Brinkmann wrote:
> Thank you so much for the answers.
> By telling me this: "MemoryContextStrdup enables you to create a copy of
> a string in an explicitly specified memory context." I better understood
> the function of MemoryContextStrdup.
> And thank you very much to Mr. Jony Cohen for giving me the reference of
> his work. I really was already researching about his work, and I
> included it in the state of the art of my thesis, but I didn't look at
> the source code.
> I will be looking at the source code for a help, and especially in this
> case to see for the moment, the maintenance of the context for its
> subsequent restoration.

For better understanding of how Postgres manages memory, you might want
to also read this:

https://github.com/postgres/postgres/blob/master/src/backend/utils/mmgr/README

and possibly browse through this:

https://github.com/postgres/postgres/blob/master/src/backend/utils/mmgr/mcxt.c

HTH,

Joe
--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

On 11/25/19 4:38 PM, Yessica Brinkmann wrote:
> Well, as I was told that I should save the
> CurrentMemoryContext before starting the SPI, Let's say it would be
> before doing SPI_connect (). Is this correct? And then I must use
> MemoryContextStrdup. As you told me the MemoryContextStrdup It is
> used to create a copy of a string in a specific memory context. Well,
> where in the source code should I use MemoryContextStrdup? After
> doing the SPI_connect () or where? I would use it from
> MemoryContextStrdup to copy the data variable as I understand it, But
> in what context would I have to create the copy? In a new context or
> what would the theme be like? Should I use AllocSetContextCreate to
> create the new context or what would the theme be like? And if I have
> to create the new context with AllocSetContextCreate, where in the
> source code will I have to create it? After doing SPI_connect () or
> where? The truth is that I also read the source code of
> https://github.com/cohenjo/pg_idx_advisor but I don't see that The
> MemoryContextStrdup that they told me to use has been used there.
> Sorry for the inconvenience and see the same thing again. But as I
> indicated, I also read the explanations they told me to read (which
> were explanations about the memory contexts in Postgresql mainly) but
> there were no examples of source code. And since there is nothing on
> the Internet of examples that will help me to use, I am asking
> again. And the truth is that I didn't find examples of this in the
> Postgres source code, just definitions, That is the source code
> where MemoryContextStrdup is defined. It may be very easy for you and
> you will see it very clearly, but for me it really is not, and there
> are no examples of use on the Internet. I really searched a lot and
> found nothing. I would greatly appreciate a help please.
Sorry but I am not going to try to address that wall of text ;-)
But here is some general information about how that stuff works:
---------------------------------
1. The most common pattern is something like this:

   MemoryContext oldcontext;

   oldcontext = MemoryContextSwitchTo(<some_memory_context>);

   /* do stuff that allocates memory
    * using PostgreSQL allocation functions
    * e.g. palloc, pstrdup, other exported
    * backend functions, etc
    */

   MemoryContextSwitchTo(oldcontext);

2. MemoryContextStrdup() is similar to the above, except in that case
   you do not need MemoryContextSwitchTo(). It directly allocates into
   the specified memory context without all the switching back and
   forth. If you are simply copying one string and need it in a context
   other than what is current, it is more convenient. But either method
   could be used.

3. When you run SPI_connect() the memory context is switched
   transparently for you to a special SPI memory context. When you run
   SPI_finish() the original memory context (the one in effect before
   SPI_connect) is restored.

4. Depending on what you are trying to do, use method #1 or method #2 if
   needed, including while doing SPI related things (in between
   SPI_connect and SPI_finish)

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development


Attachment

Re: I think that my data is saved correctly, but when printing again,other data appears

From
Yessica Brinkmann
Date:
I understand. Thank you very much for clearing things up.
It helps me a lot, especially point 3.
"3. When you run SPI_connect () the memory context is switched
    transparently for you to a special SPI memory context. When you run
    SPI_finish () the original memory context (the one in effect before
    SPI_connect) is restored. "
Best regards,
Yessica Brinkmann

El mar., 26 nov. 2019 a las 15:25, Joe Conway (<mail@joeconway.com>) escribió:
On 11/25/19 4:38 PM, Yessica Brinkmann wrote:
> Well, as I was told that I should save the
> CurrentMemoryContext before starting the SPI, Let's say it would be
> before doing SPI_connect (). Is this correct? And then I must use
> MemoryContextStrdup. As you told me the MemoryContextStrdup It is
> used to create a copy of a string in a specific memory context. Well,
> where in the source code should I use MemoryContextStrdup? After
> doing the SPI_connect () or where? I would use it from
> MemoryContextStrdup to copy the data variable as I understand it, But
> in what context would I have to create the copy? In a new context or
> what would the theme be like? Should I use AllocSetContextCreate to
> create the new context or what would the theme be like? And if I have
> to create the new context with AllocSetContextCreate, where in the
> source code will I have to create it? After doing SPI_connect () or
> where? The truth is that I also read the source code of
> https://github.com/cohenjo/pg_idx_advisor but I don't see that The
> MemoryContextStrdup that they told me to use has been used there.
> Sorry for the inconvenience and see the same thing again. But as I
> indicated, I also read the explanations they told me to read (which
> were explanations about the memory contexts in Postgresql mainly) but
> there were no examples of source code. And since there is nothing on
> the Internet of examples that will help me to use, I am asking
> again. And the truth is that I didn't find examples of this in the
> Postgres source code, just definitions, That is the source code
> where MemoryContextStrdup is defined. It may be very easy for you and
> you will see it very clearly, but for me it really is not, and there
> are no examples of use on the Internet. I really searched a lot and
> found nothing. I would greatly appreciate a help please.
Sorry but I am not going to try to address that wall of text ;-)
But here is some general information about how that stuff works:
---------------------------------
1. The most common pattern is something like this:

   MemoryContext oldcontext;

   oldcontext = MemoryContextSwitchTo(<some_memory_context>);

   /* do stuff that allocates memory
    * using PostgreSQL allocation functions
    * e.g. palloc, pstrdup, other exported
    * backend functions, etc
    */

   MemoryContextSwitchTo(oldcontext);

2. MemoryContextStrdup() is similar to the above, except in that case
   you do not need MemoryContextSwitchTo(). It directly allocates into
   the specified memory context without all the switching back and
   forth. If you are simply copying one string and need it in a context
   other than what is current, it is more convenient. But either method
   could be used.

3. When you run SPI_connect() the memory context is switched
   transparently for you to a special SPI memory context. When you run
   SPI_finish() the original memory context (the one in effect before
   SPI_connect) is restored.

4. Depending on what you are trying to do, use method #1 or method #2 if
   needed, including while doing SPI related things (in between
   SPI_connect and SPI_finish)

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development