Thread: Re: Memory Errors...

Re: Memory Errors...

From

"Nigel J. Andrews"

Date:

20 September 2002, 21:18:20

Ok, below is the original email I sent, which I can not remember seeing come
across the patches list. Please do read the assumptions since they might throw
up problems with what I have done.

I have attached the pltcl patch again, just in case. For the sake of clarity
let's say this patch superscedes the previous one.

I have also attached a patch addressing the similar memory leak problem in
plpython. This includes a slight adjustment of the tests in the source
directory. The patch also includes a cosmetic change to remove a compiler
warning although I think the change makes the code look worse though.

Once again, please read my text below and also take a quick look at the comment
I've added in the plpython patch since it may well show that that
particular change is complete rubbish.

BTW, by my reckoning the memory leak would occur with prepared plans and
without. If that is not the case then I've been barking up the wrong tree.

Of further note, I have not tested for the memory leak in plpython but the
build passes the normal and big checks. However, I have tried testing using the
test.sh script in src/pl/plpython. This seems to be generating errors where
before there were warnings. Can anyone comment on the correctness of this?
Reversing my changes doesn't really help matters so I presume it is something
else that is causing the different behaviour.

--
Nigel J. Andrews

On Fri, 20 Sep 2002, Nigel J. Andrews wrote:

> On Thu, 19 Sep 2002, Tom Lane wrote:
>
> > "Ian Harding" <ianh@tpchd.org> writes:
> > > It is pltcl [not plpgsql]
> >
> > Ah.  I don't think we've done much of any work on plugging leaks in
> > pltcl :-(.
> >
> > > It hurts when I do this:
> >
> > > drop function memleak();
> > > create function memleak() returns int as '
> > > for {set counter 1} {$counter < 100000} {incr counter} {
> > >         set sql "select ''foo''"
> > >         spi_exec "$sql"
> > > }
> > > ' language 'pltcl';
> > > select memleak();
> >
> > Yeah, I see very quick memory exhaustion also :-(.  Looks like the
> > spi_exec call is the culprit, but I'm not sure exactly why ...
> > anyone have time to look at this?
>
> Attached is a patch that frees the SPI_tuptable in all post SPI_exec
> non-elog paths in both pltcl_SPI_exec() and pltcl_SPI_execp().
>
> The fault as triggered by the above code has been fixed by this patch but
> please read my assumptions below to ensure they are correct.
>
> I have assumed that Tom's comment about this only being required in non-elog
> paths is correct, which seems a reasonable assumption to me.
>
> I have also assumed, rather than verified, that freeing the tuptable does
> indeed free the tuples as well. Tests with the above function show that the
> process does not increase it's memory footprint during it's operation, although
> if my assumption here is wrong this could be a feature of selecting
> insignificantly sized tuples.
>
> I have not worried about other uses of SPI_exec for selects in pltcl.c on the
> basis that those are not under the control of the function writer and the
> normal function management will release the storage.

Re: Memory Errors...

From

Bruce Momjian

Date:

23 September 2002, 00:53:34

[ Previous version removed from patches queue..]

Thanks for doing both interfaces.

Your patch has been added to the PostgreSQL unapplied patches list at:

    http://candle.pha.pa.us/cgi-bin/pgpatches

I will try to apply it within the next 48 hours.

---------------------------------------------------------------------------


Nigel J. Andrews wrote:
>
> Ok, below is the original email I sent, which I can not remember seeing come
> across the patches list. Please do read the assumptions since they might throw
> up problems with what I have done.
>
> I have attached the pltcl patch again, just in case. For the sake of clarity
> let's say this patch superscedes the previous one.
>
> I have also attached a patch addressing the similar memory leak problem in
> plpython. This includes a slight adjustment of the tests in the source
> directory. The patch also includes a cosmetic change to remove a compiler
> warning although I think the change makes the code look worse though.
>
> Once again, please read my text below and also take a quick look at the comment
> I've added in the plpython patch since it may well show that that
> particular change is complete rubbish.
>
> BTW, by my reckoning the memory leak would occur with prepared plans and
> without. If that is not the case then I've been barking up the wrong tree.
>
> Of further note, I have not tested for the memory leak in plpython but the
> build passes the normal and big checks. However, I have tried testing using the
> test.sh script in src/pl/plpython. This seems to be generating errors where
> before there were warnings. Can anyone comment on the correctness of this?
> Reversing my changes doesn't really help matters so I presume it is something
> else that is causing the different behaviour.
>
>
> --
> Nigel J. Andrews
>
>
> On Fri, 20 Sep 2002, Nigel J. Andrews wrote:
>
> > On Thu, 19 Sep 2002, Tom Lane wrote:
> >
> > > "Ian Harding" <ianh@tpchd.org> writes:
> > > > It is pltcl [not plpgsql]
> > >
> > > Ah.  I don't think we've done much of any work on plugging leaks in
> > > pltcl :-(.
> > >
> > > > It hurts when I do this:
> > >
> > > > drop function memleak();
> > > > create function memleak() returns int as '
> > > > for {set counter 1} {$counter < 100000} {incr counter} {
> > > >         set sql "select ''foo''"
> > > >         spi_exec "$sql"
> > > > }
> > > > ' language 'pltcl';
> > > > select memleak();
> > >
> > > Yeah, I see very quick memory exhaustion also :-(.  Looks like the
> > > spi_exec call is the culprit, but I'm not sure exactly why ...
> > > anyone have time to look at this?
> >
> > Attached is a patch that frees the SPI_tuptable in all post SPI_exec
> > non-elog paths in both pltcl_SPI_exec() and pltcl_SPI_execp().
> >
> > The fault as triggered by the above code has been fixed by this patch but
> > please read my assumptions below to ensure they are correct.
> >
> > I have assumed that Tom's comment about this only being required in non-elog
> > paths is correct, which seems a reasonable assumption to me.
> >
> > I have also assumed, rather than verified, that freeing the tuptable does
> > indeed free the tuples as well. Tests with the above function show that the
> > process does not increase it's memory footprint during it's operation, although
> > if my assumption here is wrong this could be a feature of selecting
> > insignificantly sized tuples.
> >
> > I have not worried about other uses of SPI_exec for selects in pltcl.c on the
> > basis that those are not under the control of the function writer and the
> > normal function management will release the storage.
>

Content-Description:

[ Attachment, skipping... ]

Content-Description:

[ Attachment, skipping... ]

Content-Description:

[ Attachment, skipping... ]

>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
>
> http://archives.postgresql.org

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: Memory Errors...

From

Bruce Momjian

Date:

26 September 2002, 04:23:54

Patch applied.  Thanks.

---------------------------------------------------------------------------



Nigel J. Andrews wrote:
>
> Ok, below is the original email I sent, which I can not remember seeing come
> across the patches list. Please do read the assumptions since they might throw
> up problems with what I have done.
>
> I have attached the pltcl patch again, just in case. For the sake of clarity
> let's say this patch superscedes the previous one.
>
> I have also attached a patch addressing the similar memory leak problem in
> plpython. This includes a slight adjustment of the tests in the source
> directory. The patch also includes a cosmetic change to remove a compiler
> warning although I think the change makes the code look worse though.
>
> Once again, please read my text below and also take a quick look at the comment
> I've added in the plpython patch since it may well show that that
> particular change is complete rubbish.
>
> BTW, by my reckoning the memory leak would occur with prepared plans and
> without. If that is not the case then I've been barking up the wrong tree.
>
> Of further note, I have not tested for the memory leak in plpython but the
> build passes the normal and big checks. However, I have tried testing using the
> test.sh script in src/pl/plpython. This seems to be generating errors where
> before there were warnings. Can anyone comment on the correctness of this?
> Reversing my changes doesn't really help matters so I presume it is something
> else that is causing the different behaviour.
>
>
> --
> Nigel J. Andrews
>
>
> On Fri, 20 Sep 2002, Nigel J. Andrews wrote:
>
> > On Thu, 19 Sep 2002, Tom Lane wrote:
> >
> > > "Ian Harding" <ianh@tpchd.org> writes:
> > > > It is pltcl [not plpgsql]
> > >
> > > Ah.  I don't think we've done much of any work on plugging leaks in
> > > pltcl :-(.
> > >
> > > > It hurts when I do this:
> > >
> > > > drop function memleak();
> > > > create function memleak() returns int as '
> > > > for {set counter 1} {$counter < 100000} {incr counter} {
> > > >         set sql "select ''foo''"
> > > >         spi_exec "$sql"
> > > > }
> > > > ' language 'pltcl';
> > > > select memleak();
> > >
> > > Yeah, I see very quick memory exhaustion also :-(.  Looks like the
> > > spi_exec call is the culprit, but I'm not sure exactly why ...
> > > anyone have time to look at this?
> >
> > Attached is a patch that frees the SPI_tuptable in all post SPI_exec
> > non-elog paths in both pltcl_SPI_exec() and pltcl_SPI_execp().
> >
> > The fault as triggered by the above code has been fixed by this patch but
> > please read my assumptions below to ensure they are correct.
> >
> > I have assumed that Tom's comment about this only being required in non-elog
> > paths is correct, which seems a reasonable assumption to me.
> >
> > I have also assumed, rather than verified, that freeing the tuptable does
> > indeed free the tuples as well. Tests with the above function show that the
> > process does not increase it's memory footprint during it's operation, although
> > if my assumption here is wrong this could be a feature of selecting
> > insignificantly sized tuples.
> >
> > I have not worried about other uses of SPI_exec for selects in pltcl.c on the
> > basis that those are not under the control of the function writer and the
> > normal function management will release the storage.
>

Content-Description:

[ Attachment, skipping... ]

Content-Description:

[ Attachment, skipping... ]

Content-Description:

[ Attachment, skipping... ]

>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
>
> http://archives.postgresql.org

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073