Thread: Memory leakage associated with plperl spi_prepare/spi_freeplan
I looked into the problem described here: http://www.postgresql.org/message-id/5125087D.8090105@deriva.de The core of the problem is that plperl's plperl_spi_prepare() sets up input conversion functions for the parameters of a prepared query using perm_fmgr_info(), which allocates FmgrInfo structs for the I/O functions in TopMemoryContext and makes their fn_mcxt values point to TopMemoryContext too. So when domains.c allocates assorted stuff in fn_mcxt, that stuff lives forever ... but guess what, the prepared query doesn't. So we have a memory leak on every use of spi_freeplan(). The leak is particularly egregious for this specific test case, where an 8K-or-so ExprContext is made as a consequence of domain_check_input's call to CreateStandaloneExprContext(); but it would add up to something noticeable eventually even with input functions as innocuous as int4in. Proof is to try this: create or replace function perlleak(n int) returns void as $$ my ($n) = @_; while ($n--) { my $stmt = spi_prepare('select $1', 'int'); spi_freeplan($stmt); } $$ language plperl volatile strict; with repeat count of a million or so. I'm surprised we've not seen this reported before --- maybe people don't tend to use spi_freeplan() much in plperl. I'm inclined to think the right fix is to make a small memory context for each prepared plan made by plperl_spi_prepare(). The qdesc for it could be made right in the context (getting rid of the unchecked malloc's near the top of the function), the FmgrInfos and their subsidiary data could live there too, and plperl_spi_freeplan could replace its retail free's with a single MemoryContextDelete. Not being particularly a plperl user, I don't really want to code and test this. Any takers? regards, tom lane
On Tue, Feb 26, 2013 at 3:56 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
I'm inclined to think the right fix is to make a small memory context
for each prepared plan made by plperl_spi_prepare(). The qdesc for it
could be made right in the context (getting rid of the unchecked
malloc's near the top of the function), the FmgrInfos and their
subsidiary data could live there too, and plperl_spi_freeplan could
replace its retail free's with a single MemoryContextDelete.
Seemed fairly trivial, find the above approach in the attached. I added a CHECK_FOR_INTERRUPTS() at the top of plperl_spi_prepare(), it was fairly annoying that I couldn't ctrl+c my way out of test function.
One annonce is it still leaks :-(. I tracked it down and it seemed to stem from parseTypeString(). I chased down the rabbit hole for a bit, but nothing jumped out... raw_parser() is a bit of a black box to me. Adding the seemingly obvious list_free(raw_parsetree_list); or setting the memory context before parseTypeString() didn't seem to do much.
It would be nice to squish the other leaks due to perm_fmgr_info()... but this is a start.
Attachment
Alex Hunsaker <badalex@gmail.com> writes: > On Tue, Feb 26, 2013 at 3:56 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> I'm inclined to think the right fix is to make a small memory context >> for each prepared plan made by plperl_spi_prepare(). The qdesc for it >> could be made right in the context (getting rid of the unchecked >> malloc's near the top of the function), the FmgrInfos and their >> subsidiary data could live there too, and plperl_spi_freeplan could >> replace its retail free's with a single MemoryContextDelete. > Seemed fairly trivial, find the above approach in the attached. Applied with some fixes. > I added a > CHECK_FOR_INTERRUPTS() at the top of plperl_spi_prepare(), it was fairly > annoying that I couldn't ctrl+c my way out of test function. Good idea, but it wasn't safe where it was --- needs to be inside the PG_TRY(), so as to convert from postgres to perl error handling. > One annonce is it still leaks :-(. I fixed that, at least for the function-lifespan leakage from spi_prepare() --- is that what you meant? > It would be nice to squish the other leaks due to perm_fmgr_info()... but > this is a start. Yeah, I'm sure there's more left to do in the area --- but at least you can create and free plans without it leaking. regards, tom lane
On Fri, Mar 1, 2013 at 7:38 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > Alex Hunsaker <badalex@gmail.com> writes: > > Applied with some fixes. Thanks! Your version looks much better than mine. > > One annonce is it still leaks :-(. > > I fixed that, at least for the function-lifespan leakage from > spi_prepare() --- is that what you meant? > Yep, I don't see the leak with your version.