Re: Memory use in 8.3 plpgsql with heavy use of xpath() - Mailing list pgsql-general

"Matt Magoffin" <postgresql.org@msqr.us> writes:
> Below is a test case that simulates the use of xpath() within a plpgsql
> function in my application.

I'm able to duplicate the memory leak in this function with the current
Fedora 8 libxml2 (2.6.32).  The leak is definitely inside libxml2
itself, because the bloat shows up here:

LibxmlContext: 150986752 total in 27 blocks; 6025448 free (204 chunks); 144961304 used

and that context is only used for allocations requested by libxml2.
(We reset that context at every transaction end, so the only way for
the problem to become really noticeable in Postgres is to do a whole
lot of xpath() calls in a single transaction.)

Some idle looking through the context suggests that the leakage consists
of short individually palloc'd null-terminated strings that seem to be
words from your test data.  I tried dumping the context stats at
successive entries to xpath():

LibxmlContext: 260038656 total in 40 blocks; 3934120 free (204 chunks); 256104536 used
LibxmlContext: 260038656 total in 40 blocks; 3934088 free (204 chunks); 256104568 used
LibxmlContext: 260038656 total in 40 blocks; 3934056 free (204 chunks); 256104600 used
LibxmlContext: 260038656 total in 40 blocks; 3934056 free (205 chunks); 256104600 used
LibxmlContext: 260038656 total in 40 blocks; 3934000 free (205 chunks); 256104656 used
LibxmlContext: 260038656 total in 40 blocks; 3933960 free (204 chunks); 256104696 used
LibxmlContext: 260038656 total in 40 blocks; 3933928 free (204 chunks); 256104728 used
LibxmlContext: 260038656 total in 40 blocks; 3933896 free (204 chunks); 256104760 used
LibxmlContext: 260038656 total in 40 blocks; 3933856 free (204 chunks); 256104800 used
LibxmlContext: 260038656 total in 40 blocks; 3933816 free (204 chunks); 256104840 used
LibxmlContext: 260038656 total in 40 blocks; 3933784 free (204 chunks); 256104872 used
LibxmlContext: 260038656 total in 40 blocks; 3933752 free (204 chunks); 256104904 used
LibxmlContext: 260038656 total in 40 blocks; 3933752 free (205 chunks); 256104904 used
LibxmlContext: 260038656 total in 40 blocks; 3933696 free (205 chunks); 256104960 used
LibxmlContext: 260038656 total in 40 blocks; 3933656 free (204 chunks); 256105000 used
LibxmlContext: 260038656 total in 40 blocks; 3933624 free (204 chunks); 256105032 used

which shows that not every call in this test case leaks memory, but the
majority do, and that the leakage runs between 32 and 56 bytes per call.
This is on a 64-bit machine with cassert enabled, which means that the
palloc overhead is 24 bytes per chunk.  So I think we can fairly
confidently assert that libxml is leaking exactly one short string per
xpath() call --- the leak is too small for it to be more than one palloc
chunk at a time.

I looked through the libxml2 sources a little bit but couldn't
immediately find the problem.  I'm fairly confident though that
this could be reproduced outside Postgres, by replicating the sequence
of libxml2 calls we make in xpath().  The next step should probably be
to build a reproducer program and submit it to the libxml authors,
or maybe run it under a debugging malloc package that could help
identify where the leak is.  Anyone want to do that?

            regards, tom lane