Hello,
I implemented connection pooling in the backend of postgresql (one of the
TODO items) and I've found some really interesting performance improvements
I'd like to continue to explore but unfortunately I've also uncovered a
nasty little side effect with the memory usage. It's my guess that it's
either a number of mallocs don't have corresponding frees because the
process would exit when the transaction is done (normally) or I've bypassed
some of the memory context "free"s that would have also resulted in memory
being reclaimed on a process exit or more importantly on a connection being
reclosed.
The implementation is different from front-end pooling in that the client
does not keep their connection open with the database. The socket is closed
for example. Also a basic authorization check (to make sure the client
credentials match those of an existing pooled connection) happens. So the
connection does get some of the initialization that occurs for a new
connection (but not all, a lot of it can be skipped because it has already
been setup).
Anyway, my hope is there might be someone who is familiar with either the
memory system or the connection startup and shutdown process enough that
could provide a few pointers as to where I could start looking. I've been
instrumenting the code to track malloc/pmalloc and free/pfree calls to see
which ones don't match for example but it's a bit laborious and would be
simpler if I could narrow down where I think things might be having
problems.
The implementation was done on 7.3.4. It was the latest at the time I
started and I haven't sync'd up yet because I wasn't sure if there would be
a need in terms of the general postgresql population wanting this feature.
If the improvements prove to be real and generalized versus specialized
(just a few use cases for example) then I would imagine the interest would
grow but I'm not there yet.
I can get you copies of the changes I've made. I changed 27 files (some of
the changes are minor, some more complicated) and I added 2 files as well
(for the pooling functionality).
Thank you,
Mike
