Thread: Exporting closePGconn from libpq

A minor issue has come up in creating low-level bindings to libpq for
safe garbage-collected languages,  namely that PQfinish is the only
(AFAICT) way to close a connection but also de-allocates the memory
used to represent the database connection.    It would be preferable
to call PQfinish to free the memory in a finalizer,  but appilcations
need a way to disconnect from the database at a predictable and
deterministic point in time,  whereas leaving a bit of memory around
until the GC finally gets to it is relatively harmless.    The
low-level binding has a couple of options:

1.  Ignore the issue and allow for the possibility of a segfault if
the library is used incorrectly,  which is not a good situation for
"safe" languages.

2.  Create a wrapper that tracks whether or not PQfinish has ever been
called,  so that attempts to use a connection afterwards can be turned
into native exceptions/other forms of error signaling.  This kind of
solution can introduce their own minor issues.

3.  Hack libpq to export closePGconn so that libpq can safely signal
the low-level bindings of the error when a connection is used after it
is disconnected,  reserving PQfinish to run in a GC-triggered
finalizer.  While this is a technically preferable solution,  without
getting the change into upstream sources it is also a deployment
nightmare.

Is there any particular reason why closePGconn should not be exported
from libpq?

Best,
Leon

Leon Smith <leon.p.smith@gmail.com> writes:
> ... The low-level binding has a couple of options:

> 1.  Ignore the issue and allow for the possibility of a segfault if
> the library is used incorrectly,  which is not a good situation for
> "safe" languages.

> 2.  Create a wrapper that tracks whether or not PQfinish has ever been
> called,  so that attempts to use a connection afterwards can be turned
> into native exceptions/other forms of error signaling.  This kind of
> solution can introduce their own minor issues.

> 3.  Hack libpq to export closePGconn so that libpq can safely signal
> the low-level bindings of the error when a connection is used after it
> is disconnected,  reserving PQfinish to run in a GC-triggered
> finalizer.  While this is a technically preferable solution,  without
> getting the change into upstream sources it is also a deployment
> nightmare.

> Is there any particular reason why closePGconn should not be exported
> from libpq?

Yes: it'd introduce a new externally-visible state that libpq now has to
worry about supporting in all its operations, ie connection closed but
not gone.  This state is guaranteed to be poorly tested and hence buggy.

I think you need a wrapper object.  Given the context you're describing,
I'd be willing to lay a side bet that you'll end up needing a wrapper
anyway, even if it seems like you could avoid it right now.  Language
embeddings of libpq tend to accrete features...
        regards, tom lane

On Sat, May 14, 2011 at 11:37 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Yes: it'd introduce a new externally-visible state that libpq now has to
> worry about supporting in all its operations, ie connection closed but
> not gone.  This state is guaranteed to be poorly tested and hence buggy.

If you connect to a database over an unreliable network,  you can lose
the connection without warning at any time.  Thus libpq must already
support a "connection 'closed' but not gone" state,  and I'm fine with
making the "explicitly disconnected" state indistinguishable from the
"connection lost" state.

> I think you need a wrapper object.  Given the context you're describing,
> I'd be willing to lay a side bet that you'll end up needing a wrapper
> anyway, even if it seems like you could avoid it right now.  Language
> embeddings of libpq tend to accrete features...

The intention of the low-level bindings I'm working on is to keep
features to an absolute minimum; to bind calls to C in a 1-1 fashion
and to handle memory management and error signaling associated with
foreign calls.  Of course such a library is not intended to be
particularly attractive for application development,  but rather as a
library that can be wrapped up into a higher-level database access
library that's free to accrete features.  :)

Best,
Leon

Leon,

On 05/14/2011 05:23 PM, Leon Smith wrote:
> A minor issue has come up in creating low-level bindings to libpq for
> safe garbage-collected languages,  namely that PQfinish is the only
> (AFAICT) way to close a connection but also de-allocates the memory
> used to represent the database connection.    It would be preferable
> to call PQfinish to free the memory in a finalizer,  but appilcations
> need a way to disconnect from the database at a predictable and
> deterministic point in time,  whereas leaving a bit of memory around
> until the GC finally gets to it is relatively harmless.

It's harmless, but I think it's also useless.  Or why do you want to
keep that (libpq-private) bit of memory around beyond PQfinish()?

I'm not sure what language or VM you have in mind, but your description
sounds like you are writing a wrapper by definition.

Regards

Markus Wanner

On Sat, May 14, 2011 at 11:23 AM, Leon Smith <leon.p.smith@gmail.com> wrote:
> A minor issue has come up in creating low-level bindings to libpq for
> safe garbage-collected languages,  namely that PQfinish is the only
> (AFAICT) way to close a connection but also de-allocates the memory
> used to represent the database connection.    It would be preferable
> to call PQfinish to free the memory in a finalizer,  but appilcations
> need a way to disconnect from the database at a predictable and
> deterministic point in time,  whereas leaving a bit of memory around
> until the GC finally gets to it is relatively harmless.    The
> low-level binding has a couple of options:
>
> 1.  Ignore the issue and allow for the possibility of a segfault if
> the library is used incorrectly,  which is not a good situation for
> "safe" languages.

The 'safety' of the language has nothing to do with it. It should
possible to maintain state outside of libpq without crashing.  Which
language by the way?

> 2.  Create a wrapper that tracks whether or not PQfinish has ever been
> called,  so that attempts to use a connection afterwards can be turned
> into native exceptions/other forms of error signaling.  This kind of
> solution can introduce their own minor issues.

This is probably your answer.

> 3.  Hack libpq to export closePGconn so that libpq can safely signal
> the low-level bindings of the error when a connection is used after it
> is disconnected,  reserving PQfinish to run in a GC-triggered
> finalizer.  While this is a technically preferable solution,  without
> getting the change into upstream sources it is also a deployment
> nightmare.

This is not going to be as helpful as you think -- in many cases the
connection will not go 'bad' until you attempt to use...libpq runs in
thread and does no management when you are not inside a libpq call.
Your hypothetical callback would only get fired when you are making a
libpq call anyways and you haven't made a case why you can't just wrap
the query call with a a post call check to connection status.

merlin