Re: Connection terminated by the server causes deadlock in jdbc client side connection - Mailing list pgsql-jdbc

I think there maybe multiple bugs going on here and possibly not fixed...
First I'd like to say that even though we are dealing with TCP one must treat it almost like UDP. TCP packets can be consumed by firewalls and proxy servers with no response given to the sender. So with that in mind to create a robust fault tolerant driver, timeout on all TCP communications is a must.

1) So in the first case where an IOException is thrown, the original code that is closing the connection via "close" cannot have a timeout feature enabled and that will still exist in the code with the potential to fail...

2) When an IOException is thrown as the origin of an exception this should not be lost and should always be attached to the final exception. I don't see this cascaded in the updated driver using the "abort" code...

Regards
Donald

Steffen,

I can't really comment on what is the behavior of the PG server, but at least in the scenario that I am having (replication taking priority over a local query in hot_standby) either the client does not appear to receive an RST or Java 1.7 is still trying to write to that socket even if the reading end is gone, or even worse: the server is not actually closing it but simply no longer reading (Perhaps this is indeed hinting at a bug in the server side too?)

The stack trace does not give much detail either. This is the new trace I get after patching the driver to use abort() instead of close():

...

        at java.lang.Thread.run(Unknown Source)

Caused by: org.postgresql.util.PSQLException: FATAL: terminating connection due to conflict with recovery

  Detail: User query might have needed to see row versions that must be removed.

  Hint: In a moment you should be able to reconnect to the database and repeat your command.

        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)

        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)

        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)

        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:562)

        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:420)

        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:305)

        at com.jolbox.bonecp.PreparedStatementHandle.executeQuery(PreparedStatementHandle.java:174)

        at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:56)

        ... 42 more

As you can see, the IO Exception which caused the PSQL Exception is not coming up in the stack trace. 

Just for your awareness, please note that we are now using a snapshot that Dave created for us with the change above mentioned (to use abort() instead of close()) and we no longer see the issue now.

Regards,

Leonardo

On 8 October 2015 at 13:46, Steffen Heil (Mailinglisten) <lists@steffen-heil.de> wrote:

Hi


> I was referring to the fact that the TCP connection is indeed left 'half
> open'
> with the jdbc client side still assuming it's 'properly' open and thus
> trying to
> send a graceful close command over the socket ("X" in v2 or "X4" in v3) . It
> is
> this action that indefinitely blocks the client thread, which is then no
> longer
> usable by the application.

But it should not block indefinitely. Why would it?
If the server closed the connection, it should reply with a RST and the client
should detect that and throw an IOException( "Socket closed" ).


> All of this said, I still think it is conceptually incorrect to attempt a
> graceful
> closure after an IO error in the client.

Agreed. IOExceptions on blocking sockets usually indicate that you cannot do
anything good with them anymore.


Regards,
  Steffen