Postgres Pro
Downloads
Home   >   mailing lists

Thread: connection reset error, followed by broken pipe (pg74.213.jdbc3.jar)

connection reset error, followed by broken pipe (pg74.213.jdbc3.jar)

From
"Ed Blackmore"
Date:
Hi,

I'm having trouble with a tomcat-postgres installation. Server 1 runs
postgresql and tomcat, server 2 has an identical tomcat installation
and application code running (except rebranding). Server 2 has errors
about every week with 'connection reset' in logs/appname.date.log,
futher connection attempts have a 'broken pipe' error. The solution
to get the app running again is to restart tomcat. Both servers have
low load ~50 users per week.

tomcat 5.0.28
postgresql 7.4.8
java jdk 1.5.0_02
jdbc driver pg74.213.jdbc3.jar

Stack trace errors are:

2006-04-13 15:54:03 StandardWrapperValve[AccessPortal]:
Servlet.service() for servlet AccessPortal threw exception
javax.servlet.ServletException: org.postgresql.util.PSQLException: An
I/O error has occured while flushing the output - Exception:
java.net.SocketException: Connection reset
Stack Trace:

java.net.SocketException: Connection reset
    at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
    at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
    at org.postgresql.core.PGStream.flush(PGStream.java:411)
    at
org.postgresql.core.QueryExecutor.sendQueryV3(QueryExecutor.java:337)
    at
org.postgresql.core.QueryExecutor.executeV3(QueryExecutor.java:121)
    at org.postgresql.core.QueryExecutor.execute(QueryExecutor.java:100)
    at org.postgresql.core.QueryExecutor.execute(QueryExecutor.java:43)
    at
org.postgresql.jdbc1.AbstractJdbc1Statement.execute(AbstractJdbc1State
ment.java:517)
    at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2State
ment.java:50)
    at
org.postgresql.jdbc1.AbstractJdbc1Statement.executeQuery(AbstractJdbc1
Statement.java:233)
    at
org.postgresql.jdbc1.AbstractJdbc1Statement.executeQuery(AbstractJdbc1
Statement.java:221)
    at
org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingSta
tement.java:205)
    at
com.ourdomain.ourapp.AccessPortal.getUserData(AccessPortal.java:126)
    at
com.ourdomain.ourapp.AccessPortal.processRequest(AccessPortal.java:71)
    at com.ourdomain.ourapp.AccessPortal.doGet(AccessPortal.java:88)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
cationFilterChain.java:237)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi
lterChain.java:157)
    at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa
lve.java:214)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
    at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
    at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardC
ontextValve.java:198)
    at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa
lve.java:152)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
    at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authenticat
orBase.java:540)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:102)
    at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
    at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja
va:137)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
    at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja
va:118)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:102)
    at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
    at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv
e.java:109)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
    at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
    at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
    at
org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160
)
    at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
799)
    at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proces
sConnection(Http11Protocol.java:705)
    at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:
577)
    at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPo
ol.java:683)
    at java.lang.Thread.run(Thread.java:595)
End of Stack Trace

    at
com.ourdomain.ourapp.AccessPortal.getUserData(AccessPortal.java:190)
    at
com.ourdomain.ourapp.AccessPortal.processRequest(AccessPortal.java:71)
    at com.ourdomain.ourapp.AccessPortal.doGet(AccessPortal.java:88)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
cationFilterChain.java:237)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi
lterChain.java:157)
    at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa
lve.java:214)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
    at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
    at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardC
ontextValve.java:198)
    at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa
lve.java:152)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
    at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authenticat
orBase.java:540)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:102)
    at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
    at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja
va:137)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
    at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja
va:118)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:102)
    at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
    at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv
e.java:109)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
    at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
    at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
    at
org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160
)
    at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
799)
    at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proces
sConnection(Http11Protocol.java:705)
    at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:
577)
    at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPo
ol.java:683)
    at java.lang.Thread.run(Thread.java:595)

2006-04-13 16:06:51 StandardWrapperValve[AccessPortal]:
Servlet.service() for servlet AccessPortal threw exception
javax.servlet.ServletException: org.postgresql.util.PSQLException: An
I/O error has occured while flushing the output - Exception:
java.net.SocketException: Broken pipe
Stack Trace:

java.net.SocketException: Broken pipe
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
    at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
    at org.postgresql.core.PGStream.flush(PGStream.java:411)
    at
org.postgresql.core.QueryExecutor.sendQueryV3(QueryExecutor.java:337)
    at
org.postgresql.core.QueryExecutor.executeV3(QueryExecutor.java:121)
    at org.postgresql.core.QueryExecutor.execute(QueryExecutor.java:100)
    at org.postgresql.core.QueryExecutor.execute(QueryExecutor.java:43)
    at
org.postgresql.jdbc1.AbstractJdbc1Statement.execute(AbstractJdbc1State
ment.java:517)
    at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2State
ment.java:50)
    at
org.postgresql.jdbc1.AbstractJdbc1Statement.executeQuery(AbstractJdbc1
Statement.java:233)
    at
org.postgresql.jdbc1.AbstractJdbc1Statement.executeQuery(AbstractJdbc1
Statement.java:221)
    at
org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingSta
tement.java:205)
    at
com.ourdomain.ourapp.AccessPortal.getUserData(AccessPortal.java:126)
    at
com.ourdomain.ourapp.AccessPortal.processRequest(AccessPortal.java:71)
    at com.ourdomain.ourapp.AccessPortal.doGet(AccessPortal.java:88)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
cationFilterChain.java:237)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi
lterChain.java:157)
    at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa
lve.java:214)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
    at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
    at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardC
ontextValve.java:198)
    at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa
lve.java:152)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
    at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authenticat
orBase.java:540)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:102)
    at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
    at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja
va:137)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
    at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja
va:118)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:102)
    at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
    at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv
e.java:109)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
    at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
    at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
    at
org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160
)
    at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
799)
    at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proces
sConnection(Http11Protocol.java:705)
    at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:
577)
    at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPo
ol.java:683)
    at java.lang.Thread.run(Thread.java:595)
End of Stack Trace

    at
com.ourdomain.ourapp.AccessPortal.getUserData(AccessPortal.java:190)
    at
com.ourdomain.ourapp.AccessPortal.processRequest(AccessPortal.java:71)
    at com.ourdomain.ourapp.AccessPortal.doGet(AccessPortal.java:88)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
    at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
cationFilterChain.java:237)
    at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi
lterChain.java:157)
    at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa
lve.java:214)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
    at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
    at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardC
ontextValve.java:198)
    at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa
lve.java:152)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
    at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authenticat
orBase.java:540)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:102)
    at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
    at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja
va:137)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
    at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja
va:118)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:102)
    at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
    at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv
e.java:109)
    at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
    at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
    at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
    at
org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160
)
    at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
799)
    at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proces
sConnection(Http11Protocol.java:705)
    at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:
577)
    at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPo
ol.java:683)
    at java.lang.Thread.run(Thread.java:595)


I haven't been able to find any matches for this problem on the
internet, so I'm posting here hoping someone is aware of the problem.

There is a release pg74.216.jdbc3.jar, I can't find a changelog for
it and builds 213-216, anyone know? It may be that these include a
bug fix. I'm checking pg74.216.jdbc3.jar on a development server, so
will probably try this next.

Thanks,
Ed

Re: connection reset error, followed by broken pipe (pg74.213.jdbc3.jar)

From
Markus Schaber
Date:
Hi, Ed,

Ed Blackmore wrote:

> I'm having trouble with a tomcat-postgres installation. Server 1 runs
> postgresql and tomcat, server 2 has an identical tomcat installation
> and application code running (except rebranding). Server 2 has errors
> about every week with 'connection reset' in logs/appname.date.log,
> futher connection attempts have a 'broken pipe' error.

So it seems that the network connection breaks from time to time.
Independently from making your application more robust, you might want
to put some research into that.

> The solution to get the app running again is to restart tomcat.

There was a thread here some days ago with the title "No automatic
reconnect after network error", you might have a look there.


If you're using the tomcat connection pooling, my proposed solution
there might help you:

| At least JBoss, you can use the following approach: In the Datasource
| definition, put the following:
|
| <datasources>
|   <no-tx-datasource>
|     <jndi-name>boo</jndi-name>
|    <connection-url>jdbc:postgresql://localhost:5432/bar</connection-url>
|     [ ... other attributes ... ]
|     <new-connection-sql>SELECT version()</new-connection-sql>
|     <check-valid-connection-sql>SELECT
| version()</check-valid-connection-sql>
|   </no-tx-datasource>
| </datasources>
|
| This validates the connection before it is handed out on the
| application, and throws away invalid connections. Be aware that the SQL
| is case sensitive, we had a problem with earlier versions of OpenMDX/CRX
| that had upper case letters in the tags so they were ignored by jboss.

HTH,
Markus
--
Markus Schaber | Logical Tracking&Tracing International AG
Dipl. Inf.     | Software Development GIS

Fight against software patents in EU! www.ffii.org www.nosoftwarepatents.org

Re: connection reset error, followed by broken pipe (pg74.213.jdbc3.jar)

From
"Ed Blackmore"
Date:
Hi Markus,

Thankyou for your assistance.

> So it seems that the network connection breaks from time to time
> Independently from making your application more robust, you might want
> to put some research into that.

Yes, I put a minutely ping script in place, which wasn't able to
detect a failure at a time when the connection was lost. The servers
are dedicated server (installed and hosted by another company), so
I've been trying to detect any outages to report to them.

> If you're using the tomcat connection pooling, my proposed solution
> there might help you

Thanks, I tried the following settings based on your idea looking for
similar tomcat settings on google:

In webapps/ourapp/META-INF/context.ml, adding:
    <parameter>
      <name>testOnBorrow</name>
      <value>true</value>
    </parameter>
    <parameter>
      <name>testOnReturn</name>
      <value>true</value>
    </parameter>
    <parameter>
      <name>validationQuery</name>
      <value>select version()</value>
    </parameter>

I wasn't able to nicely induce the error through our firewall or
taking connections down, but restarting postgresql did lead to a
broken pipe error. The above fix didn't help in this scenerio.

Reading other posts it seems that the jdbc driver is not expected to
reopen connections, therefore it doesn't seem using a newer version
will help. Given that our tomcat app doesn't reconnect to a
postgresql db after it (the database) has been restarted, is it
correct I need to work with the app developer as to how we use the
connection pool, or are there configuration changes that can help?

Regards,
Ed

Re: connection reset error, followed by broken pipe (pg74.213.jdbc3.jar)

From
Oliver Jowett
Date:
Ed Blackmore wrote:
> Hi Markus,
>
> Thankyou for your assistance.
>
>
>>So it seems that the network connection breaks from time to time
>>Independently from making your application more robust, you might want
>>to put some research into that.
>
>
> Yes, I put a minutely ping script in place, which wasn't able to
> detect a failure at a time when the connection was lost. The servers
> are dedicated server (installed and hosted by another company), so
> I've been trying to detect any outages to report to them.

One common cause of connection loss is if there is a stateful firewall
or NAT between your client and server .. if the firewall loses the
connection state for some reason (e.g. if the connection goes idle for a
while and the firewall times it out, or if the firewall is restarted),
then the connection will break.

-O

Re: connection reset error, followed by broken pipe (pg74.213.jdbc3.jar)

From
Markus Schaber
Date:
Hi, Ed,

Ed Blackmore wrote:

> I wasn't able to nicely induce the error through our firewall or
> taking connections down, but restarting postgresql did lead to a
> broken pipe error. The above fix didn't help in this scenerio.
>
> Reading other posts it seems that the jdbc driver is not expected to
> reopen connections, therefore it doesn't seem using a newer version
> will help. Given that our tomcat app doesn't reconnect to a
> postgresql db after it (the database) has been restarted, is it
> correct I need to work with the app developer as to how we use the
> connection pool, or are there configuration changes that can help?

Yes, that's true.

My workaround protects against a pool handing out broken connections[1],
but the Application still must get a fresh connection ittself.

Most JSP pages and Servlets I've seen get such a connection on every
incoming request, most Servlets I've written refresh their connection
when an appropriate SQL exception happens.

I think that this is the way your app has to be modified - refresh the
SQL connection in case of an Exception.



HTH,
Markus

[1] Before java 1.6, the pool has no (easy and portable) way to tell
whether a connection is still valid.

--
Markus Schaber | Logical Tracking&Tracing International AG
Dipl. Inf.     | Software Development GIS

Fight against software patents in EU! www.ffii.org www.nosoftwarepatents.org