Thread: hung postmaster when client machine dies?
We recently had an incident where a linux box went down with a kernel error. A process on the box had an open connection to a postgres session, and was in a transaction. This situation was noticed when other processes connected to postgres would not respond. We observed that there was postmaster connected to the ip address of the downed machine with an "in transaction" status. Killing that processes unblocked the other processes. Is this expected behavior? Was postgres simply waiting for a failure from the TCP/IP layer? We're now running a watchdog process that pings machines for which a postmaster is running and killing that process if the machine is not contactable for a certain period of time. Thanks to whoever made the status information show up in ps output! Unfortunately, we didn't capture the process data... if this would help we can attempt to reproduce the situation. Many TIA, Mark -- Mark Harrison Pixar Animation Studios Emeryville, CA
On Thu, 29 Jan 2004 17:00:27 -0800 Mark Harrison <mh@pixar.com> wrote: > We observed that there was postmaster connected to the ip address > of the downed machine with an "in transaction" status. Killing > that processes unblocked the other processes. > > Is this expected behavior? Was postgres simply waiting for > a failure from the TCP/IP layer? > When a machine simply "goes away" (crashed, unplugged) no packets are sent indicating the socket has closed so PG doesn't know the socket is closed and cannot rollback the transaction. (This is true of any tcp-based protocol). If that connection had a transaction open it'll likely have rows locked. Killing PG caused it to rollback that transaction freeing those locks. Eventually PG would have found out the socket was dead. Most likely if it tried to write to it. -- Jeff Trout <jeff@jefftrout.com> http://www.jefftrout.com/ http://www.stuarthamm.net/
Jeff <threshar@torgo.978.org> writes:
> Eventually PG would have found out the socket was dead.
We do enable TCP keepalive on the client socket, so eventually the
kernel will time out and notify us of a lost connection. Unfortunately
the timeouts involved are long --- IIRC the relevant RFCs specify at
least an hour.
regards, tom lane
Hope you don't mind if I disagree. Most OS's that have a tcp/ip layer also have a parameter therein called tcp_keep_alive. They also set this parameter to infinity. The purpose of tcp_keep_alive is to have the OS kernel periodicallyverify that all tcp/ip connections it is managing are still functioning every so often. Basically the OS sendsa probe packet down the line to the to the client machine. If it bounces back the connection is dead & the OS can dowhat it has to, which will then notify postmaster just like you manually did. I'd contact your OS vendor for informationof what tcp_keep_alive is set to by default and how you can change it. Dick Goulet Senior Oracle DBA Oracle Certified 8i DBA -----Original Message----- From: Jeff [mailto:threshar@torgo.978.org] Sent: Friday, January 30, 2004 10:18 AM To: Mark Harrison Cc: pgsql-admin@postgresql.org Subject: Re: [ADMIN] hung postmaster when client machine dies? On Thu, 29 Jan 2004 17:00:27 -0800 Mark Harrison <mh@pixar.com> wrote: > We observed that there was postmaster connected to the ip address > of the downed machine with an "in transaction" status. Killing > that processes unblocked the other processes. > > Is this expected behavior? Was postgres simply waiting for > a failure from the TCP/IP layer? > When a machine simply "goes away" (crashed, unplugged) no packets are sent indicating the socket has closed so PG doesn't know the socket is closed and cannot rollback the transaction. (This is true of any tcp-based protocol). If that connection had a transaction open it'll likely have rows locked. Killing PG caused it to rollback that transaction freeing those locks. Eventually PG would have found out the socket was dead. Most likely if it tried to write to it. -- Jeff Trout <jeff@jefftrout.com> http://www.jefftrout.com/ http://www.stuarthamm.net/ ---------------------------(end of broadcast)--------------------------- TIP 4: Don't 'kill -9' the postmaster
Goulet, Dick wrote: > Hope you don't mind if I disagree. Most OS's that have a tcp/ip > layer also have a parameter therein called tcp_keep_alive. They also > set this parameter to infinity. The purpose of tcp_keep_alive is to > have the OS kernel periodically verify that all tcp/ip connections it > is managing are still functioning every so often. Basically the OS > sends a probe packet down the line to the to the client machine. If > it bounces back the connection is dead & the OS can do what it has > to, which will then notify postmaster just like you manually did. > I'd contact your OS vendor for information of what tcp_keep_alive is > set to by default and how you can change it. On the other hand I suggest that you *do not* change this value, even if you know how, without being very careful understanding what it means. Changing the system-wide TCP KA time can have unforseen effects on other, unrelated functions of the systems concerned. This is simply why most long lived protocols layered over TCP also tend to have their own keep alive PDUs. Peter