Thread: hung postmaster when client machine dies?

hung postmaster when client machine dies?

From
Mark Harrison
Date:
We recently had an incident where a linux box went down with
a kernel error.  A process on the box had an open connection
to a postgres session, and was in a transaction.

This situation was noticed when other processes connected to
postgres would not respond.

We observed that there was postmaster connected to the ip address
of the downed machine with an "in transaction" status.  Killing
that processes unblocked the other processes.

Is this expected behavior?  Was postgres simply waiting for
a failure from the TCP/IP layer?

We're now running a watchdog process that pings machines for
which a postmaster is running and killing that process if
the machine is not contactable for a certain period of time.
Thanks to whoever made the status information show up in ps
output!

Unfortunately, we didn't capture the process data... if this
would help we can attempt to reproduce the situation.

Many TIA,
Mark

--
Mark Harrison
Pixar Animation Studios
Emeryville, CA


Re: hung postmaster when client machine dies?

From
Jeff
Date:
On Thu, 29 Jan 2004 17:00:27 -0800
Mark Harrison <mh@pixar.com> wrote:

> We observed that there was postmaster connected to the ip address
> of the downed machine with an "in transaction" status.  Killing
> that processes unblocked the other processes.
>
> Is this expected behavior?  Was postgres simply waiting for
> a failure from the TCP/IP layer?
>

When a machine simply "goes away" (crashed, unplugged) no packets are
sent indicating the socket has closed so PG doesn't know the socket is
closed and cannot rollback the transaction. (This is true of any
tcp-based protocol).

If that connection had a transaction open it'll likely have rows locked.
Killing PG caused it to rollback that transaction freeing those locks.
Eventually PG would have found out the socket was dead.  Most likely if
it tried to write to it.

--
Jeff Trout <jeff@jefftrout.com>
http://www.jefftrout.com/
http://www.stuarthamm.net/

Re: hung postmaster when client machine dies?

From
Tom Lane
Date:
Jeff <threshar@torgo.978.org> writes:
> Eventually PG would have found out the socket was dead.

We do enable TCP keepalive on the client socket, so eventually the
kernel will time out and notify us of a lost connection.  Unfortunately
the timeouts involved are long --- IIRC the relevant RFCs specify at
least an hour.

            regards, tom lane

Re: hung postmaster when client machine dies?

From
"Goulet, Dick"
Date:
Hope you don't mind if I disagree.  Most OS's that have a tcp/ip layer also have a parameter therein called
tcp_keep_alive. They also set this parameter to infinity.  The purpose of tcp_keep_alive is to have the OS kernel
periodicallyverify that all tcp/ip connections it is managing are still functioning every so often.  Basically the OS
sendsa probe packet down the line to the to the client machine.  If it bounces back the connection is dead & the OS can
dowhat it has to, which will then notify postmaster just like you manually did.  I'd contact your OS vendor for
informationof what tcp_keep_alive is set to by default and how you can change it. 

Dick Goulet
Senior Oracle DBA
Oracle Certified 8i DBA

-----Original Message-----
From: Jeff [mailto:threshar@torgo.978.org]
Sent: Friday, January 30, 2004 10:18 AM
To: Mark Harrison
Cc: pgsql-admin@postgresql.org
Subject: Re: [ADMIN] hung postmaster when client machine dies?


On Thu, 29 Jan 2004 17:00:27 -0800
Mark Harrison <mh@pixar.com> wrote:

> We observed that there was postmaster connected to the ip address
> of the downed machine with an "in transaction" status.  Killing
> that processes unblocked the other processes.
>
> Is this expected behavior?  Was postgres simply waiting for
> a failure from the TCP/IP layer?
>

When a machine simply "goes away" (crashed, unplugged) no packets are
sent indicating the socket has closed so PG doesn't know the socket is
closed and cannot rollback the transaction. (This is true of any
tcp-based protocol).

If that connection had a transaction open it'll likely have rows locked.
Killing PG caused it to rollback that transaction freeing those locks.
Eventually PG would have found out the socket was dead.  Most likely if
it tried to write to it.

--
Jeff Trout <jeff@jefftrout.com>
http://www.jefftrout.com/
http://www.stuarthamm.net/

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

Re: hung postmaster when client machine dies?

From
"Peter Galbavy"
Date:
Goulet, Dick wrote:
> Hope you don't mind if I disagree.  Most OS's that have a tcp/ip
> layer also have a parameter therein called tcp_keep_alive.  They also
> set this parameter to infinity.  The purpose of tcp_keep_alive is to
> have the OS kernel periodically verify that all tcp/ip connections it
> is managing are still functioning every so often.  Basically the OS
> sends a probe packet down the line to the to the client machine.  If
> it bounces back the connection is dead & the OS can do what it has
> to, which will then notify postmaster just like you manually did.
> I'd contact your OS vendor for information of what tcp_keep_alive is
> set to by default and how you can change it.

On the other hand I suggest that you *do not* change this value, even if you
know how, without being very careful understanding what it means.

Changing the system-wide TCP KA time can have unforseen effects on other,
unrelated functions of the systems concerned.

This is simply why most long lived protocols layered over TCP also tend to
have their own keep alive PDUs.

Peter