Thread: Leftover processes on shutdown - Debian+JDBC

Hi-

I've run across a problem that has a simple workaround, but I wonder if it
indicates a problem that needs fixing on the postgresql server end.
Alternately, it may be considered a feature by some...

My environment is PostgreSQL 7.2.1 on Debian Linux 2.4. On the client side,
I have Tomcat accessing the database using the 7.2 PostgreSQL JDBC Driver.

The problem is that if I attempt to do a PostgreSQL shutdown while tomcat
still has connections open, the backend processes that have been spawned to
serve tomcat don't go away. This surprises me because they are owned by
postgres & hence it seems as if they should be stopped as well.

The simple workaround is to make sure that I shutdown tomcat & make the
client connections go away first. This is a sensible procedure anyway, so it
isn't a serious problem.

Since this involves the start/stop script, I suppose it may be
Debian-specific. It also involves the JDBC driver, but I'm guessing that
this behavior could occur with other client connections.

I have tried to make the same thing happen by leaving a psql connection
open, but the shutdown is complete in that case, and attempting further
access through the psql client results in an informative error message &
then a connection reset, so in the case of psql, the situation is handled
gracefully.


Here is the error message:

morgai:/data1/postgresql# /etc/init.d/postgresql restart
Restarting PostgreSQL database: postmaster
Stopped /usr/lib/postgresql/bin/postmaster (pid 321 323 324 5465 5466 5470
5471 5472 5473 5474 5475 5476 5477).
Starting PostgreSQL postmaster.
pg_ctl: Another postmaster may be running.  Trying to start postmaster
anyway.
pg_ctl: cannot start postmaster
Examine the log output.

-Nick

--------------------------------------------------------------------------
Nick Fankhauser  nickf@ontko.com  Phone 1.765.935.4283  Fax 1.765.962.9788
Ray Ontko & Co.     Software Consulting Services     http://www.ontko.com/

"Robert M. Meyer" <rmeyer@installs.com> writes:
> I had the same problem.  I rewrote my shutdown script to call 'pg_ctl -m
> immediate'.  So far, I haven't seen any damage because of it

A much safer choice would be "-m fast".

The default ("-m smart") doesn't shut down the server until all clients
voluntarily disconnect.  If you have tomcat holding persistent
connections open then that's not what you want.

"-m fast" forcibly disconnects clients and then does a clean database
shutdown.  This is generally what you want if you have
persistent-connection clients.

"-m immediate" is the equivalent of the big red EMERGENCY STOP button
found in places like power plant control rooms.  If you are in the habit
of hitting that as a routine measure, you need to have your operator's
permit revoked.

            regards, tom lane

"Robert M. Meyer" <rmeyer@installs.com> writes:
> Hmmm... You're right.  How much longer does a '-m fast' take.  The
> reason I ask is because during a server shutdown, init doesn't wait
> around long after the script exits.  The problem with not using the '-m'
> is that pg_ctl returns quickly but the DB may not be down, causing the
> system to pull the rug out from under it.

(a) The solution to that is to use the -w option to pg_ctl, no?

(b) -m immediate is just as dangerous as letting the system pull the
rug out.  Probably more so, as the system is unlikely to get around to
killing Postgres before we've had a chance to flush shared buffers.

            regards, tom lane

"Chad R. Larson" <clarson@eldocomp.com> writes:
> We're running PostgreSQL as a backend to a J2SE server (with Macromedia
> JRun instead of Tomcat).  When we converted our Sun Enterprise servers into
> a cluster (Veritas VCS) we had similar problems.  We were using the "fast"
> option on a shutdown, but switchover in the cluster would hang until we
> changed the "fast" to "immediate".  We'd prefer "fast", but it does not
> seem to work exactly as documented.
> Solaris 2.6, PostgreSQL 7.1.2, VCS 1.1.

Hmm.  Could you look into it and figure out *why* it didn't work?

It's possible that this Solaris configuration has the same kind of
kernel bug that was recently reported against Mac OSX 10.1: normal
postmaster shutdown fails during system shutdown because fork()
stops working, thus the postmaster cannot launch a checkpoint subprocess
to clean up.  (I'm not certain that fork itself is the issue in OSX,
but the behavior was definitely that the checkpoint subprocess never
got dispatched, even though the postmaster got a success return from
fork...)

            regards, tom lane

"Chad R. Larson" <clarson@eldocomp.com> writes:
> At 08:25 AM 8/13/02 , Robert M. Meyer wrote:
>> I had the same problem.  I rewrote my shutdown script to call 'pg_ctl -m
>> immediate'.  So far, I haven't seen any damage because of it and we always
>> restart tomcat at some point in the middle of the night, anyway.

> We're running PostgreSQL as a backend to a J2SE server (with Macromedia
> JRun instead of Tomcat).  When we converted our Sun Enterprise servers into
> a cluster (Veritas VCS) we had similar problems.  We were using the "fast"
> option on a shutdown, but switchover in the cluster would hang until we
> changed the "fast" to "immediate".  We'd prefer "fast", but it does not
> seem to work exactly as documented.
> Solaris 2.6, PostgreSQL 7.1.2, VCS 1.1.

FYI, I think the cause of this has finally been identified -- a client
that sits idle for long enough to let the SI buffer overrun means that
its connected backend will ignore "fast" shutdown requests for as long
as the client continues to do nothing.  The bug exists in 7.1 through
current.  I just posted a patch to pgsql-admin.

            regards, tom lane