A 'bad' thing happened yesterday.
Postgresql 8.1.X FreeBSD 6.0
At some point in the day, ran out of space on the root
filesystem. (db is elsewhere)
Took about 10 minutes to clear enough space to make
processes stop freaking out and to slow my heart-rate
down to below 200 beats per minute.
Everything 'seemed' to be fine after that. Database
was constantly utilized with no errors for several
more hours.
Later that evening (during our peak activity time), a
process was inadvertently run that when combined with
peak activity time caused the load average on the box
to spike into the stratosphere (19). *sigh*
Database activity was taking so long (heavy web app
server use), that db connection handles skyrocketed
(over 120 when normally stays below 30) and started
getting complaints from pool management No
ManagedConnections available within configured
blocking timeout ( 30000 [ms] )
At that point, started stopping all unnecessary
activity and even restarted app server to clear all
the connections. However....the db connections all
stayed in some sort of 'frozen' state.....showing that
they were either idle (in transaction) or processing
SELECTs or whatever they were last doing. (Normally
restarting app server will cause those connections to
reset pretty quickly) Tried killing a few of the idle
ones manually...but it was like pissing on a forest
fire.
Hmmmm. It was time to do something hadn't done in a
LOOONG time, restart postgres.
Uh oh.... "postmaster does not shut down".
I tried fast, I tried immediate. No dice. Postmaster
refused to die. I don't know how many times I've
heard "don't kill -9 the postmaster", but that's what
I did (on the 'head' of the snake referenced in the
pid file)....it died, but all the other processes
remained.
*grumble*. Couldn't get a killall to get rid of the
rest (not sure why) so I was back to pissing on the
forest file typing kill kill kill. My co-hort in the
office claimed that more processes were showing up for
each one I killed....(I assume he was delirious at
that point) but this was taking too long anyway. So I
bounced the box.
I prayed.
Came back up fine...postgres, slony, etc.....
no errors reported when the database started.
I see no errors in my logs through the night when
autovacuum and backups occurred, so the database
'appears' to be intact.....
I do notice that all of my logs (all.log, etc) are
'missing' between the filesystem full issue earlier in
the day and the time I rebooted (I assume the OS was
NOT happy with me letting it run out of space on / )
So.....other than the half-dozen things that I
obviously did wrong, was there anything else I could
have done with postgres when it refused to shut down?
____________________________________________________________________________________
Be a better Globetrotter. Get better travel answers from someone who knows. Yahoo! Answers - Check it out.
http://answers.yahoo.com/dir/?link=list&sid=396545469
