I woke up this morning to a filled root partition on one of my servers.
I found that both debug and syslog were filled with over 50MB each of
postgres errors:
[what appears to be a standard cleanup cron job]
. . . .
Apr 21 04:00:03 alpha postgres[3371]: [85] DEBUG: --Relation
pg_toast_16408--
Apr 21 04:00:03 alpha postgres[3371]: [86-1] DEBUG: Pages 0: Changed 0,
Empty 0; Tup 0: Vac 0, Keep 0, UnUsed 0.
Apr 21 04:00:03 alpha postgres[3371]: [86-2] ^ITotal CPU 0.00s/0.00u sec
elapsed 0.00 sec.
Apr 22 04:00:01 alpha postgres[17926]: [1] DEBUG: connection:
host=[local] user=postgres database=template1
Apr 22 04:00:02 alpha postgres[17941]: [1] DEBUG: connection:
host=[local] user=postgres database=template1
Apr 22 04:00:02 alpha postgres[362]: [1] DEBUG: statistics collector
process (pid 364) exited with exit code 1
Apr 22 04:00:02 alpha postgres[362]: [2] DEBUG: statistics collector
process (pid 17945) exited with exit code 1
. . . .
Apr 22 04:16:32 alpha postgres[362]: [445836] DEBUG: statistics collector
process (pid 1629) exited with exit code 1
The 'statistics collector exited' error message goes on to repeat for
almost half a million times (as shown), before the files filled the root
fs and truncated. It took about 16.5 minutes to spew this much, to both
logs, before it wedged the machine and finally fully gave up.
As far as I know the machine did not have zero reported space available
(after minfree) at the time this started, but even if it did, an infinite
loop of syslogs is only going to exacerbate the problem, hence this bug
report.
I've shut down postgres on the machine (since we're not using it at this
instant afaik) and have kept the logs in question, so if someone can tell
me what to look at, I'll do what I can to gather it. Email me directly,
as I'm not on any pgsql lists.
The machine is a stock Debian Woody box, up to date, with no local changes
to the postgres configuration at all. It's running 7.2.1-2woody2.
Erik Walthinsen <omega@temple-baptist.com> - System Administrator
__
/ \ GStreamer - The only way to stream!
| | M E G A ***** http://gstreamer.net/ *****
_\ /_
