Hi All.
I've started using replication, and I'd like to monitor my logs for any errors or problems. I don't want to do it
manually,and I'm not interested in stats (a la PgBadger).
What I'd like, is the instant PG logs: "FATAL: wal segment already removed" (or some such bad thing), I'd like to get
anemail.
1st: is anyone using a program that does something like this? What do you use? How do you like it?
My thinking has been along these lines:
+ log to syslog doesnt really help, and I recall seeing somewhere "syslog may not capture everything". I still have
monitoringand log rotation problems.
+ log to stderr and write my own collector works, but then I have to duplicate what logging_collector already does
(rotating,truncating, age, size, etc). Too much work.
+ log with logging_collector, then write a thing to figure out what file its writing to and tail it, watch for
rotation,etc. This is just messy.
If there isn't a program already available (which I've searched for, believe me), I'd like to get feedback on extending
logging_collectorwith some lua scriptable event notification.
Lua is small, fast, and mostly easy to embed. It would allow an admin to customize whatever kind of monitoring they
want. When an event matches logging_collector would spawn off a different app to handle the event notification. The
appwould be launched in the background and forgotten about so that logging isn't delayed.
I'm thinking:
function checkLine(item)
if item:find('FATAL') then
launch('/usr/bin/mynotify.pl', item)
end
end
Logging_collector would then do something like (forgive the perl pseudo code):
... regular log file rotation stuff ..
open OUT
while ($line = <stderr>)
{
checkLine($line);
print OUT $line;
}
... etc, etc ...
Lua could also have another handy events defined:
OnLogRotate(newFile)
OnStartup()
OnShutdown()
Lua can also keep state, so maybe you dont want to email on the first FATAL, but on the third.
local cc = 0
function checkLine(item)
if item:find('FATAL') then
cc = cc + 1
if cc > 2 then
launch('/usr/bin/mynotify.pl', item)
cc = 0
end
end
end
Thoughts?
-Andy
