> I know, from IRC, the problem has been solved, there was no space on the
> disk ...
>
> Unfortunately, i haven't logs.
>
Yes. Thanks to everybody on the IRC who helped me out.
The suggestion that was most helpful was to call the posgres binary
directly. /usr/lib/postgresql/8.3/bin/postgres. Calling the
/etc/init.d postgres starter gave me no information at all but calling
the binary told me what the problem was.
In a nutshell somehow the restoring of the log files stopped and the
disk started filling up with the unprocessed log files. Eventually the
disk filled up.
I have since set up monitoring the disk space and the postgres
replication lag using zabbix. The zabbix agent reports the disk space
and I report the replication lag using a ruby script.
In case anybody is using zabbix here is the script.
#! /usr/local/bin/ruby
require 'date'
require 'yaml'
require 'socket'
require 'timeout'
require "base64"
def send_item_to_zabbix (server, port, host, key, value)
response = ""
request = "<req><host>"
request << Base64.encode64(host)
request << "</host><key>"
request << Base64.encode64(key)
request << "</key><data>"
request << Base64.encode64(value)
request << "</data></req>"
status = Timeout::timeout(30){
socket = TCPSocket.new(server, port)
socket.puts request
response = socket.gets
if response == "OK" || response.nil?
response = ""
else
response = "Monitor rejected key='#{key}' value='#{value}'
host='#{host}' with response #{response}"
end
} #timeout
return response
rescue Timeout::Error
return "can't connect to monitoring server at #{server}:#{port}"
rescue
return $!
end #send item to zabbix
output =`/usr/lib/postgresql/8.3/bin/pg_controldata
/var/lib/postgresql/8.3/main`
data = YAML.load output
last_checkpoint = DateTime.parse( data['Time of latest checkpoint'])
hours, mins, secs, ignore_fractions =
Date::day_fraction_to_time(DateTime.now - last_checkpoint)
lag = hours * 60 * 60 + mins * 60 + secs
output = send_item_to_zabbix 'my.zabbix..host.com', '10051',
'postgres_host_name'', 'postgres.replication.lag', lag.to_s
~
