PostgreSQL 9.1 "database system was interrupted; last known up at" - Mailing list pgsql-admin

From Boylan, Ross
Subject PostgreSQL 9.1 "database system was interrupted; last known up at"
Date
Msg-id F1F13E14A610474196571953929C02096D0E40@ex08.net.ucsf.edu
Whole thread Raw
Responses Re: PostgreSQL 9.1 "database system was interrupted; last known up at"
Re: PostgreSQL 9.1 "database system was interrupted; last known up at"
List pgsql-admin
I had to power cycle my system because it became unresponsive.  Now PosgtreSQL will not start.  I would like advice
abouthow to proceed; I think pg_resetxlog is my next step.  I have made a copy of the current database files. 

<log file="postgresql-9.1-main.log">
2015-05-25 10:44:21 PDT LOG:  database system was interrupted; last known up at 2015-05-22 09:22:25 PDT
2015-05-25 10:44:21 PDT LOG:  incomplete startup packet
2015-05-25 10:44:21 PDT FATAL:  could not open file "/etc/ssl/certs/ssl-cert-snakeoil.pem": Permission denied
2015-05-25 10:44:21 PDT LOG:  startup process (PID 5180) exited with exit code 1
2015-05-25 10:44:21 PDT LOG:  aborting startup due to startup process failure
</log>

I am running PostgreSQL 9.1 on Debian wheezy aka 7 aka oldstable.
Installed via the Debian package.  I think I accepted the defaults, and have not changed the configuration since.
Linux  3.2.0, stock Debian kernel, amd64.
Connect via emacs sql-postgresql or psql.  An init script controls startup.

When the system became unresponsive I was able to ssh in; the X process had gone crazy and could not be killed.  Most
keyfile systems had been remounted read-only, and many commands (includiing shutdown and telinit) produced errors,
oftenI/O errors, when run.  The last kern.log entries showed a process being killed.  There was quite a lot of inode
deletionand log replaying on restart.  The log message above was from just after the restart. 

I have no backups*, but could recreate the database in the worst case.  I haven't done anything with the database in at
leasta week, I think, and so if I could get back the state as of 5/22 that would be fine. 

Filesystem is ext3 on dm-crypt on LVM.

The permission error on the snakeoil cert is weird, since it is readable by all.  I'm guessing it's a side effect of
theearlier problems. 

Thanks for any guidance.

Ross Boylan

(*) It would be a bit of mess even if I did, since I use bacula with postgres as the database.  I think I was also
dumpingthe database when I had backup going. 

pgsql-admin by date:

Previous
From: Bruce Momjian
Date:
Subject: Re: Minor upgrades of postgre in aws rds?
Next
From: Tom Lane
Date:
Subject: Re: PostgreSQL 9.1 "database system was interrupted; last known up at"