Postgres won't start after disk space issue - Mailing list pgsql-admin

From Cheyne Wallace
Subject Postgres won't start after disk space issue
Date
Msg-id etPan.54ebb1ea.6b68079a.341@cheyne.local
Whole thread Raw
Responses Re: Postgres won't start after disk space issue  (Albe Laurenz <laurenz.albe@wien.gv.at>)
List pgsql-admin
 
Hey guys,

I have a problem i’m stuck on. Our Postgres server ran out of disk space last night and it fell over. We’ve resolved
thedisk space issue and there is plenty now although Postgres will not start.   

When I view the main log it reads:

2015-02-23 17:49:35 UTC LOG:  redo starts at 75/202674D8
2015-02-23 17:49:35 UTC LOG:  incomplete startup packet
2015-02-23 17:49:36 UTC FATAL:  the database system is starting up
2015-02-23 17:49:36 UTC FATAL:  the database system is starting up
2015-02-23 17:49:41 UTC FATAL:  the database system is starting up
2015-02-23 17:49:41 UTC LOG:  incomplete startup packet

When I look at ps , I can see that postgres is trying to perform a recovery, although it never finishes. 

- postgres: startup process   recovering 00000001000000750000002D   

I ran an strace on this process to try and see what it’s doing and I see these lines repeating over and over again:

read(7, 0x7fff6c82fadf, 1)              = -1 EAGAIN (Resource temporarily unavailable)
open("pg_clog/01ED", O_RDWR|O_CREAT, 0600) = 27
lseek(27, 212992, SEEK_SET)             = 212992
write(27, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192) = 8192
fsync(27)                               = 0
close(27)

When I look under pg_clog I can see that the 01ED file it’s opening here is the most recent one and has a different
filesize to the others in the dir.  
I’m assuming this is the file it was writing too when the disk space ran out and now it’s corrupt or something along
thoselines. 

Can anyone please help me figure out how to fix this?  I was told that deleting any of these pg_clog files would be
bad,how ever I had no idea how to repair it. 

p.s I also asked this question
here: http://dba.stackexchange.com/questions/93594/postgres-wont-start-incomplete-startup-packet

Cheers
Cheyne





pgsql-admin by date:

Previous
From: Simon Riggs
Date:
Subject: Re: Query cancellation on hot standby because of buffer pins
Next
From: Albe Laurenz
Date:
Subject: Re: Postgres won't start after disk space issue