Re: Postgres won't start after disk space issue - Mailing list pgsql-admin

From Albe Laurenz
Subject Re: Postgres won't start after disk space issue
Date
Msg-id A737B7A37273E048B164557ADEF4A58B3659F9D0@ntex2010i.host.magwien.gv.at
Whole thread Raw
In response to Postgres won't start after disk space issue  (Cheyne Wallace <cheyne.wallace@gmail.com>)
List pgsql-admin
Cheyne Wallace wrote:
> I have a problem i’m stuck on. Our Postgres server ran out of disk space last night and it fell over.
> We’ve resolved the disk space issue and there is plenty now although Postgres will not start.
> 
> When I view the main log it reads:
> 
> 2015-02-23 17:49:35 UTC LOG:  redo starts at 75/202674D8
> 2015-02-23 17:49:35 UTC LOG:  incomplete startup packet
> 2015-02-23 17:49:36 UTC FATAL:  the database system is starting up
> 2015-02-23 17:49:36 UTC FATAL:  the database system is starting up
> 2015-02-23 17:49:41 UTC FATAL:  the database system is starting up
> 2015-02-23 17:49:41 UTC LOG:  incomplete startup packet
> 
> When I look at ps , I can see that postgres is trying to perform a recovery, although it never
> finishes.
> 
> - postgres: startup process   recovering 00000001000000750000002D
> 
> I ran an strace on this process to try and see what it’s doing and I see these lines repeating over
> and over again:
> 
> read(7, 0x7fff6c82fadf, 1)              = -1 EAGAIN (Resource temporarily unavailable)
> open("pg_clog/01ED", O_RDWR|O_CREAT, 0600) = 27
> lseek(27, 212992, SEEK_SET)             = 212992
> write(27, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192) = 8192
> fsync(27)                               = 0
> close(27)
> 
> When I look under pg_clog I can see that the 01ED file it’s opening here is the most recent one and
> has a different file size to the others in the dir.
> I’m assuming this is the file it was writing too when the disk space ran out and now it’s corrupt or
> something along those lines.
> 
> Can anyone please help me figure out how to fix this?  I was told that deleting any of these pg_clog
> files would be bad, how ever I had no idea how to repair it.
> 
> p.s I also asked this question here: http://dba.stackexchange.com/questions/93594/postgres-wont-start-
> incomplete-startup-packet

Can you get a stack trace from the startup process?

Yours,
Laurenz Albe

pgsql-admin by date:

Previous
From: Cheyne Wallace
Date:
Subject: Postgres won't start after disk space issue
Next
From: jesper@krogh.cc
Date:
Subject: Large memory machine and PG 9.2.9