Thread: Postgres won't start after disk space issue
Hey guys,
I have a problem i’m stuck on. Our Postgres server ran out of disk space last night and it fell over. We’ve resolved
thedisk space issue and there is plenty now although Postgres will not start.
When I view the main log it reads:
2015-02-23 17:49:35 UTC LOG: redo starts at 75/202674D8
2015-02-23 17:49:35 UTC LOG: incomplete startup packet
2015-02-23 17:49:36 UTC FATAL: the database system is starting up
2015-02-23 17:49:36 UTC FATAL: the database system is starting up
2015-02-23 17:49:41 UTC FATAL: the database system is starting up
2015-02-23 17:49:41 UTC LOG: incomplete startup packet
When I look at ps , I can see that postgres is trying to perform a recovery, although it never finishes.
- postgres: startup process recovering 00000001000000750000002D
I ran an strace on this process to try and see what it’s doing and I see these lines repeating over and over again:
read(7, 0x7fff6c82fadf, 1) = -1 EAGAIN (Resource temporarily unavailable)
open("pg_clog/01ED", O_RDWR|O_CREAT, 0600) = 27
lseek(27, 212992, SEEK_SET) = 212992
write(27, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192) = 8192
fsync(27) = 0
close(27)
When I look under pg_clog I can see that the 01ED file it’s opening here is the most recent one and has a different
filesize to the others in the dir.
I’m assuming this is the file it was writing too when the disk space ran out and now it’s corrupt or something along
thoselines.
Can anyone please help me figure out how to fix this? I was told that deleting any of these pg_clog files would be
bad,how ever I had no idea how to repair it.
p.s I also asked this question
here: http://dba.stackexchange.com/questions/93594/postgres-wont-start-incomplete-startup-packet
Cheers
Cheyne
Cheyne Wallace wrote:
> I have a problem i’m stuck on. Our Postgres server ran out of disk space last night and it fell over.
> We’ve resolved the disk space issue and there is plenty now although Postgres will not start.
>
> When I view the main log it reads:
>
> 2015-02-23 17:49:35 UTC LOG: redo starts at 75/202674D8
> 2015-02-23 17:49:35 UTC LOG: incomplete startup packet
> 2015-02-23 17:49:36 UTC FATAL: the database system is starting up
> 2015-02-23 17:49:36 UTC FATAL: the database system is starting up
> 2015-02-23 17:49:41 UTC FATAL: the database system is starting up
> 2015-02-23 17:49:41 UTC LOG: incomplete startup packet
>
> When I look at ps , I can see that postgres is trying to perform a recovery, although it never
> finishes.
>
> - postgres: startup process recovering 00000001000000750000002D
>
> I ran an strace on this process to try and see what it’s doing and I see these lines repeating over
> and over again:
>
> read(7, 0x7fff6c82fadf, 1) = -1 EAGAIN (Resource temporarily unavailable)
> open("pg_clog/01ED", O_RDWR|O_CREAT, 0600) = 27
> lseek(27, 212992, SEEK_SET) = 212992
> write(27, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192) = 8192
> fsync(27) = 0
> close(27)
>
> When I look under pg_clog I can see that the 01ED file it’s opening here is the most recent one and
> has a different file size to the others in the dir.
> I’m assuming this is the file it was writing too when the disk space ran out and now it’s corrupt or
> something along those lines.
>
> Can anyone please help me figure out how to fix this? I was told that deleting any of these pg_clog
> files would be bad, how ever I had no idea how to repair it.
>
> p.s I also asked this question here: http://dba.stackexchange.com/questions/93594/postgres-wont-start-
> incomplete-startup-packet
Can you get a stack trace from the startup process?
Yours,
Laurenz Albe