Thread: Postgres fails to start

Postgres fails to start

From
Haiming Zhang
Date:

Hi All,

 

I have experienced that Postgres services was not able to start, I tried from Services and command, and both were failed. Here are the logs I got, please help. (I am using postgres 9.1 and there are plenty disk space left).

 

2015-04-07 09:01:28 EST LOG:  database system was interrupted while in recovery at 2015-04-07 08:53:41 EST

2015-04-07 09:01:28 EST HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.

2015-04-07 09:01:28 EST FATAL:  the database system is starting up

2015-04-07 09:01:28 EST LOG:  database system was not properly shut down; automatic recovery in progress

2015-04-07 09:01:28 EST LOG:  redo starts at CD1/3C1414D0

2015-04-07 09:01:29 EST FATAL:  the database system is starting up

2015-04-07 09:01:30 EST FATAL:  the database system is starting up

2015-04-07 09:01:31 EST FATAL:  the database system is starting up

2015-04-07 09:01:32 EST FATAL:  the database system is starting up

2015-04-07 09:01:33 EST FATAL:  the database system is starting up

2015-04-07 09:01:34 EST FATAL:  the database system is starting up

2015-04-07 09:01:35 EST FATAL:  the database system is starting up

2015-04-07 09:01:36 EST FATAL:  the database system is starting up

2015-04-07 09:01:37 EST FATAL:  the database system is starting up

2015-04-07 09:01:38 EST FATAL:  the database system is starting up

2015-04-07 09:01:39 EST FATAL:  the database system is starting up

2015-04-07 09:01:40 EST FATAL:  the database system is starting up

2015-04-07 09:01:41 EST FATAL:  the database system is starting up

2015-04-07 09:01:43 EST FATAL:  the database system is starting up

2015-04-07 09:01:43 EST LOG:  unrecognized win32 error code: 1392

2015-04-07 09:01:43 EST CONTEXT:  xlog redo split_r: rel 1663/2008723533/2107262657 left 375921, right 379817, next 376394, level 0, firstright 95

2015-04-07 09:01:43 EST FATAL:  could not extend file "base/2008723533/2107262657.2": Invalid argument

2015-04-07 09:01:43 EST HINT:  Check free disk space.

2015-04-07 09:01:43 EST CONTEXT:  xlog redo split_r: rel 1663/2008723533/2107262657 left 375921, right 379817, next 376394, level 0, firstright 95

2015-04-07 09:01:44 EST LOG:  startup process (PID 2652) exited with exit code 1

2015-04-07 09:01:44 EST LOG:  aborting startup due to startup process failure

 

Regards,

 

Haiming Zhang

Software Engineer

 




If you are not an authorised recipient of this e-mail, please contact me at Redflex immediately by return phone call or by email. In this case, you should not read, print, retransmit, store or act in reliance on this e-mail or any attachments, and should destroy all copies of them. This e-mail and any attachments are confidential and may contain privileged information and/or copyright material of Redflex or third parties. You should only retransmit, distribute or commercialise the material if you are authorised to do so. This notice should not be removed.

Re: Postgres fails to start

From
Michael Paquier
Date:
On Tue, Apr 7, 2015 at 9:25 AM, Haiming Zhang
<Haiming.Zhang@redflex.com.au> wrote:
> 2015-04-07 09:01:43 EST LOG:  unrecognized win32 error code: 1392
>
> [...]

In Windows world, 1392 = ERROR_FILE_CORRUPT = the file or directory is
corrupted and unreadable:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms681385%28v=vs.85%29.aspx
So first check your file system, and be sure that you have a backup around.
--
Michael


Re: Postgres fails to start

From
Haiming Zhang
Date:
Hi Michael,

Thank you for replying. The file was there, is that ok to remove the corrupted file to recover postgres?
"base/2008723533/2107262657.2"

Unfortunately, I only have a backup on February. Is there a way I can recover it without losing the recent data?

Thanks and regards,
Haiming

-----Original Message-----
From: Michael Paquier [mailto:michael.paquier@gmail.com]
Sent: Tuesday, 7 April 2015 11:36 AM
To: Haiming Zhang
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] Postgres fails to start

On Tue, Apr 7, 2015 at 9:25 AM, Haiming Zhang <Haiming.Zhang@redflex.com.au> wrote:
> 2015-04-07 09:01:43 EST LOG:  unrecognized win32 error code: 1392
>
> [...]

In Windows world, 1392 = ERROR_FILE_CORRUPT = the file or directory is corrupted and unreadable:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms681385%28v=vs.85%29.aspx
So first check your file system, and be sure that you have a backup around.
--
Michael

________________________________

If you are not an authorised recipient of this e-mail, please contact me at Redflex immediately by return phone call or
byemail. In this case, you should not read, print, retransmit, store or act in reliance on this e-mail or any
attachments,and should destroy all copies of them. This e-mail and any attachments are confidential and may contain
privilegedinformation and/or copyright material of Redflex or third parties. You should only retransmit, distribute or
commercialisethe material if you are authorised to do so. This notice should not be removed. 



Re: Postgres fails to start

From
Michael Paquier
Date:
On Tue, Apr 7, 2015 at 10:46 AM, Haiming Zhang
<Haiming.Zhang@redflex.com.au> wrote:
> Thank you for replying. The file was there, is that ok to remove the corrupted file to recover postgres?
"base/2008723533/2107262657.2"

Be careful here, I would recommend taking a file-level snapshot before
going on and do perhaps-stupid things. As that's a btree right split,
perhaps you could recover your data by ignoring this index...

> Unfortunately, I only have a backup on February. Is there a way I can recover it without losing the recent data?

What is lost is lost. A good backup strategy is essential.
--
Michael


Re: Postgres fails to start

From
Haiming Zhang
Date:
Hi Michael,

Thank you again. It seems hard to recover, I will go the hard way (lost data). Learned a lesson.

Regards,
Haiming

-----Original Message-----
From: Michael Paquier [mailto:michael.paquier@gmail.com]
Sent: Tuesday, 7 April 2015 1:04 PM
To: Haiming Zhang
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] Postgres fails to start

On Tue, Apr 7, 2015 at 10:46 AM, Haiming Zhang <Haiming.Zhang@redflex.com.au> wrote:
> Thank you for replying. The file was there, is that ok to remove the corrupted file to recover postgres?
"base/2008723533/2107262657.2"

Be careful here, I would recommend taking a file-level snapshot before going on and do perhaps-stupid things. As that's
abtree right split, perhaps you could recover your data by ignoring this index... 

> Unfortunately, I only have a backup on February. Is there a way I can recover it without losing the recent data?

What is lost is lost. A good backup strategy is essential.
--
Michael

________________________________

If you are not an authorised recipient of this e-mail, please contact me at Redflex immediately by return phone call or
byemail. In this case, you should not read, print, retransmit, store or act in reliance on this e-mail or any
attachments,and should destroy all copies of them. This e-mail and any attachments are confidential and may contain
privilegedinformation and/or copyright material of Redflex or third parties. You should only retransmit, distribute or
commercialisethe material if you are authorised to do so. This notice should not be removed.