Thread: Server continuously enters to recovery mode.

Hi everybody!

I have a problem (really huge problem), I have one server of production, but
yesterday in the night I saw this error:

    *ERROR: could not access status of transaction 2410303155
    DETAIL: Could not open file "pg_clog/08FA": No such file or directory*

Solution:
  * dd if=/dev/zero/ of=data/pg_clog/08FA bs=256K count=1*


So I ran this solutions, no problem so far, but then (after 1 or 2 hours
approximately), the server crash, I think somebody did something but did not
tell me, no one did nothing! cool!! ¬¬. I started the server and this began
in recovery mode, he started. But after some time (without apparent
pattern),  the server came into the recovery mode again.

After that, the server continuously entering recovery mode, like I said
without apparent pattern, between 3, 5 or 10 minutes run normally but then
enter in the recovery mode again. I restart the server (began in recovery
mode again), started, but after sometime he enter in recovery mode again.

Try to recover the server with the PITR and nothing. The server version is
9.0.x in a Linux SUSE. The database size is the 336 GB.

Please give me any help to recover the server!

Thanks!



--
View this message in context:
http://postgresql.1045698.n5.nabble.com/Server-continuously-enters-to-recovery-mode-tp5802321.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.

On 05/03/2014 04:29 PM, DrakoRod wrote:
> Hi everybody!
>
> I have a problem (really huge problem), I have one server of production, but
> yesterday in the night I saw this error:
>
>      *ERROR: could not access status of transaction 2410303155
>      DETAIL: Could not open file "pg_clog/08FA": No such file or directory*
>
> Solution:
>    * dd if=/dev/zero/ of=data/pg_clog/08FA bs=256K count=1*
>
>
> So I ran this solutions, no problem so far, but then (after 1 or 2 hours
> approximately), the server crash, I think somebody did something but did not
> tell me, no one did nothing! cool!! ¬¬. I started the server and this began
> in recovery mode, he started. But after some time (without apparent
> pattern),  the server came into the recovery mode again.
>
> After that, the server continuously entering recovery mode, like I said
> without apparent pattern, between 3, 5 or 10 minutes run normally but then
> enter in the recovery mode again. I restart the server (began in recovery
> mode again), started, but after sometime he enter in recovery mode again.
>
> Try to recover the server with the PITR and nothing. The server version is
> 9.0.x in a Linux SUSE. The database size is the 336 GB.
>
> Please give me any help to recover the server!

In addition to what David said about the log information, do you have
any program running against the pg_clog/ directory?

Something that might be moving the files out of it?

>
> Thanks!
>
>



--
Adrian Klaver
adrian.klaver@aklaver.com

Hi everybody Thank you for your help!

I upgrade the version of 9.0.15 to 9.0.17. But after that constantly show
this error, in all sessions:

    *WARNING:  terminating connection because of crash of another server
process
    DETAIL:  The postmaster has commanded this server process to roll back
the
    current transaction and exit, because another server process exited
    abnormally and possibly corrupted shared memory.
    HINT:  In a moment you should be able to reconnect to the database and
    repeat your command.  *

The problem was in the database template0, this had a problem with the
index, I saw in the log when server started, this error:


    *PANIC: could not open critical system index 2662
*


So, I tried REINDEX but this crash. But I saw when I tried connect to the
database template0, showed this error:

*    PANIC: could not open critical system index 2662  *

So decided drop database.

   * DROP DATABASE template0;  *


But I saw this error:

    *ERROR:  cannot drop a template database  *


Then run this query and after that DROP DATABASE:


    *UPDATE pg_database SET datistemplate='false' WHERE datname='template0';
*

After that I delete the database template0 and resolved it!!

Thanks for all!! :)




--
View this message in context:
http://postgresql.1045698.n5.nabble.com/Server-continuously-enters-to-recovery-mode-tp5802321p5802394.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.