Postgres Pro
Downloads
Home   >   mailing lists

Thread: PANIC: could not open critical system index 2662

PANIC: could not open critical system index 2662

From
C GG
Date:
This comes at the end of a long cascade of problems stemming from a lightning strike to our datacenter causing our battery backup systems to fail, and overvoltage from the commercial power company to damage our storage arrays which run some services and contain our on-site backups. 

No problem, right-- just get the backups from the DR site. Even though we have exercised our DR backups previously with no problems, this go we have a data corruption issue. Murphy's law, right?

When trying to restart the PostgreSQL 9.2 database for our JIRA installation, we're getting 
"PANIC:  could not open critical system index 2662" ... I've tried various things like turning on zero_damaged_pages but postmaster still crashes on connect when attempting to pg_dump. 

PANIC:  could not open critical system index 2662

LOG:  server process (PID 44753) was terminated by signal 6: Aborted

LOG:  terminating any other active server processes

WARNING:  terminating connection because of crash of another server process

DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

HINT:  In a moment you should be able to reconnect to the database and repeat your command.

LOG:  all server processes terminated; reinitializing

LOG:  database system was interrupted; last known up at 2018-08-04 10:04:19 EDT

LOG:  database system was not properly shut down; automatic recovery in progress

LOG:  record with zero length at C/8C7E28D0

LOG:  redo is not required

LOG:  database system is ready to accept connections

LOG:  autovacuum launcher started


Attempt to connect again, and you get the same set of messages. What other ways can you think of exist for me to tackle getting a dump of the data?

I have a disk backup of the data dir prior to performing zero_damaged_pages so we can do our worst, scrap the failed attempts and try as many other things as we can think of.

If there are any low-level or recovery tools that would potentially work on a 9.2 install, I'd love to hear about those as well. I looked at pg_hexdump but that is 9.4+ ...

I could really use a miracle right now. Thanks all.

CG

Re: PANIC: could not open critical system index 2662

From
Tom Lane
Date:
C GG <cgg0007@gmail.com> writes:
> When trying to restart the PostgreSQL 9.2 database for our JIRA
> installation, we're getting
> "PANIC:  could not open critical system index 2662" ... I've tried various
> things like turning on zero_damaged_pages but postmaster still crashes on
> connect when attempting to pg_dump.

2662 is pg_class_oid_index :-(

Try starting a single-user backend with the ignore-system-indexes flag,
and doing "REINDEX INDEX pg_class_oid_index".  If that works, I'd try
reindexing the rest of the system just to see how bad things are.

If that doesn't work, it's time to hire professional help --- there are
several companies that specialize in PG data recovery.

            regards, tom lane

Re: PANIC: could not open critical system index 2662

From
C GG
Date:


On Sat, Aug 4, 2018 at 11:13 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
C GG <cgg0007@gmail.com> writes:
> When trying to restart the PostgreSQL 9.2 database for our JIRA
> installation, we're getting
> "PANIC:  could not open critical system index 2662" ... I've tried various
> things like turning on zero_damaged_pages but postmaster still crashes on
> connect when attempting to pg_dump.

2662 is pg_class_oid_index :-(

Try starting a single-user backend with the ignore-system-indexes flag,
and doing "REINDEX INDEX pg_class_oid_index".  If that works, I'd try
reindexing the rest of the system just to see how bad things are.


I appreciate the reality check, Tom. I turned it every which way, but no matter what, before I can issue a statement it will crash. Looks pretty bleak.

If that doesn't work, it's time to hire professional help --- there are
several companies that specialize in PG data recovery.


@Community: Does anyone know of a data recovery company that will work on a per-incident basis? 


 
                        regards, tom lane

Re: PANIC: could not open critical system index 2662

From
"Ravi Krishna"
Date:
Just curious, why can't you restore the db from the backup ?