"PANIC: btree_split_redo: lost left sibling" problem - Mailing list pgsql-admin

From MStringham@tomax.com
Subject "PANIC: btree_split_redo: lost left sibling" problem
Date
Msg-id OFF3E761DE.99B8E506-ON87257515.00647BD8-87257515.00656A7C@tomax.com
Whole thread Raw
Responses Re: "PANIC: btree_split_redo: lost left sibling" problem  (Tom Lane <tgl@sss.pgh.pa.us>)
List pgsql-admin

This is my first post so hopefully I can provide all the information to get this solved.
Last night we ran out of disk space on our server and when I tried to start  pg_ctl I got this error:
 ** EnterpriseDB Dynamic Tuning Agent ************************
        * System Utilization: 66 %                                  *
        *   Database Version: 8.1.3.12                              *
        *   Operating System: Linux                                 *
        *                RAM: 7.56  GB                              *
        *      Shared Memory: 7.55  GB                              *
        * Max DB Connections: 100                                   *
        *************************************************************

2008-12-04 10:20:47 MST LOG:  database system was interrupted while in recovery at 2008-12-04 10:11:35 MST
2008-12-04 10:20:47 MST HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.
2008-12-04 10:20:47 MST LOG:  starting archive recovery
2008-12-04 10:20:47 MST LOG:  restore_command = "cp /u01/home/enterprisedb/bkp/archivedir/%f %p"
cp: cannot stat `/u01/home/enterprisedb/bkp/archivedir/00000003.history': No such file or directory
cp: cannot stat `/u01/home/enterprisedb/bkp/archivedir/00000003000000350000002D': No such file or directory
2008-12-04 10:20:47 MST LOG:  checkpoint record is at 35/B74729F0
2008-12-04 10:20:47 MST LOG:  redo record is at 35/B74729F0; undo record is at 0/0; shutdown FALSE
2008-12-04 10:20:47 MST LOG:  next transaction ID: 24775805; next OID: 730532
2008-12-04 10:20:47 MST LOG:  next MultiXactId: 2952; next MultiXactOffset: 6359
2008-12-04 10:20:47 MST LOG:  automatic recovery in progress
2008-12-04 10:20:47 MST LOG:  redo starts at 35/B7472A40
2008-12-04 10:20:47 MST PANIC:  btree_split_redo: lost left sibling
2008-12-04 10:20:47 MST LOG:  startup process (PID 2520) was terminated by signal 6
2008-12-04 10:20:47 MST LOG:  aborting startup due to startup process failure

I went to the archivedir directory and indeed the files mentioned above are not there.   I did go to the pg_xlog folder and found that the files listed above are there - this directory has a lot of data.  I wanted to do a PITR but to do this I need the database running.  Here are the details:
EnterpriseDB version:  8.1.3.12
Server:  Red Hat Enterprise Linux AS release 3 (Taroon Update 7)

I'm at a loss what to do.  I've researched and couldn't find any answers.  I don't have a lot of space to work with still because I was going to remove a bunch of old schemas that were no longer used but I cannot do that until the db is up and running.  Any ideas on what do do here?  

Matt

CONFIDENTIALITY NOTICE: This message and any attachment(s) are solely for the use of the intended recipient(s) identified above and may contain information that is proprietary, privileged, or confidential. If you are not an intended recipient, you may not review, retransmit, or otherwise use this message or any attachment. If you have received this message in error, please immediately notify the sender by reply e-mail and delete this message.

pgsql-admin by date:

Previous
From: Abubaker
Date:
Subject: Re: Remote access
Next
From: Chris Curvey
Date:
Subject: moving users/roles from one server to another?