Postgres Pro
Downloads
Home   >   mailing lists

Thread: Best way to recover from crash?

Best way to recover from crash?

From
Steve Wampler
Date:
Immediately after finally deploying our PG database up
on our mountain-top observatory, we got hit by lightening.
(The machines were *supposed* to be installed on a RUPS,
but weren't.  Sigh)

A file was lost.  Now many simple commands cause the
DB to 'crash':
----------------------------------------------------
lab.devel.configdb=# vacuum;
NOTICE:  Rel attributes_table: Uninitialized page 60523 - fixing
NOTICE:  Rel attributes_table: Uninitialized page 60524 - fixing
NOTICE:  Rel attributes_table: Uninitialized page 60525 - fixing
NOTICE:  Rel attributes_table: Uninitialized page 60526 - fixing
FATAL 2:  open of /var/lib/pgsql/data/pg_clog/0000 failed: No such file
or directory
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
The connection to the server was lost. Attempting reset: NOTICE:
Message from PostgreSQL backend:
        The Postmaster has informed me that some other backend
        died abnormally and possibly corrupted shared memory.
        I have rolled back the current transaction and am
        going to terminate your database system connection and exit.
        Please reconnect to the database system and repeat your query.
Failed.
!# \q
----------------------------------------------

I do nightly backups of the DBs so aside from observing time
lost this isn't catestrophic, but since it takes so long to
restore from backup (some of the DBs are fairly large) I was
wondering if there's a 'known procedure' for quickly recovering
from the above.

Thanks!
Steve
--
Steve Wampler -- swampler@noao.edu
The gods that smiled on your birth are now laughing out loud.

Re: Best way to recover from crash?

From
Tom Lane
Date:
Steve Wampler <swampler@noao.edu> writes:
> Immediately after finally deploying our PG database up
> on our mountain-top observatory, we got hit by lightening.
> (The machines were *supposed* to be installed on a RUPS,
> but weren't.  Sigh)

> A file was lost.  Now many simple commands cause the
> DB to 'crash':

> I do nightly backups of the DBs so aside from observing time
> lost this isn't catestrophic, but since it takes so long to
> restore from backup (some of the DBs are fairly large) I was
> wondering if there's a 'known procedure' for quickly recovering
> from the above.

I think you'd be foolish not to initdb and reload from the backups.
You have no way to know how extensive the data damage is ... you might
work around the problems you see now, only to find something else
later after you've put more data into the corrupted DB.

            regards, tom lane

Re: Best way to recover from crash?

From
Steve Wampler
Date:
On Thu, 2004-07-22 at 10:59, Tom Lane wrote:

> I think you'd be foolish not to initdb and reload from the backups.
> You have no way to know how extensive the data damage is ... you might
> work around the problems you see now, only to find something else
> later after you've put more data into the corrupted DB.

Thanks, Tom.  I did that and what you say makes sense.  I was
just overly worried about the down time since the backup database
dump was well over 4GB and looking for a quick fix.  On the
other hand, a 1-hour downtime (what it took) isn't so bad,
and knowing things are solid is worth more than that!

Thanks again,
Steve
--
Steve Wampler -- swampler@noao.edu
The gods that smiled on your birth are now laughing out loud.