Re: Postgres Server crashed - Mailing list pgsql-admin

Thank You Kevin.

Details:
---------------
Running "select version()" gives me  "PostgreSQL 8.4.4 on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 3.4.6
20060404(Red Hat 3.4.6-10), 64-bit" 

dmesg | head -10 -gives- "Linux version 2.6.18-92.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20071124 (Red
Hat4.1.2-42)) #1 SMP Tue Jun 10 18:51:06 EDT 2008" 

Database mounted on 285 GB NFS HD
---------------

I am copying it to a path as and when I write this email.

Regular backups is something I am afraid I have not been taking. But this incident sure has taught me a lesson. pg_clog
andpg_xlog are untouched. 

Thanks and Regards,
Akash.

-----Original Message-----
From: Kevin Grittner [mailto:Kevin.Grittner@wicourts.gov]
Sent: Monday, October 10, 2011 10:47 PM
To: Akash Kodibail; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] Postgres Server crashed

kash Kodibail <akash.kodibail@onmobile.com> wrote:

> Lately the postgres server crashed due to space constraints,

The very first thing to do is to copy the entire directory structure
for the database cluster to somewhere you can keep it safe until the
problem has been resolved.  If you have not yet done so, I strongly
recommend that you do so before trying anything else for recovery.

> pg_control file got lost and the server wouldn't start.

"got lost"?  It seems to me that it would be a Very Bad Thing if an
out of disk space situation allowed PostgreSQL to cause that.
Should we be looking for a bug, or is there a chance someone deleted
this in an attempt to free space?  If the latter, is there any
chance that other files critical to data integrity were deleted,
like in the pg_xlog or pg_clog directories?

>  Had a old pg_control file placed in $PGDATA/global/ path.
>
> I was able to start the server, however, the data seems to be a
> very old one (2011 Jan 07) (probably the time where pg_control I
> had pasted was last updated).

And normal maintenance may be viewing newer data as old, due to
transaction wrap-around from the old pg_control file, and removing
it as part of normal cleanup.  So you may have destroyed some of
your more recent data by doing that.

> How do I recover the rest of the data?

Well, as I said above, start by trying to preserve whatever you have
now.  After you've done that, please provide more information.  We
don't even know what version of PostgreSQL you're talking about, for
example.

http://wiki.postgresql.org/wiki/Guide_to_reporting_problems

It's somewhat likely that you'll be best off going to your last
known good backup, assuming you've been doing regular backups.

-Kevin

DISCLAIMER: The information in this message is confidential and may be legally privileged. It is intended solely for
theaddressee. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any
disclosure,copying, or distribution of the message, or any action or omission taken by you in reliance on it, is
prohibitedand may be unlawful. Please immediately contact the sender if you have received this message in error.
Further,this e-mail may contain viruses and all reasonable precaution to minimize the risk arising there from is taken
byOnMobile. OnMobile is not liable for any damage sustained by you as a result of any virus in this e-mail. All
applicablevirus checks should be carried out by you before opening this e-mail or any attachment thereto. 
Thank you - OnMobile Global Limited.