Thread: too many Fatal Error 1: shutdown messages

too many Fatal Error 1: shutdown messages

From
"Johnson, Shaunn"
Date:

Running PostgreSQL 7.2.1 on RedHat Linux 7.2

Over the past few days, the database has been acting
strange and have been trying to shutdown.  While my
postmaster is running debug level 2, when I check my
logs to see why, I get :

[snip from log]

Jan  8 09:54:16 test_srv postgres[19069]: [12354] FATAL 1: 
The database system is shutting down

[/snip from log]

There doesn't seem to be any rhyme or reason for it;
yesterday, I thought it was one of the drives having
problems and the kernel gave an error. 

Last night, I thought it was because of too many
users logging in (it had reached it's max and gave that
message).

Both seem to be red herrings.

I'm going to expand on the debug level to see the whys and
hows of some of these things, but, I'm running out of ideas.

Any suggestions?

-X

Re: too many Fatal Error 1: shutdown messages

From
"Johnson, Shaunn"
Date:

Running PostgreSQL 7.2.1 on RedHat Linux 7.2

Can someone tell me why I'm getting this error when I
try to vacuumdb -a -U postgres -v -z  (other than
there is no filed called '003' in that directory)?

[snip from log]

FATAL 2:  open of /var/lib/pgsql/data/pg_clog/0003 failed: No such file or directory
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
connection to server was lost

[/snip from log]

I have a '004' in that directory ...

Thanks!

-X

Re: too many Fatal Error 1: shutdown messages

From
Tom Lane
Date:
"Johnson, Shaunn" <SJohnson6@bcbsm.com> writes:
> Running PostgreSQL 7.2.1 on RedHat Linux 7.2

> FATAL 2:  open of /var/lib/pgsql/data/pg_clog/0003 failed: No such file or
> directory
> server closed the connection unexpectedly

You've run into one of the bugs that we fixed in 7.2.3.

I am not sure if you can recover from this :-(  Do you have a recent
backup?


Memo to onlookers: there is a reason why we put out dot-releases.
We would not have bothered with a 7.2.3 release if we hadn't found
some pretty serious bugs.  If you are running 7.2.x for x < 3, you
are living dangerously.

            regards, tom lane

Re: too many Fatal Error 1: shutdown messages

From
"Bjoern Metzdorf"
Date:
> "Johnson, Shaunn" <SJohnson6@bcbsm.com> writes:
> > Running PostgreSQL 7.2.1 on RedHat Linux 7.2
>
> > FATAL 2:  open of /var/lib/pgsql/data/pg_clog/0003 failed: No such file
or
> > directory
> > server closed the connection unexpectedly
>
> You've run into one of the bugs that we fixed in 7.2.3.
>
> I am not sure if you can recover from this :-(  Do you have a recent
> backup?

We've run into these pg_clog issues quite a lot in the last time with 7.2.1.
Good to know that 7.2.3 might fix this.

In my experience you don't necessarily need to grab the latest backup. If
you really just want to get the database up and running again, you may use a
dummy clog and then dump and restore the database.

E.g. "dd if=/dev/zero of=/path/to/pgdata/pg_clog/0003 bs=1024k count=256"

Tom pointed that out in
http://groups.google.com/groups?q=tom+lane+pg_clog&hl=en&lr=&ie=UTF-8&oe=utf
-8&selm=27106.1031148278%40sss.pgh.pa.us&rnum=5

Regards,
Bjoern