Thread: Weird postmaster crashes

Weird postmaster crashes

From
Dmitry Tkach
Date:
I am experiencing database server crashes quite frequently (sometimes,
*daily*), and I am having hard time identifying what could possibly be
causing them :-(
They seem to be happenning kinda randomly, I was unable to attribute
them to any specific database activity going on at the time...
The postgres log looks like:

2003-06-10 13:53:32 [14522]  DEBUG:  pq_recvbuf: unexpected EOF on
client connection
2003-06-10 13:53:32 [16915]  DEBUG:  pq_recvbuf: unexpected EOF on
client connection
2003-06-10 13:53:32 [14523]  DEBUG:  pq_recvbuf: unexpected EOF on
client connection
2003-06-10 13:53:32 [17095]  DEBUG:  pq_recvbuf: unexpected EOF on
client connection
2003-06-10 13:53:32 [14551]  FATAL 1:  LWLockAcquire: can't wait without
a PROC structure
2003-06-10 13:53:32 [14551]  FATAL 1:  LWLockAcquire: can't wait without
a PROC structure
2003-06-10 13:53:32 [14527]  DEBUG:  pq_recvbuf: unexpected EOF on
client connection
2003-06-10 13:53:32 [14685]  DEBUG:  pq_recvbuf: unexpected EOF on
client connection
2003-06-10 13:53:32 [17093]  DEBUG:  pq_recvbuf: unexpected EOF on
client connection
2003-06-10 13:53:32 [17092]  DEBUG:  pq_recvbuf: unexpected EOF on
client connection
.... <snip a few identical messages (with different pids)

2003-06-10 13:53:33 [14072]  DEBUG:  server process (pid 14551) exited
with exit code 1
2003-06-10 13:53:33 [14072]  DEBUG:  terminating any other active server
processes
2003-06-10 13:53:33 [1609]   NOTICE:  Message from PostgreSQL backend:
        The Postmaster has informed me that some other backend
        died abnormally and possibly corrupted shared memory.
.....


It does not even produce a core file after this - just silently exists,
and restarts itself.

Could somebody please point me to any clue what could possibly be wrong
with it?

This is 7.2.1 - I know, I need to upgrade.
Working on it, but it is going to take a while, and at the time being I
would greatly appreciate any ideas on what I can do about this thing.

Thanks a lot!

Dima







Re: Weird postmaster crashes

From
Dennis Gearon
Date:
the mantra is  to always check hardware first. Do a disk and memory check.

Dmitry Tkach wrote:

> I am experiencing database server crashes quite frequently (sometimes,
> *daily*), and I am having hard time identifying what could possibly be
> causing them :-(
> They seem to be happenning kinda randomly, I was unable to attribute
> them to any specific database activity going on at the time...
> The postgres log looks like:
>
> 2003-06-10 13:53:32 [14522]  DEBUG:  pq_recvbuf: unexpected EOF on
> client connection
> 2003-06-10 13:53:32 [16915]  DEBUG:  pq_recvbuf: unexpected EOF on
> client connection
> 2003-06-10 13:53:32 [14523]  DEBUG:  pq_recvbuf: unexpected EOF on
> client connection
> 2003-06-10 13:53:32 [17095]  DEBUG:  pq_recvbuf: unexpected EOF on
> client connection
> 2003-06-10 13:53:32 [14551]  FATAL 1:  LWLockAcquire: can't wait without
> a PROC structure
> 2003-06-10 13:53:32 [14551]  FATAL 1:  LWLockAcquire: can't wait without
> a PROC structure
> 2003-06-10 13:53:32 [14527]  DEBUG:  pq_recvbuf: unexpected EOF on
> client connection
> 2003-06-10 13:53:32 [14685]  DEBUG:  pq_recvbuf: unexpected EOF on
> client connection
> 2003-06-10 13:53:32 [17093]  DEBUG:  pq_recvbuf: unexpected EOF on
> client connection
> 2003-06-10 13:53:32 [17092]  DEBUG:  pq_recvbuf: unexpected EOF on
> client connection
> .... <snip a few identical messages (with different pids)
>
> 2003-06-10 13:53:33 [14072]  DEBUG:  server process (pid 14551) exited
> with exit code 1
> 2003-06-10 13:53:33 [14072]  DEBUG:  terminating any other active server
> processes
> 2003-06-10 13:53:33 [1609]   NOTICE:  Message from PostgreSQL backend:
>        The Postmaster has informed me that some other backend
>        died abnormally and possibly corrupted shared memory.
> .....
>
>
> It does not even produce a core file after this - just silently exists,
> and restarts itself.
>
> Could somebody please point me to any clue what could possibly be wrong
> with it?
>
> This is 7.2.1 - I know, I need to upgrade.
> Working on it, but it is going to take a while, and at the time being I
> would greatly appreciate any ideas on what I can do about this thing.
>
> Thanks a lot!
>
> Dima
>
>
>
>
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
>


Re: Weird postmaster crashes

From
Tom Lane
Date:
Dmitry Tkach <dmitry@openratings.com> writes:
> I am experiencing database server crashes quite frequently

> 2003-06-10 13:53:32 [14551]  FATAL 1:  LWLockAcquire: can't wait without
> a PROC structure

> This is 7.2.1 - I know, I need to upgrade.

Yes, you do.  This is a known bug that was fixed in .3 or .4.

            regards, tom lane

Re: Weird postmaster crashes

From
Dmitry Tkach
Date:
Tom Lane wrote:

>Dmitry Tkach <dmitry@openratings.com> writes:
>
>
>>I am experiencing database server crashes quite frequently
>>
>>
>
>
>
>>2003-06-10 13:53:32 [14551]  FATAL 1:  LWLockAcquire: can't wait without
>>a PROC structure
>>
>>
>
>
>
>>This is 7.2.1 - I know, I need to upgrade.
>>
>>
>
>Yes, you do.  This is a known bug that was fixed in .3 or .4.
>
>            regards, tom lane
>
>
Thanks, Tom!

That's kinda what I suspected....
Could you give me some idea on what circumstances cause this to happen?

Thanks again!

Dima


Re: Weird postmaster crashes

From
Tom Lane
Date:
Dmitry Tkach <dmitry@openratings.com> writes:
> 2003-06-10 13:53:32 [14551]  FATAL 1:  LWLockAcquire: can't wait without
> a PROC structure

> Could you give me some idea on what circumstances cause this to happen?

IIRC, it's an order-of-operations mistake during backend shutdown: the
proc structure is deallocated while it's still possible to receive an
interrupt from another backend --- and if you get such an interrupt, you
need the proc.  So from the user's point of view it's pretty
unpredictable.

Short answer: upgrade.  This is not the only nasty bug in 7.2.1.

            regards, tom lane

Re: Weird postmaster crashes

From
Dmitry Tkach
Date:
Makes sense. Thanks!

One more thing to clarify - when you said it was fixed in .3 and .4 did
you mean 7.3 or 7.2.3?

Thanks!

Dima

Tom Lane wrote:

>Dmitry Tkach <dmitry@openratings.com> writes:
>
>
>>2003-06-10 13:53:32 [14551]  FATAL 1:  LWLockAcquire: can't wait without
>>a PROC structure
>>
>>
>
>
>
>>Could you give me some idea on what circumstances cause this to happen?
>>
>>
>
>IIRC, it's an order-of-operations mistake during backend shutdown: the
>proc structure is deallocated while it's still possible to receive an
>interrupt from another backend --- and if you get such an interrupt, you
>need the proc.  So from the user's point of view it's pretty
>unpredictable.
>
>Short answer: upgrade.  This is not the only nasty bug in 7.2.1.
>
>            regards, tom lane
>
>



Re: Weird postmaster crashes

From
Tom Lane
Date:
Dmitry Tkach <dmitry@openratings.com> writes:
> One more thing to clarify - when you said it was fixed in .3 and .4 did
> you mean 7.3 or 7.2.3?

I meant I couldn't remember whether it was first fixed in 7.2.3 or 7.2.4.
Doesn't matter for your purposes --- as long as you're updating, you
should go to 7.2.4.

7.3.* has the fix also of course, but updating to 7.3 is a much bigger
task.

            regards, tom lane