Thread: Abnormal Exit Quandary
I'm getting this message on Postgres 7.4.1 (Mac OS 10.3) 2004-05-24 15:20:15 WARNING: terminating connection because of crash of another server process DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. HINT: In a moment you should be able to reconnect to the database and repeat your command. This message repeats itself for each back-end that shuts down. This is the second time in two days I've got this message in my log. The server has been running fine for about 5 months straight up until now. The load on the server has increased somewhat in the last few days, not in the number of back-end processes, but in the number of SELECTs each back-end performs. Can someone give me any pointers on what troubleshooting steps I can take to get my system stable again?
Joe Lester <joe_lester@sweetwater.com> writes:
> I'm getting this message on Postgres 7.4.1 (Mac OS 10.3)
> 2004-05-24 15:20:15 WARNING: terminating connection because of crash
> of another server process
> Can someone give me any pointers on what troubleshooting steps I can
> take to get my system stable again?
Well, for starters you could tell us about the actual crash, and not the
subsequent shutdown/restart. The messages you've quoted tell nothing at
all. What happened just before that?
regards, tom lane
Is it crashing around the time a pg_dump is executed? There is a similar problem with 7.4.1 on cygwin / windows. There were some patches for this applied to 7.4.2 if that is the case. On Mon, May 24, 2004 at 04:25:30PM -0500, Joe Lester wrote: > I'm getting this message on Postgres 7.4.1 (Mac OS 10.3) > > 2004-05-24 15:20:15 WARNING: terminating connection because of crash > of another server process > DETAIL: The postmaster has commanded this server process to roll back > the current transaction and exit, because another server process exited > abnormally and possibly corrupted shared memory. > HINT: In a moment you should be able to reconnect to the database and > repeat your command. > > This message repeats itself for each back-end that shuts down. > > This is the second time in two days I've got this message in my log. > The server has been running fine for about 5 months straight up until > now. The load on the server has increased somewhat in the last few > days, not in the number of back-end processes, but in the number of > SELECTs each back-end performs. > > Can someone give me any pointers on what troubleshooting steps I can > take to get my system stable again? > > > > ---------------------------(end of broadcast)--------------------------- > TIP 3: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to majordomo@postgresql.org so that your > message can get through to the mailing list cleanly
No. I dump the database every night, but this problem has happened twice in the afternoon around 3:00 PM. There's nothing special running at that time. I'll respond to Tom and hopefully provide some more detail about my case. Thanks for the response. On May 24, 2004, at 9:19 PM, Mike G wrote: > Is it crashing around the time a pg_dump is executed? There is a > similar problem with 7.4.1 on cygwin / windows. > > There were some patches for this applied to 7.4.2 if that is the case. > > On Mon, May 24, 2004 at 04:25:30PM -0500, Joe Lester wrote: >> I'm getting this message on Postgres 7.4.1 (Mac OS 10.3) >> >> 2004-05-24 15:20:15 WARNING: terminating connection because of crash >> of another server process >> DETAIL: The postmaster has commanded this server process to roll back >> the current transaction and exit, because another server process >> exited >> abnormally and possibly corrupted shared memory. >> HINT: In a moment you should be able to reconnect to the database and >> repeat your command. >> >> This message repeats itself for each back-end that shuts down. >> >> This is the second time in two days I've got this message in my log. >> The server has been running fine for about 5 months straight up until >> now. The load on the server has increased somewhat in the last few >> days, not in the number of back-end processes, but in the number of >> SELECTs each back-end performs. >> >> Can someone give me any pointers on what troubleshooting steps I can >> take to get my system stable again? >> >> >> >> ---------------------------(end of >> broadcast)--------------------------- >> TIP 3: if posting/reading through Usenet, please send an appropriate >> subscribe-nomail command to majordomo@postgresql.org so that your >> message can get through to the mailing list cleanly > > ---------------------------(end of > broadcast)--------------------------- > TIP 3: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to majordomo@postgresql.org so that your > message can get through to the mailing list cleanly >
Thank you for pointing that out Tom. This is much more revealing. I think I can take it from here. I'll check max_files_per_process in postgresql.conf and I'll check my kernal's settings. Thanks! 2004-05-24 15:02:16 LOG: out of file descriptors: Too many open files in system; release and retry 2004-05-24 15:02:16 LOG: out of file descriptors: Too many open files in system; release and retry 2004-05-24 15:20:12 LOG: out of file descriptors: Too many open files in system; release and retry 2004-05-24 15:20:13 PANIC: could not open control file "/Library/PostgreSQL/data/global/pg_control": Too many open files in system 2004-05-24 15:20:13 LOG: checkpoint process (PID 1188) was terminated by signal 6 2004-05-24 15:20:13 LOG: terminating any other active server processes 2004-05-24 15:20:13 WARNING: terminating connection because of crash of another server process DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. HINT: In a moment you should be able to reconnect to the database and repeat your command. On May 24, 2004, at 8:31 PM, Tom Lane wrote: > Joe Lester <joe_lester@sweetwater.com> writes: >> I'm getting this message on Postgres 7.4.1 (Mac OS 10.3) >> 2004-05-24 15:20:15 WARNING: terminating connection because of crash >> of another server process > > Well, for starters you could tell us about the actual crash, and not > the > subsequent shutdown/restart. The messages you've quoted tell nothing > at > all. What happened just before that? > > regards, tom lane >