Thread: Re: Getting FATAL: terminating connection due to administrator command

Re: Getting FATAL: terminating connection due to administrator command

From
Peter Hopfgartner
Date:
--------Tom Lane <tgl@sss.pgh.pa.us> wrote--------
Subject: Re: [GENERAL] Getting FATAL: terminating connection due to administrator command 
Date: 15.09.2010 16:07

>Peter Hopfgartner <peter.hopfgartner@r3-gis.com> writes:
>> Since some days we are getting the above message.
>> Also in the PostgreSQL logs we get:
>> FATAL:  terminating connection due to administrator command
>
>This is a result of something sending SIGTERM to the backend process.
>
>I have heard reports of "load management" software that SIGTERM's
>processes more or less at random whenever it decides the system is
>overloaded.  If you have any such junkware installed on your server,
>try disabling it.

The server is a rather bare bone server for web mapping, so basically PostgreSQL/PostGIS, Apache, PHP, Tomcat and
littleother stuff. The Dell software was the only which did not come from CentOS/EPEL/argeo/in-house RPM packages. I've
removedthe Dell stuff completely, but the problem is still there.
 
>
>> The server is from Dell, Dell's hardware monitoring, OpenManage, says
>that the hardware, in particular memory and disk, are ok.
>
>Never dealt with OpenManage before, but you should cast a wary eye
>upon any Dell-specific software on the machine.  This behavior is
>definitely not normal for Unix systems, so you need to look for
>nonstandard software (and what's more, nonstandard software running with
>root privileges, else it couldn't SIGTERM postgres processes).
>

Other informations: disks are costly SAS drives in a RAID 1 array, memory is with ECC.
Security level is disabled
SELinux is Permissive.
The server acts as a XEN host 

Is it reasonable to restrict the problem to kernel/hardware and/or PostgreSQL/PostGIS itself?

Can I trace where the SIGTERM comes from?

>            regards, tom lane
>

Regards,

Peter


Re: Getting FATAL: terminating connection due to administrator command

From
Tom Lane
Date:
Peter Hopfgartner <peter.hopfgartner@r3-gis.com> writes:
> --------Tom Lane <tgl@sss.pgh.pa.us> wrote--------
>> This is a result of something sending SIGTERM to the backend process.

> Can I trace where the SIGTERM comes from?

If this is a recent Red-Hat-based release, I think that systemtap could
probably be used to determine that.  There's a script here that solves
a related problem:
http://sourceware.org/systemtap/examples/process/sigmon.stp

            regards, tom lane

Re: Getting FATAL: terminating connection due to administrator command

From
"Joshua J. Kugler"
Date:
On Wednesday 15 September 2010, Peter Hopfgartner elucidated thus:
> The server is a rather bare bone server for web mapping, so basically
> PostgreSQL/PostGIS, Apache, PHP, Tomcat and little other stuff. The
> Dell software was the only which did not come from
> CentOS/EPEL/argeo/in-house RPM packages. I've removed the Dell stuff
> completely, but the problem is still there.

Are you running out of memory and getting killed by the OOM killer?

j

--
Joshua Kugler
Part-Time System Admin/Programmer
http://www.eeinternet.com - Fairbanks, AK
PGP Key: http://pgp.mit.edu/  ID 0x73B13B6A

Re: Getting FATAL: terminating connection due to administrator command

From
Tom Lane
Date:
"Joshua J. Kugler" <joshua@eeinternet.com> writes:
> On Wednesday 15 September 2010, Peter Hopfgartner elucidated thus:
>> The server is a rather bare bone server for web mapping, so basically
>> PostgreSQL/PostGIS, Apache, PHP, Tomcat and little other stuff. The
>> Dell software was the only which did not come from
>> CentOS/EPEL/argeo/in-house RPM packages. I've removed the Dell stuff
>> completely, but the problem is still there.

> Are you running out of memory and getting killed by the OOM killer?

The OOM killer hits its victims with SIGKILL (kill -9), so we can rule
that out as not matching Peter's symptoms.  This is definitely an
unwanted SIGTERM not SIGKILL.

            regards, tom lane