Re: FATAL: terminating connection due to administrator command - Mailing list pgsql-general

Hi all,

We're seeing the FATAL error message from the subject pop up in our logs at regular intervals, but I haven't been able to pinpoint what is causing it. I'm hoping for some insights here.

We run a PostgreSQL 11.9 server on CentOS 7, within a vmware environment:
 PostgreSQL 11.9 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39), 64-bit

The package was installed from the PGDG repository.

I'm not even sure I should be worried, there doesn't appear to be any impact on the servers' functioning, but it does say 'FATAL'.
What we're seeing are lines like these two instances:

2020-09-30 22:27:56.446 CEST [30659]   STATEMENT:  select count(*) from "dm_b2b"."prlwytzkofskiv1"
2020-09-30 22:27:56.446 CEST [30658]   FATAL:  terminating connection due to administrator command
2020-09-30 22:27:56.446 CEST [30658]   STATEMENT:  select count(*) from "dm_b2b"."prlwytzkofskiv1"
2020-09-30 22:27:56.446 CEST [30657]   FATAL:  terminating connection due to administrator command
2020-09-30 22:27:56.446 CEST [30657]   STATEMENT:  select count(*) from "dm_b2b"."prlwytzkofskiv1"
2020-09-30 22:27:56.446 CEST [30656]   FATAL:  terminating connection due to administrator command
2020-09-30 22:27:56.446 CEST [30656]   STATEMENT:  select count(*) from "dm_b2b"."prlwytzkofskiv1"
2020-09-30 22:27:56.446 CEST [30655]   FATAL:  terminating connection due to administrator command
2020-09-30 22:27:56.446 CEST [30655]   STATEMENT:  select count(*) from "dm_b2b"."prlwytzkofskiv1"
2020-09-30 22:27:56.459 CEST [6482]   LOG:  background worker "parallel worker" (PID 30655) exited with exit code 1
2020-09-30 22:27:56.459 CEST [6482]   LOG:  background worker "parallel worker" (PID 30656) exited with exit code 1
2020-09-30 22:27:56.459 CEST [6482]   LOG:  background worker "parallel worker" (PID 30657) exited with exit code 1
2020-09-30 22:27:56.459 CEST [6482]   LOG:  background worker "parallel worker" (PID 30658) exited with exit code 1
2020-09-30 22:27:56.459 CEST [6482]   LOG:  background worker "parallel worker" (PID 30659) exited with exit code 1
2020-09-30 22:43:08.459 CEST [8055] 172.30.2.25 selfservice_prd ERROR:  schema "somethingelse" does not exist at character 71

Apparently, something is sending SIGTERM to our pg processes. I know that I'm not doing that, certainly not at those hours, and I'm the one who set up this system and am the only DBA of it.

Advice I found on the Internet is to use systemtap with some tap-script, but the scripts that I found just displayed the PID's of processes without telling me their names, which I didn't find all that useful in figuring out who was responsible, so I made an attempt (I have no experience with stap) at modifying it to print process names of signal sender and target:

The information contained in this e-mail is intended solely for the use of the individual or entity to whom it is addressed. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or action in relation to the contents of this information is strictly prohibited and may be unlawful and request you to delete this message and any attachments and advise the sender by return e-mail. The confidentiality of this message is not warranted. Apollo Vredestein and its subsidiaries rule out any and every liability resulting from this or any other electronic transmission

Please consider the environment before printing this e-mail