Please help! Server process was terminated by signal 11: Segmentation fault - Mailing list pgsql-general

Hello Everyone,

On 2015/12/30 17:51:01, Transaction was failed on our system.

I checked the pg_log, it states as below:

2015-12-30 17:51:01 CST [11260]: [42-1] LOG:  00000: server process (PID 14272) was terminated by signal 11: Segmentation fault

<part of the pg_log>

================================================

2015-12-30 17:50:52 CST [18954]: [5165-1] LOG:  00000: statement: select se_addlog('1','99','20151230161941915390111750','O','0000','');

2015-12-30 17:50:52 CST [18954]: [5166-1] LOCATION:  exec_simple_query, postgres.c:1082

2015-12-30 17:50:52 CST [18954]: [5167-1] LOG:  00000: duration: 0.951 ms

2015-12-30 17:50:52 CST [18954]: [5168-1] LOCATION:  exec_simple_query, postgres.c:1364

2015-12-30 17:50:52 CST [18954]: [5169-1] LOG:  00000: statement: unlisten *

2015-12-30 17:50:52 CST [18954]: [5170-1] LOCATION:  exec_simple_query, postgres.c:1082

2015-12-30 17:50:52 CST [18954]: [5171-1] LOG:  00000: duration: 0.118 ms

2015-12-30 17:50:52 CST [18954]: [5172-1] LOCATION:  exec_simple_query, postgres.c:1364

2015-12-30 17:50:52 CST [18954]: [5173-1] LOG:  00000: statement: select 1

2015-12-30 17:50:52 CST [18954]: [5174-1] LOCATION:  exec_simple_query, postgres.c:1082

2015-12-30 17:50:52 CST [18954]: [5175-1] LOG:  00000: duration: 0.160 ms

2015-12-30 17:50:52 CST [18954]: [5176-1] LOCATION:  exec_simple_query, postgres.c:1364

2015-12-30 17:50:52 CST [18954]: [5179-1] LOG:  00000: duration: 0.738 ms

2015-12-30 17:50:52 CST [18954]: [5180-1] LOCATION:  exec_simple_query, postgres.c:1364

2015-12-30 17:50:52 CST [18954]: [5181-1] LOG:  00000: statement: unlisten *

2015-12-30 17:50:52 CST [18954]: [5182-1] LOCATION:  exec_simple_query, postgres.c:1082

2015-12-30 17:50:52 CST [18954]: [5183-1] LOG:  00000: duration: 0.112 ms

2015-12-30 17:50:52 CST [18954]: [5184-1] LOCATION:  exec_simple_query, postgres.c:1364

2015-12-30 17:50:52 CST [18954]: [5185-1] LOG:  00000: statement: select 1

2015-12-30 17:50:52 CST [18954]: [5186-1] LOCATION:  exec_simple_query, postgres.c:1082

2015-12-30 17:50:52 CST [18954]: [5187-1] LOG:  00000: duration: 0.159 ms

2015-12-30 17:50:52 CST [18954]: [5188-1] LOCATION:  exec_simple_query, postgres.c:1364

2015-12-30 17:50:52 CST [18954]: [5190-1] LOCATION:  exec_simple_query, postgres.c:1082

2015-12-30 17:50:52 CST [18954]: [5191-1] LOG:  00000: duration: 0.832 ms

2015-12-30 17:50:52 CST [18954]: [5192-1] LOCATION:  exec_simple_query, postgres.c:1364

2015-12-30 17:50:52 CST [18954]: [5193-1] LOG:  00000: statement: unlisten *

2015-12-30 17:50:52 CST [18954]: [5194-1] LOCATION:  exec_simple_query, postgres.c:1082

2015-12-30 17:50:52 CST [18954]: [5195-1] LOG:  00000: duration: 0.111 ms

2015-12-30 17:50:52 CST [18954]: [5196-1] LOCATION:  exec_simple_query, postgres.c:1364

★2015-12-30 17:51:02 CST [14276]: [1-1] LOG:  00000: database system was interrupted; last known up at 2015-12-30 17:50:21 CST★

★2015-12-30 17:51:02 CST [14276]: [31-1] LOG:  00000: last completed transaction was at log time 2015-12-30 17:50:52.793516+08★

2015-12-30 17:51:01 CST [14270]: [1-1] LOG:  00000: connection received: host=[local]

2015-12-30 17:51:01 CST [14270]: [2-1] LOCATION:  BackendInitialize, postmaster.c:4056

2015-12-30 17:51:01 CST [14270]: [3-1] LOG:  00000: connection authorized: user=enterprisedb database=edb

2015-12-30 17:51:01 CST [14270]: [4-1] LOCATION:  PerformAuthentication, postinit.c:265

2015-12-30 17:51:01 CST [14271]: [1-1] LOG:  00000: connection received: host=[local]

2015-12-30 17:51:01 CST [14271]: [2-1] LOCATION:  BackendInitialize, postmaster.c:4056

2015-12-30 17:51:01 CST [14271]: [3-1] LOG:  00000: connection authorized: user=enterprisedb database=edb

2015-12-30 17:51:01 CST [14271]: [4-1] LOCATION:  PerformAuthentication, postinit.c:265

2015-12-30 17:51:01 CST [14270]: [5-1] LOG:  00000: statement: select now();

2015-12-30 17:51:01 CST [14270]: [6-1] LOCATION:  exec_simple_query, postgres.c:1082

2015-12-30 17:51:01 CST [14272]: [1-1] LOG:  00000: connection received: host=[local]

2015-12-30 17:51:01 CST [14272]: [2-1] LOCATION:  BackendInitialize, postmaster.c:4056

2015-12-30 17:51:01 CST [14270]: [7-1] LOG:  00000: duration: 1.523 ms

2015-12-30 17:51:01 CST [14270]: [8-1] LOCATION:  exec_simple_query, postgres.c:1364

2015-12-30 17:51:01 CST [14270]: [9-1] LOG:  00000: statement: select current_timestamp,pid,current_timestamp - query_start as runtime,datname,usename,client_addr,query

2015-12-30 17:51:01 CST [14270]: [10-1] LOCATION:  exec_simple_query, postgres.c:1082

2015-12-30 17:51:01 CST [14272]: [3-1] LOG:  00000: connection authorized: user=enterprisedb database=edb

2015-12-30 17:51:01 CST [14272]: [4-1] LOCATION:  PerformAuthentication, postinit.c:265

2015-12-30 17:51:01 CST [14271]: [5-1] LOG:  00000: statement: select CURRENT_TIMESTAMP(0), client_addr, count(client_addr) from pg_stat_activity group by client_addr order by client_addr;

2015-12-30 17:51:01 CST [14271]: [6-1] LOCATION:  exec_simple_query, postgres.c:1082

2015-12-30 17:51:01 CST [14272]: [5-1] LOG:  00000: statement: select now();

2015-12-30 17:51:01 CST [14272]: [6-1] LOCATION:  exec_simple_query, postgres.c:1082

2015-12-30 17:51:01 CST [14272]: [7-1] LOG:  00000: duration: 0.997 ms

2015-12-30 17:51:01 CST [14272]: [8-1] LOCATION:  exec_simple_query, postgres.c:1364

2015-12-30 17:51:01 CST [14272]: [9-1] LOG:  00000: statement: SELECT bl.pid                 AS blocked_pid,

2015-12-30 17:51:01 CST [14272]: [10-1] LOCATION:  exec_simple_query, postgres.c:1082

2015-12-30 17:51:01 CST [14270]: [11-1] LOG:  00000: duration: 5.044 ms

2015-12-30 17:51:01 CST [14270]: [12-1] LOCATION:  exec_simple_query, postgres.c:1364

2015-12-30 17:51:01 CST [14270]: [13-1] LOG:  00000: disconnection: session time: 0:00:00.015 user=enterprisedb database=edb host=[local]

2015-12-30 17:51:01 CST [14270]: [14-1] LOCATION:  log_disconnections, postgres.c:5993

2015-12-30 17:51:01 CST [14271]: [7-1] LOG:  00000: duration: 7.072 ms

2015-12-30 17:51:01 CST [14271]: [8-1] LOCATION:  exec_simple_query, postgres.c:1364

2015-12-30 17:51:01 CST [14271]: [9-1] LOG:  00000: disconnection: session time: 0:00:00.015 user=enterprisedb database=edb host=[local]

2015-12-30 17:51:01 CST [14271]: [10-1] LOCATION:  log_disconnections, postgres.c:5993

★2015-12-30 17:51:01 CST [11260]: [42-1] LOG:  00000: server process (PID 14272) was terminated by signal 11: Segmentation fault★

★2015-12-30 17:51:01 CST [11260]: [43-1] DETAIL:  Failed process was running: SELECT bl.pid                 AS blocked_pid,★

2015-12-30 17:51:01 CST [11260]: [44-1] LOCATION:  LogChildExit, postmaster.c:3362

2015-12-30 17:51:01 CST [11260]: [45-1] LOG:  00000: terminating any other active server processes

2015-12-30 17:51:01 CST [11260]: [46-1] LOCATION:  HandleChildCrash, postmaster.c:3083

2015-12-30 17:51:01 CST [18954]: [5197-1] WARNING:  57P02: terminating connection because of crash of another server process

2015-12-30 17:51:01 CST [18954]: [5198-1] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

2015-12-30 17:51:01 CST [18954]: [5199-1] HINT:  In a moment you should be able to reconnect to the database and repeat your command.

2015-12-30 17:51:01 CST [18954]: [5200-1] LOCATION:  quickdie, postgres.c:3855

2015-12-30 17:51:01 CST [18953]: [4969-1] WARNING:  57P02: terminating connection because of crash of another server process

2015-12-30 17:51:01 CST [18953]: [4970-1] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

2015-12-30 17:51:01 CST [18953]: [4971-1] HINT:  In a moment you should be able to reconnect to the database and repeat your command.

2015-12-30 17:51:01 CST [18953]: [4972-1] LOCATION:  quickdie, postgres.c:3855

================================================

The server log states:

Dec 30 17:51:01 egtcdb1 kernel: edb-postgres[14272]: segfault at 30 ip 00000000006a1f5a sp 00007fff70c5e2e0 error 4 in edb-postgres[400000+5e8000]

Server information,

egtcdb1 (Linux release 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013) (0)

[enterprisedb@egtcdb1 ~]$ psql

psql.bin (9.4.1.3)

Type "help" for help.

edb=# select version ;   

LOG:  statement: select version ;

LOG:  duration: 1.603 ms

                                                      version                                                     

-------------------------------------------------------------------------------------------------------------------

EnterpriseDB 9.4.1.3 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-55), 64-bit

(1 row)

[enterprisedb@egtcdb1 ~]$ free -tm

             total       used       free     shared    buffers     cached

Mem:         15888      12506       3381          0        502      11204

-/+ buffers/cache:        799      15088

Swap:         4015         36       3979

Total:       19904      12543       7361

Cpu 4 core, memory 16GB

Postgresql.conf information,

shared_buffers = 4GB

temp_buffers = 8MB

work_mem = 56696kB

maintenance_work_mem = 256MB

effective_cache_size = 5324784kB

On my case, Please give me the instruction in steps on how to figure out the

reason why did this “signal 11” happen and how to solve “signal 11: Segmentation fault” issue.

Thank you in advance.

John