[BUGS] *** glibc detected *** postgres: SELECT: double free or corruption(!prev): 0x0000000001e59de0 *** - Mailing list pgsql-bugs
| From | Manjula Bharathan |
|---|---|
| Subject | [BUGS] *** glibc detected *** postgres: SELECT: double free or corruption(!prev): 0x0000000001e59de0 *** |
| Date | |
| Msg-id | 3976998C-8D3B-4825-9B10-69ECB70A597A@appnexus.com Whole thread Raw |
| Responses |
Re: [BUGS] *** glibc detected *** postgres: SELECT: double free orcorruption (!prev): 0x0000000001e59de0 ***
|
| List | pgsql-bugs |
Hi,
We have been seeing issues with our Postgres Primary /Master server, where we see the following message logged to the log file
*** glibc detected *** postgres: SELECT: double free or corruption (!prev): 0x0000000001e59de0 ***
This has happened thrice in the last week , twice last Sunday June 25th and once on Thursday June 29th.
and the query causing the termination twice on June 25th was the same . The query on June 29th was a different one.
Note: We recently upgraded postgresql version to 9.6.3 from version 9.5 on June 21st.. We did not encounter this error before the Upgrade.
Operating System : CentOS release 6.6
We saw a similar bug being filed for Postgresql Version 9.5.7
https://www.postgresql.org/message-id/4DD2043D33498E4B82FF57B256C47540D56618%40ex-04.aam.local
At the time ,the parent postgres process remained up, but the child processes got restarted.
postgres 86053 1 0 Jun21 ? 00:19:19 /usr/pgsql-9.6/bin/postgres -D /var/lib/pgsql/9.6/data
postgres 75590 86053 0 16:23 ? 00:00:00 postgres: autovacuum worker process ourdatabase
postgres 86054 86053 2 Jun21 ? 02:58:37 postgres: logger process
postgres 114396 86053 0 11:55 ? 00:00:25 postgres: checkpointer process
postgres 114397 86053 0 11:55 ? 00:00:03 postgres: writer process
postgres 114398 86053 0 11:55 ? 00:00:08 postgres: wal writer process
postgres 114399 86053 0 11:55 ? 00:00:00 postgres: autovacuum launcher process
postgres 114400 86053 0 11:55 ? 00:00:00 postgres: archiver process last was 0000000100000633000000BD
postgres 114401 86053 0 11:55 ? 00:01:21 postgres: stats collector process
postgres 114550 86053 0 11:55 ? 00:00:42 postgres: wal sender process replication 10.l.zz.xx(52897) streaming 633/BE3A08D8
postgres 114551 86053 0 11:55 ? 00:00:43 postgres: wal sender process replication 10.l.zz.yy(35246) streaming 633/BE3A08D8
postgres 114553 86053 0 11:55 ? 00:00:45 postgres: wal sender process replication 10.l.qq.kk(51182) streaming 633/BE3A08D8
postgres 114555 86053 0 11:55 ? 00:00:46 postgres: wal sender process replication 10.l.ww.mm(57154) streaming 633/BE3A08D8
postgres 114557 86053 0 11:55 ? 00:00:46 postgres: wal sender process replication 10.l.ee.hh(60865) streaming 633/BE3A08D8
postgres 114559 86053 0 11:55 ? 00:00:44 postgres: wal sender process replication 10.l.pp.tt(58809) streaming 633/BE3A08D8
postgres 114570 86053 0 11:55 ? 00:00:41 postgres: wal sender process replication 10.l.bb.uu(57592) streaming 633/BE3A08D8
Incident 1 (Jun 25 11:55)
-------------
-rw------- 1 postgres postgres 17075 Jun 25 11:55 postgresql-Sun-11.log
$ cat postgresql-Sun-11.log
*** glibc detected *** postgres: app_gigi_feed ourdatabase 66.11.111.11(42947) SELECT: double free or corruption (!prev): 0x0000000001e59de0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3711a75e66]
/lib64/libc.so.6[0x3711a789b3]
postgres: app_gigi_feed ourdatabase 66.11.111.11(42947) SELECT(LogicalTapeSetClose+0x15)[0x7fd645]
postgres: app_gigi_feed ourdatabase 66.11.111.11(42947) SELECT(tuplesort_end+0x3c)[0x80014c]
postgres: app_gigi_feed ourdatabase 66.11.111.11(42947) SELECT(ExecEndSort+0x2a)[0x5ea80a]
postgres: app_gigi_feed ourdatabase 66.11.111.11(42947) SELECT(ExecEndNestLoop+0x1b)[0x5e7e5b]
postgres: app_gigi_feed ourdatabase 66.11.111.11(42947) SELECT(ExecEndNestLoop+0x1b)[0x5e7e5b]
postgres: app_gigi_feed ourdatabase 66.11.111.11(42947) SELECT(ExecEndNestLoop+0x1b)[0x5e7e5b]
postgres: app_gigi_feed ourdatabase 66.11.111.11(42947) SELECT(standard_ExecutorEnd+0x2e)[0x5caa9e]
postgres: app_gigi_feed ourdatabase 66.11.111.11(42947) SELECT(PortalCleanup+0x9e)[0x592dae]
postgres: app_gigi_feed ourdatabase 66.11.111.11(42947) SELECT(PortalDrop+0x2a)[0x7faa2a]
postgres: app_gigi_feed ourdatabase 66.11.111.11(42947) SELECT[0x6df472]
postgres: app_gigi_feed ourdatabase 66.11.111.11(42947) SELECT(PostgresMain+0xdcc)[0x6e0b2c]
postgres: app_gigi_feed ourdatabase 66.11.111.11(42947) SELECT(PostmasterMain+0x1865)[0x680d55]
postgres: app_gigi_feed ourdatabase 66.11.111.11(42947) SELECT(main+0x7a8)[0x608dc8]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3711a1ed5d]
postgres: app_gigi_feed ourdatabase 66.11.111.11(42947) SELECT[0x46be39]
======= Memory map: ========
00400000-00a13000 r-xp 00000000 fd:00 18221735 /usr/pgsql-9.6/bin/postgres
00c13000-00c20000 rw-p 00613000 fd:00 18221735 /usr/pgsql-9.6/bin/postgres
00c20000-00c71000 rw-p 00000000 00:00 0
01cd4000-01d5a000 rw-p 00000000 00:00 0
01d5a000-0251c000 rw-p 00000000 00:00 0
3711600000-3711620000 r-xp 00000000 fd:00 2883590 /lib64/ld-2.12.so
371181f000-3711820000 r--p 0001f000 fd:00 2883590 /lib64/ld-2.12.so
3711820000-3711821000 rw-p 00020000 fd:00 2883590 /lib64/ld-2.12.so
3711821000-3711822000 rw-p 00000000 00:00 0
3711a00000-3711b8a000 r-xp 00000000 fd:00 2883612 /lib64/libc-2.12.so
3711b8a000-3711d8a000 ---p 0018a000 fd:00 2883612 /lib64/libc-2.12.so
3711d8a000-3711d8e000 r--p 0018a000 fd:00 2883612 /lib64/libc-2.12.so
3711d8e000-3711d8f000 rw-p 0018e000 fd:00 2883612 /lib64/libc-2.12.so
3711d8f000-3711d94000 rw-p 00000000 00:00 0
3711e00000-3711e17000 r-xp 00000000 fd:00 2883660 /lib64/libpthread-2.12.so
3711e17000-3712017000 ---p 00017000 fd:00 2883660 /lib64/libpthread-2.12.so
3712017000-3712018000 r--p 00017000 fd:00 2883660 /lib64/libpthread-2.12.so
3712018000-3712019000 rw-p 00018000 fd:00 2883660 /lib64/libpthread-2.12.so
3712019000-371201d000 rw-p 00000000 00:00 0
3712200000-3712202000 r-xp 00000000 fd:00 2883634 /lib64/libdl-2.12.so
3712202000-3712402000 ---p 00002000 fd:00 2883634 /lib64/libdl-2.12.so
3712402000-3712403000 r--p 00002000 fd:00 2883634 /lib64/libdl-2.12.so
3712403000-3712404000 rw-p 00003000 fd:00 2883634 /lib64/libdl-2.12.so
3712600000-3712607000 r-xp 00000000 fd:00 2883661 /lib64/librt-2.12.so
3712607000-3712806000 ---p 00007000 fd:00 2883661 /lib64/librt-2.12.so
3712806000-3712807000 r--p 00006000 fd:00 2883661 /lib64/librt-2.12.so
3712807000-3712808000 rw-p 00007000 fd:00 2883661 /lib64/librt-2.12.so
3712a00000-3712a83000 r-xp 00000000 fd:00 2883948 /lib64/libm-2.12.so
3712a83000-3712c82000 ---p 00083000 fd:00 2883948 /lib64/libm-2.12.so
3712c82000-3712c83000 r--p 00082000 fd:00 2883948 /lib64/libm-2.12.so
3712c83000-3712c84000 rw-p 00083000 fd:00 2883948 /lib64/libm-2.12.so
3712e00000-3712e15000 r-xp 00000000 fd:00 2883942 /lib64/libz.so.1.2.3
3712e15000-3713014000 ---p 00015000 fd:00 2883942 /lib64/libz.so.1.2.3
3713014000-3713015000 r--p 00014000 fd:00 2883942 /lib64/libz.so.1.2.3
3713015000-3713016000 rw-p 00015000 fd:00 2883942 /lib64/libz.so.1.2.3
3713200000-371321d000 r-xp 00000000 fd:00 2883949 /lib64/libselinux.so.1
371321d000-371341c000 ---p 0001d000 fd:00 2883949 /lib64/libselinux.so.1
371341c000-371341d000 r--p 0001c000 fd:00 2883949 /lib64/libselinux.so.1
371341d000-371341e000 rw-p 0001d000 fd:00 2883949 /lib64/libselinux.so.1
371341e000-371341f000 rw-p 00000000 00:00 0
3713a00000-3713a02000 r-xp 00000000 fd:00 2883966 /lib64/libkeyutils.so.1.3
3713a02000-3713c01000 ---p 00002000 fd:00 2883966 /lib64/libkeyutils.so.1.3
3713c01000-3713c02000 r--p 00001000 fd:00 2883966 /lib64/libkeyutils.so.1.3
3713c02000-3713c03000 rw-p 00002000 fd:00 2883966 /lib64/libkeyutils.so.1.3
3713e00000-3713e16000 r-xp 00000000 fd:00 2883959 /lib64/libresolv-2.12.so
3713e16000-3714016000 ---p 00016000 fd:00 2883959 /lib64/libresolv-2.12.so
3714016000-3714017000 r--p 00016000 fd:00 2883959 /lib64/libresolv-2.12.so
3714017000-3714018000 rw-p 00017000 fd:00 2883959 /lib64/libresolv-2.12.so
3714018000-371401a000 rw-p 00000000 00:00 0
3714200000-3714219000 r-xp 00000000 fd:00 17303744 /usr/lib64/libsasl2.so.2.0.23
3714219000-3714418000 ---p 00019000 fd:00 17303744 /usr/lib64/libsasl2.so.2.0.23
3714418000-3714419000 r--p 00018000 fd:00 17303744 /usr/lib64/libsasl2.so.2.0.23
3714419000-371441a000 rw-p 00019000 fd:00 17303744 /usr/lib64/libsasl2.so.2.0.23
3714600000-371460c000 r-xp 00000000 fd:00 2883803 /lib64/libpam.so.0.82.2
371460c000-371480c000 ---p 0000c000 fd:00 2883803 /lib64/libpam.so.0.82.2
371480c000-371480d000 r--p 0000c000 fd:00 2883803 /lib64/libpam.so.0.82.2
371480d000-371480e000 rw-p 0000d000 fd:00 2883803 /lib64/libpam.so.0.82.2
3714e00000-3714e03000 r-xp 00000000 fd:00 2883958 /lib64/libplds4.so
3714e03000-3715002000 ---p 00003000 fd:00 2883958 /lib64/libplds4.so
3715002000-3715003000 r--p 00002000 fd:00 2883958 /lib64/libplds4.so
3715003000-3715004000 rw-p 00003000 fd:00 2883958 /lib64/libplds4.so
3715200000-3715239000 r-xp 00000000 fd:00 2883956 /lib64/libnspr4.so
3715239000-3715439000 ---p 00039000 fd:00 2883956 /lib64/libnspr4.so
3715439000-371543a000 r--p 00039000 fd:00 2883956 /lib64/libnspr4.so
371543a000-371543c000 rw-p 0003a000 fd:00 2883956 /lib64/libnspr4.so
371543c000-371543e000 rw-p 00000000 00:00 0
3715600000-3715603000 r-xp 00000000 fd:00 2883720 /lib64/libcom_err.so.2.1
3715603000-3715802000 ---p 00003000 fd:00 2883720 /lib64/libcom_err.so.2.1
3715802000-3715803000 r--p 00002000 fd:00 2883720 /lib64/libcom_err.so.2.1
3715803000-3715804000 rw-p 00003000 fd:00 2883720 /lib64/libcom_err.so.2.1
3715a00000-3715a02000 r-xp 00000000 fd:00 2883714 /lib64/libfreebl3.so
3715a02000-3715c01000 ---p 00002000 fd:00 2883714 /lib64/libfreebl3.so
3715c01000-3715c02000 r--p 00001000 fd:00 2883714 /lib64/libfreebl3.so
3715c02000-3715c03000 rw-p 00002000 fd:00 2883714 /lib64/libfreebl3.so
3715e00000-3715e07000 r-xp 00000000 fd:00 2883798 /lib64/libcrypt-2.12.so
3715e07000-3716007000 ---p 00007000 fd:00 2883798 /lib64/libcrypt-2.12.so
3716007000-3716008000 r--p 00007000 fd:00 2883798 /lib64/libcrypt-2.12.so
3716008000-3716009000 rw-p 00008000 fd:00 2883798 /lib64/libcrypt-2.12.so
3716009000-3716037000 rw-p 00000000 00:00 0
3716200000-3716225000 r-xp 00000000 fd:00 17303337 /usr/lib64/libnssutil3.so
3716225000-3716425000 ---p 00025000 fd:00 17303337 /usr/lib64/libnssutil3.so
3716425000-371642b000 r--p 00025000 fd:00 17303337 /usr/lib64/libnssutil3.so
371642b000-371642c000 rw-p 0002b000 fd:00 17303337 /usr/lib64/libnssutil3.so
3716600000-37167b9000 r-xp 00000000 fd:00 17305030 /usr/lib64/libcrypto.so.1.0.1e
37167b9000-37169b8000 ---p 001b9000 fd:00 17305030 /usr/lib64/libcrypto.so.1.0.1e
37169b8000-37169d3000 r--p 001b8000 fd:00 17305030 /usr/lib64/libcrypto.so.1.0.1e
37169d3000-37169df000 rw-p 001d3000 fd:00 17305030 /usr/lib64/libcrypto.so.1.0.1e
37169df000-37169e3000 rw-p 00000000 00:00 0
3717600000-3717641000 r-xp 00000000 fd:00 2883970 /lib64/libgssapi_krb5.so.2.2
3717641000-3717841000 ---p 00041000 fd:00 2883970 /lib64/libgssapi_krb5.so.2.2
3717841000-3717842000 r--p 00041000 fd:00 2883970 /lib64/libgssapi_krb5.so.2.2
3717842000-3717844000 rw-p 00042000 fd:00 2883970 /lib64/libgssapi_krb5.so.2.2
3717a00000-3717adb000 r-xp 00000000 fd:00 2883969 /lib64/libkrb5.so.3.3
3717adb000-3717cda000 ---p 000db000 fd:00 2883969 /lib64/libkrb5.so.3.3
3717cda000-3717ce4000 r--p 000da000 fd:00 2883969 /lib64/libkrb5.so.3.3
3717ce4000-3717ce6000 rw-p 000e4000 fd:00 2883969 /lib64/libkrb5.so.3.3
3717e00000-3717e29000 r-xp 00000000 fd:00 2883968 /lib64/libk5crypto.so.3.1
3717e29000-3718029000 ---p 00029000 fd:00 2883968 /lib64/libk5crypto.so.3.1
3718029000-371802a000 r--p 00029000 fd:00 2883968 /lib64/libk5crypto.so.3.1
371802a000-371802b000 rw-p 0002a000 fd:00 2883968 /lib64/libk5crypto.so.3.1
371802b000-371802c000 rw-p 00000000 00:00 0
3718200000-371820a000 r-xp 00000000 fd:00 2883967 /lib64/libkrb5support.so.0.1
371820a000-3718409000 ---p 0000a000 fd:00 2883967 /lib64/libkrb5support.so.0.1
3718409000-371840a000 r--p 00009000 fd:00 2883967 /lib64/libkrb5support.so.0.1
371840a000-371840b000 rw-p 0000a000 fd:00 2883967 /lib64/libkrb5support.so.0.1
3718600000-3718749000 r-xp 00000000 fd:00 17303500 /usr/lib64/libxml2.so.2.7.6
3718749000-3718948000 ---p 00149000 fd:00 17303500 /usr/lib64/libxml2.so.2.7.6
3718948000-3718951000 rw-p 00148000 fd:00 17303500 /usr/lib64/libxml2.so.2.7.6
3718951000-3718953000 rw-p 00000000 00:00 0
3718a00000-3718a62000 r-xp 00000000 fd:00 17305217 /usr/lib64/libssl.so.1.0.1e
3718a62000-3718c61000 ---p 00062000 fd:00 17305217 /usr/lib64/libssl.so.1.0.1e
3718c61000-3718c65000 r--p 00061000 fd:00 17305217 /usr/lib64/libssl.so.1.0.1e
3718c65000-3718c6c000 rw-p 00065000 fd:00 17305217 /usr/lib64/libssl.so.1.0.1e
3d2a800000-3d2a937000 r-xp 00000000 fd:00 17305255 /usr/lib64/libnss3.so
3d2a937000-3d2ab36000 ---p 00137000 fd:00 17305255 /usr/lib64/libnss3.so
3d2ab36000-3d2ab3b000 r--p 00136000 fd:00 17305255 /usr/lib64/libnss3.so
3d2ab3b000-3d2ab3d000 rw-p 0013b000 fd:00 17305255 /usr/lib64/libnss3.so
3d2ab3d000-3d2ab3f000 rw-p 00000000 00:00 0
3d2ac00000-3d2ac3b000 r-xp 00000000 fd:00 17305256 /usr/lib64/libssl3.so
3d2ac3b000-3d2ae3a000 ---p 0003b000 fd:00 17305256 /usr/lib64/libssl3.so
3d2ae3a000-3d2ae3d000 r--p 0003a000 fd:00 17305256 /usr/lib64/libssl3.so
3d2ae3d000-3d2ae3e000 rw-p 0003d000 fd:00 17305256 /usr/lib64/libssl3.so
3d2ae3e000-3d2ae3f000 rw-p 00000000 00:00 0
3d2b000000-3d2b028000 r-xp 00000000 fd:00 17305257 /usr/lib64/libsmime3.so
3d2b028000-3d2b227000 ---p 00028000 fd:00 17305257 /usr/lib64/libsmime3.so
3d2b227000-3d2b22b000 r--p 00027000 fd:00 17305257 /usr/lib64/libsmime3.so
3d2b22b000-3d2b22c000 rw-p 0002b000 fd:00 17305257 /usr/lib64/libsmime3.so
3d2b400000-3d2b40e000 r-xp 00000000 fd:00 2883664 /lib64/liblber-2.4.so.2.10.2
3d2b40e000-3d2b60d000 ---p 0000e000 fd:00 2883664 /lib64/liblber-2.4.so.2.10.2
3d2b60d000-3d2b60e000 r--p 0000d000 fd:00 2883664 /lib64/liblber-2.4.so.2.10.2
3d2b60e000-3d2b60f000 rw-p 0000e000 fd:00 2883664 /lib64/liblber-2.4.so.2.10.2
3d2b800000-3d2b84d000 r-xp 00000000 fd:00 2883814 /lib64/libldap-2.4.so.2.10.2
3d2b84d000-3d2ba4c000 ---p 0004d000 fd:00 2883814 /lib64/libldap-2.4.so.2.10.2
3d2ba4c000-3d2ba4e000 r--p 0004c000 fd:00 2883814 /lib64/libldap-2.4.so.2.10.2
3d2ba4e000-3d2ba50000 rw-p 0004e000 fd:00 2883814 /lib64/libldap-2.4.so.2.10.2
7f3101fdd000-7f3102427000 rw-p 00000000 00:00 0
7f3102d63000-7f3102d79000 r-xp 00000000 fd:00 2883652 /lib64/libgcc_s-4.4.7-20120601.so.1
7f3102d79000-7f3102f78000 ---p 00016000 fd:00 2883652 /lib64/libgcc_s-4.4.7-20120601.so.1
7f3102f78000-7f3102f79000 rw-p 00015000 fd:00 2883652 /lib64/libgcc_s-4.4.7-20120601.so.1
7f3102f79000-7f310397b000 rw-p 00000000 00:00 0
7f310397b000-7f3103afc000 rw-p 00000000 00:00 0
7f3103afc000-7f3103b7d000 rw-p 00000000 00:00 0
7f3103b7d000-7f3103c7e000 rw-p 00000000 00:00 0
7f3103d80000-7f3104202000 rw-p 00000000 00:00 0
7f3104263000-7f31044a5000 rw-p 00000000 00:00 0
7f31044b5000-7f31048dc000 rw-p 00000000 00:00 0
7f31048dc000-7f31048e8000 r-xp 00000000 fd:00 2883613 /lib64/libnss_files-2.12.so
7f31048e8000-7f3104ae8000 ---p 0000c000 fd:00 2883613 /lib64/libnss_files-2.12.so
7f3104ae8000-7f3104ae9000 r--p 0000c000 fd:00 2883613 /lib64/libnss_files-2.12.so
7f3104ae9000-7f3104aea000 rw-p 0000d000 fd:00 2883613 /lib64/libnss_files-2.12.so
7f3104af2000-7f3520056000 rw-s 00000000 00:04 4272906649 /dev/zero (deleted)
7f3520056000-7f352005e000 r-xp 00000000 fd:00 18221940 /usr/pgsql-9.6/lib/pg_stat_statements.so
7f352005e000-7f352025d000 ---p 00008000 fd:00 18221940 /usr/pgsql-9.6/lib/pg_stat_statements.so
7f352025d000-7f352025e000 rw-p 00007000 fd:00 18221940 /usr/pgsql-9.6/lib/pg_stat_statements.so
7f352025e000-7f35260ee000 r--p 00000000 fd:00 17305075 /usr/lib/locale/locale-archive
7f35260ee000-7f35260f6000 rw-p 00000000 00:00 0
7f35260f6000-7f35260fa000 r-xp 00000000 fd:00 2883957 /lib64/libplc4.so
7f35260fa000-7f35262f9000 ---p 00004000 fd:00 2883957 /lib64/libplc4.so
7f35262f9000-7f35262fa000 r--p 00003000 fd:00 2883957 /lib64/libplc4.so
7f35262fa000-7f35262fb000 rw-p 00004000 fd:00 2883957 /lib64/libplc4.so
7f35262fb000-7f3526300000 rw-p 00000000 00:00 0
7f3526300000-7f3526317000 r-xp 00000000 fd:00 2883712 /lib64/libaudit.so.1.0.0
7f3526317000-7f3526517000 ---p 00017000 fd:00 2883712 /lib64/libaudit.so.1.0.0
7f3526517000-7f3526518000 r--p 00017000 fd:00 2883712 /lib64/libaudit.so.1.0.0
7f3526518000-7f3526523000 rw-p 00018000 fd:00 2883712 /lib64/libaudit.so.1.0.0
7f3526523000-7f3526527000 rw-p 00000000 00:00 0
7f352652a000-7f352652b000 rw-p 00000000 00:00 0
7f352652b000-7f352652e000 rw-s 00000000 00:10 4272906653 /dev/shm/PostgreSQL.1269737114
7f352652e000-7f352652f000 rw-s 00000000 00:04 3178496 /SYSV0052e2c1 (deleted)
7f352652f000-7f3526530000 rw-p 00000000 00:00 0
7fff9af75000-7fff9af8a000 rw-p 00000000 00:00 0 [stack]
7fff9afff000-7fff9b000000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Corresponding message in csv file
2017-06-25 19:55:02.088 GMT,,,86053,,594a6c83.15025,8,,2017-06-21 12:54:27 GMT,,0,LOG,00000,"server process (PID 109465) was terminated by signal 6: Aborted","Failed process was running:
Incident 2 ( Jun 25 19:55)
————————————————
-rw------- 1 postgres postgres 17074 Jun 25 19:55 postgresql-Sun-19.log
$ cat postgresql-Sun-19.log
*** glibc detected *** postgres: app_gigi_feed ourdatabase 66.11.111.11(53140) SELECT: double free or corruption (!prev): 0x0000000001e583e0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3711a75e66]
/lib64/libc.so.6[0x3711a789b3]
postgres: app_gigi_feed ourdatabase 66.11.111.11(53140) SELECT(LogicalTapeSetClose+0x15)[0x7fd645]
postgres: app_gigi_feed ourdatabase 66.11.111.11(53140) SELECT(tuplesort_end+0x3c)[0x80014c]
postgres: app_gigi_feed ourdatabase 66.11.111.11(53140) SELECT(ExecEndSort+0x2a)[0x5ea80a]
postgres: app_gigi_feed ourdatabase 66.11.111.11(53140) SELECT(ExecEndNestLoop+0x1b)[0x5e7e5b]
postgres: app_gigi_feed ourdatabase 66.11.111.11(53140) SELECT(ExecEndNestLoop+0x1b)[0x5e7e5b]
postgres: app_gigi_feed ourdatabase 66.11.111.11(53140) SELECT(ExecEndNestLoop+0x1b)[0x5e7e5b]
postgres: app_gigi_feed ourdatabase 66.11.111.11(53140) SELECT(standard_ExecutorEnd+0x2e)[0x5caa9e]
postgres: app_gigi_feed ourdatabase 66.11.111.11(53140) SELECT(PortalCleanup+0x9e)[0x592dae]
postgres: app_gigi_feed ourdatabase 66.11.111.11(53140) SELECT(PortalDrop+0x2a)[0x7faa2a]
postgres: app_gigi_feed ourdatabase 66.11.111.11(53140) SELECT[0x6df472]
postgres: app_gigi_feed ourdatabase 66.11.111.11(53140) SELECT(PostgresMain+0xdcc)[0x6e0b2c]
postgres: app_gigi_feed ourdatabase 66.11.111.11(53140) SELECT(PostmasterMain+0x1865)[0x680d55]
postgres: app_gigi_feed ourdatabase 66.11.111.11(53140) SELECT(main+0x7a8)[0x608dc8]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3711a1ed5d]
postgres: app_gigi_feed ourdatabase 66.11.111.11(53140) SELECT[0x46be39]
======= Memory map: ========
00400000-00a13000 r-xp 00000000 fd:00 18221735 /usr/pgsql-9.6/bin/postgres
00c13000-00c20000 rw-p 00613000 fd:00 18221735 /usr/pgsql-9.6/bin/postgres
00c20000-00c71000 rw-p 00000000 00:00 0
01cd4000-01d5a000 rw-p 00000000 00:00 0
01d5a000-0251a000 rw-p 00000000 00:00 0
3711600000-3711620000 r-xp 00000000 fd:00 2883590 /lib64/ld-2.12.so
371181f000-3711820000 r--p 0001f000 fd:00 2883590 /lib64/ld-2.12.so
3711820000-3711821000 rw-p 00020000 fd:00 2883590 /lib64/ld-2.12.so
3711821000-3711822000 rw-p 00000000 00:00 0
3711a00000-3711b8a000 r-xp 00000000 fd:00 2883612 /lib64/libc-2.12.so
3711b8a000-3711d8a000 ---p 0018a000 fd:00 2883612 /lib64/libc-2.12.so
3711d8a000-3711d8e000 r--p 0018a000 fd:00 2883612 /lib64/libc-2.12.so
3711d8e000-3711d8f000 rw-p 0018e000 fd:00 2883612 /lib64/libc-2.12.so
3711d8f000-3711d94000 rw-p 00000000 00:00 0
3711e00000-3711e17000 r-xp 00000000 fd:00 2883660 /lib64/libpthread-2.12.so
3711e17000-3712017000 ---p 00017000 fd:00 2883660 /lib64/libpthread-2.12.so
3712017000-3712018000 r--p 00017000 fd:00 2883660 /lib64/libpthread-2.12.so
3712018000-3712019000 rw-p 00018000 fd:00 2883660 /lib64/libpthread-2.12.so
3712019000-371201d000 rw-p 00000000 00:00 0
3712200000-3712202000 r-xp 00000000 fd:00 2883634 /lib64/libdl-2.12.so
3712202000-3712402000 ---p 00002000 fd:00 2883634 /lib64/libdl-2.12.so
3712402000-3712403000 r--p 00002000 fd:00 2883634 /lib64/libdl-2.12.so
3712403000-3712404000 rw-p 00003000 fd:00 2883634 /lib64/libdl-2.12.so
3712600000-3712607000 r-xp 00000000 fd:00 2883661 /lib64/librt-2.12.so
3712607000-3712806000 ---p 00007000 fd:00 2883661 /lib64/librt-2.12.so
3712806000-3712807000 r--p 00006000 fd:00 2883661 /lib64/librt-2.12.so
3712807000-3712808000 rw-p 00007000 fd:00 2883661 /lib64/librt-2.12.so
3712a00000-3712a83000 r-xp 00000000 fd:00 2883948 /lib64/libm-2.12.so
3712a83000-3712c82000 ---p 00083000 fd:00 2883948 /lib64/libm-2.12.so
3712c82000-3712c83000 r--p 00082000 fd:00 2883948 /lib64/libm-2.12.so
3712c83000-3712c84000 rw-p 00083000 fd:00 2883948 /lib64/libm-2.12.so
3712e00000-3712e15000 r-xp 00000000 fd:00 2883942 /lib64/libz.so.1.2.3
3712e15000-3713014000 ---p 00015000 fd:00 2883942 /lib64/libz.so.1.2.3
3713014000-3713015000 r--p 00014000 fd:00 2883942 /lib64/libz.so.1.2.3
3713015000-3713016000 rw-p 00015000 fd:00 2883942 /lib64/libz.so.1.2.3
3713200000-371321d000 r-xp 00000000 fd:00 2883949 /lib64/libselinux.so.1
371321d000-371341c000 ---p 0001d000 fd:00 2883949 /lib64/libselinux.so.1
371341c000-371341d000 r--p 0001c000 fd:00 2883949 /lib64/libselinux.so.1
371341d000-371341e000 rw-p 0001d000 fd:00 2883949 /lib64/libselinux.so.1
371341e000-371341f000 rw-p 00000000 00:00 0
3713a00000-3713a02000 r-xp 00000000 fd:00 2883966 /lib64/libkeyutils.so.1.3
3713a02000-3713c01000 ---p 00002000 fd:00 2883966 /lib64/libkeyutils.so.1.3
3713c01000-3713c02000 r--p 00001000 fd:00 2883966 /lib64/libkeyutils.so.1.3
3713c02000-3713c03000 rw-p 00002000 fd:00 2883966 /lib64/libkeyutils.so.1.3
3713e00000-3713e16000 r-xp 00000000 fd:00 2883959 /lib64/libresolv-2.12.so
3713e16000-3714016000 ---p 00016000 fd:00 2883959 /lib64/libresolv-2.12.so
3714016000-3714017000 r--p 00016000 fd:00 2883959 /lib64/libresolv-2.12.so
3714017000-3714018000 rw-p 00017000 fd:00 2883959 /lib64/libresolv-2.12.so
3714018000-371401a000 rw-p 00000000 00:00 0
3714200000-3714219000 r-xp 00000000 fd:00 17303744 /usr/lib64/libsasl2.so.2.0.23
3714219000-3714418000 ---p 00019000 fd:00 17303744 /usr/lib64/libsasl2.so.2.0.23
3714418000-3714419000 r--p 00018000 fd:00 17303744 /usr/lib64/libsasl2.so.2.0.23
3714419000-371441a000 rw-p 00019000 fd:00 17303744 /usr/lib64/libsasl2.so.2.0.23
3714600000-371460c000 r-xp 00000000 fd:00 2883803 /lib64/libpam.so.0.82.2
371460c000-371480c000 ---p 0000c000 fd:00 2883803 /lib64/libpam.so.0.82.2
371480c000-371480d000 r--p 0000c000 fd:00 2883803 /lib64/libpam.so.0.82.2
371480d000-371480e000 rw-p 0000d000 fd:00 2883803 /lib64/libpam.so.0.82.2
3714e00000-3714e03000 r-xp 00000000 fd:00 2883958 /lib64/libplds4.so
3714e03000-3715002000 ---p 00003000 fd:00 2883958 /lib64/libplds4.so
3715002000-3715003000 r--p 00002000 fd:00 2883958 /lib64/libplds4.so
3715003000-3715004000 rw-p 00003000 fd:00 2883958 /lib64/libplds4.so
3715200000-3715239000 r-xp 00000000 fd:00 2883956 /lib64/libnspr4.so
3715239000-3715439000 ---p 00039000 fd:00 2883956 /lib64/libnspr4.so
3715439000-371543a000 r--p 00039000 fd:00 2883956 /lib64/libnspr4.so
371543a000-371543c000 rw-p 0003a000 fd:00 2883956 /lib64/libnspr4.so
371543c000-371543e000 rw-p 00000000 00:00 0
3715600000-3715603000 r-xp 00000000 fd:00 2883720 /lib64/libcom_err.so.2.1
3715603000-3715802000 ---p 00003000 fd:00 2883720 /lib64/libcom_err.so.2.1
3715802000-3715803000 r--p 00002000 fd:00 2883720 /lib64/libcom_err.so.2.1
3715803000-3715804000 rw-p 00003000 fd:00 2883720 /lib64/libcom_err.so.2.1
3715a00000-3715a02000 r-xp 00000000 fd:00 2883714 /lib64/libfreebl3.so
3715a02000-3715c01000 ---p 00002000 fd:00 2883714 /lib64/libfreebl3.so
3715c01000-3715c02000 r--p 00001000 fd:00 2883714 /lib64/libfreebl3.so
3715c02000-3715c03000 rw-p 00002000 fd:00 2883714 /lib64/libfreebl3.so
3715e00000-3715e07000 r-xp 00000000 fd:00 2883798 /lib64/libcrypt-2.12.so
3715e07000-3716007000 ---p 00007000 fd:00 2883798 /lib64/libcrypt-2.12.so
3716007000-3716008000 r--p 00007000 fd:00 2883798 /lib64/libcrypt-2.12.so
3716008000-3716009000 rw-p 00008000 fd:00 2883798 /lib64/libcrypt-2.12.so
3716009000-3716037000 rw-p 00000000 00:00 0
3716200000-3716225000 r-xp 00000000 fd:00 17303337 /usr/lib64/libnssutil3.so
3716225000-3716425000 ---p 00025000 fd:00 17303337 /usr/lib64/libnssutil3.so
3716425000-371642b000 r--p 00025000 fd:00 17303337 /usr/lib64/libnssutil3.so
371642b000-371642c000 rw-p 0002b000 fd:00 17303337 /usr/lib64/libnssutil3.so
3716600000-37167b9000 r-xp 00000000 fd:00 17305030 /usr/lib64/libcrypto.so.1.0.1e
37167b9000-37169b8000 ---p 001b9000 fd:00 17305030 /usr/lib64/libcrypto.so.1.0.1e
37169b8000-37169d3000 r--p 001b8000 fd:00 17305030 /usr/lib64/libcrypto.so.1.0.1e
37169d3000-37169df000 rw-p 001d3000 fd:00 17305030 /usr/lib64/libcrypto.so.1.0.1e
37169df000-37169e3000 rw-p 00000000 00:00 0
3717600000-3717641000 r-xp 00000000 fd:00 2883970 /lib64/libgssapi_krb5.so.2.2
3717641000-3717841000 ---p 00041000 fd:00 2883970 /lib64/libgssapi_krb5.so.2.2
3717841000-3717842000 r--p 00041000 fd:00 2883970 /lib64/libgssapi_krb5.so.2.2
3717842000-3717844000 rw-p 00042000 fd:00 2883970 /lib64/libgssapi_krb5.so.2.2
3717a00000-3717adb000 r-xp 00000000 fd:00 2883969 /lib64/libkrb5.so.3.3
3717adb000-3717cda000 ---p 000db000 fd:00 2883969 /lib64/libkrb5.so.3.3
3717cda000-3717ce4000 r--p 000da000 fd:00 2883969 /lib64/libkrb5.so.3.3
3717ce4000-3717ce6000 rw-p 000e4000 fd:00 2883969 /lib64/libkrb5.so.3.3
3717e00000-3717e29000 r-xp 00000000 fd:00 2883968 /lib64/libk5crypto.so.3.1
3717e29000-3718029000 ---p 00029000 fd:00 2883968 /lib64/libk5crypto.so.3.1
3718029000-371802a000 r--p 00029000 fd:00 2883968 /lib64/libk5crypto.so.3.1
371802a000-371802b000 rw-p 0002a000 fd:00 2883968 /lib64/libk5crypto.so.3.1
371802b000-371802c000 rw-p 00000000 00:00 0
3718200000-371820a000 r-xp 00000000 fd:00 2883967 /lib64/libkrb5support.so.0.1
371820a000-3718409000 ---p 0000a000 fd:00 2883967 /lib64/libkrb5support.so.0.1
3718409000-371840a000 r--p 00009000 fd:00 2883967 /lib64/libkrb5support.so.0.1
371840a000-371840b000 rw-p 0000a000 fd:00 2883967 /lib64/libkrb5support.so.0.1
3718600000-3718749000 r-xp 00000000 fd:00 17303500 /usr/lib64/libxml2.so.2.7.6
3718749000-3718948000 ---p 00149000 fd:00 17303500 /usr/lib64/libxml2.so.2.7.6
3718948000-3718951000 rw-p 00148000 fd:00 17303500 /usr/lib64/libxml2.so.2.7.6
3718951000-3718953000 rw-p 00000000 00:00 0
3718a00000-3718a62000 r-xp 00000000 fd:00 17305217 /usr/lib64/libssl.so.1.0.1e
3718a62000-3718c61000 ---p 00062000 fd:00 17305217 /usr/lib64/libssl.so.1.0.1e
3718c61000-3718c65000 r--p 00061000 fd:00 17305217 /usr/lib64/libssl.so.1.0.1e
3718c65000-3718c6c000 rw-p 00065000 fd:00 17305217 /usr/lib64/libssl.so.1.0.1e
3d2a800000-3d2a937000 r-xp 00000000 fd:00 17305255 /usr/lib64/libnss3.so
3d2a937000-3d2ab36000 ---p 00137000 fd:00 17305255 /usr/lib64/libnss3.so
3d2ab36000-3d2ab3b000 r--p 00136000 fd:00 17305255 /usr/lib64/libnss3.so
3d2ab3b000-3d2ab3d000 rw-p 0013b000 fd:00 17305255 /usr/lib64/libnss3.so
3d2ab3d000-3d2ab3f000 rw-p 00000000 00:00 0
3d2ac00000-3d2ac3b000 r-xp 00000000 fd:00 17305256 /usr/lib64/libssl3.so
3d2ac3b000-3d2ae3a000 ---p 0003b000 fd:00 17305256 /usr/lib64/libssl3.so
3d2ae3a000-3d2ae3d000 r--p 0003a000 fd:00 17305256 /usr/lib64/libssl3.so
3d2ae3d000-3d2ae3e000 rw-p 0003d000 fd:00 17305256 /usr/lib64/libssl3.so
3d2ae3e000-3d2ae3f000 rw-p 00000000 00:00 0
3d2b000000-3d2b028000 r-xp 00000000 fd:00 17305257 /usr/lib64/libsmime3.so
3d2b028000-3d2b227000 ---p 00028000 fd:00 17305257 /usr/lib64/libsmime3.so
3d2b227000-3d2b22b000 r--p 00027000 fd:00 17305257 /usr/lib64/libsmime3.so
3d2b22b000-3d2b22c000 rw-p 0002b000 fd:00 17305257 /usr/lib64/libsmime3.so
3d2b400000-3d2b40e000 r-xp 00000000 fd:00 2883664 /lib64/liblber-2.4.so.2.10.2
3d2b40e000-3d2b60d000 ---p 0000e000 fd:00 2883664 /lib64/liblber-2.4.so.2.10.2
3d2b60d000-3d2b60e000 r--p 0000d000 fd:00 2883664 /lib64/liblber-2.4.so.2.10.2
3d2b60e000-3d2b60f000 rw-p 0000e000 fd:00 2883664 /lib64/liblber-2.4.so.2.10.2
3d2b800000-3d2b84d000 r-xp 00000000 fd:00 2883814 /lib64/libldap-2.4.so.2.10.2
3d2b84d000-3d2ba4c000 ---p 0004d000 fd:00 2883814 /lib64/libldap-2.4.so.2.10.2
3d2ba4c000-3d2ba4e000 r--p 0004c000 fd:00 2883814 /lib64/libldap-2.4.so.2.10.2
3d2ba4e000-3d2ba50000 rw-p 0004e000 fd:00 2883814 /lib64/libldap-2.4.so.2.10.2
7f3101fdd000-7f3102427000 rw-p 00000000 00:00 0
7f3102d63000-7f3102d79000 r-xp 00000000 fd:00 2883652 /lib64/libgcc_s-4.4.7-20120601.so.1
7f3102d79000-7f3102f78000 ---p 00016000 fd:00 2883652 /lib64/libgcc_s-4.4.7-20120601.so.1
7f3102f78000-7f3102f79000 rw-p 00015000 fd:00 2883652 /lib64/libgcc_s-4.4.7-20120601.so.1
7f3102f79000-7f310397b000 rw-p 00000000 00:00 0
7f310397b000-7f3103afc000 rw-p 00000000 00:00 0
7f3103afc000-7f3103b7d000 rw-p 00000000 00:00 0
7f3103b7d000-7f3103c7e000 rw-p 00000000 00:00 0
7f3103d80000-7f3104202000 rw-p 00000000 00:00 0
7f3104263000-7f31044a5000 rw-p 00000000 00:00 0
7f31044b5000-7f31048dc000 rw-p 00000000 00:00 0
7f31048dc000-7f31048e8000 r-xp 00000000 fd:00 2883613 /lib64/libnss_files-2.12.so
7f31048e8000-7f3104ae8000 ---p 0000c000 fd:00 2883613 /lib64/libnss_files-2.12.so
7f3104ae8000-7f3104ae9000 r--p 0000c000 fd:00 2883613 /lib64/libnss_files-2.12.so
7f3104ae9000-7f3104aea000 rw-p 0000d000 fd:00 2883613 /lib64/libnss_files-2.12.so
7f3104af2000-7f3520056000 rw-s 00000000 00:04 2427386 /dev/zero (deleted)
7f3520056000-7f352005e000 r-xp 00000000 fd:00 18221940 /usr/pgsql-9.6/lib/pg_stat_statements.so
7f352005e000-7f352025d000 ---p 00008000 fd:00 18221940 /usr/pgsql-9.6/lib/pg_stat_statements.so
7f352025d000-7f352025e000 rw-p 00007000 fd:00 18221940 /usr/pgsql-9.6/lib/pg_stat_statements.so
7f352025e000-7f35260ee000 r--p 00000000 fd:00 17305075 /usr/lib/locale/locale-archive
7f35260ee000-7f35260f6000 rw-p 00000000 00:00 0
7f35260f6000-7f35260fa000 r-xp 00000000 fd:00 2883957 /lib64/libplc4.so
7f35260fa000-7f35262f9000 ---p 00004000 fd:00 2883957 /lib64/libplc4.so
7f35262f9000-7f35262fa000 r--p 00003000 fd:00 2883957 /lib64/libplc4.so
7f35262fa000-7f35262fb000 rw-p 00004000 fd:00 2883957 /lib64/libplc4.so
7f35262fb000-7f3526300000 rw-p 00000000 00:00 0
7f3526300000-7f3526317000 r-xp 00000000 fd:00 2883712 /lib64/libaudit.so.1.0.0
7f3526317000-7f3526517000 ---p 00017000 fd:00 2883712 /lib64/libaudit.so.1.0.0
7f3526517000-7f3526518000 r--p 00017000 fd:00 2883712 /lib64/libaudit.so.1.0.0
7f3526518000-7f3526523000 rw-p 00018000 fd:00 2883712 /lib64/libaudit.so.1.0.0
7f3526523000-7f3526527000 rw-p 00000000 00:00 0
7f352652a000-7f352652b000 rw-p 00000000 00:00 0
7f352652b000-7f352652e000 rw-s 00000000 00:10 2427390 /dev/shm/PostgreSQL.851836827
7f352652e000-7f352652f000 rw-s 00000000 00:04 3211264 /SYSV0052e2c1 (deleted)
7f352652f000-7f3526530000 rw-p 00000000 00:00 0
7fff9af75000-7fff9af8a000 rw-p 00000000 00:00 0 [stack]
7fff9afff000-7fff9b000000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Message from cvs file with query
2017-06-25 11:55:02.565 GMT,,,86053,,594a6c83.15025,3,,2017-06-21 12:54:27 GMT,,0,LOG,00000,"server process (PID 113514) was terminated by signal 6: Aborted","Failed process was running:
Incident 3 (Jun 29 18:55)
——————
-rw------- 1 postgres postgres 17631 Jun 29 18:55 postgresql-Thu-18.log
$ cat postgresql-Thu-18.log
*** glibc detected *** postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT: double free or corruption (!prev): 0x0000000002210f70 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3711a75e66]
/lib64/libc.so.6[0x3711a789b3]
postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT(ExecHashJoin+0x323)[0x5e0693]
postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT(ExecProcNode+0x208)[0x5cd368]
postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT(ExecSort+0xf9)[0x5ea9e9]
postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT(ExecProcNode+0x1e8)[0x5cd348]
postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT[0x5f00ec]
postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT(ExecWindowAgg+0x639)[0x5f1359]
postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT(ExecProcNode+0x278)[0x5cd3d8]
postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT[0x5eb34c]
postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT(ExecScan+0x187)[0x5d4697]
postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT(ExecProcNode+0xe8)[0x5cd248]
postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT(ExecNestLoop+0xb0)[0x5e8070]
postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT(ExecProcNode+0x228)[0x5cd388]
postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT(standard_ExecutorRun+0x17a)[0x5cae5a]
/usr/pgsql-9.6/lib/pg_stat_statements.so(+0x44fb)[0x7f352005a4fb]
postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT[0x6e1567]
postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT(PortalRun+0x188)[0x6e2738]
postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT[0x6df461]
postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT(PostgresMain+0xdcc)[0x6e0b2c]
postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT(PostmasterMain+0x1865)[0x680d55]
postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT(main+0x7a8)[0x608dc8]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3711a1ed5d]
postgres: app_feefee_monitor feedlogs 10.1.11.11(53362) SELECT[0x46be39]
======= Memory map: ========
00400000-00a13000 r-xp 00000000 fd:00 18221735 /usr/pgsql-9.6/bin/postgres
00c13000-00c20000 rw-p 00613000 fd:00 18221735 /usr/pgsql-9.6/bin/postgres
00c20000-00c71000 rw-p 00000000 00:00 0
01cd4000-01d5a000 rw-p 00000000 00:00 0
01d5a000-02350000 rw-p 00000000 00:00 0
3711600000-3711620000 r-xp 00000000 fd:00 2883590 /lib64/ld-2.12.so
371181f000-3711820000 r--p 0001f000 fd:00 2883590 /lib64/ld-2.12.so
3711820000-3711821000 rw-p 00020000 fd:00 2883590 /lib64/ld-2.12.so
3711821000-3711822000 rw-p 00000000 00:00 0
3711a00000-3711b8a000 r-xp 00000000 fd:00 2883612 /lib64/libc-2.12.so
3711b8a000-3711d8a000 ---p 0018a000 fd:00 2883612 /lib64/libc-2.12.so
3711d8a000-3711d8e000 r--p 0018a000 fd:00 2883612 /lib64/libc-2.12.so
3711d8e000-3711d8f000 rw-p 0018e000 fd:00 2883612 /lib64/libc-2.12.so
3711d8f000-3711d94000 rw-p 00000000 00:00 0
3711e00000-3711e17000 r-xp 00000000 fd:00 2883660 /lib64/libpthread-2.12.so
3711e17000-3712017000 ---p 00017000 fd:00 2883660 /lib64/libpthread-2.12.so
3712017000-3712018000 r--p 00017000 fd:00 2883660 /lib64/libpthread-2.12.so
3712018000-3712019000 rw-p 00018000 fd:00 2883660 /lib64/libpthread-2.12.so
3712019000-371201d000 rw-p 00000000 00:00 0
3712200000-3712202000 r-xp 00000000 fd:00 2883634 /lib64/libdl-2.12.so
3712202000-3712402000 ---p 00002000 fd:00 2883634 /lib64/libdl-2.12.so
3712402000-3712403000 r--p 00002000 fd:00 2883634 /lib64/libdl-2.12.so
3712403000-3712404000 rw-p 00003000 fd:00 2883634 /lib64/libdl-2.12.so
3712600000-3712607000 r-xp 00000000 fd:00 2883661 /lib64/librt-2.12.so
3712607000-3712806000 ---p 00007000 fd:00 2883661 /lib64/librt-2.12.so
3712806000-3712807000 r--p 00006000 fd:00 2883661 /lib64/librt-2.12.so
3712807000-3712808000 rw-p 00007000 fd:00 2883661 /lib64/librt-2.12.so
3712a00000-3712a83000 r-xp 00000000 fd:00 2883948 /lib64/libm-2.12.so
3712a83000-3712c82000 ---p 00083000 fd:00 2883948 /lib64/libm-2.12.so
3712c82000-3712c83000 r--p 00082000 fd:00 2883948 /lib64/libm-2.12.so
3712c83000-3712c84000 rw-p 00083000 fd:00 2883948 /lib64/libm-2.12.so
3712e00000-3712e15000 r-xp 00000000 fd:00 2883942 /lib64/libz.so.1.2.3
3712e15000-3713014000 ---p 00015000 fd:00 2883942 /lib64/libz.so.1.2.3
3713014000-3713015000 r--p 00014000 fd:00 2883942 /lib64/libz.so.1.2.3
3713015000-3713016000 rw-p 00015000 fd:00 2883942 /lib64/libz.so.1.2.3
3713200000-371321d000 r-xp 00000000 fd:00 2883949 /lib64/libselinux.so.1
371321d000-371341c000 ---p 0001d000 fd:00 2883949 /lib64/libselinux.so.1
371341c000-371341d000 r--p 0001c000 fd:00 2883949 /lib64/libselinux.so.1
371341d000-371341e000 rw-p 0001d000 fd:00 2883949 /lib64/libselinux.so.1
371341e000-371341f000 rw-p 00000000 00:00 0
3713a00000-3713a02000 r-xp 00000000 fd:00 2883966 /lib64/libkeyutils.so.1.3
3713a02000-3713c01000 ---p 00002000 fd:00 2883966 /lib64/libkeyutils.so.1.3
3713c01000-3713c02000 r--p 00001000 fd:00 2883966 /lib64/libkeyutils.so.1.3
3713c02000-3713c03000 rw-p 00002000 fd:00 2883966 /lib64/libkeyutils.so.1.3
3713e00000-3713e16000 r-xp 00000000 fd:00 2883959 /lib64/libresolv-2.12.so
3713e16000-3714016000 ---p 00016000 fd:00 2883959 /lib64/libresolv-2.12.so
3714016000-3714017000 r--p 00016000 fd:00 2883959 /lib64/libresolv-2.12.so
3714017000-3714018000 rw-p 00017000 fd:00 2883959 /lib64/libresolv-2.12.so
3714018000-371401a000 rw-p 00000000 00:00 0
3714200000-3714219000 r-xp 00000000 fd:00 17303744 /usr/lib64/libsasl2.so.2.0.23
3714219000-3714418000 ---p 00019000 fd:00 17303744 /usr/lib64/libsasl2.so.2.0.23
3714418000-3714419000 r--p 00018000 fd:00 17303744 /usr/lib64/libsasl2.so.2.0.23
3714419000-371441a000 rw-p 00019000 fd:00 17303744 /usr/lib64/libsasl2.so.2.0.23
3714600000-371460c000 r-xp 00000000 fd:00 2883803 /lib64/libpam.so.0.82.2
371460c000-371480c000 ---p 0000c000 fd:00 2883803 /lib64/libpam.so.0.82.2
371480c000-371480d000 r--p 0000c000 fd:00 2883803 /lib64/libpam.so.0.82.2
371480d000-371480e000 rw-p 0000d000 fd:00 2883803 /lib64/libpam.so.0.82.2
3714e00000-3714e03000 r-xp 00000000 fd:00 2883958 /lib64/libplds4.so
3714e03000-3715002000 ---p 00003000 fd:00 2883958 /lib64/libplds4.so
3715002000-3715003000 r--p 00002000 fd:00 2883958 /lib64/libplds4.so
3715003000-3715004000 rw-p 00003000 fd:00 2883958 /lib64/libplds4.so
3715200000-3715239000 r-xp 00000000 fd:00 2883956 /lib64/libnspr4.so
3715239000-3715439000 ---p 00039000 fd:00 2883956 /lib64/libnspr4.so
3715439000-371543a000 r--p 00039000 fd:00 2883956 /lib64/libnspr4.so
371543a000-371543c000 rw-p 0003a000 fd:00 2883956 /lib64/libnspr4.so
371543c000-371543e000 rw-p 00000000 00:00 0
3715600000-3715603000 r-xp 00000000 fd:00 2883720 /lib64/libcom_err.so.2.1
3715603000-3715802000 ---p 00003000 fd:00 2883720 /lib64/libcom_err.so.2.1
3715802000-3715803000 r--p 00002000 fd:00 2883720 /lib64/libcom_err.so.2.1
3715803000-3715804000 rw-p 00003000 fd:00 2883720 /lib64/libcom_err.so.2.1
3715a00000-3715a02000 r-xp 00000000 fd:00 2883714 /lib64/libfreebl3.so
3715a02000-3715c01000 ---p 00002000 fd:00 2883714 /lib64/libfreebl3.so
3715c01000-3715c02000 r--p 00001000 fd:00 2883714 /lib64/libfreebl3.so
3715c02000-3715c03000 rw-p 00002000 fd:00 2883714 /lib64/libfreebl3.so
3715e00000-3715e07000 r-xp 00000000 fd:00 2883798 /lib64/libcrypt-2.12.so
3715e07000-3716007000 ---p 00007000 fd:00 2883798 /lib64/libcrypt-2.12.so
3716007000-3716008000 r--p 00007000 fd:00 2883798 /lib64/libcrypt-2.12.so
3716008000-3716009000 rw-p 00008000 fd:00 2883798 /lib64/libcrypt-2.12.so
3716009000-3716037000 rw-p 00000000 00:00 0
3716200000-3716225000 r-xp 00000000 fd:00 17303337 /usr/lib64/libnssutil3.so
3716225000-3716425000 ---p 00025000 fd:00 17303337 /usr/lib64/libnssutil3.so
3716425000-371642b000 r--p 00025000 fd:00 17303337 /usr/lib64/libnssutil3.so
371642b000-371642c000 rw-p 0002b000 fd:00 17303337 /usr/lib64/libnssutil3.so
3716600000-37167b9000 r-xp 00000000 fd:00 17305030 /usr/lib64/libcrypto.so.1.0.1e
37167b9000-37169b8000 ---p 001b9000 fd:00 17305030 /usr/lib64/libcrypto.so.1.0.1e
37169b8000-37169d3000 r--p 001b8000 fd:00 17305030 /usr/lib64/libcrypto.so.1.0.1e
37169d3000-37169df000 rw-p 001d3000 fd:00 17305030 /usr/lib64/libcrypto.so.1.0.1e
37169df000-37169e3000 rw-p 00000000 00:00 0
3717600000-3717641000 r-xp 00000000 fd:00 2883970 /lib64/libgssapi_krb5.so.2.2
3717641000-3717841000 ---p 00041000 fd:00 2883970 /lib64/libgssapi_krb5.so.2.2
3717841000-3717842000 r--p 00041000 fd:00 2883970 /lib64/libgssapi_krb5.so.2.2
3717842000-3717844000 rw-p 00042000 fd:00 2883970 /lib64/libgssapi_krb5.so.2.2
3717a00000-3717adb000 r-xp 00000000 fd:00 2883969 /lib64/libkrb5.so.3.3
3717adb000-3717cda000 ---p 000db000 fd:00 2883969 /lib64/libkrb5.so.3.3
3717cda000-3717ce4000 r--p 000da000 fd:00 2883969 /lib64/libkrb5.so.3.3
3717ce4000-3717ce6000 rw-p 000e4000 fd:00 2883969 /lib64/libkrb5.so.3.3
3717e00000-3717e29000 r-xp 00000000 fd:00 2883968 /lib64/libk5crypto.so.3.1
3717e29000-3718029000 ---p 00029000 fd:00 2883968 /lib64/libk5crypto.so.3.1
3718029000-371802a000 r--p 00029000 fd:00 2883968 /lib64/libk5crypto.so.3.1
371802a000-371802b000 rw-p 0002a000 fd:00 2883968 /lib64/libk5crypto.so.3.1
371802b000-371802c000 rw-p 00000000 00:00 0
3718200000-371820a000 r-xp 00000000 fd:00 2883967 /lib64/libkrb5support.so.0.1
371820a000-3718409000 ---p 0000a000 fd:00 2883967 /lib64/libkrb5support.so.0.1
3718409000-371840a000 r--p 00009000 fd:00 2883967 /lib64/libkrb5support.so.0.1
371840a000-371840b000 rw-p 0000a000 fd:00 2883967 /lib64/libkrb5support.so.0.1
3718600000-3718749000 r-xp 00000000 fd:00 17303500 /usr/lib64/libxml2.so.2.7.6
3718749000-3718948000 ---p 00149000 fd:00 17303500 /usr/lib64/libxml2.so.2.7.6
3718948000-3718951000 rw-p 00148000 fd:00 17303500 /usr/lib64/libxml2.so.2.7.6
3718951000-3718953000 rw-p 00000000 00:00 0
3718a00000-3718a62000 r-xp 00000000 fd:00 17305217 /usr/lib64/libssl.so.1.0.1e
3718a62000-3718c61000 ---p 00062000 fd:00 17305217 /usr/lib64/libssl.so.1.0.1e
3718c61000-3718c65000 r--p 00061000 fd:00 17305217 /usr/lib64/libssl.so.1.0.1e
3718c65000-3718c6c000 rw-p 00065000 fd:00 17305217 /usr/lib64/libssl.so.1.0.1e
3d2a800000-3d2a937000 r-xp 00000000 fd:00 17305255 /usr/lib64/libnss3.so
3d2a937000-3d2ab36000 ---p 00137000 fd:00 17305255 /usr/lib64/libnss3.so
3d2ab36000-3d2ab3b000 r--p 00136000 fd:00 17305255 /usr/lib64/libnss3.so
3d2ab3b000-3d2ab3d000 rw-p 0013b000 fd:00 17305255 /usr/lib64/libnss3.so
3d2ab3d000-3d2ab3f000 rw-p 00000000 00:00 0
3d2ac00000-3d2ac3b000 r-xp 00000000 fd:00 17305256 /usr/lib64/libssl3.so
3d2ac3b000-3d2ae3a000 ---p 0003b000 fd:00 17305256 /usr/lib64/libssl3.so
3d2ae3a000-3d2ae3d000 r--p 0003a000 fd:00 17305256 /usr/lib64/libssl3.so
3d2ae3d000-3d2ae3e000 rw-p 0003d000 fd:00 17305256 /usr/lib64/libssl3.so
3d2ae3e000-3d2ae3f000 rw-p 00000000 00:00 0
3d2b000000-3d2b028000 r-xp 00000000 fd:00 17305257 /usr/lib64/libsmime3.so
3d2b028000-3d2b227000 ---p 00028000 fd:00 17305257 /usr/lib64/libsmime3.so
3d2b227000-3d2b22b000 r--p 00027000 fd:00 17305257 /usr/lib64/libsmime3.so
3d2b22b000-3d2b22c000 rw-p 0002b000 fd:00 17305257 /usr/lib64/libsmime3.so
3d2b400000-3d2b40e000 r-xp 00000000 fd:00 2883664 /lib64/liblber-2.4.so.2.10.2
3d2b40e000-3d2b60d000 ---p 0000e000 fd:00 2883664 /lib64/liblber-2.4.so.2.10.2
3d2b60d000-3d2b60e000 r--p 0000d000 fd:00 2883664 /lib64/liblber-2.4.so.2.10.2
3d2b60e000-3d2b60f000 rw-p 0000e000 fd:00 2883664 /lib64/liblber-2.4.so.2.10.2
3d2b800000-3d2b84d000 r-xp 00000000 fd:00 2883814 /lib64/libldap-2.4.so.2.10.2
3d2b84d000-3d2ba4c000 ---p 0004d000 fd:00 2883814 /lib64/libldap-2.4.so.2.10.2
3d2ba4c000-3d2ba4e000 r--p 0004c000 fd:00 2883814 /lib64/libldap-2.4.so.2.10.2
3d2ba4e000-3d2ba50000 rw-p 0004e000 fd:00 2883814 /lib64/libldap-2.4.so.2.10.2
7f310306a000-7f3103080000 r-xp 00000000 fd:00 2883652 /lib64/libgcc_s-4.4.7-20120601.so.1
7f3103080000-7f310327f000 ---p 00016000 fd:00 2883652 /lib64/libgcc_s-4.4.7-20120601.so.1
7f310327f000-7f3103280000 rw-p 00015000 fd:00 2883652 /lib64/libgcc_s-4.4.7-20120601.so.1
7f3103280000-7f3103a81000 rw-p 00000000 00:00 0
7f3103a81000-7f3103c42000 rw-p 00000000 00:00 0
7f3103c42000-7f3104043000 rw-p 00000000 00:00 0
7f31041c4000-7f31043c5000 rw-p 00000000 00:00 0
7f3104486000-7f3104587000 rw-p 00000000 00:00 0
7f31045e8000-7f3104669000 rw-p 00000000 00:00 0
7f310469a000-7f31046fc000 rw-p 00000000 00:00 0
7f31047fd000-7f31048dc000 rw-p 00000000 00:00 0
7f31048dc000-7f31048e8000 r-xp 00000000 fd:00 2883613 /lib64/libnss_files-2.12.so
7f31048e8000-7f3104ae8000 ---p 0000c000 fd:00 2883613 /lib64/libnss_files-2.12.so
7f3104ae8000-7f3104ae9000 r--p 0000c000 fd:00 2883613 /lib64/libnss_files-2.12.so
7f3104ae9000-7f3104aea000 rw-p 0000d000 fd:00 2883613 /lib64/libnss_files-2.12.so
7f3104af2000-7f3520056000 rw-s 00000000 00:04 4312619 /dev/zero (deleted)
7f3520056000-7f352005e000 r-xp 00000000 fd:00 18221940 /usr/pgsql-9.6/lib/pg_stat_statements.so
7f352005e000-7f352025d000 ---p 00008000 fd:00 18221940 /usr/pgsql-9.6/lib/pg_stat_statements.so
7f352025d000-7f352025e000 rw-p 00007000 fd:00 18221940 /usr/pgsql-9.6/lib/pg_stat_statements.so
7f352025e000-7f35260ee000 r--p 00000000 fd:00 17305075 /usr/lib/locale/locale-archive
7f35260ee000-7f35260f6000 rw-p 00000000 00:00 0
7f35260f6000-7f35260fa000 r-xp 00000000 fd:00 2883957 /lib64/libplc4.so
7f35260fa000-7f35262f9000 ---p 00004000 fd:00 2883957 /lib64/libplc4.so
7f35262f9000-7f35262fa000 r--p 00003000 fd:00 2883957 /lib64/libplc4.so
7f35262fa000-7f35262fb000 rw-p 00004000 fd:00 2883957 /lib64/libplc4.so
7f35262fb000-7f3526300000 rw-p 00000000 00:00 0
7f3526300000-7f3526317000 r-xp 00000000 fd:00 2883712 /lib64/libaudit.so.1.0.0
7f3526317000-7f3526517000 ---p 00017000 fd:00 2883712 /lib64/libaudit.so.1.0.0
7f3526517000-7f3526518000 r--p 00017000 fd:00 2883712 /lib64/libaudit.so.1.0.0
7f3526518000-7f3526523000 rw-p 00018000 fd:00 2883712 /lib64/libaudit.so.1.0.0
7f3526523000-7f3526527000 rw-p 00000000 00:00 0
7f352652a000-7f352652b000 rw-p 00000000 00:00 0
7f352652b000-7f352652e000 rw-s 00000000 00:10 4312623 /dev/shm/PostgreSQL.658076755
7f352652e000-7f352652f000 rw-s 00000000 00:04 3244032 /SYSV0052e2c1 (deleted)
7f352652f000-7f3526530000 rw-p 00000000 00:00 0
7fff9af75000-7fff9af8a000 rw-p 00000000 00:00 0 [stack]
7fff9afff000-7fff9b000000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Corresponding message from csv file
2017-06-29 18:55:03.346 GMT,,,86053,,594a6c83.15025,13,,2017-06-21 12:54:27 GMT,,0,LOG,00000,"server process (PID 92659) was terminated by signal 6: Aborted","Failed process was running: WITH ranked AS (
pgsql-bugs by date: