Thread: [BUGS] BUG #14683: *** glibc detected *** SELECT: double free or corruption

[BUGS] BUG #14683: *** glibc detected *** SELECT: double free or corruption

From
ray.warren@artsalliancemedia.com
Date:
The following bug has been logged on the website:

Bug reference:      14683
Logged by:          Ray Warren
Email address:      ray.warren@artsalliancemedia.com
PostgreSQL version: 9.5.7
Operating system:   CentOS release 6.9
Description:

We are experiencing some performance issues and we had this error on Sunday.
Not sure whether the 2 things are related or not as we have been making some
changes to the application and load. However, I assume the double free or
corruption error is a bug?

2017-05-28 17:54:53 BST [14898]: [3-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.110 LOG:  duration:
13.117 ms  statement: UPDATE monitoring_log SET timestamp=1495990491.0,
value='0', seq_id=nextval('monitoring_seq_num') WHERE
monitoring_log.device_uuid = '135d016b-7a0b-43dc-b6aa-cd0a812dc533' AND
monitoring_log.monitoring_type_uuid =
'01bdc171-9869-5d51-bce7-3117c4f3dde4'
2017-05-28 17:54:53 BST [14749]: [11-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.63 LOG:  temporary file:
path "base/pgsql_tmp/pgsql_tmp14749.3", size 30453160
2017-05-28 17:54:53 BST [14749]: [12-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.63 STATEMENT:  SELECT
cpl_metadata.uuid AS cpl_metadata_uuid, cpl_metadata.created AS
cpl_metadata_created, cpl_metadata.last_modified AS cpl_met
adata_last_modified, cpl_metadata.cpl_uuid AS cpl_metadata_cpl_uuid,
cpl_metadata.name AS cpl_metadata_name, cpl_metadata.date AS
cpl_metadata_date, cpl_metadata.content_kind AS cpl_metadata_content_kind,
cpl_metadata.credits_offset AS c
pl_metadata_credits_offset, cpl_metadata.intermission_offset AS
cpl_metadata_intermission_offset, cpl_metadata.playback_mode AS
cpl_metadata_playback_mode, cpl_metadata.rating_hardlocked AS
cpl_metadata_rating_hardlocked, cpl_metadata.te
rritory AS cpl_metadata_territory, cpl_metadata.rating AS
cpl_metadata_rating, cpl_metadata.audio_type AS cpl_metadata_audio_type,
cpl_metadata.audio_language AS cpl_metadata_audio_language,
cpl_metadata.subtitled AS cpl_metadata_subtitl
ed, cpl_metadata.subtitle_language AS cpl_metadata_subtitle_language,
cpl_metadata.accessibility AS cpl_metadata_accessibility,
cpl_metadata.aspect_ratio AS cpl_metadata_aspect_ratio,
cpl_metadata.resolution AS cpl_metadata_resolution, c
pl_metadata.studio AS cpl_metadata_studio, cpl_metadata.facility AS
cpl_metadata_facility, cpl_metadata.motion_simulator_format AS
cpl_metadata_motion_simulator_format, cpl_metadata.ghostbusted AS
cpl_metadata_ghostbusted, cpl_metadata.s
tate AS cpl_metadata_state, cpl_metadata.is_master AS
cpl_metadata_is_master, cpl_metadata.is_edited AS cpl_metadata_is_edited,
cpl_metadata.next_cpl_uuid AS cpl_metadata_next_cpl_uuid,
cpl_metadata.seq_id AS cpl_metadata_seq_id, cpl.uui
d AS cpl_uuid, cpl.created AS cpl_created, cpl.last_modified AS
cpl_last_modified, cpl.content_title_text AS cpl_content_title_text,
cpl.content_title AS cpl_content_title, cpl.is_encrypted AS
cpl_is_encrypted, cpl.duration_in_frames AS
cpl_duration_in_frames, cpl.duration_in_seconds AS cpl_duration_in_seconds,
cpl.edit_rate_a AS cpl_edit_rate_a, cpl.edit_rate_b AS cpl_edit_rate_b,
cpl.package_type AS cpl_package_type, cpl.physically_exists AS
cpl_physically_exists, cpl
.available_on_courier AS cpl_available_on_courier, cpl.courier_uploaders AS
cpl_courier_uploaders, cpl.seq_id AS cpl_seq_id, greatest(cpl.seq_id,
cpl_metadata.seq_id) AS combined_seq_id       FROM cpl_metadata JOIN cpl ON cpl_metadata.cpl_uuid = cpl.uuid
WHEREcpl_metadata.is_master = true AND greatest(cpl.seq_id, 
cpl_metadata.seq_id) >= 14321152 ORDER BY combined_seq_id        LIMIT 1000
*** glibc detected *** postgres: postgres etms 172.30.8.63(42412) SELECT:
double free or corruption (!prev): 0x00000000033c5770 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75dee)[0x7f1d6ba7edee]
/lib64/libc.so.6(+0x78c80)[0x7f1d6ba81c80]
postgres: postgres etms 172.30.8.63(42412)
SELECT(ExecHashJoin+0x31b)[0x5ce60b]
postgres: postgres etms 172.30.8.63(42412)
SELECT(ExecProcNode+0x168)[0x5bc9f8]
postgres: postgres etms 172.30.8.63(42412) SELECT(ExecSort+0xf9)[0x5d86c9]
postgres: postgres etms 172.30.8.63(42412)
SELECT(ExecProcNode+0x148)[0x5bc9d8]
postgres: postgres etms 172.30.8.63(42412)
SELECT(ExecLimit+0xa8)[0x5d13a8]
postgres: postgres etms 172.30.8.63(42412)
SELECT(ExecProcNode+0x90)[0x5bc920]
postgres: postgres etms 172.30.8.63(42412)
SELECT(standard_ExecutorRun+0xd2)[0x5bb2c2]
/usr/pgsql-9.5/lib/pg_stat_statements.so(+0x448b)[0x7f1d62b2848b]
postgres: postgres etms 172.30.8.63(42412) SELECT[0x6bd417]
postgres: postgres etms 172.30.8.63(42412)
SELECT(PortalRun+0x188)[0x6be5d8]
postgres: postgres etms 172.30.8.63(42412) SELECT[0x6bb3be]
postgres: postgres etms 172.30.8.63(42412)
SELECT(PostgresMain+0xd35)[0x6bca25]
postgres: postgres etms 172.30.8.63(42412)
SELECT(PostmasterMain+0x1875)[0x663165]
postgres: postgres etms 172.30.8.63(42412) SELECT(main+0x7a8)[0x5f4b28]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f1d6ba27d1d]
postgres: postgres etms 172.30.8.63(42412) SELECT[0x467549]
======= Memory map: ========
00400000-009cf000 r-xp 00000000 fd:01 788841
/usr/pgsql-9.5/bin/postgres
00bcf000-00bdc000 rw-p 005cf000 fd:01 788841
/usr/pgsql-9.5/bin/postgres
00bdc000-00c2b000 rw-p 00000000 00:00 0
0173e000-01785000 rw-p 00000000 00:00 0
01785000-035d5000 rw-p 00000000 00:00 0
7f1b0c000000-7f1b0c021000 rw-p 00000000 00:00 0
7f1b0c021000-7f1b10000000 ---p 00000000 00:00 0
7f1b12280000-7f1b12296000 r-xp 00000000 fd:01 1308180
/lib64/libgcc_s-4.4.7-20120601.so.1
7f1b12296000-7f1b12495000 ---p 00016000 fd:01 1308180
/lib64/libgcc_s-4.4.7-20120601.so.1
7f1b12495000-7f1b12496000 rw-p 00015000 fd:01 1308180
/lib64/libgcc_s-4.4.7-20120601.so.1
7f1b12496000-7f1b124a3000 r-xp 00000000 fd:01 1308307
/lib64/libnss_files-2.12.so
7f1b124a3000-7f1b126a2000 ---p 0000d000 fd:01 1308307
/lib64/libnss_files-2.12.so
7f1b126a2000-7f1b126a3000 r--p 0000c000 fd:01 1308307
/lib64/libnss_files-2.12.so
7f1b126a3000-7f1b126a4000 rw-p 0000d000 fd:01 1308307
/lib64/libnss_files-2.12.so
7f1b126a4000-7f1d62b24000 rw-s 00000000 00:04 440003533
/dev/zero (deleted)
7f1d62b24000-7f1d62b2c000 r-xp 00000000 fd:01 659776
/usr/pgsql-9.5/lib/pg_stat_statements.so
7f1d62b2c000-7f1d62d2b000 ---p 00008000 fd:01 659776
/usr/pgsql-9.5/lib/pg_stat_statements.so
7f1d62d2b000-7f1d62d2c000 rw-p 00007000 fd:01 659776
/usr/pgsql-9.5/lib/pg_stat_statements.so
7f1d62d2c000-7f1d68bc1000 r--p 00000000 fd:01 659094
/usr/lib/locale/locale-archive
7f1d68bc1000-7f1d68bde000 r-xp 00000000 fd:01 1308211
/lib64/libselinux.so.1
7f1d68bde000-7f1d68ddd000 ---p 0001d000 fd:01 1308211
/lib64/libselinux.so.1
7f1d68ddd000-7f1d68dde000 r--p 0001c000 fd:01 1308211
/lib64/libselinux.so.1
7f1d68dde000-7f1d68ddf000 rw-p 0001d000 fd:01 1308211
/lib64/libselinux.so.1
7f1d68ddf000-7f1d68de0000 rw-p 00000000 00:00 0
7f1d68de0000-7f1d68de2000 r-xp 00000000 fd:01 1308204
/lib64/libfreebl3.so
7f1d68de2000-7f1d68fe1000 ---p 00002000 fd:01 1308204
/lib64/libfreebl3.so
7f1d68fe1000-7f1d68fe2000 r--p 00001000 fd:01 1308204
/lib64/libfreebl3.so
7f1d68fe2000-7f1d68fe3000 rw-p 00002000 fd:01 1308204
/lib64/libfreebl3.so
7f1d68fe3000-7f1d6901c000 r-xp 00000000 fd:01 1308378
/lib64/libnspr4.so
7f1d6901c000-7f1d6921c000 ---p 00039000 fd:01 1308378
/lib64/libnspr4.so
7f1d6921c000-7f1d6921d000 r--p 00039000 fd:01 1308378
/lib64/libnspr4.so
7f1d6921d000-7f1d6921f000 rw-p 0003a000 fd:01 1308378
/lib64/libnspr4.so
7f1d6921f000-7f1d69221000 rw-p 00000000 00:00 0
7f1d69221000-7f1d69225000 r-xp 00000000 fd:01 1308380
/lib64/libplc4.so
7f1d69225000-7f1d69424000 ---p 00004000 fd:01 1308380
/lib64/libplc4.so
7f1d69424000-7f1d69425000 r--p 00003000 fd:01 1308380
/lib64/libplc4.so
7f1d69425000-7f1d69426000 rw-p 00004000 fd:01 1308380
/lib64/libplc4.so
7f1d69426000-7f1d69429000 r-xp 00000000 fd:01 1308382
/lib64/libplds4.so
7f1d69429000-7f1d69628000 ---p 00003000 fd:01 1308382
/lib64/libplds4.so
7f1d69628000-7f1d69629000 r--p 00002000 fd:01 1308382
/lib64/libplds4.so
7f1d69629000-7f1d6962a000 rw-p 00003000 fd:01 1308382
/lib64/libplds4.so
7f1d6962a000-7f1d69650000 r-xp 00000000 fd:01 656260
/usr/lib64/libnssutil3.so
7f1d69650000-7f1d6984f000 ---p 00026000 fd:01 656260
/usr/lib64/libnssutil3.so
7f1d6984f000-7f1d69856000 r--p 00025000 fd:01 656260
/usr/lib64/libnssutil3.so
7f1d69856000-7f1d69857000 rw-p 0002c000 fd:01 656260
/usr/lib64/libnssutil3.so
7f1d69857000-7f1d69991000 r-xp 00000000 fd:01 656907
/usr/lib64/libnss3.so
7f1d69991000-7f1d69b90000 ---p 0013a000 fd:01 656907
/usr/lib64/libnss3.so
7f1d69b90000-7f1d69b96000 r--p 00139000 fd:01 656907
/usr/lib64/libnss3.so
7f1d69b96000-7f1d69b98000 rw-p 0013f000 fd:01 656907
/usr/lib64/libnss3.so
7f1d69b98000-7f1d69b9a000 rw-p 00000000 00:00 0
7f1d69b9a000-7f1d69bc2000 r-xp 00000000 fd:01 659119
/usr/lib64/libsmime3.so
7f1d69bc2000-7f1d69dc1000 ---p 00028000 fd:01 659119
/usr/lib64/libsmime3.so
7f1d69dc1000-7f1d69dc5000 r--p 00027000 fd:01 659119
/usr/lib64/libsmime3.so
7f1d69dc5000-7f1d69dc6000 rw-p 0002b000 fd:01 659119
/usr/lib64/libsmime3.so
7f1d69dc6000-7f1d69e0d000 r-xp 00000000 fd:01 659120
/usr/lib64/libssl3.so
7f1d69e0d000-7f1d6a00d000 ---p 00047000 fd:01 659120
/usr/lib64/libssl3.so
7f1d6a00d000-7f1d6a011000 r--p 00047000 fd:01 659120
/usr/lib64/libssl3.so
7f1d6a011000-7f1d6a012000 rw-p 0004b000 fd:01 659120
/usr/lib64/libssl3.so
7f1d6a012000-7f1d6a013000 rw-p 00000000 00:00 0
7f1d6a013000-7f1d6a02c000 r-xp 00000000 fd:01 655285
/usr/lib64/libsasl2.so.2.0.23
7f1d6a02c000-7f1d6a22b000 ---p 00019000 fd:01 655285
/usr/lib64/libsasl2.so.2.0.23
7f1d6a22b000-7f1d6a22c000 r--p 00018000 fd:01 655285
/usr/lib64/libsasl2.so.2.0.23
7f1d6a22c000-7f1d6a22d000 rw-p 00019000 fd:01 655285
/usr/lib64/libsasl2.so.2.0.23
7f1d6a22d000-7f1d6a23b000 r-xp 00000000 fd:01 1308168
/lib64/liblber-2.4.so.2.10.3
7f1d6a23b000-7f1d6a43a000 ---p 0000e000 fd:01 1308168
/lib64/liblber-2.4.so.2.10.3
7f1d6a43a000-7f1d6a43b000 r--p 0000d000 fd:01 1308168
/lib64/liblber-2.4.so.2.10.3
7f1d6a43b000-7f1d6a43c000 rw-p 0000e000 fd:01 1308168
/lib64/liblber-2.4.so.2.10.3
7f1d6a43c000-7f1d6a453000 r-xp 00000000 fd:01 1308197
/lib64/libpthread-2.12.so
7f1d6a453000-7f1d6a653000 ---p 00017000 fd:01 1308197
/lib64/libpthread-2.12.so
7f1d6a653000-7f1d6a654000 r--p 00017000 fd:01 1308197
/lib64/libpthread-2.12.so
7f1d6a654000-7f1d6a655000 rw-p 00018000 fd:01 1308197
/lib64/libpthread-2.12.so
7f1d6a655000-7f1d6a659000 rw-p 00000000 00:00 0
7f1d6a659000-7f1d6a66f000 r-xp 00000000 fd:01 1308318
/lib64/libresolv-2.12.so
7f1d6a66f000-7f1d6a86f000 ---p 00016000 fd:01 1308318
/lib64/libresolv-2.12.so
7f1d6a86f000-7f1d6a870000 r--p 00016000 fd:01 1308318
/lib64/libresolv-2.12.so
7f1d6a870000-7f1d6a871000 rw-p 00017000 fd:01 1308318
/lib64/libresolv-2.12.so
7f1d6a871000-7f1d6a873000 rw-p 00000000 00:00 0
7f1d6a873000-7f1d6a875000 r-xp 00000000 fd:01 1308340
/lib64/libkeyutils.so.1.3
7f1d6a875000-7f1d6aa74000 ---p 00002000 fd:01 1308340
/lib64/libkeyutils.so.1.3
7f1d6aa74000-7f1d6aa75000 r--p 00001000 fd:01 1308340
/lib64/libkeyutils.so.1.3
7f1d6aa75000-7f1d6aa76000 rw-p 00002000 fd:01 1308340
/lib64/libkeyutils.so.1.3
7f1d6aa76000-7f1d6aa80000 r-xp 00000000 fd:01 1308358
/lib64/libkrb5support.so.0.1
7f1d6aa80000-7f1d6ac7f000 ---p 0000a000 fd:01 1308358
/lib64/libkrb5support.so.0.1
7f1d6ac7f000-7f1d6ac80000 r--p 00009000 fd:01 1308358
/lib64/libkrb5support.so.0.1
7f1d6ac80000-7f1d6ac81000 rw-p 0000a000 fd:01 1308358
/lib64/libkrb5support.so.0.1
7f1d6ac81000-7f1d6acaa000 r-xp 00000000 fd:01 1308344
/lib64/libk5crypto.so.3.1
7f1d6acaa000-7f1d6aeaa000 ---p 00029000 fd:01 1308344
/lib64/libk5crypto.so.3.1
7f1d6aeaa000-7f1d6aeab000 r--p 00029000 fd:01 1308344
/lib64/libk5crypto.so.3.1
7f1d6aeab000-7f1d6aeac000 rw-p 0002a000 fd:01 1308344
/lib64/libk5crypto.so.3.1
7f1d6aeac000-7f1d6aead000 rw-p 00000000 00:00 0
7f1d6aead000-7f1d6aeb0000 r-xp 00000000 fd:01 1308233
/lib64/libcom_err.so.2.1
7f1d6aeb0000-7f1d6b0af000 ---p 00003000 fd:01 1308233
/lib64/libcom_err.so.2.1
7f1d6b0af000-7f1d6b0b0000 r--p 00002000 fd:01 1308233
/lib64/libcom_err.so.2.1
7f1d6b0b0000-7f1d6b0b1000 rw-p 00003000 fd:01 1308233
/lib64/libcom_err.so.2.1
7f1d6b0b1000-7f1d6b18d000 r-xp 00000000 fd:01 1308355
/lib64/libkrb5.so.3.3
7f1d6b18d000-7f1d6b38c000 ---p 000dc000 fd:01 1308355
/lib64/libkrb5.so.3.3
7f1d6b38c000-7f1d6b396000 r--p 000db000 fd:01 1308355
/lib64/libkrb5.so.3.3
7f1d6b396000-7f1d6b398000 rw-p 000e5000 fd:01 1308355
/lib64/libkrb5.so.3.3
7f1d6b398000-7f1d6b39f000 r-xp 00000000 fd:01 1308177
/lib64/libcrypt-2.12.so
7f1d6b39f000-7f1d6b59f000 ---p 00007000 fd:01 1308177
/lib64/libcrypt-2.12.so
7f1d6b59f000-7f1d6b5a0000 r--p 00007000 fd:01 1308177
/lib64/libcrypt-2.12.so
7f1d6b5a0000-7f1d6b5a1000 rw-p 00008000 fd:01 1308177
/lib64/libcrypt-2.12.so
7f1d6b5a1000-7f1d6b5cf000 rw-p 00000000 00:00 0
7f1d6b5cf000-7f1d6b5e7000 r-xp 00000000 fd:01 1308482
/lib64/libaudit.so.1.0.0
7f1d6b5e7000-7f1d6b7e6000 ---p 00018000 fd:01 1308482
/lib64/libaudit.so.1.0.0
7f1d6b7e6000-7f1d6b7e8000 r--p 00017000 fd:01 1308482
/lib64/libaudit.so.1.0.0
7f1d6b7e8000-7f1d6b7f3000 rw-p 00019000 fd:01 1308482
/lib64/libaudit.so.1.0.0
7f1d6b7f3000-7f1d6b808000 r-xp 00000000 fd:01 1308226
/lib64/libz.so.1.2.3
7f1d6b808000-7f1d6ba07000 ---p 00015000 fd:01 1308226
/lib64/libz.so.1.2.3
7f1d6ba07000-7f1d6ba08000 r--p 00014000 fd:01 1308226
/lib64/libz.so.1.2.3
7f1d6ba08000-7f1d6ba09000 rw-p 00015000 fd:01 1308226
/lib64/libz.so.1.2.3
7f1d6ba09000-7f1d6bb93000 r-xp 00000000 fd:01 1308173
/lib64/libc-2.12.so
7f1d6bb93000-7f1d6bd93000 ---p 0018a000 fd:01 1308173
/lib64/libc-2.12.so
7f1d6bd93000-7f1d6bd97000 r--p 0018a000 fd:01 1308173
/lib64/libc-2.12.so
7f1d6bd97000-7f1d6bd99000 rw-p 0018e000 fd:01 1308173
/lib64/libc-2.12.so
7f1d6bd99000-7f1d6bd9d000 rw-p 00000000 00:00 0
7f1d6bd9d000-7f1d6bdeb000 r-xp 00000000 fd:01 1308613
/lib64/libldap-2.4.so.2.10.3
7f1d6bdeb000-7f1d6bfea000 ---p 0004e000 fd:01 1308613
/lib64/libldap-2.4.so.2.10.3
7f1d6bfea000-7f1d6bfec000 r--p 0004d000 fd:01 1308613
/lib64/libldap-2.4.so.2.10.3
7f1d6bfec000-7f1d6bfee000 rw-p 0004f000 fd:01 1308613
/lib64/libldap-2.4.so.2.10.3
7f1d6bfee000-7f1d6c071000 r-xp 00000000 fd:01 1308290
/lib64/libm-2.12.so
7f1d6c071000-7f1d6c270000 ---p 00083000 fd:01 1308290
/lib64/libm-2.12.so
7f1d6c270000-7f1d6c271000 r--p 00082000 fd:01 1308290
/lib64/libm-2.12.so
7f1d6c271000-7f1d6c272000 rw-p 00083000 fd:01 1308290
/lib64/libm-2.12.so
7f1d6c272000-7f1d6c274000 r-xp 00000000 fd:01 1308282
/lib64/libdl-2.12.so
7f1d6c274000-7f1d6c474000 ---p 00002000 fd:01 1308282
/lib64/libdl-2.12.so
7f1d6c474000-7f1d6c475000 r--p 00002000 fd:01 1308282
/lib64/libdl-2.12.so
7f1d6c475000-7f1d6c476000 rw-p 00003000 fd:01 1308282
/lib64/libdl-2.12.so
7f1d6c476000-7f1d6c47d000 r-xp 00000000 fd:01 1308320
/lib64/librt-2.12.so
7f1d6c47d000-7f1d6c67c000 ---p 00007000 fd:01 1308320
/lib64/librt-2.12.so
7f1d6c67c000-7f1d6c67d000 r--p 00006000 fd:01 1308320
/lib64/librt-2.12.so
7f1d6c67d000-7f1d6c67e000 rw-p 00007000 fd:01 1308320
/lib64/librt-2.12.so
7f1d6c67e000-7f1d6c6bf000 r-xp 00000000 fd:01 1308350
/lib64/libgssapi_krb5.so.2.2
7f1d6c6bf000-7f1d6c8bf000 ---p 00041000 fd:01 1308350
/lib64/libgssapi_krb5.so.2.2
7f1d6c8bf000-7f1d6c8c0000 r--p 00041000 fd:01 1308350
/lib64/libgssapi_krb5.so.2.2
7f1d6c8c0000-7f1d6c8c2000 rw-p 00042000 fd:01 1308350
/lib64/libgssapi_krb5.so.2.2
7f1d6c8c2000-7f1d6ca7c000 r-xp 00000000 fd:01 656262
/usr/lib64/libcrypto.so.1.0.1e
7f1d6ca7c000-7f1d6cc7c000 ---p 001ba000 fd:01 656262
/usr/lib64/libcrypto.so.1.0.1e
7f1d6cc7c000-7f1d6cc97000 r--p 001ba000 fd:01 656262
/usr/lib64/libcrypto.so.1.0.1e
7f1d6cc97000-7f1d6cca3000 rw-p 001d5000 fd:01 656262
/usr/lib64/libcrypto.so.1.0.1e
7f1d6cca3000-7f1d6cca7000 rw-p 00000000 00:00 0
7f1d6cca7000-7f1d6cd09000 r-xp 00000000 fd:01 659112
/usr/lib64/libssl.so.1.0.1e
7f1d6cd09000-7f1d6cf09000 ---p 00062000 fd:01 659112
/usr/lib64/libssl.so.1.0.1e
7f1d6cf09000-7f1d6cf0d000 r--p 00062000 fd:01 659112
/usr/lib64/libssl.so.1.0.1e
7f1d6cf0d000-7f1d6cf13000 rw-p 00066000 fd:01 659112
/usr/lib64/libssl.so.1.0.1e
7f1d6cf13000-7f1d6cf1f000 r-xp 00000000 fd:01 1308338
/lib64/libpam.so.0.82.2
7f1d6cf1f000-7f1d6d11f000 ---p 0000c000 fd:01 1308338
/lib64/libpam.so.0.82.2
7f1d6d11f000-7f1d6d120000 r--p 0000c000 fd:01 1308338
/lib64/libpam.so.0.82.2
7f1d6d120000-7f1d6d121000 rw-p 0000d000 fd:01 1308338
/lib64/libpam.so.0.82.2
7f1d6d121000-7f1d6d26a000 r-xp 00000000 fd:01 655397
/usr/lib64/libxml2.so.2.7.6
7f1d6d26a000-7f1d6d469000 ---p 00149000 fd:01 655397
/usr/lib64/libxml2.so.2.7.6
7f1d6d469000-7f1d6d473000 rw-p 00148000 fd:01 655397
/usr/lib64/libxml2.so.2.7.6
7f1d6d473000-7f1d6d474000 rw-p 00000000 00:00 0
7f1d6d474000-7f1d6d494000 r-xp 00000000 fd:01 1308202
/lib64/ld-2.12.so
7f1d6d578000-7f1d6d5f9000 rw-p 00000000 00:00 0
7f1d6d61a000-7f1d6d67c000 rw-p 00000000 00:00 0
7f1d6d67c000-7f1d6d68d000 rw-p 00000000 00:00 0
7f1d6d68e000-7f1d6d68f000 rw-p 00000000 00:00 0
7f1d6d68f000-7f1d6d691000 rw-s 00000000 00:0f 440003629
/dev/shm/PostgreSQL.1035112127
7f1d6d691000-7f1d6d692000 rw-s 00000000 00:04 196609
/SYSV0052e2c1 (deleted)
7f1d6d692000-7f1d6d693000 rw-p 00000000 00:00 0
7f1d6d693000-7f1d6d695000 r--p 0001f000 fd:01 1308202
/lib64/ld-2.12.so
7f1d6d695000-7f1d6d696000 rw-p 00021000 fd:01 1308202
/lib64/ld-2.12.so
7f1d6d696000-7f1d6d697000 rw-p 00000000 00:00 0
7ffd0ee14000-7ffd0ee29000 rw-p 00000000 00:00 0
[stack]
7ffd0efc1000-7ffd0efc2000 r-xp 00000000 00:00 0
[vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]
2017-05-28 17:54:54 BST [2886]: [16-1] user=,db=,app=,client= LOG:  server
process (PID 14749) was terminated by signal 6: Aborted
2017-05-28 17:54:54 BST [2886]: [17-1] user=,db=,app=,client= DETAIL:
Failed process was running: SELECT cpl_metadata.uuid AS cpl_metadata_uuid,
cpl_metadata.created AS cpl_metadata_created, cpl_metadata.last_modified AS
cpl_metadata_la
st_modified, cpl_metadata.cpl_uuid AS cpl_metadata_cpl_uuid,
cpl_metadata.name AS cpl_metadata_name, cpl_metadata.date AS
cpl_metadata_date, cpl_metadata.content_kind AS cpl_metadata_content_kind,
cpl_metadata.credits_offset AS cpl_metad
ata_credits_offset, cpl_metadata.intermission_offset AS
cpl_metadata_intermission_offset, cpl_metadata.playback_mode AS
cpl_metadata_playback_mode, cpl_metadata.rating_hardlocked AS
cpl_metadata_rating_hardlocked, cpl_metadata.territory
AS cpl_metadata_territory, cpl_metadata.rating AS cpl_metadata_rating,
cpl_metadata.audio_type AS cpl_metadata_audio_type,
cpl_metadata.audio_language AS cpl_metadata_audio_language,
cpl_metadata.subtitled AS cpl_metadata_subtitled, cpl_
metadata.subtitle_language AS cpl_metadata_subtitle_language,
cpl_metadata.accessibility AS cpl_metadata_accessibility,
cpl_metadata.aspect_ratio AS cpl_metadata_aspect_ratio
2017-05-28 17:54:54 BST [2886]: [18-1] user=,db=,app=,client= LOG:
terminating any other active server processes
2017-05-28 17:54:54 BST [14905]: [4-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.100 WARNING:
terminating connection because of crash of another server process
2017-05-28 17:54:54 BST [14905]: [5-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.100 DETAIL:  The
postmaster has commanded this server process to roll back the current
transaction and exit, because another server process exitedabnormally and possibly corrupted shared memory.
2017-05-28 17:54:54 BST [14905]: [6-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.100 HINT:  In a moment
you should be able to reconnect to the database and repeat your command.
2017-05-28 17:54:54 BST [14894]: [4-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.68 WARNING:  terminating
connection because of crash of another server process
2017-05-28 17:54:54 BST [14894]: [5-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.68 DETAIL:  The
postmaster has commanded this server process to roll back the current
transaction and exit, because another server process exited
abnormally and possibly corrupted shared memory.
2017-05-28 17:54:54 BST [14894]: [6-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.68 HINT:  In a moment
you should be able to reconnect to the database and repeat your command.
2017-05-28 17:54:54 BST [14887]: [4-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.65 WARNING:  terminating
connection because of crash of another server process
2017-05-28 17:54:54 BST [14887]: [5-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.65 DETAIL:  The
postmaster has commanded this server process to roll back the current
transaction and exit, because another server process exited
abnormally and possibly corrupted shared memory.
2017-05-28 17:54:54 BST [14887]: [6-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.65 HINT:  In a moment
you should be able to reconnect to the database and repeat your command.
2017-05-28 17:54:54 BST [14884]: [5-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.110 WARNING:
terminating connection because of crash of another server process
2017-05-28 17:54:54 BST [14884]: [6-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.110 DETAIL:  The
postmaster has commanded this server process to roll back the current
transaction and exit, because another server process exitedabnormally and possibly corrupted shared memory.
2017-05-28 17:54:54 BST [14884]: [7-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.110 HINT:  In a moment
you should be able to reconnect to the database and repeat your command.
2017-05-28 17:54:54 BST [14884]: [8-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.110 CONTEXT:  SQL
statement "UPDATE monitoring_log SET search_data = sdq.search_data



2017-05-28 17:54:57 BST [15007]: [2-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.111 FATAL:  the database
system is in recovery mode
2017-05-28 17:54:57 BST [15004]: [2-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.111 FATAL:  the database
system is in recovery mode
2017-05-28 17:54:57 BST [15010]: [1-1]
user=[unknown],db=[unknown],app=[unknown],client=172.30.8.110 LOG:
connection received: host=172.30.8.110 port=52270
2017-05-28 17:54:57 BST [15008]: [2-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.111 FATAL:  the database
system is in recovery mode
2017-05-28 17:54:57 BST [15005]: [1-1]
user=[unknown],db=[unknown],app=[unknown],client=172.30.8.110 LOG:
connection received: host=172.30.8.110 port=52268
2017-05-28 17:54:57 BST [15009]: [1-1]
user=[unknown],db=[unknown],app=[unknown],client=172.30.8.67 LOG:
connection received: host=172.30.8.67 port=48328



--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs

Re: [BUGS] BUG #14683: *** glibc detected *** SELECT: double free or corruption

From
Michael Paquier
Date:
On Thu, Jun 1, 2017 at 1:11 AM,  <ray.warren@artsalliancemedia.com> wrote:
> We are experiencing some performance issues and we had this error on Sunday.
> Not sure whether the 2 things are related or not as we have been making some
> changes to the application and load. However, I assume the double free or
> corruption error is a bug?

Could you come up with a self-contained test case? It will be
difficult to dig into this issue otherwise.
-- 
Michael


-- 
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs

Hi Michael,

I can't create a self-contained test case. This seems to be a random crash. It is happening daily at different time of
day.The SQL running at the time isn't the same.
 

Is this likely a bug in Postgres? Could we have something misconfigured? This is a production system so we need to get
thisresolved.
 

Thanks ... Ray


-----Original Message-----
From: Michael Paquier [mailto:michael.paquier@gmail.com] 
Sent: 02 June 2017 02:18
To: Ray Warren
Cc: PostgreSQL mailing lists
Subject: Re: [BUGS] BUG #14683: *** glibc detected *** SELECT: double free or corruption

On Thu, Jun 1, 2017 at 1:11 AM,  <ray.warren@artsalliancemedia.com> wrote:
> We are experiencing some performance issues and we had this error on Sunday.
> Not sure whether the 2 things are related or not as we have been 
> making some changes to the application and load. However, I assume the 
> double free or corruption error is a bug?

Could you come up with a self-contained test case? It will be difficult to dig into this issue otherwise.
--
Michael

--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs

Hi Michael,

In fact I have noticed that the other error is different:-

*** glibc detected *** postgres: postgres etms 172.30.8.63(53338) SELECT: corrupted double-linked list:
0x00000000034119a0***
 
2017-05-31 04:04:41 BST [2703]: [7195-1] user=postgres,db=etms,app=[unknown],client=172.30.8.67 LOG:  duration: 15.793
ms statement: SELECT playback.uuid AS playback_uuid, playback.created AS playback_created, playback.last_modified AS
 
playback_last_modified, playback.deleted AS playback_deleted, playback.complex_id AS playback_complex_id,
playback.startAS playback_start, playback."end" AS playback_end, playback.duration AS playback_duration,
playback.intermission_sta
rt AS playback_intermission_start, playback.intermission_end AS playback_intermission_end, playback.accuracy AS
playback_accuracy,playback.screen_identifier AS playback_screen_identifier, playback.dnqualifier AS
playback_dnqualifier,pl
 
ayback.device_ip_address AS playback_device_ip_address, playback.serial AS playback_serial, playback.complex_identifier
ASplayback_complex_identifier, playback.feature_cpl_uuid AS playback_feature_cpl_uuid, playback.cpl_uuid AS playback
 
_cpl_uuid, playback.merge_error_code AS playback_merge_error_code, playback.worst_confidence_level AS
playback_worst_confidence_level,playback.verified AS playback_verified, playback.tms_last_modified AS
playback_tms_last_modified,play
 
back.tms_version AS playback_tms_version, playback.pack_uuid AS playback_pack_uuid, playback.pos_uuid AS
playback_pos_uuid,playback.seats_sold AS playback_seats_sold, playback.meta AS playback_meta, playback.seq_id AS
playback_seq_id      FROM playback       WHERE playback.uuid = '89cb645e-72e4-4570-add8-c797a1391e76'
 
2017-05-31 04:04:41 BST [8011]: [1350-1] user=postgres,db=etms,app=[unknown],client=172.30.8.67 LOG:  duration: 15.925
ms statement: SELECT playback.uuid AS playback_uuid, playback.created AS playback_created, playback.last_modified AS
 
playback_last_modified, playback.deleted AS playback_deleted, playback.complex_id AS playback_complex_id,
playback.startAS playback_start, playback."end" AS playback_end, playback.duration AS playback_duration,
playback.intermission_sta
rt AS playback_intermission_start, playback.intermission_end AS playback_intermission_end, playback.accuracy AS
playback_accuracy,playback.screen_identifier AS playback_screen_identifier, playback.dnqualifier AS
playback_dnqualifier,pl
 
ayback.device_ip_address AS playback_device_ip_address, playback.serial AS playback_serial, playback.complex_identifier
ASplayback_complex_identifier, playback.feature_cpl_uuid AS playback_feature_cpl_uuid, playback.cpl_uuid AS playback
 
_cpl_uuid, playback.merge_error_code AS playback_merge_error_code, playback.worst_confidence_level AS
playback_worst_confidence_level,playback.verified AS playback_verified, playback.tms_last_modified AS
playback_tms_last_modified,play
 
back.tms_version AS playback_tms_version, playback.pack_uuid AS playback_pack_uuid, playback.pos_uuid AS
playback_pos_uuid,playback.seats_sold AS playback_seats_sold, playback.meta AS playback_meta, playback.seq_id AS
playback_seq_id      FROM playback       WHERE playback.uuid = '1f29ddf2-3266-495e-ac33-2a21d00e4a1d'
 
======= Backtrace: =========
/lib64/libc.so.6(+0x75dee)[0x7f1d6ba7edee]
/lib64/libc.so.6(+0x78ff6)[0x7f1d6ba81ff6]
/lib64/libc.so.6(+0x7babe)[0x7f1d6ba84abe]
/lib64/libc.so.6(realloc+0x158)[0x7f1d6ba84f08]
postgres: postgres etms 172.30.8.63(53338) SELECT[0x7c8d72]
postgres: postgres etms 172.30.8.63(53338) SELECT(repalloc+0x25)[0x7c9295]
postgres: postgres etms 172.30.8.63(53338) SELECT(enlargeStringInfo+0x4e)[0x5e505e]
postgres: postgres etms 172.30.8.63(53338) SELECT(appendStringInfoChar+0x28)[0x5e5178]
postgres: postgres etms 172.30.8.63(53338) SELECT[0x7a6724]
postgres: postgres etms 172.30.8.63(53338) SELECT(EmitErrorReport+0xa2f)[0x7a963f]
postgres: postgres etms 172.30.8.63(53338) SELECT(errfinish+0x8d)[0x7a97dd]
postgres: postgres etms 172.30.8.63(53338) SELECT(FileClose+0x1c5)[0x6966d5]
postgres: postgres etms 172.30.8.63(53338) SELECT(BufFileClose+0x32)[0x698652]
postgres: postgres etms 172.30.8.63(53338) SELECT(ExecHashJoin+0x31b)[0x5ce60b]
postgres: postgres etms 172.30.8.63(53338) SELECT(ExecProcNode+0x168)[0x5bc9f8]
postgres: postgres etms 172.30.8.63(53338) SELECT(ExecSort+0xf9)[0x5d86c9]
postgres: postgres etms 172.30.8.63(53338) SELECT(ExecProcNode+0x148)[0x5bc9d8]
postgres: postgres etms 172.30.8.63(53338) SELECT(ExecLimit+0xa8)[0x5d13a8]
postgres: postgres etms 172.30.8.63(53338) SELECT(ExecProcNode+0x90)[0x5bc920]
postgres: postgres etms 172.30.8.63(53338) SELECT(standard_ExecutorRun+0xd2)[0x5bb2c2]
/usr/pgsql-9.5/lib/pg_stat_statements.so(+0x448b)[0x7f1d62b2848b]
postgres: postgres etms 172.30.8.63(53338) SELECT[0x6bd417]
postgres: postgres etms 172.30.8.63(53338) SELECT(PortalRun+0x188)[0x6be5d8]
postgres: postgres etms 172.30.8.63(53338) SELECT[0x6bb3be]
postgres: postgres etms 172.30.8.63(53338) SELECT(PostgresMain+0xd35)[0x6bca25]
postgres: postgres etms 172.30.8.63(53338) SELECT(PostmasterMain+0x1875)[0x663165]
postgres: postgres etms 172.30.8.63(53338) SELECT(main+0x7a8)[0x5f4b28]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f1d6ba27d1d]
postgres: postgres etms 172.30.8.63(53338) SELECT[0x467549]
======= Memory map: ========
00400000-009cf000 r-xp 00000000 fd:01 788841                             /usr/pgsql-9.5/bin/postgres
00bcf000-00bdc000 rw-p 005cf000 fd:01 788841                             /usr/pgsql-9.5/bin/postgres
00bdc000-00c2b000 rw-p 00000000 00:00 0
0173e000-01785000 rw-p 00000000 00:00 0
01785000-0362a000 rw-p 00000000 00:00 0
7f1b0c000000-7f1b0c021000 rw-p 00000000 00:00 0
7f1b0c021000-7f1b10000000 ---p 00000000 00:00 0
7f1b12280000-7f1b12296000 r-xp 00000000 fd:01 1308180                    /lib64/libgcc_s-4.4.7-20120601.so.1
7f1b12296000-7f1b12495000 ---p 00016000 fd:01 1308180                    /lib64/libgcc_s-4.4.7-20120601.so.1
7f1b12495000-7f1b12496000 rw-p 00015000 fd:01 1308180                    /lib64/libgcc_s-4.4.7-20120601.so.1
7f1b12496000-7f1b124a3000 r-xp 00000000 fd:01 1308307                    /lib64/libnss_files-2.12.so
7f1b124a3000-7f1b126a2000 ---p 0000d000 fd:01 1308307                    /lib64/libnss_files-2.12.so
7f1b126a2000-7f1b126a3000 r--p 0000c000 fd:01 1308307                    /lib64/libnss_files-2.12.so
7f1b126a3000-7f1b126a4000 rw-p 0000d000 fd:01 1308307                    /lib64/libnss_files-2.12.so
7f1b126a4000-7f1d62b24000 rw-s 00000000 00:04 492918692                  /dev/zero (deleted)
7f1d62b24000-7f1d62b2c000 r-xp 00000000 fd:01 659776                     /usr/pgsql-9.5/lib/pg_stat_statements.so
7f1d62b2c000-7f1d62d2b000 ---p 00008000 fd:01 659776                     /usr/pgsql-9.5/lib/pg_stat_statements.so
7f1d62d2b000-7f1d62d2c000 rw-p 00007000 fd:01 659776                     /usr/pgsql-9.5/lib/pg_stat_statements.so
7f1d62d2c000-7f1d68bc1000 r--p 00000000 fd:01 659094                     /usr/lib/locale/locale-archive
7f1d68bc1000-7f1d68bde000 r-xp 00000000 fd:01 1308211                    /lib64/libselinux.so.1
7f1d68bde000-7f1d68ddd000 ---p 0001d000 fd:01 1308211                    /lib64/libselinux.so.1
7f1d68ddd000-7f1d68dde000 r--p 0001c000 fd:01 1308211                    /lib64/libselinux.so.1
7f1d68dde000-7f1d68ddf000 rw-p 0001d000 fd:01 1308211                    /lib64/libselinux.so.1
7f1d68ddf000-7f1d68de0000 rw-p 00000000 00:00 0
7f1d68de0000-7f1d68de2000 r-xp 00000000 fd:01 1308204                    /lib64/libfreebl3.so
7f1d68de2000-7f1d68fe1000 ---p 00002000 fd:01 1308204                    /lib64/libfreebl3.so
7f1d68fe1000-7f1d68fe2000 r--p 00001000 fd:01 1308204                    /lib64/libfreebl3.so
7f1d68fe2000-7f1d68fe3000 rw-p 00002000 fd:01 1308204                    /lib64/libfreebl3.so
7f1d68fe3000-7f1d6901c000 r-xp 00000000 fd:01 1308378                    /lib64/libnspr4.so
7f1d6901c000-7f1d6921c000 ---p 00039000 fd:01 1308378                    /lib64/libnspr4.so
7f1d6921c000-7f1d6921d000 r--p 00039000 fd:01 1308378                    /lib64/libnspr4.so
7f1d6921d000-7f1d6921f000 rw-p 0003a000 fd:01 1308378                    /lib64/libnspr4.so
7f1d6921f000-7f1d69221000 rw-p 00000000 00:00 0
7f1d69221000-7f1d69225000 r-xp 00000000 fd:01 1308380                    /lib64/libplc4.so
7f1d69225000-7f1d69424000 ---p 00004000 fd:01 1308380                    /lib64/libplc4.so
7f1d69424000-7f1d69425000 r--p 00003000 fd:01 1308380                    /lib64/libplc4.so
7f1d69425000-7f1d69426000 rw-p 00004000 fd:01 1308380                    /lib64/libplc4.so
7f1d69426000-7f1d69429000 r-xp 00000000 fd:01 1308382                    /lib64/libplds4.so
7f1d69429000-7f1d69628000 ---p 00003000 fd:01 1308382                    /lib64/libplds4.so
7f1d69628000-7f1d69629000 r--p 00002000 fd:01 1308382                    /lib64/libplds4.so
7f1d69629000-7f1d6962a000 rw-p 00003000 fd:01 1308382                    /lib64/libplds4.so
7f1d6962a000-7f1d69650000 r-xp 00000000 fd:01 656260                     /usr/lib64/libnssutil3.so
7f1d69650000-7f1d6984f000 ---p 00026000 fd:01 656260                     /usr/lib64/libnssutil3.so
7f1d6984f000-7f1d69856000 r--p 00025000 fd:01 656260                     /usr/lib64/libnssutil3.so
7f1d69856000-7f1d69857000 rw-p 0002c000 fd:01 656260                     /usr/lib64/libnssutil3.so
7f1d69857000-7f1d69991000 r-xp 00000000 fd:01 656907                     /usr/lib64/libnss3.so
7f1d69991000-7f1d69b90000 ---p 0013a000 fd:01 656907                     /usr/lib64/libnss3.so
7f1d69b90000-7f1d69b96000 r--p 00139000 fd:01 656907                     /usr/lib64/libnss3.so
7f1d69b96000-7f1d69b98000 rw-p 0013f000 fd:01 656907                     /usr/lib64/libnss3.so
7f1d69b98000-7f1d69b9a000 rw-p 00000000 00:00 0
7f1d69b9a000-7f1d69bc2000 r-xp 00000000 fd:01 659119                     /usr/lib64/libsmime3.so
7f1d69bc2000-7f1d69dc1000 ---p 00028000 fd:01 659119                     /usr/lib64/libsmime3.so
7f1d69dc1000-7f1d69dc5000 r--p 00027000 fd:01 659119                     /usr/lib64/libsmime3.so
7f1d69dc5000-7f1d69dc6000 rw-p 0002b000 fd:01 659119                     /usr/lib64/libsmime3.so
7f1d69dc6000-7f1d69e0d000 r-xp 00000000 fd:01 659120                     /usr/lib64/libssl3.so
7f1d69e0d000-7f1d6a00d000 ---p 00047000 fd:01 659120                     /usr/lib64/libssl3.so
7f1d6a00d000-7f1d6a011000 r--p 00047000 fd:01 659120                     /usr/lib64/libssl3.so
7f1d6a011000-7f1d6a012000 rw-p 0004b000 fd:01 659120                     /usr/lib64/libssl3.so
7f1d6a012000-7f1d6a013000 rw-p 00000000 00:00 0
7f1d6a013000-7f1d6a02c000 r-xp 00000000 fd:01 655285                     /usr/lib64/libsasl2.so.2.0.23
7f1d6a02c000-7f1d6a22b000 ---p 00019000 fd:01 655285                     /usr/lib64/libsasl2.so.2.0.23
7f1d6a22b000-7f1d6a22c000 r--p 00018000 fd:01 655285                     /usr/lib64/libsasl2.so.2.0.23
7f1d6a22c000-7f1d6a22d000 rw-p 00019000 fd:01 655285                     /usr/lib64/libsasl2.so.2.0.23
7f1d6a22d000-7f1d6a23b000 r-xp 00000000 fd:01 1308168                    /lib64/liblber-2.4.so.2.10.3
7f1d6a23b000-7f1d6a43a000 ---p 0000e000 fd:01 1308168                    /lib64/liblber-2.4.so.2.10.3
7f1d6a43a000-7f1d6a43b000 r--p 0000d000 fd:01 1308168                    /lib64/liblber-2.4.so.2.10.3
7f1d6a43b000-7f1d6a43c000 rw-p 0000e000 fd:01 1308168                    /lib64/liblber-2.4.so.2.10.3
7f1d6a43c000-7f1d6a453000 r-xp 00000000 fd:01 1308197                    /lib64/libpthread-2.12.so
7f1d6a453000-7f1d6a653000 ---p 00017000 fd:01 1308197                    /lib64/libpthread-2.12.so
7f1d6a653000-7f1d6a654000 r--p 00017000 fd:01 1308197                    /lib64/libpthread-2.12.so
7f1d6a654000-7f1d6a655000 rw-p 00018000 fd:01 1308197                    /lib64/libpthread-2.12.so
7f1d6a655000-7f1d6a659000 rw-p 00000000 00:00 0
7f1d6a659000-7f1d6a66f000 r-xp 00000000 fd:01 1308318                    /lib64/libresolv-2.12.so
7f1d6a66f000-7f1d6a86f000 ---p 00016000 fd:01 1308318                    /lib64/libresolv-2.12.so
7f1d6a86f000-7f1d6a870000 r--p 00016000 fd:01 1308318                    /lib64/libresolv-2.12.so
7f1d6a870000-7f1d6a871000 rw-p 00017000 fd:01 1308318                    /lib64/libresolv-2.12.so
7f1d6a871000-7f1d6a873000 rw-p 00000000 00:00 0
7f1d6a873000-7f1d6a875000 r-xp 00000000 fd:01 1308340                    /lib64/libkeyutils.so.1.3
7f1d6a875000-7f1d6aa74000 ---p 00002000 fd:01 1308340                    /lib64/libkeyutils.so.1.3
7f1d6aa74000-7f1d6aa75000 r--p 00001000 fd:01 1308340                    /lib64/libkeyutils.so.1.3
7f1d6aa75000-7f1d6aa76000 rw-p 00002000 fd:01 1308340                    /lib64/libkeyutils.so.1.3
7f1d6aa76000-7f1d6aa80000 r-xp 00000000 fd:01 1308358                    /lib64/libkrb5support.so.0.1
7f1d6aa80000-7f1d6ac7f000 ---p 0000a000 fd:01 1308358                    /lib64/libkrb5support.so.0.1
7f1d6ac7f000-7f1d6ac80000 r--p 00009000 fd:01 1308358                    /lib64/libkrb5support.so.0.1
7f1d6ac80000-7f1d6ac81000 rw-p 0000a000 fd:01 1308358                    /lib64/libkrb5support.so.0.1
7f1d6ac81000-7f1d6acaa000 r-xp 00000000 fd:01 1308344                    /lib64/libk5crypto.so.3.1
7f1d6acaa000-7f1d6aeaa000 ---p 00029000 fd:01 1308344                    /lib64/libk5crypto.so.3.1
7f1d6aeaa000-7f1d6aeab000 r--p 00029000 fd:01 1308344                    /lib64/libk5crypto.so.3.1
7f1d6aeab000-7f1d6aeac000 rw-p 0002a000 fd:01 1308344                    /lib64/libk5crypto.so.3.1
7f1d6aeac000-7f1d6aead000 rw-p 00000000 00:00 0
7f1d6aead000-7f1d6aeb0000 r-xp 00000000 fd:01 1308233                    /lib64/libcom_err.so.2.1
7f1d6aeb0000-7f1d6b0af000 ---p 00003000 fd:01 1308233                    /lib64/libcom_err.so.2.1
7f1d6b0af000-7f1d6b0b0000 r--p 00002000 fd:01 1308233                    /lib64/libcom_err.so.2.1
7f1d6b0b0000-7f1d6b0b1000 rw-p 00003000 fd:01 1308233                    /lib64/libcom_err.so.2.1
7f1d6b0b1000-7f1d6b18d000 r-xp 00000000 fd:01 1308355                    /lib64/libkrb5.so.3.3
7f1d6b18d000-7f1d6b38c000 ---p 000dc000 fd:01 1308355                    /lib64/libkrb5.so.3.3
7f1d6b38c000-7f1d6b396000 r--p 000db000 fd:01 1308355                    /lib64/libkrb5.so.3.3
7f1d6b396000-7f1d6b398000 rw-p 000e5000 fd:01 1308355                    /lib64/libkrb5.so.3.3
7f1d6b398000-7f1d6b39f000 r-xp 00000000 fd:01 1308177                    /lib64/libcrypt-2.12.so
7f1d6b39f000-7f1d6b59f000 ---p 00007000 fd:01 1308177                    /lib64/libcrypt-2.12.so
7f1d6b59f000-7f1d6b5a0000 r--p 00007000 fd:01 1308177                    /lib64/libcrypt-2.12.so
7f1d6b5a0000-7f1d6b5a1000 rw-p 00008000 fd:01 1308177                    /lib64/libcrypt-2.12.so
7f1d6b5a1000-7f1d6b5cf000 rw-p 00000000 00:00 0
7f1d6b5cf000-7f1d6b5e7000 r-xp 00000000 fd:01 1308482                    /lib64/libaudit.so.1.0.0
7f1d6b5e7000-7f1d6b7e6000 ---p 00018000 fd:01 1308482                    /lib64/libaudit.so.1.0.0
7f1d6b7e6000-7f1d6b7e8000 r--p 00017000 fd:01 1308482                    /lib64/libaudit.so.1.0.0
7f1d6b7e8000-7f1d6b7f3000 rw-p 00019000 fd:01 1308482                    /lib64/libaudit.so.1.0.0
7f1d6b7f3000-7f1d6b808000 r-xp 00000000 fd:01 1308226                    /lib64/libz.so.1.2.3
7f1d6b808000-7f1d6ba07000 ---p 00015000 fd:01 1308226                    /lib64/libz.so.1.2.3
7f1d6ba07000-7f1d6ba08000 r--p 00014000 fd:01 1308226                    /lib64/libz.so.1.2.3
7f1d6ba08000-7f1d6ba09000 rw-p 00015000 fd:01 1308226                    /lib64/libz.so.1.2.3
7f1d6ba09000-7f1d6bb93000 r-xp 00000000 fd:01 1308173                    /lib64/libc-2.12.so
7f1d6bb93000-7f1d6bd93000 ---p 0018a000 fd:01 1308173                    /lib64/libc-2.12.so
7f1d6bd93000-7f1d6bd97000 r--p 0018a000 fd:01 1308173                    /lib64/libc-2.12.so
7f1d6bd97000-7f1d6bd99000 rw-p 0018e000 fd:01 1308173                    /lib64/libc-2.12.so
7f1d6bd99000-7f1d6bd9d000 rw-p 00000000 00:00 0
7f1d6bd9d000-7f1d6bdeb000 r-xp 00000000 fd:01 1308613                    /lib64/libldap-2.4.so.2.10.3
7f1d6bdeb000-7f1d6bfea000 ---p 0004e000 fd:01 1308613                    /lib64/libldap-2.4.so.2.10.3
7f1d6bfea000-7f1d6bfec000 r--p 0004d000 fd:01 1308613                    /lib64/libldap-2.4.so.2.10.3
7f1d6bfec000-7f1d6bfee000 rw-p 0004f000 fd:01 1308613                    /lib64/libldap-2.4.so.2.10.3
7f1d6bfee000-7f1d6c071000 r-xp 00000000 fd:01 1308290                    /lib64/libm-2.12.so
7f1d6c071000-7f1d6c270000 ---p 00083000 fd:01 1308290                    /lib64/libm-2.12.so
7f1d6c270000-7f1d6c271000 r--p 00082000 fd:01 1308290                    /lib64/libm-2.12.so
7f1d6c271000-7f1d6c272000 rw-p 00083000 fd:01 1308290                    /lib64/libm-2.12.so
7f1d6c272000-7f1d6c274000 r-xp 00000000 fd:01 1308282                    /lib64/libdl-2.12.so
7f1d6c274000-7f1d6c474000 ---p 00002000 fd:01 1308282                    /lib64/libdl-2.12.so
7f1d6c474000-7f1d6c475000 r--p 00002000 fd:01 1308282                    /lib64/libdl-2.12.so
7f1d6c475000-7f1d6c476000 rw-p 00003000 fd:01 1308282                    /lib64/libdl-2.12.so
7f1d6c476000-7f1d6c47d000 r-xp 00000000 fd:01 1308320                    /lib64/librt-2.12.so
7f1d6c47d000-7f1d6c67c000 ---p 00007000 fd:01 1308320                    /lib64/librt-2.12.so
7f1d6c67c000-7f1d6c67d000 r--p 00006000 fd:01 1308320                    /lib64/librt-2.12.so
7f1d6c67d000-7f1d6c67e000 rw-p 00007000 fd:01 1308320                    /lib64/librt-2.12.so
7f1d6c67e000-7f1d6c6bf000 r-xp 00000000 fd:01 1308350                    /lib64/libgssapi_krb5.so.2.2
7f1d6c6bf000-7f1d6c8bf000 ---p 00041000 fd:01 1308350                    /lib64/libgssapi_krb5.so.2.2
7f1d6c8bf000-7f1d6c8c0000 r--p 00041000 fd:01 1308350                    /lib64/libgssapi_krb5.so.2.2
7f1d6c8c0000-7f1d6c8c2000 rw-p 00042000 fd:01 1308350                    /lib64/libgssapi_krb5.so.2.2
7f1d6c8c2000-7f1d6ca7c000 r-xp 00000000 fd:01 656262                     /usr/lib64/libcrypto.so.1.0.1e
7f1d6ca7c000-7f1d6cc7c000 ---p 001ba000 fd:01 656262                     /usr/lib64/libcrypto.so.1.0.1e
7f1d6cc7c000-7f1d6cc97000 r--p 001ba000 fd:01 656262                     /usr/lib64/libcrypto.so.1.0.1e
7f1d6cc97000-7f1d6cca3000 rw-p 001d5000 fd:01 656262                     /usr/lib64/libcrypto.so.1.0.1e
7f1d6cca3000-7f1d6cca7000 rw-p 00000000 00:00 0
7f1d6cca7000-7f1d6cd09000 r-xp 00000000 fd:01 659112                     /usr/lib64/libssl.so.1.0.1e
7f1d6cd09000-7f1d6cf09000 ---p 00062000 fd:01 659112                     /usr/lib64/libssl.so.1.0.1e
7f1d6cf09000-7f1d6cf0d000 r--p 00062000 fd:01 659112                     /usr/lib64/libssl.so.1.0.1e
7f1d6cf0d000-7f1d6cf13000 rw-p 00066000 fd:01 659112                     /usr/lib64/libssl.so.1.0.1e
7f1d6cf13000-7f1d6cf1f000 r-xp 00000000 fd:01 1308338                    /lib64/libpam.so.0.82.2
7f1d6cf1f000-7f1d6d11f000 ---p 0000c000 fd:01 1308338                    /lib64/libpam.so.0.82.2
7f1d6d11f000-7f1d6d120000 r--p 0000c000 fd:01 1308338                    /lib64/libpam.so.0.82.2
7f1d6d120000-7f1d6d121000 rw-p 0000d000 fd:01 1308338                    /lib64/libpam.so.0.82.2
7f1d6d121000-7f1d6d26a000 r-xp 00000000 fd:01 655397                     /usr/lib64/libxml2.so.2.7.6
7f1d6d26a000-7f1d6d469000 ---p 00149000 fd:01 655397                     /usr/lib64/libxml2.so.2.7.6
7f1d6d469000-7f1d6d473000 rw-p 00148000 fd:01 655397                     /usr/lib64/libxml2.so.2.7.6
7f1d6d473000-7f1d6d474000 rw-p 00000000 00:00 0
7f1d6d474000-7f1d6d494000 r-xp 00000000 fd:01 1308202                    /lib64/ld-2.12.so
7f1d6d498000-7f1d6d61a000 rw-p 00000000 00:00 0
7f1d6d63b000-7f1d6d67c000 rw-p 00000000 00:00 0
7f1d6d67c000-7f1d6d68d000 rw-p 00000000 00:00 0
7f1d6d68e000-7f1d6d68f000 rw-p 00000000 00:00 0
7f1d6d68f000-7f1d6d691000 rw-s 00000000 00:0f 492918696                  /dev/shm/PostgreSQL.1856781077
7f1d6d691000-7f1d6d692000 rw-s 00000000 00:04 229377                     /SYSV0052e2c1 (deleted)
7f1d6d692000-7f1d6d693000 rw-p 00000000 00:00 0
7f1d6d693000-7f1d6d695000 r--p 0001f000 fd:01 1308202                    /lib64/ld-2.12.so
7f1d6d695000-7f1d6d696000 rw-p 00021000 fd:01 1308202                    /lib64/ld-2.12.so
7f1d6d696000-7f1d6d697000 rw-p 00000000 00:00 0
7ffd0ee14000-7ffd0ee29000 rw-p 00000000 00:00 0                          [stack]
7ffd0efc1000-7ffd0efc2000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]




-----Original Message-----
From: Michael Paquier [mailto:michael.paquier@gmail.com] 
Sent: 02 June 2017 02:18
To: Ray Warren
Cc: PostgreSQL mailing lists
Subject: Re: [BUGS] BUG #14683: *** glibc detected *** SELECT: double free or corruption

On Thu, Jun 1, 2017 at 1:11 AM,  <ray.warren@artsalliancemedia.com> wrote:
> We are experiencing some performance issues and we had this error on Sunday.
> Not sure whether the 2 things are related or not as we have been 
> making some changes to the application and load. However, I assume the 
> double free or corruption error is a bug?

Could you come up with a self-contained test case? It will be difficult to dig into this issue otherwise.
--
Michael

--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs

Ray Warren <Ray.Warren@artsalliancemedia.com> writes:
> ======= Backtrace: =========
> /lib64/libc.so.6(+0x75dee)[0x7f1d6ba7edee]
> /lib64/libc.so.6(+0x78ff6)[0x7f1d6ba81ff6]
> /lib64/libc.so.6(+0x7babe)[0x7f1d6ba84abe]
> /lib64/libc.so.6(realloc+0x158)[0x7f1d6ba84f08]
> postgres: postgres etms 172.30.8.63(53338) SELECT[0x7c8d72]
> postgres: postgres etms 172.30.8.63(53338) SELECT(repalloc+0x25)[0x7c9295]
> postgres: postgres etms 172.30.8.63(53338) SELECT(enlargeStringInfo+0x4e)[0x5e505e]
> postgres: postgres etms 172.30.8.63(53338) SELECT(appendStringInfoChar+0x28)[0x5e5178]
> postgres: postgres etms 172.30.8.63(53338) SELECT[0x7a6724]
> postgres: postgres etms 172.30.8.63(53338) SELECT(EmitErrorReport+0xa2f)[0x7a963f]
> postgres: postgres etms 172.30.8.63(53338) SELECT(errfinish+0x8d)[0x7a97dd]
> postgres: postgres etms 172.30.8.63(53338) SELECT(FileClose+0x1c5)[0x6966d5]
> postgres: postgres etms 172.30.8.63(53338) SELECT(BufFileClose+0x32)[0x698652]
> postgres: postgres etms 172.30.8.63(53338) SELECT(ExecHashJoin+0x31b)[0x5ce60b]
> postgres: postgres etms 172.30.8.63(53338) SELECT(ExecProcNode+0x168)[0x5bc9f8]
> postgres: postgres etms 172.30.8.63(53338) SELECT(ExecSort+0xf9)[0x5d86c9]
> postgres: postgres etms 172.30.8.63(53338) SELECT(ExecProcNode+0x148)[0x5bc9d8]

These stack traces would be a lot more useful if you had debug symbols
installed, I think.

This trace is rather unusual because it appears to show FileClose trying to
log something.  That would be expected if you have log_temp_files enabled,
but otherwise it would indicate an unexpected failure.  More generally,
what nondefault configuration settings are you using, and do you have any
extensions installed?
        regards, tom lane


--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs

Hi Tom,

Not sure how to install debug symbols and whether it would be a good idea on a Production system.

The information you asked for is below. Note that we have had increased logging configured for a while now so that we
canrun pgbadger. These issues seem to have happened when the load was increased by adding new functionality. We have
switchedthis extra functionality off for the time being so that the database is stable again. 

NON DEFAUL PARAMETERS
work_mem = '32MB'
listen_addresses = '*'
max_connections = '300'
superuser_reserved_connections = '13'
shared_buffers = '9GB'
effective_cache_size = '18GB'
maintenance_work_mem = '256MB'
wal_level = 'hot_standby'
archive_mode = 'on'
archive_command = 'cp /var/lib/pgsql/9.5/data/%p /var/lib/pgsql/9.5/wal_archive/%f'
max_wal_senders = '10'
wal_keep_segments = '5000'
hot_standby = 'on'
max_stack_depth = '4MB'
search_path = '"$user", public, repmgr_dbp01_cluster'
shared_preload_libraries = 'pg_stat_statements'
pg_stat_statements.max = '5000'
pg_stat_statements.track = 'all'
log_min_duration_statement = '10'
log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,app=%a,client=%h '
log_checkpoints = 'on'
log_connections = 'on'
log_disconnections = 'on'
log_lock_waits = 'on'
log_temp_files = '0'
log_statement = 'none'
log_autovacuum_min_duration = '0'
log_directory = '/mnt/pg_log'
timezone = 'UTC'
synchronous_commit = 'off'
checkpoint_timeout = '30min'
checkpoint_completion_target = '0.9'
max_wal_size = '18GB'

EXTENSIONS       name
--------------------plpgsqlpg_stat_statementspg_buffercacheuuid-ossp


Thanks for helping.

Kind regards ... Ray

-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: 02 June 2017 16:10
To: Ray Warren
Cc: Michael Paquier; PostgreSQL mailing lists
Subject: Re: [BUGS] BUG #14683: *** glibc detected *** SELECT: double free or corruption

Ray Warren <Ray.Warren@artsalliancemedia.com> writes:
> ======= Backtrace: =========
> /lib64/libc.so.6(+0x75dee)[0x7f1d6ba7edee]
> /lib64/libc.so.6(+0x78ff6)[0x7f1d6ba81ff6]
> /lib64/libc.so.6(+0x7babe)[0x7f1d6ba84abe]
> /lib64/libc.so.6(realloc+0x158)[0x7f1d6ba84f08]
> postgres: postgres etms 172.30.8.63(53338) SELECT[0x7c8d72]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(repalloc+0x25)[0x7c9295]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(enlargeStringInfo+0x4e)[0x5e505e]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(appendStringInfoChar+0x28)[0x5e5178]
> postgres: postgres etms 172.30.8.63(53338) SELECT[0x7a6724]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(EmitErrorReport+0xa2f)[0x7a963f]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(errfinish+0x8d)[0x7a97dd]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(FileClose+0x1c5)[0x6966d5]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(BufFileClose+0x32)[0x698652]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(ExecHashJoin+0x31b)[0x5ce60b]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(ExecProcNode+0x168)[0x5bc9f8]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(ExecSort+0xf9)[0x5d86c9]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(ExecProcNode+0x148)[0x5bc9d8]

These stack traces would be a lot more useful if you had debug symbols installed, I think.

This trace is rather unusual because it appears to show FileClose trying to log something.  That would be expected if
youhave log_temp_files enabled, but otherwise it would indicate an unexpected failure.  More generally, what nondefault
configurationsettings are you using, and do you have any extensions installed? 
        regards, tom lane



--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs