Thread: [BUGS] BUG #14683: *** glibc detected *** SELECT: double free or corruption
[BUGS] BUG #14683: *** glibc detected *** SELECT: double free or corruption
From
ray.warren@artsalliancemedia.com
Date:
The following bug has been logged on the website:
Bug reference: 14683
Logged by: Ray Warren
Email address: ray.warren@artsalliancemedia.com
PostgreSQL version: 9.5.7
Operating system: CentOS release 6.9
Description:
We are experiencing some performance issues and we had this error on Sunday.
Not sure whether the 2 things are related or not as we have been making some
changes to the application and load. However, I assume the double free or
corruption error is a bug?
2017-05-28 17:54:53 BST [14898]: [3-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.110 LOG: duration:
13.117 ms statement: UPDATE monitoring_log SET timestamp=1495990491.0,
value='0', seq_id=nextval('monitoring_seq_num') WHERE
monitoring_log.device_uuid = '135d016b-7a0b-43dc-b6aa-cd0a812dc533' AND
monitoring_log.monitoring_type_uuid =
'01bdc171-9869-5d51-bce7-3117c4f3dde4'
2017-05-28 17:54:53 BST [14749]: [11-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.63 LOG: temporary file:
path "base/pgsql_tmp/pgsql_tmp14749.3", size 30453160
2017-05-28 17:54:53 BST [14749]: [12-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.63 STATEMENT: SELECT
cpl_metadata.uuid AS cpl_metadata_uuid, cpl_metadata.created AS
cpl_metadata_created, cpl_metadata.last_modified AS cpl_met
adata_last_modified, cpl_metadata.cpl_uuid AS cpl_metadata_cpl_uuid,
cpl_metadata.name AS cpl_metadata_name, cpl_metadata.date AS
cpl_metadata_date, cpl_metadata.content_kind AS cpl_metadata_content_kind,
cpl_metadata.credits_offset AS c
pl_metadata_credits_offset, cpl_metadata.intermission_offset AS
cpl_metadata_intermission_offset, cpl_metadata.playback_mode AS
cpl_metadata_playback_mode, cpl_metadata.rating_hardlocked AS
cpl_metadata_rating_hardlocked, cpl_metadata.te
rritory AS cpl_metadata_territory, cpl_metadata.rating AS
cpl_metadata_rating, cpl_metadata.audio_type AS cpl_metadata_audio_type,
cpl_metadata.audio_language AS cpl_metadata_audio_language,
cpl_metadata.subtitled AS cpl_metadata_subtitl
ed, cpl_metadata.subtitle_language AS cpl_metadata_subtitle_language,
cpl_metadata.accessibility AS cpl_metadata_accessibility,
cpl_metadata.aspect_ratio AS cpl_metadata_aspect_ratio,
cpl_metadata.resolution AS cpl_metadata_resolution, c
pl_metadata.studio AS cpl_metadata_studio, cpl_metadata.facility AS
cpl_metadata_facility, cpl_metadata.motion_simulator_format AS
cpl_metadata_motion_simulator_format, cpl_metadata.ghostbusted AS
cpl_metadata_ghostbusted, cpl_metadata.s
tate AS cpl_metadata_state, cpl_metadata.is_master AS
cpl_metadata_is_master, cpl_metadata.is_edited AS cpl_metadata_is_edited,
cpl_metadata.next_cpl_uuid AS cpl_metadata_next_cpl_uuid,
cpl_metadata.seq_id AS cpl_metadata_seq_id, cpl.uui
d AS cpl_uuid, cpl.created AS cpl_created, cpl.last_modified AS
cpl_last_modified, cpl.content_title_text AS cpl_content_title_text,
cpl.content_title AS cpl_content_title, cpl.is_encrypted AS
cpl_is_encrypted, cpl.duration_in_frames AS
cpl_duration_in_frames, cpl.duration_in_seconds AS cpl_duration_in_seconds,
cpl.edit_rate_a AS cpl_edit_rate_a, cpl.edit_rate_b AS cpl_edit_rate_b,
cpl.package_type AS cpl_package_type, cpl.physically_exists AS
cpl_physically_exists, cpl
.available_on_courier AS cpl_available_on_courier, cpl.courier_uploaders AS
cpl_courier_uploaders, cpl.seq_id AS cpl_seq_id, greatest(cpl.seq_id,
cpl_metadata.seq_id) AS combined_seq_id FROM cpl_metadata JOIN cpl ON cpl_metadata.cpl_uuid = cpl.uuid
WHEREcpl_metadata.is_master = true AND greatest(cpl.seq_id,
cpl_metadata.seq_id) >= 14321152 ORDER BY combined_seq_id LIMIT 1000
*** glibc detected *** postgres: postgres etms 172.30.8.63(42412) SELECT:
double free or corruption (!prev): 0x00000000033c5770 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75dee)[0x7f1d6ba7edee]
/lib64/libc.so.6(+0x78c80)[0x7f1d6ba81c80]
postgres: postgres etms 172.30.8.63(42412)
SELECT(ExecHashJoin+0x31b)[0x5ce60b]
postgres: postgres etms 172.30.8.63(42412)
SELECT(ExecProcNode+0x168)[0x5bc9f8]
postgres: postgres etms 172.30.8.63(42412) SELECT(ExecSort+0xf9)[0x5d86c9]
postgres: postgres etms 172.30.8.63(42412)
SELECT(ExecProcNode+0x148)[0x5bc9d8]
postgres: postgres etms 172.30.8.63(42412)
SELECT(ExecLimit+0xa8)[0x5d13a8]
postgres: postgres etms 172.30.8.63(42412)
SELECT(ExecProcNode+0x90)[0x5bc920]
postgres: postgres etms 172.30.8.63(42412)
SELECT(standard_ExecutorRun+0xd2)[0x5bb2c2]
/usr/pgsql-9.5/lib/pg_stat_statements.so(+0x448b)[0x7f1d62b2848b]
postgres: postgres etms 172.30.8.63(42412) SELECT[0x6bd417]
postgres: postgres etms 172.30.8.63(42412)
SELECT(PortalRun+0x188)[0x6be5d8]
postgres: postgres etms 172.30.8.63(42412) SELECT[0x6bb3be]
postgres: postgres etms 172.30.8.63(42412)
SELECT(PostgresMain+0xd35)[0x6bca25]
postgres: postgres etms 172.30.8.63(42412)
SELECT(PostmasterMain+0x1875)[0x663165]
postgres: postgres etms 172.30.8.63(42412) SELECT(main+0x7a8)[0x5f4b28]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f1d6ba27d1d]
postgres: postgres etms 172.30.8.63(42412) SELECT[0x467549]
======= Memory map: ========
00400000-009cf000 r-xp 00000000 fd:01 788841
/usr/pgsql-9.5/bin/postgres
00bcf000-00bdc000 rw-p 005cf000 fd:01 788841
/usr/pgsql-9.5/bin/postgres
00bdc000-00c2b000 rw-p 00000000 00:00 0
0173e000-01785000 rw-p 00000000 00:00 0
01785000-035d5000 rw-p 00000000 00:00 0
7f1b0c000000-7f1b0c021000 rw-p 00000000 00:00 0
7f1b0c021000-7f1b10000000 ---p 00000000 00:00 0
7f1b12280000-7f1b12296000 r-xp 00000000 fd:01 1308180
/lib64/libgcc_s-4.4.7-20120601.so.1
7f1b12296000-7f1b12495000 ---p 00016000 fd:01 1308180
/lib64/libgcc_s-4.4.7-20120601.so.1
7f1b12495000-7f1b12496000 rw-p 00015000 fd:01 1308180
/lib64/libgcc_s-4.4.7-20120601.so.1
7f1b12496000-7f1b124a3000 r-xp 00000000 fd:01 1308307
/lib64/libnss_files-2.12.so
7f1b124a3000-7f1b126a2000 ---p 0000d000 fd:01 1308307
/lib64/libnss_files-2.12.so
7f1b126a2000-7f1b126a3000 r--p 0000c000 fd:01 1308307
/lib64/libnss_files-2.12.so
7f1b126a3000-7f1b126a4000 rw-p 0000d000 fd:01 1308307
/lib64/libnss_files-2.12.so
7f1b126a4000-7f1d62b24000 rw-s 00000000 00:04 440003533
/dev/zero (deleted)
7f1d62b24000-7f1d62b2c000 r-xp 00000000 fd:01 659776
/usr/pgsql-9.5/lib/pg_stat_statements.so
7f1d62b2c000-7f1d62d2b000 ---p 00008000 fd:01 659776
/usr/pgsql-9.5/lib/pg_stat_statements.so
7f1d62d2b000-7f1d62d2c000 rw-p 00007000 fd:01 659776
/usr/pgsql-9.5/lib/pg_stat_statements.so
7f1d62d2c000-7f1d68bc1000 r--p 00000000 fd:01 659094
/usr/lib/locale/locale-archive
7f1d68bc1000-7f1d68bde000 r-xp 00000000 fd:01 1308211
/lib64/libselinux.so.1
7f1d68bde000-7f1d68ddd000 ---p 0001d000 fd:01 1308211
/lib64/libselinux.so.1
7f1d68ddd000-7f1d68dde000 r--p 0001c000 fd:01 1308211
/lib64/libselinux.so.1
7f1d68dde000-7f1d68ddf000 rw-p 0001d000 fd:01 1308211
/lib64/libselinux.so.1
7f1d68ddf000-7f1d68de0000 rw-p 00000000 00:00 0
7f1d68de0000-7f1d68de2000 r-xp 00000000 fd:01 1308204
/lib64/libfreebl3.so
7f1d68de2000-7f1d68fe1000 ---p 00002000 fd:01 1308204
/lib64/libfreebl3.so
7f1d68fe1000-7f1d68fe2000 r--p 00001000 fd:01 1308204
/lib64/libfreebl3.so
7f1d68fe2000-7f1d68fe3000 rw-p 00002000 fd:01 1308204
/lib64/libfreebl3.so
7f1d68fe3000-7f1d6901c000 r-xp 00000000 fd:01 1308378
/lib64/libnspr4.so
7f1d6901c000-7f1d6921c000 ---p 00039000 fd:01 1308378
/lib64/libnspr4.so
7f1d6921c000-7f1d6921d000 r--p 00039000 fd:01 1308378
/lib64/libnspr4.so
7f1d6921d000-7f1d6921f000 rw-p 0003a000 fd:01 1308378
/lib64/libnspr4.so
7f1d6921f000-7f1d69221000 rw-p 00000000 00:00 0
7f1d69221000-7f1d69225000 r-xp 00000000 fd:01 1308380
/lib64/libplc4.so
7f1d69225000-7f1d69424000 ---p 00004000 fd:01 1308380
/lib64/libplc4.so
7f1d69424000-7f1d69425000 r--p 00003000 fd:01 1308380
/lib64/libplc4.so
7f1d69425000-7f1d69426000 rw-p 00004000 fd:01 1308380
/lib64/libplc4.so
7f1d69426000-7f1d69429000 r-xp 00000000 fd:01 1308382
/lib64/libplds4.so
7f1d69429000-7f1d69628000 ---p 00003000 fd:01 1308382
/lib64/libplds4.so
7f1d69628000-7f1d69629000 r--p 00002000 fd:01 1308382
/lib64/libplds4.so
7f1d69629000-7f1d6962a000 rw-p 00003000 fd:01 1308382
/lib64/libplds4.so
7f1d6962a000-7f1d69650000 r-xp 00000000 fd:01 656260
/usr/lib64/libnssutil3.so
7f1d69650000-7f1d6984f000 ---p 00026000 fd:01 656260
/usr/lib64/libnssutil3.so
7f1d6984f000-7f1d69856000 r--p 00025000 fd:01 656260
/usr/lib64/libnssutil3.so
7f1d69856000-7f1d69857000 rw-p 0002c000 fd:01 656260
/usr/lib64/libnssutil3.so
7f1d69857000-7f1d69991000 r-xp 00000000 fd:01 656907
/usr/lib64/libnss3.so
7f1d69991000-7f1d69b90000 ---p 0013a000 fd:01 656907
/usr/lib64/libnss3.so
7f1d69b90000-7f1d69b96000 r--p 00139000 fd:01 656907
/usr/lib64/libnss3.so
7f1d69b96000-7f1d69b98000 rw-p 0013f000 fd:01 656907
/usr/lib64/libnss3.so
7f1d69b98000-7f1d69b9a000 rw-p 00000000 00:00 0
7f1d69b9a000-7f1d69bc2000 r-xp 00000000 fd:01 659119
/usr/lib64/libsmime3.so
7f1d69bc2000-7f1d69dc1000 ---p 00028000 fd:01 659119
/usr/lib64/libsmime3.so
7f1d69dc1000-7f1d69dc5000 r--p 00027000 fd:01 659119
/usr/lib64/libsmime3.so
7f1d69dc5000-7f1d69dc6000 rw-p 0002b000 fd:01 659119
/usr/lib64/libsmime3.so
7f1d69dc6000-7f1d69e0d000 r-xp 00000000 fd:01 659120
/usr/lib64/libssl3.so
7f1d69e0d000-7f1d6a00d000 ---p 00047000 fd:01 659120
/usr/lib64/libssl3.so
7f1d6a00d000-7f1d6a011000 r--p 00047000 fd:01 659120
/usr/lib64/libssl3.so
7f1d6a011000-7f1d6a012000 rw-p 0004b000 fd:01 659120
/usr/lib64/libssl3.so
7f1d6a012000-7f1d6a013000 rw-p 00000000 00:00 0
7f1d6a013000-7f1d6a02c000 r-xp 00000000 fd:01 655285
/usr/lib64/libsasl2.so.2.0.23
7f1d6a02c000-7f1d6a22b000 ---p 00019000 fd:01 655285
/usr/lib64/libsasl2.so.2.0.23
7f1d6a22b000-7f1d6a22c000 r--p 00018000 fd:01 655285
/usr/lib64/libsasl2.so.2.0.23
7f1d6a22c000-7f1d6a22d000 rw-p 00019000 fd:01 655285
/usr/lib64/libsasl2.so.2.0.23
7f1d6a22d000-7f1d6a23b000 r-xp 00000000 fd:01 1308168
/lib64/liblber-2.4.so.2.10.3
7f1d6a23b000-7f1d6a43a000 ---p 0000e000 fd:01 1308168
/lib64/liblber-2.4.so.2.10.3
7f1d6a43a000-7f1d6a43b000 r--p 0000d000 fd:01 1308168
/lib64/liblber-2.4.so.2.10.3
7f1d6a43b000-7f1d6a43c000 rw-p 0000e000 fd:01 1308168
/lib64/liblber-2.4.so.2.10.3
7f1d6a43c000-7f1d6a453000 r-xp 00000000 fd:01 1308197
/lib64/libpthread-2.12.so
7f1d6a453000-7f1d6a653000 ---p 00017000 fd:01 1308197
/lib64/libpthread-2.12.so
7f1d6a653000-7f1d6a654000 r--p 00017000 fd:01 1308197
/lib64/libpthread-2.12.so
7f1d6a654000-7f1d6a655000 rw-p 00018000 fd:01 1308197
/lib64/libpthread-2.12.so
7f1d6a655000-7f1d6a659000 rw-p 00000000 00:00 0
7f1d6a659000-7f1d6a66f000 r-xp 00000000 fd:01 1308318
/lib64/libresolv-2.12.so
7f1d6a66f000-7f1d6a86f000 ---p 00016000 fd:01 1308318
/lib64/libresolv-2.12.so
7f1d6a86f000-7f1d6a870000 r--p 00016000 fd:01 1308318
/lib64/libresolv-2.12.so
7f1d6a870000-7f1d6a871000 rw-p 00017000 fd:01 1308318
/lib64/libresolv-2.12.so
7f1d6a871000-7f1d6a873000 rw-p 00000000 00:00 0
7f1d6a873000-7f1d6a875000 r-xp 00000000 fd:01 1308340
/lib64/libkeyutils.so.1.3
7f1d6a875000-7f1d6aa74000 ---p 00002000 fd:01 1308340
/lib64/libkeyutils.so.1.3
7f1d6aa74000-7f1d6aa75000 r--p 00001000 fd:01 1308340
/lib64/libkeyutils.so.1.3
7f1d6aa75000-7f1d6aa76000 rw-p 00002000 fd:01 1308340
/lib64/libkeyutils.so.1.3
7f1d6aa76000-7f1d6aa80000 r-xp 00000000 fd:01 1308358
/lib64/libkrb5support.so.0.1
7f1d6aa80000-7f1d6ac7f000 ---p 0000a000 fd:01 1308358
/lib64/libkrb5support.so.0.1
7f1d6ac7f000-7f1d6ac80000 r--p 00009000 fd:01 1308358
/lib64/libkrb5support.so.0.1
7f1d6ac80000-7f1d6ac81000 rw-p 0000a000 fd:01 1308358
/lib64/libkrb5support.so.0.1
7f1d6ac81000-7f1d6acaa000 r-xp 00000000 fd:01 1308344
/lib64/libk5crypto.so.3.1
7f1d6acaa000-7f1d6aeaa000 ---p 00029000 fd:01 1308344
/lib64/libk5crypto.so.3.1
7f1d6aeaa000-7f1d6aeab000 r--p 00029000 fd:01 1308344
/lib64/libk5crypto.so.3.1
7f1d6aeab000-7f1d6aeac000 rw-p 0002a000 fd:01 1308344
/lib64/libk5crypto.so.3.1
7f1d6aeac000-7f1d6aead000 rw-p 00000000 00:00 0
7f1d6aead000-7f1d6aeb0000 r-xp 00000000 fd:01 1308233
/lib64/libcom_err.so.2.1
7f1d6aeb0000-7f1d6b0af000 ---p 00003000 fd:01 1308233
/lib64/libcom_err.so.2.1
7f1d6b0af000-7f1d6b0b0000 r--p 00002000 fd:01 1308233
/lib64/libcom_err.so.2.1
7f1d6b0b0000-7f1d6b0b1000 rw-p 00003000 fd:01 1308233
/lib64/libcom_err.so.2.1
7f1d6b0b1000-7f1d6b18d000 r-xp 00000000 fd:01 1308355
/lib64/libkrb5.so.3.3
7f1d6b18d000-7f1d6b38c000 ---p 000dc000 fd:01 1308355
/lib64/libkrb5.so.3.3
7f1d6b38c000-7f1d6b396000 r--p 000db000 fd:01 1308355
/lib64/libkrb5.so.3.3
7f1d6b396000-7f1d6b398000 rw-p 000e5000 fd:01 1308355
/lib64/libkrb5.so.3.3
7f1d6b398000-7f1d6b39f000 r-xp 00000000 fd:01 1308177
/lib64/libcrypt-2.12.so
7f1d6b39f000-7f1d6b59f000 ---p 00007000 fd:01 1308177
/lib64/libcrypt-2.12.so
7f1d6b59f000-7f1d6b5a0000 r--p 00007000 fd:01 1308177
/lib64/libcrypt-2.12.so
7f1d6b5a0000-7f1d6b5a1000 rw-p 00008000 fd:01 1308177
/lib64/libcrypt-2.12.so
7f1d6b5a1000-7f1d6b5cf000 rw-p 00000000 00:00 0
7f1d6b5cf000-7f1d6b5e7000 r-xp 00000000 fd:01 1308482
/lib64/libaudit.so.1.0.0
7f1d6b5e7000-7f1d6b7e6000 ---p 00018000 fd:01 1308482
/lib64/libaudit.so.1.0.0
7f1d6b7e6000-7f1d6b7e8000 r--p 00017000 fd:01 1308482
/lib64/libaudit.so.1.0.0
7f1d6b7e8000-7f1d6b7f3000 rw-p 00019000 fd:01 1308482
/lib64/libaudit.so.1.0.0
7f1d6b7f3000-7f1d6b808000 r-xp 00000000 fd:01 1308226
/lib64/libz.so.1.2.3
7f1d6b808000-7f1d6ba07000 ---p 00015000 fd:01 1308226
/lib64/libz.so.1.2.3
7f1d6ba07000-7f1d6ba08000 r--p 00014000 fd:01 1308226
/lib64/libz.so.1.2.3
7f1d6ba08000-7f1d6ba09000 rw-p 00015000 fd:01 1308226
/lib64/libz.so.1.2.3
7f1d6ba09000-7f1d6bb93000 r-xp 00000000 fd:01 1308173
/lib64/libc-2.12.so
7f1d6bb93000-7f1d6bd93000 ---p 0018a000 fd:01 1308173
/lib64/libc-2.12.so
7f1d6bd93000-7f1d6bd97000 r--p 0018a000 fd:01 1308173
/lib64/libc-2.12.so
7f1d6bd97000-7f1d6bd99000 rw-p 0018e000 fd:01 1308173
/lib64/libc-2.12.so
7f1d6bd99000-7f1d6bd9d000 rw-p 00000000 00:00 0
7f1d6bd9d000-7f1d6bdeb000 r-xp 00000000 fd:01 1308613
/lib64/libldap-2.4.so.2.10.3
7f1d6bdeb000-7f1d6bfea000 ---p 0004e000 fd:01 1308613
/lib64/libldap-2.4.so.2.10.3
7f1d6bfea000-7f1d6bfec000 r--p 0004d000 fd:01 1308613
/lib64/libldap-2.4.so.2.10.3
7f1d6bfec000-7f1d6bfee000 rw-p 0004f000 fd:01 1308613
/lib64/libldap-2.4.so.2.10.3
7f1d6bfee000-7f1d6c071000 r-xp 00000000 fd:01 1308290
/lib64/libm-2.12.so
7f1d6c071000-7f1d6c270000 ---p 00083000 fd:01 1308290
/lib64/libm-2.12.so
7f1d6c270000-7f1d6c271000 r--p 00082000 fd:01 1308290
/lib64/libm-2.12.so
7f1d6c271000-7f1d6c272000 rw-p 00083000 fd:01 1308290
/lib64/libm-2.12.so
7f1d6c272000-7f1d6c274000 r-xp 00000000 fd:01 1308282
/lib64/libdl-2.12.so
7f1d6c274000-7f1d6c474000 ---p 00002000 fd:01 1308282
/lib64/libdl-2.12.so
7f1d6c474000-7f1d6c475000 r--p 00002000 fd:01 1308282
/lib64/libdl-2.12.so
7f1d6c475000-7f1d6c476000 rw-p 00003000 fd:01 1308282
/lib64/libdl-2.12.so
7f1d6c476000-7f1d6c47d000 r-xp 00000000 fd:01 1308320
/lib64/librt-2.12.so
7f1d6c47d000-7f1d6c67c000 ---p 00007000 fd:01 1308320
/lib64/librt-2.12.so
7f1d6c67c000-7f1d6c67d000 r--p 00006000 fd:01 1308320
/lib64/librt-2.12.so
7f1d6c67d000-7f1d6c67e000 rw-p 00007000 fd:01 1308320
/lib64/librt-2.12.so
7f1d6c67e000-7f1d6c6bf000 r-xp 00000000 fd:01 1308350
/lib64/libgssapi_krb5.so.2.2
7f1d6c6bf000-7f1d6c8bf000 ---p 00041000 fd:01 1308350
/lib64/libgssapi_krb5.so.2.2
7f1d6c8bf000-7f1d6c8c0000 r--p 00041000 fd:01 1308350
/lib64/libgssapi_krb5.so.2.2
7f1d6c8c0000-7f1d6c8c2000 rw-p 00042000 fd:01 1308350
/lib64/libgssapi_krb5.so.2.2
7f1d6c8c2000-7f1d6ca7c000 r-xp 00000000 fd:01 656262
/usr/lib64/libcrypto.so.1.0.1e
7f1d6ca7c000-7f1d6cc7c000 ---p 001ba000 fd:01 656262
/usr/lib64/libcrypto.so.1.0.1e
7f1d6cc7c000-7f1d6cc97000 r--p 001ba000 fd:01 656262
/usr/lib64/libcrypto.so.1.0.1e
7f1d6cc97000-7f1d6cca3000 rw-p 001d5000 fd:01 656262
/usr/lib64/libcrypto.so.1.0.1e
7f1d6cca3000-7f1d6cca7000 rw-p 00000000 00:00 0
7f1d6cca7000-7f1d6cd09000 r-xp 00000000 fd:01 659112
/usr/lib64/libssl.so.1.0.1e
7f1d6cd09000-7f1d6cf09000 ---p 00062000 fd:01 659112
/usr/lib64/libssl.so.1.0.1e
7f1d6cf09000-7f1d6cf0d000 r--p 00062000 fd:01 659112
/usr/lib64/libssl.so.1.0.1e
7f1d6cf0d000-7f1d6cf13000 rw-p 00066000 fd:01 659112
/usr/lib64/libssl.so.1.0.1e
7f1d6cf13000-7f1d6cf1f000 r-xp 00000000 fd:01 1308338
/lib64/libpam.so.0.82.2
7f1d6cf1f000-7f1d6d11f000 ---p 0000c000 fd:01 1308338
/lib64/libpam.so.0.82.2
7f1d6d11f000-7f1d6d120000 r--p 0000c000 fd:01 1308338
/lib64/libpam.so.0.82.2
7f1d6d120000-7f1d6d121000 rw-p 0000d000 fd:01 1308338
/lib64/libpam.so.0.82.2
7f1d6d121000-7f1d6d26a000 r-xp 00000000 fd:01 655397
/usr/lib64/libxml2.so.2.7.6
7f1d6d26a000-7f1d6d469000 ---p 00149000 fd:01 655397
/usr/lib64/libxml2.so.2.7.6
7f1d6d469000-7f1d6d473000 rw-p 00148000 fd:01 655397
/usr/lib64/libxml2.so.2.7.6
7f1d6d473000-7f1d6d474000 rw-p 00000000 00:00 0
7f1d6d474000-7f1d6d494000 r-xp 00000000 fd:01 1308202
/lib64/ld-2.12.so
7f1d6d578000-7f1d6d5f9000 rw-p 00000000 00:00 0
7f1d6d61a000-7f1d6d67c000 rw-p 00000000 00:00 0
7f1d6d67c000-7f1d6d68d000 rw-p 00000000 00:00 0
7f1d6d68e000-7f1d6d68f000 rw-p 00000000 00:00 0
7f1d6d68f000-7f1d6d691000 rw-s 00000000 00:0f 440003629
/dev/shm/PostgreSQL.1035112127
7f1d6d691000-7f1d6d692000 rw-s 00000000 00:04 196609
/SYSV0052e2c1 (deleted)
7f1d6d692000-7f1d6d693000 rw-p 00000000 00:00 0
7f1d6d693000-7f1d6d695000 r--p 0001f000 fd:01 1308202
/lib64/ld-2.12.so
7f1d6d695000-7f1d6d696000 rw-p 00021000 fd:01 1308202
/lib64/ld-2.12.so
7f1d6d696000-7f1d6d697000 rw-p 00000000 00:00 0
7ffd0ee14000-7ffd0ee29000 rw-p 00000000 00:00 0
[stack]
7ffd0efc1000-7ffd0efc2000 r-xp 00000000 00:00 0
[vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]
2017-05-28 17:54:54 BST [2886]: [16-1] user=,db=,app=,client= LOG: server
process (PID 14749) was terminated by signal 6: Aborted
2017-05-28 17:54:54 BST [2886]: [17-1] user=,db=,app=,client= DETAIL:
Failed process was running: SELECT cpl_metadata.uuid AS cpl_metadata_uuid,
cpl_metadata.created AS cpl_metadata_created, cpl_metadata.last_modified AS
cpl_metadata_la
st_modified, cpl_metadata.cpl_uuid AS cpl_metadata_cpl_uuid,
cpl_metadata.name AS cpl_metadata_name, cpl_metadata.date AS
cpl_metadata_date, cpl_metadata.content_kind AS cpl_metadata_content_kind,
cpl_metadata.credits_offset AS cpl_metad
ata_credits_offset, cpl_metadata.intermission_offset AS
cpl_metadata_intermission_offset, cpl_metadata.playback_mode AS
cpl_metadata_playback_mode, cpl_metadata.rating_hardlocked AS
cpl_metadata_rating_hardlocked, cpl_metadata.territory
AS cpl_metadata_territory, cpl_metadata.rating AS cpl_metadata_rating,
cpl_metadata.audio_type AS cpl_metadata_audio_type,
cpl_metadata.audio_language AS cpl_metadata_audio_language,
cpl_metadata.subtitled AS cpl_metadata_subtitled, cpl_
metadata.subtitle_language AS cpl_metadata_subtitle_language,
cpl_metadata.accessibility AS cpl_metadata_accessibility,
cpl_metadata.aspect_ratio AS cpl_metadata_aspect_ratio
2017-05-28 17:54:54 BST [2886]: [18-1] user=,db=,app=,client= LOG:
terminating any other active server processes
2017-05-28 17:54:54 BST [14905]: [4-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.100 WARNING:
terminating connection because of crash of another server process
2017-05-28 17:54:54 BST [14905]: [5-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.100 DETAIL: The
postmaster has commanded this server process to roll back the current
transaction and exit, because another server process exitedabnormally and possibly corrupted shared memory.
2017-05-28 17:54:54 BST [14905]: [6-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.100 HINT: In a moment
you should be able to reconnect to the database and repeat your command.
2017-05-28 17:54:54 BST [14894]: [4-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.68 WARNING: terminating
connection because of crash of another server process
2017-05-28 17:54:54 BST [14894]: [5-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.68 DETAIL: The
postmaster has commanded this server process to roll back the current
transaction and exit, because another server process exited
abnormally and possibly corrupted shared memory.
2017-05-28 17:54:54 BST [14894]: [6-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.68 HINT: In a moment
you should be able to reconnect to the database and repeat your command.
2017-05-28 17:54:54 BST [14887]: [4-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.65 WARNING: terminating
connection because of crash of another server process
2017-05-28 17:54:54 BST [14887]: [5-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.65 DETAIL: The
postmaster has commanded this server process to roll back the current
transaction and exit, because another server process exited
abnormally and possibly corrupted shared memory.
2017-05-28 17:54:54 BST [14887]: [6-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.65 HINT: In a moment
you should be able to reconnect to the database and repeat your command.
2017-05-28 17:54:54 BST [14884]: [5-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.110 WARNING:
terminating connection because of crash of another server process
2017-05-28 17:54:54 BST [14884]: [6-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.110 DETAIL: The
postmaster has commanded this server process to roll back the current
transaction and exit, because another server process exitedabnormally and possibly corrupted shared memory.
2017-05-28 17:54:54 BST [14884]: [7-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.110 HINT: In a moment
you should be able to reconnect to the database and repeat your command.
2017-05-28 17:54:54 BST [14884]: [8-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.110 CONTEXT: SQL
statement "UPDATE monitoring_log SET search_data = sdq.search_data
2017-05-28 17:54:57 BST [15007]: [2-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.111 FATAL: the database
system is in recovery mode
2017-05-28 17:54:57 BST [15004]: [2-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.111 FATAL: the database
system is in recovery mode
2017-05-28 17:54:57 BST [15010]: [1-1]
user=[unknown],db=[unknown],app=[unknown],client=172.30.8.110 LOG:
connection received: host=172.30.8.110 port=52270
2017-05-28 17:54:57 BST [15008]: [2-1]
user=postgres,db=etms,app=[unknown],client=172.30.8.111 FATAL: the database
system is in recovery mode
2017-05-28 17:54:57 BST [15005]: [1-1]
user=[unknown],db=[unknown],app=[unknown],client=172.30.8.110 LOG:
connection received: host=172.30.8.110 port=52268
2017-05-28 17:54:57 BST [15009]: [1-1]
user=[unknown],db=[unknown],app=[unknown],client=172.30.8.67 LOG:
connection received: host=172.30.8.67 port=48328
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
Re: [BUGS] BUG #14683: *** glibc detected *** SELECT: double free or corruption
From
Michael Paquier
Date:
On Thu, Jun 1, 2017 at 1:11 AM, <ray.warren@artsalliancemedia.com> wrote: > We are experiencing some performance issues and we had this error on Sunday. > Not sure whether the 2 things are related or not as we have been making some > changes to the application and load. However, I assume the double free or > corruption error is a bug? Could you come up with a self-contained test case? It will be difficult to dig into this issue otherwise. -- Michael -- Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-bugs
Re: [BUGS] BUG #14683: *** glibc detected *** SELECT: double freeor corruption
From
Ray Warren
Date:
Hi Michael, I can't create a self-contained test case. This seems to be a random crash. It is happening daily at different time of day.The SQL running at the time isn't the same. Is this likely a bug in Postgres? Could we have something misconfigured? This is a production system so we need to get thisresolved. Thanks ... Ray -----Original Message----- From: Michael Paquier [mailto:michael.paquier@gmail.com] Sent: 02 June 2017 02:18 To: Ray Warren Cc: PostgreSQL mailing lists Subject: Re: [BUGS] BUG #14683: *** glibc detected *** SELECT: double free or corruption On Thu, Jun 1, 2017 at 1:11 AM, <ray.warren@artsalliancemedia.com> wrote: > We are experiencing some performance issues and we had this error on Sunday. > Not sure whether the 2 things are related or not as we have been > making some changes to the application and load. However, I assume the > double free or corruption error is a bug? Could you come up with a self-contained test case? It will be difficult to dig into this issue otherwise. -- Michael -- Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-bugs
Re: [BUGS] BUG #14683: *** glibc detected *** SELECT: double freeor corruption
From
Ray Warren
Date:
Hi Michael, In fact I have noticed that the other error is different:- *** glibc detected *** postgres: postgres etms 172.30.8.63(53338) SELECT: corrupted double-linked list: 0x00000000034119a0*** 2017-05-31 04:04:41 BST [2703]: [7195-1] user=postgres,db=etms,app=[unknown],client=172.30.8.67 LOG: duration: 15.793 ms statement: SELECT playback.uuid AS playback_uuid, playback.created AS playback_created, playback.last_modified AS playback_last_modified, playback.deleted AS playback_deleted, playback.complex_id AS playback_complex_id, playback.startAS playback_start, playback."end" AS playback_end, playback.duration AS playback_duration, playback.intermission_sta rt AS playback_intermission_start, playback.intermission_end AS playback_intermission_end, playback.accuracy AS playback_accuracy,playback.screen_identifier AS playback_screen_identifier, playback.dnqualifier AS playback_dnqualifier,pl ayback.device_ip_address AS playback_device_ip_address, playback.serial AS playback_serial, playback.complex_identifier ASplayback_complex_identifier, playback.feature_cpl_uuid AS playback_feature_cpl_uuid, playback.cpl_uuid AS playback _cpl_uuid, playback.merge_error_code AS playback_merge_error_code, playback.worst_confidence_level AS playback_worst_confidence_level,playback.verified AS playback_verified, playback.tms_last_modified AS playback_tms_last_modified,play back.tms_version AS playback_tms_version, playback.pack_uuid AS playback_pack_uuid, playback.pos_uuid AS playback_pos_uuid,playback.seats_sold AS playback_seats_sold, playback.meta AS playback_meta, playback.seq_id AS playback_seq_id FROM playback WHERE playback.uuid = '89cb645e-72e4-4570-add8-c797a1391e76' 2017-05-31 04:04:41 BST [8011]: [1350-1] user=postgres,db=etms,app=[unknown],client=172.30.8.67 LOG: duration: 15.925 ms statement: SELECT playback.uuid AS playback_uuid, playback.created AS playback_created, playback.last_modified AS playback_last_modified, playback.deleted AS playback_deleted, playback.complex_id AS playback_complex_id, playback.startAS playback_start, playback."end" AS playback_end, playback.duration AS playback_duration, playback.intermission_sta rt AS playback_intermission_start, playback.intermission_end AS playback_intermission_end, playback.accuracy AS playback_accuracy,playback.screen_identifier AS playback_screen_identifier, playback.dnqualifier AS playback_dnqualifier,pl ayback.device_ip_address AS playback_device_ip_address, playback.serial AS playback_serial, playback.complex_identifier ASplayback_complex_identifier, playback.feature_cpl_uuid AS playback_feature_cpl_uuid, playback.cpl_uuid AS playback _cpl_uuid, playback.merge_error_code AS playback_merge_error_code, playback.worst_confidence_level AS playback_worst_confidence_level,playback.verified AS playback_verified, playback.tms_last_modified AS playback_tms_last_modified,play back.tms_version AS playback_tms_version, playback.pack_uuid AS playback_pack_uuid, playback.pos_uuid AS playback_pos_uuid,playback.seats_sold AS playback_seats_sold, playback.meta AS playback_meta, playback.seq_id AS playback_seq_id FROM playback WHERE playback.uuid = '1f29ddf2-3266-495e-ac33-2a21d00e4a1d' ======= Backtrace: ========= /lib64/libc.so.6(+0x75dee)[0x7f1d6ba7edee] /lib64/libc.so.6(+0x78ff6)[0x7f1d6ba81ff6] /lib64/libc.so.6(+0x7babe)[0x7f1d6ba84abe] /lib64/libc.so.6(realloc+0x158)[0x7f1d6ba84f08] postgres: postgres etms 172.30.8.63(53338) SELECT[0x7c8d72] postgres: postgres etms 172.30.8.63(53338) SELECT(repalloc+0x25)[0x7c9295] postgres: postgres etms 172.30.8.63(53338) SELECT(enlargeStringInfo+0x4e)[0x5e505e] postgres: postgres etms 172.30.8.63(53338) SELECT(appendStringInfoChar+0x28)[0x5e5178] postgres: postgres etms 172.30.8.63(53338) SELECT[0x7a6724] postgres: postgres etms 172.30.8.63(53338) SELECT(EmitErrorReport+0xa2f)[0x7a963f] postgres: postgres etms 172.30.8.63(53338) SELECT(errfinish+0x8d)[0x7a97dd] postgres: postgres etms 172.30.8.63(53338) SELECT(FileClose+0x1c5)[0x6966d5] postgres: postgres etms 172.30.8.63(53338) SELECT(BufFileClose+0x32)[0x698652] postgres: postgres etms 172.30.8.63(53338) SELECT(ExecHashJoin+0x31b)[0x5ce60b] postgres: postgres etms 172.30.8.63(53338) SELECT(ExecProcNode+0x168)[0x5bc9f8] postgres: postgres etms 172.30.8.63(53338) SELECT(ExecSort+0xf9)[0x5d86c9] postgres: postgres etms 172.30.8.63(53338) SELECT(ExecProcNode+0x148)[0x5bc9d8] postgres: postgres etms 172.30.8.63(53338) SELECT(ExecLimit+0xa8)[0x5d13a8] postgres: postgres etms 172.30.8.63(53338) SELECT(ExecProcNode+0x90)[0x5bc920] postgres: postgres etms 172.30.8.63(53338) SELECT(standard_ExecutorRun+0xd2)[0x5bb2c2] /usr/pgsql-9.5/lib/pg_stat_statements.so(+0x448b)[0x7f1d62b2848b] postgres: postgres etms 172.30.8.63(53338) SELECT[0x6bd417] postgres: postgres etms 172.30.8.63(53338) SELECT(PortalRun+0x188)[0x6be5d8] postgres: postgres etms 172.30.8.63(53338) SELECT[0x6bb3be] postgres: postgres etms 172.30.8.63(53338) SELECT(PostgresMain+0xd35)[0x6bca25] postgres: postgres etms 172.30.8.63(53338) SELECT(PostmasterMain+0x1875)[0x663165] postgres: postgres etms 172.30.8.63(53338) SELECT(main+0x7a8)[0x5f4b28] /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f1d6ba27d1d] postgres: postgres etms 172.30.8.63(53338) SELECT[0x467549] ======= Memory map: ======== 00400000-009cf000 r-xp 00000000 fd:01 788841 /usr/pgsql-9.5/bin/postgres 00bcf000-00bdc000 rw-p 005cf000 fd:01 788841 /usr/pgsql-9.5/bin/postgres 00bdc000-00c2b000 rw-p 00000000 00:00 0 0173e000-01785000 rw-p 00000000 00:00 0 01785000-0362a000 rw-p 00000000 00:00 0 7f1b0c000000-7f1b0c021000 rw-p 00000000 00:00 0 7f1b0c021000-7f1b10000000 ---p 00000000 00:00 0 7f1b12280000-7f1b12296000 r-xp 00000000 fd:01 1308180 /lib64/libgcc_s-4.4.7-20120601.so.1 7f1b12296000-7f1b12495000 ---p 00016000 fd:01 1308180 /lib64/libgcc_s-4.4.7-20120601.so.1 7f1b12495000-7f1b12496000 rw-p 00015000 fd:01 1308180 /lib64/libgcc_s-4.4.7-20120601.so.1 7f1b12496000-7f1b124a3000 r-xp 00000000 fd:01 1308307 /lib64/libnss_files-2.12.so 7f1b124a3000-7f1b126a2000 ---p 0000d000 fd:01 1308307 /lib64/libnss_files-2.12.so 7f1b126a2000-7f1b126a3000 r--p 0000c000 fd:01 1308307 /lib64/libnss_files-2.12.so 7f1b126a3000-7f1b126a4000 rw-p 0000d000 fd:01 1308307 /lib64/libnss_files-2.12.so 7f1b126a4000-7f1d62b24000 rw-s 00000000 00:04 492918692 /dev/zero (deleted) 7f1d62b24000-7f1d62b2c000 r-xp 00000000 fd:01 659776 /usr/pgsql-9.5/lib/pg_stat_statements.so 7f1d62b2c000-7f1d62d2b000 ---p 00008000 fd:01 659776 /usr/pgsql-9.5/lib/pg_stat_statements.so 7f1d62d2b000-7f1d62d2c000 rw-p 00007000 fd:01 659776 /usr/pgsql-9.5/lib/pg_stat_statements.so 7f1d62d2c000-7f1d68bc1000 r--p 00000000 fd:01 659094 /usr/lib/locale/locale-archive 7f1d68bc1000-7f1d68bde000 r-xp 00000000 fd:01 1308211 /lib64/libselinux.so.1 7f1d68bde000-7f1d68ddd000 ---p 0001d000 fd:01 1308211 /lib64/libselinux.so.1 7f1d68ddd000-7f1d68dde000 r--p 0001c000 fd:01 1308211 /lib64/libselinux.so.1 7f1d68dde000-7f1d68ddf000 rw-p 0001d000 fd:01 1308211 /lib64/libselinux.so.1 7f1d68ddf000-7f1d68de0000 rw-p 00000000 00:00 0 7f1d68de0000-7f1d68de2000 r-xp 00000000 fd:01 1308204 /lib64/libfreebl3.so 7f1d68de2000-7f1d68fe1000 ---p 00002000 fd:01 1308204 /lib64/libfreebl3.so 7f1d68fe1000-7f1d68fe2000 r--p 00001000 fd:01 1308204 /lib64/libfreebl3.so 7f1d68fe2000-7f1d68fe3000 rw-p 00002000 fd:01 1308204 /lib64/libfreebl3.so 7f1d68fe3000-7f1d6901c000 r-xp 00000000 fd:01 1308378 /lib64/libnspr4.so 7f1d6901c000-7f1d6921c000 ---p 00039000 fd:01 1308378 /lib64/libnspr4.so 7f1d6921c000-7f1d6921d000 r--p 00039000 fd:01 1308378 /lib64/libnspr4.so 7f1d6921d000-7f1d6921f000 rw-p 0003a000 fd:01 1308378 /lib64/libnspr4.so 7f1d6921f000-7f1d69221000 rw-p 00000000 00:00 0 7f1d69221000-7f1d69225000 r-xp 00000000 fd:01 1308380 /lib64/libplc4.so 7f1d69225000-7f1d69424000 ---p 00004000 fd:01 1308380 /lib64/libplc4.so 7f1d69424000-7f1d69425000 r--p 00003000 fd:01 1308380 /lib64/libplc4.so 7f1d69425000-7f1d69426000 rw-p 00004000 fd:01 1308380 /lib64/libplc4.so 7f1d69426000-7f1d69429000 r-xp 00000000 fd:01 1308382 /lib64/libplds4.so 7f1d69429000-7f1d69628000 ---p 00003000 fd:01 1308382 /lib64/libplds4.so 7f1d69628000-7f1d69629000 r--p 00002000 fd:01 1308382 /lib64/libplds4.so 7f1d69629000-7f1d6962a000 rw-p 00003000 fd:01 1308382 /lib64/libplds4.so 7f1d6962a000-7f1d69650000 r-xp 00000000 fd:01 656260 /usr/lib64/libnssutil3.so 7f1d69650000-7f1d6984f000 ---p 00026000 fd:01 656260 /usr/lib64/libnssutil3.so 7f1d6984f000-7f1d69856000 r--p 00025000 fd:01 656260 /usr/lib64/libnssutil3.so 7f1d69856000-7f1d69857000 rw-p 0002c000 fd:01 656260 /usr/lib64/libnssutil3.so 7f1d69857000-7f1d69991000 r-xp 00000000 fd:01 656907 /usr/lib64/libnss3.so 7f1d69991000-7f1d69b90000 ---p 0013a000 fd:01 656907 /usr/lib64/libnss3.so 7f1d69b90000-7f1d69b96000 r--p 00139000 fd:01 656907 /usr/lib64/libnss3.so 7f1d69b96000-7f1d69b98000 rw-p 0013f000 fd:01 656907 /usr/lib64/libnss3.so 7f1d69b98000-7f1d69b9a000 rw-p 00000000 00:00 0 7f1d69b9a000-7f1d69bc2000 r-xp 00000000 fd:01 659119 /usr/lib64/libsmime3.so 7f1d69bc2000-7f1d69dc1000 ---p 00028000 fd:01 659119 /usr/lib64/libsmime3.so 7f1d69dc1000-7f1d69dc5000 r--p 00027000 fd:01 659119 /usr/lib64/libsmime3.so 7f1d69dc5000-7f1d69dc6000 rw-p 0002b000 fd:01 659119 /usr/lib64/libsmime3.so 7f1d69dc6000-7f1d69e0d000 r-xp 00000000 fd:01 659120 /usr/lib64/libssl3.so 7f1d69e0d000-7f1d6a00d000 ---p 00047000 fd:01 659120 /usr/lib64/libssl3.so 7f1d6a00d000-7f1d6a011000 r--p 00047000 fd:01 659120 /usr/lib64/libssl3.so 7f1d6a011000-7f1d6a012000 rw-p 0004b000 fd:01 659120 /usr/lib64/libssl3.so 7f1d6a012000-7f1d6a013000 rw-p 00000000 00:00 0 7f1d6a013000-7f1d6a02c000 r-xp 00000000 fd:01 655285 /usr/lib64/libsasl2.so.2.0.23 7f1d6a02c000-7f1d6a22b000 ---p 00019000 fd:01 655285 /usr/lib64/libsasl2.so.2.0.23 7f1d6a22b000-7f1d6a22c000 r--p 00018000 fd:01 655285 /usr/lib64/libsasl2.so.2.0.23 7f1d6a22c000-7f1d6a22d000 rw-p 00019000 fd:01 655285 /usr/lib64/libsasl2.so.2.0.23 7f1d6a22d000-7f1d6a23b000 r-xp 00000000 fd:01 1308168 /lib64/liblber-2.4.so.2.10.3 7f1d6a23b000-7f1d6a43a000 ---p 0000e000 fd:01 1308168 /lib64/liblber-2.4.so.2.10.3 7f1d6a43a000-7f1d6a43b000 r--p 0000d000 fd:01 1308168 /lib64/liblber-2.4.so.2.10.3 7f1d6a43b000-7f1d6a43c000 rw-p 0000e000 fd:01 1308168 /lib64/liblber-2.4.so.2.10.3 7f1d6a43c000-7f1d6a453000 r-xp 00000000 fd:01 1308197 /lib64/libpthread-2.12.so 7f1d6a453000-7f1d6a653000 ---p 00017000 fd:01 1308197 /lib64/libpthread-2.12.so 7f1d6a653000-7f1d6a654000 r--p 00017000 fd:01 1308197 /lib64/libpthread-2.12.so 7f1d6a654000-7f1d6a655000 rw-p 00018000 fd:01 1308197 /lib64/libpthread-2.12.so 7f1d6a655000-7f1d6a659000 rw-p 00000000 00:00 0 7f1d6a659000-7f1d6a66f000 r-xp 00000000 fd:01 1308318 /lib64/libresolv-2.12.so 7f1d6a66f000-7f1d6a86f000 ---p 00016000 fd:01 1308318 /lib64/libresolv-2.12.so 7f1d6a86f000-7f1d6a870000 r--p 00016000 fd:01 1308318 /lib64/libresolv-2.12.so 7f1d6a870000-7f1d6a871000 rw-p 00017000 fd:01 1308318 /lib64/libresolv-2.12.so 7f1d6a871000-7f1d6a873000 rw-p 00000000 00:00 0 7f1d6a873000-7f1d6a875000 r-xp 00000000 fd:01 1308340 /lib64/libkeyutils.so.1.3 7f1d6a875000-7f1d6aa74000 ---p 00002000 fd:01 1308340 /lib64/libkeyutils.so.1.3 7f1d6aa74000-7f1d6aa75000 r--p 00001000 fd:01 1308340 /lib64/libkeyutils.so.1.3 7f1d6aa75000-7f1d6aa76000 rw-p 00002000 fd:01 1308340 /lib64/libkeyutils.so.1.3 7f1d6aa76000-7f1d6aa80000 r-xp 00000000 fd:01 1308358 /lib64/libkrb5support.so.0.1 7f1d6aa80000-7f1d6ac7f000 ---p 0000a000 fd:01 1308358 /lib64/libkrb5support.so.0.1 7f1d6ac7f000-7f1d6ac80000 r--p 00009000 fd:01 1308358 /lib64/libkrb5support.so.0.1 7f1d6ac80000-7f1d6ac81000 rw-p 0000a000 fd:01 1308358 /lib64/libkrb5support.so.0.1 7f1d6ac81000-7f1d6acaa000 r-xp 00000000 fd:01 1308344 /lib64/libk5crypto.so.3.1 7f1d6acaa000-7f1d6aeaa000 ---p 00029000 fd:01 1308344 /lib64/libk5crypto.so.3.1 7f1d6aeaa000-7f1d6aeab000 r--p 00029000 fd:01 1308344 /lib64/libk5crypto.so.3.1 7f1d6aeab000-7f1d6aeac000 rw-p 0002a000 fd:01 1308344 /lib64/libk5crypto.so.3.1 7f1d6aeac000-7f1d6aead000 rw-p 00000000 00:00 0 7f1d6aead000-7f1d6aeb0000 r-xp 00000000 fd:01 1308233 /lib64/libcom_err.so.2.1 7f1d6aeb0000-7f1d6b0af000 ---p 00003000 fd:01 1308233 /lib64/libcom_err.so.2.1 7f1d6b0af000-7f1d6b0b0000 r--p 00002000 fd:01 1308233 /lib64/libcom_err.so.2.1 7f1d6b0b0000-7f1d6b0b1000 rw-p 00003000 fd:01 1308233 /lib64/libcom_err.so.2.1 7f1d6b0b1000-7f1d6b18d000 r-xp 00000000 fd:01 1308355 /lib64/libkrb5.so.3.3 7f1d6b18d000-7f1d6b38c000 ---p 000dc000 fd:01 1308355 /lib64/libkrb5.so.3.3 7f1d6b38c000-7f1d6b396000 r--p 000db000 fd:01 1308355 /lib64/libkrb5.so.3.3 7f1d6b396000-7f1d6b398000 rw-p 000e5000 fd:01 1308355 /lib64/libkrb5.so.3.3 7f1d6b398000-7f1d6b39f000 r-xp 00000000 fd:01 1308177 /lib64/libcrypt-2.12.so 7f1d6b39f000-7f1d6b59f000 ---p 00007000 fd:01 1308177 /lib64/libcrypt-2.12.so 7f1d6b59f000-7f1d6b5a0000 r--p 00007000 fd:01 1308177 /lib64/libcrypt-2.12.so 7f1d6b5a0000-7f1d6b5a1000 rw-p 00008000 fd:01 1308177 /lib64/libcrypt-2.12.so 7f1d6b5a1000-7f1d6b5cf000 rw-p 00000000 00:00 0 7f1d6b5cf000-7f1d6b5e7000 r-xp 00000000 fd:01 1308482 /lib64/libaudit.so.1.0.0 7f1d6b5e7000-7f1d6b7e6000 ---p 00018000 fd:01 1308482 /lib64/libaudit.so.1.0.0 7f1d6b7e6000-7f1d6b7e8000 r--p 00017000 fd:01 1308482 /lib64/libaudit.so.1.0.0 7f1d6b7e8000-7f1d6b7f3000 rw-p 00019000 fd:01 1308482 /lib64/libaudit.so.1.0.0 7f1d6b7f3000-7f1d6b808000 r-xp 00000000 fd:01 1308226 /lib64/libz.so.1.2.3 7f1d6b808000-7f1d6ba07000 ---p 00015000 fd:01 1308226 /lib64/libz.so.1.2.3 7f1d6ba07000-7f1d6ba08000 r--p 00014000 fd:01 1308226 /lib64/libz.so.1.2.3 7f1d6ba08000-7f1d6ba09000 rw-p 00015000 fd:01 1308226 /lib64/libz.so.1.2.3 7f1d6ba09000-7f1d6bb93000 r-xp 00000000 fd:01 1308173 /lib64/libc-2.12.so 7f1d6bb93000-7f1d6bd93000 ---p 0018a000 fd:01 1308173 /lib64/libc-2.12.so 7f1d6bd93000-7f1d6bd97000 r--p 0018a000 fd:01 1308173 /lib64/libc-2.12.so 7f1d6bd97000-7f1d6bd99000 rw-p 0018e000 fd:01 1308173 /lib64/libc-2.12.so 7f1d6bd99000-7f1d6bd9d000 rw-p 00000000 00:00 0 7f1d6bd9d000-7f1d6bdeb000 r-xp 00000000 fd:01 1308613 /lib64/libldap-2.4.so.2.10.3 7f1d6bdeb000-7f1d6bfea000 ---p 0004e000 fd:01 1308613 /lib64/libldap-2.4.so.2.10.3 7f1d6bfea000-7f1d6bfec000 r--p 0004d000 fd:01 1308613 /lib64/libldap-2.4.so.2.10.3 7f1d6bfec000-7f1d6bfee000 rw-p 0004f000 fd:01 1308613 /lib64/libldap-2.4.so.2.10.3 7f1d6bfee000-7f1d6c071000 r-xp 00000000 fd:01 1308290 /lib64/libm-2.12.so 7f1d6c071000-7f1d6c270000 ---p 00083000 fd:01 1308290 /lib64/libm-2.12.so 7f1d6c270000-7f1d6c271000 r--p 00082000 fd:01 1308290 /lib64/libm-2.12.so 7f1d6c271000-7f1d6c272000 rw-p 00083000 fd:01 1308290 /lib64/libm-2.12.so 7f1d6c272000-7f1d6c274000 r-xp 00000000 fd:01 1308282 /lib64/libdl-2.12.so 7f1d6c274000-7f1d6c474000 ---p 00002000 fd:01 1308282 /lib64/libdl-2.12.so 7f1d6c474000-7f1d6c475000 r--p 00002000 fd:01 1308282 /lib64/libdl-2.12.so 7f1d6c475000-7f1d6c476000 rw-p 00003000 fd:01 1308282 /lib64/libdl-2.12.so 7f1d6c476000-7f1d6c47d000 r-xp 00000000 fd:01 1308320 /lib64/librt-2.12.so 7f1d6c47d000-7f1d6c67c000 ---p 00007000 fd:01 1308320 /lib64/librt-2.12.so 7f1d6c67c000-7f1d6c67d000 r--p 00006000 fd:01 1308320 /lib64/librt-2.12.so 7f1d6c67d000-7f1d6c67e000 rw-p 00007000 fd:01 1308320 /lib64/librt-2.12.so 7f1d6c67e000-7f1d6c6bf000 r-xp 00000000 fd:01 1308350 /lib64/libgssapi_krb5.so.2.2 7f1d6c6bf000-7f1d6c8bf000 ---p 00041000 fd:01 1308350 /lib64/libgssapi_krb5.so.2.2 7f1d6c8bf000-7f1d6c8c0000 r--p 00041000 fd:01 1308350 /lib64/libgssapi_krb5.so.2.2 7f1d6c8c0000-7f1d6c8c2000 rw-p 00042000 fd:01 1308350 /lib64/libgssapi_krb5.so.2.2 7f1d6c8c2000-7f1d6ca7c000 r-xp 00000000 fd:01 656262 /usr/lib64/libcrypto.so.1.0.1e 7f1d6ca7c000-7f1d6cc7c000 ---p 001ba000 fd:01 656262 /usr/lib64/libcrypto.so.1.0.1e 7f1d6cc7c000-7f1d6cc97000 r--p 001ba000 fd:01 656262 /usr/lib64/libcrypto.so.1.0.1e 7f1d6cc97000-7f1d6cca3000 rw-p 001d5000 fd:01 656262 /usr/lib64/libcrypto.so.1.0.1e 7f1d6cca3000-7f1d6cca7000 rw-p 00000000 00:00 0 7f1d6cca7000-7f1d6cd09000 r-xp 00000000 fd:01 659112 /usr/lib64/libssl.so.1.0.1e 7f1d6cd09000-7f1d6cf09000 ---p 00062000 fd:01 659112 /usr/lib64/libssl.so.1.0.1e 7f1d6cf09000-7f1d6cf0d000 r--p 00062000 fd:01 659112 /usr/lib64/libssl.so.1.0.1e 7f1d6cf0d000-7f1d6cf13000 rw-p 00066000 fd:01 659112 /usr/lib64/libssl.so.1.0.1e 7f1d6cf13000-7f1d6cf1f000 r-xp 00000000 fd:01 1308338 /lib64/libpam.so.0.82.2 7f1d6cf1f000-7f1d6d11f000 ---p 0000c000 fd:01 1308338 /lib64/libpam.so.0.82.2 7f1d6d11f000-7f1d6d120000 r--p 0000c000 fd:01 1308338 /lib64/libpam.so.0.82.2 7f1d6d120000-7f1d6d121000 rw-p 0000d000 fd:01 1308338 /lib64/libpam.so.0.82.2 7f1d6d121000-7f1d6d26a000 r-xp 00000000 fd:01 655397 /usr/lib64/libxml2.so.2.7.6 7f1d6d26a000-7f1d6d469000 ---p 00149000 fd:01 655397 /usr/lib64/libxml2.so.2.7.6 7f1d6d469000-7f1d6d473000 rw-p 00148000 fd:01 655397 /usr/lib64/libxml2.so.2.7.6 7f1d6d473000-7f1d6d474000 rw-p 00000000 00:00 0 7f1d6d474000-7f1d6d494000 r-xp 00000000 fd:01 1308202 /lib64/ld-2.12.so 7f1d6d498000-7f1d6d61a000 rw-p 00000000 00:00 0 7f1d6d63b000-7f1d6d67c000 rw-p 00000000 00:00 0 7f1d6d67c000-7f1d6d68d000 rw-p 00000000 00:00 0 7f1d6d68e000-7f1d6d68f000 rw-p 00000000 00:00 0 7f1d6d68f000-7f1d6d691000 rw-s 00000000 00:0f 492918696 /dev/shm/PostgreSQL.1856781077 7f1d6d691000-7f1d6d692000 rw-s 00000000 00:04 229377 /SYSV0052e2c1 (deleted) 7f1d6d692000-7f1d6d693000 rw-p 00000000 00:00 0 7f1d6d693000-7f1d6d695000 r--p 0001f000 fd:01 1308202 /lib64/ld-2.12.so 7f1d6d695000-7f1d6d696000 rw-p 00021000 fd:01 1308202 /lib64/ld-2.12.so 7f1d6d696000-7f1d6d697000 rw-p 00000000 00:00 0 7ffd0ee14000-7ffd0ee29000 rw-p 00000000 00:00 0 [stack] 7ffd0efc1000-7ffd0efc2000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] -----Original Message----- From: Michael Paquier [mailto:michael.paquier@gmail.com] Sent: 02 June 2017 02:18 To: Ray Warren Cc: PostgreSQL mailing lists Subject: Re: [BUGS] BUG #14683: *** glibc detected *** SELECT: double free or corruption On Thu, Jun 1, 2017 at 1:11 AM, <ray.warren@artsalliancemedia.com> wrote: > We are experiencing some performance issues and we had this error on Sunday. > Not sure whether the 2 things are related or not as we have been > making some changes to the application and load. However, I assume the > double free or corruption error is a bug? Could you come up with a self-contained test case? It will be difficult to dig into this issue otherwise. -- Michael -- Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-bugs
Ray Warren <Ray.Warren@artsalliancemedia.com> writes:
> ======= Backtrace: =========
> /lib64/libc.so.6(+0x75dee)[0x7f1d6ba7edee]
> /lib64/libc.so.6(+0x78ff6)[0x7f1d6ba81ff6]
> /lib64/libc.so.6(+0x7babe)[0x7f1d6ba84abe]
> /lib64/libc.so.6(realloc+0x158)[0x7f1d6ba84f08]
> postgres: postgres etms 172.30.8.63(53338) SELECT[0x7c8d72]
> postgres: postgres etms 172.30.8.63(53338) SELECT(repalloc+0x25)[0x7c9295]
> postgres: postgres etms 172.30.8.63(53338) SELECT(enlargeStringInfo+0x4e)[0x5e505e]
> postgres: postgres etms 172.30.8.63(53338) SELECT(appendStringInfoChar+0x28)[0x5e5178]
> postgres: postgres etms 172.30.8.63(53338) SELECT[0x7a6724]
> postgres: postgres etms 172.30.8.63(53338) SELECT(EmitErrorReport+0xa2f)[0x7a963f]
> postgres: postgres etms 172.30.8.63(53338) SELECT(errfinish+0x8d)[0x7a97dd]
> postgres: postgres etms 172.30.8.63(53338) SELECT(FileClose+0x1c5)[0x6966d5]
> postgres: postgres etms 172.30.8.63(53338) SELECT(BufFileClose+0x32)[0x698652]
> postgres: postgres etms 172.30.8.63(53338) SELECT(ExecHashJoin+0x31b)[0x5ce60b]
> postgres: postgres etms 172.30.8.63(53338) SELECT(ExecProcNode+0x168)[0x5bc9f8]
> postgres: postgres etms 172.30.8.63(53338) SELECT(ExecSort+0xf9)[0x5d86c9]
> postgres: postgres etms 172.30.8.63(53338) SELECT(ExecProcNode+0x148)[0x5bc9d8]
These stack traces would be a lot more useful if you had debug symbols
installed, I think.
This trace is rather unusual because it appears to show FileClose trying to
log something. That would be expected if you have log_temp_files enabled,
but otherwise it would indicate an unexpected failure. More generally,
what nondefault configuration settings are you using, and do you have any
extensions installed?
regards, tom lane
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
Re: [BUGS] BUG #14683: *** glibc detected *** SELECT: double freeor corruption
From
Ray Warren
Date:
Hi Tom,
Not sure how to install debug symbols and whether it would be a good idea on a Production system.
The information you asked for is below. Note that we have had increased logging configured for a while now so that we
canrun pgbadger. These issues seem to have happened when the load was increased by adding new functionality. We have
switchedthis extra functionality off for the time being so that the database is stable again.
NON DEFAUL PARAMETERS
work_mem = '32MB'
listen_addresses = '*'
max_connections = '300'
superuser_reserved_connections = '13'
shared_buffers = '9GB'
effective_cache_size = '18GB'
maintenance_work_mem = '256MB'
wal_level = 'hot_standby'
archive_mode = 'on'
archive_command = 'cp /var/lib/pgsql/9.5/data/%p /var/lib/pgsql/9.5/wal_archive/%f'
max_wal_senders = '10'
wal_keep_segments = '5000'
hot_standby = 'on'
max_stack_depth = '4MB'
search_path = '"$user", public, repmgr_dbp01_cluster'
shared_preload_libraries = 'pg_stat_statements'
pg_stat_statements.max = '5000'
pg_stat_statements.track = 'all'
log_min_duration_statement = '10'
log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,app=%a,client=%h '
log_checkpoints = 'on'
log_connections = 'on'
log_disconnections = 'on'
log_lock_waits = 'on'
log_temp_files = '0'
log_statement = 'none'
log_autovacuum_min_duration = '0'
log_directory = '/mnt/pg_log'
timezone = 'UTC'
synchronous_commit = 'off'
checkpoint_timeout = '30min'
checkpoint_completion_target = '0.9'
max_wal_size = '18GB'
EXTENSIONS name
--------------------plpgsqlpg_stat_statementspg_buffercacheuuid-ossp
Thanks for helping.
Kind regards ... Ray
-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: 02 June 2017 16:10
To: Ray Warren
Cc: Michael Paquier; PostgreSQL mailing lists
Subject: Re: [BUGS] BUG #14683: *** glibc detected *** SELECT: double free or corruption
Ray Warren <Ray.Warren@artsalliancemedia.com> writes:
> ======= Backtrace: =========
> /lib64/libc.so.6(+0x75dee)[0x7f1d6ba7edee]
> /lib64/libc.so.6(+0x78ff6)[0x7f1d6ba81ff6]
> /lib64/libc.so.6(+0x7babe)[0x7f1d6ba84abe]
> /lib64/libc.so.6(realloc+0x158)[0x7f1d6ba84f08]
> postgres: postgres etms 172.30.8.63(53338) SELECT[0x7c8d72]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(repalloc+0x25)[0x7c9295]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(enlargeStringInfo+0x4e)[0x5e505e]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(appendStringInfoChar+0x28)[0x5e5178]
> postgres: postgres etms 172.30.8.63(53338) SELECT[0x7a6724]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(EmitErrorReport+0xa2f)[0x7a963f]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(errfinish+0x8d)[0x7a97dd]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(FileClose+0x1c5)[0x6966d5]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(BufFileClose+0x32)[0x698652]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(ExecHashJoin+0x31b)[0x5ce60b]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(ExecProcNode+0x168)[0x5bc9f8]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(ExecSort+0xf9)[0x5d86c9]
> postgres: postgres etms 172.30.8.63(53338)
> SELECT(ExecProcNode+0x148)[0x5bc9d8]
These stack traces would be a lot more useful if you had debug symbols installed, I think.
This trace is rather unusual because it appears to show FileClose trying to log something. That would be expected if
youhave log_temp_files enabled, but otherwise it would indicate an unexpected failure. More generally, what nondefault
configurationsettings are you using, and do you have any extensions installed?
regards, tom lane
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs