Re: libgcc double-free, backend won't die - Mailing list pgsql-performance

Tom Lane wrote:
> Craig James <craig_james@emolecules.com> writes:
>> GNU gdb Red Hat Linux (6.5-15.fc6rh)
>> Copyright (C) 2006 Free Software Foundation, Inc.
>> GDB is free software, covered by the GNU General Public License, and you are
>> welcome to change it and/or distribute copies of it under certain conditions.
>
>> [snip - a bunch of symbol table stuff]
>
> Please show that stuff you snipped --- it might have some relevant
> information.  The stack trace looks a bit like a threading problem...

# gdb -p 31442
GNU gdb Red Hat Linux (6.5-15.fc6rh)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
Attaching to process 31442
Reading symbols from /usr/local/pgsql/bin/postgres...(no debugging symbols found)...done.
Using host libthread_db library "/lib/libthread_db.so.1".
Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /usr/lib/libreadline.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libreadline.so.5
Reading symbols from /lib/libtermcap.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libtermcap.so.2
Reading symbols from /lib/libcrypt.so.1...
(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libm.so.6...
(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /usr/local/pgsql/lib/libchmoogle.so...done.
Loaded symbols for /usr/local/pgsql/lib/libchmoogle.so
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/jaguarformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/jaguarformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/libopenbabel.so.2...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/libopenbabel.so.2
Reading symbols from /usr/lib/libstdc++.so.6...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/fastaformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/fastaformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/cansmilesformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/cansmilesformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/APIInterface.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/APIInterface.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/mmodformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/mmodformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/molreportformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/molreportformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/fhformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/fhformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/chemkinformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/chemkinformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/mmcifformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/mmcifformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/thermoformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/thermoformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/carformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/carformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/ghemicalformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/ghemicalformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/turbomoleformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/turbomoleformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/xmlformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/xmlformat.so
Reading symbols from /usr/lib/libxml2.so.2...done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/rxnformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/rxnformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/reportformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/reportformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/acrformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/acrformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/nwchemformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/nwchemformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/hinformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/hinformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/bgfformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/bgfformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/shelxformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/shelxformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/yasaraformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/yasaraformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/viewmolformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/viewmolformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/mdlformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/mdlformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/CSRformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/CSRformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/cacaoformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/cacaoformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/gaussformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/gaussformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/titleformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/titleformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/gamessformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/gamessformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/zindoformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/zindoformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/fingerprintformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/fingerprintformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/balstformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/balstformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/cssrformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/cssrformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/cdxmlformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/cdxmlformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/crkformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/crkformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/xedformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/xedformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/chemdrawcdxformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/chemdrawcdxformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/cmlformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/cmlformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/mpdformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/mpdformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/amberformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/amberformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/smilesformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/smilesformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/chemtoolformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/chemtoolformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/pubchem.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/pubchem.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/fchkformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/fchkformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/qchemformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/qchemformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/mopacformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/mopacformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/PQSformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/PQSformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/fastsearchformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/fastsearchformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/freefracformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/freefracformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/chem3dformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/chem3dformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/inchiformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/inchiformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/cccformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/cccformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/mpqcformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/mpqcformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/copyformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/copyformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/cifformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/cifformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/unichemformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/unichemformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/boxformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/boxformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/mol2format.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/mol2format.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/tinkerformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/tinkerformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/featformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/featformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/alchemyformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/alchemyformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/pngformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/pngformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/pcmodelformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/pcmodelformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/dmolformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/dmolformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/gausscubeformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/gausscubeformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/povrayformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/povrayformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/xyzformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/xyzformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/cacheformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/cacheformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/chemdrawctformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/chemdrawctformat.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/gromos96format.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/gromos96format.so
Reading symbols from /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/pdbformat.so...done.
Loaded symbols for /usr/local/openbabel-inst/openbabel-dev-2-1-x/lib/openbabel/pdbformat.so

0x00110402 in __kernel_vsyscall ()
(gdb) bt
#0  0x00110402 in __kernel_vsyscall ()
#1  0x0082fb8e in __lll_mutex_lock_wait () from /lib/libc.so.6
#2  0x007bfce8 in _L_lock_14096 () from /lib/libc.so.6
#3  0x007befa4 in free () from /lib/libc.so.6
#4  0x00744f93 in _dl_map_object_deps () from /lib/ld-linux.so.2
#5  0x0074989d in dl_open_worker () from /lib/ld-linux.so.2
#6  0x00745c36 in _dl_catch_error () from /lib/ld-linux.so.2
#7  0x00749222 in _dl_open () from /lib/ld-linux.so.2
#8  0x00858712 in do_dlopen () from /lib/libc.so.6
#9  0x00745c36 in _dl_catch_error () from /lib/ld-linux.so.2
#10 0x008588c5 in __libc_dlopen_mode () from /lib/libc.so.6
#11 0x00836139 in init () from /lib/libc.so.6
#12 0x008362d3 in backtrace () from /lib/libc.so.6
#13 0x007b3e11 in __libc_message () from /lib/libc.so.6
#14 0x007bba96 in _int_free () from /lib/libc.so.6
#15 0x007befb0 in free () from /lib/libc.so.6
#16 0x001f943a in DeleteByteCode (node=0x890ff4) at chains.cpp:477
#17 0x00780859 in exit () from /lib/libc.so.6
#18 0x081a6064 in proc_exit ()
#19 0x081b5b9d in PostgresMain ()
#20 0x0818e34b in ServerLoop ()
#21 0x0818f1de in PostmasterMain ()
#22 0x08152369 in main ()
(gdb)