Thread: double free corruption?

double free corruption?

From

marcelo Cortez

Date:

28 December 2007, 14:33:11

Folks

 i received the follow message from backend ,it's this
a bug?

best regards
and happy new year
MDC


pd: any clue are welcomed.




*** glibc detected *** postgres: postgres richelet
201.235.11.133(2504) SELECT: double free or corruption
(!prev): 0x0845d7e8 ***
======= Backtrace: =========
/lib/libc.so.6[0xb7e0e930]
/lib/libc.so.6(__libc_free+0x89)[0xb7e0ff99]
postgres: postgres richelet 201.235.11.133(2504)
SELECT[0x82b1c0b]
postgres: postgres richelet 201.235.11.133(2504)
SELECT(MemoryContextDelete+0x42)[0x82b2152]
postgres: postgres richelet 201.235.11.133(2504)
SELECT(MemoryContextDeleteChildren+0x28)[0x82b2198]
postgres: postgres richelet 201.235.11.133(2504)
SELECT(MemoryContextDelete+0x12)[0x82b2122]
postgres: postgres richelet 201.235.11.133(2504)
SELECT(MemoryContextDeleteChildren+0x28)[0x82b2198]
postgres: postgres richelet 201.235.11.133(2504)
SELECT(AtAbort_Portals+0x6f)[0x82b281f]
postgres: postgres richelet 201.235.11.133(2504)
SELECT[0x80adef3]
postgres: postgres richelet 201.235.11.133(2504)
SELECT(AbortCurrentTransaction+0x25)[0x80ae115]
postgres: postgres richelet 201.235.11.133(2504)
SELECT(PostgresMain+0x25c6)[0x81f7226]
postgres: postgres richelet 201.235.11.133(2504)
SELECT[0x81ca226]
postgres: postgres richelet 201.235.11.133(2504)
SELECT(PostmasterMain+0x81d)[0x81caf0d]
postgres: postgres richelet 201.235.11.133(2504)
SELECT(main+0x1c7)[0x8182e67]
/lib/libc.so.6(__libc_start_main+0xd8)[0xb7dc0838]
postgres: postgres richelet 201.235.11.133(2504)
SELECT[0x807fa81]
======= Memory map: ========
08048000-0836a000 r-xp 00000000 03:03 715320
/usr/local/pgsql/bin/postgres
0836a000-08373000 rw-p 00321000 03:03 715320
/usr/local/pgsql/bin/postgres
08373000-0846d000 rw-p 08373000 00:00 0
[heap]
b5f00000-b5f21000 rw-p b5f00000 00:00 0
b5f21000-b6000000 ---p b5f21000 00:00 0
b60c4000-b60ce000 r-xp 00000000 03:03 744303
/usr/lib/gcc/i686-pc-linux-gnu/4.1.2/libgcc_s.so.1
b60ce000-b60cf000 rw-p 00009000 03:03 744303
/usr/lib/gcc/i686-pc-linux-gnu/4.1.2/libgcc_s.so.1
b60d7000-b60d9000 r-xp 00000000 03:03 708661
/usr/lib/gconv/ISO8859-1.so
b60d9000-b60db000 rw-p 00001000 03:03 708661
/usr/lib/gconv/ISO8859-1.so
b60db000-b60e3000 r-xp 00000000 03:03 527740
/usr/local/pgsql/lib/fuzzystrmatch.so
b60e3000-b60e4000 rw-p 00007000 03:03 527740
/usr/local/pgsql/lib/fuzzystrmatch.so
b60e4000-b6146000 rw-p b60e4000 00:00 0
b6146000-b6154000 r-xp 00000000 03:03 709308
/lib/libresolv-2.5.so
b6154000-b6156000 rw-p 0000d000 03:03 709308
/lib/libresolv-2.5.so
b6156000-b6158000 rw-p b6156000 00:00 0
b6158000-b615c000 r-xp 00000000 03:03 709745
/lib/libnss_dns-2.5.so
b615c000-b615e000 rw-p 00003000 03:03 709745
/lib/libnss_dns-2.5.so
b615e000-b6166000 r-xp 00000000 03:03 708470
/lib/libnss_files-2.5.so
b6166000-b6168000 rw-p 00007000 03:03 708470
/lib/libnss_files-2.5.so
b6169000-b6170000 r--s 00000000 03:03 6427
/usr/lib/gconv/gconv-modules.cache
b6170000-b61a3000 r--p 00000000 03:03 8975
/usr/lib/locale/es_AR/LC_CTYPE
b61a3000-b61a8000 r--p 00000000 03:03 16329
/usr/lib/locale/es_AR/LC_COLLATE
b61a8000-b7daa000 rw-s 00000000 00:08 114456
/SYSV0052e2c1 (deleted)
b7daa000-b7dab000 rw-p b7daa000 00:00 0
b7dab000-b7ecd000 r-xp 00000000 03:03 709248
/lib/libc-2.5.so
b7ecd000-b7ece000 r--p 00122000 03:03 709248
/lib/libc-2.5.so
b7ece000-b7ed0000 rw-p 00123000 03:03 709248
/lib/libc-2.5.so
b7ed0000-b7ed3000 rw-p b7ed0000 00:00 0
b7ed3000-b7ef6000 r-xp 00000000 03:03 709734
/lib/libm-2.5.so
b7ef6000-b7ef8000 rw-p 00022000 03:03 709734
/lib/libm-2.5.so
b7ef8000-b7efa000 r-xp 00000000 03:03 709751
/lib/libdl-2.5.so
b7efa000-b7efc000 rw-p 00001000 03:03 709751
/lib/libdl-2.5.so
b7efc000-b7f01000 r-xp 00000000 03:03 709885
/lib/libcrypt-2.5.so
b7f01000-b7f03000 rw-p 00004000 03:03 709885
/lib/libcrypt-2.5.so
b7f03000-b7f2b000 rw-p b7f03000 00:00 0
b7f2d000-b7f2e000 r-xp 00000000 03:03 715438
/usr/local/pgsql/lib/utf8_and_iso8859_1.so
b7f2e000-b7f2f000 rw-p 00000000 03:03 715438
/usr/local/pgsql/lib/utf8_and_iso8859_1.so
b7f2f000-b7f30000 r--p 00000000 03:03 206641
/usr/lib/locale/es_AR/LC_TIME
b7f30000-b7f31000 r--p 00000000 03:03 16760
/usr/lib/locale/es_AR/LC_NUMERIC
b7f31000-b7f32000 r--p 00000000 03:03 206642
/usr/lib/locale/es_AR/LC_MONETARY
b7f32000-b7f33000 r--p 00000000 03:03 16336
/usr/lib/locale/es_AR/LC_MESSAGES/SYS_LC_MESSAGES
b7f33000-b7f4d000 r-xp 00000000 03:03 709923
/lib/ld-2.5.so
b7f4d000-b7f4e000 r--p 00019000 03:03 709923
/lib/ld-2.5.so
b7f4e000-b7f4f000 rw-p 0001a000 03:03 709923
/lib/ld-2.5.so
bfdc4000-bfdda000 rw-p bfdc4000 00:00 0
[stack]
ffffe000-fffff000 r-xp 00000000 00:00 0
[vdso]
LOG:  server process (PID 15558) was terminated by
signal 6: Aborted
LOG:  terminating any other active server processes
WARNING:  terminating connection because of crash of
another server process
DETAIL:  The postmaster has commanded this server
process to roll back the current transaction and exit,
because another server process exited abnormally and
possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to
the database and repeat your command.
WARNING:  terminating connection because of crash of
another server process
DETAIL:  The postmaster has commanded this server
process to roll back the current transaction and exit,
because another server process exited abnormally and
possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to
the database and repeat your command.
LOG:  all server processes terminated; reinitializing
LOG:  database system was interrupted; last known up
at 2007-12-28 09:20:37 ART
LOG:  database system was not properly shut down;
automatic recovery in progress
LOG:  record with zero length at 0/20AC262C
LOG:  redo is not required
LOG:  autovacuum launcher started
LOG:  database system is ready to accept connections




      Los referentes más importantes en compra/ venta de autos se juntaron:
Demotores y Yahoo!
Ahora comprar o vender tu auto es más fácil. Vistá ar.autos.yahoo.com/

Re: double free corruption?

From

Erik Jones

Date:

28 December 2007, 14:53:48

On Dec 28, 2007, at 9:33 AM, marcelo Cortez wrote:

> Folks
>
>  i received the follow message from backend ,it's this
> a bug?
>
> best regards
> and happy new year
> MDC
>
>
> pd: any clue are welcomed.
>
>
>
>
> *** glibc detected *** postgres: postgres richelet
> 201.235.11.133(2504) SELECT: double free or corruption
> (!prev): 0x0845d7e8 ***
> ======= Backtrace: =========
> /lib/libc.so.6[0xb7e0e930]
> /lib/libc.so.6(__libc_free+0x89)[0xb7e0ff99]
> postgres: postgres richelet 201.235.11.133(2504)
> SELECT[0x82b1c0b]
> postgres: postgres richelet 201.235.11.133(2504)
> SELECT(MemoryContextDelete+0x42)[0x82b2152]
> postgres: postgres richelet 201.235.11.133(2504)
> SELECT(MemoryContextDeleteChildren+0x28)[0x82b2198]
> postgres: postgres richelet 201.235.11.133(2504)
> SELECT(MemoryContextDelete+0x12)[0x82b2122]
> postgres: postgres richelet 201.235.11.133(2504)
> SELECT(MemoryContextDeleteChildren+0x28)[0x82b2198]
> postgres: postgres richelet 201.235.11.133(2504)
> SELECT(AtAbort_Portals+0x6f)[0x82b281f]
> postgres: postgres richelet 201.235.11.133(2504)
> SELECT[0x80adef3]
> postgres: postgres richelet 201.235.11.133(2504)
> SELECT(AbortCurrentTransaction+0x25)[0x80ae115]
> postgres: postgres richelet 201.235.11.133(2504)
> SELECT(PostgresMain+0x25c6)[0x81f7226]
> postgres: postgres richelet 201.235.11.133(2504)
> SELECT[0x81ca226]
> postgres: postgres richelet 201.235.11.133(2504)
> SELECT(PostmasterMain+0x81d)[0x81caf0d]
> postgres: postgres richelet 201.235.11.133(2504)
> SELECT(main+0x1c7)[0x8182e67]
> /lib/libc.so.6(__libc_start_main+0xd8)[0xb7dc0838]
> postgres: postgres richelet 201.235.11.133(2504)
> SELECT[0x807fa81]
> ======= Memory map: ========
> 08048000-0836a000 r-xp 00000000 03:03 715320
> /usr/local/pgsql/bin/postgres
> 0836a000-08373000 rw-p 00321000 03:03 715320
> /usr/local/pgsql/bin/postgres
> 08373000-0846d000 rw-p 08373000 00:00 0
> [heap]
> b5f00000-b5f21000 rw-p b5f00000 00:00 0
> b5f21000-b6000000 ---p b5f21000 00:00 0
> b60c4000-b60ce000 r-xp 00000000 03:03 744303
> /usr/lib/gcc/i686-pc-linux-gnu/4.1.2/libgcc_s.so.1
> b60ce000-b60cf000 rw-p 00009000 03:03 744303
> /usr/lib/gcc/i686-pc-linux-gnu/4.1.2/libgcc_s.so.1
> b60d7000-b60d9000 r-xp 00000000 03:03 708661
> /usr/lib/gconv/ISO8859-1.so
> b60d9000-b60db000 rw-p 00001000 03:03 708661
> /usr/lib/gconv/ISO8859-1.so
> b60db000-b60e3000 r-xp 00000000 03:03 527740
> /usr/local/pgsql/lib/fuzzystrmatch.so
> b60e3000-b60e4000 rw-p 00007000 03:03 527740
> /usr/local/pgsql/lib/fuzzystrmatch.so
> b60e4000-b6146000 rw-p b60e4000 00:00 0
> b6146000-b6154000 r-xp 00000000 03:03 709308
> /lib/libresolv-2.5.so
> b6154000-b6156000 rw-p 0000d000 03:03 709308
> /lib/libresolv-2.5.so
> b6156000-b6158000 rw-p b6156000 00:00 0
> b6158000-b615c000 r-xp 00000000 03:03 709745
> /lib/libnss_dns-2.5.so
> b615c000-b615e000 rw-p 00003000 03:03 709745
> /lib/libnss_dns-2.5.so
> b615e000-b6166000 r-xp 00000000 03:03 708470
> /lib/libnss_files-2.5.so
> b6166000-b6168000 rw-p 00007000 03:03 708470
> /lib/libnss_files-2.5.so
> b6169000-b6170000 r--s 00000000 03:03 6427
> /usr/lib/gconv/gconv-modules.cache
> b6170000-b61a3000 r--p 00000000 03:03 8975
> /usr/lib/locale/es_AR/LC_CTYPE
> b61a3000-b61a8000 r--p 00000000 03:03 16329
> /usr/lib/locale/es_AR/LC_COLLATE
> b61a8000-b7daa000 rw-s 00000000 00:08 114456
> /SYSV0052e2c1 (deleted)
> b7daa000-b7dab000 rw-p b7daa000 00:00 0
> b7dab000-b7ecd000 r-xp 00000000 03:03 709248
> /lib/libc-2.5.so
> b7ecd000-b7ece000 r--p 00122000 03:03 709248
> /lib/libc-2.5.so
> b7ece000-b7ed0000 rw-p 00123000 03:03 709248
> /lib/libc-2.5.so
> b7ed0000-b7ed3000 rw-p b7ed0000 00:00 0
> b7ed3000-b7ef6000 r-xp 00000000 03:03 709734
> /lib/libm-2.5.so
> b7ef6000-b7ef8000 rw-p 00022000 03:03 709734
> /lib/libm-2.5.so
> b7ef8000-b7efa000 r-xp 00000000 03:03 709751
> /lib/libdl-2.5.so
> b7efa000-b7efc000 rw-p 00001000 03:03 709751
> /lib/libdl-2.5.so
> b7efc000-b7f01000 r-xp 00000000 03:03 709885
> /lib/libcrypt-2.5.so
> b7f01000-b7f03000 rw-p 00004000 03:03 709885
> /lib/libcrypt-2.5.so
> b7f03000-b7f2b000 rw-p b7f03000 00:00 0
> b7f2d000-b7f2e000 r-xp 00000000 03:03 715438
> /usr/local/pgsql/lib/utf8_and_iso8859_1.so
> b7f2e000-b7f2f000 rw-p 00000000 03:03 715438
> /usr/local/pgsql/lib/utf8_and_iso8859_1.so
> b7f2f000-b7f30000 r--p 00000000 03:03 206641
> /usr/lib/locale/es_AR/LC_TIME
> b7f30000-b7f31000 r--p 00000000 03:03 16760
> /usr/lib/locale/es_AR/LC_NUMERIC
> b7f31000-b7f32000 r--p 00000000 03:03 206642
> /usr/lib/locale/es_AR/LC_MONETARY
> b7f32000-b7f33000 r--p 00000000 03:03 16336
> /usr/lib/locale/es_AR/LC_MESSAGES/SYS_LC_MESSAGES
> b7f33000-b7f4d000 r-xp 00000000 03:03 709923
> /lib/ld-2.5.so
> b7f4d000-b7f4e000 r--p 00019000 03:03 709923
> /lib/ld-2.5.so
> b7f4e000-b7f4f000 rw-p 0001a000 03:03 709923
> /lib/ld-2.5.so
> bfdc4000-bfdda000 rw-p bfdc4000 00:00 0
> [stack]
> ffffe000-fffff000 r-xp 00000000 00:00 0
> [vdso]
> LOG:  server process (PID 15558) was terminated by
> signal 6: Aborted
> LOG:  terminating any other active server processes
> WARNING:  terminating connection because of crash of
> another server process
> DETAIL:  The postmaster has commanded this server
> process to roll back the current transaction and exit,
> because another server process exited abnormally and
> possibly corrupted shared memory.
> HINT:  In a moment you should be able to reconnect to
> the database and repeat your command.
> WARNING:  terminating connection because of crash of
> another server process
> DETAIL:  The postmaster has commanded this server
> process to roll back the current transaction and exit,
> because another server process exited abnormally and
> possibly corrupted shared memory.
> HINT:  In a moment you should be able to reconnect to
> the database and repeat your command.
> LOG:  all server processes terminated; reinitializing
> LOG:  database system was interrupted; last known up
> at 2007-12-28 09:20:37 ART
> LOG:  database system was not properly shut down;
> automatic recovery in progress
> LOG:  record with zero length at 0/20AC262C
> LOG:  redo is not required
> LOG:  autovacuum launcher started
> LOG:  database system is ready to accept connections

Well, if Postgres had killed the proc itself it would have written
out a nicely formatted Postgres-style memory context report along
with an ERROR message along the lines of OUT OF MEMORY and the
request size and Postgres would not have bounced.  Since the
postmaster dropped into recovery mode when the proc received the
SIGABRT and died, that means that the signal came from somewhere
else, OOM killer?

Erik Jones

Software Developer | Emma®
erik@myemma.com
800.595.4401 or 615.292.5888
615.292.0777 (fax)

Emma helps organizations everywhere communicate & market in style.
Visit us online at http://www.myemma.com

Re: double free corruption?

From

Tom Lane

Date:

28 December 2007, 14:58:34

marcelo Cortez <jmdc_marcelo@yahoo.com.ar> writes:
> *** glibc detected *** postgres: postgres richelet
> 201.235.11.133(2504) SELECT: double free or corruption
> (!prev): 0x0845d7e8 ***

What PG version is this?  Can you provide a reproducible test case?

            regards, tom lane

Re: double free corruption?

From

marcelo Cortez

Date:

28 December 2007, 15:28:59

Folks

sorry i forgot to mention
i'm developing one c external program, may be fault is
my code , but surprise to  me the message, what bad
practice generate this behavior?
fail seems to be not to reproducible all times,

i'm using beta3 version, it's this important?

select version:
"PostgreSQL 8.3beta3 on i686-pc-linux-gnu, compiled by
GCC gcc (GCC) 4.1.2 (Gentoo 4.1.2 p1.0.1)"

i'ts correct GCC version?

best regards.
MDC

pd: gdb can help? ( for debugging my own code)
links about howto debug?

--- Tom Lane <tgl@sss.pgh.pa.us> escribió:

> marcelo Cortez <jmdc_marcelo@yahoo.com.ar> writes:
> > *** glibc detected *** postgres: postgres richelet
> > 201.235.11.133(2504) SELECT: double free or
> corruption
> > (!prev): 0x0845d7e8 ***
>
> What PG version is this?  Can you provide a
> reproducible test case?
>
>             regards, tom lane
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 5: don't forget to increase your free space map
> settings
>

      Tarjeta de crédito Yahoo! de Banco Supervielle.
Solicitá tu nueva Tarjeta de crédito. De tu PC directo a tu casa. www.tuprimeratarjeta.com.ar

Re: double free corruption?

From

Tom Lane

Date:

28 December 2007, 15:41:28

Erik Jones <erik@myemma.com> writes:
> Well, if Postgres had killed the proc itself it would have written
> out a nicely formatted Postgres-style memory context report along
> with an ERROR message along the lines of OUT OF MEMORY and the
> request size and Postgres would not have bounced.  Since the
> postmaster dropped into recovery mode when the proc received the
> SIGABRT and died, that means that the signal came from somewhere
> else, OOM killer?

No, an abort() is expected when glibc's malloc code detects a problem,
and all that other junk is stuff that malloc helpfully prints on stderr
before committing hara-kiri.

This seems clearly a memory-stomp bug of some kind (although there's
a very small probability that it was a transient RAM glitch).  Not much
we can do about it without a test case, though.

            regards, tom lane

Re: double free corruption?

From

Tom Lane

Date:

28 December 2007, 15:49:49

marcelo Cortez <jmdc_marcelo@yahoo.com.ar> writes:
> sorry i forgot to mention
> i'm developing one c external program, may be fault is
> my code , but surprise to  me the message, what bad
> practice generate this behavior?
> fail seems to be not to reproducible all times,

> i'm using beta3 version, it's this important?

What it looks like to me is something clobbering memory, eg writing more
data into a palloc'd memory chunk than will fit; which results in
overwriting malloc's own data structures, causing malloc to complain
when it notices.

Whether it's your bug or something wrong in the beta is hard to say.

> pd: gdb can help? ( for debugging my own code)

--enable-cassert would probably help more, since it would turn on some
clobber-detection support in PG.

            regards, tom lane

Re: double free corruption?

From

Reece Hart

Date:

31 December 2007, 17:29:16

On Fri, 2007-12-28 at 12:33 -0300, marcelo Cortez wrote:

i received the follow message from backend ,it's this
a bug?

...

*** glibc detected *** postgres: postgres richelet
201.235.11.133(2504) SELECT: double free or corruption
(!prev): 0x0845d7e8 ***
======= Backtrace: =========

Does this happen to be on Novell SLES/SLED 10? I saw this bug sporadically and in several applications 6-9 months ago. I don't think I ever knew the cause or even what specifically tickled this problem, but my recollection is that a Novell patch set fixed it.

I remember that I saw a log of these in /var/log/messages (assuming you're logging locally). Consider looking there to see if you have other instances of this bug with other applications.

-Reece

-- 
Reece Hart, http://harts.net/reece/, GPG:0x25EC91A0