Thread: more on high load on postgres 7.4.16

From:
Geoffrey
Date:

We are trying to attack this problem from multiple avenues, thus I'm
starting a separate thread.  This is with regard to the problem posted
via thread:

http://archives.postgresql.org/pgsql-performance/2007-04/msg00120.php

One thing we are seeing with this move to the new hardware (and rhas 4)
is database connection processes that are left over by users who have
exited the application.  I've attached to these processes via gdb and
find they all have the same backtrace.  Any insights into what might be
causing this issue would be appreciated.  Understand, we did not have
this problem on the previous hardware running on rhes 3.  Here is the
backtrace:

#0  0x00ba47a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x0019f1de in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
#2  0x0019ca7a in _L_mutex_lock_23 () from /lib/tls/libpthread.so.0
#3  0xbfed9438 in ?? ()
#4  0x00c96a4e in pthread_cond_destroy@@GLIBC_2.3.2 () from
/lib/tls/libc.so.6
#5  0x00c96a4e in pthread_cond_destroy@@GLIBC_2.3.2 () from
/lib/tls/libc.so.6
#6  0x0015243f in critSec::~critSec () from /usr/local/pcm170/libdalkutil.so
#7  0x003a48b8 in Comp_ZipFiles () from /usr/local/pcm170/libcompress.so
#8  0x00bec527 in exit () from /lib/tls/libc.so.6
#9  0x0816a52f in proc_exit ()


--
Until later, Geoffrey

Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
  - Benjamin Franklin

From:
Stefan Kaltenbrunner
Date:

Geoffrey wrote:
> We are trying to attack this problem from multiple avenues, thus I'm
> starting a separate thread.  This is with regard to the problem posted
> via thread:
>
> http://archives.postgresql.org/pgsql-performance/2007-04/msg00120.php
>
> One thing we are seeing with this move to the new hardware (and rhas 4)
> is database connection processes that are left over by users who have
> exited the application.  I've attached to these processes via gdb and
> find they all have the same backtrace.  Any insights into what might be
> causing this issue would be appreciated.  Understand, we did not have
> this problem on the previous hardware running on rhes 3.  Here is the
> backtrace:
>
> #0  0x00ba47a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
> #1  0x0019f1de in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
> #2  0x0019ca7a in _L_mutex_lock_23 () from /lib/tls/libpthread.so.0
> #3  0xbfed9438 in ?? ()
> #4  0x00c96a4e in pthread_cond_destroy@@GLIBC_2.3.2 () from
> /lib/tls/libc.so.6
> #5  0x00c96a4e in pthread_cond_destroy@@GLIBC_2.3.2 () from
> /lib/tls/libc.so.6
> #6  0x0015243f in critSec::~critSec () from
> /usr/local/pcm170/libdalkutil.so
> #7  0x003a48b8 in Comp_ZipFiles () from /usr/local/pcm170/libcompress.so

/usr/local on RHEL should only contain software installed directly from
source - what exactly is pcm170/libdalkutil ?
beside that - is pg actually compiled with debugging symbols on that
platform ?


Stefan

From:
Geoffrey
Date:

Stefan Kaltenbrunner wrote:
> Geoffrey wrote:
>> We are trying to attack this problem from multiple avenues, thus I'm
>> starting a separate thread.  This is with regard to the problem posted
>> via thread:
>>
>> http://archives.postgresql.org/pgsql-performance/2007-04/msg00120.php
>>
>> One thing we are seeing with this move to the new hardware (and rhas 4)
>> is database connection processes that are left over by users who have
>> exited the application.  I've attached to these processes via gdb and
>> find they all have the same backtrace.  Any insights into what might be
>> causing this issue would be appreciated.  Understand, we did not have
>> this problem on the previous hardware running on rhes 3.  Here is the
>> backtrace:
>>
>> #0  0x00ba47a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
>> #1  0x0019f1de in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
>> #2  0x0019ca7a in _L_mutex_lock_23 () from /lib/tls/libpthread.so.0
>> #3  0xbfed9438 in ?? ()
>> #4  0x00c96a4e in pthread_cond_destroy@@GLIBC_2.3.2 () from
>> /lib/tls/libc.so.6
>> #5  0x00c96a4e in pthread_cond_destroy@@GLIBC_2.3.2 () from
>> /lib/tls/libc.so.6
>> #6  0x0015243f in critSec::~critSec () from
>> /usr/local/pcm170/libdalkutil.so
>> #7  0x003a48b8 in Comp_ZipFiles () from /usr/local/pcm170/libcompress.so
>
> /usr/local on RHEL should only contain software installed directly from
> source - what exactly is pcm170/libdalkutil ?

It is a third party package that we have build into the backend.
pcmiler.  We do not have source to it though.

> beside that - is pg actually compiled with debugging symbols on that
> platform ?

Not yet, I'm building it now, but I was hoping that the limited info
above might get us some insights.  I plan to try and recreate the
problem and reproduce a more useful backtrace after rebuilding
postgresql with debugging symbols.

--
Until later, Geoffrey

Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
  - Benjamin Franklin