Thread: BUG #6578: Deadlock in libpq after upgrading from 8.4.7 to 8.4.11
The following bug has been logged on the website: Bug reference: 6578 Logged by: Deadlock in libpq Email address: rnysmile@yahoo.com PostgreSQL version: 8.4.11 Operating system: Ubuntu 11.04 Description:=20=20=20=20=20=20=20=20 We wrote a C++ program using libpq to access postgres. it is a heavy load service, about 100 concurrent postgres access. It works very well for two years. This month, we upgrade the libpq5 from 8.4.7 to 8.4.11.(libpq-dev 8.4.11-0ubuntu0.11.04) Our service program become unstable after upgrading. gdb shows that everything is halt at libpq (100 threads halt at the same position). #0 0x00007f0cec7377c4 in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f0cec7325b4 in _L_lock_944 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f0cec7323ca in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x00007f0ced10eaa0 in ?? () from /usr/lib/libpq.so.5 #4 0x00007f0cebb31085 in engine_table_select () from /lib/libcrypto.so.0.9.8 #5 0x00007f0cebb47228 in EVP_CipherInit_ex () from /lib/libcrypto.so.0.9.8 we tried to upgraded the libpq to 9.1.3, it will not deadlock but crash from time to time. After search the mailing list, we found someone else have the similar problem: http://archives.postgresql.org/pgsql-general/2011-03/msg00895.php=20
rnysmile@yahoo.com writes: > This month, we upgrade the libpq5 from 8.4.7 to 8.4.11.(libpq-dev > 8.4.11-0ubuntu0.11.04) Our service program become unstable after upgrading. It seems rather improbable that a deadlock inside glibc would be our bug. Also, I've just looked very carefully through all the diffs between 8.4.7 and 8.4.11 (or at least, all the diffs affecting libpq) and I don't see anything that could plausibly lead to this behavior. > After search the mailing list, we found someone else have the similar > problem: > http://archives.postgresql.org/pgsql-general/2011-03/msg00895.php I can't help observing that that person is also using ubuntu, and that we have not heard similar complaints from anybody else. Ubuntu-specific glibc bug perhaps? A slightly different theory is an ubuntu-specific openssl bug; if there were something wrong in the internal locking in libcrypto, it could well lead to a backtrace like this. regards, tom lane
On Wed, Apr 11, 2012 at 2:04 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > rnysmile@yahoo.com writes: >> This month, we upgrade the libpq5 from =A08.4.7 to 8.4.11.(libpq-dev >> 8.4.11-0ubuntu0.11.04) Our service program become unstable after upgradi= ng. > > It seems rather improbable that a deadlock inside glibc would be our > bug. =A0Also, I've just looked very carefully through all the diffs > between 8.4.7 and 8.4.11 (or at least, all the diffs affecting libpq) > and I don't see anything that could plausibly lead to this behavior. Is it's possible it's this? It's fixed in a version of libc that post-dates Ubuntu 11.0.4 as far as I can tell. https://bugs.launchpad.net/ubuntu/+source/eglibc/+bug/838975 Otherwise, just as a comment, in my experience pthread problems often are triggered by generic memory handling bugs like buffer overruns or writing to freed or realloced blocks. These kinds of bugs can corrupt any dynamically managed memory block and typically pthreads includes such memory blocks internally. --=20 greg
> > This month, we upgrade the libpq5 from=A0 8.4.7 to 8.4.11.(libpq-dev > > 8.4.11-0ubuntu0.11.04) Our service program become unstable after upgrad= ing. > > After search the mailing list, we found someone else have the similar > > problem: > > http://archives.postgresql.org/pgsql-general/2011-03/msg00895.php=20 > I can't help observing that that person is also using ubuntu, and that > we have not heard similar complaints from anybody else.=A0 Ubuntu-specific > glibc bug perhaps?=A0 A slightly different theory is an ubuntu-specific > openssl bug; if there were something wrong in the internal locking > in libcrypto, it could well lead to a backtrace like this. Thank you for you advice. We move back from Ubuntu 11.04 x64 to Ubuntu 10.0= 4 x64, but the problem still exist. We finally FIX that deadlock by install a older version of openssl using th= e following command apt-get install libssl0.9.8=3D0.9.8k-7ubuntu8 Our program=A0 runs 6 Days 21:55:30 till now without any problem. The conclusion is : Deadlock bug in openssl version 0.9.8k-7ubuntu8.10, newer version of ubuntu= (we tested 11.04, 11.10) still have this bug. The working version of openssl is 0.9.8k-7ubuntu8=