Re: Postmaster hangs - Mailing list pgsql-bugs

Sorry for the delay in responding, and thanks for your help.


> You didn't actually request a backtrace (bt), so all it shows is the top
> stack frame. That doesn't tell us anything except that it's busy in a
> system call in the kernel.

For a newly-started psql process that locks as soon as the password is
entered:

#0  0x00fbe416 in __kernel_vsyscall ()
#1  0x00c7673b in poll () from /lib/libc.so.6
#2  0x00d20eaa in ?? () from /usr/lib/libpq.so.5
#3  0x00d20faa in ?? () from /usr/lib/libpq.so.5
#4  0x00d1b344 in ?? () from /usr/lib/libpq.so.5
#5  0x00d1d571 in PQsetdbLogin () from /usr/lib/libpq.so.5
#6  0x08055662 in geteuid ()
#7  0x00bb36e5 in __libc_start_main () from /lib/libc.so.6
#8  0x0804a9c1 in geteuid ()

And for the process that ran the update commands that locked it:

#0  0x00e12416 in __kernel_vsyscall ()
#1  0x00c7673b in poll () from /lib/libc.so.6
#2  0x00d20eaa in ?? () from /usr/lib/libpq.so.5
#3  0x00d20faa in ?? () from /usr/lib/libpq.so.5
#4  0x00d21023 in ?? () from /usr/lib/libpq.so.5
#5  0x00d2062a in PQgetResult () from /usr/lib/libpq.so.5
#6  0x00d20853 in ?? () from /usr/lib/libpq.so.5
#7  0x0804e97f in geteuid ()
#8  0x080528bb in geteuid ()
#9  0x0804b37c in geteuid ()
#10 0x08055856 in geteuid ()
#11 0x00bb36e5 in __libc_start_main () from /lib/libc.so.6
#12 0x0804a9c1 in geteuid ()


> > > You can find out a bit more about what the kernel is doing using the
> > > "magic" keyboard sequence "ALT-SysRQ-T" from a vconsole (not under X).
> >
> > Nothing happened.  Nothing useful in dmesg -- certainly no stacktraces.
>
> Your kernel might not have the "magic sysrq key" enabled. Run:
>
>   sudo sysctl -w kernel.sysrq=1
>
> and try again. Note that on some systems with weird keyboards you might
> have to hold the "Fn" key (if you have one) or disable "F-Lock" (if you
> have it) to get SysRq to be recognised. The print screen / PrtScn key is
> usually shared with SysRq even if it's not marked as such.

That sysctl command did the trick.  Here's the relevant sections:

psql          S c1ff7c00     0 13794  13755
       eb383bcc 00200082 f11b1c00 c1ff7c00 eb383bdc c087c67c c087fc00
c087fc00
       c087fc00 c24ecce0 c24ecf54 c1ff7c00 00000002 c1ff7c00 c24ecf54
c1ff7c00
       00000002 c1ff7c00 e7f75b40 c24ecf54 05f35f1a 7cc8a1e3 00005a28
c24ecf54
Call Trace:
 [<c06aaccb>] schedule_timeout+0x17/0xbc
 [<c049e1ef>] ? __pollwait+0xad/0xb6
 [<c0691aa2>] ? unix_poll+0x17/0x8a
 [<c049d41c>] do_sys_poll+0x292/0x348
 [<c049e142>] ? __pollwait+0x0/0xb6
 [<c0428230>] ? default_wake_function+0x0/0xd
 [<c051b115>] ? __next_cpu+0x15/0x25
 [<c0425b31>] ? tg_shares_up+0x165/0x178
 [<c04259cc>] ? tg_shares_up+0x0/0x178
 [<c041fa2f>] ? walk_tg_tree+0x62/0x72
 [<c041fa91>] ? tg_nop+0x0/0x5
 [<c04149bb>] ? apic_wait_icr_idle+0x17/0x21
 [<c04221dc>] ? task_rq_lock+0x49/0x73
 [<c041445e>] ? __send_IPI_dest_field+0x49/0x4e
 [<c0414595>] ? send_IPI_mask_bitmask+0x5e/0x79
 [<c041fc04>] ? update_curr+0x8d/0xf0
 [<c04226d4>] ? dequeue_entity+0x15/0x13a
 [<c0422bd8>] ? dequeue_task_fair+0x3d/0x42
 [<c04036bf>] ? __switch_to+0xb9/0x139
 [<c042b045>] ? finish_task_switch+0x2f/0xb0
 [<c06aa9bf>] ? schedule+0x6ee/0x70d
 [<c045a5c0>] ? utrace_reset+0x1e7/0x1ef
 [<c045af80>] ? utrace_stop+0x19b/0x1ac
 [<c045b00f>] ? finish_resume_report+0x7e/0x8f
 [<c045b85c>] ? utrace_get_signal+0x4a8/0x5b1
 [<c04a2d5b>] ? touch_atime+0xa0/0xba
 [<c0434ef2>] ? ptrace_report_signal+0x0/0x253
 [<c04393da>] ? dequeue_signal+0x92/0x105
 [<c043970f>] ? get_signal_to_deliver+0x2c2/0x2d6
 [<c0403f10>] ? do_notify_resume+0x71/0x679
 [<c042262a>] ? set_next_entity+0x8b/0xf7
 [<c04036bf>] ? __switch_to+0xb9/0x139
 [<c042b045>] ? finish_task_switch+0x2f/0xb0
 [<c06aa9bf>] ? schedule+0x6ee/0x70d
 [<c0461e8f>] ? audit_syscall_entry+0xf9/0x123
 [<c049d74d>] do_restart_poll+0x2f/0x52
 [<c0437017>] sys_restart_syscall+0x11/0x13
 [<c0404c8a>] syscall_call+0x7/0xb
 =======================
psql          S c087fc00     0 18118  12388
       ee7a0bdc 00200082 f7836680 c087fc00 c087fc00 c087c67c c087fc00
c087fc00
       c087fc00 c44bc010 c44bc284 c1fecc00 00000001 c1fecc00 cdb9adb4
c44bc284
       05f2d5b0 001280d2 c07fc90c c44bc284 05f53997 ee7a0bc8 ee7a0bc8
00000000
Call Trace:
 [<c06aaccb>] schedule_timeout+0x17/0xbc
 [<c049e1ef>] ? __pollwait+0xad/0xb6
 [<c0691aa2>] ? unix_poll+0x17/0x8a
 [<c049d41c>] do_sys_poll+0x292/0x348
 [<c049e142>] ? __pollwait+0x0/0xb6
 [<c0428230>] ? default_wake_function+0x0/0xd
 [<c051b115>] ? __next_cpu+0x15/0x25
 [<c0425b31>] ? tg_shares_up+0x165/0x178
 [<c04259cc>] ? tg_shares_up+0x0/0x178
 [<c041fa2f>] ? walk_tg_tree+0x62/0x72
 [<c041fa91>] ? tg_nop+0x0/0x5
 [<c06abe09>] ? _spin_lock+0x8/0xb
 [<c04221dc>] ? task_rq_lock+0x49/0x73
 [<c041fc04>] ? update_curr+0x8d/0xf0
 [<c04226d4>] ? dequeue_entity+0x15/0x13a
 [<c0422bd8>] ? dequeue_task_fair+0x3d/0x42
 [<c04036bf>] ? __switch_to+0xb9/0x139
 [<c042b045>] ? finish_task_switch+0x2f/0xb0
 [<c06aa9bf>] ? schedule+0x6ee/0x70d
 [<c040a793>] ? save_i387+0x103/0x19e
 [<c048f651>] ? kmem_cache_free+0x8c/0xa7
 [<c0403e30>] ? setup_sigcontext+0xfa/0x169
 [<c040aaf5>] ? restore_i387+0xdd/0x12f
 [<c040467c>] ? restore_sigcontext+0x151/0x1a8
 [<c0461e8f>] ? audit_syscall_entry+0xf9/0x123
 [<c049d6ee>] sys_poll+0x3a/0x6a
 [<c0404c8a>] syscall_call+0x7/0xb
 =======================
postmaster    D c1fc8a00     0 18125   2877
       eb1eeda8 00200082 eb1eed4c c1fc8a00 eb1eed4c c087c67c c087fc00
c087fc00
       c087fc00 c44be680 c44be8f4 c1fecc00 00000001 c1fecc00 f750ae38
eb1eed88
       c041f325 00000000 c1fc8a00 c44be8f4 05edcf8f 00200246 eb1eeddc
eb1eeda8
Call Trace:
 [<c041f325>] ? kunmap_atomic+0x87/0xa7
 [<c043f1e8>] ? prepare_to_wait+0x51/0x58
 [<f8853904>] start_this_handle+0x2cc/0x3dd [jbd2]
 [<c043efe2>] ? autoremove_wake_function+0x0/0x33
 [<f8853b7f>] jbd2_journal_start+0x8c/0xb9 [jbd2]
 [<f894de9b>] ext4_journal_start_sb+0x40/0x42 [ext4]
 [<f89420f8>] ext4_dirty_inode+0x24/0x97 [ext4]
 [<c04ac266>] __mark_inode_dirty+0x29/0x143
 [<c04a2d6f>] touch_atime+0xb4/0xba
 [<c04718ab>] generic_file_aio_read+0x4ff/0x53d
 [<c0492b62>] do_sync_read+0xab/0xe9
 [<c0480500>] ? __vma_link+0x6e/0x73
 [<c043efe2>] ? autoremove_wake_function+0x0/0x33
 [<c04f683e>] ? security_file_permission+0xf/0x11
 [<c0492ab7>] ? do_sync_read+0x0/0xe9
 [<c0493418>] vfs_read+0x81/0xdc
 [<c049350c>] sys_read+0x3b/0x60
 [<c0404c8a>] syscall_call+0x7/0xb
 =======================
postmaster    D 00000002     0 13813   2877
       eb199da8 00200086 c44b9cd8 00000002 eb199d58 c087c67c c087fc00
c087fc00
       c087fc00 c44b99a0 c44b9c14 c1ff7c00 00000002 c1ff7c00 ef2d1900
c087c67c
       c087fc00 ec018600 eb199d94 c44b9c14 05e4138d 00200246 eb199ddc
eb199da8
Call Trace:
 [<c043f1e8>] ? prepare_to_wait+0x51/0x58
 [<f8853904>] start_this_handle+0x2cc/0x3dd [jbd2]
 [<c043efe2>] ? autoremove_wake_function+0x0/0x33
 [<f8853b7f>] jbd2_journal_start+0x8c/0xb9 [jbd2]
 [<f894de9b>] ext4_journal_start_sb+0x40/0x42 [ext4]
 [<f89420f8>] ext4_dirty_inode+0x24/0x97 [ext4]
 [<c04ac266>] __mark_inode_dirty+0x29/0x143
 [<c04a2d6f>] touch_atime+0xb4/0xba
 [<c04718ab>] generic_file_aio_read+0x4ff/0x53d
 [<c0492b62>] do_sync_read+0xab/0xe9
 [<c0446572>] ? clockevents_program_event+0xd4/0xe3
 [<c043efe2>] ? autoremove_wake_function+0x0/0x33
 [<c041f842>] ? need_resched+0x18/0x22
 [<c06aaab2>] ? _cond_resched+0x8/0x32
 [<c04f683e>] ? security_file_permission+0xf/0x11
 [<c0492ab7>] ? do_sync_read+0x0/0xe9
 [<c0493418>] vfs_read+0x81/0xdc
 [<c049350c>] sys_read+0x3b/0x60
 [<c0404c8a>] syscall_call+0x7/0xb
 =======================
httpd         D e8859ec0     0 16604   3285
       e8859f24 00000082 00000060 e8859ec0 c041f842 c087c67c c087fc00
c087fc00
       c087fc00 eaaf0000 eaaf0274 c2002c00 00000003 c2002c00 c4e99d2e
00005b61
       00000901 00000000 00000000 eaaf0274 000041fd 00000002 00000284
000001f4
Call Trace:
 [<c041f842>] ? need_resched+0x18/0x22
 [<c06ab0ce>] __mutex_lock_common+0x8a/0xd9
 [<c06ab12f>] __mutex_lock_killable_slowpath+0x12/0x14
 [<c06ab15e>] ? mutex_lock_killable+0x2d/0x34
 [<c06ab15e>] mutex_lock_killable+0x2d/0x34
 [<c049ce61>] vfs_readdir+0x41/0x8f
 [<c049cbd0>] ? filldir64+0x0/0xcd
 [<c049cf12>] sys_getdents64+0x63/0xa1
 [<c0404c8a>] syscall_call+0x7/0xb
 =======================
httpd         D e7db6ec0     0 16605   3285
       e7db6f24 00000082 00000060 e7db6ec0 c041f842 c087c67c c087fc00
c087fc00
       c087fc00 eaaf0cd0 eaaf0f44 c2002c00 00000003 c2002c00 bff09b8c
e7db6f60
       00000901 00000000 00000000 eaaf0f44 05f7ea2e 00000002 00000284
000001f4
Call Trace:
 [<c041f842>] ? need_resched+0x18/0x22
 [<c06ab0ce>] __mutex_lock_common+0x8a/0xd9
 [<c06ab12f>] __mutex_lock_killable_slowpath+0x12/0x14
 [<c06ab15e>] ? mutex_lock_killable+0x2d/0x34
 [<c06ab15e>] mutex_lock_killable+0x2d/0x34
 [<c049ce61>] vfs_readdir+0x41/0x8f
 [<c049cbd0>] ? filldir64+0x0/0xcd
 [<c049cf12>] sys_getdents64+0x63/0xa1
 [<c0404c8a>] syscall_call+0x7/0xb
 =======================
httpd         D ea951d80     0 16606   3285
       ea951ddc 00000086 ea951da8 ea951d80 c0498e59 c087c67c c087fc00
c087fc00
       c087fc00 eaaf6680 eaaf68f4 c2002c00 00000003 c2002c00 c04a1bec
ea951e04
       f740c088 00000006 94f29ee3 eaaf68f4 05f8a91b ea951e10 f740c088
ea951e10
Call Trace:
 [<c0498e59>] ? path_to_nameidata+0x11/0x2a
 [<c04a1bec>] ? __d_lookup+0xb3/0xe7
 [<c04a22b2>] ? dput+0x34/0x107
 [<c06ab0ce>] __mutex_lock_common+0x8a/0xd9
 [<c06ab1bf>] __mutex_lock_slowpath+0x12/0x15
 [<c06ab040>] ? mutex_lock+0x29/0x2d
 [<c06ab040>] mutex_lock+0x29/0x2d
 [<c0499133>] do_lookup+0x6b/0x11e
 [<c049a5a1>] __link_path_walk+0x57e/0x6b5
 [<c049aac4>] path_walk+0x4c/0x9b
 [<c049acc0>] do_path_lookup+0x12d/0x175
 [<c049bdbb>] user_path_at+0x3c/0x62
 [<c043efe2>] ? autoremove_wake_function+0x0/0x33
 [<c0495b75>] vfs_stat_fd+0x19/0x40
 [<c0495c4b>] vfs_stat+0x11/0x13
 [<c0495c61>] sys_stat64+0x14/0x2b
 [<c0493694>] ? generic_file_llseek+0x3a/0x46
 [<c0461e8f>] ? audit_syscall_entry+0xf9/0x123
 [<c040b098>] ? syscall_trace_enter+0xb5/0xc9
 [<c0404c8a>] syscall_call+0x7/0xb
 =======================
httpd         D db54cec0     0 18182   3285
       db54cf24 00000082 00000060 db54cec0 c041f842 c087c67c c087fc00
c087fc00
       c087fc00 c44bb340 c44bb5b4 c2002c00 00000003 c2002c00 e2abbec3
00005b5c
       00000901 00000000 00000000 c44bb5b4 000041fd 00000002 00000284
000001f4
Call Trace:
 [<c041f842>] ? need_resched+0x18/0x22
 [<c06ab0ce>] __mutex_lock_common+0x8a/0xd9
 [<c06ab12f>] __mutex_lock_killable_slowpath+0x12/0x14
 [<c06ab15e>] ? mutex_lock_killable+0x2d/0x34
 [<c06ab15e>] mutex_lock_killable+0x2d/0x34
 [<c049ce61>] vfs_readdir+0x41/0x8f
 [<c049cbd0>] ? filldir64+0x0/0xcd
 [<c049cf12>] sys_getdents64+0x63/0xa1
 [<c0404c8a>] syscall_call+0x7/0xb
 =======================
httpd         D db755d80     0 21298   3285
       db755ddc 00000082 db755da8 db755d80 c0498e59 c087c67c c087fc00
c087fc00
       c087fc00 ec4d2670 ec4d28e4 c2002c00 00000003 c2002c00 3c45a442
00005b4e
       f740c088 00000006 94f29ee3 ec4d28e4 ea5cfcc0 db755e10 f740c088
db755e10
Call Trace:
 [<c0498e59>] ? path_to_nameidata+0x11/0x2a
 [<c04a22b2>] ? dput+0x34/0x107
 [<c06ab0ce>] __mutex_lock_common+0x8a/0xd9
 [<c06ab1bf>] __mutex_lock_slowpath+0x12/0x15
 [<c06ab040>] ? mutex_lock+0x29/0x2d
 [<c06ab040>] mutex_lock+0x29/0x2d
 [<c0499133>] do_lookup+0x6b/0x11e
 [<c049a5a1>] __link_path_walk+0x57e/0x6b5
 [<c049aac4>] path_walk+0x4c/0x9b
 [<c049acc0>] do_path_lookup+0x12d/0x175
 [<c049bdbb>] user_path_at+0x3c/0x62
 [<c043efe2>] ? autoremove_wake_function+0x0/0x33
 [<c0495b75>] vfs_stat_fd+0x19/0x40
 [<c0495c4b>] vfs_stat+0x11/0x13
 [<c0495c61>] sys_stat64+0x14/0x2b
 [<c0493694>] ? generic_file_llseek+0x3a/0x46
 [<c0461e8f>] ? audit_syscall_entry+0xf9/0x123
 [<c040b098>] ? syscall_trace_enter+0xb5/0xc9
 [<c0404c8a>] syscall_call+0x7/0xb
 =======================
httpd         D cc21c700     0 21309   3285
       ee7afeb8 00000086 f893f20d cc21c700 ca80d110 c087c67c c087fc00
c087fc00
       c087fc00 ec4d6680 ec4d68f4 c1fe1c00 00000000 c1fe1c00 00000000
b557abd0
       ee7aff94 ee7afea4 c049cc67 ec4d68f4 05f3a48d 00000246 ee7afeec
ee7afeb8
Call Trace:
 [<f893f20d>] ? ext4_htree_store_dirent+0xdb/0xe6 [ext4]
 [<c049cc67>] ? filldir64+0x97/0xcd
 [<c043f1e8>] ? prepare_to_wait+0x51/0x58
 [<f8853904>] start_this_handle+0x2cc/0x3dd [jbd2]
 [<c043efe2>] ? autoremove_wake_function+0x0/0x33
 [<f8853b7f>] jbd2_journal_start+0x8c/0xb9 [jbd2]
 [<f894de9b>] ext4_journal_start_sb+0x40/0x42 [ext4]
 [<f89420f8>] ext4_dirty_inode+0x24/0x97 [ext4]
 [<c04ac266>] __mark_inode_dirty+0x29/0x143
 [<c04a2d6f>] touch_atime+0xb4/0xba
 [<c049ce96>] vfs_readdir+0x76/0x8f
 [<c049cbd0>] ? filldir64+0x0/0xcd
 [<c049cf12>] sys_getdents64+0x63/0xa1
 [<c0404c8a>] syscall_call+0x7/0xb
 =======================
httpd         D c17cf780     0 21310   3285
       ee7c0e14 00000082 c065fe70 c17cf780 ee7c0dc8 c087c67c c087fc00
c087fc00
       c087fc00 ec4d19a0 ec4d1c14 c1fe1c00 00000000 c1fe1c00 00000040
c06d9348
       ee7c0e54 ee7c0e0c c04acb17 ec4d1c14 05f6ab0b 00000246 ee7c0e48
ee7c0e14
Call Trace:
 [<c065fe70>] ? tcp_sendpage+0x0/0x39b
 [<c04acb17>] ? pipe_to_sendpage+0x56/0x61
 [<c043f1e8>] ? prepare_to_wait+0x51/0x58
 [<f8853904>] start_this_handle+0x2cc/0x3dd [jbd2]
 [<c043efe2>] ? autoremove_wake_function+0x0/0x33
 [<f8853b7f>] jbd2_journal_start+0x8c/0xb9 [jbd2]
 [<f894de9b>] ext4_journal_start_sb+0x40/0x42 [ext4]
 [<f89420f8>] ext4_dirty_inode+0x24/0x97 [ext4]
 [<c04ac266>] __mark_inode_dirty+0x29/0x143
 [<c04a2d6f>] touch_atime+0xb4/0xba
 [<c04ad5fe>] splice_direct_to_actor+0x144/0x187
 [<c04ad156>] ? direct_splice_actor+0x0/0x1c
 [<c04ad6a8>] do_splice_direct+0x67/0x84
 [<c0492dd5>] do_sendfile+0x188/0x237
 [<c0492ebe>] sys_sendfile64+0x3a/0x7a
 [<c0404c8a>] syscall_call+0x7/0xb
 =======================
httpd         D c1867000     0 21311   3285
       f12b8e14 00000082 c065fe70 c1867000 f12b8dc8 c087c67c c087fc00
c087fc00
       c087fc00 d4da0000 d4da0274 c1fe1c00 00000000 c1fe1c00 64fb23f0
00005a99
       f12b8e54 f12b8e0c c04acb17 d4da0274 f12b8df8 00000246 f12b8e48
f12b8e14
Call Trace:
 [<c065fe70>] ? tcp_sendpage+0x0/0x39b
 [<c04acb17>] ? pipe_to_sendpage+0x56/0x61
 [<c043f1e8>] ? prepare_to_wait+0x51/0x58
 [<f8853904>] start_this_handle+0x2cc/0x3dd [jbd2]
 [<c043efe2>] ? autoremove_wake_function+0x0/0x33
 [<f8853b7f>] jbd2_journal_start+0x8c/0xb9 [jbd2]
 [<f894de9b>] ext4_journal_start_sb+0x40/0x42 [ext4]
 [<f89420f8>] ext4_dirty_inode+0x24/0x97 [ext4]
 [<c04ac266>] __mark_inode_dirty+0x29/0x143
 [<c04a2d6f>] touch_atime+0xb4/0xba
 [<c04ad5fe>] splice_direct_to_actor+0x144/0x187
 [<c04ad156>] ? direct_splice_actor+0x0/0x1c
 [<c04ad6a8>] do_splice_direct+0x67/0x84
 [<c0492dd5>] do_sendfile+0x188/0x237
 [<c0492ebe>] sys_sendfile64+0x3a/0x7a
 [<c0404c8a>] syscall_call+0x7/0xb
 =======================
httpd         D c1867000     0   681   3285
       e7df7e14 00000086 c065fe70 c1867000 e7df7dc8 c087c67c c087fc00
c087fc00
       c087fc00 cc328cd0 cc328f44 c2002c00 00000003 c2002c00 a29652c3
00005aa0
       e7df7e54 e7df7e0c c04acb17 cc328f44 e7df7df8 00000246 e7df7e48
e7df7e14
Call Trace:
 [<c065fe70>] ? tcp_sendpage+0x0/0x39b
 [<c04acb17>] ? pipe_to_sendpage+0x56/0x61
 [<c043f1e8>] ? prepare_to_wait+0x51/0x58
 [<f8853904>] start_this_handle+0x2cc/0x3dd [jbd2]
 [<c043efe2>] ? autoremove_wake_function+0x0/0x33
 [<f8853b7f>] jbd2_journal_start+0x8c/0xb9 [jbd2]
 [<f894de9b>] ext4_journal_start_sb+0x40/0x42 [ext4]
 [<f89420f8>] ext4_dirty_inode+0x24/0x97 [ext4]
 [<c04ac266>] __mark_inode_dirty+0x29/0x143
 [<c04a2d6f>] touch_atime+0xb4/0xba
 [<c04ad5fe>] splice_direct_to_actor+0x144/0x187
 [<c04ad156>] ? direct_splice_actor+0x0/0x1c
 [<c04ad6a8>] do_splice_direct+0x67/0x84
 [<c0492dd5>] do_sendfile+0x188/0x237
 [<c0492ebe>] sys_sendfile64+0x3a/0x7a
 [<c0404c8a>] syscall_call+0x7/0xb
httpd         D e9287ec0     0 14218   3285
       e9287f24 00000082 00000060 e9287ec0 c041f842 c087c67c c087fc00
c087fc00
       c087fc00 eb2c99a0 eb2c9c14 c2002c00 00000003 c2002c00 92108055
00005a55
       00000901 00000000 00000000 eb2c9c14 000041ff 00000009 00000284
000001f4
Call Trace:
 [<c041f842>] ? need_resched+0x18/0x22
 [<c06ab0ce>] __mutex_lock_common+0x8a/0xd9
 [<c06ab12f>] __mutex_lock_killable_slowpath+0x12/0x14
 [<c06ab15e>] ? mutex_lock_killable+0x2d/0x34
 [<c06ab15e>] mutex_lock_killable+0x2d/0x34
 [<c049ce61>] vfs_readdir+0x41/0x8f
 [<c049cbd0>] ? filldir64+0x0/0xcd
 [<c049cf12>] sys_getdents64+0x63/0xa1
 [<c0404c8a>] syscall_call+0x7/0xb
 =======================
httpd         S c06aaab2     0 18474   3285
       ef333d94 00000082 ef333d2c c06aaab2 ef333d38 c087c67c c087fc00
c087fc00
       c087fc00 ec6319a0 ec631c14 c1ff7c00 00000002 c1ff7c00 c087c8e8
ef333dc4
       ef333d80 c041f4f5 ef333d6c ec631c14 05f8a33e c04ec457 ef333dc8
ee32dea8
Call Trace:
 [<c06aaab2>] ? _cond_resched+0x8/0x32
 [<c041f4f5>] ? kmap_atomic_prot+0x1b0/0x1da
 [<c04ec457>] ? ipc_lock+0x2b/0x45
 [<c04ee244>] sys_semtimedop+0x4ca/0x579
 [<c047d8fb>] ? handle_mm_fault+0x2e8/0x6ec
 [<c041f325>] ? kunmap_atomic+0x87/0xa7
 [<c06adcde>] ? do_page_fault+0x3ac/0x710
 [<c041f8b4>] ? resched_task+0x3a/0x6e
 [<c06abfcb>] ? _spin_unlock_irqrestore+0x22/0x38
 [<c0422597>] ? __dequeue_entity+0x73/0x7b
 [<c048edad>] ? __slab_alloc+0x56/0x45b
 [<c04036bf>] ? __switch_to+0xb9/0x139
 [<c06abe09>] ? _spin_lock+0x8/0xb
 [<c043f21a>] ? add_wait_queue+0x2b/0x30
 [<c04b99b5>] ? ep_ptable_queue_proc+0x58/0x69
 [<c0407c02>] sys_ipc+0x48/0x149
 [<c040b098>] ? syscall_trace_enter+0xb5/0xc9
 [<c0404c8a>] syscall_call+0x7/0xb
 =======================
Sched Debug Version: v0.07, 2.6.27.37-170.2.104.fc10.i686 #1
now at 100508767.613095 msecs
  .sysctl_sched_latency                    : 60.000000
  .sysctl_sched_min_granularity            : 12.000000
  .sysctl_sched_wakeup_granularity         : 15.000000
  .sysctl_sched_child_runs_first           : 0.000001
  .sysctl_sched_features                   : 15999

cpu#0, 2397.603 MHz
  .nr_running                    : 3
  .load                          : 5169
  .nr_switches                   : 39955626
  .nr_load_updates               : 31722464
  .nr_uninterruptible            : 4294962483
  .jiffies                       : 100208693
  .next_balance                  : 100.207487
  .curr->pid                     : 1419
  .clock                         : 100508693.131789
  .cpu_load[0]                   : 0
  .cpu_load[1]                   : 0
  .cpu_load[2]                   : 0
  .cpu_load[3]                   : 0
  .cpu_load[4]                   : 0

cfs_rq[0]:/
  .exec_clock                    : 7190027.411929
  .MIN_vruntime                  : 9629754.177543
  .min_vruntime                  : 9630117.494520
  .max_vruntime                  : 9630057.048647
  .spread                        : 302.871104
  .spread0                       : 0.000000
  .nr_running                    : 3
  .load                          : 5169
  .yld_exp_empty                 : 0
  .yld_act_empty                 : 0
  .yld_both_empty                : 0
  .yld_count                     : 5798
  .sched_switch                  : 0
  .sched_count                   : 57208443
  .sched_goidle                  : 33335034
  .ttwu_count                    : 21522783
  .ttwu_local                    : 19234927
  .bkl_count                     : 257158
  .nr_spread_over                : 227460
  .shares                        : 0

rt_rq[0]:/
  .rt_nr_running                 : 0
  .rt_throttled                  : 0
  .rt_time                       : 0.000000
  .rt_runtime                    : 950.000000

runnable tasks:
            task   PID         tree-key  switches  prio     exec-runtime
-------------------------------------------------------------------------------
        events/0    15   9630057.048647    166379   115   9630057.048647
6
           httpd  3387   9629754.177543     11379   120   9629754.177543
6
R nspluginviewer  1419   9632731.819953   3910675   120   9632731.819953
782

cpu#1, 2397.603 MHz
  .nr_running                    : 0
  .load                          : 0
  .nr_switches                   : 44557404
  .nr_load_updates               : 26043206
  .nr_uninterruptible            : 4294951028
  .jiffies                       : 100208693
  .next_balance                  : 100.208709
  .curr->pid                     : 0
  .clock                         : 100508772.127200
  .cpu_load[0]                   : 0
  .cpu_load[1]                   : 0
  .cpu_load[2]                   : 0
  .cpu_load[3]                   : 0
  .cpu_load[4]                   : 0

cfs_rq[1]:/
  .exec_clock                    : 10760814.641896
  .MIN_vruntime                  : 0.000001
  .min_vruntime                  : 16030451.392284
  .max_vruntime                  : 0.000001
  .spread                        : 0.000000
  .spread0                       : 6400333.897764
  .nr_running                    : 0
  .load                          : 0
  .yld_exp_empty                 : 0
  .yld_act_empty                 : 0
  .yld_both_empty                : 0
  .yld_count                     : 6012
  .sched_switch                  : 0
  .sched_count                   : 46573288
  .sched_goidle                  : 17609300
  .ttwu_count                    : 24786657
  .ttwu_local                    : 23332714
  .bkl_count                     : 329871
  .nr_spread_over                : 592231
  .shares                        : 0

rt_rq[1]:/
  .rt_nr_running                 : 0
  .rt_throttled                  : 0
  .rt_time                       : 0.000000
  .rt_runtime                    : 950.000000

runnable tasks:
            task   PID         tree-key  switches  prio     exec-runtime
-------------------------------------------------------------------------------

cpu#2, 2397.603 MHz
  .nr_running                    : 0
  .load                          : 0
  .nr_switches                   : 37194792
  .nr_load_updates               : 18136776
  .nr_uninterruptible            : 4294963669
  .jiffies                       : 100208693
  .next_balance                  : 100.208694
  .curr->pid                     : 0
  .clock                         : 100508693.131564
  .cpu_load[0]                   : 0
  .cpu_load[1]                   : 521
  .cpu_load[2]                   : 654
  .cpu_load[3]                   : 654
  .cpu_load[4]                   : 694

cfs_rq[2]:/
  .exec_clock                    : 4935114.000916
  .MIN_vruntime                  : 0.000001
  .min_vruntime                  : 7676888.540213
  .max_vruntime                  : 0.000001
  .spread                        : 0.000000
  .spread0                       : -1953228.954307
  .nr_running                    : 0
  .load                          : 0
  .yld_exp_empty                 : 0
  .yld_act_empty                 : 0
  .yld_both_empty                : 0
  .yld_count                     : 8319
  .sched_switch                  : 0
  .sched_count                   : 38119479
  .sched_goidle                  : 15526926
  .ttwu_count                    : 21012904
  .ttwu_local                    : 19121428
  .bkl_count                     : 108453
  .nr_spread_over                : 356716
  .shares                        : 0

rt_rq[2]:/
  .rt_nr_running                 : 0
  .rt_throttled                  : 0
  .rt_time                       : 0.000000
  .rt_runtime                    : 950.000000

runnable tasks:
            task   PID         tree-key  switches  prio     exec-runtime
-------------------------------------------------------------------------------

cpu#3, 2397.603 MHz
  .nr_running                    : 0
  .load                          : 0
  .nr_switches                   : 39364990
  .nr_load_updates               : 19956577
  .nr_uninterruptible            : 24732
  .jiffies                       : 100208693
  .next_balance                  : 100.208698
  .curr->pid                     : 0
  .clock                         : 100508778.377187
  .cpu_load[0]                   : 0
  .cpu_load[1]                   : 0
  .cpu_load[2]                   : 0
  .cpu_load[3]                   : 0
  .cpu_load[4]                   : 0

cfs_rq[3]:/
  .exec_clock                    : 7390465.685135
  .MIN_vruntime                  : 0.000001
  .min_vruntime                  : 8633311.798439
  .max_vruntime                  : 0.000001
  .spread                        : 0.000000
  .spread0                       : -996805.696081
  .nr_running                    : 0
  .load                          : 0
  .yld_exp_empty                 : 0
  .yld_act_empty                 : 0
  .yld_both_empty                : 0
  .yld_count                     : 4123
  .sched_switch                  : 0
  .sched_count                   : 39982547
  .sched_goidle                  : 15706527
  .ttwu_count                    : 21587076
  .ttwu_local                    : 20679894
  .bkl_count                     : 314888
  .nr_spread_over                : 391794
  .shares                        : 0

rt_rq[3]:/
  .rt_nr_running                 : 0
  .rt_throttled                  : 0
  .rt_time                       : 0.000000
  .rt_runtime                    : 950.000000

runnable tasks:
            task   PID         tree-key  switches  prio     exec-runtime
-------------------------------------------------------------------------------

It's greek to me, but I hope this helps.  Thanks again!

    - Karen