Re: hung backends stuck in spinlock heavy endless loop - Mailing list pgsql-hackers
| From | Merlin Moncure |
|---|---|
| Subject | Re: hung backends stuck in spinlock heavy endless loop |
| Date | |
| Msg-id | CAHyXU0xZ1pUjtnGYnpVPG231Auaf9s_cssPNSqQDVXWAL1aVmA@mail.gmail.com Whole thread Raw |
| In response to | Re: hung backends stuck in spinlock heavy endless loop (Merlin Moncure <mmoncure@gmail.com>) |
| Responses |
Re: hung backends stuck in spinlock heavy endless loop
|
| List | pgsql-hackers |
On Wed, Jan 14, 2015 at 8:03 AM, Merlin Moncure <mmoncure@gmail.com> wrote: > Here's a backtrace: > > #0 0x0000000000750a97 in spin_delay () > #1 0x0000000000750b19 in s_lock () > #2 0x0000000000750844 in LWLockRelease () > #3 0x0000000000730000 in LockBuffer () > #4 0x00000000004b2db4 in _bt_relandgetbuf () > #5 0x00000000004b7116 in _bt_moveright () > #6 0x00000000004b6e02 in _bt_search () > #7 0x00000000004b8006 in _bt_first () > #8 0x00000000004b5aad in btgettuple () > #9 0x0000000000896656 in FunctionCall2Coll () > #10 0x00000000004acd55 in index_getnext_tid () > #11 0x00000000004acfad in index_getnext () There were seven process with that backtrace exact backtrace (except that randomly they are sleeping in the spinloop). Something else interesting: autovacuum has been running all night as well. Unlike the other process however, cpu utilization does not register on top. I backtraced them as well. They are not identical. One of them looks like this: #0 0x00007f89c1b21a97 in semop () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00000000006de073 in PGSemaphoreLock () #2 0x000000000074e398 in ProcWaitForSignal () #3 0x00000000007302cb in LockBufferForCleanup () #4 0x00000000004b699e in btvacuumpage () #5 0x00000000004b6627 in btvacuumscan () #6 0x00000000004b6389 in btbulkdelete () #7 0x00000000008968d7 in FunctionCall4Coll () #8 0x00000000004ad196 in index_bulk_delete () #9 0x000000000060b9ed in lazy_vacuum_index () #10 0x000000000060b019 in lazy_scan_heap () #11 0x0000000000609836 in lazy_vacuum_rel () #12 0x0000000000609383 in vacuum_rel () #13 0x0000000000607e4a in vacuum () #14 0x00000000006e2d40 in autovacuum_do_vac_analyze () #15 0x00000000006e2127 in do_autovacuum () #16 0x00000000006e0f19 in AutoVacWorkerMain () #17 0x00000000006e0b06 in StartAutoVacWorker () #18 0x00000000006f4fac in StartAutovacuumWorker () #19 0x00000000006f4816 in sigusr1_handler () #20 <signal handler called> #21 0x00007f89c1b17dc3 in select () from /lib/x86_64-linux-gnu/libc.so.6 #22 0x00000000006f0619 in ServerLoop () #23 0x00000000006efdec in PostmasterMain () #24 0x00000000006594ec in main () The other autovacuum process (0 cpu usage) looks like this: #0 0x00007f89c1b17dc3 in select () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00000000008d3ac4 in pg_usleep () #2 0x0000000000750b69 in s_lock () #3 0x0000000000750844 in LWLockRelease () #4 0x0000000000730000 in LockBuffer () #5 0x00000000004b2db4 in _bt_relandgetbuf () #6 0x00000000004b7116 in _bt_moveright () #7 0x00000000004b6e02 in _bt_search () #8 0x00000000004b8006 in _bt_first () #9 0x00000000004b5aad in btgettuple () #10 0x0000000000896656 in FunctionCall2Coll () #11 0x00000000004acd55 in index_getnext_tid () #12 0x00000000004acfad in index_getnext () #13 0x00000000004ac120 in systable_getnext () #14 0x0000000000878ce4 in ScanPgRelation () #15 0x000000000087bcc4 in RelationReloadIndexInfo () #16 0x000000000087c1f1 in RelationClearRelation () #17 0x000000000087ca37 in RelationCacheInvalidate () #18 0x0000000000875fce in InvalidateSystemCaches () #19 0x0000000000741d81 in ReceiveSharedInvalidMessages () #20 0x0000000000876080 in AcceptInvalidationMessages () #21 0x0000000000745ce3 in LockRelationOid () #22 0x000000000049510b in relation_open () #23 0x0000000000495359 in heap_open () #24 0x0000000000608694 in vac_update_relstats () #25 0x000000000060bb98 in lazy_cleanup_index () #26 0x000000000060b0b6 in lazy_scan_heap () #27 0x0000000000609836 in lazy_vacuum_rel () #28 0x0000000000609383 in vacuum_rel () #29 0x0000000000607e4a in vacuum () #30 0x00000000006e2d40 in autovacuum_do_vac_analyze () #31 0x00000000006e2127 in do_autovacuum () #32 0x00000000006e0f19 in AutoVacWorkerMain () #33 0x00000000006e0b06 in StartAutoVacWorker () #34 0x00000000006f4fac in StartAutovacuumWorker () #35 0x00000000006f4816 in sigusr1_handler () #36 <signal handler called> #37 0x00007f89c1b17dc3 in select () from /lib/x86_64-linux-gnu/libc.so.6 #38 0x00000000006f0619 in ServerLoop () #39 0x00000000006efdec in PostmasterMain () #40 0x00000000006594ec in main () merlin
pgsql-hackers by date: