Re: hung backends stuck in spinlock heavy endless loop - Mailing list pgsql-hackers

From Merlin Moncure
Subject Re: hung backends stuck in spinlock heavy endless loop
Date
Msg-id CAHyXU0zPQk_y7SohNhbGrT9xiPTY8w+n0Sd0P2UFPQ=ruY1g+Q@mail.gmail.com
Whole thread Raw
In response to Re: hung backends stuck in spinlock heavy endless loop  (Andres Freund <andres@2ndquadrant.com>)
Responses Re: hung backends stuck in spinlock heavy endless loop
List pgsql-hackers
On Wed, Jan 14, 2015 at 9:30 AM, Andres Freund <andres@2ndquadrant.com> wrote:
> If you gdb in, and type 'fin' a couple times, to wait till the function
> finishes, is there actually any progress? I'm wondering whether it's
> just many catalog accesses + contention, or some other
> problem. Alternatively set a breakpoint on ScanPgRelation() or so and
> see how often it's hit.

well, i restarted the database, forgetting my looper was running which
immediately spun up and it got stuck again with a similar profile
(lots of cpu in spinlock):

Samples: 3K of event 'cycles', Event count (approx.): 269572322831.16%  postgres            [.] s_lock22.32%  postgres
         [.] tas12.13%  postgres            [.] tas 5.93%  postgres            [.] spin_delay 5.69%  postgres
[.] LWLockRelease 3.75%  postgres            [.] LWLockAcquireCommon 3.61%  perf                [.] 0x00000000000526c4
2.51% postgres            [.] FunctionCall2Coll 1.48%  libc-2.17.so        [.] 0x000000000016a132
 

> If you gdb in, and type 'fin' a couple times,
(gdb) fin
Run till exit from #0  0x00007ff4c63f7a97 in semop () from
/lib/x86_64-linux-gnu/libc.so.6
0x00000000006de073 in PGSemaphoreLock ()
(gdb) fin
Run till exit from #0  0x00000000006de073 in PGSemaphoreLock ()

It returned once.  Second time, it didn't at least so far (minute or so).

>> I can send the code off-list if you guys think it'd help.
>
> Might be interesting.

sent (off-list).

merlin



pgsql-hackers by date:

Previous
From: Tom Lane
Date:
Subject: Re: Merging postgresql.conf and postgresql.auto.conf
Next
From: Andres Freund
Date:
Subject: Re: hung backends stuck in spinlock heavy endless loop