Re: [HACKERS] Dynamic instrumentation of lwlock wait times (lwlock flamegraphs) - Mailing list pgsql-hackers

From Andres Freund
Subject Re: [HACKERS] Dynamic instrumentation of lwlock wait times (lwlock flamegraphs)
Date
Msg-id 20200813004233.hdsdfvufqrbdwzgr@alap3.anarazel.de
Whole thread Raw
In response to [HACKERS] Dynamic instrumentation of lwlock wait times (lwlock flamegraphs)  (Andres Freund <andres@anarazel.de>)
List pgsql-hackers
Hi,

On 2017-06-22 14:08:45 -0700, Andres Freund wrote:
> At pgcon some people were talking about the difficulty of instrumenting
> the time actually spent waiting for lwlocks and related measurements.
> I'd mentioned that linux these days provides infrastructure to measure
> such things in unmodified binaries.
> 
> Attached is a prototype of a script that measures the time spent inside
> PGSemaphoreLock(), aggregates that inside the kernel, grouped by pid and
> stacktrace.  That allows one to generate nice flame graphs showing which
> part of the code waits how long for lwlocks.
> 
> The attached script clearly needs improvements, but I thought I'd show
> some of the results it can get.  To run it you need the the python
> library of the 'bcc' project [1], and a sufficiently new kernel.  Some
> distributions, e.g. newer debian versions, package this as python-bpfcc
> and similar.
> 
> The output of the script can be turned into a flamegraph with
> https://github.com/brendangregg/FlameGraph 's flamegraph.pl.

The script has bitrot slightly, due to python3 and postgres changes (the
move to posix semaphores). Updated version attached.

Based on the discussion in
https://www.postgresql.org/message-id/20200813003934.yrm4qqngfygr6ii7%40alap3.anarazel.de

Greetings,

Andres Freund

Attachment

pgsql-hackers by date:

Previous
From: Andres Freund
Date:
Subject: Re: Add LWLock blocker(s) information
Next
From: Peter Geoghegan
Date:
Subject: Re: Add LWLock blocker(s) information