Re: CPU spikes and transactions - Mailing list pgsql-performance

From Tony Kay
Subject Re: CPU spikes and transactions
Date
Msg-id CAB=fRcqq1SU9zFiKMkkf0wnTS0eEosLEf=B6_teRvvoVkr=x9w@mail.gmail.com
Whole thread Raw
In response to Re: CPU spikes and transactions  (Merlin Moncure <mmoncure@gmail.com>)
List pgsql-performance
Thanks for the tip. I forgot there were kernel stats on spinlocks.

I'm not sure we'll be able to get it to tip in a test environment, and we're unwilling to revert the code in production in order to have our users trigger it. We'll try triggering it on our test server, and if we manage, I'll get you the stats.

Thanks!

Tony




On Tue, Oct 15, 2013 at 6:00 AM, Merlin Moncure <mmoncure@gmail.com> wrote:
On Mon, Oct 14, 2013 at 6:45 PM, Tomas Vondra <tv@fuzzy.cz> wrote:
> On 15.10.2013 01:26, Tony Kay wrote:
>> Hi Calvin,
>>
>> Yes, I have sar data on all systems going back for years.
>>
>> Since others are going to probably want to be assured I am really
>> "reading the data" right:
>>
>> - This is 92% user CPU time, 5% sys, and 1% soft
>> - On some of the problems, I _do_ see a short spike of pgswpout's
>> (memory pressure), but again, not enough to end up using much system time
>> - The database disks are idle (all data being used is in RAM)..and are
>> SSDs....average service times are barely measurable in ms.
>
> OK. Can you share the data? Maybe we'll notice something suspicious.
>
>> If I had to guess, I'd say it was spinlock misbehavior....I cannot
>> understand why ekse a transaction blocking other things would drive
>> the CPUs so hard into the ground with user time.
>
> Have you tried running perf, to verify the time is actually spent on
> spinlocks?

+1 this.  It is almost certainly spinlocks, but we need to know which
one and why.  plz install debug symbols and run a perf during normal
and high load conditions.

merlin


--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

pgsql-performance by date:

Previous
From: Tony Kay
Date:
Subject: Re: CPU spikes and transactions
Next
From: Julien Cigar
Date:
Subject: Re: CPU spikes and transactions