Postgres Pro
Downloads
Home   >   mailing lists

Re: CPU load spikes when CentOS tries to reclaim 'cached' memory - Mailing list pgsql-performance

From Deron
Subject Re: CPU load spikes when CentOS tries to reclaim 'cached' memory
Date
Msg-id CAF3Lvs5HPj9=8UV8bFuXqXk5G-=8spg9MoYkkt11oZd6SXFe4g@mail.gmail.com
Whole thread Raw
In response to Re: CPU load spikes when CentOS tries to reclaim 'cached' memory  (Merlin Moncure <mmoncure@gmail.com>)
Responses Re: CPU load spikes when CentOS tries to reclaim 'cached' memory
List pgsql-performance
Tree view
We saw very similar issues with a CentOS server with 40 cores (32 virtualized) when moving from a physical server to a virtual server (I think it had 128GB RAM).   Never had the problem on a physical server.  We checked the same things as noted here, but never found a bug.   We really thought it had something to do with NUMA zone reclaim, but could never prove that.    In our case it was all kernel time in the guest, all CPUs at 100%.   Sometimes it would last for a few seconds or minutes.  Sometimes we would go days without a problem, and then it would completely tank.  

If you figure out what is going on, I would like to know  (especially if it is virtualized).

Deron



On Thu, Jun 5, 2014 at 12:23 PM, Merlin Moncure <mmoncure@gmail.com> wrote:
On Thu, Jun 5, 2014 at 10:58 AM, Jeff Janes <jeff.janes@gmail.com> wrote:
> This sounds like a kernel problem, probably either the zone reclaim issue,
> or the transparent huge pages issue.

I at first thought maybe same, but I don't think THP was introduced
until 2.6.38...OP is running 2.6.32-431.11.2.el6.x86_6.  Maybe it's
NUMA related, but would not be idiomatic of NUMA issues as I
understand them (poor memory utilization/high IO utilization).  Would
be a very cheap/easy thing to try though.

Is this server virtualized?

merlin


--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

pgsql-performance by date:

Previous
From: Weinzierl Stefan
Date:
Subject: Seqscan on big table, when an Index-Usage should be possible
Next
From: Merlin Moncure
Date:
Subject: Re: CPU load spikes when CentOS tries to reclaim 'cached' memory