Thread: High load average in 64-core server , no I/O wait and CPU is idle

Dear List ,

We are having scalability issues with a high end hardware

The  hardware is
CPU  = 4 *  opteron 6272 with 16 cores ie Total = 64 cores.
RAM  = 128 GB DDR3
Disk = High performance RAID10 with lots of 15K spindles and a working BBU Cache.

normally the 1 min load average of the system remains between 0.5 to 1.0 .

The problem is that  sometimes there are spikes of load avg which
jumps to > 50 very rapidly ( ie from 0.5 to 50  within 10 secs) and
it remains there for sometime and slowly reduces to normal value.

During such times of high load average we observe that there is no IO wait
in system and even CPU is 50% idle. In any case the IO Wait always remains < 1.0 % and
is mostly 0. Hence the load is not due to high I/O wait which was generally
the case with our previous hardware.

We are  puzzled why the CPU and DISK I/O system are not being utilized
fully and would seek lists' wisdom on that.

We have setup sar to poll the system parameters every minute and
the data of which is graphed with cacti. If required any of the
system parameters or postgresql parameter can easily be  put under
cacti monitoring and can be graphed.

The query load is mostly read only.

It is also possible to replicate the problem with pg_bench to some
extent . I choose -s = 100 and -t=10000 , the load does shoot but not
that spectacularly as achieved by the real world usage.

any help shall be greatly appreciated.

just a thought, will it be a good idea to partition the host hardware
to 4 equal  virtual environments , ie 1 for master (r/w) and 3 slaves r/o
and distribute the r/o load on the 3 slaves ?


regds
mallah

On Thu, May 24, 2012 at 12:39 AM, Rajesh Kumar. Mallah
<mallah@tradeindia.com> wrote:
> The problem is that  sometimes there are spikes of load avg which
> jumps to > 50 very rapidly ( ie from 0.5 to 50  within 10 secs) and
> it remains there for sometime and slowly reduces to normal value.
>
> During such times of high load average we observe that there is no IO wait
> in system and even CPU is 50% idle. In any case the IO Wait always remains < 1.0 % and
> is mostly 0. Hence the load is not due to high I/O wait which was generally
> the case with our previous hardware.

Do you experience decreased query performance?

Load can easily get to 64 (1 per core) without reaching its capacity.
So, unless you're experiencing decreased performance I wouldn't think
much of it.

Do you have mcelog running? as a cron or a daemon?
Sometimes, mcelog tends to crash in that way. We had to disable it in
our servers because it misbehaved like that. It only makes load avg
meaningless, no performance impact, but being unable to accurately
measure load is bad enough.

----- "Claudio Freire" <klaussfreire@gmail.com> wrote:

| From: "Claudio Freire" <klaussfreire@gmail.com>
| To: "Rajesh Kumar. Mallah" <mallah@tradeindia.com>
| Cc: pgsql-performance@postgresql.org
| Sent: Thursday, May 24, 2012 9:23:43 AM
| Subject: Re: [PERFORM] High load average in 64-core server , no I/O wait and CPU is idle
|
| On Thu, May 24, 2012 at 12:39 AM, Rajesh Kumar. Mallah
| <mallah@tradeindia.com> wrote:
| > The problem is that  sometimes there are spikes of load avg which
| > jumps to > 50 very rapidly ( ie from 0.5 to 50  within 10 secs) and
| > it remains there for sometime and slowly reduces to normal value.
| >
| > During such times of high load average we observe that there is no
| IO wait
| > in system and even CPU is 50% idle. In any case the IO Wait always
| remains < 1.0 % and
| > is mostly 0. Hence the load is not due to high I/O wait which was
| generally
| > the case with our previous hardware.
|
| Do you experience decreased query performance?


Yes we do experience substantial application performance degradations.


|
| Load can easily get to 64 (1 per core) without reaching its capacity.
| So, unless you're experiencing decreased performance I wouldn't think
| much of it.

I far as i understand ,
Load Avg is the average number of processes waiting to be run in past 1 ,
5 or 15 mins. A number > 1 would mean that countable number of processes
were waiting to be run. how can load of more than 1 and upto 64 be OK
for a 64 core machine ?



|
| Do you have mcelog running? as a cron or a daemon?


No we do not have mcelog.

BTW the Postgresql version is : 9.1.3 which i forgot to mention
in my last email.


regds
mallah.

| Sometimes, mcelog tends to crash in that way. We had to disable it in
| our servers because it misbehaved like that. It only makes load avg
| meaningless, no performance impact, but being unable to accurately
| measure load is bad enough.

On Thu, May 24, 2012 at 2:26 AM, Rajesh Kumar. Mallah
<mallah@tradeindia.com> wrote:
> |
> | Load can easily get to 64 (1 per core) without reaching its capacity.
> | So, unless you're experiencing decreased performance I wouldn't think
> | much of it.
>
> I far as i understand ,
> Load Avg is the average number of processes waiting to be run in past 1 ,
> 5 or 15 mins. A number > 1 would mean that countable number of processes
> were waiting to be run. how can load of more than 1 and upto 64 be OK
> for a 64 core machine ?

Load avg is the number of processes in the running queue, which can be
either waiting to be run or actually running.

So if you had 100% CPU usage, then you'd most definitely have a load
avg of 64, which is neither good or bad. It may simply mean that
you're using your hardware's full potential.

If your processes are waiting but not using CPU or I/O time... all I
can think of is mcelog (it's the only application I've ever witnessed
doing that). Do check ps/top and try to find out which processes are
in a waiting state to have a little more insight.

On 05/24/2012 12:26 AM, Rajesh Kumar. Mallah wrote:
>
> ----- "Claudio Freire"<klaussfreire@gmail.com>  wrote:
>
> | From: "Claudio Freire"<klaussfreire@gmail.com>
> | To: "Rajesh Kumar. Mallah"<mallah@tradeindia.com>
> | Cc: pgsql-performance@postgresql.org
> | Sent: Thursday, May 24, 2012 9:23:43 AM
> | Subject: Re: [PERFORM] High load average in 64-core server , no I/O wait and CPU is idle
> |
> | On Thu, May 24, 2012 at 12:39 AM, Rajesh Kumar. Mallah
> |<mallah@tradeindia.com>  wrote:
> |>  The problem is that  sometimes there are spikes of load avg which
> |>  jumps to>  50 very rapidly ( ie from 0.5 to 50  within 10 secs) and
> |>  it remains there for sometime and slowly reduces to normal value.
> |>
> |>  During such times of high load average we observe that there is no
> | IO wait
> |>  in system and even CPU is 50% idle. In any case the IO Wait always
> | remains<  1.0 % and
> |>  is mostly 0. Hence the load is not due to high I/O wait which was
> | generally
> |>  the case with our previous hardware.
> |
> | Do you experience decreased query performance?
>
>
> Yes we do experience substantial application performance degradations.
>
>

Maybe you are hitting some locks?   If its not IO and not CPU then maybe something is getting locked and queries are
pilingup. 

-Andy

Rajesh,

* Rajesh Kumar. Mallah (mallah@tradeindia.com) wrote:
> We are  puzzled why the CPU and DISK I/O system are not being utilized
> fully and would seek lists' wisdom on that.

What OS is this?  What kernel version?

> just a thought, will it be a good idea to partition the host hardware
> to 4 equal  virtual environments , ie 1 for master (r/w) and 3 slaves r/o
> and distribute the r/o load on the 3 slaves ?

Actually, it might help with 9.1, if you're really running into some
scalability issues in our locking area..  You might review this:

http://rhaas.blogspot.com/2012/04/did-i-say-32-cores-how-about-64.html

That's a pretty contrived test case, but I suppose it's possible your
case is actually close enough to be getting affected also..

    Thanks,

        Stephen

----- "Stephen Frost" <sfrost@snowman.net> wrote:

| From: "Stephen Frost" <sfrost@snowman.net>
| To: "Rajesh Kumar. Mallah" <mallah@tradeindia.com>
| Cc: pgsql-performance@postgresql.org
| Sent: Thursday, May 24, 2012 9:27:37 PM
| Subject: Re: [PERFORM] High load average in 64-core server ,  no I/O wait and CPU is idle
|
| Rajesh,
|
| * Rajesh Kumar. Mallah (mallah@tradeindia.com) wrote:
| > We are  puzzled why the CPU and DISK I/O system are not being
| utilized
| > fully and would seek lists' wisdom on that.
|
| What OS is this?  What kernel version?

Dear Frost ,

We are running linux with kernel 3.2.X
(which has the lseek improvements)

|
| > just a thought, will it be a good idea to partition the host
| hardware
| > to 4 equal  virtual environments , ie 1 for master (r/w) and 3
| slaves r/o
| > and distribute the r/o load on the 3 slaves ?
|
| Actually, it might help with 9.1, if you're really running into some
| scalability issues in our locking area..  You might review this:
|
| http://rhaas.blogspot.com/2012/04/did-i-say-32-cores-how-about-64.html
|
| That's a pretty contrived test case, but I suppose it's possible your
| case is actually close enough to be getting affected also..

Thanks for the reference , even i thought so (LockManager) ,
but we are actually also running out db max connections (also)
( which is currently at 600) , when that happens  something at
the beginning of the application stack also gets dysfunctional and it
changes the very input to the system. ( think of negative feedback systems )

It is sort of complicated but i will definitely update list ,
when i get to  the point of putting the blame on DB  :-) .

Regds
Mallah.

|
|     Thanks,
|
|         Stephen

* Rajesh Kumar. Mallah (mallah@tradeindia.com) wrote:
> We are running linux with kernel 3.2.X
> (which has the lseek improvements)

Ah, good.

> Thanks for the reference , even i thought so (LockManager) ,
> but we are actually also running out db max connections (also)
> ( which is currently at 600) , when that happens  something at
> the beginning of the application stack also gets dysfunctional and it
> changes the very input to the system. ( think of negative feedback systems )

Oh.  Yeah, have you considered pgbouncer?

> It is sort of complicated but i will definitely update list ,
> when i get to  the point of putting the blame on DB  :-) .

Ok. :)

    Thanks,

        Stephen

On Thu, May 24, 2012 at 2:09 PM, Stephen Frost <sfrost@snowman.net> wrote:
> * Rajesh Kumar. Mallah (mallah@tradeindia.com) wrote:
>> We are running linux with kernel 3.2.X
>> (which has the lseek improvements)
>
> Ah, good.
>
>> Thanks for the reference , even i thought so (LockManager) ,
>> but we are actually also running out db max connections (also)
>> ( which is currently at 600) , when that happens  something at
>> the beginning of the application stack also gets dysfunctional and it
>> changes the very input to the system. ( think of negative feedback systems )
>
> Oh.  Yeah, have you considered pgbouncer?

Or pooling at the application level. Many ORMs support connection
pooling and limiting out-of-the-box.

In essence, postgres should never bounce connections, it should all be
handled by the application or a previous pgbouncer, both of which
would do it more efficient and effectively.