Thread: Slow application response on lightly loaded server?

Slow application response on lightly loaded server?

From
Mike Blackwell
Date:
We're seeing slow application performance on a PostgreSQL 9.1 server which appears to be relatively lightly loaded.  Some graphs from pgstatview are at  http://www2.uptimeforce.com/pgstatview/e35ba4e7db0842a1b9cf2e10a4c03d91/   These cover approximately 40 minutes, during which there was some activity from a web application and two bulk loads in process.  

The machine running the bulk loads (perl scripts) is also running at about 70% idle with very little iowait.  That seems to suggest network latency to me.

Am I missing something in the server stats that would indicate a problem?  If not, where should I look next?  

__________________________________________________________________________________
Mike Blackwell | Technical Analyst, Distribution Services/Rollout Management | RR Donnelley
1750 Wallace Ave | St Charles, IL 60174-3401
Office: 630.313.7818
Mike.Blackwell@rrd.com
http://www.rrdonnelley.com



Re: Slow application response on lightly loaded server?

From
Scott Marlowe
Date:
On Tue, Jul 17, 2012 at 10:27 AM, Mike Blackwell <mike.blackwell@rrd.com> wrote:
> We're seeing slow application performance on a PostgreSQL 9.1 server which
> appears to be relatively lightly loaded.  Some graphs from pgstatview are at
> http://www2.uptimeforce.com/pgstatview/e35ba4e7db0842a1b9cf2e10a4c03d91/
> These cover approximately 40 minutes, during which there was some activity
> from a web application and two bulk loads in process.
>
> The machine running the bulk loads (perl scripts) is also running at about
> 70% idle with very little iowait.  That seems to suggest network latency to
> me.
>
> Am I missing something in the server stats that would indicate a problem?
> If not, where should I look next?

I'd run vmstat and look for high cs or int numbers (100k and above) to
see if you're maybe seeing an issue with that.  A lot of times a
"slow" server is just too much process switching.  But yeah, the
graphs you've posted don't seem overly bad.

Re: Slow application response on lightly loaded server?

From
Mike Blackwell
Date:



On Tue, Jul 17, 2012 at 11:35 AM, Scott Marlowe <scott.marlowe@gmail.com> wrote:

I'd run vmstat and look for high cs or int numbers (100k and above) to
see if you're maybe seeing an issue with that.  A lot of times a
"slow" server is just too much process switching.  But yeah, the
graphs you've posted don't seem overly bad.


Thanks for the tip.  Here's a quick look at those numbers under that same load.  Watching it for a while longer didn't show any spikes.  That doesn't seem to be it, either.  

$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 3  0  11868  34500  16048 3931436    0    0     4     2    0     0  6  2 91  1
 2  0  11868  21964  16088 3931396    0    0     0   212 8667  8408 15  3 80  2
 0  0  11868  37772  16112 3932152    0    0     2   249 9109  8811 34  2 62  1
 2  0  11868  34068  16124 3932400    0    0     1   168 9142  9165 12  3 84  1
 1  0  11868  38036  16124 3932920    0    0     8   155 9995 10904 16  4 80  1
 1  0  11868  40212  16124 3933440    0    0     0   146 9586  9825 13  3 83  1

__________________________________________________________________________________
Mike Blackwell | Technical Analyst, Distribution Services/Rollout Management | RR Donnelley
1750 Wallace Ave | St Charles, IL 60174-3401
Office: 630.313.7818
Mike.Blackwell@rrd.com
http://www.rrdonnelley.com



Re: Slow application response on lightly loaded server?

From
Scott Marlowe
Date:
On Tue, Jul 17, 2012 at 11:37 AM, Mike Blackwell <mike.blackwell@rrd.com> wrote:
>
> On Tue, Jul 17, 2012 at 11:35 AM, Scott Marlowe <scott.marlowe@gmail.com>
> wrote:
>
>> I'd run vmstat and look for high cs or int numbers (100k and above) to
>> see if you're maybe seeing an issue with that.  A lot of times a
>> "slow" server is just too much process switching.  But yeah, the
>> graphs you've posted don't seem overly bad.
>
>
>
> Thanks for the tip.  Here's a quick look at those numbers under that same
> load.  Watching it for a while longer didn't show any spikes.  That doesn't
> seem to be it, either.

Yep it all looks good to me.  Are you sure you're not getting network
lag or something like that?

Re: Slow application response on lightly loaded server?

From
Mike Blackwell
Date:
I'm wondering about that.  However, the database server and the server doing the bulk loads are on the same subnet.  Traceroute shows only a single hop.  Traceroute and ping both show reply times in the area of .25 - .50 ms or so.  Is that reasonable?

__________________________________________________________________________________
Mike Blackwell | Technical Analyst, Distribution Services/Rollout Management | RR Donnelley
1750 Wallace Ave | St Charles, IL 60174-3401
Office: 630.313.7818
Mike.Blackwell@rrd.com
http://www.rrdonnelley.com





On Tue, Jul 17, 2012 at 12:49 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote:
On Tue, Jul 17, 2012 at 11:37 AM, Mike Blackwell <mike.blackwell@rrd.com> wrote:
>
> On Tue, Jul 17, 2012 at 11:35 AM, Scott Marlowe <scott.marlowe@gmail.com>
> wrote:
>
>> I'd run vmstat and look for high cs or int numbers (100k and above) to
>> see if you're maybe seeing an issue with that.  A lot of times a
>> "slow" server is just too much process switching.  But yeah, the
>> graphs you've posted don't seem overly bad.
>
>
>
> Thanks for the tip.  Here's a quick look at those numbers under that same
> load.  Watching it for a while longer didn't show any spikes.  That doesn't
> seem to be it, either.

Yep it all looks good to me.  Are you sure you're not getting network
lag or something like that?

Re: Slow application response on lightly loaded server?

From
Scott Marlowe
Date:
Yeah seems reasonable.  The last thing I'd look at is something like
improperly configured dns service.  Are you connecting by IP or by
host name?

On Tue, Jul 17, 2012 at 12:10 PM, Mike Blackwell <mike.blackwell@rrd.com> wrote:
> I'm wondering about that.  However, the database server and the server doing
> the bulk loads are on the same subnet.  Traceroute shows only a single hop.
> Traceroute and ping both show reply times in the area of .25 - .50 ms or so.
> Is that reasonable?
>
> __________________________________________________________________________________
> Mike Blackwell | Technical Analyst, Distribution Services/Rollout Management
> | RR Donnelley
> 1750 Wallace Ave | St Charles, IL 60174-3401
> Office: 630.313.7818
> Mike.Blackwell@rrd.com
> http://www.rrdonnelley.com
>
>
>
>
>
> On Tue, Jul 17, 2012 at 12:49 PM, Scott Marlowe <scott.marlowe@gmail.com>
> wrote:
>>
>> On Tue, Jul 17, 2012 at 11:37 AM, Mike Blackwell <mike.blackwell@rrd.com>
>> wrote:
>> >
>> > On Tue, Jul 17, 2012 at 11:35 AM, Scott Marlowe
>> > <scott.marlowe@gmail.com>
>> > wrote:
>> >
>> >> I'd run vmstat and look for high cs or int numbers (100k and above) to
>> >> see if you're maybe seeing an issue with that.  A lot of times a
>> >> "slow" server is just too much process switching.  But yeah, the
>> >> graphs you've posted don't seem overly bad.
>> >
>> >
>> >
>> > Thanks for the tip.  Here's a quick look at those numbers under that
>> > same
>> > load.  Watching it for a while longer didn't show any spikes.  That
>> > doesn't
>> > seem to be it, either.
>>
>> Yep it all looks good to me.  Are you sure you're not getting network
>> lag or something like that?
>
>



--
To understand recursion, one must first understand recursion.

Re: Slow application response on lightly loaded server?

From
Mike Blackwell
Date:
On Tue, Jul 17, 2012 at 2:36 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote:
 
Yeah seems reasonable.  The last thing I'd look at is something like
improperly configured dns service.  Are you connecting by IP or by
host name?

 
Interesting possibility.  We're currently connecting by host name.  I could try temporarily using the IP from one of the servers to see if that helps.  I'm not familiar enough with DNS services to do any diagnostics other than using dig to see where something points.

Thanks for your help, BTW!

__________________________________________________________________________________
Mike Blackwell | Technical Analyst, Distribution Services/Rollout Management | RR Donnelley
1750 Wallace Ave | St Charles, IL 60174-3401
Office: 630.313.7818
Mike.Blackwell@rrd.com
http://www.rrdonnelley.com


Re: Slow application response on lightly loaded server?

From
Scott Marlowe
Date:
Well if it suddenly gets faster when connecting by IP, you'll know
where your problem lies.  DNS issues are more common in windows
installs, due to Windows having more interesting ways to misconfigure
dns etc.

On Tue, Jul 17, 2012 at 1:48 PM, Mike Blackwell <mike.blackwell@rrd.com> wrote:
> On Tue, Jul 17, 2012 at 2:36 PM, Scott Marlowe <scott.marlowe@gmail.com>
> wrote:
>
>
>>
>> Yeah seems reasonable.  The last thing I'd look at is something like
>> improperly configured dns service.  Are you connecting by IP or by
>> host name?
>>
>
> Interesting possibility.  We're currently connecting by host name.  I could
> try temporarily using the IP from one of the servers to see if that helps.
> I'm not familiar enough with DNS services to do any diagnostics other than
> using dig to see where something points.
>
> Thanks for your help, BTW!
>
> __________________________________________________________________________________
> Mike Blackwell | Technical Analyst, Distribution Services/Rollout Management
> | RR Donnelley
> 1750 Wallace Ave | St Charles, IL 60174-3401
> Office: 630.313.7818
> Mike.Blackwell@rrd.com
> http://www.rrdonnelley.com
>
>



--
To understand recursion, one must first understand recursion.