Re: High load and iowait but no disk access

From: Michael Stone
Subject: Re: High load and iowait but no disk access
Date: ,
Msg-id: 20050830162525.GU14921@mathom.us
(view: Whole thread, Raw)
In response to: High load and iowait but no disk access  (Rémy Beaumont)
List: pgsql-performance

Tree view

High load and iowait but no disk access  (Rémy Beaumont, )
 Re: High load and iowait but no disk access  (Rémy Beaumont, )
  Re: High load and iowait but no disk access  (Tom Lane, )
   Re: High load and iowait but no disk access  (Rémy Beaumont, )
 Re: High load and iowait but no disk access  (Michael Stone, )
 Re: High load and iowait but no disk access  (Tom Lane, )
 Re: High load and iowait but no disk access  (Josh Berkus, )
  Re: High load and iowait but no disk access  (Rémy Beaumont, )
 Re: High load and iowait but no disk access  ("Woody Woodring", )
 Re: High load and iowait but no disk access  (Rémy Beaumont, )
 Re: High load and iowait but no disk access  ("Anjan Dave", )
  Re: High load and iowait but no disk access  (, )

On Mon, Aug 29, 2005 at 09:42:46AM -0400, Rémy Beaumont wrote:
>We have been trying to pinpoint what originally seem to be a I/O
>bottleneck but which now seems to be an issue with either Postgresql or
>RHES 3.

Nope, it's an IO bottleneck.

>The behavior we see is that when running queries that do random reads
>on disk, IOWAIT goes over 80% and actual disk IO falls to a crawl at a
>throughput bellow 3000kB/s

That's the sign of an IO bottleneck.

>The stats of the NetApp do confirm that it is sitting idle. Doing an
>strace on the Postgresql process shows that is it doing seeks and
>reads.
>
>So my question is where is this iowait time spent ?

Waiting for the seeks. postgres doesn't do async io, so it requests a
block, waits for it to come in, then requests another block, etc. The
utilization on the netapp isn't going to be high because it doesn't have
a queue of requests and can't do readahead because the IO is random. The
only way to improve the situation would be to reduce the latency of the
seeks. If I read the numbers right you're only getting about 130
seeks/s, which ain't great. I don't know how much latency the netapp
adds in the this configuration; have you tried benchmarking
direct-attach disks?

Mike Stone


pgsql-performance by date:

From: Ralph Mason
Date:
Subject: 'Real' auto vacuum?
From: mark@mark.mielke.cc
Date:
Subject: Re: 'Real' auto vacuum?