Re: Blocking I/O, async I/O and io_uring - Mailing list pgsql-hackers

From Craig Ringer
Subject Re: Blocking I/O, async I/O and io_uring
Date
Msg-id CAGRY4nzG_U=nnSFq8FwuJhhYVFWNOtdTNTKt_8rrRDqpwTSGyg@mail.gmail.com
Whole thread Raw
In response to Re: Blocking I/O, async I/O and io_uring  (Andres Freund <andres@anarazel.de>)
List pgsql-hackers
On Tue, 8 Dec 2020 at 15:04, Andres Freund <andres@anarazel.de> wrote:
Hi,

On 2020-12-08 04:24:44 +0000, tsunakawa.takay@fujitsu.com wrote:
> I'm looking forward to this from the async+direct I/O, since the
> throughput of some write-heavy workload decreased by half or more
> during checkpointing (due to fsync?)

Depends on why that is. The most common, I think, cause is that your WAL
volume increases drastically just after a checkpoint starts, because
initially all page modification will trigger full-page writes.  There's
a significant slowdown even if you prevent the checkpointer from doing
*any* writes at that point.  I got the WAL AIO stuff to the point that I
see a good bit of speedup at high WAL volumes, and I see it helping in
this scenario.

There's of course also the issue that checkpoint writes cause other IO
(including WAL writes) to slow down and, importantly, cause a lot of
jitter leading to unpredictable latencies.  I've seen some good and some
bad results around this with the patch, but there's a bunch of TODOs to
resolve before delving deeper really makes sense (the IO depth control
is not good enough right now).

A third issue is that sometimes checkpointer can't really keep up - and
that I think I've seen pretty clearly addressed by the patch. I have
managed to get to ~80% of my NVMe disks top write speed (> 2.5GB/s) by
the checkpointer, and I think I know what to do for the remainder.


Thanks for explaining this. I'm really glad you're looking into it. If I get the chance I'd like to try to apply some wait-analysis and blocking stats tooling to it. I'll report back if I make any progress there.

pgsql-hackers by date:

Previous
From: Kyotaro Horiguchi
Date:
Subject: Re: [Patch] Optimize dropping of relation buffers using dlist
Next
From: Andres Freund
Date:
Subject: Occasional tablespace.sql failures in check-world -jnn