Re: Asynchronous and "direct" IO support for PostgreSQL. - Mailing list pgsql-hackers
| From | Wenjing Zeng |
|---|---|
| Subject | Re: Asynchronous and "direct" IO support for PostgreSQL. |
| Date | |
| Msg-id | 0DA61802-058F-421A-9C25-BA6588692599@gmail.com Whole thread Raw |
| In response to | Re: Asynchronous and "direct" IO support for PostgreSQL. (Andres Freund <andres@anarazel.de>) |
| List | pgsql-hackers |
2021年9月1日 13:56，Andres Freund <andres@anarazel.de> 写道：Hi,
Attached is an updated patch AIO series. The major changes are:
- rebased onto master (Andres)
- lots of progress on posix AIO backend (Thomas)
- lots of progress towards a windows native AIO implementation - not yet quite
merged (Thomas & David)
- considerably improved "worker" io_method (Thomas)
- some preliminary patches merged (Thomas) and thus dropped
- error handling overhaul, AIO references now use resource owners
- quite a few more localized bugfixes
- further CI improvements
Unfortunately there's a few tests that don't pass on windows. At least some of
those failures also happen on master - hence the alternative output file added
in the last commit.
Thanks to Thomas there's now a new wiki page for AIO support:
https://wiki.postgresql.org/wiki/AIO
It's currently mostly a shared todo list....
My own next steps are to try to get some of the preliminary patches merged
into master, and to address some of the higher level things that aren't yet
quite right with the AIO interface, and to split the "main" AIO patch into
smaller patches.
I hope that we soon send in a new version with native AIO support for
windows. I'm mostly interested in that to make sure that we get the shared
infrastructure right.
Melanie has some work improving bitmap heap scan AIO support and some IO stats
/ explain improvements.
I think a decent and reasonably simple example for the way the AIO interface
can be used to do faster IO is
v3-0028-aio-Use-AIO-in-nbtree-vacuum-scan.patch.gz which adds AIO for nbtree
vacuum. It's not perfectly polished, but I think it shows that it's not too
hard to add AIO usage to individual once the general infrastructure is in
place.
I've attached the code for posterity, but the series is large enough that I
don't think it makes sense to do that all that often... The code is at
https://github.com/anarazel/postgres/tree/aio
HI Andres:
I noticed this feature and did some testing.
code in GitHub's aio branch:
Function
static void pgaio_write_smgr_retry(PgAioInProgress *io) { uint32 off; AioBufferTag *tag = &io->scb_data.write_smgr.tag; SMgrRelation reln = smgropen(tag->rlocator.locator, tag->rlocator.backend); io->op_data.read.fd = smgrfd(reln, tag->forkNum, tag->blockNum, &off); Assert(off == io->op_data.read.offset); }
seems should to be: io->op_data.write.fd = smgrfd(reln, tag->forkNum, tag->blockNum, &off); Assert(off == io->op_data.write.offset);
Best regards,
Wenjing
Greetings,
Andres Freund
<v3-0001-windows-Only-consider-us-to-be-running-as-service.patch.gz><v3-0002-WIP-Fix-non-aio-bug-leading-to-checkpointer-not-s.patch.gz><v3-0003-aio-WIP-align-PGAlignedBlock-to-page-size.patch.gz><v3-0004-Add-allocator-support-for-larger-allocation-align.patch.gz><v3-0005-ilist.h-debugging-improvements.patch.gz><v3-0006-lwlock-xlog-Report-caller-wait-event-for-LWLockWa.patch.gz><v3-0007-heapam-Don-t-re-inquire-block-number-for-each-tup.patch.gz><v3-0008-Add-pg_prefetch_mem-macro-to-load-cache-lines.patch.gz><v3-0009-heapam-WIP-cacheline-prefetching-for-hot-pruning.patch.gz><v3-0010-WIP-Change-instr_time-to-just-store-nanoseconds-t.patch.gz><v3-0011-aio-Add-some-error-checking-around-pinning.patch.gz><v3-0012-aio-allow-lwlocks-to-be-unowned.patch.gz><v3-0013-condvar-add-ConditionVariableCancelSleepEx.patch.gz><v3-0014-Use-a-global-barrier-to-fix-DROP-TABLESPACE-on-Wi.patch.gz><v3-0015-pg_buffercache-Add-pg_buffercache_stats.patch.gz><v3-0016-bufmgr-Add-LockBufHdr-fastpath.patch.gz><v3-0017-lwlock-WIP-add-extended-locking-functions.patch.gz><v3-0018-io-Add-O_DIRECT-non-buffered-IO-mode.patch.gz><v3-0019-io-Increase-default-ringbuffer-size.patch.gz><v3-0020-Use-aux-process-resource-owner-in-walsender.patch.gz><v3-0021-Ensure-a-resowner-exists-for-all-paths-that-may-p.patch.gz><v3-0022-aio-Add-asynchronous-IO-infrastructure.patch.gz><v3-0023-aio-Use-AIO-in-pg_prewarm.patch.gz><v3-0024-aio-Use-AIO-in-bulk-relation-extension.patch.gz><v3-0025-aio-Use-AIO-in-checkpointer-bgwriter.patch.gz><v3-0026-aio-bitmap-heap-scan-Minimal-and-hacky-improvemen.patch.gz><v3-0027-aio-Use-AIO-in-heap-vacuum-s-lazy_scan_heap-and-l.patch.gz><v3-0028-aio-Use-AIO-in-nbtree-vacuum-scan.patch.gz><v3-0029-aio-Use-AIO-for-heap-table-scans.patch.gz><v3-0030-aio-Use-AIO-in-SyncDataDirectory.patch.gz><v3-0031-aio-Use-AIO-in-ProcessSyncRequests.patch.gz><v3-0032-aio-wal-concurrent-WAL-flushes.patch.gz><v3-0033-wal-Use-LWLockAcquireOrWait-in-AdvanceXLInsertBuf.patch.gz><v3-0034-wip-wal-async-commit-reduce-frequency-of-latch-se.patch.gz><v3-0035-aio-wal-padding-of-partial-records.patch.gz><v3-0036-aio-wal-extend-pg_stat_wal.patch.gz><v3-0037-aio-initial-sketch-for-design-document.patch.gz><v3-0038-aio-CI-and-README.md.patch.gz><v3-0039-XXX-Add-temporary-workaround-for-partition_prune-.patch.gz>
pgsql-hackers by date: