Thread: pgsql: Prefetch data referenced by the WAL, take II.
Prefetch data referenced by the WAL, take II. Introduce a new GUC recovery_prefetch. When enabled, look ahead in the WAL and try to initiate asynchronous reading of referenced data blocks that are not yet cached in our buffer pool. For now, this is done with posix_fadvise(), which has several caveats. Since not all OSes have that system call, "try" is provided so that it can be enabled where available. Better mechanisms for asynchronous I/O are possible in later work. Set to "try" for now for test coverage. Default setting to be finalized before release. The GUC wal_decode_buffer_size limits the distance we can look ahead in bytes of decoded data. The existing GUC maintenance_io_concurrency is used to limit the number of concurrent I/Os allowed, based on pessimistic heuristics used to infer that I/Os have begun and completed. We'll also not look more than maintenance_io_concurrency * 4 block references ahead. Reviewed-by: Julien Rouhaud <rjuju123@gmail.com> Reviewed-by: Tomas Vondra <tomas.vondra@2ndquadrant.com> Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> (earlier version) Reviewed-by: Andres Freund <andres@anarazel.de> (earlier version) Reviewed-by: Justin Pryzby <pryzby@telsasoft.com> (earlier version) Tested-by: Tomas Vondra <tomas.vondra@2ndquadrant.com> (earlier version) Tested-by: Jakub Wartak <Jakub.Wartak@tomtom.com> (earlier version) Tested-by: Dmitry Dolgov <9erthalion6@gmail.com> (earlier version) Tested-by: Sait Talha Nisanci <Sait.Nisanci@microsoft.com> (earlier version) Discussion: https://postgr.es/m/CA%2BhUKGJ4VJN8ttxScUFM8dOKX0BrBiboo5uz1cq%3DAovOddfHpA%40mail.gmail.com Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/5dc0418fab281d017a61a5756240467af982bdfd Modified Files -------------- doc/src/sgml/config.sgml | 64 ++ doc/src/sgml/monitoring.sgml | 86 +- doc/src/sgml/wal.sgml | 12 + src/backend/access/transam/Makefile | 1 + src/backend/access/transam/xlog.c | 2 + src/backend/access/transam/xlogprefetcher.c | 1082 +++++++++++++++++++++++++ src/backend/access/transam/xlogreader.c | 27 +- src/backend/access/transam/xlogrecovery.c | 179 ++-- src/backend/access/transam/xlogutils.c | 27 +- src/backend/catalog/system_views.sql | 14 + src/backend/storage/buffer/bufmgr.c | 4 + src/backend/storage/freespace/freespace.c | 3 +- src/backend/storage/ipc/ipci.c | 3 + src/backend/storage/smgr/md.c | 6 +- src/backend/utils/adt/pgstatfuncs.c | 5 +- src/backend/utils/misc/guc.c | 55 +- src/backend/utils/misc/postgresql.conf.sample | 6 + src/include/access/xlog.h | 1 + src/include/access/xlogprefetcher.h | 53 ++ src/include/access/xlogreader.h | 8 + src/include/access/xlogutils.h | 3 +- src/include/catalog/catversion.h | 2 +- src/include/catalog/pg_proc.dat | 7 + src/include/utils/guc.h | 4 + src/include/utils/guc_tables.h | 1 + src/test/regress/expected/rules.out | 11 + src/tools/pgindent/typedefs.list | 6 + 27 files changed, 1595 insertions(+), 77 deletions(-)
Hi, On 2022-04-07 07:44:20 +0000, Thomas Munro wrote: > Prefetch data referenced by the WAL, take II. https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&dt=2022-04-07%2008%3A17%3A27 thinks that xlogpretcher.h doesn't include enough... I had added those checks to CI, but apparently somehow screwed that up. Or maybe it's the scripts that are screwed up? Because I don't see any error checking in headerscheck/cpluspluscheck. And indeed, locally they show the errors, but exit with 0. Greetings, Andres Freund
On 4/7/22 04:36, Andres Freund wrote: > Hi, > > On 2022-04-07 07:44:20 +0000, Thomas Munro wrote: >> Prefetch data referenced by the WAL, take II. > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&dt=2022-04-07%2008%3A17%3A27 > thinks that xlogpretcher.h doesn't include enough... > > I had added those checks to CI, but apparently somehow screwed that up. > > Or maybe it's the scripts that are screwed up? Because I don't see any error > checking in headerscheck/cpluspluscheck. And indeed, locally they show the > errors, but exit with 0. Yeah, you can't rely on the exit status, if they produce output that should be regarded as a failure (that's what crake does). cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
Andrew Dunstan <andrew@dunslane.net> writes:
> On 4/7/22 04:36, Andres Freund wrote:
>> Or maybe it's the scripts that are screwed up? Because I don't see any error
>> checking in headerscheck/cpluspluscheck. And indeed, locally they show the
>> errors, but exit with 0.
> Yeah, you can't rely on the exit status, if they produce output that
> should be regarded as a failure (that's what crake does).
Yeah, those scripts were only intended to be run by hand. If someone
feels like upgrading them to also produce a useful exit status,
I'm for it.
regards, tom lane