Re: [PATCH] Batched clock sweep to reduce cross-socket atomic contention - Mailing list pgsql-hackers

On Fri, Jul 10, 2026, at 10:00 AM, Greg Burd wrote:
> Hello again,
>
> Removing the freelist [1] turned out to be a good idea, so why not remove
> and simplify more?
>
> Included is a three-patch series (v3, on 2e6578292a9 cf-7000) that
> replaces the 0..5 usage_count clock sweep with a cooling-stage clock,
> and because this evictor algorithm is scan resistance 0003 removes
> the BufferAccessStrategy ring buffers entirely.  The net is slightly
> faster (TPS), hotter buffers (fewer misses) and a lot less code to
> maintain (no special case for scan resistant work) going forward.
>
>   0001  Batch the clock sweep to reduce nextVictimBuffer atomic contention
>   0002  Replace the usage_count clock sweep with a cooling-stage evictor
>   0003  Remove BufferAccessStrategy; scan resistance is now intrinsic
>
> Net it is +510/-1642 over 83 files; almost all of the deletion is 0003.
>
> On a 6-NUMA-node r8i.metal-96xl under continuous eviction, this change is
> a consistent +4–5% on read-heavy pgbench with equal-or-better hit ratios.
> The same test on a 2-node box is flat, which is the tell that the win
> is the cross-socket clock-hand contention the batched sweep removes. Under
> real storage IO (working set > RAM on local NVMe), the scan-resistance
> path is +1.6–3.9% with measurably fewer heap reads.
>
> I'll put the design decisions and the benchmark methodology on the table
> first, then talk through the results because if the model is wrong on
> principle, or the benchmarks are measuring the wrong thing, I would much
> rather hear it now.
>
> Contention arises on high core-count systems from the of cost advancing
> nextVictimBuffer in StrategyGetBuffer() via a fetch-add of 1 per tick.
> This contention is exacerbated on a multi-socket box.  That one cache
> line bounces over the interconnect on every eviction; under pressure
> with hundreds of backends it is the dominant cost of the sweep.  This is
> why we're investing time on NUMA optimizations.
>
>
> == What each patch does, and why ==
>
> 0001 -- Batch the clock sweep.
>
>   This is the setup/baseline commit and is an incremental improvement
>   over the existing batched approach, but it is not the final goal and
>   should not be committed without 0002.
>
>   Each backend will claim a run of consecutive buffer IDs with a single
>   fetch-add and iterate them privately.  Global sweep order is preserved
>   -- each buffer is visited exactly once per pass -- only the intra-pass
>   ordering of visits changes, which the algorithm does not depend on.
>   The atomic fires ~1/batch as often.
>
>   The batch is one cache line's worth of hand values (PG_CACHE_LINE_SIZE
>   / sizeof(uint32)), capped at NBuffers.  It is gated on multi-node NUMA
>   (pg_numa_get_max_node() >= 1); on a single socket the batch is 1 and
>   the path is byte-identical to today.  This is essentially Jim's patch;
>   the only substantive change from his v1/v2 is deriving the batch size
>   from the cache-line size rather than a fixed 64/tiered constant.
>
> 0002 -- Replace usage_count with a cooling-stage evictor.
>
>   This is the heart of the series and where I most want the design
>   attacked.
>
>   A buffer is HOT (recently used) or COOL (an eviction candidate);
>   "pinned" is the existing refcount.  There is no per-buffer 0..5
>   counter.  A demand-loaded page is admitted COOL (probationary); a
>   second access promotes it COOL->HOT.  So a page touched once -- as is
>   the case with a sequential scan -- fills and drains the COOL stage and
>   is evicted from it without displacing the HOT working set.  This is
>   the LeanStore / 2Q-A1 idea, and it is the whole reason 0003 can exist:
>   scan resistance stops being a ring-buffer bolt-on and becomes a
>   property of the replacement algorithm.
>
>   The 4-bit usage_count field is reinterpreted in place -- bit 0
>   HOT/COOL, bit 1 a reference bit, the top 2 bits unused -- so the 64-bit
>   buffer-state layout, its refcount / flag / lock offsets.  The only
>   change to the StaticAsserts is a new one asserting the field is at
>   least 2 bits wide, since we now use two of its bits.  The eviction
>   claim (COOL, unpinned -> pinned) stays a CAS so a racing PinBuffer
>   always wins; promotion and demotion are single-bit transitions.
>
>   The one non-obvious decision, which I got wrong first and want to flag
>   loudly: *Who* demotes HOT->COOL.  My first cut was "prefer-COOL": the
>   foreground sweep skips HOT buffers hunting for an already-COOL victim
>   and only cools HOT buffers once a full pass finds none.  That
>   collapses under an ordinary OLTP workload where the working set is
>   larger than shared_buffers (no scan at all).  Every access promotes
>   its buffer to HOT, so there are almost no COOL buffers, the "cool a
>   full pass" fallback fires on nearly every allocation, and each victim
>   search becomes a ~2x full-pool scan of scattered BufferDescs.  I
>   measured 3-17x throughput loss vs stock -- a cliff.
>
>   HOT->COOL demotion is done during the background writer's existing LRU
>   scan, which already runs ahead of the clock hand.  It demotes just
>   enough HOT buffers to keep a supply of COOL victims (bounded by the
>   predicted next-cycle allocation, so it does not cool the whole pool),
>   under the buffer header lock it already holds.  A single reference bit
>   gives a recently-accessed buffer one reprieve before it is cooled,
>   which keeps the genuinely-hot set out of the COOL stage under scan
>   pressure.  With that, the foreground finds a victim in a single pass
>   and the cliff is gone (data below).
>
>   I chose prefer-COOL-plus-bgwriter-precooling because it can protect a
>   hot buffer slightly longer (the pre-cooler, tuned to a budget, decides
>   when to demote rather than demoting on contact), which should help hit
>   ratio on a stable hot set -- IF the pre-cooler keeps up.  When it lags
>   (bgwriter off or behind), the foreground force_cool is still there as
>   a correctness fallback, but it is the expensive path.
>
> 0003 -- Remove BufferAccessStrategy.
>
>   With scan resistance intrinsic, the BAS_BULKREAD/BULKWRITE/VACUUM
>   rings are dead weight, so this removes them end to end: the type and
>   enum, the ring machinery in freelist.c, the strategy parameter
>   threaded through ReadBufferExtended / the ExtendBufferedRel* family /
>   read_stream / every scan/vacuum/analyze/index-AM caller, the strategy
>   fields on the scan and bulk-insert descriptors, and
>   _hash_getbuf_with_strategy.  pg_stat_io's per-strategy IO contexts
>   collapse to normal/init (IOOP_REUSE only ever happened while recycling
>   a ring buffer, so it is gone), and the vacuum_buffer_usage_limit GUC /
>   VACUUM (BUFFER_USAGE_LIMIT ...) option / vacuumdb --buffer-usage-limit
>   go with it.
>
>   This is the patch most likely to be contentious for reasons unrelated
>   to the sweep: it removes a user-visible GUC and changes pg_stat_io's
>   shape.  I have kept it as its own commit precisely so 0001+0002 can be
>   judged on the algorithm without swallowing the removal.  If the
>   consensus is that the cooling model is fine but the ring machinery
>   should stay for other reasons (BUFFER_USAGE_LIMIT as an operator
>   control, say), 0003 can simply be dropped and 0002 still stands -- the
>   rings just become redundant rather than removed.  I would like to know
>   if that is where people land.
>
>
> == Benchmarks ==
>
> I want to be careful here, because Andres's central criticism of the
> original batched-sweep numbers [2] was that they measured the wrong
> thing -- an all-in-page-cache config where the only bottleneck left is
> the atomic, which is not how anyone runs a 384-vCPU box.  That criticism
> is correct and I have tried to design around it, but I have almost
> certainly not fully escaped it, so the methodology is laid out below in
> enough detail to shoot at.
>
> I refer to the COOL/HOT approach (the changes in this patch set) as
> "bcs", I've forgotten what that stands for... "buffer cache solution?"
> I really don't remember (ha!).
>
> Hardware / method (both instances bare-metal, Amazon Linux 2023):
>
> - m6i.metal      -- 128 vCPU, 2 sockets, 2 NUMA nodes (distances 10/20), 503GB
> - r8i.metal-96xl -- 384 vCPU, 2 sockets, 6 NUMA nodes via SNC3, 3TB
>
> Builds: --buildtype=debugoptimized (-O2), cassert off, --with-libnuma,
> both branches from the same tree.  Postmaster pinned with numactl
> --cpunodebind=0 --membind=0 (without it stock TPS varied ~30% by launch
> node -- worth flagging for anyone reproducing).  pgbench dataset loaded
> once per build into its own datadir.
>
> The regime I settled on: a dataset that fits fully in OS page cache but
> exceeds shared_buffers, warmed before each cell, caches NOT dropped
> between cells.  The point is a sweep that runs continuously with
> sub-millisecond read latency and NO storage IO in the critical path --
> so what I am measuring is the eviction machinery itself, not the disk
> (as best as I can tell).
>
> I sweep the ratio (working-set / shared_buffers) from 0.8 (fits, no
> eviction, a control) up to 8x by varying shared_buffers against a fixed
> 63GB dataset.
>
> 3 iterations per cell, medians reported, 256 clients on m6i / 384 on r8i.
>
> Two workloads: uniform pgbench -S (the pure eviction-churn case, and the
> worst case for the cooling model -- no hot set to protect), and
> "hotscan" (a Zipfian hot set plus a handful of clients running large
> range scans -- the scan-resistance case).
>
> I want the regime itself critiqued.  It deliberately removes storage IO
> to expose the sweep; the flip side is it is not a production
> configuration, and the read-path win (fewer misses under scan
> resistance) shows up as read count, not TPS, because a "miss" here is an
> OS-cache memcpy, not a device read.
>
> r8i.metal-96xl, uniform pgbench -S, 384 clients, TPS (median of 3):
>
>   ratio  SB(GB)   stock       bcs        delta   stock/bcs hit%
>   0.8    ~400     1,623,536   1,692,090   +4.2%   99.2 / 99.9
>   1.25   ~320     1,443,343   1,519,290   +5.3%   94.8 / 95.2
>   1.5    ~266     1,344,051   1,417,462   +5.5%   91.9 / 92.1
>   2      ~200     1,293,871   1,358,550   +5.0%   87.9 / 87.8
>   4      ~100     1,194,405   1,240,334   +3.8%   77.1 / 78.5
>   8      ~50      1,090,873   1,140,839   +4.6%   69.7 / 70.6
>
> Consistent +4-5% across the eviction range, growing with pressure, with
> equal-or-better hit ratio (better at 4x/8x).  bcs also showed lower
> cache-miss rate in perf stat (~28-35% vs ~31-37%), which is the batched
> sweep's reduced cross-node line bouncing showing through.
>
> r8i, hotscan (Zipfian hot set + range scanners), TPS / hit% / heap reads:
>
>   ratio  stock TPS   bcs TPS    stock hit  bcs hit   stock reads  bcs reads
>   1.25   1,833,675   1,871,289   99.43     99.57     7,309,457    5,503,927
>   1.5    1,876,846   1,942,877   99.35     99.42     8,335,144    7,565,667
>   2      1,868,591   1,853,547   99.12     99.15    11,173,831   10,987,371
>
> The scan-resistance signal: at 1.25-1.5x, bcs holds a higher hit ratio
> and does up to 25% fewer heap reads (24.7% at 1.25x, ~9% at 1.5x) -- it
> is keeping the hot set resident through the scans where stock lets them
> flush it.  Muted in TPS only because everything is in OS cache (a miss
> is cheap); on real storage this read reduction is where the win would
> land.  At 2x it washes out (enough pressure that both evict heavily).
>
> m6i.metal (2 nodes), uniform, 256 clients -- the smaller box, for contrast:
> essentially parity, bcs -2% to +1% across ratios.  The 2-node box barely
> exercises the atomic, so 0001's contention win does not appear; that it does
> not regress is the result that matters here.
>
> Huge pages on vs off (r8i, uniform, medians): I ran this because the
> original thread flagged it as uncharacterized.  bcs won by +3-7% both
> ways, no regression without huge pages -- the win is from cutting the
> frequency of atomic ops on the counter line, which does not depend on
> where the descriptors physically live.  (This is why the batching gate
> is NUMA-only and not also huge-pages-gated.)
>
> The "prefer-COOL cliff" I mentioned under 0002, so the failure mode is
> on the record: BEFORE moving cooling into the bgwriter, the m6i uniform
> run at 256 clients was bcs 274K vs stock 840K at ratio 2 (-67%), and
> 42.8K vs 762K at 8x (-94%), with cache-miss rate exploding to ~40%.
> That is the shape of getting the demotion policy wrong; the r8i +5%
> table above is after the fix.
>
> Reproduction: the whole harness (instance launch, OS tuning, per-build
> load, the ratio sweep, perf stat capture) is scripted; I will attach it
> as a DO-NOT-MERGE commit / put it in the CF entry so the methodology can
> be reproduced and picked apart rather than taken on faith.  Raw per-run
> CSVs and perf output likewise.
>
>
> A Real IO (working set > RAM, evictions hitting storage) Benchmark
>
> This is the regime Andres asked for, and the one I flagged earlier as not yet
> done cleanly.  The earlier attempt was EBS-latency-bound; this one uses local
> NVMe so eviction reads hit real storage at ~microsecond, not ~15ms, latency --
> during the run the array sat at 100% utilization and ~145K read IOPS, so the
> eviction path is genuinely storage-bound, not cache-served.
>
> m6id.metal -- 128 vCPU, 2 sockets, 2 NUMA nodes, 503GB, 4x1.9TB local NVMe in
> RAID0.  Dataset ~700GB (pgbench scale 47000), i.e. LARGER than RAM, so the
> working set cannot sit in the OS page cache.  shared_buffers is a small window
> over it -- 63GB (11x) and 31GB (22x) -- caches dropped per cell, 256 clients, 3
> iterations, medians.  Same builds/method as the in-cache runs otherwise.
>
> hotscan (Zipfian hot set + range scanners), median of 3:
>
>   ratio  SB(GB)   stock TPS   bcs TPS     dTPS    stock/bcs hit   reads d
>   11     63       877,357     911,363    +3.9%    96.30 / 96.43   -2.2%
>   22     31       873,880     888,010    +1.6%    94.02 / 94.22   -1.5%
>
> This is the result the in-cache runs could only hint at: under real storage IO
> the scan-resistance read reduction converts to throughput.  The bcs approach
> keeps a higher hit ratio and does 1.5-2.2% fewer heap reads, and here -- unlike
> in cache, where a miss is a cheap memcpy -- a read it avoids is an NVMe round
> trip, so the read reduction shows up as +1.6-3.9% TPS.
>
> uniform pgbench -S (pure eviction churn, no hot set to protect), median of 3:
>
>   ratio  SB(GB)   stock TPS   bcs TPS     dTPS    stock/bcs hit
>   11     63       605,656     616,247    +1.7%    67.0 / 67.3
>   22     31       605,517     611,312    +1.0%    63.5 / 63.6
>
> Hit ratio here is 63-67% -- a third of accesses miss and hit NVMe (190M-220M
> evictions per run), so this is deep, genuinely storage-bound churn.  bcs is
> +1-1.7%, i.e. flat-to-slightly-positive, which is the honest production picture:
> with no hot set to protect, scan resistance has nothing to do, and the win is
> just the batched sweep's reduced contention showing faintly through the IO wait.
> Notably bcs does not regress even when its policy has no advantage to exploit.
>
>
> == Side note for the curious... ==
>
> Separately, Dhruv Aron has proposed restructuring the shared-buffer lookup table
> [2], replacing dynahash with a flat two-array structure. That attacks the
> other hot cost on a buffer miss — resolving a page to its buffer — where this
> series attacks the eviction that a miss triggers. They touch buf_table.c and a
> lock-ordering change in InvalidateBuffer(); this series touches freelist.c and
> the per-buffer replacement state, and removes BufferAccessStrategy. The two are
> complementary and should compound on the miss path; the only overlap is
> InvalidateBuffer()/GetVictimBuffer(), where their extended buffer-header-lock
> hold and this series' CAS-claim + bgwriter pre-cooling both take that lock, and
> would want reconciling if both land. I have not benchmarked them together (yet).
>
> == What I have not done, honestly ==
>
>   - Hardening the foreground force_cool fallback to be cheap when it
>     fires, rather than relying on the bgwriter pre-cooler never lagging.
>
>   - Anything on single-socket beyond "does not regress"; the design is
>     not trying to help there.
>
>
> == The ask ==
>
>   1. 0002's demotion policy: is prefer-COOL + bgwriter pre-cooling the
>      right call, or is the other team's cool-in-place the more robust
>      default given it has no cliff and no background-process dependency?
>      This is the decision everything else hangs on.
>
>   2. Is admitting demand-loaded pages COOL (probationary,
>      promote-on-second- touch) an acceptable basis for scan resistance
>      in the core buffer manager, i.e. is it OK to make scan resistance
>      an algorithm property and retire the strategy rings (0003)?  Or
>      should the rings stay and 0002 ride alongside them?
>
>   3. The benchmark methodology: where is the in-OS-cache regime
>      misleading, and what would you want measured instead?  I am most
>      worried I am flattering the sweep by removing the IO that would
>      otherwise hide it.
>
>   4. Reinterpreting the usage_count field as {HOT/COOL, ref} bits and
>      collapsing pg_stat_io's contexts -- acceptable, or is there a
>      cleaner representation the project would want before this is worth
>      pursuing?
>
> I have measured that the 0..5 count is overhead and provides no
> meaningful signal at all, that a HOT/COLD approach provides a simpler
> more stable and better performing eviction model for the buffer pool.
> If you dispute that, let's dig in and compare notes. :)
>
> I would be remiss if I didn't point out the thread [3] by Tomas et. al.,
> whose NUMA investigation targets the same bottlenecks, and inspired the
> work that led to this set of ideas.
>
> Thanks for reading this far.  I look forward to the critique.
>
> best.
>
> -greg
>
> [1] Reconsidering the freelist
>
> https://www.postgresql.org/message-id/f0e3c02e-e217-4f04-8dab-1e7e80a228c0%40burd.me
> [2] Re: Restructured Shared Buffer Hash Table
>
> https://www.postgresql.org/message-id/dbbd1998-19ff-4ac2-b4b1-a39f4ec1b0f5@iki.fi
> [3] Adding basic NUMA awareness (Tomas Vondra)
>
> https://www.postgresql.org/message-id/099b9433-2855-4f1b-b421-d078a5d82017%40vondra.me
> Attachments:
> * v3-0001-Batch-the-clock-sweep-to-reduce-nextVictimBuffer-.patch
> * v3-0002-Replace-the-usage_count-clock-sweep-with-a-coolin.patch
> * v3-0003-Remove-BufferAccessStrategy-scan-resistance-is-no.patch

Rebased v4 onto c71d43025d7.

-greg