== HOT with Selective Index Updates ==

=== Request for Comments [RFC] NOT MERGED ===

HOT is one of the most important performance optimizations in PostgreSQL
history, but it has a significant limitation: it applies only when no indexed
columns are modified. This blocks optimization on wide tables where only a few
columns are indexed. Various efforts have tried to lift this limitation,
including selective index updates proposals, deferred index maintenance, and
partial HOT chains. However, none have been committed to core PostgreSQL.

The flaw in those previous efforts falls into two categories:

* '''Design complexity:''' Some proposals required undo-chain navigation
* (difficult without a full UNDO infrastructure). Others used background
* workers to defer index updates (but queries see stale indexes during
* maintenance windows). Still others tried to track per-column index
* eligibility at the tuple level (too much memory overhead).

* '''Correctness gaps:''' Previous attempts didn't adequately handle concurrent
* prune/vacuum scenarios, leading to orphaned references or index corruption
* under load. They also struggled with replication safety—subscriber indexes
* can diverge from publisher indexes, and prior proposals didn't address this
* systematically.

This proposal, HOT-indexed updates, differs in three ways:

1. '''Attribute-bitmap staleness, not a value recheck.''' A HOT-indexed update
stays on the HOT chain and maintains only the indexes whose attributes changed.
Each new tuple records, inline in its tail, which indexed attributes changed at
that hop; a reader unions the bitmaps of the hops it crosses and drops an entry
whose index columns overlap that union.  This is access-method agnostic and,
crucially, correct under a value cycled away and back (ABA) &mdash; the case
that sank WARM's value recheck.

2. '''Collapse to xid-free stubs, no "convert back" step.''' Prune rewrites a
dead chain prefix into xid-free forwarding stubs that preserve each surviving
hop's bitmap; VACUUM later reclaims the stubs and re-points the root redirect,
collapsing back to classic HOT.  There is no persistent per-tuple state to
reconcile.

3. '''Replication safety:''' a per-subscription option
<code>hot_indexed_on_apply</code> (off / subset_only / always) gates the apply
path, so a HOT-indexed update of a replica-identity attribute leaves a stale
leaf only when the apply worker's RI lookup can tolerate it.

---

== Measured performance (indicative) ==

A/B run of two release (<code>cassert=off</code>) builds &mdash;
<code>origin/master</code> vs the SIU series &mdash; on a single Apple Silicon
laptop (macOS), pgbench, scale 5 (<code>siu_table</code> = 500k rows with 3
secondary indexes; <code>wide_table</code> = 5k rows with 16 secondary indexes
+ PK), 8 clients / 4 threads, 20 s per cell.  pgbench runs for a fixed time, so
each variant completes a different number of updates; the write-amplification
signal is therefore reported as WAL bytes per update.

{| class="wikitable"
|-
! Workload (indexed cols changed) !! TPS master&rarr;tepid !! WAL/update master&rarr;tepid
|-
| simple_update (control; HOT both) || 32.6k &rarr; 32.1k (0%) || 265 &rarr; 265 B (0%)
|-
| hot_indexed_update (1 of 4) || 58.2k &rarr; 68.6k (+18%) || 636 &rarr; 487 B (&minus;23%)
|-
| wide, 1 of 17 indexes || 33.6k &rarr; 41.7k (+24%) || 1466 &rarr; 598 B (&minus;59%)
|-
| wide, 8 of 17 indexes || 37.4k &rarr; 47.3k (+26%) || 1498 &rarr; 1015 B (&minus;32%)
|-
| wide, 16 of 17 indexes || 36.3k &rarr; 37.3k (+3%) || 1530 &rarr; 1490 B (&minus;3%)
|-
| read_indexscan (read-only) || 164.4k &rarr; 161.3k (&minus;2%) || n/a (no writes)
|}

Reading the table:

* The win scales with how many indexes are skipped.  Changing one indexed
  column on a 17-index table cuts WAL per update by ~59% and lifts throughput
  ~24%; changing all but one (16 of 17) leaves almost nothing to skip, so SIU
  converges to master (within noise).

* <code>simple_update</code> changes no indexed column (HOT on both variants);
  it is a control, and the identical WAL/update and throughput confirm SIU adds
  no overhead to the existing HOT path.

* <code>read_indexscan</code> is read-only on a freshly reset table with no
  stale entries.  master and tepid are at parity (&minus;2%, within run-to-run
  noise): the crossed-attribute read path adds no per-scan key comparison, and
  &mdash; after the read path no longer materializes the leaf IndexTuple
  &mdash; no per-scan itup cost either.

'''Caveats:''' a single short run on a laptop, so absolute TPS and post-run
index sizes are noise- and autovacuum-sensitive.  WAL-per-update and the read
parity are the robust signals; the directional throughput gains are consistent
with them.  A multi-run sweep on dedicated hardware (the harness in
<code>src/test/benchmarks/siu</code> supports it) remains future work.

These numbers predate the v70 change that replaced the per-entry bit-14 marker
with the per-block <code>VISIBILITYMAP_LOCATOR_SPLIT</code> signal.  That change
adds visibility-map work to the HOT-indexed write path (a VM pin, and the VM
page lock while the bit is set), so the write-side figures need re-measuring
before they are quoted for v70.

---

== The Problem ==

Heap bloat and index bloat due to the MVCC model in heap pushes the cost of
in-place updates into the VACUUM (or pruning) process.  Avoiding as much of that
bloat as possible saves space and I/O.

PostgreSQL's heap-only tuple (HOT) optimization works when a new version of a
row has room to co-exist on the same page as the old version.  When that is the
case then you can avoid updating indexes as long as you've not modified any
indexed attributues.  Change one and you're forced to update all indexes - even
those that don't reference any changed attributes.  A small optimization was
made to address summarizing indexes, which always need updates, in cases where
no non-summarizing indexes were updated.  As it is before this patch, HOT can
avoide index updates when only non-key columns change. This is fantastic, but it
still updates every index, even if the index doesn't care about the columns that
changed.

Consider a table with fifty columns and twenty indexes (one on
<code>supplier_id</code>, another on <code>price</code>, and then 18 more). An
UPDATE that changes only <code>price</code> but not <code>supplier_id</code>
(or other indexed columns) still updates 20 indexes, not one. For wide tables
with many indexes, this creates unnecessary write amplification (aka bloat) in
the index that must be cleaned up by a VACUUM and reconciled during index
scans.

The solution is simple: if an index doesn't reference a column, then the index
doesn't care whether that column changed. We should be able to skip updating
such indexes... in theory.

---

== How This Works ==

=== Step 1: Identify What Indexed Attributes Changed ===

When we UPDATE a row, the executor (<code>ExecUpdateModifiedIdxAttrs</code>,
which replaces heapam's old <code>HeapDetermineColumnsInfo</code>) compares the
old and new tuples over the relation's indexed-attribute set and builds a
bitmap of the indexed attributes whose values actually changed
(<code>modified_idx_attrs</code>):
* changed an indexed attribute &rarr; set its bit
* otherwise &rarr; leave it clear

The executor itself stays access-method agnostic: it hands the candidate
indexed-attribute set and the old/new slots to a table-AM callback through the
<code>table_modified_attrs()</code> wrapper, which returns the subset that
actually changed.  The heap AM implements this as
<code>heapam_modified_attrs</code>, which owns the notion of attribute equality
(comparing slot values with <code>datum_image_eq</code>) and the heap-specific
system-column semantics.  The callback is optional: an AM that leaves it NULL
is treated as though every candidate attribute changed (the conservative
"maintain all indexes" answer &mdash; the wrapper returns the input set
unchanged).

<code>HeapUpdateHotAllowable</code> then classifies the update from that bitmap:

* '''HEAP_UPDATE_ALL_INDEXES''' &mdash; not HOT (e.g. all indexed attributes
  changed, an expression-index input changed, a system catalog, or the
  per-subscription apply gate); a new tuple and an entry in every index.

* '''HEAP_HEAP_ONLY_UPDATE''' &mdash; no non-summarizing indexed attribute
  changed (only summarizing ones, if any); classic HOT.

* '''HEAP_SELECTIVE_INDEX_UPDATE''' &mdash; some, but not all, indexed
  attributes changed; stay on the HOT chain and maintain only the changed
  indexes.

The table AM reports back to the executor, through
<code>table_tuple_update</code>'s <code>row_moved</code> output flag, whether
it stored the new tuple such that existing index entries no longer locate it
(for heap, a non-HOT update at a new TID).  The executor derives
index-maintenance policy from that fact together with the changed-attribute
bitmap: when <code>row_moved</code> is true every index needs a fresh entry;
otherwise only the indexes whose attributes overlap the bitmap do.  This keeps
the boundary AM-neutral&mdash;the AM states a fact ("the row moved") rather
than a heap-specific instruction, and the executor owns the which-indexes
decision.

A simple table with two indexes:
<pre>
Table: users(id, email, bio, status)
Indexes: idx_email(email), idx_status(status)
</pre>

And then an UPDATE where an one of the two indexed columns is mutated.
<pre>
UPDATE users SET status = 'active' WHERE id = 1;
</pre>

We note that a subset of indexes overlap with the changed attributes, so we can
selectively update those.
<pre>
  changed indexed attrs = {status}     (email unchanged)
  not empty, not "all indexed"         => HEAP_SELECTIVE_INDEX_UPDATE
  maintain idx_status (insert a fresh entry); skip idx_email
</pre>

=== Step 2: Mark the New Tuple and Record Which Attributes Changed ===

The new version is a heap-only tuple linked into the chain via
<code>t_ctid</code>, exactly like classic HOT, plus the
<code>HEAP_INDEXED_UPDATED</code> bit (<code>t_infomask2</code> 0x0800) and,
appended after its attribute data, a fixed-size bitmap of the attributes that
changed at this hop.  The executor inserts a fresh index entry only into the
changed indexes, and '''that fresh entry points at the new heap-only tuple's
own TID''', not at the chain root. Unchanged indexes are not touched: their
existing entries still resolve through the chain.

There is no "tombstone" line pointer.  The bitmap is inline in the data-bearing
tuple; its length is ceil(natts/8) bytes, sized by the tuple's own attribute
count at write time, so it survives ADD COLUMN (see [[#Bitmap sizing across
DDL]]).

<pre>
INSERT (1, 'a@x', 'active'):
  LP[1] = T1(email='a@x', status='active')   root; idx_email,idx_status -> LP[1]

UPDATE status='paused' (HOT-indexed, change {status}):
  LP[1] = T1   root, HEAP_HOT_UPDATED, t_ctid -> 2
  LP[2] = T2(email='a@x', status='paused')   heap-only, INDEXED_UPDATED{status}
  idx_status gains a fresh entry ('paused') -> LP[2]
  idx_email is NOT touched: its entry ('a@x') -> LP[1] still resolves the chain
</pre>

=== Step 3: Reads Drop Stale Entries by the Crossed-Attribute Bitmap ===

A pre-update index entry can now be ''stale'': it chain-leads to a live tuple
whose current key differs.  The read side detects this without any value
comparison.  <code>heap_hot_search_buffer</code> walks the chain to the live
tuple and unions the per-hop bitmaps of every hop crossed ''after'' the
arriving entry's own tuple (the entry's own producing hop does not count
&mdash; a fresh entry is never stale for its own index).  The index-access
layer tests that union against the arriving index's key columns: overlap &rArr;
stale, drop; disjoint &rArr; current, return.  The row a stale entry would have
surfaced is re-supplied by the fresh entry the same update planted.

<pre>
Scan status='paused' via idx_status -> LP[2]:
  arrive AT LP[2] (own hop {status} not counted); no later hop crossed
  crossed = {}    => current => return T2.                              OK
Scan status='active' via idx_status -> LP[1] (stale):
  cross ->2 {status};  crossed = {status};  {status} & {status} = {status}
                  => stale => drop (T2 is supplied by the 'paused' entry).  OK
</pre>

This is access-method agnostic: it never reconstructs or compares an index key,
so btree, hash, GiST, GIN and SP-GiST all work, and a scan never has to
materialize the leaf IndexTuple for staleness purposes.

=== Index-only scans: the all-visible page problem and the per-entry staleness check ===

Index-only scans (IOS) need two things from SIU that an ordinary index scan
does not, because an IOS deliberately tries to answer a query ''without''
touching the heap &mdash; it serves the result columns out of the index tuple
(<code>xs_itup</code>) and skips the heap fetch whenever the target page is
marked all-visible in the visibility map (VM).  Both of those shortcuts are
unsafe over a HOT-indexed chain unless we account for them.

'''(1) All-visible pages, or the lack of them over redirect/stub chains.'''
A stale leaf points (via the chain) at a live tuple whose current key differs
from the leaf's stored key.  If the page holding that chain were marked
all-visible, an IOS would take the VM fast path, skip the heap fetch, and
return the leaf's '''stale''' key as the answer &mdash; wrong results, with no
opportunity to detect the problem.  SIU prevents this from the prune side: a
page that carries anything a stale leaf can still resolve through &mdash; a
preserved live <code>HEAP_INDEXED_UPDATED</code> member, an
<code>LP_REDIRECT</code> that forwards into one, or a collapse-survivor stub
&mdash; is '''deliberately kept out of the visibility map''' (prune forces
<code>set_all_visible = false</code> for it; see
<code>heap_prune_record_redirect</code> and the stub recorders, with the same
guard re-applied in <code>heap_page_would_be_all_visible</code>).  So a page
that could surface a stale entry is never all-visible, and an IOS over it is
forced down the heap-fetch path below, where staleness can be detected.
Conversely, a page that genuinely is all-visible cannot hold a live SIU chain,
so the VM fast path stays correct and needs no extra check.

'''(2) Re-checking the entry against the live tuple during the scan.'''
Once forced to fetch the heap, the IOS still intends to return values from
<code>xs_itup</code>, not from the heap tuple it just read.  That is exactly
where a stale leaf would do damage: <code>xs_itup</code> holds the ''old'' key,
so returning it would surface a value the live row no longer has.  The scan
therefore re-checks, per entry, whether the leaf it arrived through is stale
for this index before trusting <code>xs_itup</code>.  For the read path that
check is the crossed-attribute bitmap test, not a literal key comparison: the
chain walk has accumulated the union of the modified-attrs bitmaps it crossed,
and the index-access layer sets <code>xs_entry_needs_recheck</code> iff that union
overlaps this index's columns.  If stale, IOS drops the entry
(<code>ExecClearTuple</code> + continue); the fresh entry the same update
planted returns the row with correct values via its own path.

<pre>
IOS for an indexed column, entry arrived via a stale leaf:
  VM_ALL_VISIBLE(page)? -> no (prune kept the SIU page out of the VM)
  index_fetch_heap()    -> walks the chain, accumulates crossed = {changed attrs}
  xs_entry_needs_recheck  -> crossed overlaps this index's columns => true
  => drop the entry (do NOT return xs_itup's stale key); the fresh entry serves it.
</pre>

Why a bitmap test and not an actual key comparison here: the read path is
access-method agnostic and must not depend on reconstructing or comparing keys
(that would defeat the whole design and would not work uniformly across AMs).
The one place SIU ''does'' compare keys is the unique-insert check
(<code>_bt_check_unique</code>), where a missed conflict is corruption and the
bitmap's lossy "something changed" verdict is not enough (see Appendix A); that
is a write-side correctness gate, not the IOS read path.

=== Bitmap scans: the mid-chain TID that BitmapAnd would drop ===

The per-entry staleness test above runs on the way to the heap.  Bitmap scans
have an earlier hazard that the crossed-attribute test cannot reach: a
<code>BitmapAnd</code> intersects two indexes' TID sets in the TID-bitmap layer
(<code>tidbitmap.c</code>) at raw block+offset granularity, ''before'' either
side ever touches the heap.

A HOT-indexed fresh entry points at the new heap-only tuple, while an unchanged
index's entry for the same row still points at the chain root.  For a query
with a predicate on each &mdash; <code>WHERE changed_col = x AND unchanged_col
= y</code> &mdash; the two bitmap index scans therefore contribute
''different'' offsets (mid-chain member vs. root) for the one logical row, the
exact-mode intersection finds nothing in common, and the row is dropped.  A
false negative: bitmap scans tolerate false positives but not false negatives.
(Reported by Alexander Korotkov; it is the same weakness that sank WARM.)

Two facts shape the fix.  The disagreement never crosses a heap block, because
a HOT or HOT-indexed chain never leaves its page, so both entries always name
the same block.  And only <code>BitmapAnd</code> is affected:
<code>BitmapOr</code> unions, keeping both offsets, and
<code>BitmapHeapScan</code> resolves each through the chain and de-duplicates
them.  So what <code>BitmapAnd</code> needs to know is not "which entry is
inexact" but "on which block might two entries for one row name different
offsets".  That is a property of the heap block, not of any index entry.

The heap records it as a per-block visibility-map bit,
<code>VISIBILITYMAP_LOCATOR_SPLIT</code>.  The visibility map widens from 2 to
4 bits per heap block to carry it (3 would do, but integer division packs 3 and
4 bits both at two blocks per byte), and <code>pg_upgrade</code> rewrites each
<code>_vm</code> fork from the old layout, as it did for 9.6's frozen bit.  The
bit is not part of visibility and is excluded from
<code>VISIBILITYMAP_VALID_BITS</code>.

* '''Set''': <code>heap_update</code> sets it on the block when it performs a
  HOT-indexed update that plants a fresh index entry, inside the same critical
  section, and logs it with a flag on the existing update WAL record
  (<code>XLH_UPDATE_NEW_LOCATOR_SPLIT</code>); redo sets it too.  A promoted
  update that changes no indexed column plants no entry and does not set it.
* '''Clear''': prune clears it when it sets the page all-visible.  The
  carve-outs that already keep a page with a redirect-to-HOT-indexed member or
  a stub out of the visibility map guarantee no disagreeing entry remains once
  the page is all-visible, so the bit is stale by then.  The clear rides the
  existing prune/freeze WAL record.
* '''Read''': the table AM exposes the bit through the locator descriptor's
  <code>bitmap_and_inexact</code> hook.  The executor records the relation on
  the <code>BitmapAnd</code> accumulator (<code>tbm_set_relation</code>), and
  <code>tbm_intersect_page</code> asks the hook once per block, lock-free, the
  way index-only scans read <code>ALL_VISIBLE</code>.  For a flagged block it
  unions the two sides' offsets and forces a recheck instead of intersecting;
  <code>BitmapHeapScan</code> then resolves the chain and the per-entry
  crossed-attribute test makes the final call.

The lock-free read is safe for the same reason index-only scans' is.  The bit
is set before the fresh index entry is inserted and both precede commit, and a
snapshot that can see the new row version was taken after a full barrier
(<code>ProcArrayLock</code>) that follows the set.  A bit set after the
snapshot concerns only a version the snapshot cannot see, whose old entries
still agree, so intersecting them remains exact.

A table whose AM leaves the hook NULL &mdash; every non-heap AM &mdash; takes the
identical exact-intersection path it always did.  No index entry is marked, no
<code>ItemPointer</code> bit is reserved, no index access method changes, and
<code>BitmapOr</code> is untouched.  amcheck's <code>heapallindexed</code>
fingerprints the plain stored TIDs with no stripping.

The cost is precision, not correctness: <code>BitmapAnd</code> rechecks the
offsets either index named on a flagged block, rather than intersecting them,
until prune clears the bit.  That is narrower than the whole-page lossy
fallback an earlier version of this series used, and it costs nothing per TID.

An earlier version reserved a bit of a stored TID's offset as a per-entry
"inexact" marker and had <code>tbm_add_tuples</code> add a marked entry's page
lossily.  That put a heap-ism into the generic <code>ItemPointer</code> type,
required every offset reader to strip the bit, taxed every bitmap scan with a
per-TID test, and made <code>BitmapOr</code> pages lossy for no reason.  The
per-block signal replaces it entirely.

=== Step 4: Prune Collapses Dead Chains to Xid-Free Stubs ===

A dead mid-chain HOT-indexed tuple cannot be reclaimed to LP_UNUSED while a
not-yet-swept stale entry can still arrive at it, and its bitmap is what later
readers union.  Prune collapses a dead prefix: each preserved dead key tuple is
rewritten in place as an xid-free '''stub''' (LP_NORMAL, HEAP_INDEXED_UPDATED,
natts == 0, frozen XMIN/XMAX_INVALID, <code>t_ctid.offnum</code> forwarding to
the next survivor, carrying the same inline bitmap); a dead member whose
attributes are wholly subsumed by later hops is reclaimed outright instead.
The root becomes an LP_REDIRECT to the first survivor.  Readers step through
stubs transparently and still cross every surviving hop's bitmap.

The collapse rides the existing prune/freeze WAL; no new record type.  Once
VACUUM's index cleanup has swept the stale leaves and the whole chain is dead, a
later prune reclaims the stubs and re-points the root redirect, collapsing back
to classic HOT.

Reclamation is amortized onto reads via the existing opportunistic
<code>heap_page_prune_opt</code> sites, which prune a heap page while it is
pinned for a scan rather than deferring everything to VACUUM.  The sequential,
bitmap, and index-fetch scan paths already do this, and HOT-indexed chains are
reclaimed through those sites and by VACUUM.

---

== Why This Is Correct ==

The correctness argument rests on the crossed-attribute bitmap being the
staleness authority, plus the placement of fresh entries:

1. '''Exactly one entry per index resolves the row.'''  A fresh entry points at
the heap-only tuple whose key it matched, so its walk to the live tuple crosses
no later hop that changed its index's key &mdash; the union is disjoint and it
is kept.  A stale entry's walk ''does'' cross such a hop &mdash; the union
overlaps and it is dropped.  No duplicates, no lost rows.

2. '''ABA is handled.'''  If an indexed value is cycled away and back
(X &rarr; Y &rarr; X), the live tuple's key equals both the ancestor entry's
key and the fresh entry's key.  A value recheck would keep both and return the
row twice; the bitmap drops the ancestor because the column ''changed'' after
it, regardless of the coincident value (worked trace below).

3. '''The union is complete.'''  Every crossed live hop and every
collapse-survivor stub contributes its bitmap, and collapse reclaims a dead
member only when its attributes are a subset of the surviving later hops &mdash;
so a reader crossing the survivors still sees every collapsed hop's attributes.
Disjointness therefore reliably means current.

4. '''Unique checks compare values with the opclass comparator.'''
<code>_bt_check_unique</code> fetches the conflicting tuple under SnapshotDirty
and, when the chain walk crossed a HOT-indexed hop, compares its live index key
against the arriving leaf using the index's own ordering procedure
(<code>_bt_heap_keys_equal_leaf</code>, BTORDER_PROC under each column's
collation).  Using the opclass comparator &mdash; not a bitwise comparison
&mdash; recognizes a cycled key as the same logical row while still detecting a
genuinely live duplicate that is opclass-equal but not bitwise-identical
(numeric 1.0 vs 1.00, float -0.0 vs 0.0).  The recheck is required for
correctness, not just an optimization: in the in-flight window of a restoring
(Y&rarr;X) update the fresh X entry is not yet inserted, so the stale ancestor
leaf is the only witness of the conflict, and the recheck routes that hit into
<code>_bt_doinsert</code>'s xwait wait-and-recheck (a bitmap-only verdict would
skip it and admit a duplicate).  It reads plain key columns straight from the
heap slot and evaluates no indexed expression: an UPDATE touching an
expression-index attribute is disqualified from HOT-indexed
(<code>HeapUpdateHotAllowable</code>), so an expression index is never the one
receiving the fresh entry whose insert runs this check.  This helper is
internal to nbtree and is not on the read path.

5. '''A stub-bearing page is never PD_ALL_VISIBLE''', so the all-visible seqscan
and index-only-scan fast paths cannot surface stub bytes as a phantom row;
amcheck enforces this as an on-disk invariant.  WAL replay is verified
byte-for-byte under <code>wal_consistency_checking</code>, including the
collapse/stub records.

---

== Worked Example: Selective Maintenance, ABA, and Chain Collapse ==

Notation follows the in-tree README and RFC: <code>LP[n]</code> is the line
pointer at offset n; <code>{a,b}</code> is a modified-attrs bitmap; <code>-&gt;k</code>
is the same-page <code>t_ctid</code> successor; a tuple's bitmap = the attrs
that changed on the hop ''into'' it.

=== Example 1: selective maintenance and a stale drop ===

Table <code>t(id PK, a, b, c)</code>, indexes <code>t_a(a)</code>,
<code>t_b(b)</code>, <code>t_c(c)</code>, fillfactor 50.
<code>INSERT (1,10,20,30); UPDATE a=11; UPDATE b=21; UPDATE c=31.</code>

<pre>
Chain:
  LP[1] v1(a=10,b=20,c=30)  root, HEAP_HOT_UPDATED, ->2          dead
  LP[2] v2(a=11,b=20,c=30)  heap-only, INDEXED_UPDATED{a}, ->3   dead
  LP[3] v3(a=11,b=21,c=30)  heap-only, INDEXED_UPDATED{b}, ->4   dead
  LP[4] v4(a=11,b=21,c=31)  heap-only, INDEXED_UPDATED{c}        live

Index entries (fresh entries point mid-chain at the tuple they matched):
  t_a:  (10)->LP[1] stale     (11)->LP[2] fresh
  t_b:  (20)->LP[1] stale     (21)->LP[3] fresh
  t_c:  (30)->LP[1] stale     (31)->LP[4] fresh

Scan a=11 via t_a -> LP[2]:
  arrive AT LP[2] (own hop {a} not counted); cross ->3 {b}, ->4 {c}.
  crossed={b,c};  t_a keys={a};  {a} & {b,c} = {}  => fresh => return v4.  OK
Scan a=10 via t_a -> LP[1] (stale):
  cross ->2 {a}, ->3 {b}, ->4 {c};  crossed={a,b,c};  {a}&{a,b,c}={a} => drop.  OK
  (v4 is supplied once, by the fresh (11)->LP[2] entry.)
</pre>

=== Example 2: ABA &mdash; the case a value recheck gets wrong ===

<code>INSERT (1,a=10); UPDATE a=11; UPDATE a=10.</code> (a cycles 10 &rarr; 11 &rarr; 10)

<pre>
  LP[1] v1(a=10) root ->2 dead
  LP[2] v2(a=11) {a} ->3 dead
  LP[3] v3(a=10) {a}     live
  t_a:  (10)->LP[1] stale     (11)->LP[2] stale     (10)->LP[3] fresh

Scan a=10 finds TWO entries with key 10 (LP[1] and LP[3]):
  via LP[3]: zero hops crossed => fresh => return v3.                  OK
  via LP[1]: cross ->2 {a}, ->3 {a}; crossed={a}; {a}&{a}={a} => drop.  OK
Returned exactly once.  A value recheck would compare leaf key 10 against live
a=10 for BOTH entries and keep both -> duplicate.  The bitmap drops the
ancestor because a *changed* after LP[1], regardless of the coincident value.
</pre>

=== Example 3: collapse to xid-free stubs (back toward classic HOT) ===

From Example 1, VACUUM finds <code>LP[1..3]</code> dead, <code>LP[4]</code>
live.  Walking the dead prefix from the live end and accumulating the union of
later hops (<code>laterattrs</code>): a member is reclaimed if its bitmap is a
subset of later hops (its entries are already superseded), otherwise it is kept
as a stub.

<pre>
  seed laterattrs from the live remainder LP[4]: {c}.
  LP[3] {b}: {b} not-subset {c}   -> keep as stub forwarding ->4.  laterattrs={b,c}.
  LP[2] {a}: {a} not-subset {b,c} -> keep as stub forwarding ->3.  laterattrs={a,b,c}.
  LP[1] root -> LP_REDIRECT ->2 (first survivor).

Result:
  LP[1] redirect ->2
  LP[2] stub{a}  forward ->3      (xid-free, natts==0)
  LP[3] stub{b}  forward ->4      (xid-free, natts==0)
  LP[4] live v4

Scan a=11 via t_a (11)->LP[2]:
  arrive AT LP[2] stub (own segment {a} not counted); forward ->3 stub {b},
  ->4 {c};  crossed={b,c};  {a}&{b,c}={} => fresh => return v4.        OK
Scan a=10 via t_a (10)->LP[1] redirect ->2:
  follow redirect to LP[2] (now a crossed segment) {a}, ->3 {b}, ->4 {c};
  crossed={a,b,c};  {a}&{a,b,c}={a} => stale => drop.                 OK
</pre>

There is no "redirect-with-data": the bitmap lives on the stub itself, not on
the redirect.  Once every entry into the chain is swept by ambulkdelete and the
whole chain is dead, VACUUM reclaims the stubs to LP_UNUSED and re-points the
root redirect straight at the live tuple &mdash; the page is back to classic
HOT, with no metadata remaining.

=== Example 3a: prune and vacuum, step by step ===

A fuller trace of the same chain that separates what '''prune''' does (the
collapse) from what '''VACUUM''' does (the index sweep and the final reclaim).
Note there is no "redirect-with-data": the root becomes a plain LP_REDIRECT and
the per-hop bitmaps live on the stubs, which the reader crosses one by one.

Table <code>siu_collapse(id, a, b, c)</code> with indexes <code>siu_coll_a(a)</code>,
<code>siu_coll_b(b)</code>, <code>siu_coll_c(c)</code>:
<code>INSERT (1,10,20,30); UPDATE a=11; UPDATE b=21; UPDATE c=31;</code>

'''(0) Chain after the three HOT-indexed updates, before any prune.'''  Each
new version is a heap-only tuple carrying the bitmap of what changed at its
hop; each changed index got a fresh entry at the new tuple's own TID, and the
pre-update entries remain (now stale).

<pre>
  LP[1] v1(a=10,b=20,c=30)  root, HEAP_HOT_UPDATED, ->2     dead
  LP[2] v2(a=11,b=20,c=30)  heap-only, {a}, ->3            dead
  LP[3] v3(a=11,b=21,c=30)  heap-only, {b}, ->4            dead
  LP[4] v4(a=11,b=21,c=31)  heap-only, {c}                 live

  siu_coll_a:  (10)->LP[1] stale     (11)->LP[2] fresh
  siu_coll_b:  (20)->LP[1] stale     (21)->LP[3] fresh
  siu_coll_c:  (30)->LP[1] stale     (31)->LP[4] fresh
</pre>

'''(1) PRUNE collapses the dead prefix''' (on-access via heap_page_prune_opt,
or in VACUUM's first pass).  It finds LP[1..3] dead, LP[4] live, and walks from
the live end accumulating laterattrs (the union of later hops):

<pre>
  seed laterattrs = LP[4] {c}
  LP[3] {b}: {b} not subset of {c}     -> keep as stub ->4;  laterattrs={b,c}
  LP[2] {a}: {a} not subset of {b,c}   -> keep as stub ->3;  laterattrs={a,b,c}
  LP[1] root                           -> LP_REDIRECT ->2 (first survivor)
</pre>

A dead member is reclaimed outright (LP_DEAD) instead of stubbed only when its
bitmap is a subset of the later hops &mdash; then no live entry references it
and a later survivor still carries its attributes.  Here none qualify, so all
three are kept.  Result:

<pre>
  LP[1] redirect ->2
  LP[2] stub {a}  forward ->3   (xid-free: XMIN/XMAX_INVALID, natts==0)
  LP[3] stub {b}  forward ->4   (xid-free)
  LP[4] live v4
</pre>

The page is kept non-all-visible while a stub remains, so index-only scans
heap-fetch through it.  The stale and fresh leaves still point where they did;
only the heap changed.

'''(2) Reads against the collapsed page:'''

<pre>
Query a=11 via siu_coll_a, fresh entry (11)->LP[2]:
  arrive AT LP[2] stub (its own {a} is the entry's own hop, not counted);
  cross ->3 {b}, ->4 {c};  crossed={b,c};  key {a};  {a}&{b,c}={}
  => current => return v4.                                         OK
Query b=21 via siu_coll_b, fresh entry (21)->LP[3]:
  arrive AT LP[3] stub; cross ->4 {c};  crossed={c};  {b}&{c}={}
  => current => return v4.                                         OK
Query a=10 via siu_coll_a, STALE entry (10)->LP[1]:
  LP[1] is a plain redirect -> follow to LP[2]; now crossing the collapsed
  segment: LP[2] {a}, ->3 {b}, ->4 {c};  crossed={a,b,c};  {a}&{a,b,c}={a}
  => stale => drop (v4 is supplied once, by the fresh (11)->LP[2] entry).  OK
</pre>

'''(3) VACUUM index cleanup''' (ambulkdelete) removes the now-removable stale
leaves (10/20/30 -> LP[1]); kill_prior_tuple and bottom-up deletion also remove
them opportunistically.  VACUUM's heap second pass (lazy_vacuum_heap_page) does
NOT collapse or re-point anything; it only turns LP_DEAD line pointers into
LP_UNUSED.

'''(4) Final reclaim.'''  Once every entry into the chain has been swept and
the whole chain is dead, a later PRUNE reclaims the stubs to LP_UNUSED and
re-points the root redirect straight at the live tuple:

<pre>
  LP[1] redirect ->4      (or reclaimed if no entry references the root)
  LP[2] LP_UNUSED
  LP[3] LP_UNUSED
  LP[4] live v4
</pre>

No SIU metadata remains on the page; it is indistinguishable from a
classic-HOT chain that has been pruned.

=== Bitmap sizing across DDL ===

The inline bitmap is ceil(natts/8) bytes, sized by the tuple's '''own''' natts
at write time, not the relation's current natts.  ADD COLUMN raises the
relation's natts without rewriting existing tuples, so a chain can hold hops
sized for different natts; the sharp case is crossing an 8-attribute boundary,
where the byte count grows.  Every consumer locates a hop's bitmap from that
hop's own write-time natts (<code>HotIndexedTupleBitmapNatts</code>:
<code>HeapTupleHeaderGetNatts</code> for a live tuple, the stub's stashed natts
otherwise &mdash; a stub keeps its write-time natts in the unused block half of
<code>t_ctid</code>, since the offset half is the forward link).

<pre>
t(c1 PK,...,c7, payload)  -- exactly 8 attrs; t_c2(c2), t_c7(c7)
  UPDATE c7=71; UPDATE c7=72;        chain bitmaps are 1 byte (natts=8)
  ALTER TABLE t ADD COLUMN c9 int;   relation natts 8 -> 9; ceil 1 -> 2
  UPDATE c7=73;                      this hop's bitmap is 2 bytes (natts=9)

  LP[1] v1(c7=70) root ->2 dead   (1-byte)
  LP[2] v2(c7=71) {c7} ->3 dead   (1-byte)
  LP[3] v3(c7=72) {c7} ->4 dead   (1-byte)
  LP[4] v4(c7=73) {c7}     live   (2-byte)

Scan an unchanged c2 via t_c2 -> LP[1] (stale): cross ->2,->3,->4, each located
  by its own write-time natts; crossed={c7}; {c2}&{c7}={} => current => return v4.
  Sizing LP[2,3] with the relation's *current* natts=9 would misread a data
  byte as bitmap and could wrongly drop the current c2 entry; per-hop sizing
  avoids it.  OK
</pre>

DROP COLUMN keeps the attnum slot (it never renumbers), so bit positions and
natts are unchanged and existing bitmaps stay aligned.  CREATE INDEX/REINDEX
over a live chain indexes each live tuple under its own TID, so the new
entries cross no later hop and are never stale.

---

== Testing ==

The feature is covered by a regression suite, an adversarial isolation spec, a
crash-recovery TAP test, replication and logical-decoding tests, a pg_upgrade
test, and the amcheck/pg_surgery guards (all passing with
<code>cassert=true</code>):

* '''Regression''' (<code>hot_indexed_updates.sql</code>): eligibility and
  classification; selective maintenance across multiple/composite indexes; the
  crossed-attribute read path for equality and range scans; a key cycled away
  and back (ABA); TOASTed indexed columns; partial-index predicate flips;
  non-btree access methods (hash incl. ABA, GIN, GiST); CREATE INDEX/REINDEX
  over an existing chain; and DDL after a chain exists (CREATE/DROP INDEX,
  ADD COLUMN crossing a bitmap-size boundary, DROP COLUMN).
* '''Isolation''' (<code>hot_indexed_adversarial.spec</code>): concurrent
  UPDATE/VACUUM/prune and index scans, key cycling, aborts, and reader
  consistency across a concurrent collapse.  It also races a HOT-indexed update against a forced <code>BitmapAnd</code> over the changed and an unchanged index, committed and concurrent, to pin the per-block split signal's lock-free read.
* '''Recovery''' (<code>055_hot_indexed_recovery.pl</code>): crash + WAL replay
  of chains and stub collapse, byte-identical under
  <code>wal_consistency_checking = 'all'</code>.
* '''Replication''' (<code>039_hot_indexed_apply.pl</code>,
  <code>040_hot_indexed_replica_identity.pl</code>, test_decoding): the
  per-subscription apply modes, replica identity FULL and USING INDEX (incl. a
  cycled USING INDEX key), and logical decoding over chains.
* '''pg_upgrade''' (<code>009_hot_indexed.pl</code>): a relation with chains,
  an ABA-cycled column, a TOASTed indexed column, and VACUUM-collapsed stubs
  carried across a major-version upgrade and re-verified.  The 2-bit to 4-bit <code>_vm</code> rewrite was verified separately by upgrading a cluster built with the old layout (863 all-visible, all-frozen pages preserved; <code>pg_check_visible</code> reports no mismatches).
* '''amcheck / pg_surgery''': verify_heapam recognizes HOT-indexed tuples and
  stubs; pg_surgery refuses to freeze/kill a stub.

---

== Patch series ==

The series is staged so each commit is a complete, correct layer; no commit
fixes an earlier one.  The two generic contracts the feature rests on come
first, as preparatory patches with no HOT-indexed caller, so they can be
reviewed (and accepted or rejected) on their own terms.

  0001  Name the locator contract between table and index access methods
  0002  Rename TM_FailureData.traversed to retargeted
  0003  Let an index AM store a variable-width locator
  0004  Support table AMs that update rows in place
  0005  Add a per-block locator-split signal for BitmapAnd
  0006  Improve test coverage for heap-only tuple (HOT) update behavior
  0007  Identify modified indexed attributes in the executor on UPDATE
  0008  Implement the heap AM's modified-attrs comparison callback
  0009  Add the selective-indexed on-disk format: inline attr bitmap and stubs
  0010  Add selective-indexed partially-HOT updates
  0011  Reclaim dead selective-indexed redirect chains during prune/vacuum
  0012  Extend heap statistics to include selective-indexed updates
  0013  Teach amcheck to recognize HOT-indexed chains and collapse stubs
  0014  Add a per-subscription hot_indexed_on_apply option

The locator contract that patches 0001 through 0004 build is split so each
piece can be judged on its own.  Patch 0001 replaces convention with a declared
contract.  A table AM returns a per-relation ''locator descriptor''
(<code>relation_locator</code>, reached through
<code>RelationGetLocatorDesc</code>): the locator's width (positive for a fixed
size, negative for a variable one whose magnitude bounds it), whether the row
keeps its locator across an UPDATE (<code>stable</code>), and two per-operation
bitmap hooks (below).  <code>CREATE INDEX</code> checks the pairing, and the
planner offers bitmap paths as before, when the AM has the bitmap scan
callback.  Heap answers every question the way core code already assumes, so
for heap the patch is a no-op.

Patch 0002 is a pure rename: <code>TM_FailureData.traversed</code> becomes
<code>retargeted</code> and gains <code>table_locator_is_stable()</code>, since
its readers care whether the tuple they locked is the one they named, not
whether a chain was walked.  For heap the two coincide; for an AM whose locator
is stable they do not.

Patch 0003 lets an index AM store a locator wider than a <code>tid</code>
(<code>amcanvarlocator</code>, reported through <code>pg_indexam_has_property</code>
as <code>can_var_locator</code>); <code>CREATE INDEX</code> refuses a wide
locator on an index AM that cannot store it.  No in-core index AM sets the
flag, so nothing changes for existing AMs.

Patch 0004 makes the in-place-update properties load-bearing.  The descriptor's
<code>old_version_retained</code> is false for an AM that overwrites a row's
storage on UPDATE; when it is false the executor copies the pre-update image
before the write, after-trigger events carry the row images for the life of the
transaction instead of re-fetching them by TID, and EvalPlanQual uses
<code>ROW_MARK_COPY</code> for such a relation.  No in-core AM exercises those
paths; they exist for out-of-core AMs that update in place, as zheap, zedstore
and OrioleDB all do.

Patch 0005 is the per-block signal described in the bitmap-scan section above:
the visibility-map widening, the <code>pg_upgrade</code> rewrite, and the
<code>BitmapAnd</code> consultation through the descriptor's two per-operation
hooks, <code>bitmap_and_inexact</code> and <code>bitmap_or_inexact</code> (heap
sets only the first; the second lets a future AM whose union is not exact force
a recheck).  It has no heap caller; the feature commit is what sets the bit and
prune is what clears it.

Patches 0007 and 0008 are the contract the feature rests on: the executor
determines which indexed attributes an UPDATE actually changed, and the table AM
supplies the comparison (<code>modified_attrs</code>).  That deliberately moves
the determination out of <code>heap_update()</code> &mdash; where upstream does it
today, via <code>HeapDetermineColumnsInfo()</code> with the old tuple already
pinned &mdash; and up into the executor behind the table-AM boundary.  The price is
that the non-executor path (<code>simple_heap_update()</code>, used by catalog
updates, which has no <code>ResultRelInfo</code>/<code>EState</code> to ask) must
fetch the old tuple and compare it itself.  That is an accepted trade, confined
to catalog DDL, reading an already-warm buffer, and measuring at noise level.

=== Relationship to the slot-based table-AM index scan interface ===

The read path is written against the slot-based index-scan interface added by
<code>ddce1da5</code> ("Add slot-based table AM index scan interface").  That
commit removed <code>table_index_fetch_tuple</code> and moved per-scan table
state into the AM-private <code>xs_table_opaque</code>, so the HOT-indexed
staleness state (the crossed-attribute bitmap and its verdict) lives in heap's
<code>IndexScanHeapData</code>, and the drop decision is made inside heap's
<code>getnext_slot</code> implementations rather than in the executor nodes.
Constraint enforcement, which holds a TID taken from an index but performs no
index scan, uses a <code>fetch_tid_check</code> table-AM callback that reports
the same verdict alongside the fetched tuple.  The bitmap-scan signal does not
depend on a future <code>amgetbatch</code> interface: it lives on the heap
block, not in the index entry, so nothing needs to travel alongside each TID.

---

== Scope ==

Supported and tested under any access method (the read path is access-method
agnostic): selective maintenance across multiple indexes; partial indexes
(including predicate flips); exclusion constraints, including a temporal
<code>PRIMARY KEY ... WITHOUT OVERLAPS</code> (the exclusion recheck remains
authoritative for conflict detection while the bitmap drops stale entries);
summarizing indexes (BRIN, via the existing summarizing path); partitioned
tables (a within-partition update is HOT-indexed on the leaf heap; a
partition-key change is a cross-partition delete+insert and is never HOT);
TOASTed indexed columns; and replica identity FULL / USING INDEX on the apply
path.

Deliberately ineligible (carve-outs): system catalogs; an UPDATE that changes
an expression-index input (the bitmap is attribute-granular and cannot tell
whether the expression's value changed &mdash; expression-aware selective
maintenance is deferred); an UPDATE that changes ''every'' indexed attribute
(nothing to skip); and the logical-replication apply path unless permitted by
<code>hot_indexed_on_apply</code>.

---

== The Three Transitions: Where the Bitmap Lives ==

The modified-attrs bitmap is never persisted in WAL records or in a permanent
tuple-header field.  It is born inline on the heap-only tuple, is preserved
(re-homed) on a stub when prune collapses the chain, and disappears entirely
once the chain is fully dead.  Here are the three transitions, in the same
notation as the worked examples.

=== Transition 1: Bitmap inline on the new heap-only tuple (the UPDATE) ===

A HOT-indexed UPDATE links a heap-only tuple into the chain and appends, in the
final ceil(natts/8) bytes of its item, the bitmap of attributes changed at this
hop.  A fresh index entry for each changed index points at this tuple's own TID.

<pre>
Initial:
  LP[1] = T1(a=10,b=20,c=30)   root

After UPDATE c=31 (change {c}):
  LP[1] = T1   root, HEAP_HOT_UPDATED, ->2          (still live here)
  LP[2] = T2(a=10,b=20,c=31)   heap-only, INDEXED_UPDATED, inline bitmap {c}
  t_c gains a fresh entry (31) -> LP[2]; t_a, t_b untouched.
</pre>

The bitmap is on the data-bearing tuple itself &mdash; there is no separate
tombstone line pointer, and nothing about it is written to WAL beyond the
ordinary heap-update record that already carries the new tuple's bytes.

=== Transition 2: Bitmap re-homed on an xid-free stub (prune/collapse) ===

When the dead prefix is collapsed, each preserved dead key tuple is rewritten
in place as an xid-free stub that ''keeps its own inline bitmap'' and forwards
to the next survivor; a member whose attributes are wholly subsumed by later
hops is reclaimed instead.  The root becomes a plain LP_REDIRECT.

<pre>
After UPDATE c=31 {c}, UPDATE b=21 {b}, then T1,T2 die (T3 live):
  LP[1] = LP_REDIRECT ->2            (first survivor; no payload)
  LP[2] = stub, INDEXED_UPDATED, natts==0, bitmap {c}, forward ->3, frozen
  LP[3] = T3(a=10,b=21,c=31)   live, INDEXED_UPDATED, bitmap {b}
</pre>

The bitmap did not move into the redirect (there is no "redirect-with-data");
it stays on the stub, which a reader crosses just like a live hop.  The stub is
XMIN/XMAX_INVALID, so it holds back nothing for freezing, and it preserves its
write-time natts in the unused block half of <code>t_ctid</code> so the bitmap
stays correctly sized after a later ADD COLUMN.

=== Transition 3: Bitmap gone (back to classic HOT) ===

Once ambulkdelete has swept every stale leaf and the whole chain is dead, a
later prune reclaims the stubs to LP_UNUSED and re-points the root redirect
straight at the live tuple.

<pre>
  LP[1] = LP_REDIRECT ->3      (or reclaimed, depending on remaining refs)
  LP[2] = LP_UNUSED
  LP[3] = T3   live
</pre>

No bitmap remains anywhere; the page is indistinguishable from a classic-HOT
chain.  The metadata was ephemeral throughout: inline on a tuple, then on a
stub, then nothing &mdash; never encoded permanently in tuple headers or WAL,
and never replicated as per-column metadata (a subscriber sees only ordinary
UPDATE/prune records).


== Eligibility ==

There is no cost heuristic and no GUC: any UPDATE that changes a
non-summarizing indexed attribute is HEAP_SELECTIVE_INDEX_UPDATE unless a carve-out
applies.  <code>HeapUpdateHotAllowable</code> decides this from the
<code>modified_idx_attrs</code> bitmap and the per-relation indexed-attribute
set (<code>RelationGetIndexedAttrs</code>).

The carve-outs that force HEAP_UPDATE_ALL_INDEXES are:

* '''Every indexed attribute changed.'''  Nothing can be skipped, so a plain
  non-HOT update is cheaper (it avoids the chain walk and bitmap overhead).
  This is an exact test, not a percentage.
* '''An expression-index input changed.'''  The bitmap is attribute-granular
  and cannot tell whether the expression's value changed; expression-aware
  selective maintenance is deferred.  (This is the same kind of restriction the
  partial-index one used to be, and is expected to be liftable once tested.)
* '''System catalogs.'''  A catalog UPDATE that changes a non-summarizing
  indexed attribute stays classic HOT but never takes the HOT-indexed path:
  catalogs are reached through access paths (systable scans, SnapshotDirty
  unique checks) not yet proven safe.
* '''The logical-replication apply path''', gated per subscription by
  <code>hot_indexed_on_apply</code> (off / subset_only (default) / always): a
  HOT-indexed update of a replica-identity attribute leaves a stale leaf the
  apply worker's RI lookup must tolerate, which it does only when the indexed
  attributes are a subset of (or equal to, for off) the replica identity.

Everything else &mdash; multiple/partial/composite indexes, any access method,
summarizing (BRIN) columns, exclusion constraints, partitioned tables, TOASTed
columns &mdash; is eligible and tested (see Scope and Testing).  An earlier
value-recheck draft needed several extra restrictions (partial-index
predicates, a chain-length cap); they were removed once the crossed-attribute
bitmap became the staleness authority &mdash; the cap in particular never
bounded anything (it measured from the chain tail) and growth is instead
bounded naturally by the page and by prune/collapse.


== Why Previous Efforts Failed ==

=== WARM (Write-Amplification Reduction Mechanism) ===

The [https://wiki.postgresql.org/wiki/WARM WARM] proposal by the author of HOT had an "alternating update" model: tuples in a chain would alternate between being "hot updatable" (new versions can be added without index updates) and "cold" (fully indexed). The idea: on write-heavy workloads, a series of updates would alternate, some skipping indexes and some updating them. Indexes maintain references to the root LP, not mid-chain.

Why it failed:

* '''Unpredictable benefit:''' Whether alternation helps depends on access patterns. A read-heavy query might hit a "cold" tuple and still need index scans. A write-heavy workload might be updating the same tuple repeatedly, making the alternating state machine ineffective.

* '''Operator confusion:''' In production, having some index entries be fresh and others stale is hard to reason about. When does alternation trigger? When does it not? The state machine adds cognitive overhead.

* '''Incomplete solution:''' WARM didn't address what happens when the chain is pruned or when readers encounter stale entries. Does alternation survive VACUUM? Can readers rely on it?

The community decided the complexity wasn't justified by the unpredictable payoff. WARM was never committed.

=== PHOT (Partial HOT) ===

[https://wiki.postgresql.org/wiki/Partial_HOT PHOT]
([https://pg.ddx.io/pgsql-hackers/2ECBBCA0-4D8D-4841-8872-4A5BBDC063D2@amazon.com/ "partial heap only tuples", Nathan Bossart, pgsql-hackers, 2021-02-09])
tracked per-tuple which
indexed columns were modified. When a new version skipped updating indexes, PHOT
would record which columns actually changed. Readers would consult this metadata
during index scans to validate stale entries.

Why it was promising but ultimately failed:

* '''Metadata persistence problem:''' PHOT needed to track "which columns were
modified" using LP_DEAD space on the page identified during pruning as well as
in WAL records. This couples index metadata (which columns are indexed?) with
tuple state (which columns changed?). When replicating to a subscriber with
different indexes, this becomes intractable.

* '''Chain walking after prune:''' PHOT required readers to walk HOT chains
consulting per-tuple metadata. But if VACUUM prunes a chain member before all
readers process it, readers might miss tuples or hit dangling pointers. PHOT
deferred this problem explicitly, acknowledging a correctness gap as a WIP.

* '''Concurrent prune/vacuum races:''' PHOT had no robust answer for: What if
VACUUM removes a chain member while a reader is traversing it? What if prune
modifies metadata while readers consult it? The per-tuple metadata made
concurrency reasoning harder, not easier.

* '''WAL encoding complexity:''' Encoding per-tuple column-modification info in
every WAL record bloats the log and couples the replication stream to index
structure. Subscriber index changes break assumptions made at publication time.

* '''Replication safety:''' Under logical replication, subscriber indexes can
diverge from publisher indexes (drop index, add index, change index
columns). PHOT had no mechanism to detect divergence or prevent corruption.

PHOT was ambitious but required solving too many hard problems
simultaneously. The proposal stalled around 2016.

=== Why HOT-indexed Updates Is Different ===

This work takes a different approach:

* '''Metadata is ephemeral, not persistent.''' Like PHOT, we track which
indexed columns were modified&mdash;but only inline on the heap-only tuple (and,
after collapse, on its xid-free stub), never replicated across the chain or
encoded permanently in WAL.  A reader consults the bitmaps it crosses during
chain traversal; the ordinary heap-update and prune records already carry
everything needed, so the replication stream is not coupled to index structure.

* '''Chain walking is deterministic and the bitmap is positional per hop.''' A
fresh entry points at the tuple whose key it matched, so it crosses no later
key-changing hop; a stale entry does.  Staleness is the crossed-attribute
overlap, not a value recheck, so an ABA cycle cannot fool it.

* '''Collapse needs no "convert back" bookkeeping.''' Prune rewrites the dead
prefix to stubs that preserve their bitmaps and forward to the next survivor;
once the chain is fully dead VACUUM reclaims the stubs and re-points the root
redirect.  There is no alternating state machine and no per-tuple flag to clear.

* '''Replication safety is explicit.''' We provide a per-subscription option to
control whether HOT-indexed updates are used on the apply side. Administrators
choose: safe (always off), compatible (on only when index sets match), or risky
(always on, requires manual sync). No silent corruption.

---


== Acknowledgements ==

This series builds directly on prior work by:

* '''Pavan Deolasee''' and '''Tom Lane''' &mdash; classic HOT (2007), commit <code>282d2a03dd3</code>.
* '''Pavan Deolasee''' and '''Gokulakannan Somasundaram''' &mdash; original HOT design (2007).
* '''Pavan Deolasee''' &mdash; WARM (2017), the structural template this series consciously moves away from: it replaces WARM's value recheck with an attribute-bitmap staleness test on mid-chain pointers.
* '''Nathan Bossart''' &mdash; [https://pg.ddx.io/pgsql-hackers/2ECBBCA0-4D8D-4841-8872-4A5BBDC063D2@amazon.com/ PHOT (2021)], the structural template for "mid-chain pointers" that this series finishes.
* '''Matthias van de Meent''', '''Tomas Vondra''', '''Josef Simanek''', '''Álvaro Herrera''' &mdash; <code>amsummarizing</code> and the per-index update-decision relaxation (PostgreSQL 17, commit <code>19d8e2308bc</code>) &mdash; the first relaxation of HOT's I1 invariant.  (This series replaces the <code>TU_UpdateIndexes</code> result code with an executor-side modified-attrs bitmap, determined via the table AM's <code>modified_attrs</code> callback.)
* '''Peter Geoghegan''' &mdash; <code>indexUnchanged</code> hint and bottom-up btree deletion (PostgreSQL 14, commit <code>d168b666823</code>) &mdash; the per-index hint mechanism this series builds on.
* '''Álvaro Herrera''' &mdash; BRIN (PostgreSQL 9.5, commit <code>7516f525941</code>) &mdash; the conceptual split between summarizing and per-tuple indexes.

== Discussion ==

The pgsql-hackers thread for this proposal has not yet been started. When posted, the thread URL will be added here.

This wiki page is the design preview; the in-tree <code>src/backend/access/heap/README.HOT-INDEXED</code> is the authoritative reference and is kept current with the code.

== Links and References ==

=== PostgreSQL ===

* [[HOT|HOT wiki page]] &mdash; https://wiki.postgresql.org/wiki/HOT
* [https://www.postgresql.org/docs/current/storage-hot.html PostgreSQL documentation: Heap-Only Tuples (HOT)]
* Commit <code>282d2a03dd3</code>: HOT updates (Tom Lane, 2007-09-20)
* Commit <code>7516f525941</code>: BRIN: Block Range Indexes (Álvaro Herrera, 2014-11-07)
* Commit <code>d168b666823</code>: Enhance nbtree index tuple deletion (Peter Geoghegan, 2021-01-13)
* Commit <code>19d8e2308bc</code>: Ignore BRIN indexes when checking for HOT updates (Tomas Vondra, 2023-03-20)

=== Prior proposals ===

* [https://www.postgresql.org/message-id/CABOikdN1QLxMVkFrq7kGe4CVzaPHMSRdJYaRVes9HjG1%2BtV7Gg%40mail.gmail.com WARM proposal on pgsql-hackers (2017)] &mdash; Pavan Deolasee, EnterpriseDB
* [https://pg.ddx.io/pgsql-hackers/2ECBBCA0-4D8D-4841-8872-4A5BBDC063D2@amazon.com/ "partial heap only tuples" (PHOT) on pgsql-hackers (2021)] &mdash; Nathan Bossart, Amazon

=== In-tree documentation ===

* <code>src/backend/access/heap/README.HOT-INDEXED</code> &mdash; design reference
* <code>src/backend/access/heap/README.HOT</code> &mdash; classic HOT reference (unchanged)
* <code>src/test/regress/sql/hot_indexed_updates.sql</code> &mdash; regression tests
* <code>src/test/benchmarks/siu/</code> &mdash; A/B benchmark harness

[[Category:Development]]
[[Category:Internals]]
[[Category:Storage]]
[[Category:Proposed Features]]
