Re: BUG #19636: heap_fill_tuple misses short varlena conversion after EOH_flatten_into, causing 3-byte waste per row - Mailing list pgsql-bugs
| From | Andrey Rachitskiy |
|---|---|
| Subject | Re: BUG #19636: heap_fill_tuple misses short varlena conversion after EOH_flatten_into, causing 3-byte waste per row |
| Date | |
| Msg-id | CAB8bMivoqD0ScXgr3yv-Sq=8jVsGSS+UaEYA+1hs1WBTsm1kvQ@mail.gmail.com Whole thread |
| In response to | BUG #19636: heap_fill_tuple misses short varlena conversion after EOH_flatten_into, causing 3-byte waste per row (PG Bug reporting form <noreply@postgresql.org>) |
| List | pgsql-bugs |
пт, 21 авг. 2026 г. в 19:37, Andrey Rachitskiy <pl0h0yp1@gmail.com>:
History
-------
Short headers date from 3e23b68dac0 (2007, "Support varlena fields with
single-byte headers and unaligned storage"). heap_form_tuple has packed
inline 4-byte-header varlenas via VARATT_CAN_MAKE_SHORT ever since the
modern heap_form_tuple path.
Expanded objects arrived in 1dc5ebc9077 (2015, "Support expanded
objects, particularly arrays, for better performance"). That commit
added an EXTERNAL_EXPANDED branch in heap_compute_data_size / fill_val
that flattens with EOH_flatten_into and writes the result as-is.
Flatteners are required to produce an inline 4-byte-header varlena
(see expandeddatum.h). The new branch never applied the existing
short-header conversion afterward. So this looks like an omission from
day one of expanded objects, not a later regression.
Proposal Fix
---
In those EXTERNAL_EXPANDED arms, if the attribute is packable and the
flat size from EOH_get_flat_size() would fit a short header, flatten
into a temporary palloc buffer and emit the short form into the tuple.
Otherwise keep the previous path (align and flatten straight into the
tuple).
VARATT_CAN_MAKE_SHORT cannot be used on the expanded toast pointer
itself. It requires a 4B_U varlena. Before flattening we only have
the flat size, so the patch uses a size-only helper matching that
macro's length rule.
The temporary buffer is needed because EOH_flatten_into expects a
maxaligned destination (same constraint as datumSerialize), while short
packing writes at an unaligned data pointer. The temp is not pfree'd.
fill_val can run with CurrentMemoryContext as a BumpContext. On
master / REL_19, RecursiveUnion UNION DISTINCT stores hashed tuples in
a BumpContext tuplescxt (c106ef08071, "Use BumpContext contexts in
TupleHashTables"). LookupTupleHashEntry switches to that context
before ExecCopySlotMinimalTupleExtra → heap_form_minimal_tuple. Bump
does not support pfree. A variant that pfree'd the temp failed the
"with" regress test on 19/master with:
ERROR: pfree is not supported by the bump memory allocator
REL_18 still used AllocSet for that context, so the same pfree passed
there. The short-packable flat size is at most about 130 bytes. The
chunk is reclaimed when the context is reset.
Alternatives considered: Stack buffer was considered (fits the size bound), but palloc with EOH_flatten_into seemed better to me.
A follow-up on the repro and on the regress in v1.
The heaptuple omission is real, but the reporter's one-shot example often
does not reach it. With the default plan_cache_mode = auto, the first
executions of the PL/pgSQL INSERT use a custom plan. The bound array
parameter is substituted during planning. datumCopy flattens the
expanded object there, so fill_val later sees an ordinary 4-byte-header
varlena and packs it to short as usual. Both rows then show
pg_column_size 33 even without the fix.
I checked both cases with gdb on an unpatched build, breaking on
EOH_flatten_into.
1. Default plan_cache_mode (auto), one-shot PL/pgSQL INSERT — custom plan
EOH_flatten_into is called once, with allocated_size 36, from datumCopy
during planning (parameter substitution in eval_const_expressions):
#0 EOH_flatten_into (... allocated_size=36)
#1 datumCopy
#3 eval_const_expressions_mutator
...
# BuildCachedPlan / GetCachedPlan
# SPI_execute_plan_with_paramlist
# plpgsql exec_stmt_execsql
There is no fill_val frame on that hit. heap_form_tuple later sees an
ordinary flat 4-byte-header varlena and applies VARATT_CAN_MAKE_SHORT as
usual. pg_column_size is 33 for the plain INSERT and 33 for the
PL/pgSQL INSERT.
2. Same SQL with a generic plan
SET plan_cache_mode = force_generic_plan;
EOH_flatten_into is again called with allocated_size 36, but the caller
is the EXTERNAL_EXPANDED arm in fill_val:
#0 EOH_flatten_into (... allocated_size=36)
#1 fill_val (... heaptuple.c)
#2 heap_fill_tuple
#3 heap_form_tuple
...
# ExecModifyTable / SPI / plpgsql
That arm writes the flattener output as-is, so the PL/pgSQL row keeps
the 4-byte header. Without the fix pg_column_size is 33 (plain INSERT)
and 36 (PL/pgSQL). With the fix both are 33.
Under plan_cache_mode = auto the same fill_val path appears after the
plan cache switches from custom to generic (after several executions of
the INSERT). force_generic_plan just makes that path reliable on the
first call.
v2 of the patch is attached. The heaptuple change is the same as v1.
The plpgsql regress now sets force_generic_plan around the test so case
2 is what we cover. Case 1 would pass even without the fix.
The heaptuple omission is real, but the reporter's one-shot example often
does not reach it. With the default plan_cache_mode = auto, the first
executions of the PL/pgSQL INSERT use a custom plan. The bound array
parameter is substituted during planning. datumCopy flattens the
expanded object there, so fill_val later sees an ordinary 4-byte-header
varlena and packs it to short as usual. Both rows then show
pg_column_size 33 even without the fix.
I checked both cases with gdb on an unpatched build, breaking on
EOH_flatten_into.
1. Default plan_cache_mode (auto), one-shot PL/pgSQL INSERT — custom plan
EOH_flatten_into is called once, with allocated_size 36, from datumCopy
during planning (parameter substitution in eval_const_expressions):
#0 EOH_flatten_into (... allocated_size=36)
#1 datumCopy
#3 eval_const_expressions_mutator
...
# BuildCachedPlan / GetCachedPlan
# SPI_execute_plan_with_paramlist
# plpgsql exec_stmt_execsql
There is no fill_val frame on that hit. heap_form_tuple later sees an
ordinary flat 4-byte-header varlena and applies VARATT_CAN_MAKE_SHORT as
usual. pg_column_size is 33 for the plain INSERT and 33 for the
PL/pgSQL INSERT.
2. Same SQL with a generic plan
SET plan_cache_mode = force_generic_plan;
EOH_flatten_into is again called with allocated_size 36, but the caller
is the EXTERNAL_EXPANDED arm in fill_val:
#0 EOH_flatten_into (... allocated_size=36)
#1 fill_val (... heaptuple.c)
#2 heap_fill_tuple
#3 heap_form_tuple
...
# ExecModifyTable / SPI / plpgsql
That arm writes the flattener output as-is, so the PL/pgSQL row keeps
the 4-byte header. Without the fix pg_column_size is 33 (plain INSERT)
and 36 (PL/pgSQL). With the fix both are 33.
Under plan_cache_mode = auto the same fill_val path appears after the
plan cache switches from custom to generic (after several executions of
the INSERT). force_generic_plan just makes that path reliable on the
first call.
v2 of the patch is attached. The heaptuple change is the same as v1.
The plpgsql regress now sets force_generic_plan around the test so case
2 is what we cover. Case 1 would pass even without the fix.
Regards,
Rachitskiy Andrey
Rachitskiy Andrey
Attachment
pgsql-bugs by date: