BUG #19557: Parallel leader stuckat ParallelFinish leaked InterruptHoldoffCount - Mailing list pgsql-bugs

The following bug has been logged on the website:

Bug reference:      19557
Logged by:          Jacob Mørk Brejnbjerg
Email address:      jacob@brejnbjerg.com
PostgreSQL version: 18.4
Operating system:   Ubuntu 24.04 (kernel 6.8.0-124-generic)
Description:

A backend executing a parallel query has been stuck for 40+ hours in
wait_event IPC/ParallelFinish. All of its parallel workers exited cleanly
(exit code 0, nothing logged). pg_cancel_backend() and
pg_terminate_backend() have no effect. We kept the backend alive and
debugged it with gdb + debug symbols; the evidence shows a leaked
InterruptHoldoffCount, plus a design interaction that turns any such leak
into a permanently unkillable backend.

Possibly relevant configuration: io_method = io_uring,
max_parallel_workers_per_gather = 48,
shared_preload_libraries = 'timescaledb,pg_parquet'
(timescaledb 2.27.2, pg_parquet 0.5.1).
PostgreSQL 18.4 (Ubuntu 18.4-1.pgdg24.04+1).

The query is a COPY (SELECT <aggregation over one table> GROUP BY ...)
TO STDOUT (FORMAT parquet) issued via pg_parquet's COPY hook; the inner
select runs a parallel plan (Gather, 10+ workers planned) over
TimescaleDB compressed-hypertable chunks, i.e. an AIO-read-heavy scan.

Backtrace of the stuck leader (main thread; the only other thread is an
io_uring kernel worker, "iou-wrk-<pid>"):

#0  epoll_wait (epfd=754, events=..., maxevents=1, timeout=-1)
#1  WaitEventSetWait ()
#2  WaitLatch ()
#3  WaitForParallelWorkersToFinish ()
#4  ExecParallelFinish ()
#5-#8  (Gather shutdown path, static functions)
#9  standard_ExecutorRun ()
#10-#11 PortalRun ()
#12-#17 pg_parquet.so (COPY TO hook driving the portal)
#18 prev_ProcessUtility (timescaledb process_utility.c:124)
#19 timescaledb_ddl_command_start (...)
#20-#22 PortalRun ()
#23-#24 PostgresMain ()

Key state read from the live process via gdb:

InterruptPending       = 1
ProcDiePending         = 1    <- pg_terminate_backend arrived, never
serviced
QueryCancelPending     = 1    <- pg_cancel_backend arrived, never serviced
InterruptHoldoffCount  = 1    <- held, forever
CritSectionCount       = 0
num_held_lwlocks       = 0    <- NO LWLock is held
ParallelMessagePending = 1    <- worker messages queued, unprocessable

/proc signal state: SigPnd/ShdPnd are all zero - the SIGTERM/SIGINT were
delivered and consumed by the handlers (which set the flags above); the
backend wakes from WaitLatch, but CHECK_FOR_INTERRUPTS() is a no-op with
InterruptHoldoffCount > 0, so it just re-sleeps.

Why this is a permanent hang and not just an uncancellable wait:
worker-exit accounting is itself interrupt-driven. The workers' detach
notifications are only processed via CHECK_FOR_INTERRUPTS() ->
ProcessParallelMessages(), which is what clears each worker's error_mqh.
With the holdoff leaked, ProcessParallelMessages never runs
(ParallelMessagePending stays 1 - verified in the live process), so
WaitForParallelWorkersToFinish() believes the workers are still attached
and waits forever. The same leaked holdoff blocks ProcDiePending. One
leaked holdoff therefore produces both symptoms; only SIGKILL
(crash-restart) or a debugger can clear the backend.

On the origin of the leak: num_held_lwlocks = 0 rules out a stuck
LWLockAcquire, and the server log shows NO error from this backend for
its entire lifetime, which we believe rules out the
"ereport(ERROR) raised under HOLD, caught by PG_TRY" pattern (the raise
would have been logged by errfinish). That points to a straight-line
code path that called HOLD_INTERRUPTS() and returned without
RESUME_INTERRUPTS(). We audited the read path: bufmgr.c, read_stream.c,
shm_mq.c, condition_variable.c contain no bare holds; timescaledb 2.27.2
and pg_parquet 0.5.1 (and pgrx 0.16.1) contain no HOLD_INTERRUPTS at
all. The remaining bare holds on this workload's path are in
src/backend/storage/aio/aio.c (four HOLD/RESUME regions, including
pgaio_io_reclaim which runs completion callbacks under HOLD) and the
parallel-message path itself. Given io_method = io_uring and an
AIO-heavy scan we suspect an AIO path, but we could not identify the
exact site from the live process.

Frequency: once in ~6 days of heavy production use of this COPY path
(thousands of executions/day). Not deterministically reproducible so
far.

We have a gcore of the stuck backend and can keep it (and/or the live
process) available - happy to run additional gdb commands against it.

Separately, it may be worth considering a defensive measure: because
WaitForParallelWorkersToFinish depends on interrupt processing for its
own termination condition, any holdoff leak from any subsystem converts
into a silent, unkillable backend. Detecting "waiting on ParallelFinish
with INTERRUPTS_CAN_BE_PROCESSED() false and ParallelMessagePending set"
and at least logging (or treating it as an invariant violation) would
make this failure mode visible instead of silent.