Re: pg19b1: TRAP: failed Assert("pgstat_bktype_io_stats_valid(bktype_shstats, MyBackendType)") - Mailing list pgsql-hackers
| From | Andrey Rachitskiy |
|---|---|
| Subject | Re: pg19b1: TRAP: failed Assert("pgstat_bktype_io_stats_valid(bktype_shstats, MyBackendType)") |
| Date | |
| Msg-id | CAB8bMiuzYi3+eOd-Z5RFF9aFccAsajPbJ12YfsHtrEKaF7pVrQ@mail.gmail.com Whole thread |
| In response to | pg19b1: TRAP: failed Assert("pgstat_bktype_io_stats_valid(bktype_shstats, MyBackendType)") (Justin Pryzby <pryzby@telsasoft.com>) |
| Responses |
Re: pg19b1: TRAP: failed Assert("pgstat_bktype_io_stats_valid(bktype_shstats, MyBackendType)")
|
| List | pgsql-hackers |
I think the assert in pgstat_io_flush_cb() after 999dec9ec6a comes from IO stats on the foreign IO path.
When AsyncReadBuffers() finds IO already in progress, it sets foreign_io and returns without counting a read. But WaitReadBuffers() still calls:
```
pgstat_count_io_op_time(..., IOOP_READ, io_start, 0, 0);
```
The comment there assumes the read was already counted in AsyncReadBuffers(). That holds for IO we started ourselves, but not when we're waiting on another backend's IO. With track_io_timing = on, we get non-zero IOOP_READ time and zero count, and pgstat_bktype_io_stats_valid() trips on backend exit.
fix proposal:
```
- pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
- io_start, 0, 0);
+ if (!operation->foreign_io)
+ pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
+ io_start, 0, 0);
```
When AsyncReadBuffers() finds IO already in progress, it sets foreign_io and returns without counting a read. But WaitReadBuffers() still calls:
```
pgstat_count_io_op_time(..., IOOP_READ, io_start, 0, 0);
```
The comment there assumes the read was already counted in AsyncReadBuffers(). That holds for IO we started ourselves, but not when we're waiting on another backend's IO. With track_io_timing = on, we get non-zero IOOP_READ time and zero count, and pgstat_bktype_io_stats_valid() trips on backend exit.
fix proposal:
```
- pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
- io_start, 0, 0);
+ if (!operation->foreign_io)
+ pgstat_count_io_op_time(io_object, io_context, IOOP_READ,
+ io_start, 0, 0);
```
ср, 8 июл. 2026 г. в 19:58, Justin Pryzby <pryzby@telsasoft.com>:
While restoring from backup to v19, I hit a crash adding a unique
constraint, since 999dec9ec6a "aio: Don't wait for already in-progress
IO."
The crash is not consistent, and I don't have an easy reproducer, but I
can tell you that the crash involves the counts<=0 case, with
IOOBJECT_RELATION / IOCONTEXT_NORMAL / IOOP_READ, and depends on
maintenance_work_mem=512MB and track_io_timing=1, and always seems to
happens in a parallel worker.
LOG: background worker "parallel worker" (PID 1475078) was terminated by signal 6: Aborted
DETAIL: Failed process was running: ALTER TABLE ONLY public.counter_latest_data
ADD CONSTRAINT counter_latest_data_counter_id_site_id_key UNIQUE (counter_id, site_id);
ALTER TABLE public.counter_latest_data CLUSTER ON counter_latest_data_counter_id_site_id_key;
(gdb) bt
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
#1 0x00007f28d65d49ff in __pthread_kill_internal (threadid=<optimized out>, signo=6) at ./nptl/pthread_kill.c:89
#2 0x00007f28d657fcc2 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3 0x00007f28d65684ac in __GI_abort () at ./stdlib/abort.c:73
#4 0x00005596eca9bec6 in ExceptionalCondition (conditionName=conditionName@entry=0x5596ecc47230 "pgstat_bktype_io_stats_valid(bktype_shstats, MyBackendType)",
fileName=fileName@entry=0x5596ecc471d8 "../postgresql/src/backend/utils/activity/pgstat_io.c", lineNumber=lineNumber@entry=228) at ../postgresql/src/backend/utils/error/assert.c:65
#5 0x00005596ec979b6d in pgstat_io_flush_cb (nowait=<optimized out>) at ../postgresql/src/backend/utils/activity/pgstat_io.c:228
#6 0x00005596ec9758ba in pgstat_report_stat (force=force@entry=true) at ../postgresql/src/backend/utils/activity/pgstat.c:810
#7 0x00005596ec975a9c in pgstat_shutdown_hook (code=<optimized out>, arg=<optimized out>) at ../postgresql/src/backend/utils/activity/pgstat.c:646
#8 0x00005596ec92e901 in shmem_exit (code=code@entry=0) at ../postgresql/src/backend/storage/ipc/ipc.c:250
#9 0x00005596ec92ea74 in proc_exit_prepare (code=code@entry=0) at ../postgresql/src/backend/storage/ipc/ipc.c:199
#10 0x00005596ec92eb2e in proc_exit (code=code@entry=0) at ../postgresql/src/backend/storage/ipc/ipc.c:112
#11 0x00005596ec89f2f0 in BackgroundWorkerMain (startup_data=<optimized out>, startup_data_len=<optimized out>) at ../postgresql/src/backend/postmaster/bgworker.c:853
#12 0x00005596ec8a1a4d in postmaster_child_launch (child_type=child_type@entry=B_BG_WORKER, child_slot=239, startup_data=startup_data@entry=0x55972a7244d8, startup_data_len=startup_data_len@entry=1472,
client_sock=client_sock@entry=0x0) at ../postgresql/src/backend/postmaster/launch_backend.c:263
#13 0x00005596ec8a34bd in StartBackgroundWorker (rw=0x55972a7244d8) at ../postgresql/src/backend/postmaster/postmaster.c:4176
#14 maybe_start_bgworkers () at ../postgresql/src/backend/postmaster/postmaster.c:4341
#15 0x00005596ec8a4e05 in LaunchMissingBackgroundProcesses () at ../postgresql/src/backend/postmaster/postmaster.c:3416
#16 ServerLoop () at ../postgresql/src/backend/postmaster/postmaster.c:1728
#17 0x00005596ec8a7257 in PostmasterMain (argc=argc@entry=12, argv=argv@entry=0x55972a6a0ec0) at ../postgresql/src/backend/postmaster/postmaster.c:1403
#18 0x00005596ec5cbd1c in main (argc=12, argv=0x55972a6a0ec0) at ../postgresql/src/backend/main/main.c:231
--
Justin
pgsql-hackers by date: