Re: toast table corrupted by vacuum - missing chunk number 0 for toast value - Mailing list pgsql-hackers

From Pavel Stehule
Subject Re: toast table corrupted by vacuum - missing chunk number 0 for toast value
Date
Msg-id CAFj8pRALRFQ=jsenN+0V4NeOOxgbgLy43A=a8wvLbPw7yAQEvQ@mail.gmail.com
Whole thread
In response to toast table corrupted by vacuum - missing chunk number 0 for toast value  (Pavel Stehule <pavel.stehule@gmail.com>)
Responses Re: glob support in extension_control_path/dynamic_library_path?
List pgsql-hackers


pá 21. 8. 2026 v 19:13 odesílatel Tomas Vondra <tomas@vondra.me> napsal:


On 8/21/26 17:47, Pavel Stehule wrote:
> Hi
>
> pá 21. 8. 2026 v 12:31 odesílatel Andrey Borodin <x4mmm@yandex-team.ru
> <mailto:x4mmm@yandex-team.ru>> napsal:
>
>     Hi Pavel,
>
>     On Thu, Aug 20, 2026 at 9:56 PM Pavel Stehule wrote:
>     > We didn't find any way how this issue can be reproduced. We manually
>     > executed VACUUM on the database that was restored to the time close
>     > before this issue - and the data was not broken - but now, I am not
>     > sure what was the minor version that we used for this test.
>
>     I wonder whether the missing TOAST data could be a secondary symptom,
>     rather than evidence that VACUUM removed TOAST belonging to the current
>     row version.
>
>
> I have dumped pages
>
> https://www.postgresql.org/message-id/attachment/199738/
> investigation.txt <https://www.postgresql.org/message-id/
> attachment/199738/investigation.txt>
>
> xmin, xmax was same on main table, and toast table. These rows was not
> updated.
>

Thanks. Interesting. It's weird the TOAST rows have XID 2779428784 and
flags {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}, yet the
VACUUM apparently decided all of them are dead. Judging by:

> rmgr: Heap2       len (rec/tot):     59/    99, tx:          0, lsn:
> D6E/8A8A6AD0, prev D6E/8A8A6A68, desc: PRUNE latestRemovedXid 0
> nredirected 0 ndead 4, blkref #0: rel 1663/4267629/16352341 blk
17880358 FPW

I mean, ndead=4 seems to suggest that. But how could that be with
HEAP_XMIN_COMMITTED?

I wonder if the FPW from the WAL matches what you saw as corrupted, or
if it's correct (it should, given the ndead=4). But maybe try dumping it
using pg_waldump - although, 14 doesn't have that :-(.

What happened with the XID 2779428784? Did it commit sometime right
before the vacuum, or what? Judging by the page LSN, it must have been
pretty far back.

What does the clog say about this transaction? I think it should be:
pg_xact/0A5A, page 21, byte offset 175596, bits 0-1, so

$ od -An -tx1 -j 175596 -N 1 $PGDATA/pg_xact/0A5A

Also, what does the RUNNING_XACTS right before/after vacuum say?

Unfortunately these backups was overwritten month ago

Regards

Pavel
 

--
Tomas Vondra

pgsql-hackers by date:

Previous
From: "贾明伟"
Date:
Subject: Subject: [RFC] Umbra: Reducing Full-Page Write Amplification through Physical Page Remapping
Next
From: "cca5507"
Date:
Subject: Walreceiver create temp slot more than once when timeline switch