This is a follow up to
https://www.postgresql.org/message-id/CANQ55Tsoa6%3Dvk2YkeVUN7qO-2YdqJf_AMVQxqsVTYJm0qqQQuw%40mail.gmail.com, which I am at the same company as the original poster.
Our architecture is similar but all of the servers are now on ZFS now and Postgres 13.8 with Ubuntu 18.04+ and still doing streaming replication, all with ECC memory and 26-64 cores with 192gb ram+ on top of a ZPOOL made out of NVMe PCIe SSDs.
A101 (primary) -> A201 (replica) -> B101(primary) -> B201 (replica).
We are seeing this error occur about once per week (across all postgres clusters/chains). It is the same pattern we have been seeing for a number of years now.
Possibly relevant configuration options:
wal_init_zero=on
The last time this occurred I grabbed the good WAL file from the parent, and the corrupted WAL file from the descendant. Comparing them showed no differences until 12Mb in where the "good" WAL file continued, and the "bad" WAL file was zeros until the end of the file.
When the "good" WAL file is copied from the parent and to the descendant, replication resumes and the descendant becomes a healthy replica once again.
I am almost done on a tool that will tail the postgresql journal and walk up the chain to the primary and get the WAL file name for the LSN in the error message, then sync and overwrite the bad WAL file from the parent, but I hope I can track down the actual bug here, rather than relying upon a process to fix this after the fact.
--
John Bolliger