Hello.
(Mmm. Sorry for annoying, but added some persons in Cc:)
This is the same issue with what is discussed in [1] and recently
reported by [2].
[1] https://www.postgresql.org/message-id/E63E5670-6CC3-4B09-9686-A77CF94FE4A8%40amazon.com
[2] https://www.postgresql.org/message-id/3f9c466d-d143-472c-a961-66406172af96.mengjuan.cmj@alibaba-inc.com
At Thu, 25 Mar 2021 00:23:52 +0000, "egashira.yusuke@fujitsu.com" <egashira.yusuke@fujitsu.com> wrote in
> > The replication between "NODE-A" and "NODE-B" is synchronous replication,
> > and between "NODE-B" and "NODE-C" is asynchronous.
> >
> > "NODE-A" <-[synchronous]-> "NODE-B" <-[non-synchronous]-> "NODE-C"
> >
> > When the primary server "NODE-A" crashed due to full WAL storage and
> > "NODE-B" promoted, the downstream standby server "NODE-C" aborted with
> > following messages.
> >
> > 2021-03-11 11:26:28.470 JST [85228] LOG: invalid contrecord length 26 at
> > 0/5FFFFF0
> > 2021-03-11 11:26:28.470 JST [85232] FATAL: terminating walreceiver process
> > due to administrator command
> > 2021-03-11 11:26:28.470 JST [85228] PANIC: could not open file
> > "pg_wal/000000020000000000000005": No such file or directory
> > 2021-03-11 11:26:28.492 JST [85260] LOG: started streaming WAL from primary
> > at 0/5000000 on timeline 2
> > 2021-03-11 11:26:29.260 JST [85227] LOG: startup process (PID 85228) was
> > terminated by signal 6: Aborted
>
> I would like to clarify the conditions under which this "abort" occurred to explain to the customer.
>
> By the result of pg_waldump, I think that the conditions are followings.
>
> 1) A partially written (across the following segment files) WAL record is recorded at the end of the WAL segment
file.and
> 2) The WAL segment file of 1) is the last WAL segment file that standby server received, and
> 3) The standby server promoted.
>
> I think that the above conditions will be met only when the primary server crashed due to full WAL storage.
>
> Is my idea correct?
The diagnosis looks correct to me, but the cause of the crash is
irrelevant. A disk full just makes the crash hit accurately on the
vital point.
--
Kyotaro Horiguchi
NTT Open Source Software Center
