Re: Requested WAL segment xxx has already been removed - Mailing list pgsql-hackers

From wenhui qiu
Subject Re: Requested WAL segment xxx has already been removed
Date
Msg-id CAGjGUALfTQz4aCfN38ZRtiPmtzbdmacAm=Pse4aLbxYX4j0Cjw@mail.gmail.com
Whole thread Raw
In response to Re: Requested WAL segment xxx has already been removed  (Japin Li <japinli@hotmail.com>)
List pgsql-hackers
HI Japin 
   Thank you for your working on this.It is useful ,when a standby node has hardware issue repaired ,wal log usually has been archived.The wal_keep_size parameter is difficult to estimate accurately, as hardware repair or replacement times are often unpredictable. If the machine can be fixed in a few days, the archived WAL files are likely still available in the archive directory.One small regret is that postgresql currently lacks a speed limit for sending wal logs.

Thanks

On Tue, Jul 15, 2025 at 2:11 PM Japin Li <japinli@hotmail.com> wrote:
On Mon, 14 Jul 2025 at 20:33, Fujii Masao <masao.fujii@oss.nttdata.com> wrote:
> On 2025/07/14 17:08, Japin Li wrote:
>> Hi all,
>> I recently hit an error with our streaming replication setup:
>>    2025-07-14 11:52:59.361
>> CST,"replicator","",728458,"10.9.9.74:35724",68747f1b.b1d8a,1,"START_REPLICATION",2025-07-14
>> 11:52:59 CST,3/0,0,ERROR,58P01,"requested WAL segment
>> 00000001000000000000000C has already been
>> removed",,,,,,"START_REPLICATION 0/C000000 TIMELINE
>> 1",,,"standby","walsender",,0
>> It appears the requested WAL segment 00000001000000000000000C had
>> already been
>> archived, and I confirmed its presence in the archive directory. However, when
>> the standby tried to request this file, the primary only searched for it in
>> pg_wal and didn't check the archive directory. I had to manually copy the
>> segment into pg_wal to get streaming replication working again.
>> My question is: Can we make the primary automatically search the
>> archive if
>> restore_command is set?
>> I found that Fujii Masao also requested this feature [1], but it
>> seems there
>> wasn't a consensus.
>
> Yeah, I still like this idea. It's useful, for example, when we want to
> temporarily retain WAL files, such as during planned standby maintenance,
> to avoid "requested WAL segment ... removed." error.
>
> Using a replication slot is one way to retain WAL files in pg_wal,
> but it requires the pg_wal directory to be large enough to hold all
> WAL generated during that time, which isn't always practical.
>

Agreed.  Here is a patch that fixes this.

--
Regards,
Japin Li

pgsql-hackers by date:

Previous
From: Etsuro Fujita
Date:
Subject: Odd behavior of statement triggers with transition tables on partitions
Next
From: "Hayato Kuroda (Fujitsu)"
Date:
Subject: RE: pg_logical_slot_get_changes waits continously for a partial WAL record spanning across 2 pages