Duplicate history file? - Mailing list pgsql-hackers

From Kyotaro Horiguchi
Subject Duplicate history file?
Date
Msg-id 20210531.165825.921389284096975508.horikyota.ntt@gmail.com
Whole thread Raw
In response to Re: Race condition in recovery?  (Tatsuro Yamada <tatsuro.yamada.tf@nttcom.co.jp>)
Responses Re: Duplicate history file?
Re: Duplicate history file?
List pgsql-hackers
So, I started a thread for this topic diverged from the following
thread.

https://www.postgresql.org/message-id/4698027d-5c0d-098f-9a8e-8cf09e36a555@nttcom.co.jp_1

At Mon, 31 May 2021 11:52:05 +0900, Tatsuro Yamada <tatsuro.yamada.tf@nttcom.co.jp> wrote in 
> Since the above behavior is different from the behavior of the
> test command in the following example in postgresql.conf, I think
> we should write a note about this example.
> 
> # e.g. 'test ! -f /mnt/server/archivedir/%f && cp %p
> # /mnt/server/archivedir/%f'
>
> Let me describe the problem we faced.
> - When archive_mode=always, archive_command is (sometimes) executed
>   in a situation where the history file already exists on the standby
>   side.
> 
> - In this case, if "test ! -f" is written in the archive_command of
>   postgresql.conf on the standby side, the command will keep failing.
> 
>   Note that this problem does not occur when archive_mode=on.
> 
> So, what should we do for the user? I think we should put some notes
> in postgresql.conf or in the documentation. For example, something
> like this:

I'm not sure about the exact configuration you have in mind, but that
would happen on the cascaded standby in the case where the upstream
promotes. In this case, the history file for the new timeline is
archived twice.  walreceiver triggers archiving of the new history
file at the time of the promotion, then startup does the same when it
restores the file from archive.  Is it what you complained about?

The same workaround using the alternative archive script works for the
case.

We could check pg_wal before fetching archive, however, archiving is
not controlled so strictly that duplicate archiving never happens and
I think we choose possible duplicate archiving than having holes in
archive. (so we suggest the "test ! -f" script)

> ====
> Note: If you use archive_mode=always, the archive_command on the
> standby side should not be used "test ! -f".
> ====

It could be one workaround. However, I would suggest not to overwrite
existing files (with a file with different content) to protect archive
from corruption.

We might need to write that in the documentation...

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



pgsql-hackers by date:

Previous
From: Andrey Borodin
Date:
Subject: Re: Different compression methods for FPI
Next
From: Emre Hasegeli
Date:
Subject: postgres_fdw: Handle boolean comparison predicates