Request a Demo
Home   >   mailing lists

Re: Making WAL archiving faster — multi-file support and async ideas - Mailing list pgsql-hackers

From Alyona Vinter
Subject Re: Making WAL archiving faster — multi-file support and async ideas
Date
Msg-id CAGWv16+8LMRpWzhgf1FDGLyruyZG+wbVmDOJHDO1B6epoF2agg@mail.gmail.com
Whole thread Raw
In response to Making WAL archiving faster — multi-file support and async ideas  (Stepan Neretin <slpmcf@gmail.com>)
Responses Re: Making WAL archiving faster — multi-file support and async ideas
List pgsql-hackers
Tree view
Hi,
I have some concerns about the parallel archiver due to the requirement for sequential WAL archiving. The fundamental rule is that WAL segments must be archived in strict sequential order for a successful restore. Consider a scenario where PostgreSQL has segments 1, 2, 3, and 4 ready, and the parallel archiver successfully copies segments 1, 2, and 4, but misses segment 3. The user might be unaware of the gap and could attempt a restore using the incomplete archive. While we hope this would cause a clear error during recovery, there is a risk that partial application of non-sequential segments might lead to silent corruption or other unforeseen issues.
 
As far as I know, tools like cp naturally process files in the order they are received. This ensures that files are processed correctly, and it seems easier and more reliable to build on this concept than to introduce a new parallel paradigm. Given that the archiver already uses a priority queue, this should not be difficult to implement.

Thanks for considering my feedback.
--
Best regards,
Alyona Vinter

On Mon, 25 Aug 2025 at 11:00, Stepan Neretin <slpmcf@gmail.com> wrote:

Hi hackers,

We’ve been thinking about how to make WAL archiving faster.

This topic was previously discussed in [1], and we’ve taken a first step by implementing the attached patch, which adds support for archiving multiple WAL files in one go.

The idea is straightforward: instead of invoking the archive command or callback once per WAL file, we allow passing a batch of files. The patch introduces support for new placeholders:

  • %F – list of WAL file names

  • %P – list of their full paths

  • %N – number of files

Since PostgreSQL already reads multiple files into memory and caches them before archiving, this change avoids repeated fork() calls and reduces overhead in high-throughput setups.

Of course, there are trade-offs. After discussing with Andrey Borodin, we noted that if even one file in the batch fails to archive, we currently have to retry the whole batch. While it’s technically possible to return a list of successfully archived files, that would complicate the API and introduce messy edge cases.

So we’re also exploring a more flexible idea: an asynchronous archiver mode.

The idea is to have PostgreSQL write WAL file names (marked .ready) into a FIFO or pipe, and let an archive process or library asynchronously consume and archive them. It would send back confirmations (or failures) through another pipe, allowing PostgreSQL to retry failed files as needed. This could decouple archiving from the archiver loop and open the door to more efficient and parallel implementations.

We’d appreciate feedback on both directions:

  • Do you think the idea in the attached patch — batching WAL files for archiving — is viable? Is it something worth pursuing?

  • What do you think about the async archiver concept? Would it fit PostgreSQL’s architecture and operational expectations?

Thanks,
Stepan Neretin

[1] https://www.postgresql.org/message-id/flat/BC335D75-105B-403F-9473-976C8BBC32E3%40yandex-team.ru#d45caa9d1075734567164f73371baf00

pgsql-hackers by date:

Previous
From: Ashutosh Bapat
Date:
Subject: Re: Report reorder buffer size
Next
From: Chao Li
Date:
Subject: Re: Fixes a trivial bug in dumped parse/query/plan trees