We have addressed the O(n^2) problem which involves directory scan for archiving individual WAL files by maintaining a WAL counter to identify the next WAL file in a sequence.
WAL archiver scans the status directory to identify the next WAL file which needs to be archived. This directory scan can be minimized by maintaining the log segment number of the current file which is being archived
and incrementing it by '1' to get the next WAL file in a sequence. Archiver
can check the availability of the next file in status directory and in case if the
file is not available then it should fall-back to directory scan to get the oldest
WAL file.
Please find attached patch v1.
Thanks,
Dipesh
On Fri, May 7, 2021 at 1:31 AM Andres Freund <andres@anarazel.de> wrote:
Hi,
On 2021-05-06 21:23:36 +0200, Hannu Krosing wrote: > How are you envisioning the shared-memory signaling should work in the > original sample case, where the archiver had been failing for half a > year ?
If we leave history files and gaps in the .ready sequence aside for a second, we really only need an LSN or segment number describing the current "archive position". Then we can iterate over the segments between the "archive position" and the flush position (which we already know). Even if we needed to keep statting .ready/.done files (to handle gaps due to archive command mucking around with .ready/done), it'd still be a lot cheaper than what we do today. It probably would even still be cheaper if we just statted all potentially relevant timeline history files all the time to send them first.
> Or should we perhaps have a system table for ready-to-archive WAL > files to get around limitation sof file system to return just the > needed files with ORDER BY ... LIMIT as we already know how to make > lookups in database fast ?
Archiving needs to work on a standby so that doesn't seem like an option.
