Limiting WAL retained for archiving, like max_slot_wal_keep_size - Mailing list pgsql-hackers

From Joao Detomini
Subject Limiting WAL retained for archiving, like max_slot_wal_keep_size
Date
Msg-id CABH8dKwtgJ9+LD73dr+0e=jE0k0qiQzrFq=7VWE-Li90jCkpKA@mail.gmail.com
Whole thread
Responses Re: Limiting WAL retained for archiving, like max_slot_wal_keep_size
List pgsql-hackers
Hi,

While looking at barman-wal-archive we ran into something that seems
to be missing in the server.  If archiving breaks or can't keep up,
the unarchived WAL just piles up in pg_wal until the disk fills, and
max_slot_wal_keep_size doesn't help because it only covers
replication slots.  We considered making the archive command drop WAL
once its queue gets too big, but then nothing tells the server, or
anyone else, that a segment was skipped on purpose, so PITR breaks
without a trace.  Returning true from an archive_library for a segment
it didn't archive has the same problem, since the callback only knows
how to say "done" or "try again".

What do people think about a setting like max_slot_wal_keep_size, but
for archiving?  Off by default, and once it's exceeded the server
would let the oldest unarchived segments go and record that it did,
maybe in pg_stat_archiver, so tools can notice the gap and take a new
base backup.  We haven't written anything yet and wanted to ask first
whether this is something you'd want in core.

Thanks,
João Marcelo

pgsql-hackers by date:

Previous
From: Joao Detomini
Date:
Subject: Re: pg_resetwal: refuse to run when backup_label exists
Next
From: Rui Zhao
Date:
Subject: Re: Serverside SNI support in libpq