With pg_probackup3, you can configure retention policy to remove redundant backups, clean up unneeded WAL files, as well as pin specific backups to ensure they are kept for the specified time, as explained in the sections below. All these actions can be combined together in any way.
By default, all backup copies created with pg_probackup3 are stored in the specified backup catalog. To save disk space, you can configure retention policy to remove redundant backup copies.
To configure retention policy, set one or more of the following variables in the pg_probackup3.conf file via set-config:
--retention-redundancy=redundancy
Specifies the number of full backup copies to keep in the backup catalog.
--retention-window=window
Defines the earliest point in time for which pg_probackup3 can complete the recovery. This option is set in the number of days from the current moment. For example, if retention-window=6, pg_probackup3 must keep at least one backup copy that is older than six days, with all the corresponding WAL files, and all the backups that follow.
If both --retention-redundancy and --retention-window options are set, both these conditions have to be taken into account when purging the backup catalog. For example, if you set --retention-redundancy=2 and --retention-window=6, pg_probackup3 has to keep two full backup copies, as well as all the backups required to ensure recoverability for the last six days:
pg_probackup3 set-config -Bbackup_dir--instance=instance_name--retention-redundancy=2 --retention-window=6
It is recommended to always keep at least two last parent full backups to avoid errors when creating incremental backups.
To clean up the backup catalog in accordance with retention policy, you have to run the retention command with retention flags, as shown below.
For example, to remove all backup copies that no longer satisfy the defined retention policy, run the following command with the --delete-expired flag:
pg_probackup3 retention -Bbackup_dir--instance=instance_name--delete-expired
If you would like to also remove the WAL files that are no longer required for any of the backups, you should also specify the --delete-wal flag:
pg_probackup3 retention -Bbackup_dir--instance=instance_name--delete-expired --delete-wal
You can also set or override the current retention policy by specifying --retention-redundancy and --retention-window options directly when running the retention command:
pg_probackup3 retention -Bbackup_dir--instance=instance_name--delete-expired --retention-window=6 --retention-redundancy=2
Since incremental backups require that their parent full backup and all the preceding incremental backups are available, if any of such backups expire, they still cannot be removed while at least one incremental backup in this chain satisfies the retention policy. To avoid keeping expired backups that are still required to restore an active incremental one, you can merge them with this backup using the --merge-expired flag when running the retention command.
Suppose you have backed up the node instance in the backup_dir directory, with the --retention-window option set to 6 and --retention-redundancy option set to 2, and you have the following backups available on February 11, 2025:
BACKUP INSTANCE 'dev', version 3 =================================================================================================================================== Instance Version ID End time Mode WAL Mode TLI Duration Data WAL Zalg Zratio Start LSN Stop LSN Status =================================================================================================================================== dev 17 full-1 2024-10-18 21:02:28+0000 FULL ARCHIVE 1 87MB - none 1.00 0/10000028 0/10000128 OK dev 17 delta-1-1 2024-11-11 00:36:01+0000 DELTA ARCHIVE 1 23MB - none 1.00 0/12000028 0/12000128 OK dev 17 delta-1-2 2024-11-15 15:43:01+0000 DELTA ARCHIVE 1 22MB - none 1.00 0/14000028 0/14000128 OK dev 17 full-2 2024-11-22 14:24:04+0000 FULL ARCHIVE 1 98MB - none 1.00 0/17000028 0/17000128 OK dev 17 delta-2-1 2024-11-23 18:10:55+0000 DELTA ARCHIVE 1 23MB - none 1.00 0/19000028 0/19000128 OK ----------------------------------------------------------retention window----------------------------------------------------------- dev 17 delta-2-2 2025-02-06 23:44:33+0000 DELTA ARCHIVE 1 33MB - none 1.00 0/1C000028 0/1C000128 OK dev 17 full-3 2025-02-08 03:31:33+0000 FULL ARCHIVE 1 120MB - none 1.00 0/1F000028 0/1F000128 OK dev 17 delta-3-1 2025-02-09 07:18:31+0000 DELTA ARCHIVE 1 23MB - none 1.00 0/21000028 0/21000128 OK dev 17 delta-3-2 2025-02-10 11:05:17+0000 DELTA ARCHIVE 1 23MB - none 1.00 0/23000028 0/23000128 OK dev 17 full-4 2025-02-11 15:00:38+0000 FULL ARCHIVE 1 1s 123MB - none 1.00 0/25000028 0/25000128 OK
If you run the retention command with the --delete-expired flag, the backups with IDs full-1, delta-1-1, and delta-1-2 will be removed as they are expired both according to the retention window and due to redundancy (the required set of full backups has already been retained). delta-1-1 and delta-1-2 will also be removed since the base full backup is expired.
Running the retention command with the --merge-expired flag will merge backups full-2 and delta-2-1 with delta-2-2. The merge will occur with delta-2-2 as it is the first non-expired delta backup, which can be merged with expired delta backups delta-2-1 and expired full backup full-2. The new full backup ID will take the value of the current timestamp.
pg_probackup3 retention -Bbackup_dir--instance=node--delete-expired --merge-expired pg_probackup3 show -Bbackup_dir
BACKUP INSTANCE 'dev', version 3 ================================================================================================================================================= Instance Version ID End time Mode WAL Mode TLI Duration Data WAL Zalg Zratio Start LSN Stop LSN Status ================================================================================================================================================= dev 17 2025-02-11-11-14-18-254 2025-02-11 11:14:18+0000 FULL ARCHIVE 1 108MB - none 1.00 0/17000028 0/19000128 OK dev 17 full-3 2025-02-08 03:31:33+0000 FULL ARCHIVE 1 120MB - none 1.00 0/1F000028 0/1F000128 OK dev 17 delta-3-1 2025-02-09 07:18:31+0000 DELTA ARCHIVE 1 23MB - none 1.00 0/21000028 0/21000128 OK dev 17 delta-3-2 2025-02-10 11:05:17+0000 DELTA ARCHIVE 1 23MB - none 1.00 0/23000028 0/23000128 OK dev 17 full-4 2025-02-11 15:00:38+0000 FULL ARCHIVE 1 1s 123MB - none 1.00 0/25000028 0/25000128 OK
The Duration field for the merged backup displays the time required for the merge.
If you need to keep certain backups longer than the established retention policy allows, you can pin them for arbitrary time. For example:
pg_probackup3 set-backup -Bbackup_dir--instance=instance_name-ibackup_id--ttl=30d
This command sets the expiration time of the specified backup to 30 days starting from the time indicated in its recovery-time attribute.
You can also explicitly set the expiration time for a backup using the --expire-time option. For example:
pg_probackup3 set-backup -Bbackup_dir--instance=instance_name-ibackup_id--expire-time="2027-04-09 18:21:32+00"
Alternatively, you can use the --ttl and --expire-time options with the backup command to pin the newly created backup:
pg_probackup3 backup -Bbackup_dir--instance=instance_name-b FULL --ttl=30d pg_probackup3 backup -Bbackup_dir--instance=instance_name-b FULL --expire-time="2027-04-09 18:21:32+00"
To check if the backup is pinned, run the show command:
pg_probackup3 show -Bbackup_dir--instance=instance_name-ibackup_id
If the backup is pinned, it has the expire-time attribute that displays its expiration time:
... recovery-time = '2024-04-09 18:21:32+00' expire-time = '2027-04-09 18:21:32+00' data-bytes = 22288792 ...
You can unpin the backup by setting the --ttl option to zero:
pg_probackup3 set-backup -Bbackup_dir--instance=instance_name-ibackup_id--ttl=0
Note
A pinned incremental backup implicitly pins all its parent backups. If you unpin such a backup later, its implicitly pinned parents will also be automatically unpinned.
When continuous WAL archiving is enabled, archived WAL segments can take a lot of disk space. Even if you delete old backup copies from time to time, the --delete-wal flag can purge only those WAL segments that do not apply to any of the remaining backups in the backup catalog. However, if point-in-time recovery is critical only for the most recent backups, you can configure WAL archive retention policy to keep WAL archive of limited depth and win back some more disk space.
Suppose you have backed up the node instance in the backup_dir directory and configured continuous WAL archiving:
pg_probackup3 show -Bbackup_dir--instance=node
BACKUP INSTANCE 'dev', version 3 ================================================================================================================================================= Instance Version ID End time Mode WAL Mode TLI Duration Data WAL Zalg Zratio Start LSN Stop LSN Status ================================================================================================================================================= dev 17 2025-02-11-15-13-36-756 2025-02-11 15:13:37+0000 FULL ARCHIVE 1 1s 38MB - none 1.00 0/17000028 0/19000128 OK dev 17 2025-02-11-14-51-12-937 2025-02-06 23:44:33+0000 DELTA ARCHIVE 1 33MB - none 1.00 0/1C000028 0/1C000128 OK dev 17 2025-02-11-14-51-33-367 2025-02-08 03:31:33+0000 FULL ARCHIVE 1 120MB - none 1.00 0/1F000028 0/1F000128 OK dev 17 2025-02-11-14-51-51-220 2025-02-09 07:18:31+0000 DELTA ARCHIVE 1 23MB - none 1.00 0/21000028 0/21000128 OK dev 17 2025-02-11-14-51-57-473 2025-02-10 11:05:17+0000 DELTA ARCHIVE 1 23MB - none 1.00 0/23000028 0/23000128 OK dev 17 2025-02-11-15-00-37-815 2025-02-11 15:00:38+0000 FULL ARCHIVE 1 1s 123MB - none 1.00 0/25000028 0/25000128 OK
You can check the state of the WAL archive by running the show command with the --archive flag:
pg_probackup3 show -B backup_dir --instance=node --archive
BACKUP INSTANCE 'dev', version 3 ====================================================================================================================== TLI Parent TLI Switchpoint Min Segno Max Segno N segments Size Zratio N backups Status ====================================================================================================================== 1 0/0 000000010000000000000001 000000010000000000000025 37 592MB 1.41 6 OK
To purge all unused WAL files (that do not apply to any of the remaining backups in the backup catalog) run the following command:
pg_probackup3 retention -B backup_dir --instance=node --delete-wal
[2025-02-11 15:23:30.422696] [14218] [128670453549440] [info] command: ./pg_probackup3 retention -B /work/backup --instance dev --delete-wal [2025-02-11 15:23:30.422738] [14218] [128670453549440] [info] execute command: 'retention', instance 'dev' [2025-02-11 15:23:30.426167] [14218] [128670453549440] [info] WAL file 000000010000000000000001 removed [2025-02-11 15:23:30.428095] [14218] [128670453549440] [info] WAL file 000000010000000000000002 removed [2025-02-11 15:23:30.429776] [14218] [128670453549440] [info] WAL file 000000010000000000000003 removed [2025-02-11 15:23:30.431838] [14218] [128670453549440] [info] WAL file 000000010000000000000004 removed [2025-02-11 15:23:30.434124] [14218] [128670453549440] [info] WAL file 000000010000000000000005 removed [2025-02-11 15:23:30.434196] [14218] [128670453549440] [info] WAL file 000000010000000000000005.00000028.backup removed [2025-02-11 15:23:30.435852] [14218] [128670453549440] [info] WAL file 000000010000000000000006 removed [2025-02-11 15:23:30.437579] [14218] [128670453549440] [info] WAL file 000000010000000000000007 removed [2025-02-11 15:23:30.441360] [14218] [128670453549440] [info] WAL file 000000010000000000000008 removed [2025-02-11 15:23:30.441815] [14218] [128670453549440] [info] WAL file 000000010000000000000008.00000028.backup removed [2025-02-11 15:23:30.444488] [14218] [128670453549440] [info] WAL file 000000010000000000000009 removed [2025-02-11 15:23:30.446902] [14218] [128670453549440] [info] WAL file 00000001000000000000000A removed [2025-02-11 15:23:30.446961] [14218] [128670453549440] [info] WAL file 00000001000000000000000A.00000028.backup removed [2025-02-11 15:23:30.448960] [14218] [128670453549440] [info] WAL file 00000001000000000000000B removed [2025-02-11 15:23:30.450991] [14218] [128670453549440] [info] WAL file 00000001000000000000000C removed [2025-02-11 15:23:30.451069] [14218] [128670453549440] [info] WAL file 00000001000000000000000C.00000028.backup removed [2025-02-11 15:23:30.453236] [14218] [128670453549440] [info] WAL file 00000001000000000000000D removed [2025-02-11 15:23:30.455291] [14218] [128670453549440] [info] WAL file 00000001000000000000000E removed [2025-02-11 15:23:30.455462] [14218] [128670453549440] [info] WAL file 00000001000000000000000E.00000028.backup removed [2025-02-11 15:23:30.458088] [14218] [128670453549440] [info] WAL file 00000001000000000000000F removed [2025-02-11 15:23:30.459755] [14218] [128670453549440] [info] WAL file 000000010000000000000010 removed [2025-02-11 15:23:30.459794] [14218] [128670453549440] [info] WAL file 000000010000000000000010.00000028.backup removed [2025-02-11 15:23:30.461135] [14218] [128670453549440] [info] WAL file 000000010000000000000011 removed [2025-02-11 15:23:30.462603] [14218] [128670453549440] [info] WAL file 000000010000000000000012 removed [2025-02-11 15:23:30.462637] [14218] [128670453549440] [info] WAL file 000000010000000000000012.00000028.backup removed [2025-02-11 15:23:30.464003] [14218] [128670453549440] [info] WAL file 000000010000000000000013 removed [2025-02-11 15:23:30.465522] [14218] [128670453549440] [info] WAL file 000000010000000000000014 removed [2025-02-11 15:23:30.465555] [14218] [128670453549440] [info] WAL file 000000010000000000000014.00000028.backup removed [2025-02-11 15:23:30.466910] [14218] [128670453549440] [info] WAL file 000000010000000000000015 removed [2025-02-11 15:23:30.468572] [14218] [128670453549440] [info] WAL file 000000010000000000000016 removed [2025-02-11 15:23:30.468600] [14218] [128670453549440] [info] 30 WAL files removed.
You can check the state of the WAL archive by running the show command with the --archive flag:
pg_probackup3 show -B backup_dir --instance=node --archive
BACKUP INSTANCE 'dev', version 3 ====================================================================================================================== TLI Parent TLI Switchpoint Min Segno Max Segno N segments Size Zratio N backups Status ====================================================================================================================== 1 0/0 000000010000000000000017 000000010000000000000025 15 240MB 1.47 6 OK