Request a Demo
3.9. Cloning and Synchronizing a Postgres Pro Instance
Prev UpChapter 3. Use CasesHome Next

3.9. Cloning and Synchronizing a Postgres Pro Instance #

3.9.1. Considerations and Limitations of Using catchup
3.9.2. Using catchup
3.9.3. Troubleshooting

pg_probackup3 can create a copy of a Postgres Pro instance directly, without using the backup catalog. To do this, you can run the catchup command. It can be useful in the following cases:

  • To add a new standby server.

    If the data directory of the destination instance is empty, the catchup command works similarly to the backup command in PRO backup data source mode, but it can be faster if run in parallel mode.

    Note

    catchup operations are currently supported only in PRO and BASE modes, which do not require direct access to PGDATA.

  • To have a fallen-behind standby server “catch up” with the primary.

    Under write-intensive load, replicas may fail to replay WAL fast enough to keep up with the primary and hence may lag behind. A usual solution to create a new replica and switch to it requires a lot of extra space and data transfer. The catchup command allows you to update an existing replica much faster by fetching differences from the primary.

3.9.1. Considerations and Limitations of Using catchup #

Note the following specifics and limitations of catchup operations:

  • The backup catalog is not required for catchup operations.

  • DDL commands CREATE TABLESPACE/DROP TABLESPACE cannot be run simultaneously with catchup.

  • catchup takes configuration files, such as postgresql.conf, postgresql.auto.conf, or pg_hba.conf, from the source server and overwrites them on the target server. The --exclude-path option allows keeping the configuration files intact.

  • catchup always operates in STREAM WAL delivery mode, streaming all necessary WAL files from the server via the replication protocol.

  • catchup does not support:

    • DIRECT data source mode

    • copying external directories

    • remote mode

  • PRO data source mode requires the pgpro_bindump module to be installed and configured on the source server.

  • BASE data source mode does not support PTRACK mode.

  • The user must have the privileges of the pg_read_all_settings role to run catchup in BASE data source mode: 

    GRANT pg_read_all_settings TO backup;

  • DELTA and PTRACK modes should be used only when the following conditions are met:

    • The target server was created using pg_probackup3 catchup -b FULL or pg_probackup3 restore.

    • The target server has never been started as a primary (only as a replica).

    • The pg_rewind, pg_resetwal, or pg_upgrade commands have not been executed on the target server.

    • The pg_control file of the target server has not been manually modified.

    For more information about possible synchronization issues and ways to resolve them, refer to the Troubleshooting section.

3.9.2. Using catchup #

To prepare for cloning/synchronizing a Postgres Pro instance, set up the source server as follows:

Before cloning/synchronizing a Postgres Pro instance, ensure that the source server is running and accepting connections. To clone/synchronize a Postgres Pro instance, on the server with the destination instance, run the catchup command as follows: 

pg_probackup3 catchup -b catchup_mode --destination-pgdata=path [-s | --backup-source=catchup_source] [connection_options]

catchup can run in one of the following synchronization modes set by the -b/--backup-mode option:

  • FULL: Creates a full copy of the Postgres Pro instance. The data directory of the target instance must be empty for this mode. In FULL mode, all data files are copied from the source instance regardless of their state on the target server. This mode should be used in the following cases:

    • When creating a new standby server.

    • After operations that change the LSN value in pg_control.

    • After starting the target server as a primary.

  • DELTA: Reads all data files in the data directory and creates an incremental copy for pages that have changed since the target server was shut down. pg_probackup3 reads the position of the target instance from pg_control and copies all changed blocks from the source instance that are newer than this position by LSN.

  • PTRACK: Tracking page changes on the fly, only reads and copies pages that have changed since the point of divergence of the source and target instances. PTRACK mode does not require scanning all data files; instead, it uses information about changed blocks from the PTRACK bitmap.

You can explicitly specify the data source mode for synchronization using the -s/--backup-source option. Since catchup operations do not yet support DIRECT, -s can either take the pro (PRO mode) or base (BASE mode) value.

If catchup_source is not specified, pg_probackup3 defaults to PRO mode.

Important

If the default PRO mode is unavailable (pgpro_bindump is not configured), the utility falls back to BASE mode. In this case, FULL catchup operation will continue normally in BASE mode while DELTA and PTRACKcatchup operations will terminate with an error.

You can use connection_options to specify the connection to the source database cluster.

If the source database cluster contains tablespaces that must be located in different directories, additionally specify the -T/--tablespace-mapping option: 

pg_probackup3 catchup -b catchup_mode --destination-pgdata=path -T old_dir=new_dir

Note

Note the syntax for multiple tablespace mappings: instead of multiple -T flags, use a single flag with colon-separated mapping pairs. For example: 

-T old_dir1=new_dir1:old_dir2=new_dir2

To run the catchup command on parallel threads, specify the number of threads with the -j/--threads or --num-write-threads and --num-validate-threads options: 

pg_probackup3 catchup -b catchup_mode --destination-pgdata=path --threads=num_threads

3.9.3. Troubleshooting #

Important

Using DELTA or PTRACK modes after executing pg_rewind, pg_resetwal, pg_upgrade, or starting the target server as a primary may lead to synchronization issues and incorrect results. It is recommended to use FULL mode in such cases.

These operations change the LSN in the pg_control file of the target server. pg_probackup3 automatically checks that the LSN of the source server is not less than the LSN of the target server, and if this check fails, synchronization is aborted with the error “Current START LSN source_lsn is lower than expected START LSN dest_lsn. It may indicate that we are trying to backup PostgreSQL instance from the past.”.

However, the utility cannot detect violations if the source server advances in LSN after pg_control is modified. For example, after a failover, when the old primary becomes a replica via pg_rewind, the source server may continue processing transactions before the catchup operation starts. If the source server's LSN exceeds the modified LSN of the target server, the check passes and pg_probackup3 performs synchronization in DELTA or PTRACK mode but only copies blocks with LSN greater than or equal to the new (modified) value from pg_control, which may lead to data loss or inconsistency.

Example of an incorrect command sequence: 

pg_rewind --target-pgdata=/path/to/replica --source-server='host=master'
pg_probackup3 catchup -b DELTA ...

The correct command sequence: 

pg_rewind --target-pgdata=/path/to/replica --source-server='host=master'
pg_probackup3 catchup -b FULL ...

Similarly, after starting the target server as a primary, only FULL mode should be used.

When attempting to perform synchronization in DELTA or PTRACK mode, the utility checks for the absence of the backup_label file in the target server's data directory. If this file exists, synchronization is aborted with the error “Destination directory contains backup_label file”.

The presence of the backup_label file means that the target server is in an inconsistent state after an incomplete recovery from a backup. In this case, the contents of the pg_control file do not match the actual state of the data on disk.

Warning

Under no circumstances should you manually delete the backup_label file to bypass this check. Doing so will result in the use of an incorrect LSN from pg_control and guaranteed data loss.

To resolve the issue, follow these steps:

  1. Start the target server as a replica (WAL files must be accessible via streaming from the primary or from the archive).

  2. Wait for the startup process to complete, which will apply all WAL files, finish data synchronization, and bring pg_control into compliance with the actual state.

  3. Stop the server.

You can now safely use catchup in DELTA or PTRACK mode.

Prev Up Next
3.8. Configuring Retention Policy Home 3.10. More Examples
epub pdf pdf