3.5. Running pg_probackup3 on Parallel Threads (Multithreading) #
backup, restore, merge, delete, catchup, archive-push, archive-get, and validate processes can be executed on several parallel threads. This can significantly speed up pg_probackup3 operation given enough resources (CPU cores, disk, and network bandwidth).
Parallel execution is controlled by the -j/--threads, --num-write-threads, and --num-validate-threads command-line options. These options must be non-negative integers.
Note
The restore command does not support these options yet. The number of threads will match the number of segments in the backup.
Parallel restore applies only to copying data from the backup catalog to the data directory of the cluster. When Postgres Pro server is started, WAL records need to be replayed, and this cannot be done in parallel.
If these options are not specified or set to zero, pg_probackup3 defaults to the number of CPU cores. If the core count cannot be determined, a single thread will be used.
When specified, --num-write-threads and --num-validate-threads override -j.
If the requested threads exceed the system limit (e.g., from /proc/sys/kernel/threads-max), a warning will be displayed, and the system limit value will be used instead. If no limit is found, the value specified by the user will be applied.
The thread limit can also be controlled by the PG_PROBACKUP_MAX_THREADS environment variable. If the -j option is not specified, but PG_PROBACKUP_MAX_THREADS is set, the thread count is automatically determined based on available CPU cores but is capped by the PG_PROBACKUP_MAX_THREADS value.
In the PRO mode, the number of read threads must be less than the value of the max_wal_senders server parameter.
For example, to create a backup using four parallel threads, run the following command:
pg_probackup3 backup -Bbackup_dir--instance=instance_name-b FULL -j 4
The archive-push command is called by Postgres Pro as part of the archive_command parameter and archive-get — as part of restore_command. Although triggered for one WAL file, a single pg_probackup3 process can handle multiple WAL files (scanning the pg_wal directory in case of archive-push and copying from the backup catalog in case of archive-get) in parallel using the specified number of threads. You can also specify the --batch-size option to process WAL segments in batches of the specified size. For more details, refer to the archive-push and archive-get command descriptions.