62.3. Managing Mass Instance Operations #
- 62.3.1. Creating a Mass Instance Operation
- 62.3.2. Viewing Mass Instance Operations
- 62.3.3. Viewing Detailed Mass Operation Information
- 62.3.4. Viewing Mass Instance Operation Steps
- 62.3.5. Viewing Detailed Step Information
- 62.3.6. Restarting a Mass Instance Operation Step
- 62.3.7. Canceling a Mass Instance Operation
Mass instance operations allow you to execute an operation on a group of instances at once instead of doing this on each instance separately.
Important
The mass instance operations functionality is in the beta phase. Existing limitations are specified in the corresponding instructions.
62.3.1. Creating a Mass Instance Operation #
PPEM supports the following operation types:
Applying a configuration preset.
A selected configuration preset is applied to one or more instances.
Specified WAL archiving parameters are configured on the selected instance. Continuous WAL archiving must be configured for point-in-time recovery (PITR) to work.
62.3.1.1. Creating a Mass Instance Operation for Applying a Configuration Preset #
In the navigation panel, go to Task console → Operations.
In the top-right corner of the page, click Create operation.
In Operation, enter the basic parameters of the new mass instance operation (parameters marked with an asterisk are required):
Operation: Applying configuration preset.
Configuration preset: The configuration preset that will be applied to the instances.
If a configuration preset both increases and decreases configuration parameters, split it into two separate presets: one for increasing values and one for decreasing values.
Then apply both presets individually.
Error limit: The count of errors that can occur when executing the operation on an instance. If the error limit is exceeded, the operation is canceled.
Currently, the error limit cannot be configured, its value is automatically set to
-1.Step execution timeout, s: The maximum step duration. If a step is not completed within this time, it is terminated with the Error status.
A step in the Error status can be restarted later if the operation is in the Executing status.
Default value: 60 seconds
Click Next.
In Instances, select the instances to which the configuration preset will be applied.
The configuration preset is applied to the instances in the order you select them.
Important
For nodes of a standard primary-standby or BiHA cluster, consider the following:
If a preset increases configuration parameter values, first apply it to standby/follower nodes.
If a preset decreases configuration parameter values, first apply it to primary/leader nodes.
Applying a configuration preset in the wrong order may cause standby/follower nodes to shut down.
Click Next.
In Compatibility check, review the list of instances and configure compatibility check:
Review the list of instances and the order in which the configuration preset is applied to them.
Select whether to verify compatibility between the configuration preset and the instance before applying the preset. If the compatibility check fails, the configuration preset is not applied to the instance.
The compatibility check does not guarantee that no errors will occur when applying the configuration preset.
You can select one of the following values:
For each instance
No check
Click Execute operation.
62.3.1.2. Creating a Mass Instance Operation for Configuring WAL Archiving #
To configure WAL archiving on an instance, ensure that the following prerequisites are met:
pg_probackup is installed and configured on the server that hosts the instance.
A local backup storage exists and is bound to the instance.
If not, either create a new local storage bound to the instance, or bind an existing local storage to the instance.
In the navigation panel, go to Task console → Operations.
In the top-right corner of the page, click Create operation.
In Operation, enter the basic parameters of the new mass instance operation (parameters marked with an asterisk are required):
Operation: Configuring WAL archiving.
Error limit: The count of errors that can occur when executing the operation on an instance. If the error limit is exceeded, the operation is canceled.
Currently, the error limit cannot be configured, its value is automatically set to
-1.Step execution timeout, s: The maximum step duration. If a step is not completed within this time, it is terminated with the Error status.
A step in the Error status can be restarted later if the operation is in the Executing status.
Default value: 60 seconds
Click Next.
In Parameters, enter the parameters of the new mass instance operation (parameters marked with an asterisk are required):
Archiving mode.
Possible values:
On: Archive WAL only for the selected instance.
This value can be selected for the instance with one of the following roles:
primary,standalone.Always: Archive WAL for both the selected instance and its associated instance (either the primary, cascade, or standby one).
This value can be selected for the instance with one of the following roles:
primary,standalone,standby,cascade.
Archive level: The amount of data written to the WAL.
For more information, refer to the official Postgres Pro documentation on the
wal_levelconfiguration parameter.Possible values:
replica: Data sufficient for WAL archiving is written.
logical: Data sufficient for both WAL archiving and logical decoding is written.
Instance: The instance for which to configure WAL archiving.
WAL archiving cannot be configured for the instance with
refereeormaintenancerole.Storage: The local storage of the instance.
This storage is used for WAL archives.
When selected, its core parameters are displayed under Storage parameters.
The following group of parameters controls the behavior of the
archive-pusharchive command that PPEM uses when working with pg_probackup:Number of threads: The number of threads to use when copying WAL segments.
Copying WAL segments in batches using multiple threads may speed up archiving.
Value type:
positive integer. Default value:1.WAL files batch size: The number of WAL segments to copy simultaneously.
Copying WAL segments in batches may speed up archiving.
Value type:
positive integer. Default value:1.Wait time, sec: The time in seconds to wait before discarding an existing temporary file.
Each WAL file is copied to a temporary file with the
.partsuffix. This parameter controls when to discard the existing temporary file.Value type:
positive integer. Default value:empty. When empty, the wait time is300seconds.Rename status files in archive_status: Rename status files in the
archive_statusdirectory.Value type:
boolean. Default value:false/unchecked (the files are not renamed).Do not sync copied WAL files to disk: Do not execute
fsync()after WAL files are written to the disk.Value type:
boolean. Default value:false/unchecked (the files are synced usingfsync()).Warning
While skipping syncing of WAL files may speed up archiving process, use this parameter with caution.
If the parameter is used, the WAL archive may be corrupted in case of an operating system or hardware crash during WAL archiving.
Allow overwriting archived WAL files: Overwrite existing archived WAL files if their newer copies appear.
By default, if the specified subdirectory of the backup catalog already contains WAL files, and these files need to be replaced with newer copies, the archiving operation is aborted. This parameter allows overwriting existing files.
If the file to replace has not changed, this file is skipped regardless of the parameter value.
Value type:
boolean. Default value:false/unchecked (the files are not overwritten).Compression algorithm.
Possible values:
Disabled: no compression is used for WAL files.
zlib.
lz4.
zstd.
Compression level. Depends on the selected compression algorithm.
Possible values:
For zlib: 0, 1–9.
For lz4: 0, 1–12.
zstd: -131072–22.
Value type:
integerwithin the ranges above. Default value:0(the1compression level is set).
PPEM displays a message at the bottom of the parameter list if the instance must be restarted to apply the parameters.
Click Next.
In Confirmation, review the WAL archiving parameters.
Click Create and start operation.
62.3.2. Viewing Mass Instance Operations #
In the navigation panel, go to Task console → Operations.
The table of mass instance operations with the following columns will be displayed:
Name.
Status.
A progress bar is displayed next to this status. Use it to monitor the progress of the mass instance operation.
Possible values:
Status: The status of the mass instance operation and the number of performed steps.
A progress bar is displayed next to this status. Use it to monitor the progress of the operation.
Possible values:
Ready for execution: The operation is scheduled.
Executing: The operation is being executed.
Error: An error occurred when executing the operation.
Canceling: The operation is being canceled.
Canceled: The operation was canceled.
Completed: The operation was executed.
Owner: The user that executed the mass instance operation.
Last execution: The date and time when the mass instance operation started executing.
Duration: The duration of the mass instance operation.
This column includes additional information:
Completed: The date and time when the mass instance operation stopped executing.
Timeout: The configured step execution timeout.
Actions.
For more information about available actions, refer to other instructions in this section.
62.3.3. Viewing Detailed Mass Operation Information #
In the navigation panel, go to Task console → Operations.
Click
next to the mass operation. The window with detailed information about the mass instance operation will open. The displayed parameters depend on the type of the mass instance operation and are described below:
62.3.3.1. Applying Configuration Preset Operation #
The following detailed information is displayed:
Operation: Applying configuration preset.
Status: The status of the mass instance operation and the number of performed steps.
A progress bar is displayed next to this status. Use it to monitor the progress of the operation.
Possible values:
Ready for execution: The operation is scheduled.
Executing: The operation is being executed.
Error: An error occurred when executing the operation.
Canceling: The operation is being canceled.
Canceled: The operation was canceled.
Completed: The operation was executed.
Configuration preset: The configuration preset being applied to the instances.
Step execution timeout, sec: The configured step execution timeout.
Compatibility check: Specifies whether compatibility between the configuration preset and the instance is verified before applying the preset. If the compatibility check fails, the configuration preset is not applied to the instance.
The compatibility check does not guarantee that no errors will occur when applying the configuration preset.
Possible values:
For each instance
No check
Selected instances: The list of instances and the order in which the configuration preset is applied to them.
62.3.3.2. Configuring WAL Archiving Operation #
The following detailed information is displayed:
Operation: Configuring WAL archiving.
Instance: The name of the instance where the archiving is configured.
Status: The status of the mass instance operation and the number of performed steps.
A progress bar is displayed next to this status. Use it to monitor the progress of the operation.
Possible values:
Ready for execution: The operation is scheduled.
Executing: The operation is being executed.
Error: An error occurred when executing the operation.
Canceling: The operation is being canceled.
Canceled: The operation was canceled.
Completed: The operation was executed.
Step execution timeout, sec: The configured step execution timeout.
62.3.4. Viewing Mass Instance Operation Steps #
Every mass instance operation consists of one or more steps. You can view the step list to monitor an operation progress and to manage individual steps.
To view steps of a mass instance operation:
In the navigation panel, go to Task console → Operations.
In Status, click the status of the mass instance operation.
The table of steps being executed with the mass instance operation will be displayed.
The block with the summary data is located above the table, displaying the following information:
The status of the mass instance operation.
The number of errors, if any.
The table has the following columns:
Step number: The step sequence number.
Name: The descriptive step name, which is generated automatically.
Status: The step status.
Possible values:
Not started: The step execution did not start.
Executing: The step is being executed.
Error: An error occurred when executing the step.
Canceling: The step execution is being canceled.
Canceled: The step execution is canceled.
Skipped: The step execution is skipped.
Completed: The step execution is completed.
Instance: The instance where the step is being executed.
Start: The date and time the step execution started.
Duration: The step execution duration.
This column includes additional information.
Completed at: The date and time the step execution ended.
Actions.
For more information about available actions, refer to other instructions in this section.
62.3.5. Viewing Detailed Step Information #
In the navigation panel, go to Task console → Operations.
In Status, click the status of the mass instance operation.
Click
next to the step. The window with detailed information about the step will open. The displayed parameters depend on the type of the mass instance operation and are described below:
62.3.5.1. Applying Configuration Preset Operation #
The following detailed information is displayed:
Status: The step status.
Possible values:
Not started: The step execution did not start.
Executing: The step is being executed.
Error: An error occurred when executing the step.
Canceling: The step execution is being canceled.
Canceled: The step execution is canceled.
Skipped: The step execution is skipped.
Completed: The step execution is completed.
Start: The date and time the step execution started.
End: The date and time the step execution ended.
Duration: The step execution duration.
Cluster: The cluster where the step is being executed.
Instance: The instance where the step is being executed.
Role: The role of the instance.
Preset: The configuration preset being applied to the instance.
Instance ID.
Preset ID.
Compatibility check: Specifies whether compatibility between the configuration preset and the instance is verified before applying the preset. If the compatibility check fails, the configuration preset is not applied to the instance.
The compatibility check does not guarantee that no errors will occur when applying the configuration preset.
Possible values:
For each instance
No check
62.3.5.2. Configuring WAL Archiving Operation #
The following detailed information is displayed:
Status: The step status.
Possible values:
Not started: The step execution did not start.
Executing: The step is being executed.
Error: An error occurred when executing the step.
Canceling: The step execution is being canceled.
Canceled: The step execution is canceled.
Skipped: The step execution is skipped.
Completed: The step execution is completed.
Start: The date and time the step execution started.
End: The date and time the step execution ended.
Duration: The step execution duration.
Cluster: The cluster where the step is being executed.
Instance: The instance where the step is being executed.
The parameters that were specified when creating the mass instance operation.
These parameters are displayed only for the
wal_archive_config_setupstep.
62.3.6. Restarting a Mass Instance Operation Step #
When you restart a step, its execution begins again.
A step cannot be restarted if any of the following conditions is met:
The operation that includes the step is not completed (not in the Completed or Error status).
The step is not in the Not started or Error status.
The previous step, if any, is not in the Completed status.
To restart a mass instance operation step:
In the navigation panel, go to Task console → Operations.
In Status, click the status of the mass instance operation.
Click
next to the step. Click Restart.
62.3.7. Canceling a Mass Instance Operation #
When you cancel a mass instance operation, its execution stops, but already executed steps are not reverted.
To cancel a mass instance operation:
In the navigation panel, go to Task console → Operations.
Click
next to the mass operation. Alternatively, on the mass operation steps page, in the top-right corner of the page, click Cancel operation.
Click Cancel.