Pre-Configuring Alerts

To work with alerts, you must first pre-configure them in the ppem-manager.yml manager configuration file.

You can specify the following parameters:

alerts:
  metrics:
    request_chunk_size: number_of_instance_IDs
  cleanup_grace_period: alert_cleanup_interval_if_no_data_is_received
  scheduler:
    interval: interval_for_checking_new_alerts
    initial_delay: delay_for_starting_alert_scheduler
    timeout: timeout_for_updating_alert_trigger_rules
  delayed_data:
    is_enabled: true or false
    data_delay: default_data_arrival_delay_for_all_sources
    datasource_delays:
      metrics: delay_for_metrics_arrival
      logs: delay_for_log_arrival
    max_delay: maximum_allowed_data_arrival_delay
    is_adaptive_delay: true or false
  notifier:
    num_workers: number_of_concurrent_workers
    worker_batch_size: number_of_alerts_in_one_batch
    worker_interval: interval_for_checking_new_alerts
    backoff_base: exponential_backoff_calculation_duration
    max_retries: maximum_number_of_alert_attempts
    notification_timeout: alert_timeout
    janitor_interval: janitor_worker_polling_interval
    stale_processing_timeout: stale_alert_processing_timeout
  email:
    is_enabled: true or false
    smtp:
      host: SMTP_server_hostname_or_IP
      port: SMTP_server_port
      username: username_for_SMTP_server_authentication
      password: password_for_SMTP_server_authentication
      from: alert_sender_email
      timeout: SMTP_server_connection_timeout
      use_starttls: true or false
      use_ssl: true or false
      tls:
        insecure_skip_verify: true or false
        root_ca_path: path_to_root_CA

Where:

Creating an Alert

1. In the navigation panel, go to Monitoring → Alerts.

2. In the top-right corner of the page, click Create alert.

3. Enter parameters of the new alert (parameters marked with an asterisk are required):

   • Name.

   • Severity: The alert priority.

     Possible values:

     • Low

     • Medium

     • High

   • Datasource type: The type of the metrics that will be used for the alert trigger.

     Currently, you can only use pgpro-otel-collector metrics.

   • Datasource.

   • State: The state of the alert after creation.

     Possible values:

     • Disabled

     • Enabled

   • Check interval, sec.: The time interval in seconds for verifying the data source of the alert trigger.

     Minimum value: 60.

   • Flap check, count: The number of repeated triggers required for stopping the alert.

     The value of 0 means that this limitation is disabled.

   • Notify after, sec.: The time in seconds during which the trigger must continually fire for the alert to be sent.

   • Cooldown period, sec.: The time in seconds during which the alert will not be sent after the last firing trigger.

     0 means that this limitation is disabled.

4. Click Next, and then specify additional parameters (parameters marked with an asterisk are required):

   • Metric name: The name of the metric without any additional characters that will be used for the alert trigger.

     You can use the following pgpro-otel-collector metrics from the monitoring.metrics table of the repository database:

     • postgresql.archiver.archived_count

     • postgresql.archiver.failed_count

     • postgresql.bgwriter.buffers_checkpoint

     • postgresql.bgwriter.buffers_clean

     • postgresql.bgwriter.buffers_backend

     • postgresql.bgwriter.buffers_allocated

     • postgresql.bgwriter.maxwritten_clean

     • postgresql.bgwriter.buffers_backend_fsync

     • postgresql.bgwriter.checkpoints_requested

     • postgresql.bgwriter.checkpoints_timed

     • postgresql.bgwriter.checkpoint_sync_time_milliseconds

     • postgresql.bgwriter.checkpoint_write_time_milliseconds

     • postgresql.databases.blocks_hit

     • postgresql.databases.blocks_read

     • postgresql.databases.conflicts

     • postgresql.databases.deadlocks

     • postgresql.databases.checksum_failures

     • postgresql.databases.tuples_fetched

     • postgresql.databases.tuples_returned

     • postgresql.databases.tuples_inserted

     • postgresql.databases.tuples_updated

     • postgresql.databases.tuples_deleted

     • postgresql.databases.temp_bytes

     • postgresql.databases.temp_files

     • postgresql.wal.bytes

     • postgresql.databases.rollbacks

     • system.cpu.utilization

     • system.memory.usage

     • system.paging.usage

     • postgresql.wal.records

     • postgresql.databases.commits

   • Operator • Threshold value: The alert trigger rule condition containing a logical operator and value.

     Possible logical operators:

     • = (eq)

     • > (gt)

     • >= (gte)

     • < (lt)

     • <= (lte)

     • != (neq)

     For example, if you select > and specify 0, the alert is sent when the value of the specified metric exceeds 0.

     You can add multiple alert trigger rule conditions by clicking Add.

   • Rule condition: The logical connectives for the specified alert trigger rule conditions.

     Possible values:

     • And

     • Or

     This parameter is available only if you added multiple alert trigger rule conditions.

   • Instances to check.

     Possible values:

     • Check all.

     • Select instances.

       For this value, from Instances, select the instances.

   • Notify users: The users that will receive alerts.

   • Notify groups: The user groups that will receive alerts.

   • Alert template: The template of the alert text.

     You can use the following variables in the alert text:

     • {{.Title}}: The name of the metric used for the alert trigger.

     • {{.Timestamp}}: The date and time when the alert trigger fired.

     • {{.HostName}}: The name of the host where the trigger fired.

     • {{.AgentName}}: The agent that caused the trigger to fire.

     • {{.InstanceName}}: The name of the instance where the trigger fired.

     • {{.Status}}: The status of the alert trigger.

   • Notify resolved: Specifies whether the alert is sent once the trigger is resolved.

     Possible values:

     • Enabled.

       For this value, in Resolved template, enter the template of the alert text.

       You can use the same variables in this alert text as in Alert template.

     • Disabled.

5. Click Next and then Save to confirm the selected parameters and create the alert.

Viewing Alerts

In the navigation panel, go to Monitoring → Alerts.

The table of alerts with the following columns will be displayed:

Viewing Detailed Alert Information

1. In the navigation panel, go to Monitoring → Alerts.

2. Click → View details next to the alert.

   The alert editing window with the following tabs will be displayed:

   • History: The history of trigger firing.

   • Parameters. For more information about alert parameters, refer to Creating an Alert.

3. Click Close.

Disabling and Enabling Alerts

1. In the navigation panel, go to Monitoring → Alerts.

2. Click or next to the alert.

Editing an Alert

1. In the navigation panel, go to Monitoring → Alerts.

2. Click → Edit next to the alert.

3. Edit alert parameters.

4. Click Save.

Editing Alert Recipients

1. In the navigation panel, go to Monitoring → Alerts.

2. Click → Edit recepients next to the alert.

3. Edit users and user groups that receive alerts.

4. Click Save.

Clearing Alert History

1. In the navigation panel, go to Monitoring → Alerts.

2. Click → View details next to the alert.

3. Click Clear.

Deleting an Alert

To delete an alert:

1. In the navigation panel, go to Monitoring → Alerts.

2. Click next to the alert.

3. Confirm the operation and click Delete.