mamonsu

mamonsu — a monitoring agent for collecting Postgres Pro and system metrics.

Synopsis

mamonsu agent [agent_action]

mamonsu bootstrap -M mamonsu_user -d mamonsu_database [connection_options]

mamonsu export {template | config} filename [export_options]

mamonsu report [report_options]

mamonsu tune [tuning_options]

mamonsu zabbix {template | host | hostgroup} server_action

mamonsu zabbix item {error | lastvalue | lastclock} host_id

Description

mamonsu is a monitoring agent for collecting Postgres Pro and system metrics that can be visualized on the Zabbix server. Unlike the native Zabbix Agent configured to collect Postgres Pro metrics, mamonsu uses a single database connection, which allows to minimize performance impact on the monitored system.

The mamonsu agent includes the following components:

  • A supervisor process that monitors database and system activity.

  • Plugins that specify which Postgres Pro and system metrics to collect.

  • Customizable configuration and template files that define which plugins to use and how to visualize the collected data.

mamonsu is an active agent, which means that it sends the data to the Zabbix server once it is collected. Pre-configured to monitor multiple Postgres Pro and system metrics out of the box, mamonsu can be extended with your own custom plugins to track other metrics critical for your system.

mamonsu also provides the command-line interface for updating some Zabbix server settings, as well as getting an overview of the monitored system configuration and tuning Postgres Pro and system settings on the fly. You can find the list of all mamonsu commands in the section called “Command-Line Reference”.

Installation and Setup

To use mamonsu, you must create a Zabbix account and set up a Zabbix server as explained in Zabbix documentation. Naturally, you must also have a Postgres Pro instance up and running if you are going to monitor Postgres Pro metrics. If you are configuring your Postgres Pro cluster from scratch, see Chapter 16 and Section 17.2 for Postgres Pro installation and setup instructions, respectively.

Note

While mamonsu can collect Postgres Pro metrics from a remote cluster, system metrics are only collected locally. If you choose to collect Postgres Pro metrics remotely, make sure to disable collection of system metrics to avoid confusion, as they will be displayed under the same host in Zabbix.

A pre-built mamonsu package is provided together with Postgres Pro Standard, but has a separate installer. Once you have installed mamonsu, complete the following steps to set up metrics collection:

  1. Optionally, bootstrap mamonsu

    If you omit this step, metrics can only be collected on behalf of a superuser, which is not recommended.

    1. Create a non-privileged database user for mamonsu. For example:

      CREATE USER mamonsu_user WITH PASSWORD 'mamonsu_password';
      

    2. Create a database that will be used for connection to Postgres Pro. For example:

      CREATE DATABASE mamonsu_database OWNER mamonsu_user;
      

    3. Run the following command to bootstrap mamonsu:

      mamonsu bootstrap -M mamonsu_user [connection_options]
      

      where connection_options can be --host, --port, --dbname, --username, and --password. These options provide connection parameters for the Postgres Pro cluster you are going to monitor. The --dbname option should specify the mamonsu_database created for monitoring purposes. Note that the --username option must specify a superuser that can access the cluster.

      If you omit connection options, mamonsu checks the corresponding environment variables for these settings. If they are missing, mamonsu tries to connect to the postgres database of the server instance running locally on port 5432, on behalf of the postgres user. Make sure to provide the actual connection parameters if your Postgres Pro cluster is located elsewhere.

    As the result of this operation, monitoring functions are created in the mamonsu_database, and the right to execute them is granted to the mamonsu_user. Thus, a superuser connection is no longer required. mamonsu also creates several tables with the mamonsu prefix in the specified database. Do not delete these tables as they are required for mamonsu to work.

  2. Configure mamonsu

    Edit the agent.conf configuration file, which is located in the /etc/mamonsu/ directory by default.

    • Configure Zabbix-related settings. The address field must point to the running Zabbix server, while the client field must provide the name of the Zabbix host. You can find the list of hosts available for your account in the Zabbix web interface under Configuration > Hosts.

      [zabbix]
      ; enabled by default
      enabled = True
      client = zabbix_host_name
      address = zabbix_server
      
    • By default, mamonsu will collect both Postgres Pro and system metrics. If required, you can disable metrics collection of either type by setting the enabled parameter to False in the [postgres] or [system] section of the agent.conf file, respectively.

      [system]
      ; enabled by default
      enabled = True
      

      Note

      While mamonsu can collect Postgres Pro metrics from a remote cluster, system metrics are only collected locally. If you choose to collect Postgres Pro metrics remotely, make sure to disable collection of system metrics to avoid confusion, as they will be displayed under the same host in Zabbix.

    • If you are going to collect Postgres Pro metrics, specify connection parameters for the Postgres Pro cluster you are going to monitor. In the user, password, and database fields, you must specify the mamonsu_user, mamonsu_password, and the mamonsu_database used for bootstrap, respectively. If you skipped the bootstrap, specify a superuser credentials and the database to connect to.

      [postgres]
      ; enabled by default
      enabled = True
      user = mamonsu_user
      database = mamonsu_database
      password = mamonsu_password
      port = 5432
      

    These are the main mamonsu settings to get started. You can also fine-tune other mamonsu settings as explained in the section called “Configuration Parameters”.

  3. Configure how to display metrics on the Zabbix server

    1. Generate a template that defines how to visualize the collected metrics on the Zabbix server:

      mamonsu export template template.xml
      

      mamonsu generates the template.xml file in your current directory. By default, the name of the template that will be displayed in the Zabbix account is PostgresPro-OS, where OS is the name of your operating system. To get a template with a different display name, you can run the above command with the --template-name option.

    2. Optionally, specify your Zabbix account settings in the following environment variables on your monitoring system:

      • Set the ZABBIX_USER and ZABBIX_PASSWD variables to the login and password of your Zabbix account, respectively.

      • Set the ZABBIX_URL to http://zabbix/

      If you skip this step, you will have to add the following options to all mamonsu zabbix commands that you run:

      --url=http://zabbix/ --user=zabbix_login --password=zabbix_password
      
    3. Upload the template.xml to the Zabbix server

      mamonsu zabbix template export template.xml
      

      Alternatively, you can upload the template through the Zabbix web interface: log in to your Zabbix account and select Templates > Import.

    4. Link the generated template to the host to be monitored.

      In the Zabbix web interface, select your host, go to Templates > Add, select your template, and click Update.

      Tip

      If you would like to link a template with a new Zabbix host, you can do it from the command line using mamonsu zabbix commands. See the section called “Managing Zabbix Server Settings from the Command Line” for details.

When the setup is complete, start mamonsu. For example, on Linux systems, you can start mamonsu as a service with the following command:

service mamonsu start

mamonsu picks up all the parameters from the mamonsu configuration file and starts monitoring your system.

Command-Line Reference

agent

Syntax:

mamonsu agent { metric-list | metric-get metric_name | version }

Provides information on the collected metrics from the command line. You can specify one of the following parameters:

metric-list

Show the list of metrics that mamonsu is collecting. The output of this command provides the item key of each metric, its latest value, and the time when this value was received.

metric-get metric_name

Check the latest value for the specified metric. You can get the list of available metrics using the metric-list option.

version

Display mamonsu version.

bootstrap

Syntax:

mamonsu bootstrap -M mamonsu_user [connection_options]

Bootstrap mamonsu. The -M option must specify a non-privileged user that will own all mamonsu processes.

connection_options can be --host, --port, --dbname, --username, and --password. These options provide connection parameters for the Postgres Pro cluster you are going to monitor. Note that the --username option must specify a superuser that can access the cluster.

If you omit connection options, mamonsu checks the corresponding environment variables for these settings. If they are missing, mamonsu tries to connect to the postgres database of the server instance running locally on port 5432, on behalf of the postgres user. Make sure to provide the actual connection parameters if your Postgres Pro cluster is located elsewhere.

export

Syntax:

mamonsu export config filename.conf  [--add-plugins=plugin_directory]
mamonsu export template filename.xml [--add-plugins=plugin_directory]
                                     [--template-name=template_name]
                                     [--application=application_name]

Generate a template or configuration file for metrics collection. The optional parameters to customize metrics collection are as follows:

--add-plugins=plugin_directory

Collect metrics that are defined in custom plugins located in the specified plugin_directory. If you are going to use custom plugins, you must provide this option when generating both the configuration file and the template.

--template-name=template_name

Specify the name of the template that will be displayed on the Zabbix server.

Default: PostgresPro-OS, where OS is the name of your operating system

--application=application_name

Specify an identifier under which the collected metrics will be displayed on the Zabbix server.

Default: App-PostgresPro-OS, where OS is the name of your operating system

export zabbix-parameters

Syntax:

mamonsu export zabbix-parameters filename.conf [--add-plugins=plugin_directory]
                                     [--plugin-type={pg | sys | all}]
                                     [--pg-version=version]
                                     [--config=config_file]

Export metrics configuration for use with the native Zabbix agent. The optional parameters to customize metrics collection are as follows:

--add-plugins=plugin_directory

Collect metrics that are defined in custom plugins located in the specified plugin_directory. If you are going to use custom plugins, you must provide this option when generating both the configuration file and the template.

--plugin-type={pg | sys | all}

Specify the type of metrics to collect:

  • pg for Postgres Pro metrics.

  • sys for system metrics.

  • all for both Postgres Pro and system metrics.

Default: all

--pg-version=version

Specify server version for which to configure metrics collection. Depending on the product you are going to monitor, the version format can differ, as follows:

  • For vanilla PostgreSQL, specify its major version. mamonsu supports vanilla PostgreSQL 9.5 or higher.

  • For Postgres Pro Standard, specify its major version with the PGPRO_ prefix. For example, PGPRO_10.

  • For Postgres Pro Enterprise, specify its major version with the PGEE_ prefix. For example, PGEE_10.

Default: 10

--config=config_file

Specify the agent.conf file to be used as the source for metrics definitions.

Default: /etc/mamonsu/agent.conf

export zabbix-template

Syntax:

mamonsu export zabbix-template filename.conf [--add-plugins=plugin_directory]
                                     [--plugin-type={pg | sys | all}]
                                     [--pg-version=version]
                                     [--template-name=template_name]
                                     [--application=application_name]
                                     [--config=config_file]

Export a template for use with the native Zabbix agent. The optional parameters to customize metrics collection are as follows:

--add-plugins=plugin_directory

Collect metrics that are defined in custom plugins located in the specified plugin_directory. If you are going to use custom plugins, you must provide this option when generating both the configuration file and the template.

--plugin-type={pg | sys | all}

Specify the type of metrics to collect:

  • pg configures Postgres Pro metrics.

  • sys configures system metrics.

  • all configures both Postgres Pro and system metrics.

Default: all

--pg-version=version

Specify server version for which to configure metrics collection. Depending on the product you are going to monitor, the version format can differ, as follows:

  • For vanilla PostgreSQL, specify its major version. mamonsu supports vanilla PostgreSQL 9.5 or higher.

  • For Postgres Pro Standard, specify its major version with the PGPRO_ prefix. For example, PGPRO_10.

  • For Postgres Pro Enterprise, specify its major version with the PGEE_ prefix. For example, PGEE_10.

Default: 10

--template-name=template_name

Specify the name of the template that will be displayed on the Zabbix server.

Default: PostgresPro-OS, where OS is the name of your operating system

--application=application_name

Specify an identifier under which the collected metrics will be displayed on the Zabbix server.

Default: App-PostgresPro-OS, where OS is the name of your operating system

--config=config_file

Specify the agent.conf file to be used as the source for metrics definitions.

Default: /etc/mamonsu/agent.conf

report

Syntax:

mamonsu report [--run-system=Boolean] [--run-postgres=Boolean]
               [--print-report=Boolean] [--report-path=report_file]
               [--disable-sudo] [connection_options]

Display detailed information about the hardware, operating system, memory usage and other parameters of the monitored system, as well as Postgres Pro configuration. The following optional parameters customize the report:

--run-system=Boolean

Include system information into the generated report.

Default: True

--run-postgres=Boolean

Include information on Postgres Pro into the generated report.

Default: True

--print-report=Boolean

Print the report to stdout.

Default: True

--report-path=report_file

Save the report into the specified file.

Default: /tmp/report.txt

--disable-sudo

Do not report data that can only be received with superuser rights. This option is only available for Linux systems.

connection_options

Provide connection parameters for the Postgres Pro cluster you are going to monitor. connection_options can be --host, --port, --dbname, --username, and --password. Note that the --username option must specify a superuser that can access the cluster.

If you omit connection options, mamonsu checks the corresponding environment variables for these settings. If they are missing, mamonsu tries to connect to the postgres database of the server instance running locally on port 5432, on behalf of the postgres user. Make sure to provide the actual connection parameters if your Postgres Pro cluster is located elsewhere.

tune

Syntax:

mamonsu tune [--dry-run] [--disable-sudo] [--log-level {INFO|DEBUG|WARN}]
             [--dont-tune-pgbadger] [--dont-reload-postgresql]

Optimize Postgres Pro and system configuration based on the collected statistics. You can use the following options:

--dry-run

Display the settings to be tuned without changing the actual system and Postgres Pro configuration.

--disable-sudo

Do not tune the settings that can only be changed by a superuser. This option is only available for Linux systems.

--dont-tune-pgbadger

Do not tune pgbadger parameters.

--log-level { INFO | DEBUG | WARN}

Change the logging level.

Default: INFO

--dont-reload-postgresql

Forbid mamonsu to run the pg_reload_conf() function. If you specify this option, the modified settings that require reloading Postgres Pro configuration do not take effect immediately.

zabbix item

Syntax:

mamonsu zabbix item {error | lastvalue | lastclock } host_name

View the specified property of the latest metrics data received by Zabbix for the specified host.

zabbix host

Syntax:

mamonsu zabbix host list
mamonsu zabbix host show host_name
mamonsu zabbix host id host_name
mamonsu zabbix host delete host_id
mamonsu zabbix host create host_name hostgroup_id template_id mamonsu_address
mamonsu zabbix host info {templates | hostgroups | graphs | items} host_id

Manage Zabbix hosts using one of the actions described in the section called “Zabbix Server Actions”.

zabbix hostgroup

Syntax:

mamonsu zabbix hostgroup list
mamonsu zabbix hostgroup show hostgroup_name
mamonsu zabbix hostgroup id hostgroup_name
mamonsu zabbix hostgroup delete hostgroup_id
mamonsu zabbix hostgroup create hostgroup_name

Manage Zabbix host groups using one of the actions described in the section called “Zabbix Server Actions”.

zabbix template

Syntax:

mamonsu zabbix template list
mamonsu zabbix template show template_name
mamonsu zabbix template id template_name
mamonsu zabbix template delete template_id
mamonsu zabbix template export file

Manage Zabbix templates using one of the actions described in the section called “Zabbix Server Actions”.

Zabbix Server Actions

Using mamonsu, you can control some of the Zabbix server functionality from the command line. Specifically, you can create or delete Zabbix hosts and host groups, as well as generate, import, and delete Zabbix templates using one of the following commands. The object_name to use must correspond to the type of the Zabbix object specified in the command: template, host, or hostgroup.

list

Display the list of available templates, hosts, or host groups.

show object_name

Display the details about the specified template, host, or host group.

id object_name

Show the ID of the specified object, which is assigned automatically by the Zabbix server.

delete object_id

Delete the object with the specified ID.

create hostgroup_name
create host_name hostgroup_id template_id mamonsu_address

Create a new host or a host group.

export template_name

Generate a Zabbix template.

info {templates | hostgroups | graphs | items} host_id

Display detailed information about the templates, host groups, graphs, and metrics available on the host with the specified ID.

Configuration Parameters

The agent.conf configuration file is located in the /etc/mamonsu directory by default. It provides several groups of parameters that control which metrics to collect and how to log the collected data:

All parameters must be specified in the parameter = value format.

Connection Parameters

[postgres]

The [postgres] section controls Postgres Pro metrics collection and can contain the following parameters:

enabled

Enables/disables Postgres Pro metrics collection if set to True or False, respectively.

Default: True

user

The user on behalf of which the cluster will be monitored. It must be the same user that you specified in the -M option of the bootstrap command, or a superuser if you skipped the bootstrap.

Default: postgres

password

The password for the specified user.

database

The database to connect to for metrics collection.

Default: postgres

host

The server address to connect to.

Default: localhost

port

The port to connect to.

Default: 5432

application_name

Application name that identifies mamonsu connected to the Postgres Pro cluster.

Default: mamonsu

query_timeout

statement_timeout for the mamonsu session, in seconds. If a Postgres Pro metric query does not complete within this time interval, it gets terminated.

Default: 10

[system]

The [system] section controls system metrics collection and can contain the following parameters:

enabled

Enables/disables system metrics collection if set to True or False, respectively.

Default: True

[zabbix]

The [zabbix] section provides connection settings for the Zabbix server and can contain the following parameters:

enabled

Enables/disables sending the collected metric data to the Zabbix server if set to True or False, respectively.

Default: True

client

The name of the Zabbix host.

address

The address of the Zabbix server.

Default: 127.0.0.1

port

The port of the Zabbix server.

Default: 10051

[agent]

The [agent] section specifies the location of mamonsu and whether it is allowed to access metrics from the command line:

enabled

Enables/disables metrics collection from the command line using the agent command.

Default: True

host

The address of the system on which mamonsu is running.

Default: 127.0.0.1

port

The port on which mamonsu is running.

Default: 10052

[sender]

The [sender] section controls the queue size of the data to be sent to the Zabbix server:

queue

The maximum number of collected metric values that can be accumulated locally before mamonsu sends them to the Zabbix server. Once the accumulated data is sent, the queue is cleared.

Default: 2048

Logging Parameters

[metric_log]

The [metric_log] section enables storing the collected metric data in text files locally. This section can contain the following parameters:

enabled

Enables/disables storing the collected metric data in a text file. If this option is set to True, mamonsu creates the localhost.log file for storing metric values.

Default: False

directory

Specifies the directory where log files with metric data will be stored.

Default: /var/log/mamonsu

max_size_mb

The maximum size of a log file, in MB. When the specified size is reached, it is renamed to localhost.log.archive, and an empty localhost.log file is created.

Default: 1024

[log]

The [log] section specifies logging settings for mamonsu and can contain the following parameters:

file

Specifies the log filename, which can be preceded by the full path.

level

Specifies the debug level. This option can take DEBUG, ERROR, or INFO values.

Default: INFO

format

The format of the logged data.

Default: [%(levelname)s] %(asctime)s - %(name)s - %(message)s

where levelname is the debug level, asctime returns the current time, name specifies the plugin that emitted this log entry or root otherwise, and message provides the actual log message.

Plugin Parameters

[plugins]

The [plugins] section specifies custom plugins to be added for metrics collection and can contain the following parameters:

enabled

Enables/disables using custom plugins for metrics collection if set to True or False, respectively.

Default: False

directory

Specifies the directory that contains custom plugins for metrics collection. Setting this parameter to None forbids using custom plugins.

Default: /etc/mamonsu/plugins

Individual Plugin Sections

To disable any plugin, you can add a new section with the plugin name and set the enabled option to False. The plugin name corresponds to the name of the Python class in the plugin script, but should be written in lowercase. For example, to stop collecting connection metrics, add the following section:

[connections]
enabled = False

You can also modify the default settings for some of the plugins. All configurable settings are listed in the agent.conf file.

Usage

Collecting and Viewing Metrics Data

Once started, mamonsu begins collecting system and Postgres Pro metrics. The agent command enables you to get an overview of the collected metrics from the command line in real time.

To view the list of available metrics, use the agent metric-list command:

mamonsu agent metric-list

The output of this command provides the item key of each metric, its latest value, and the time when this value was received. For example:

system.memory[active]      7921004544    1564570818
system.memory[swap_cache]      868352    1564570818
pgsql.connections[idle]           6.0    1564570818
pgsql.archive_command[failed_trying_to_archive]    0    1564570818

To view the current value for a specific metric, you can use the agent metric-get command:

mamonsu agent metric-get metric_name

where metric_name is the item key of the metric to monitor, as returned by the metric-list command. For example, pgsql.connections[idle].

You can view graphs for the collected metrics in the Zabbix web interface under the Graphs menu. For details on working with Zabbix, see its official documentation at https://www.zabbix.com/documentation/current/.

Adding Custom Plugins

You can extend mamonsu with your own custom plugins, as follows:

  1. Save all custom plugins in a single directory, such as /etc/mamonsu/plugins.

  2. Make sure this directory is specified in your configuration file under the [plugins] section:

    [plugins]
    directory = /etc/mamonsu/plugins
    

  3. Generate a new Zabbix template to include custom plugins:

    mamonsu export template template.xml --add-plugins=/etc/mamonsu/plugins
    

  4. Upload the generated template.xml to the Zabbix server as explained in the section called “Installation and Setup”.

Tuning Postgres Pro and System Configuration

Based on the collected metrics data, mamonsu can tune your Postgres Pro and system configuration for optimal performance.

You can get detailed information about the hardware, operating system, memory usage and other parameters of the monitored system, as well as Postgres Pro configuration, as follows:

mamonsu report

To view the suggested optimizations without applying them, run the tune command with the --dry-run option:

mamonsu tune --dry-run

To apply all the suggested changes, run the tune command without any parameters:

mamonsu tune

You can exclude some settings from the report or disable some of the optimizations by passing optional parameters. See the section called “Command-Line Reference” for the full list of parameters available for report and tune commands.

Managing Zabbix Server Settings from the Command Line

mamonsu enables you to work with the Zabbix server from the command line. You can upload template files to Zabbix, create and delete Zabbix hosts and host groups, link templates with hosts and check the latest metric values.

To set up your Zabbix host from scratch, you can follow these steps:

  1. Optionally, specify your Zabbix account settings in the following environment variables on your monitoring system:

    • Set the ZABBIX_USER and ZABBIX_PASSWD variables to the login and password of your Zabbix account, respectively.

    • Set the ZABBIX_URL to http://zabbix/

    If you skip this step, you will have to add the following options to all mamonsu zabbix commands that you run:

    --url=http://zabbix/ --user=zabbix_login --password=zabbix_password
    
  2. Generate a new template file and upload it to the Zabbix server:

    mamonsu export template template.xml
    mamonsu zabbix template export template.xml
    

  3. Create a new host group:

    mamonsu zabbix hostgroup create hostgroup_name
    

  4. Check the IDs for this host group and the uploaded template, which are assigned automatically by the Zabbix server:

    mamonsu zabbix hostgroup id hostgroup_name
    mamonsu zabbix template id template_name
    

  5. Create a host associated with this group and link it with the uploaded template using a single command:

    mamonsu zabbix host create host_name hostgroup_id template_id mamonsu_address
           

    where host_name is the name of the host, hostgroup_id and template_id are the unique IDs returned in the previous step, and mamonsu_address is the IP address of the system on which mamonsu runs.

Once your Zabbix host is configured, complete the setup of your monitoring system as explained in the section called “Installation and Setup”.

Exporting Metrics for Native Zabbix Agent

Using mamonsu, you can convert all system and Postgres Pro metrics definitions to the format supported by the native Zabbix agent.

This feature is unavailable on Windows systems.

To collect metrics provided by mamonsu using the native Zabbix agent, do the following:

  1. Generate a configuration file that is compatible with the native Zabbix agent and place it under /etc/zabbix/zabbix_agent.d/. You can prepend the filename with the required path:

    mamonsu export zabbix-parameters /etc/zabbix/zabbix_agent.d/zabbix_agent.conf
    

    For all possible options of the export zabbix-parameters command, see the section called “Command-Line Reference”.

  2. Generate a template for the native Zabbix agent:

    mamonsu export zabbix-template template.xml
    

    For all possible options of the export zabbix-template command, see the section called “Command-Line Reference”.

  3. Upload the generated template to the Zabbix server, as explained in the section called “Installation and Setup”.

  4. If you are going to collect Postgres Pro metrics, change the following macros in the template after the upload:

    • For {$PG_CONNINFO}, provide connection parameters for the Postgres Pro server to be monitored.

    • For {$PG_PATH}, specify psql installation directory.