A.10. Postgres Pro Shardman 14.11.1 #

Release date: 2024-03-14

This release provides new features, optimizations and bug fixes. Major changes are as follows:

A.10.1. Core and Extensions #

  • Enhanced the pgpro_stats extension to give a better understanding of what system resources are used for distributed queries. Now the regular pgpro_stats_statements view shows the gathered statistics for individual statements on the current Shardman node (they can be part of some distributed query), while the pgpro_stats_sdm_statements view shows the gathered statistics for the distributed queries originating from the current node, that is, aggregated from all the participating nodes.

  • Added the pgpro_pwr package compatible with Shardman. This allows Shardman users to build workload reports.

  • Improved the EXPLAIN output. If a query plan contains ForeignScan nodes, the EXPLAIN output for queries executed on the remote server can now be included.

  • Added a new configuration parameter enable_partition_pruning_extra that enables extended subplan pruning logic when building and executing generic plans where the set of useful partitions depends on the prepared query parameters. This allows Shardman to do initial pruning of complex subplans, joins and partial aggregates, in particular.

  • Added metrics to the shardman.pg_stat_csn view that show delays of the global horizon and the transaction that may cause that delay. They may be useful to research autovacuum issues.

A.10.2. Management Utilities #

  • Considerably improved backup and restore with the shardmanctl probackup command. Notable changes are as follows:

    • Added support of backups to an S3-compatible object storage.

    • Implemented selective WAL archiving on the specified shards by the probackup subcommand.

    • Added two new commands shardmanctl probackup delete and shardmanctl probackup merge. The delete command deletes a backup with a specified ID and the archived WAL files that are no longer in use. The merge command merges the backups that belong to a common incremental backup chain.

    • Added new shardmanctl probackup set-config command that adds the specified settings to the pg_probackup.conf or modifies the existing ones.

    • Added a new option log-to-console for the validate subcommand. Set the log rotation file size to 20 MB. If this value is reached, the log file is rotated once a validate or backup subcommand is launched.

    • Increased the number of retries for some subcommands to avoid backup failures caused by large database sizes.

    • Added topology compatibility checks between the current Shardman cluster and the one in the backup directory to backup and restore subcommands.

    • Set the default value for the number of concurrent pg_probackup processes to the number of logical CPUs of the system.

    • Fixed data cleanup after a failure of a backup subcommand. Previously, some data of a failed backup could still remain in the repository.

    • Fixed hanging that could occur during metadata-only restore of a Shardman cluster.

    • Fixed the pg_probackup issue that could occur during the schema recovery process.

    • Changed the behavior of metadata-only restore to avoid losing a cluster. Now the cluster is stopped before such a restore and restarted after it, a cluster that has no nodes cannot be restored from the etcd dump, and if cluster IDs of the dump and the current cluster are different, the user is asked whether restoring the cluster with the changed ID is OK.

    • Added new options for archive-command: --compress, --compress-algorithm, --compress-level, --batch-size, and -j|--jobs. This helps to reduce the WAL size.

    • Improved the show subcommand output. Added new flags -archive to output the log information, -instance and -i|--backup-id to output information for the specified backups and instances.

  • Updated the getconnstr and cluster topology commands so that they do not issue a lock on other processes. Previously, some commands failed to receive a connection string because of the locks.

  • Fixed a panic that could occur on a Shardman cluster configured with PlacementPolicy = manual when a user executed the command shardmanctl cluster repfactor set.

  • Hid uninformative warnings that pg_dump displayed during execution of shardmanctl nodes add and shardmanctl probackup backup commands.

  • Removed a lock that was required by the shardmanctl status command. Previously shardmanctl status did not provide any useful information in case a process hung as it was waiting for the lock from that process.

  • Added the forceSuUserLocalPeerAuth configuration parameter. When enabled, it sets a peer authentication via unix socket for the postgres user unless strictUserHBA is set to true. See sdmspec.json for details.

  • Added a URL for Prometheus automatic service discovery metrics to shardmand.

pdf