27.4. Reference for the biha Extension #
biha is a Postgres Pro extension that allows managing a BiHA cluster.
This section contains information about the biha extension configuration parameters, functions, and views.
27.4.1. Configuration Parameters #
The biha extension provides a set of configuration parameters described below that are specific to the BiHA cluster.
27.4.1.1. Cluster Configuration Parameters #
Important
When setting the cluster configuration parameters, you must ensure network reliability for the changes to affect all cluster nodes without any errors.
It is not recommended to modify any cluster configuration parameters while one of the nodes is in the
NODE_ERRORstate, as the changes may fail to be applied on that node.
biha.autorewind(boolean) #An optional parameter that controls the automatic rewind policy for that node, against which the
pg_rewindmust be executed. For example, for the old leader when synchronizing it with the new leader. The default value isfalsemeaning that the automatic rewind is not performed. When the value is set totrue, the automatic rewind is performed after the error that usually causes theNODE_ERRORstate of the node.For more information, see Automatic Cluster Synchronization after Failover.
biha.autowaltrim(boolean) #Manages the automatic WAL trimming policy within a single node. When set to
true(default), BiHA automatically schedules and executes the WAL trimming procedure if possible. If set tofalse, the WAL trimming procedure is not executed and cluster synchronization is performed by means of pg_rewind.For more information, see Automatic Cluster Synchronization after Failover.
biha.callbacks_timeout(integer) #Sets time for execution of all callbacks for a single event, in milliseconds. The default value is
10000(10 seconds). The minimum value is1000(1 second).The value of the
biha.callbacks_timeoutparameter can be different for different nodes. Changing the value for the leader does not change it for the followers.biha.can_be_leader(boolean) #Determines the ability of a node to become the leader. The default value is
true. If set tofalse, the node cannot stand as a candidate in elections of the new leader.A node that cannot become the leader may have the largest WAL. In this case, if elections are required, those nodes that can stand as candidates attempt to receive missing data from that node. If they succeed, one of these nodes becomes the leader. If they fail to receive missing data, the automatic rewind process is launched on the node that cannot become the leader. If biha.autorewind is not enabled, the node state changes to
NODE_ERROR.biha.flw_ro(boolean) #Determines whether a follower is available for read operations. If set to
off, reading from this follower is prohibited. The default value ison.biha.heartbeat_max_lost(integer) #Specifies the maximum number of heartbeats that can be missed before the node is considered offline. This parameter can be set with the biha.set_heartbeat_max_lost function. The default value is
10.Nodes in all states use the heartbeat timeout value calculated as
biha.heartbeat_max_lost* biha.heartbeat_send_period to determine if other nodes are online. For example, if Node A receives no heartbeats from Node B within the heartbeat timeout, the Node B state changes toUNKNOWNin the biha.status_v view of Node A indicating that Node A considers Node B offline. The leader in theLEADER_RWstate additionally uses the timeout specified in the biha.no_wal_on_follower parameter.biha.heartbeat_send_period(integer) #Specifies the heartbeat sending frequency, in milliseconds. This parameter can be set with the biha.set_heartbeat_send_period function. The default value is
1000.Nodes in all states use the heartbeat timeout value calculated as biha.heartbeat_max_lost *
biha.heartbeat_send_periodto determine if other nodes are online. For example, if Node A receives no heartbeats from Node B within the heartbeat timeout, the Node B state changes toUNKNOWNin the biha.status_v view of Node A indicating that Node A considers Node B offline. The leader in theLEADER_RWstate additionally uses the timeout specified in the biha.no_wal_on_follower parameter.biha.host(text) #Specifies the host of the cluster node. It is not recommended to modify this parameter.
biha.id(integer) #Specifies the ID of the high-availability cluster node. This parameter is unique for each node of the cluster. It is not recommended to modify this parameter.
biha.no_wal_on_follower(integer) #Specifies the timeout of the replication slot position advancement, in milliseconds. This parameter can be set with the biha.set_no_wal_on_follower function. The default value is
20000.BiHA uses this configuration parameter for the following purposes:
When you nominate a follower as the new leader using the biha.set_leader function, the follower state changes to
FOLLOWER_OFFERED. The follower in this state usesbiha.no_wal_on_followerto determine if it must stop waiting for missing WAL records from the current leader and can become the new leader.A follower in the
FOLLOWER_OFFEREDstate checks whether its WAL location has been updated. Update of WAL location confirms that the follower keeps receiving replication data from the source. If the follower does not receive WAL records longer thanbiha.no_wal_on_follower, the follower state changes back toFOLLOWER. In this case, the log contains the following message: “Timed out waiting for new WAL records from current source”.The leader in the
LEADER_RWstate usesbiha.no_wal_on_followerin conjunction with the heartbeat timeout value (which is calculated as biha.heartbeat_max_lost * biha.heartbeat_send_period) to determine if other nodes are online.The leader checks the replication slot positions of other nodes. If the replication slot position updates, it means that the node receives replication data from the leader and the node is online. If the leader does not receive heartbeats from the node within the above mentioned heartbeat timeout and its replication slot position does not change within the
biha.no_wal_on_followertimeout value, the leader considers the node offline and its state changes toUNKNOWNin the biha.status_v view of the leader.
biha.port(integer) #Specifies the port used to exchange service information between nodes. This parameter is required to establish a connection with the cluster. It is not recommended to modify this parameter.
biha.proxima_status(integer) #Indicates whether the proxima extension is installed on a node and specifies how it was enabled. The following values are available:
0: the extension is not installed. This is the default value. You can set this value by using the biha.disable_proxima function.1: the extension is installed using the biha.enable_proxima function.2: the extension is installed using the bihactl cluster init command with the --enable-proxima option.
biha.use_ssl(boolean) #Specifies whether to use the protected mode of the service information exchange between cluster nodes with SSL/TLS over the biha control channel. It is not recommended to modify this parameter.
biha.watchdog_timeout(integer) #Specifies the timeout for a node to stop accepting queries if the biha process does not respond, in seconds. The default value is 2. The minimum value is 1.
27.4.1.2. Configuration Parameters Critical for GDBiHA #
The following BiHA configuration parameters are mainly critical for GDBiHA clusters. Unlike general cluster configuration parameters, these configuration parameters are stores in the pg_biha/biha.conf file and cannot be changed via ALTER SYSTEM SET or viewed via SHOW. To modify configuration parameters listed below, you must use special functions mentioned in parameter descriptions.
can_vote(boolean) #Determines whether a node is allowed to vote. The default value is
true. If set tofalse, the node cannot vote, as well as cannot stand as a candidate in elections of the new leader.minnodes(integer) #Specifies the minimum number of operational nodes for the leader node to be open for write transactions. This parameter is set for a segment and inherited by nodes within this segment. This parameter can be set with the biha.set_minnodes function. If you do not specify the --minnodes option in bihactl cluster init command, the
minnodesvalue equals thenquorumvalue.When setting up this value, consider the split-brain risk. It is recommended to use the following formula: (
total_number_of_nodes+ 1)/2. For example, if your cluster has 3 nodes, theminnodesvalue must be 2.Nodes with the can_vote parameter set to
falseare ignored.max_replicas(integer) #Specifies the maximum number of biha replication connections to a node, i.e. the maximum number of walsenders where
application_nameisbiha_node_*. The value of this parameter must not exceed themax_wal_sendersvalue. You can modify this parameter value only by the biha.set_max_replicas function.Possible values are:
0,INT_MAX. The default value isINT_MAX, the number of connections is unlimited.nquorum(integer) #Specifies the minimum number of nodes, which must vote for the new leader node if the current leader is down. This parameter is set for a segment and inherited by nodes within this segment. This parameter can be set with the biha.set_nquorum function.
When setting up this value, consider the split-brain risk. It is recommended to use the following formula: (
total_number_of_nodes+ 1)/2. For example, if your cluster has 3 nodes, thenquorumvalue must be 2.Nodes with the can_vote parameter set to
falseare ignored.preferred_roles(text) #Specifies the preferred role of a node for replication in a BiHA cluster with cascading replication. You can modify this parameter value only by the biha.set_pref_roles function.
Possible values are: combinations of
L(leader),F(follower), andR(referee). The value must be one to three symbols long and must not contain repeatable symbols. For example:L,F,LFR, orLF.The default value is
L, meaning that data is replicated from the leader or front follower depending on the segment where the node is located.priority(integer) #Sets the node weight that affects both elections and replication in a cluster, in milliseconds. Possible values are:
0,INT_MAX. The default value is-1, meaning that the parameter is ignored. You can modify this parameter value only by the biha.set_priority function.BiHA uses this configuration parameter for the following purposes:
To set the replication start timeout when selecting a replication source in a BiHA cluster with configured cascading replication. The higher the value, the later the node starts replication and allows connection of nodes with lower priority. The parameter is required for the nodes to establish the cascading replication schema automatically.
To set the node priority in a cluster with synchronous replication. The value determines the timeout that must be reached before the node stands as a candidate in elections. The zero value indicates the highest priority.
Important
To ensure correct operation of this parameter, set the --sync-standbys value one unit less than the total number of cluster nodes.
27.4.1.3. biha Logging Levels #
27.4.1.3.1. General Logging Levels #
General logging levels are used for biha components that have no designated logging levels.
biha.BihaLog_log_level(enum) #Specifies the logging level to provide general information about the operation of biha components. The default value is
LOG.biha.BihaWarn_log_level(enum) #Specifies the logging level to provide warnings of likely problems about the operation of biha components. The default value is
WARNING.biha.BihaDetails_log_level(enum) #Specifies the logging level to provide detailed information about the operation of biha components. The default value is
DEBUG1.biha.BihaDebug_log_level(enum) #Specifies the logging level to provide debugging information about the operation of biha components. The default value is
DEBUG2.biha.BihaDebug2_log_level(enum) #Specifies the logging level to provide debugging information about the operation of biha components. The default value is
DEBUG3.
27.4.1.3.2. BCP and Transport Logging Levels #
biha.BcpTransportLog_log_level(enum) #Specifies the logging level to provide general information about the control channel operation. The default value is
LOG.biha.BcpTransportWarn_log_level(enum) #Specifies the logging level to provide warnings of likely problems in the control channel. The default value is
WARNING.biha.BcpTransportDetails_log_level(enum) #Specifies the logging level to provide detailed information about the control channel operation. The default value is
DEBUG1.biha.BcpTransportDebug_log_level(enum) #Specifies the logging level to provide debugging information about the control channel operation. The default value is
DEBUG2.
27.4.1.3.3. Node Controller Logging Levels #
The following logging levels provide information about the Node Controller component.
biha.NodeControllerLog_log_level(enum) #Specifies the logging level to provide general information about the the
Node Controllercomponent. The default value isLOG.biha.NodeControllerWarn_log_level(enum) #Specifies the logging level to provide warnings of likely problems in the
Node Controllercomponent. The default value isWARNING.biha.NodeControllerDetails_log_level(enum) #Specifies the logging level to provide detailed information about operation of the
Node Controllercomponent. The default value isDEBUG1.biha.NodeControllerDebug_log_level(enum) #Specifies the logging level to provide debugging information about the operation of the
Node Controllercomponent. The default value isDEBUG2.biha.NodeControllerDebug2_log_level(enum) #Specifies the logging level to provide debugging information about the operation of the
Node Controllercomponent. The default value isDEBUG3.
27.4.1.3.4. Postgres Controller Logging Levels #
The following logging levels provide information about the Postgres Controller component.
biha.PgControllerLog_log_level(enum) #Specifies the logging level to provide general information about the
Postgres Controllercomponent. The default value isLOG.biha.PgControllerWarn_log_level(enum) #Specifies the logging level to provide warnings of likely problems in the
Postgres Controllercomponent. The default value isWARNING.biha.PgControllerDetails_log_level(enum) #Specifies the logging level to provide detailed information about operation of the
Postgres Controllercomponent. The default value isDEBUG1.biha.PgControllerDebug_log_level(enum) #Specifies the logging level to provide debugging information about the operation of the
Postgres Controllercomponent. The default value isDEBUG2.biha.PgControllerDebug2_log_level(enum) #Specifies the logging level to provide debugging information about the operation of the
Postgres Controllercomponent. The default value isDEBUG3.
27.4.1.3.5. Config Controller Logging Levels #
The following logging levels provide information about the Config Controller component.
biha.BihaConfigLog_log_level(enum) #Specifies the logging level to provide general information about the
Config Controllercomponent. The default value isLOG.biha.BihaConfigWarn_log_level(enum) #Specifies the logging level to provide warnings of likely problems in the
Config Controllercomponent. The default value isWARNING.biha.BihaConfigDetails_log_level(enum) #Specifies the logging level to provide detailed information about operation of the
Config Controllercomponent. The default value isDEBUG1.biha.BihaConfigDebug_log_level(enum) #Specifies the logging level to provide debugging information about the operation the
Config Controllercomponent. The default value isDEBUG2.biha.BihaConfigDebug2_log_level(enum) #Specifies the logging level to provide debugging information about the operation of the
Config Controllercomponent. The default value isDEBUG3.
27.4.2. Functions #
All functions listed below must be called from the biha_db database, for example:
psql biha_db -c "SELECT biha.set_leader(2)"
27.4.2.1. Cluster Composition #
biha.add_node (id integer, parent_id integer) returns boolean#Adds a segment to the GDBiHA cluster. If
parent_idis not specified, by default the root node with ID 1111 is assigned as the parent.biha.apply_topology() returns boolean#Applies the new GDBiHA cluster composition after you change node parents by biha.set_parent. This function must be called on the leader node.
biha.set_leader (id integer) returns boolean#Sets the leader, front follower, or the leader segment manually. It is recommended to call this function on the node that you want to set as the new leader or front follower.
For more information about the manual switchover procedure, see Manual Switchover.
Note
Calling the
biha.set_leaderfunction on the current leader is not recommended. If you call the function on the current leader in theLEADER_RWstate, in case of a successful switchover request, there may not be enough time for the request result to be sent to the client before the current leader reboots for demotion.biha.set_parent (node_id integer, parent_id integer) returns boolean#Allows to unite nodes under a certain segment or move a node from one segment to another. This function must be called on the leader node.
parent_idmust belong to a segment.node_idandparent_idmust not be equal.After using
biha.set_parent, execute biha.apply_topology for the changes to take effect.biha.remove_node (id integer) returns boolean#Removes a node or a segment from the cluster. Before removing, the node must be stopped. You cannot remove a node if it is a parent of another node. This function can only be called on the leader node.
For example:
biha_db=# SELECT biha.remove_node(3); remove_node ------------- t (1 row)
27.4.2.2. Common Cluster Configuration #
The following functions are used to change configuration parameters that must have the same values on all cluster nodes. These functions must only be called on the leader node. The changes are applied to all cluster nodes.
You do not need to restart the cluster nodes after calling a function for the changes to take effect unless the function description explicitly specifies otherwise.
biha.disable_proxima () returns boolean#Important
After calling this function, you must stop and start the cluster nodes using pg_ctl for the changes to take effect. The leader node must be the last to stop and first to start.
Disables the proxima extension that was enabled by the biha.enable_proxima function or the --enable-proxima option. As a result, the proxima extension is removed from
shared_preload_librariesand the proxima.cluster_mode configuration parameter is set tostandalone.biha.enable_proxima () returns boolean#Important
After calling this function, you must stop and start the cluster nodes using pg_ctl for the changes to take effect. The leader node must be the last to stop and first to start.
Enables the proxima extension in the existing BiHA cluster by adding the proxima extension to
shared_preload_librariesand setting the proxima.cluster_mode configuration parameter tobiha.biha.set_heartbeat_max_lost (integer) returns boolean#Sets the biha.heartbeat_max_lost value.
biha.set_heartbeat_send_period (integer) returns boolean#Sets the biha.heartbeat_send_period value, in milliseconds.
biha.set_no_wal_on_follower (integer) returns boolean#Sets the biha.no_wal_on_follower value, in milliseconds.
biha.set_minnodes (node_id integer, minnodes integer) returns boolean#Sets the minnodes value. If
node_idis a physical node, thenquorumandminnodesvalues are set for the segment where the node is located. Ifnode_idis a segment, the values are set for this node and its children. You can set different values for each segment.biha.set_nquorum (node_id integer, nquorum integer) returns boolean#Sets the nquorum value. If
node_idis a physical node, thenquorumandminnodesvalues are set for the segment where the node is located. Ifnode_idis a segment, the values are set for this node and its children. You can set different values for each segment.biha.set_nquorum_and_minnodes (node_id integer, nquorum integer, minnodes integer) returns boolean#Sets nquorum and minnodes values. If
node_idis a physical node, thenquorumandminnodesvalues are set for the segment where the node is located. Ifnode_idis a segment, the values are set for this node and its children. You can set different values for each segment.For example:
biha_db=# SELECT biha.set_nquorum_and_minnodes(3, 3, 1); set_nquorum_and_minnodes -------------------------- t (1 row)
27.4.2.3. Replication #
The following functions are used to configure and manage replication.
biha.add_to_ssn (id integer) returns boolean#Note
You can call this function only when quorum-based synchronous replication is enabled.
Adds a node with the specified ID to the list of synchronous standbys of the synchronous_standby_names parameter. You must specify the node which name is not yet in the list.
biha.get_ssn() returns varchar#Displays the current value of the synchronous_standby_names parameter.
biha.set_can_vote(node_id integer, can_vote integer) returns boolean#Sets the value of the can_vote configuration parameter.
biha.set_max_replicas(node_id integer, value integer) returns boolean#Sets the value of the max_replicas configuration parameter.
biha.set_pref_roles(node_id integer, value text) returns boolean#Sets the value of the preferred_roles configuration parameter.
biha.set_priority(node_id integer, value integer) returns boolean#Sets the value of the priority configuration parameter.
biha.remove_from_ssn (id integer) returns boolean#Note
You can call this function only when quorum-based synchronous replication is enabled.
Removes the node with the specified ID from the list of synchronous standbys of the synchronous_standby_names parameter. You must specify the node which name is already in the list.
biha.set_ssn (VARIADIC ids integer) returns boolean#Note
You can call this function only when quorum-based synchronous replication is enabled.
Sets nodes with the IDs specified in the
idsarray as the list of synchronous standbys of the synchronous_standby_names parameter.biha.set_sync_standbys (integer) returns boolean#Enables quorum-based synchronous replication by setting the synchronous_standby_names parameter and specifying the number of synchronous standbys (quorum) with the
ANYmethod. The value must be positive, higher than theMINfield value of the synchronous_standby_names parameter if set, and must not exceed the number of followers excluding referee. For more information, see Replication Configuration.Important
Use this function carefully, as modifying the number of synchronous standbys may affect operation of the priority system or cause hanging of client application when requesting data from a lagging node.
biha.set_sync_standbys_min (integer) returns boolean#Note
You can call this function only when quorum-based synchronous replication is enabled.
Sets the
MINfield value of the synchronous_standby_names parameter, i.e. the minimum number of synchronous standbys that must be available for the leader node to continue operation.Using this function, you can also manage relaxed quorum-based synchronous replication. To disable relaxed quorum-based synchronous replication, set the
-1value.
27.4.2.4. Cluster Monitoring #
biha.config () returns setof record#Returns the cluster configuration parameter values:
id,term,nquorum,minnodes,heartbeat_send_period,heartbeat_max_lost,no_wal_on_follower,sync_standbys_min,priority,can_be_leader,can_vote,mode,proxima_status,name,repl_pref_roles,max_replicas,config_version,level,parent_id.biha.error_details () returns setof record#Returns the description of why the node transferred to the
NODE_ERRORstate. The returned record contains the type of the error, its details, the place it occurred specifyingbegin_lsn,end_lsn, and identifier of the current and the next timeline, as well asreplay_lsn.For example:
biha_db=# \x biha_db=# SELECT * FROM biha.error_details(); -[ RECORD 1 ]--+------------------------------------------------------------------------------------- │ type | TIMELINE_ERROR │ details | Timeline divergence detected: current timeline ended before replayLSN, rewind needed │ next_tli | 3 │ next_begin_lsn | │ next_end_lsn | │ cur_tli | 2 │ cur_begin_lsn | 0/17F53A0 │ cur_end_lsn | 0/3455FC8 │ replay_lsn | 0/34593B8
biha.monitoring () returns setof record#Displays a table that contains the monitoring data about the current status of cluster nodes. The table consists of the following columns:
id: The unique identifier of the node.time: The time where the monitoring data was received.biha_state: The current state of the node. For more information about possible states, see biha.status_v.pg_state: The current state of Postgres Pro. Possible values:Prestartup,Startup,Recovery,Recovery_Pause,Promoting,Promoted.online: Shows whether the node is online.startup_progress: Shows percentage of the current startup progress whenpg_stateisStartup.
You can call the function from any node.
biha.nodes () returns setof record#Defines the
biha.nodes_vview, which is described in detail in the biha.nodes_v section.biha.status () returns setof record#Defines the
biha.status_vview, which is described in detail in the biha.status_v section. It is not recommended to use this function as it provides raw data for the view.
27.4.2.5. Callback Management #
biha.register_callback (event text, func text, database text, executor text, priority integer) returns integer#Adds a new callback and returns its unique ID. This function can only be called from the leader node in the
LEADER_RWstate. The new callback is replicated to the followers.Note
On the biha side, there is no check that
funcexists indatabase. If the specified function does not exist, the callback fails.See Registering Callbacks for the example of using the
biha.register_callbackfunction.Table 27.3. Variable Definitions
Name Type Description eventtext Event of the BiHA cluster that triggers a callback. For more information about events and corresponding callback types, see Callback Types. functext Name of the SQL function that biha executes when eventhappens. This function must be located indatabase, otherwise it cannot be executed.databasetext Database where funcis executed.executortext User that executes
func. This parameter is optional. The default value isbiha_callbacks_user.priorityinteger The lower the value, the sooner the callback is executed. This parameter is optional. The default value is 0.biha.unregister_callback(callback_id)#Deletes a callback. This function can only be called on the leader node in the
LEADER_RWstate. See Unregistering Callbacks for the example of using thebiha.unregister_callbackfunction.
27.4.2.6. Other Functions #
biha.get_magic_string () returns string#Generates a magic string for the cluster node. For example:
biha_db=# SELECT biha.get_magic_string(); get_magic_string ---------------------------------------------------------------------- dmVyc2lvbj0xIGhvc3Q9bG9jYWxob3N0IHBvcnQ9NjU0MzIgYmloYS1wb3J0PTEwMDAx (1 row)biha.reset_node_error () returns boolean#Resets the
NODE_ERRORstate on a node. Use this function after you fix the issue that caused theNODE_ERRORstate. For more information, see Restoring the Node from the NODE_ERROR State.biha.service_mode (#enableboolean,forceboolean) returns booleanEnables the service mode in a BiHA cluster. This function can only be called on the leader node. The function takes the following parameters:
enable: when set totrue, the service mode is enabled. When not set or set tofalse, the service mode is disabled. Default isfalse.force: when set totrue, allows to disable the service mode ignoring all errors. When not set or set tofalse, the service mode cannot be disabled until you resolve all errors. Default isfalse.
27.4.3. Views #
To query the views listed below, you must first connect to the biha_db database.
27.4.3.1. biha.nodes_v #
This view displays the connection status of nodes in the cluster. For the node that the view is queried on, the following columns contain NULL: state, since_conn_start, conn_count.
Table 27.4. The biha.nodes_v view
| Column Name | Description |
|---|---|
id | The node ID. |
host | The host of the node. |
port | The port of the node. |
state | The connection state of the node. This column may contain one of the following values: ACTIVE, CONNECTING, IDLE, or INIT. |
since_conn_start | The time since the node connection. |
conn_count | The number of times the node was connected since the start of the cluster. |
27.4.3.2. biha.status_v #
This view displays the state of nodes in the cluster.
Table 27.5. The biha.status_v View
| Column Name | Description |
|---|---|
id | The node ID. |
leader_id | The leader node ID. |
term | The term of the node. This is used for the purposes of the leader election. The data type is integer. |
online | Shows whether the node is online. The data type is boolean (t or f). |
state | The state of the node. This column may contain one of the following values:
|
last_known_state | The last known state of the node. |
since_last_hb | The time since the last received heartbeat message. The data type is interval. |
27.4.3.3. biha.super_status_v #
The extended version of the biha.status_v view that additionally displays data for GDBiHA cluster.
The biha.super_status_v view contains the following additional columns:
name: The name of the node.level: Displays the level where the node operates. Possible values: 1 (physical level), 2 (segment level), 3 (cluster level).parent_id: The unique identifier of the node that is the parent to the current node.segment_name: The name of the segment where the node belongs to.