Re: Simplifying "standby mode" - Mailing list pgsql-hackers

On Wed, 2006-09-06 at 12:01 -0400, Bruce Momjian wrote:
> Simon Riggs wrote:
> > 1. Notes on restartable recovery

Previously submitted

> > 2. Notes on standby functionality
> > 3. discussion on rolling your own record-level polling using
> > pg_xlogfile_name_offset()

Given below, but not in SGML yet. Looking for general pointers/feedback
before I drop those angle-brackets in place.


Warm Standby Servers for High Availability 
==========================================

Overview
--------

Continuous Archiving can also be used to create a High Availability (HA)
cluster configuration with one or more Standby Servers ready to take
over operations in the case that the Primary Server fails. This
capability is more widely known as Warm Standby Log Shipping.

The Primary and Standby Server work together to provide this capability,
though the servers are only loosely coupled. The Primary Server operates
in Continuous Archiving mode, while the Standby Server operates in a
continuous Recovery mode, reading the WAL files from the Primary. No
changes to the database tables are required to enable this capability,
so it offers a low administration overhead in comparison with other
replication approaches. This configuration also has a very low
performance impact on the Primary server.

Directly moving WAL or "log" records from one database server to another
is typically described as Log Shipping. PostgreSQL implements file-based
Log Shipping, meaning WAL records are batched one file at a time. WAL
files can be shipped easily and cheaply over any distance, whether it be
to an adjacent system, another system on the same site or another system
on the far side of the globe. The bandwidth required for this technique
varies according to the transaction rate of the Primary Server.
Record-based Log Shipping is also possible with custom-developed
procedures, discussed in a later section. Future developments are likely
to include options for synchronous and/or integrated record-based log
shipping.

It should be noted that the log shipping is asynchronous, i.e. the WAL
records are shipped after transaction commit. As a result there can be a
small window of data loss, should the Primary Server suffer a
catastrophic failure. The window of data loss is minimised by the use of
the archive_timeout parameter, which can be set as low as a few seconds
if required. A very low setting can increase the bandwidth requirements
for file shipping.

The Standby server is not available for access, since it is continually
performing recovery processing. Recovery performance is sufficiently
good that the Standby will typically be only minutes away from full
availability once it has been activated. As a result, we refer to this
capability as a Warm Standby configuration that offers High
Availability. Restoring a server from an archived base backup and
rollforward can take considerably longer and so that technique only
really offers a solution for Disaster Recovery, not HA.

Other mechanisms for High Availability replication are available, both
commercially and as open-source software.  

In general, log shipping between servers running different release
levels will not be possible. However, it may be possible for servers
running different minor release levels e.g. 8.2.1 and 8.2.2 to
inter-operate successfully. No formal support for that is offered and
there may be minor releases where that is not possible, so it is unwise
to rely on that capability.

Planning
--------

On the Standby server all tablespaces and paths will refer to similarly
named mount points, so it is important to create the Primary and Standby
servers so that they are as similar as possible, at least from the
perspective of the database server. Furthermore, any CREATE TABLESPACE
commands will be passed across as-is, so any new mount points must be
created on both servers before they are used on the Primary. Hardware
need not be the same, but experience shows that maintaining two
identical systems is easier than maintaining two dissimilar ones over
the whole lifetime of the application and system.

There is no special mode required to enable a Standby server. The
operations that occur on both Primary and Standby servers are entirely
normal continuous archiving and recovery tasks. The primary point of
contact between the two database servers is the archive of WAL files
that both share: Primary writing to the archive, Standby reading from
the archive. Care must be taken to ensure that WAL archives for separate
servers do not become mixed together or confused.

The magic that makes the two loosely coupled servers work together is
simply a restore_command that waits for the next WAL file to be archived
from the Primary. The restore_command is specified in the recovery.conf
file on the Standby Server. Normal recovery processing would request a
file from the WAL archive, causing an error if the file was unavailable.
For Standby processing it is normal for the next file to be unavailable,
so we must be patient and wait for it to appear. A waiting
restore_command can be written as a custom script that loops after
polling for the existence of the next WAL file. There must also be some
way to trigger failover, which should interrupt the restore_command,
break the loop and return a file not found error to the Standby Server.
This then ends recovery and the Standby will then come up as a normal
server.

Sample code for the C version of the restore_command would be be:

triggered = false;
while (!NextWALFileReady() && !triggered)
{for (i = 0; i < 10; i++){    sleep(100000L);        // wait for ~0.1 sec    if (CheckForExternalTrigger())    {
triggered= true;        break;    }}
 
}

if (!triggered)CopyWALFileForRecovery();

PostgreSQL does not provide the system software required to identify a
failure on the Primary and notify the Standby system and then the
Standby database server. Many such tools exist and are well integrated
with other aspects of a system failover, such as ip address migration.

Triggering failover is an important part of planning and design. The
restore_command is executed in full once for each WAL file. The process
running the restore_command is therefore created and dies for each file,
so there is no daemon or server process and so we cannot use signals and
a signal handler. A more permanent notification is required to trigger
the failover. It is possible to use a simple timeout facility,
especially if used in conjunction with a known archive_timeout setting
on the Primary. This is somewhat error prone since a network or busy
Primary server might be sufficient to initiate failover. A notification
mechanism such as the explicit creation of a trigger file is less error
prone, if this can be arranged.

Configuration
-------------

The short procedure for configuring a Standby Server is as follows. Full
details of each step are given previously in this chapter.

1. Set up Primary and Standby systems as near identically as possible,
including two identical copies of PostgreSQL at same release level.
2. Set up Continuous Archiving from the Primary to a WAL archive located
in a directory on the Standby Server. Ensure that both archive_command
and archive_delay are set.
3. Create a base backup of Primary Server
4. Restore the base backup onto the Standby Server
5. Begin recovery on the Standby Server from the local WAL archive,
using a recovery.conf that specifies a restore_command that waits as
described previously.

Recovery treats the WAL Archive as read-only, so once a WAL file has
been copied to the Standby system it can be copied to tape at the same
time as it is being used by the Standby database server to recover.
Thus, running a Standby Server for High Availability can be performed at
the same time as files are stored for longer term Disaster Recovery
purposes. 

For testing purposes, it is possible to run both Primary and Standby
servers on the same system. This does not provide any worthwhile
improvement on server robustness, nor would it be described as HA.

Failover
--------

If the Primary Server fails then the Standby Server should take begin
failover procedures.

If the Standby Server fails then no failover need take place. If the
Standby Server can be restarted, then the recovery process can also be
immediately restarted, taking advantage of Restartable Recovery.

If the Primary Server fails and then immediately restarts, you must have
a mechanism for informing it that it is no longer the Primary. This is
sometimes known as STONITH (Should the Other Node In The Head), which is
necessary to avoid situations where both systems think they are the
Primary, which can lead to confusion and ultimately data loss.

Many failover systems use just two systems, the Primary and the Standby,
connected by some kind of heartbeat mechanism to continually verify the
connectivity between the two and the viability of the Primary. It is
also possible to use a third system, known as a Witness Server to avoid
some problems of inappropriate failover, but the additional complexity
may not be worthwhile unless it is set-up with sufficient care and
rigorous testing.

At the instant that failover takes place to the Standby, we have only a
single server in operation. The former Standby is now the Primary, but
the former Primary is down and may stay down. We must now fully
re-create a Standby server, either on the former Primary system when it
comes up, or on a third, possibly new, system. Once complete the Primary
and Standby can be considered to have switched roles. Some people choose
to use a third server to provide additional protection across the
failover interval, though clearly this complicates the system
configuration and operational processes (and this can also act as a
Witness Server).

So, switching from Primary to Standby Server can be fast, but requires
some time to re-prepare the failover cluster. Regular switching from
Primary to Standby is encouraged, since it allows the regular downtime
one each system required to maintain HA. This also acts as a test of the
failover so that it definitely works when you really need it. Written
administration procedures are advised.

Record-based Log Shipping
-------------------------
The main features for Log Shipping in this release are based around the
file-based Log Shipping described above. It is also possible to
implement record-based Log Shipping using the pg_xlogfile_name_offset()
function, though this requires custom development.

An external program can call pg_xlogfile_name_offset() to find out the
filename and the exact byte offset within it of the latest WAL pointer.
If the external program regularly polls the server it can find out how
far forward the pointer has moved. It can then access the WAL file
directly and copy those bytes across to a less up-to-date copy on a
Standby Server.

--  Simon Riggs              EnterpriseDB   http://www.enterprisedb.com