Infrastructure changes for recovery (v8) - Mailing list pgsql-hackers

Patch now includes all previous agreed changes, plus I've found what
looks to be a workable method of removing the shutdown checkpoint
without loss of robustness.

Patch summary

Tuning
* Bgwriter performs dirty block cleaning during recovery
* Bgwriter performs restartpoints, offloading this task from Startup
process to allow it to continue with recovery actions
* Shutdown checkpoint removed at end of recovery. Bgwriter performs
immediate checkpoint instead, so we have same protection, but
connections and transactions can be started earlier than previously.
* PreAllocXLogs() not performed by startup process, so we do not delay
startup while we write zeroes to next WAL file. bgwriter does that now.
* XLogCtl structure padding for enhanced scalability

Recovery State Changes
* If archive recovery proceeds past a safe stopping point we signal the
postmaster that database is now in a consistent state, PM_RECOVERY. This
state change is also linked to startup of the bgwriter and stats
processes (and will in the future be the place where read only backends
may connect also)
* optional recovery_safe_start_location parameter now provided in
recovery.conf, to allow a consistency point to be manually defined if a
base backup was not taken using standard pg_start/stop backup functions
* New minSafeStopPoint added to controlfile to allow us to determine
consistency if archive recovery crashes/restarts. Value is updated each
time we access new WAL file.
* stats file removed earlier in recovery, so we may accumulate new stats
during recovery
* End of recovery is now marked by a clear global state change. Change
is global, atomic and fast - tested for using IsRecoveryProcessingMode()

Additional Safeguards
* Locks are placed around all ControlFile operations
* XLogInsert() and AssignTransactionId() now have specific checks to
prevent their use during recovery
* Makes StartupMultiXact() atomic. Adds comments to show that
StartCLOG() is already atomic, though StartupSUBTRANS() is not (this
will be addressed in a later patch, so not touched here)
* recovery.conf is not removed until slightly later now, to protect
against crash at the end of startup
* New WAL record XLOG_RECOVERY_END is now only place where timelineid
may change

Other Changes
* log_restartpoints removed, use log_checkpoints in postgresql.conf
* pg_controldata and pg_resetxlog changed to show safe start point
* designed to work in EXEC_BACKEND mode for Windows
* additional function signature for pg_start_backup('label', true |
false) to allow definition of immediate checkpoint/not
* doc changes for recovery.conf parameters
* fixes bug discovered while other testing: if pg_stop_backup() is run
when xlogswitch has just occurred then we do not switch log files, yet
we return current filename even though nothing of value in it. If
archive_timeout not enabled we would wait forever for pg_stop_backup()
to return.
* Substantial comments throughout

Patch is now v8.

 doc/src/sgml/backup.sgml                 |   30 !
 doc/src/sgml/func.sgml                   |   12
 src/backend/access/transam/clog.c        |    3
 src/backend/access/transam/multixact.c   |   14
 src/backend/access/transam/subtrans.c    |    3
 src/backend/access/transam/xact.c        |    3
 src/backend/access/transam/xlog.c        |  783 ++++++++++++++-!!!!!!!!!!!!!!!
 src/backend/postmaster/bgwriter.c        |  418 +++--!!!!!!!!!
 src/backend/postmaster/postmaster.c      |   62 +!
 src/backend/storage/buffer/README        |    9
 src/bin/pg_controldata/pg_controldata.c  |    3
 src/bin/pg_resetxlog/pg_resetxlog.c      |    2
 src/include/access/xlog.h                |   14
 src/include/access/xlog_internal.h       |    4
 src/include/catalog/pg_control.h         |    3
 src/include/postmaster/bgwriter.h        |    6
 src/include/storage/pmsignal.h           |    1
 src/test/regress/expected/opr_sanity.out |    7
 18 files changed, 579 insertions(+), 79 deletions(-), 719 modifications(!)

Please review everybody. Many thanks.

--
 Simon Riggs           www.2ndQuadrant.com
 PostgreSQL Training, Services and Support