Hot Standby (commit fest version - v5) - Mailing list pgsql-hackers

Hot Standby patch, including all major planned features.

Allows users to connect to server in archive recovery and run queries.

Applies cleanly, passes make check.

There's no flaky code, kludges or other last minute rush-for-deadline
stuff. "It all works", though really for a patch this size and scope I
expect many bugs. As a result I am describing this as "WIP", though it
is much more than a prototype. All the code has been planned out in
advance, so there's been very little on-the-fly changes required during
final development. I'm fully committed to making all required changes
and fixes in reasonable times. I will continue detailed testing over the
next few weeks to re-check everything prior to commit.

Initially, I would ask people to spend time thinking about this
conceptually to check that I have all the correct subsystems and cater
for all the little side tweaks that exist in various parts of the server

When you test this, please do it on a server built with --enable-cassert
and keep a detailed log using trace_recovery_messages = DEBUG4 (or 2-3).

Code has been specifically designed to be performance neutral or better
for normal workloads, so the WAL volume, number of times we take WAL
locks etc should be identical (on 64-bit systems). The patch includes
further tuning of subtransaction commits, so I hope the patch may even
be a few % win on more complex workloads with many PL/pgSQL functions
using EXCEPTION. Enabling the bgwriter during recovery seems likely to
be a huge gain on performance for larger shared_buffers settings, which
should offset a considerable increase in CPU usage during recovery.

Performance test results would be appreciated
* for normal running - just to test that it really is neutral
* for recovery - if query access not used is it faster than before

Points of significant note for detailed reviewers

* Prepared transactions not implemented yet. No problems foreseen, but
want to wait to see if other refactorings are required.

* Touching every block of a btree index during a replay of VACUUM is not
yet implemented. Would like some second opinions that it is even
required. I have code prototyped for it, but it feels too wacky even
after Heikki and I agreed it was required. Do we still think that?

* locking correctness around flat file refresh still not enabled

* need some discussiona round how to handle max_connections changes
cleanly.

Overview of code changes
========================

Docs
 doc/src/sgml/config.sgml                  |   26
 doc/src/sgml/func.sgml                    |  191 +++
 src/backend/storage/buffer/README         |    9

Transaction emulation in recovery
 src/backend/access/transam/xact.c         | 1022 ++++++++++++++++++-
 src/backend/storage/ipc/procarray.c       |  868 +++++++++++++++!
 src/backend/utils/time/tqual.c            |   62 !
 src/backend/access/heap/pruneheap.c       |   18
 src/backend/access/transam/clog.c         |    3
 src/backend/access/transam/multixact.c    |   14
 src/backend/access/transam/slru.c         |   16
 src/backend/access/transam/subtrans.c     |   27
 src/backend/access/transam/twophase.c     |    4
 src/backend/access/transam/xlogutils.c    |   18
 src/backend/storage/lmgr/proc.c           |  110 ++
 src/include/storage/proc.h                |   15
 src/include/storage/procarray.h           |   29
 src/include/access/heapam.h               |    4
 src/include/access/htup.h                 |   18
 src/include/access/xact.h                 |   56 +
 src/include/access/xlog.h                 |   54 !
 src/include/access/xlog_internal.h        |   16
 src/include/access/xlogutils.h            |    3
 src/include/storage/bufmgr.h              |    7
 src/include/utils/snapshot.h              |   82 +

Locking and invalidation in recovery
 src/backend/utils/cache/inval.c           |  581 +++++++++++
 src/backend/access/transam/rmgr.c         |    3
 src/backend/storage/ipc/sinvaladt.c       |   10
 src/backend/storage/lmgr/lock.c           |   44
 src/backend/utils/init/flatfiles.c        |   55
 src/backend/utils/init/postinit.c         |   10
 src/include/storage/sinval.h              |   38
 src/include/storage/sinvaladt.h           |    2
 src/include/utils/flatfiles.h             |    9
 src/include/utils/inval.h                 |    4
 src/include/access/rmgr.h                 |    1

Cleanup in recovery
 src/backend/access/heap/heapam.c          |   88 +
 src/backend/commands/vacuum.c             |   29
 src/backend/commands/vacuumlazy.c         |   42

State changes and bgwriter
 src/backend/access/transam/xlog.c         | 1533 +++++++--!!!!!!!!!!!!!
 src/backend/postmaster/bgwriter.c         |  418 +-!!!!
 src/backend/postmaster/postmaster.c       |   97 +
 src/backend/storage/buffer/bufmgr.c       |   58 +
 src/include/miscadmin.h                   |    6
 src/include/postmaster/bgwriter.h         |    6
 src/include/catalog/pg_control.h          |   15
 src/include/storage/pmsignal.h            |    1

Protections and commands refused
 src/backend/commands/discard.c            |    3
 src/backend/commands/indexcmds.c          |    2
 src/backend/commands/lockcmds.c           |   12
 src/backend/commands/sequence.c           |    2
 src/backend/tcop/postgres.c               |    2
 src/backend/tcop/utility.c                |   20
 src/backend/utils/adt/txid.c              |    6

New functions (in xlog.c)
 src/include/catalog/pg_proc.h             |   22

API changes knock-on effects
 src/backend/bootstrap/bootstrap.c         |    3
 src/backend/storage/freespace/freespace.c |    5
 src/bin/pg_controldata/pg_controldata.c   |    3
 src/bin/pg_resetxlog/pg_resetxlog.c       |    2

New parameters
 src/backend/utils/error/elog.c            |   17
 src/backend/utils/misc/guc.c              |   12

 src/test/regress/parallel_schedule        |    4
 src/test/regress/serial_schedule          |    2
 62 files changed, 4114 insertions(+), 139 deletions(-), 1586
modifications(!)

--
 Simon Riggs           www.2ndQuadrant.com
 PostgreSQL Training, Services and Support