pgsql: Add support for prepared transactions to built-in logical replic - Mailing list pgsql-committers

Add support for prepared transactions to built-in logical replication.

To add support for streaming transactions at prepare time into the
built-in logical replication, we need to do the following things:

* Modify the output plugin (pgoutput) to implement the new two-phase API
callbacks, by leveraging the extended replication protocol.

* Modify the replication apply worker, to properly handle two-phase
transactions by replaying them on prepare.

* Add a new SUBSCRIPTION option "two_phase" to allow users to enable
two-phase transactions. We enable the two_phase once the initial data sync
is over.

We however must explicitly disable replication of two-phase transactions
during replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover, we don't have a replication connection open so we don't know
where to send the data anyway.

The streaming option is not allowed with this new two_phase option. This
can be done as a separate patch.

We don't allow to toggle two_phase option of a subscription because it can
lead to an inconsistent replica. For the same reason, we don't allow to
refresh the publication once the two_phase is enabled for a subscription
unless copy_data option is false.

Author: Peter Smith, Ajin Cherian and Amit Kapila based on previous work by Nikhil Sontakke and Stas Kelvich
Reviewed-by: Amit Kapila, Sawada Masahiko, Vignesh C, Dilip Kumar, Takamichi Osumi, Greg Nancarrow
Tested-By: Haiying Tang
Discussion: https://postgr.es/m/02DA5F5E-CECE-4D9C-8B4B-418077E2C010@postgrespro.ru
Discussion: https://postgr.es/m/CAA4eK1+opiV4aFTmWWUF9h_32=HfPOW9vZASHarT0UA5oBrtGw@mail.gmail.com

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/a8fd13cab0ba815e9925dc9676e6309f699b5f72

Modified Files
--------------
contrib/test_decoding/test_decoding.c              |  12 +-
doc/src/sgml/catalogs.sgml                         |  12 +
doc/src/sgml/protocol.sgml                         | 291 ++++++++++++++++-
doc/src/sgml/ref/alter_subscription.sgml           |   5 +
doc/src/sgml/ref/create_subscription.sgml          |  37 +++
doc/src/sgml/ref/pg_dump.sgml                      |   7 +-
src/backend/access/transam/twophase.c              |  68 ++++
src/backend/catalog/pg_subscription.c              |  34 ++
src/backend/catalog/system_views.sql               |   2 +-
src/backend/commands/subscriptioncmds.c            | 131 +++++++-
.../libpqwalreceiver/libpqwalreceiver.c            |  10 +-
src/backend/replication/logical/decode.c           |  11 +-
src/backend/replication/logical/logical.c          |  31 +-
src/backend/replication/logical/origin.c           |   7 +-
src/backend/replication/logical/proto.c            | 217 ++++++++++++-
src/backend/replication/logical/reorderbuffer.c    |  25 +-
src/backend/replication/logical/snapbuild.c        |  33 +-
src/backend/replication/logical/tablesync.c        | 197 +++++++++--
src/backend/replication/logical/worker.c           | 347 +++++++++++++++++++-
src/backend/replication/pgoutput/pgoutput.c        | 201 +++++++++---
src/backend/replication/slot.c                     |   1 +
src/backend/replication/walreceiver.c              |   2 +-
src/bin/pg_dump/pg_dump.c                          |  20 +-
src/bin/pg_dump/pg_dump.h                          |   1 +
src/bin/psql/describe.c                            |   8 +-
src/bin/psql/tab-complete.c                        |   2 +-
src/include/access/twophase.h                      |   2 +
src/include/catalog/catversion.h                   |   2 +-
src/include/catalog/pg_subscription.h              |  11 +
src/include/catalog/pg_subscription_rel.h          |   1 +
src/include/replication/logical.h                  |  10 +
src/include/replication/logicalproto.h             |  73 ++++-
src/include/replication/pgoutput.h                 |   1 +
src/include/replication/reorderbuffer.h            |   8 +-
src/include/replication/slot.h                     |   7 +-
src/include/replication/snapbuild.h                |   5 +-
src/include/replication/walreceiver.h              |   7 +-
src/include/replication/worker_internal.h          |   3 +
src/test/regress/expected/subscription.out         | 109 ++++---
src/test/regress/sql/subscription.sql              |  25 ++
src/test/subscription/t/021_twophase.pl            | 359 +++++++++++++++++++++
src/test/subscription/t/022_twophase_cascade.pl    | 235 ++++++++++++++
src/tools/pgindent/typedefs.list                   |   3 +
43 files changed, 2382 insertions(+), 191 deletions(-)