Thread: [COMMITTERS] pgsql: Logical replication support for initial data copy
Logical replication support for initial data copy Add functionality for a new subscription to copy the initial data in the tables and then sync with the ongoing apply process. For the copying, add a new internal COPY option to have the COPY source data provided by a callback function. The initial data copy works on the subscriber by receiving COPY data from the publisher and then providing it locally into a COPY that writes to the destination table. A WAL receiver can now execute full SQL commands. This is used here to obtain information about tables and publications. Several new options were added to CREATE and ALTER SUBSCRIPTION to control whether and when initial table syncing happens. Change pg_dump option --no-create-subscription-slots to --no-subscription-connect and use the new CREATE SUBSCRIPTION ... NOCONNECT option for that. Author: Petr Jelinek <petr.jelinek@2ndquadrant.com> Tested-by: Erik Rijkers <er@xs4all.nl> Branch ------ master Details ------- http://git.postgresql.org/pg/commitdiff/7c4f52409a8c7d85ed169bbbc1f6092274d03920 Modified Files -------------- contrib/file_fdw/file_fdw.c | 5 +- doc/src/sgml/catalogs.sgml | 78 ++ doc/src/sgml/config.sgml | 25 + doc/src/sgml/logical-replication.sgml | 55 +- doc/src/sgml/monitoring.sgml | 9 +- doc/src/sgml/protocol.sgml | 9 +- doc/src/sgml/ref/alter_subscription.sgml | 50 +- doc/src/sgml/ref/create_subscription.sgml | 38 + doc/src/sgml/ref/pg_dump.sgml | 15 +- src/backend/catalog/Makefile | 1 + src/backend/catalog/heap.c | 6 + src/backend/catalog/pg_publication.c | 4 +- src/backend/catalog/pg_subscription.c | 282 +++++++ src/backend/catalog/system_views.sql | 1 + src/backend/commands/copy.c | 23 +- src/backend/commands/subscriptioncmds.c | 468 ++++++++++-- src/backend/parser/gram.y | 39 +- src/backend/postmaster/pgstat.c | 6 + .../libpqwalreceiver/libpqwalreceiver.c | 203 ++++- src/backend/replication/logical/Makefile | 2 +- src/backend/replication/logical/launcher.c | 130 +++- src/backend/replication/logical/relation.c | 7 + src/backend/replication/logical/snapbuild.c | 85 ++- src/backend/replication/logical/tablesync.c | 840 +++++++++++++++++++++ src/backend/replication/logical/worker.c | 203 +++-- src/backend/replication/repl_gram.y | 32 +- src/backend/replication/repl_scanner.l | 5 +- src/backend/replication/walsender.c | 104 ++- src/backend/tcop/postgres.c | 5 +- src/backend/utils/adt/misc.c | 20 + src/backend/utils/cache/syscache.c | 14 +- src/backend/utils/misc/guc.c | 12 + src/bin/pg_dump/pg_backup.h | 2 +- src/bin/pg_dump/pg_dump.c | 9 +- src/bin/pg_dump/t/002_pg_dump.pl | 2 +- src/include/catalog/catversion.h | 2 +- src/include/catalog/indexing.h | 7 +- src/include/catalog/pg_proc.h | 5 +- src/include/catalog/pg_subscription_rel.h | 78 ++ src/include/commands/copy.h | 5 +- src/include/nodes/nodes.h | 1 + src/include/nodes/parsenodes.h | 13 + src/include/nodes/replnodes.h | 9 + src/include/parser/kwlist.h | 1 + src/include/pgstat.h | 4 +- src/include/replication/logical.h | 13 +- src/include/replication/logicallauncher.h | 1 + src/include/replication/snapbuild.h | 1 + src/include/replication/walreceiver.h | 67 +- src/include/replication/walsender.h | 12 +- src/include/replication/worker_internal.h | 30 +- src/include/utils/syscache.h | 1 + src/test/regress/expected/object_address.out | 3 +- src/test/regress/expected/rules.out | 3 +- src/test/regress/expected/sanity_check.out | 1 + src/test/regress/expected/subscription.out | 45 +- src/test/regress/sql/object_address.sql | 2 +- src/test/regress/sql/subscription.sql | 11 +- src/test/subscription/t/001_rep_changes.pl | 36 +- src/test/subscription/t/002_types.pl | 6 + src/test/subscription/t/003_constraints.pl | 2 +- src/test/subscription/t/004_sync.pl | 159 ++++ 62 files changed, 2966 insertions(+), 341 deletions(-)
I am looking into buildfarm failures. Looks like the clang issue (mylodon, longfin) is because we are missing extern in include/replication/worker_internal.h for ApplyCacheContext. Still analyzing the rest. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On 23/03/17 14:47, Petr Jelinek wrote: > I am looking into buildfarm failures. > > Looks like the clang issue (mylodon, longfin) is because we are missing > extern in include/replication/worker_internal.h for ApplyCacheContext. > > Still analyzing the rest. > Ah missing fields in copy/equal function for AlterSubscriptionStmt. Attached patch adds the above mentioned extern and fixes the copy and equal functions. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Attachment
Petr, * Petr Jelinek (petr.jelinek@2ndquadrant.com) wrote: > On 23/03/17 14:47, Petr Jelinek wrote: > > I am looking into buildfarm failures. > > > > Looks like the clang issue (mylodon, longfin) is because we are missing > > extern in include/replication/worker_internal.h for ApplyCacheContext. > > > > Still analyzing the rest. > > Ah missing fields in copy/equal function for AlterSubscriptionStmt. > > Attached patch adds the above mentioned extern and fixes the copy and > equal functions. Pretty sure I saw a complaint about a trailing comma in a struct or enum too.. Thanks! Stephen
Attachment
On 23/03/17 15:17, Stephen Frost wrote: > Petr, > > * Petr Jelinek (petr.jelinek@2ndquadrant.com) wrote: >> On 23/03/17 14:47, Petr Jelinek wrote: >>> I am looking into buildfarm failures. >>> >>> Looks like the clang issue (mylodon, longfin) is because we are missing >>> extern in include/replication/worker_internal.h for ApplyCacheContext. >>> >>> Still analyzing the rest. >> >> Ah missing fields in copy/equal function for AlterSubscriptionStmt. >> >> Attached patch adds the above mentioned extern and fixes the copy and >> equal functions. > > Pretty sure I saw a complaint about a trailing comma in a struct or enum > too.. > Ah, missed that in the wall of text, thanks, fixed that too. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Attachment
Re: [COMMITTERS] pgsql: Logical replication support for initial datacopy
From
Peter Eisentraut
Date:
On 3/23/17 10:12, Petr Jelinek wrote: > On 23/03/17 14:47, Petr Jelinek wrote: >> I am looking into buildfarm failures. >> >> Looks like the clang issue (mylodon, longfin) is because we are missing >> extern in include/replication/worker_internal.h for ApplyCacheContext. >> >> Still analyzing the rest. >> > Ah missing fields in copy/equal function for AlterSubscriptionStmt. > > Attached patch adds the above mentioned extern and fixes the copy and > equal functions. Fixed. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: [COMMITTERS] pgsql: Logical replication support for initial datacopy
From
Peter Eisentraut
Date:
On 3/23/17 10:25, Petr Jelinek wrote: > On 23/03/17 15:17, Stephen Frost wrote: >> Petr, >> >> * Petr Jelinek (petr.jelinek@2ndquadrant.com) wrote: >>> On 23/03/17 14:47, Petr Jelinek wrote: >>>> I am looking into buildfarm failures. >>>> >>>> Looks like the clang issue (mylodon, longfin) is because we are missing >>>> extern in include/replication/worker_internal.h for ApplyCacheContext. >>>> >>>> Still analyzing the rest. >>> >>> Ah missing fields in copy/equal function for AlterSubscriptionStmt. >>> >>> Attached patch adds the above mentioned extern and fixes the copy and >>> equal functions. >> >> Pretty sure I saw a complaint about a trailing comma in a struct or enum >> too.. >> > > Ah, missed that in the wall of text, thanks, fixed that too. Also fixed. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Fri, Mar 24, 2017 at 12:59 AM, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote: > On 3/23/17 10:25, Petr Jelinek wrote: >> On 23/03/17 15:17, Stephen Frost wrote: >>> Petr, >>> >>> * Petr Jelinek (petr.jelinek@2ndquadrant.com) wrote: >>>> On 23/03/17 14:47, Petr Jelinek wrote: >>>>> I am looking into buildfarm failures. >>>>> >>>>> Looks like the clang issue (mylodon, longfin) is because we are missing >>>>> extern in include/replication/worker_internal.h for ApplyCacheContext. >>>>> >>>>> Still analyzing the rest. >>>> >>>> Ah missing fields in copy/equal function for AlterSubscriptionStmt. >>>> >>>> Attached patch adds the above mentioned extern and fixes the copy and >>>> equal functions. >>> >>> Pretty sure I saw a complaint about a trailing comma in a struct or enum >>> too.. >>> >> >> Ah, missed that in the wall of text, thanks, fixed that too. > > Also fixed. > The copyright in some files(tablesync.c and pg_subscriptiion_rel.h) introduced by this commit are old. Is that intentional? Attached patch. Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center