Thread: Seeded Replication
Hi,
We are working on a project to set up replication from on premises Postgresql 10.4 to Azure Postgresql services. However, our database is about 200G and it will take a long time to complete the initial data copy. We would like to manually seed the subscriber database with data dump and then turn on the subscription like depesz showed in this post https://www.depesz.com/2017/02/07/waiting-for-postgresql-10-logical-replication/ . It works for the small testing database but when I applied same steps with entire database, I am getting "ERROR: duplicate key value violates unique constraint" errors. Basically postgresql does not recognize the primary key.
Any thought / suggestion is helpful!
| Lou Tseng |
| ltseng@advancedpricing.com |
 |
| Advanced Medical Pricing Solutions 35 Technology Parkway South, Suite. 100 Peachtree Corners, GA 30092 |
Hi Lou,
Did you restore data on the subscriber during the first time. If yes this error is likely to occur.
As a remedy, you need to restore only structural dump during initial building of subscription.
Earlier the same issue was faced by me later it got resolved following the above said approach.
Kindly revert back if any queries.
Regards,
Pavan Teja,
9841380956
On Wed, 3 Apr, 2019, 5:15 PM Lou Tseng, <ltseng@advancedpricing.com> wrote:
Hi,We are working on a project to set up replication from on premises Postgresql 10.4 to Azure Postgresql services. However, our database is about 200G and it will take a long time to complete the initial data copy. We would like to manually seed the subscriber database with data dump and then turn on the subscription like depesz showed in this post https://www.depesz.com/2017/02/07/waiting-for-postgresql-10-logical-replication/ . It works for the small testing database but when I applied same steps with entire database, I am getting "ERROR: duplicate key value violates unique constraint" errors. Basically postgresql does not recognize the primary key.Any thought / suggestion is helpful!
Lou Tseng ltseng@advancedpricing.com Advanced Medical Pricing Solutions
35 Technology Parkway South, Suite. 100
Peachtree Corners, GA 30092
Pavan,
This is the steps I took. I think I did it right:
[Master]
pg_dump -s -f schema.sql
pg_dump -a -f data.sql
CREATE PUBLICATION
[Client/Slave]
psql \i schema.sql
CREATE SUBSCRIPTION ... WITH (enabled = false)
SET session_replication_role = replica; -- turn off trigger
psql \i data.sql
ALTER SUBSCRIPTION ... ENABLE
--> errors occur here
Thoughts?
| Lou Tseng |
| ltseng@advancedpricing.com |
|
| Advanced Medical Pricing Solutions 35 Technology Parkway South, Suite. 100 Peachtree Corners, GA 30092 |
From: Pavan Teja <pavan.postgresdba@gmail.com>
Sent: Wednesday, April 3, 2019 6:48 AM
To: Lou Tseng
Cc: pgsql-general@postgresql.org >> PG-General Mailing List
Subject: Re: Seeded Replication
Sent: Wednesday, April 3, 2019 6:48 AM
To: Lou Tseng
Cc: pgsql-general@postgresql.org >> PG-General Mailing List
Subject: Re: Seeded Replication
Hi Lou,
Did you restore data on the subscriber during the first time. If yes this error is likely to occur.
As a remedy, you need to restore only structural dump during initial building of subscription.
Earlier the same issue was faced by me later it got resolved following the above said approach.
Kindly revert back if any queries.
Regards,
Pavan Teja,
9841380956
On Wed, 3 Apr, 2019, 5:15 PM Lou Tseng, <ltseng@advancedpricing.com> wrote:
Hi,We are working on a project to set up replication from on premises Postgresql 10.4 to Azure Postgresql services. However, our database is about 200G and it will take a long time to complete the initial data copy. We would like to manually seed the subscriber database with data dump and then turn on the subscription like depesz showed in this post https://www.depesz.com/2017/02/07/waiting-for-postgresql-10-logical-replication/ . It works for the small testing database but when I applied same steps with entire database, I am getting "ERROR: duplicate key value violates unique constraint" errors. Basically postgresql does not recognize the primary key.Any thought / suggestion is helpful!
Lou Tseng ltseng@advancedpricing.com Advanced Medical Pricing Solutions
35 Technology Parkway South, Suite. 100
Peachtree Corners, GA 30092
Thank you Adrian. copy_data = false did the trick!
| Lou Tseng |
| ltseng@advancedpricing.com |
|
| Advanced Medical Pricing Solutions 35 Technology Parkway South, Suite. 100 Peachtree Corners, GA 30092 |
From: Adrian Klaver <adrian.klaver@aklaver.com>
Sent: Wednesday, April 3, 2019 7:10 PM
To: Lou Tseng; pgsql-general@postgresql.org
Subject: Re: Seeded Replication
Sent: Wednesday, April 3, 2019 7:10 PM
To: Lou Tseng; pgsql-general@postgresql.org
Subject: Re: Seeded Replication
On 4/3/19 4:44 AM, Lou Tseng wrote:
> Hi,
>
> We are working on a project to set up replication from on premises
> Postgresql 10.4 to Azure Postgresql services. However, our database is
> about 200G and it will take a long time to complete the initial data
> copy. We would like to manually seed the subscriber database with data
> dump and then turn on the subscription like depesz showed in this post
> https://www.depesz.com/2017/02/07/waiting-for-postgresql-10-logical-replication/ .
> It works for the small testing database but when I applied same steps
> with entire database, I am getting "ERROR: duplicate key value violates
> unique constraint" errors. Basically postgresql does not recognize the
> primary key.
Actually it does which why you get the error, which by the way is the
same error depesz got in his demo:)
From your subsequent post:
CREATE SUBSCRIPTION ... WITH (enabled = false)
Did the above also have?:
https://www.postgresql.org/docs/11/sql-createsubscription.html
copy_data = false
If not then you would copying over the data again and that would trigger
the duplicate key error.
>
> Any thought / suggestion is helpful!
>
> Lou Tseng
>
> ltseng@advancedpricing.com <mailto:ltseng@advancedpricing.com>
> <http://www.advancedpricing.com/>
> Advanced Medical Pricing Solutions <http://advancedpricing.com/>
> 35 Technology Parkway South, Suite. 100
> Peachtree Corners, GA 30092
>
>
--
Adrian Klaver
adrian.klaver@aklaver.com
> Hi,
>
> We are working on a project to set up replication from on premises
> Postgresql 10.4 to Azure Postgresql services. However, our database is
> about 200G and it will take a long time to complete the initial data
> copy. We would like to manually seed the subscriber database with data
> dump and then turn on the subscription like depesz showed in this post
> https://www.depesz.com/2017/02/07/waiting-for-postgresql-10-logical-replication/ .
> It works for the small testing database but when I applied same steps
> with entire database, I am getting "ERROR: duplicate key value violates
> unique constraint" errors. Basically postgresql does not recognize the
> primary key.
Actually it does which why you get the error, which by the way is the
same error depesz got in his demo:)
From your subsequent post:
CREATE SUBSCRIPTION ... WITH (enabled = false)
Did the above also have?:
https://www.postgresql.org/docs/11/sql-createsubscription.html
copy_data = false
If not then you would copying over the data again and that would trigger
the duplicate key error.
>
> Any thought / suggestion is helpful!
>
> Lou Tseng
>
> ltseng@advancedpricing.com <mailto:ltseng@advancedpricing.com>
> <http://www.advancedpricing.com/>
> Advanced Medical Pricing Solutions <http://advancedpricing.com/>
> 35 Technology Parkway South, Suite. 100
> Peachtree Corners, GA 30092
>
>
--
Adrian Klaver
adrian.klaver@aklaver.com