Logical replication troubles - Mailing list pgsql-general

From Anders Bøgh Bruun
Subject Logical replication troubles
Date
Msg-id CAMnwvPwBgkmaMZapP_wgXZwxGZymuaYuqNykGLiMG4K+rGmBaQ@mail.gmail.com
Whole thread Raw
Responses Re: Logical replication troubles
List pgsql-general
Hi,

I have run into a (to me) weird issue with logical replication. We are running Zalandos postgres-operator in our Kubernetes clusters and have recently had a use-case where we wanted to start doing logical replication of select tables to a data warehouse, also running postgres. It worked as expected at first, but then after a pod-restart in Kubernetes, the replication slots that were created for the subscription were gone. A bit of reading later, and I learn we need to tell Patroni which slots should be permanently available, so we specify a slot and try to set this up, but then run into an error which says the publication does not exist, even though we can verify that it does. At first I suspected Patroni handling the replication slots to be the cause of the problem, but about a week's worth of learning and experimenting later, I can now reliably replicate the problem in pure postgres. Patroni is kind of the catalyst, since my findings are that if the replication slot is created before data is inserted into the source database, and a publication is created, then it breaks. If the replication slot is created after data is inserted and the publication is created, then it works. We just can't tell Patroni to not create it until some arbitrary point in time. I am guessing this is either a bug or a case of us not knowing what we are doing...

I have created a Github gist demonstrating the problem: https://gist.github.com/drzero42/02b4082ce002c1d90ddd64f5fe03aee0

Anybody able to help? :)

--
Anders Bøgh Bruun

Infrastructure Architect

CellPoint digital
cellpointdigital.com
WE MAKE TRAVEL EASIER™

M: +45 31 14 87 41
E: anders@cellpointdigital.com

Chicago | Copenhagen | Dubai | London | Miami | Pune | Singapore

pgsql-general by date:

Previous
From: Tory M Blue
Date:
Subject: Huge tables, trying to delete OID's taking 6+hours per table
Next
From: Tory M Blue
Date:
Subject: Re: Huge tables, trying to delete OID's taking 6+hours per table