Re: CDC/ETL system on top of logical replication with pgoutput, custom client - Mailing list pgsql-hackers

From Ashutosh Bapat
Subject Re: CDC/ETL system on top of logical replication with pgoutput, custom client
Date
Msg-id CAExHW5ttk51J5taF6vJUF5sSrgf=TxSF134tA4Q5+KMwLZ=PXQ@mail.gmail.com
Whole thread Raw
In response to RE: CDC/ETL system on top of logical replication with pgoutput, custom client  (José Neves <rafaneves3@msn.com>)
Responses RE: CDC/ETL system on top of logical replication with pgoutput, custom client
List pgsql-hackers
On Tue, Oct 24, 2023 at 8:53 PM José Neves <rafaneves3@msn.com> wrote:
>
> Hi there, hope to find you well.
>
> I have a follow-up question to this already long thread.
>
> Upon deploying my PostgreSQL logical replication fed application on a stale database, I ended up running out of
space,as the replication slot is being held back till the next time that we receive a data-changing event, and we
advanceto that new LSN offset. 
> I think that the solution for this is to advance our LSN offset every time a keep-alive message is received ('k' //
107).
> My doubt is, can the keep-alive messages be received in between open transaction events? I think not, but I would
liketo get your input to be extra sure as if this happens, and I commit that offset, I may introduce again faulty logic
leadingto data loss. 
>
> In sum, something like this wouldn't happen:
> BEGIN LSN001
> INSERT LSN002
> KEEP LIVE LSN003
> UPDATE LSN004
> COMMIT LSN005
>

If the downstream acknowledges receipt of LSN003 and saves it locally
and crashes, upon restart the upstream will resend all the
transactions that committed after LSN003 including the one ended at
LSN005. So this is safe.

--
Best Wishes,
Ashutosh



pgsql-hackers by date:

Previous
From: Smolkin Grigory
Date:
Subject: race condition in pg_class
Next
From: José Neves
Date:
Subject: RE: CDC/ETL system on top of logical replication with pgoutput, custom client