Re: Logical decoding without slots: decoding in lockstep with recovery - Mailing list pgsql-hackers

From Amit Kapila
Subject Re: Logical decoding without slots: decoding in lockstep with recovery
Date
Msg-id CAA4eK1KAYH4hZFoLESMdyMACD5TjqbGJhZ0JaPVHkcSRYv9g3A@mail.gmail.com
Whole thread Raw
In response to Logical decoding without slots: decoding in lockstep with recovery  (Craig Ringer <craig.ringer@enterprisedb.com>)
Responses Re: Logical decoding without slots: decoding in lockstep with recovery
List pgsql-hackers
On Wed, Dec 23, 2020 at 12:26 PM Craig Ringer
<craig.ringer@enterprisedb.com> wrote:
>
> Hi all
>
> I want to share an idea I've looked at a few times where I've run into situations where logical slots were
inadvertentlydropped, or where it became necessary to decode changes in the past on a slot.
 
>
> As most of you will know you can't just create a logical slot in the past. Even if it was permitted, it'd be unsafe
dueto catalog_xmin retention requirements and missing WAL.
 
>
> But if we can arrange a physical replica to replay the WAL of interest and decode each commit as soon as it's
replayedby the startup process, we know the needed catalog rows must all exist, so it's safe to decode the change.
 
>
> So it should be feasible to run logical decoding in standby, even without a replication slot, so long as we:
>
> * pause startup process after each xl_xact_commit
> * wake the walsender running logical decoding
> * decode and process until ReorderBufferCommit for the just-committed xact returns
> * wake the startup process to decode the up to the next commit
>

How will you deal with subscriber restart?  I think you need some way
to remember confirmed_flush_lsn and restart_lsn and then need to teach
WAL machinery to restart from some previous point.

-- 
With Regards,
Amit Kapila.



pgsql-hackers by date:

Previous
From: Masahiko Sawada
Date:
Subject: Re: Deadlock between backend and recovery may not be detected
Next
From: Pavel Borisov
Date:
Subject: Re: [PATCH] Automatic HASH and LIST partition creation