Re: confirmed_flush_lsn shows LSN of the data that has not yet been received by the logical subscriber. - Mailing list pgsql-hackers

On Thu, Sep 8, 2022 at 8:32 PM Ashutosh Sharma <ashu.coek88@gmail.com> wrote:
>
> On Thu, Sep 8, 2022 at 6:23 PM Ashutosh Bapat
> <ashutosh.bapat.oss@gmail.com> wrote:
> >
> > On Thu, Sep 8, 2022 at 4:14 PM Ashutosh Sharma <ashu.coek88@gmail.com> wrote:
> > >
> > > Hi All,
> > >
> > > The logically decoded data are sent to the logical subscriber at the time of transaction commit, assuming that
thedata is small. However, before the transaction commit is performed, the LSN representing the data that is yet to be
receivedby the logical subscriber appears in the confirmed_flush_lsn column of pg_replication_slots catalog. Isn't the
informationseen in the confirmed_flush_lsn column while the transaction is in progress incorrect ? esp considering the
descriptiongiven in the pg doc for this column. 
> > >
> > > Actually, while the transaction is running, the publisher keeps on sending keepalive messages containing LSN of
thelast decoded data saved in reorder buffer and the subscriber responds with the same LSN as the last received LSN
whichis then updated as confirmed_flush_lsn by the publisher. I think the LSN that we are sending with the keepalive
messageshould be the one representing the transaction begin message, not the LSN of the last decoded data which is yet
tobe sent. Please let me know if I am missing something here. 
> >
> > The transactions with commit lsn < confirmed_flush_lsn are confirmed
> > to be received (and applied by the subscriber. Setting LSN
> > corresponding to a WAL record within a transaction in progress as
> > confirmed_flush should be ok. Since the transactions are interleaved
> > in WAL stream, it's quite possible that LSNs of some WAL records of an
> > inflight transaction are lesser than commit LSN of some another
> > transaction. So setting commit LSN of another effectively same as
> > setting it to any of the LSNs of any previous WAL record irrespective
> > of the transaction that it belongs to.
>
> Thank you Ashutosh for the explanation. I still feel that the
> documentation on confirmed_flush_lsn needs some improvement. It
> actually claims that all the data before the confirmed_flush_lsn has
> been received by the logical subscriber, but that's not the case. It
> actually means that all the data belonging to the transactions with
> commit lsn < confirmed_flush_lsn has been received and applied by the
> subscriber. So setting confirmed_flush_lsn to the lsn of wal records
> generated by running transaction might make people think that the wal
> records belonging to previous data of the same running transaction has
> already been received and applied by the subscriber node, but that's
> not true.
>

Can you please point to the documentation.

It's true that it needs to be clarified. But what you are saying may
not be entirely true in case of streamed transaction. In that case we
might send logically decoded changes of an ongoing transaction as
well. They may even get applied but not necessarily committed. It's a
bit complicated. :)

--
Best Wishes,
Ashutosh Bapat