Re: Skipping logical replication transactions on subscriber side - Mailing list pgsql-hackers

From Amit Kapila
Subject Re: Skipping logical replication transactions on subscriber side
Date
Msg-id CAA4eK1J6VNwofqvs+kLCdf_wWztFLzet0d0N64xOoyx9DgivAw@mail.gmail.com
Whole thread Raw
In response to Re: Skipping logical replication transactions on subscriber side  (Masahiko Sawada <sawada.mshk@gmail.com>)
Responses Re: Skipping logical replication transactions on subscriber side
List pgsql-hackers
On Wed, Jan 26, 2022 at 8:55 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Wed, Jan 26, 2022 at 11:51 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Wed, Jan 26, 2022 at 11:28 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > IIUC, the proposal is to compare the skip_xid with the very
> > > transaction the apply worker received to apply and raise a warning if
> > > it doesn't match with skip_xid and then continue. This seems like a
> > > reasonable idea but can we guarantee that it is always the first
> > > transaction that we want to skip? We seem to guarantee that we won't
> > > get something again once it is written durably/flushed on the
> > > subscriber side. I guess here it can happen that before the errored
> > > transaction, there is some empty xact, or maybe part of the stream
> > > (consider streaming transactions) of some xact, or there could be
> > > other cases as well where the server will send those xacts again.
> >
> > Good point.
> >
> > I guess that in the situation the worker entered an error loop, we can
> > guarantee that the worker fails while applying the first non-empty
> > transaction since starting logical replication. And the transaction is
> > what we’d like to skip. If the transaction that can be applied without
> > an error is resent after a restart, it’s a problem of logical
> > replication. As you pointed out, it's possible that there are some
> > empty transactions before the transaction in question since we don't
> > advance replication origin LSN if the transaction is empty. Also,
> > probably the same is true for a streamed transaction that is rolled
> > back or ROLLBACK-PREPARED transactions. So, we can also skip clearing
> > subskipxid if the transaction is empty? That is, we make sure to clear
> > it after applying the first non-empty transaction. We would need to
> > carefully think about this solution otherwise ALTER SUBSCRIPTION SKIP
> > ends up not working at all in some cases.

I think it is okay to clear after the first successful application of
any transaction. What I was not sure was about the idea of giving
WARNING/ERROR if the first xact to be applied is not the same as
skip_xid.

>
> Probably, we also need to consider the case where the tablesync worker
> entered an error loop and the user wants to skip the transaction? The
> apply worker is also running at the same time but it should not clear
> subskipxid. Similarly, the tablesync worker should not clear
> subskipxid if the apply worker wants to skip the transaction.
>

I think for tablesync workers, the skip_xid set via this mechanism
won't work as we don't have any remote_xid for them, and neither any
XID is reported in the view for them.

--
With Regards,
Amit Kapila.



pgsql-hackers by date:

Previous
From: "wangw.fnst@fujitsu.com"
Date:
Subject: RE: Logical replication timeout problem
Next
From: Masahiko Sawada
Date:
Subject: Re: Skipping logical replication transactions on subscriber side