Postgres Pro
Downloads
Home   >   mailing lists

Thread: Local visibility with logical decoding

Local visibility with logical decoding

From
Marko Tiikkaja
Date:
Hi,

It appears that when logical decoding sends out the data from the output plugin, it is not guaranteed that the decoded transaction's effects are visible on the source server.  Is this the way it's supposed to work?

If so, would doing something like this in the output plugin be reasonable?

    TransactionId xid = transaction->xid;
    if (transaction->is_known_as_subxact)
        xid = transaction->toplevel_xid;

    if (TransactionIdIsInProgress(xid))
        XactLockTableWait(xid, NULL, NULL, XLTW_None);


-marko

Re: Local visibility with logical decoding

From
Antonin Houska
Date:
Marko Tiikkaja <marko@joh.to> wrote:

> It appears that when logical decoding sends out the data from the output
> plugin, it is not guaranteed that the decoded transaction's effects are
> visible on the source server.  Is this the way it's supposed to work?

Can you please share the test that indicates this behavior? As far as I
understand, the transaction must have been committed before the output plugin
starts to receive the changes.

-- 
Antonin Houska
Web: https://www.cybertec-postgresql.com

Re: Local visibility with logical decoding

From
Marko Tiikkaja
Date:
On Mon, Jul 20, 2020 at 7:36 PM Antonin Houska <ah@cybertec.at> wrote:
Marko Tiikkaja <marko@joh.to> wrote:

> It appears that when logical decoding sends out the data from the output
> plugin, it is not guaranteed that the decoded transaction's effects are
> visible on the source server.  Is this the way it's supposed to work?

Can you please share the test that indicates this behavior? As far as I
understand, the transaction must have been committed before the output plugin
starts to receive the changes.

I don't have a reliable test program, but you can reproduce quite easily with test_decoding if you put a breakpoint before the SyncRepWaitForLSN() call in src/backend/access/transam/xact.c.  pg_logicalrecv will see the changes while the session is sitting on the breakpoint, and not finishing its commit.


-marko

Re: Local visibility with logical decoding

From
Andres Freund
Date:
Hi,

On 2020-07-20 17:27:30 +0300, Marko Tiikkaja wrote:
> It appears that when logical decoding sends out the data from the output
> plugin, it is not guaranteed that the decoded transaction's effects are
> visible on the source server.  Is this the way it's supposed to work?

At the moment the visibility behaviour is basically the same as crash
recovery / standbys. And they just look at the WAL...


> If so, would doing something like this in the output plugin be reasonable?
> 
>     TransactionId xid = transaction->xid;
>     if (transaction->is_known_as_subxact)
>         xid = transaction->toplevel_xid;
> 
>     if (TransactionIdIsInProgress(xid))
>         XactLockTableWait(xid, NULL, NULL, XLTW_None);

I'd not be surprised if this had a potential to cause deadlocks.

Greetings,

Andres Freund