Re: BUG #13725: Logical Decoding - wrong results with large transactions and unfortunate timing - Mailing list pgsql-bugs

From Andres Freund
Subject Re: BUG #13725: Logical Decoding - wrong results with large transactions and unfortunate timing
Date
Msg-id 20151026123935.GC29811@alap3.anarazel.de
Whole thread Raw
In response to BUG #13725: Logical Decoding - wrong results with large transactions and unfortunate timing  (ofir.manor@gmail.com)
Responses Re: BUG #13725: Logical Decoding - wrong results with large transactions and unfortunate timing  (<ofir.manor@gmail.com>)
List pgsql-bugs
Hi,

On 2015-10-26 14:09:24 +0200, ofir.manor@gmail.com wrote:
> Regarding my expectations...
> 1. I have observed that pg_logical_slot_get_changes always return changes
> at the transaction boundary.

Right, it's documented to do so.

> For example, asking for 1 change (third parameter, replace NULL a number)
> can still return million of changes (if the next transaction is big), as
> the check is likely done at the end of each transaction.
> I'm actually OK with that (weird but doesn't hurt me).

The reason for that is that we can't return changes until they're
committed, and that we currently can't specify anything but transaction
boundaries as "progress" point.

> 2. I have validated that a single call to pg_logical_slot_get_changes
> returns a result set with duplicates, going back to the start (I've seen it
> with a Java debugger, looping over the forward-only cursor of the SELECT
> from the replication slot). That is the bug I'm reporting - not across
> calls but within a single call.

That'd be rather weird. I'll look into it, but due to the pgconf.eu
conference this week I can't promise I'll get to it this week.

> 3. I also don't understand the "end of WAL" thing, since the call must wait
> to see if the transaction commits or rolls back.

The SQL interface doesn't ever block waiting for WAL, it decodes as far
as it can with the existing data.

> Maybe if it reaches "end of WAL", it starts over internally, adding the
> same changes again to some internal buffer that later gets sent?

No.

Andres

pgsql-bugs by date:

Previous
From: "Shulgin, Oleksandr"
Date:
Subject: Re: BUG #13725: Logical Decoding - wrong results with large transactions and unfortunate timing
Next
From: chenhj
Date:
Subject: Re: BUG #13723: "duplicate key" error occurs when update delete and insert the same row concurrently