Re: Proposal: Conflict log history table for Logical Replication - Mailing list pgsql-hackers

From Nisha Moond
Subject Re: Proposal: Conflict log history table for Logical Replication
Date
Msg-id CABdArM6_FvTMr2Pz-n5Af9bxXBSHjsCVn1931XTCK=6sMHQ3JA@mail.gmail.com
Whole thread
In response to Re: Proposal: Conflict log history table for Logical Replication  (kedar anavardekar <kedar.anavardekar@gmail.com>)
List pgsql-hackers
On Wed, Sep 30, 2026 at 4:40 PM kedar anavardekar
<kedar.anavardekar@gmail.com> wrote:
>
> This is a slightly different topic than the one discussed in the email.
>
> opposite to what mentioned in the docs:
> +     If a key value is larger than 1kB it is not recorded.  In its place the
> +     conflict log stores a small object marking it as omitted and giving its
> +     size, and sets <literal>has_omitted_values</literal> for that row:
> +<programlisting>
> + {"id":"1","tag":"abc","doc":{"omitted":true,"length":4096}}
> +</programlisting>
> +     Large values are dropped rather than shortened, because a shortened value
> +     would still look like a real one and would give wrong answers when
> +     compared or joined against the table.
>
> For the case you mentioned in Case:A, the oversized value is PK itself and
>  The contents logged to the CLT are like:
>
> relid | conflict_type  | replica_identity_full |
> replica_identity              | local_conflicts | has_om
> itted_values
>
-------+----------------+-----------------------+-------------------------------------------+-----------------+-------
> -------------
>  16432 | delete_missing | f                     |
> {"b":{"omitted":true,"length":200000000}} |                 | t
>
> Would logging a shortened value or preview (which would fit within the column
> size limits) to the CLT provide more information or value to the user?
>

Thanks for trying this out. I don't think logging a trimmed value is a
good idea, for two reasons.

1) As the docs mention, a trimmed value can still look like a real
value. A query comparing the CLT key with the table could then
incorrectly match or miss a row. This was also discussed previously in
[1].

2) One option could be to put the prefix inside the marker to avoid
search mismatches, something like:
    {"b":{"omitted":true,"length":200000000,"prefix":"xxxx"}}

But to create such a prefix for types like arrays, composites, and
jsonb, the whole value must be rendered first, which could itself
exceed the 1GB allocation limit and make the apply fail. This is
exactly what the 1kB cap is intended to prevent.

So, I think we should keep the current marker.

> As from the current information which is getting logged as shown above
> user might not get useful
> information about the conflict.

Such cases are rare, and the row is still useful. It has the relation,
conflict type, other key columns, value length, and remote xid, commit
LSN, and commit timestamp, which should be enough to track down the
change on the publisher. Also, a key that large couldn't be stored in
a local B-tree index anyway, so it would only show up in a *_missing
conflict like the one in the example.

[1] https://www.postgresql.org/message-id/CAA4eK1LvgUX53XpEdWqcSiADg%3DqmXyCJ8YEZEzvh7G%3D%3DFUfLvA%40mail.gmail.com

--
Thanks,
Nisha



pgsql-hackers by date:

Previous
From: Antonin Houska
Date:
Subject: Re: REPACK (CONCURRENTLY) might keep dropped-column data
Next
From: Narayanan Venkateswaran
Date:
Subject: Re: Proposal: Conflict log history table for Logical Replication