Re: Report bytes and transactions actually sent downtream - Mailing list pgsql-hackers

On Wed, Sep 24, 2025 at 1:55 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Hi,
>
> On Wed, Sep 24, 2025 at 12:51:29PM +0530, Ashutosh Bapat wrote:
> > On Wed, Sep 24, 2025 at 12:32 PM Bertrand Drouvot
> > <bertranddrouvot.pg@gmail.com> wrote:
> > > > > - create a table and use pg_logical_slot_get_changes with ('skip-empty-xacts', '1')
> > > > > then I don't see plugin_sent_bytes increasing (which makes sense) but I also don't
> > > > > see plugin_filtered_bytes increasing. I think that would make sense to also increase
> > > > > plugin_filtered_bytes in this case (and for the other options that would skip
> > > > > sending data). Thoughts?
> > > >
> > > > Thanks for bringing this up. I don't think we discussed this
> > > > explicitly in the thread. The changes which are filtered out by the
> > > > core itself e.g. changes to the catalogs or changes to other databases
> > > > or changes from undesired origins are not added to the reorder buffer.
> > > > They are not counted in total_bytes. The transactions containing only
> > > > such changes are not added to reorder buffer, so even total_txns does
> > > > not count such empty transactions. If we count these changes and
> > > > transactions in plugin_filtered_bytes, and plugin_filtered_txns, that
> > > > would create an anomaly - filtered counts being higher than total
> > > > counts. Further since core does not add these changes and transactions
> > > > to the reorder buffer, there is no way for a plugin to know about
> > > > their existence and hence count them. Does that make sense?
> > >
> > > Yes. Do you think that the doc in the patch is clear enough regarding this point?
> > > I mean the doc looks correct (mentioning the output plugin) but would that make
> > > sense to insist that core filtering is not taken into account?
> >
> > Do you mean, should we mention in the docs that core filtering is not
> > taken into account?
> > I would question whether that's called filtering
> > at all, in the context of logical decoding. The view should be read in
> > the context of logical decoding. For example, we aren't mentioning
> > that total_bytes does not include changes from other database.
>
> Right. But, in the example above, do you consider "skip-empty-xacts" as "core"
> or "plugin" filtering?
>
> It's an option part of the "test_decoding" plugin, so it's the plugin choice to
> not display empty xacts (should the option be set accordingly). Then should it
> be reported in plugin_filtered_bytes? (one could write a plugin, decide to
> skip/filter empty xacts or whatever in the plugin callbacks: should that be
> reported as plugin_filtered_bytes?)

If a transaction becomes empty because the plugin filtered all the
changes then plugin_filtered_bytes will be incremented by the amount
of filtered changes. If the transaction was empty because core didn't
send any of the changes to the output plugin, there was nothing
filtered by the output plugin so plugin_filtered_bytes will not be
affected.

skip_empty_xacts controls whether BEGIN and COMMIT are sent for an
empty transaction or not. It does not filter "changes". It affects
"sent_bytes".

--
Best Wishes,
Ashutosh Bapat