Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables - Mailing list pgsql-hackers

From Amit Kapila
Subject Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables
Date
Msg-id CAA4eK1JZ6FAOSN3E9N91oQD_byBUsymmufnWa19yHxZVDNwrxw@mail.gmail.com
Whole thread Raw
In response to Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables  (Dilip Kumar <dilipbalaut@gmail.com>)
Responses Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables  (Keisuke Kuroda <keisuke.kuroda.3862@gmail.com>)
Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables  (Dilip Kumar <dilipbalaut@gmail.com>)
List pgsql-hackers
On Wed, Sep 30, 2020 at 12:16 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Mon, Sep 28, 2020 at 2:22 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > I don't have a patch for this idea but you can refer [1]
> > (v25-0002-Issue-individual-invalidations-with-wal_level-lo) to see
> > what I was trying to say about registering the invalidations in the
> > form of ReorderBufferChange. We have something along the lines of what
> > I described above in that patch but we have removed it because we need
> > all the invalidations to be accumulated to handle aborts and we don't
> > want two different mechanisms to store invalidations.
>
> In some of the intermediate version of the logical-decoding, we had
> collected in form of changes and later we changed it so that we can
> execute all the invalidation on abort.  I just browsed old patch and
> fetch the changes to collect the invalidation in form of changes.  I
> have rebased on the current head so that we collect both in form of
> changes as well as collection of the invalidation.  So if we get
> individiaual invalidation we execute them and on abort we execute all
> the invalidation together.
>

Yeah, this is in-line with what I had in mind but please update the
comments in ReorderBufferAddInvalidations why we need to collect this
in both the forms. The comments are there but expand them a bit more.

And you might need to update the below comment as well:
typedef struct ReorderBufferTXN
{
..
/*
* List of ReorderBufferChange structs, including new Snapshots and new
* CommandIds
*/
dlist_head changes;

I have tried to think of a way to capture these changes only in one
form to serve our purpose but couldn't come up with any good ideas but
if you can think of some idea to achieve that I am all ears.

Keisuke-San, can you please test Dilip's patch to see if this has
fixed your problem and whether you see any other problem with this?

-- 
With Regards,
Amit Kapila.



pgsql-hackers by date:

Previous
From: Dilip Kumar
Date:
Subject: Re: [HACKERS] logical decoding of two-phase transactions
Next
From: Amit Kapila
Date:
Subject: Re: [HACKERS] logical decoding of two-phase transactions