Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication - Mailing list pgsql-hackers

From Melih Mutlu
Subject Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication
Date
Msg-id CAGPVpCSk4v-V1WbFDy8a5dL7Es5z8da6hoQbuVyrqP5s3Yh6Cg@mail.gmail.com
Whole thread Raw
In response to Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication  (Peter Smith <smithpb2250@gmail.com>)
Responses Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication
List pgsql-hackers
Hi Peter,

Peter Smith <smithpb2250@gmail.com>, 11 Tem 2023 Sal, 05:59 tarihinde şunu yazdı:
> Even if patches 0003 and 0002 are to be combined, I think that should
> not happen until after the "reuse" design is confirmed which way is
> best.
>
> e.g. IMO it might be easier to compare the different PoC designs for
> patch 0002 if there is no extra logic involved.
>
> PoC design#1 -- each tablesync decides for itself what to do next
> after it finishes
> PoC design#2 -- reuse tablesync using a "pool" of available workers

Right. I made a patch 0003 to change 0002 so that tables will be assigned to sync workers by apply worker.
It's a rough POC and ignores some edge cases. But this is what I think how apply worker would take the responsibility of table assignments. Hope the implementation makes sense and I'm not missing anything that may cause degraded perforrmance.

PoC design#1 --> apply only patch 0001 and 0002
PoC design#2 --> apply all patches, 0001, 0002 and 0003

Here are some quick numbers with 100 empty tables.

+--------------+----------------+----------------+----------------+
|              | 2 sync workers | 4 sync workers | 8 sync workers |
+--------------+----------------+----------------+----------------+
| POC design#1 | 1909.873 ms    | 986.261 ms     | 552.404 ms     |
+--------------+----------------+----------------+----------------+
| POC design#2 | 4962.208 ms    | 1240.503 ms    | 1165.405 ms    |
+--------------+----------------+----------------+----------------+
| master       | 2666.008 ms    | 1462.012 ms    | 986.848 ms     |
+--------------+----------------+----------------+----------------+


Seems like design#1 is better than both design#2 and master overall. It's surprising to see that even master beats design#2 in some cases though. Not sure if that is expected or there are some places to improve design#2 even more.

What do you think?

PS: I only attached the related patches and not the whole patch set. 0001 and 0002 may contain some of your earlier reviews, but I'll send a proper updated set soon.

Thanks,
--
Melih Mutlu
Microsoft
Attachment

pgsql-hackers by date:

Previous
From: Gurjeet Singh
Date:
Subject: Re: Fix search_path for all maintenance commands
Next
From: "David G. Johnston"
Date:
Subject: Re: Fix search_path for all maintenance commands