Re: adding partitioned tables to publications - Mailing list pgsql-hackers

From Mark Zhao
Subject Re: adding partitioned tables to publications
Date
Msg-id tencent_BC6941FB423FC8AB628371AD53FA0B1F4905@qq.com
Whole thread Raw
In response to Re: adding partitioned tables to publications  (Amit Kapila <amit.kapila16@gmail.com>)
Responses Re: adding partitioned tables to publications
List pgsql-hackers
Thanks for your reply. The patch is exactly what I want.
My English name is Mark Zhao, which should be the current email name.

Thanks,
Mark Zhao


------------------ Original ------------------
From:  "Amit Kapila";<amit.kapila16@gmail.com>;
Send time: Monday, Jan 11, 2021 8:12 PM
To: "赵锐"<875941708@qq.com>;
Cc: "Peter Eisentraut"<peter.eisentraut@2ndquadrant.com>; "Amit Langote"<amitlangote09@gmail.com>; "Petr Jelinek"<petr@2ndquadrant.com>; "Rafia Sabih"<rafia.pghackers@gmail.com>; "PostgreSQL-development"<pgsql-hackers@postgresql.org>;
Subject:  Re: adding partitioned tables to publications

On Wed, Dec 30, 2020 at 8:03 PM 赵锐 <875941708@qq.com> wrote:
>
> The first file of Amit's patch can not only re-range the code, but also fix a hidden bug.
> To make it easy to see, I attach another patch.
> "RelationIdGetRelation" will increase ref on owner->relrefarr, without "RelationClose", the owner->relrefarr will enlarge and re-hash.
> When the capacity of owner->relrefarr is over than 10 million, enlarge and re-hash takes serial hours. And what's worse, increase ref will also take minutes, as the hash collision resolution is based on looking up an array in order.
> When we want to publish 10 billion data under one partition table, it takes serial days up to increase ref, enlarge and re-hash, and CPU is always 99%.
> After applying my patch, 10 billion will be published in 10 minutes.
>

It is a clear relation descriptor leak. The proposed fix seems correct
to me. The patch wasn't getting applied to HEAD. So, I have prepared
the separate patches for HEAD and 13. There are minor modifications in
the patch like I have used RelationIsValid before closing the
relation. I have not added any test because I see that there is
already a test in src/test/subscription/t/013_partition.

Kindly let me know your English name so that I can give you credit as
a co-author?

--
With Regards,
Amit Kapila.

pgsql-hackers by date:

Previous
From: "Tang, Haiying"
Date:
Subject: RE: [POC] Fast COPY FROM command for the table with foreign partitions
Next
From: 曾文旌
Date:
Subject: Re: Proposal: Global Index