Re: adding partitioned tables to publications

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: adding partitioned tables to publications

赵锐
The first file of Amit's patch can not only re-range the code, but also fix a hidden bug.
To make it easy to see, I attach another patch.
"RelationIdGetRelation" will increase ref on owner->relrefarr, without "RelationClose", the owner->relrefarr will enlarge and re-hash.
When the capacity of owner->relrefarr is over than 10 million, enlarge and re-hash takes serial hours. And what's worse, increase ref will also take minutes, as the hash collision resolution is based on looking up an array in order.
When we want to publish 10 billion data under one partition table, it takes serial days up to increase ref, enlarge and re-hash, and CPU is always 99%.
After applying my patch, 10 billion will be published in 10 minutes.


------------------ Original ------------------
From:  "Amit Langote";<[hidden email]>;
Send time: Friday, Apr 17, 2020 10:58 PM
To: "Peter Eisentraut"<[hidden email]>;
Cc: "Petr Jelinek"<[hidden email]>; "Rafia Sabih"<[hidden email]>; "PostgreSQL-development"<[hidden email]>;
Subject:  Re: adding partitioned tables to publications

On Fri, Apr 17, 2020 at 10:23 PM Peter Eisentraut
<[hidden email]> wrote:
> On 2020-04-09 09:28, Amit Langote wrote:
> > While figuring this out, I thought the nearby code could be rearranged
> > a bit, especially to de-duplicate the code.  Also, I think
> > get_rel_sync_entry() may be a better place to set the map, rather than
> > maybe_send_schema().  Thoughts?
>
> because I didn't really have an opinion on that at the time, but if you
> still want it considered or have any open thoughts on this thread,
> please resend or explain.

Sure, thanks for taking care of the bug.

Rebased the code rearrangement patch.  Also resending the patch to fix
TAP tests for improving coverage as described in:
https://www.postgresql.org/message-id/CA%2BHiwqFyydvQ5g%3Dqa54UM%2BXjm77BdhX-nM4dXQkNOgH%3DzvDjoA%40mail.gmail.com

To summarize:
1. Missing coverage for a couple of related blocks in
apply_handle_tuple_routing()
2. Missing coverage report for the code in pgoutput.c added by 83fd4532

--
Amit Langote
EnterpriseDB: http://www.enterprisedb.com


0001-RelationClose-after-RelationIdGetRelation.patch (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: adding partitioned tables to publications

akapila
On Wed, Dec 30, 2020 at 8:03 PM 赵锐 <[hidden email]> wrote:
>
> The first file of Amit's patch can not only re-range the code, but also fix a hidden bug.
> To make it easy to see, I attach another patch.
> "RelationIdGetRelation" will increase ref on owner->relrefarr, without "RelationClose", the owner->relrefarr will enlarge and re-hash.
> When the capacity of owner->relrefarr is over than 10 million, enlarge and re-hash takes serial hours. And what's worse, increase ref will also take minutes, as the hash collision resolution is based on looking up an array in order.
> When we want to publish 10 billion data under one partition table, it takes serial days up to increase ref, enlarge and re-hash, and CPU is always 99%.
> After applying my patch, 10 billion will be published in 10 minutes.
>

It is a clear relation descriptor leak. The proposed fix seems correct
to me. The patch wasn't getting applied to HEAD. So, I have prepared
the separate patches for HEAD and 13. There are minor modifications in
the patch like I have used RelationIsValid before closing the
relation. I have not added any test because I see that there is
already a test in src/test/subscription/t/013_partition.

Kindly let me know your English name so that I can give you credit as
a co-author?

--
With Regards,
Amit Kapila.

v1-0001-Fix-relation-descriptor-leak.HEAD.patch (3K) Download Attachment
v1-0001-Fix-relation-descriptor-leak.13.patch (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: adding partitioned tables to publications

赵锐
Thanks for your reply. The patch is exactly what I want.
My English name is Mark Zhao, which should be the current email name.

Thanks,
Mark Zhao


------------------ Original ------------------
From:  "Amit Kapila";<[hidden email]>;
Send time: Monday, Jan 11, 2021 8:12 PM
To: "赵锐"<[hidden email]>;
Cc: "Peter Eisentraut"<[hidden email]>; "Amit Langote"<[hidden email]>; "Petr Jelinek"<[hidden email]>; "Rafia Sabih"<[hidden email]>; "PostgreSQL-development"<[hidden email]>;
Subject:  Re: adding partitioned tables to publications

On Wed, Dec 30, 2020 at 8:03 PM 赵锐 <[hidden email]> wrote:
>
> The first file of Amit's patch can not only re-range the code, but also fix a hidden bug.
> To make it easy to see, I attach another patch.
> "RelationIdGetRelation" will increase ref on owner->relrefarr, without "RelationClose", the owner->relrefarr will enlarge and re-hash.
> When the capacity of owner->relrefarr is over than 10 million, enlarge and re-hash takes serial hours. And what's worse, increase ref will also take minutes, as the hash collision resolution is based on looking up an array in order.
> When we want to publish 10 billion data under one partition table, it takes serial days up to increase ref, enlarge and re-hash, and CPU is always 99%.
> After applying my patch, 10 billion will be published in 10 minutes.
>

It is a clear relation descriptor leak. The proposed fix seems correct
to me. The patch wasn't getting applied to HEAD. So, I have prepared
the separate patches for HEAD and 13. There are minor modifications in
the patch like I have used RelationIsValid before closing the
relation. I have not added any test because I see that there is
already a test in src/test/subscription/t/013_partition.

Kindly let me know your English name so that I can give you credit as
a co-author?

--
With Regards,
Amit Kapila.

Reply | Threaded
Open this post in threaded view
|

Re: adding partitioned tables to publications

akapila
On Mon, Jan 11, 2021 at 5:44 PM Mark Zhao <[hidden email]> wrote:
>
> Thanks for your reply. The patch is exactly what I want.
> My English name is Mark Zhao, which should be the current email name.
>

Pushed the fix.

--
With Regards,
Amit Kapila.


Reply | Threaded
Open this post in threaded view
|

Re: adding partitioned tables to publications

Amit Langote
On Tue, Jan 12, 2021 at 5:09 PM Amit Kapila <[hidden email]> wrote:
>
> On Mon, Jan 11, 2021 at 5:44 PM Mark Zhao <[hidden email]> wrote:
> >
> > Thanks for your reply. The patch is exactly what I want.
> > My English name is Mark Zhao, which should be the current email name.
> >
>
> Pushed the fix.

Thanks Amit and Mark.  I hadn't realized at the time that the relation
descriptor leak was occurring, but good to see it tested and fixed.

--
Amit Langote
EDB: http://www.enterprisedb.com