Postgres Pro
Downloads
Home   >   mailing lists

Thread: Number of buckets/partitions of dshash

Number of buckets/partitions of dshash

From
"Ideriha, Takeshi"
Date:

Hi, let me clarify my understanding about the $title.

 

It seems that the number of hash partitions is fixed at 128 in dshash and

right now we cannot change it unless dshash.c code itself is changed, right?

 

According to the comment of dshash.c, DSHASH_NUM_PARTITIONS could be runtime parameter in future.

Are there someone working on it?

(I want to do it, but TBH I cannot promise it because of some other work.)

 

In my current development for allocating catalog cache on the shared memory[1]

I'm trying to put hash table of each CatCache to the shared memory using dshash.

The number of buckets for CatCache is pre-defined by cacheinfo and most of them is under 128 like 8 or 16.

This would cause some waste of memory on DSA because some partitions (buckets) is allocated but not used.

 

So I'm thinking that current dshash design is still ok but flexible size of partition is appropriate

for some use cases like mine.

 

Do you have any thoughts?

 

[1] https://www.postgresql.org/message-id/4E72940DA2BF16479384A86D54D0988A567B9245%40G01JPEXMBKW04

 

===================================================

Takeshi Ideriha

Fujitsu Limited

 

Re: Number of buckets/partitions of dshash

From
Kyotaro HORIGUCHI
Date:
Hello.

At Fri, 12 Oct 2018 06:19:12 +0000, "Ideriha, Takeshi" <ideriha.takeshi@jp.fujitsu.com> wrote in
<4E72940DA2BF16479384A86D54D0988A6F1C1779@G01JPEXMBKW04>
> Hi, let me clarify my understanding about the $title.
> 
> It seems that the number of hash partitions is fixed at 128 in dshash and
> right now we cannot change it unless dshash.c code itself is changed, right?
> 
> According to the comment of dshash.c, DSHASH_NUM_PARTITIONS could be runtime parameter in future.
> Are there someone working on it?
> (I want to do it, but TBH I cannot promise it because of some other work.)
> 
> In my current development for allocating catalog cache on the shared memory[1]
> I'm trying to put hash table of each CatCache to the shared memory using dshash.
> The number of buckets for CatCache is pre-defined by cacheinfo and most of them is under 128 like 8 or 16.
> This would cause some waste of memory on DSA because some partitions (buckets) is allocated but not used.
> 
> So I'm thinking that current dshash design is still ok but flexible size of partition is appropriate
> for some use cases like mine.
> 
> Do you have any thoughts?

We could do that easily, but shouldn't we find a way to reduce or
eliminate the impact of locking first? dshash needs to hold
partition lock while the caller is examining a returned entry.


> [1] https://www.postgresql.org/message-id/4E72940DA2BF16479384A86D54D0988A567B9245%40G01JPEXMBKW04

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

RE: Number of buckets/partitions of dshash

From
"Ideriha, Takeshi"
Date:
>From: Kyotaro HORIGUCHI [mailto:horiguchi.kyotaro@lab.ntt.co.jp]

>> This would cause some waste of memory on DSA because some partitions (buckets)
>is allocated but not used.
>>
>> So I'm thinking that current dshash design is still ok but flexible
>> size of partition is appropriate for some use cases like mine.
>>
>> Do you have any thoughts?
>
>We could do that easily, but shouldn't we find a way to reduce or eliminate the impact
>of locking first? dshash needs to hold partition lock while the caller is examining a
>returned entry.

Thanks for the comment. 
I agreed.
It would take a long time to achieve it but as you've pointed out
finding way to minimize the locking time seems benefit for everyone and first priority.

Regards,
Takeshi Ideriha