RE: Global shared meta cache - Mailing list pgsql-hackers

From serge@rielau.com
Subject RE: Global shared meta cache
Date
Msg-id 20180626100350.9cf3801d03770ada01bb39dc8f52321d.c634ca2421.mailapi@email18.godaddy.com
Whole thread Raw
In response to Global shared meta cache  ("Ideriha, Takeshi" <ideriha.takeshi@jp.fujitsu.com>)
Responses RE: Global shared meta cache
List pgsql-hackers
Takeshi-san,

>My customer created hundreds of thousands of partition tables and tried to select data from hundreds of applications,
>which resulted in enormous consumption of memory because it consumed # of backend multiplied by #
> of local memory (ex. 100 backends X 1GB = 100GB).
>Relation caches are loaded on each backend local memory.
My team and I have been working to make caches shared for the past two years, but the system and rel caches we have chosen not to share..
Reason being that these caches play a big role in transactional DDL processing.
When you do DDL your backend can see all the changes since you update your own cache, but no anyone else's until you commit.
You will find that dealing with that will be the true complexity.
 
 
Have you tried to simply cap the size of these caches?
That's a rather straight forward piece of work and will get you quite far.
We run with a 20MB syscache and a 10MB relcache with 100k+ objects and hundreds of backends
A dumb LRU is plenty good for the purpose. 
 
That being said I would love to see these caches shared. :-)
 
Cheers
Serge
Salesforce 
 
 

pgsql-hackers by date:

Previous
From: "David G. Johnston"
Date:
Subject: Re: Unexpected behavior of DROP VIEW/TABLE IF EXISTS
Next
From: Tom Lane
Date:
Subject: Re: Unexpected behavior of DROP VIEW/TABLE IF EXISTS