Re: Protect syscache from bloating with negative cache entries - Mailing list pgsql-hackers
| From | Kyotaro Horiguchi |
|---|---|
| Subject | Re: Protect syscache from bloating with negative cache entries |
| Date | |
| Msg-id | 20200122.143819.1183954869198108813.horikyota.ntt@gmail.com Whole thread Raw |
| In response to | Re: Protect syscache from bloating with negative cache entries (Tomas Vondra <tomas.vondra@2ndquadrant.com>) |
| List | pgsql-hackers |
At Tue, 21 Jan 2020 17:29:47 +0100, Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote in > I see this patch is stuck in WoA since 2019/12/01, although there's a > new patch version from 2020/01/14. But the patch seems to no longer > apply, at least according to https://commitfest.cputube.org :-( So at > this point the status is actually correct. > > Not sure about the appveyor build (it seems to be about > jsonb_set_lax), > but on travis it fails like this: > > catcache.c:820:1: error: no previous prototype for > ‘CatalogCacheFlushCatalog2’ [-Werror=missing-prototypes] I changed my mind to attach the benchmark patch as .txt file, expecting the checkers not picking up it as a part of the patchset. I have in the precise performance measurement mode for a long time, but I think it's settled. I'd like to return to normal mode and explain this patch. === Motive of the patch System cache is a mechanism that accelerates access to system catalogs Basically the entries in a cache is removed via invalidation machanism when corresponding system catalog entry is removed. On the other hand the system cache also holds "negative" entries that indicates that the object is nonexistent, which entry accelerates the response for nonexistent objects. But the negative cache doesn't have a chance of removal. On a long-lived session that accepts a wide variety of queries on many objects, system cache holds the cache entries for many objects that are accessed once or a few times. Suppose every object is accessed once per, say, 30 minutes, and the query doesn't needed to run in a very short time. Such cache entries are almost useless but occupy a large amount of memory. === Possible solutions. Many caching system has expiration mechanism, which removes "useless" entries to keep the size under a certain limit. The limit is typically defined by memory usage or expiration time, in a hard or soft way. Since we don't implement an detailed accouting of memory usage by cache for the performance reasons, we can use coarse memory accounting or expiration time. This patch uses expiration time because it can be detemined on a rather clearer basis. === Pruning timing The next point is when to prune cache entries. Apparently it's not reasonable to do on every cache access time, since pruning takes a far longer time than cache access. The system cache is implemented on a hash. When there's no room for a new cache entry, it gets twice in size and rehashesall entries. If pruning made some space for the new entry, rehashing can be avoided, so this patch tries pruningjust before enlarging hash table. A system cache can be shrinked if less than a half of the size is used, but this patch doesn't that. It is because we cannot predict if the system cache that have just shrinked is going to enlarged just after and I don't want get this patch that complex. === Performance The pruning mechanism adds several entries to cache entry and updates System cache is a very light-weight machinery so that inserting one branch affects performance apparently. So in this patch, the new stuff is isolated from existing code path using indirect call. After trials on some call-points that can be indirect calls, I found that SearchCatCache[1-4]() is the only point that doesn't affect performance. (Please see upthread for details.) That configuraion also allows future implements of system caches, such like shared system caches. The alternative SearchCatCache[1-4] functions get a bit slower because it maintains access timestamp and access counter. Addition to that pruning puts a certain amount of additional time if no entries are not pruned off. === Pruning criteria At the pruning time described above, every entry is examined agianst the GUC variable catalog_cache_prune_min_age. The pruning mechanism involves a clock-sweep-like mechanism where an entry lives longer if it had accessed. Entry of which access counter is zero is pruned after catalog_cache_prune_min_age. Otherwise an entry survives the pruning round and its counter is decremented. All the timestamp used by the stuff is "catcacheclock", which is updated at every transaction start. === Concise test The attached test1.pl can be used to replay the syscache-bloat caused by negative entries. Setting $prune_age to -1, pruning is turned of and you can see that the backend unlimitedly takes more and more memory as time proceeds. Setting it to 10 or so, the memory size of backend process will stops raising at certain amount. === The patch The attached following are the patch. They have been separated for the benchmarking reasons but that seems to make the patch more easy to read so I leave it alone. I forgot its correct version through a long time of benchmarking so I started from v1 now. - v1-0001-base_change.patch Adds new members to existing structs and catcacheclock-related code. - v1-0002-Make-CatCacheSearchN-indirect-functions.patch Changes SearchCatCacheN functions to be called by indirect calls. - v1-0003-CatCache-expiration-feature.patch The core code of the patch. - catcache-benchmark-extension.patch.txt The benchmarking extension that was used for benchmarking upthread. Just for information. - test1.pl Test script to make syscache bloat. The patchset doesn't contain documentaion for the new GUC option. I will add it later. regards. -- Kyotaro Horiguchi NTT Open Source Software Center From fb80260907ac4ac0ff330806632f095484772fd1 Mon Sep 17 00:00:00 2001 From: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Date: Fri, 10 Jan 2020 15:02:26 +0900 Subject: [PATCH v1 1/3] base_change Adds struct members needed by catcache expiration feature and a GUC variable that controls the behavior of the feature. But no substantial code is not added yet. If existence of some variables alone can cause degradation, benchmarking after this patch shows that. --- src/backend/access/transam/xact.c | 3 +++ src/backend/utils/cache/catcache.c | 15 +++++++++++++++ src/backend/utils/misc/guc.c | 13 +++++++++++++ src/include/utils/catcache.h | 20 ++++++++++++++++++++ 4 files changed, 51 insertions(+) diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c index 017f03b6d8..1268a7fb80 100644 --- a/src/backend/access/transam/xact.c +++ b/src/backend/access/transam/xact.c @@ -1067,6 +1067,9 @@ ForceSyncCommit(void) static void AtStart_Cache(void) { + if (xactStartTimestamp != 0) + SetCatCacheClock(xactStartTimestamp); + AcceptInvalidationMessages(); } diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 64776e3209..7248bd0d41 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -60,9 +60,18 @@ #define CACHE_elog(...) #endif +/* + * GUC variable to define the minimum age of entries that will be considered + * to be evicted in seconds. -1 to disable the feature. + */ +int catalog_cache_prune_min_age = 300; + /* Cache management header --- pointer is NULL until created */ static CatCacheHeader *CacheHdr = NULL; +/* Clock for the last accessed time of a catcache entry. */ +TimestampTz catcacheclock = 0; + static inline HeapTuple SearchCatCacheInternal(CatCache *cache, int nkeys, Datum v1, Datum v2, @@ -99,6 +108,12 @@ static void CatCacheFreeKeys(TupleDesc tupdesc, int nkeys, int *attnos, static void CatCacheCopyKeys(TupleDesc tupdesc, int nkeys, int *attnos, Datum *srckeys, Datum *dstkeys); +/* GUC assign function */ +void +assign_catalog_cache_prune_min_age(int newval, void *extra) +{ + catalog_cache_prune_min_age = newval; +} /* * internal support functions diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index e44f71e991..3029e44d7a 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -83,6 +83,8 @@ #include "tsearch/ts_cache.h" #include "utils/builtins.h" #include "utils/bytea.h" +#include "utils/catcache.h" +#include "utils/guc_tables.h" #include "utils/float.h" #include "utils/guc_tables.h" #include "utils/memutils.h" @@ -2293,6 +2295,17 @@ static struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"catalog_cache_prune_min_age", PGC_USERSET, RESOURCES_MEM, + gettext_noop("System catalog cache entries that live unused for longer than this seconds are considered forremoval."), + gettext_noop("The value of -1 turns off pruning."), + GUC_UNIT_S + }, + &catalog_cache_prune_min_age, + 300, -1, INT_MAX, + NULL, assign_catalog_cache_prune_min_age, NULL + }, + /* * We use the hopefully-safely-small value of 100kB as the compiled-in * default for max_stack_depth. InitializeGUCOptions will increase it if diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h index f4aa316604..3d3870f05a 100644 --- a/src/include/utils/catcache.h +++ b/src/include/utils/catcache.h @@ -22,6 +22,7 @@ #include "access/htup.h" #include "access/skey.h" +#include "datatype/timestamp.h" #include "lib/ilist.h" #include "utils/relcache.h" @@ -61,6 +62,7 @@ typedef struct catcache slist_node cc_next; /* list link */ ScanKeyData cc_skey[CATCACHE_MAXKEYS]; /* precomputed key info for heap * scans */ + TimestampTz cc_oldest_ts; /* timestamp of the oldest tuple in the hash */ /* * Keep these at the end, so that compiling catcache.c with CATCACHE_STATS @@ -119,6 +121,8 @@ typedef struct catctup bool dead; /* dead but not yet removed? */ bool negative; /* negative cache entry? */ HeapTupleData tuple; /* tuple management header */ + unsigned int naccess; /* # of access to this entry */ + TimestampTz lastaccess; /* timestamp of the last usage */ /* * The tuple may also be a member of at most one CatCList. (If a single @@ -189,6 +193,22 @@ typedef struct catcacheheader /* this extern duplicates utils/memutils.h... */ extern PGDLLIMPORT MemoryContext CacheMemoryContext; +/* for guc.c, not PGDLLPMPORT'ed */ +extern int catalog_cache_prune_min_age; + +/* source clock for access timestamp of catcache entries */ +extern TimestampTz catcacheclock; + +/* SetCatCacheClock - set catcache timestamp source clodk */ +static inline void +SetCatCacheClock(TimestampTz ts) +{ + catcacheclock = ts; +} + + +extern void assign_catalog_cache_prune_min_age(int newval, void *extra); + extern void CreateCacheMemoryContext(void); extern CatCache *InitCatCache(int id, Oid reloid, Oid indexoid, -- 2.23.0 From 2b4449372acfbdf728a79f43dec0a0109c30228d Mon Sep 17 00:00:00 2001 From: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Date: Thu, 9 Jan 2020 19:22:18 +0900 Subject: [PATCH v1 2/3] Make CatCacheSearchN indirect functions After some expriments showed that the best way to add a new feature to the current CatCacheSearch path is making SearchCatCacheN functions replacable using indirect calling. This patch does that. If the change of how to call the functions alone causes degradataion, benchmarking after this patch applied shows that. --- src/backend/utils/cache/catcache.c | 42 +++++++++++++++++++++++------- src/include/utils/catcache.h | 40 ++++++++++++++++++++++++---- 2 files changed, 67 insertions(+), 15 deletions(-) diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 7248bd0d41..a4e3676a89 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -84,6 +84,15 @@ static pg_noinline HeapTuple SearchCatCacheMiss(CatCache *cache, Datum v1, Datum v2, Datum v3, Datum v4); +static HeapTuple SearchCatCacheb(CatCache *cache, + Datum v1, Datum v2, Datum v3, Datum v4); +static HeapTuple SearchCatCache1b(CatCache *cache, Datum v1); +static HeapTuple SearchCatCache2b(CatCache *cache, Datum v1, Datum v2); +static HeapTuple SearchCatCache3b(CatCache *cache, + Datum v1, Datum v2, Datum v3); +static HeapTuple SearchCatCache4b(CatCache *cache, + Datum v1, Datum v2, Datum v3, Datum v4); + static uint32 CatalogCacheComputeHashValue(CatCache *cache, int nkeys, Datum v1, Datum v2, Datum v3, Datum v4); static uint32 CatalogCacheComputeTupleHashValue(CatCache *cache, int nkeys, @@ -108,6 +117,16 @@ static void CatCacheFreeKeys(TupleDesc tupdesc, int nkeys, int *attnos, static void CatCacheCopyKeys(TupleDesc tupdesc, int nkeys, int *attnos, Datum *srckeys, Datum *dstkeys); +static SearchCatCacheFuncsType catcache_base = { + SearchCatCacheb, + SearchCatCache1b, + SearchCatCache2b, + SearchCatCache3b, + SearchCatCache4b +}; + +SearchCatCacheFuncsType *SearchCatCacheFuncs = NULL; + /* GUC assign function */ void assign_catalog_cache_prune_min_age(int newval, void *extra) @@ -817,6 +836,9 @@ InitCatCache(int id, CacheHdr = (CatCacheHeader *) palloc(sizeof(CatCacheHeader)); slist_init(&CacheHdr->ch_caches); CacheHdr->ch_ntup = 0; + + SearchCatCacheFuncs = &catcache_base; + #ifdef CATCACHE_STATS /* set up to dump stats at backend exit */ on_proc_exit(CatCachePrintStats, 0); @@ -1158,8 +1180,8 @@ IndexScanOK(CatCache *cache, ScanKey cur_skey) * the caller need not go to the trouble of converting it to a fully * null-padded NAME. */ -HeapTuple -SearchCatCache(CatCache *cache, +static HeapTuple +SearchCatCacheb(CatCache *cache, Datum v1, Datum v2, Datum v3, @@ -1175,32 +1197,32 @@ SearchCatCache(CatCache *cache, * bit faster than SearchCatCache(). */ -HeapTuple -SearchCatCache1(CatCache *cache, +static HeapTuple +SearchCatCache1b(CatCache *cache, Datum v1) { return SearchCatCacheInternal(cache, 1, v1, 0, 0, 0); } -HeapTuple -SearchCatCache2(CatCache *cache, +static HeapTuple +SearchCatCache2b(CatCache *cache, Datum v1, Datum v2) { return SearchCatCacheInternal(cache, 2, v1, v2, 0, 0); } -HeapTuple -SearchCatCache3(CatCache *cache, +static HeapTuple +SearchCatCache3b(CatCache *cache, Datum v1, Datum v2, Datum v3) { return SearchCatCacheInternal(cache, 3, v1, v2, v3, 0); } -HeapTuple -SearchCatCache4(CatCache *cache, +static HeapTuple +SearchCatCache4b(CatCache *cache, Datum v1, Datum v2, Datum v3, Datum v4) { return SearchCatCacheInternal(cache, 4, v1, v2, v3, v4); diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h index 3d3870f05a..f9e9889339 100644 --- a/src/include/utils/catcache.h +++ b/src/include/utils/catcache.h @@ -189,6 +189,36 @@ typedef struct catcacheheader int ch_ntup; /* # of tuples in all caches */ } CatCacheHeader; +typedef HeapTuple (*SearchCatCache_fn)(CatCache *cache, + Datum v1, Datum v2, Datum v3, Datum v4); +typedef HeapTuple (*SearchCatCache1_fn)(CatCache *cache, Datum v1); +typedef HeapTuple (*SearchCatCache2_fn)(CatCache *cache, Datum v1, Datum v2); +typedef HeapTuple (*SearchCatCache3_fn)(CatCache *cache, Datum v1, Datum v2, + Datum v3); +typedef HeapTuple (*SearchCatCache4_fn)(CatCache *cache, + Datum v1, Datum v2, Datum v3, Datum v4); + +typedef struct SearchCatCacheFuncsType +{ + SearchCatCache_fn SearchCatCache; + SearchCatCache1_fn SearchCatCache1; + SearchCatCache2_fn SearchCatCache2; + SearchCatCache3_fn SearchCatCache3; + SearchCatCache4_fn SearchCatCache4; +} SearchCatCacheFuncsType; + +extern PGDLLIMPORT SearchCatCacheFuncsType *SearchCatCacheFuncs; + +#define SearchCatCache(cache, v1, v2, v3, v4) \ + SearchCatCacheFuncs->SearchCatCache(cache, v1, v2, v3, v4) +#define SearchCatCache1(cache, v1) \ + SearchCatCacheFuncs->SearchCatCache1(cache, v1) +#define SearchCatCache2(cache, v1, v2) \ + SearchCatCacheFuncs->SearchCatCache2(cache, v1, v2) +#define SearchCatCache3(cache, v1, v2, v3) \ + SearchCatCacheFuncs->SearchCatCache3(cache, v1, v2, v3) +#define SearchCatCache4(cache, v1, v2, v3, v4) \ + SearchCatCacheFuncs->SearchCatCache4(cache, v1, v2, v3, v4) /* this extern duplicates utils/memutils.h... */ extern PGDLLIMPORT MemoryContext CacheMemoryContext; @@ -216,15 +246,15 @@ extern CatCache *InitCatCache(int id, Oid reloid, Oid indexoid, int nbuckets); extern void InitCatCachePhase2(CatCache *cache, bool touch_index); -extern HeapTuple SearchCatCache(CatCache *cache, +extern HeapTuple (*SearchCatCache)(CatCache *cache, Datum v1, Datum v2, Datum v3, Datum v4); -extern HeapTuple SearchCatCache1(CatCache *cache, +extern HeapTuple (*SearchCatCache1)(CatCache *cache, Datum v1); -extern HeapTuple SearchCatCache2(CatCache *cache, +extern HeapTuple (*SearchCatCache2)(CatCache *cache, Datum v1, Datum v2); -extern HeapTuple SearchCatCache3(CatCache *cache, +extern HeapTuple (*SearchCatCache3)(CatCache *cache, Datum v1, Datum v2, Datum v3); -extern HeapTuple SearchCatCache4(CatCache *cache, +extern HeapTuple (*SearchCatCache4)(CatCache *cache, Datum v1, Datum v2, Datum v3, Datum v4); extern void ReleaseCatCache(HeapTuple tuple); -- 2.23.0 From e348af29c3cae63212dcb3d982e419e53bc86517 Mon Sep 17 00:00:00 2001 From: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Date: Fri, 10 Jan 2020 15:08:54 +0900 Subject: [PATCH v1 3/3] CatCache expiration feature. This adds the catcache expiration feature to the catcache mechanism. Current catcache doesn't remove an entry and there's a case where many hash entries occupy large amont of memory , being not accessed ever after. This can be a quite serious issue on the cases of long-running sessions. The expiration feature keeps process memory usage below certain amount, in exchange of some extent of degradation if it is turned on. --- src/backend/utils/cache/catcache.c | 343 +++++++++++++++++++++++++++-- 1 file changed, 326 insertions(+), 17 deletions(-) diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index a4e3676a89..29bc980d8e 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -72,10 +72,11 @@ static CatCacheHeader *CacheHdr = NULL; /* Clock for the last accessed time of a catcache entry. */ TimestampTz catcacheclock = 0; -static inline HeapTuple SearchCatCacheInternal(CatCache *cache, - int nkeys, - Datum v1, Datum v2, - Datum v3, Datum v4); +/* basic catcache search functions */ +static inline HeapTuple SearchCatCacheInternalb(CatCache *cache, + int nkeys, + Datum v1, Datum v2, + Datum v3, Datum v4); static pg_noinline HeapTuple SearchCatCacheMiss(CatCache *cache, int nkeys, @@ -93,6 +94,23 @@ static HeapTuple SearchCatCache3b(CatCache *cache, static HeapTuple SearchCatCache4b(CatCache *cache, Datum v1, Datum v2, Datum v3, Datum v4); +/* catcache search functions with expiration feature */ +static inline HeapTuple SearchCatCacheInternale(CatCache *cache, + int nkeys, + Datum v1, Datum v2, + Datum v3, Datum v4); + +static HeapTuple SearchCatCachee(CatCache *cache, + Datum v1, Datum v2, Datum v3, Datum v4); +static HeapTuple SearchCatCache1e(CatCache *cache, Datum v1); +static HeapTuple SearchCatCache2e(CatCache *cache, Datum v1, Datum v2); +static HeapTuple SearchCatCache3e(CatCache *cache, + Datum v1, Datum v2, Datum v3); +static HeapTuple SearchCatCache4e(CatCache *cache, + Datum v1, Datum v2, Datum v3, Datum v4); + +static bool CatCacheCleanupOldEntries(CatCache *cp); + static uint32 CatalogCacheComputeHashValue(CatCache *cache, int nkeys, Datum v1, Datum v2, Datum v3, Datum v4); static uint32 CatalogCacheComputeTupleHashValue(CatCache *cache, int nkeys, @@ -125,13 +143,35 @@ static SearchCatCacheFuncsType catcache_base = { SearchCatCache4b }; +static SearchCatCacheFuncsType catcache_expire = { + SearchCatCachee, + SearchCatCache1e, + SearchCatCache2e, + SearchCatCache3e, + SearchCatCache4e +}; + SearchCatCacheFuncsType *SearchCatCacheFuncs = NULL; +/* set catcache function set according to guc variables */ +static void +set_catcache_functions(void) +{ + if (catalog_cache_prune_min_age < 0) + SearchCatCacheFuncs = &catcache_base; + else + SearchCatCacheFuncs = &catcache_expire; +} + + /* GUC assign function */ void assign_catalog_cache_prune_min_age(int newval, void *extra) { catalog_cache_prune_min_age = newval; + + /* choose corresponding function set */ + set_catcache_functions(); } /* @@ -837,7 +877,7 @@ InitCatCache(int id, slist_init(&CacheHdr->ch_caches); CacheHdr->ch_ntup = 0; - SearchCatCacheFuncs = &catcache_base; + set_catcache_functions(); #ifdef CATCACHE_STATS /* set up to dump stats at backend exit */ @@ -900,6 +940,10 @@ RehashCatCache(CatCache *cp) int newnbuckets; int i; + /* try removing old entries before expanding hash */ + if (CatCacheCleanupOldEntries(cp)) + return; + elog(DEBUG1, "rehashing catalog cache id %d for %s; %d tups, %d buckets", cp->id, cp->cc_relname, cp->cc_ntup, cp->cc_nbuckets); @@ -1187,7 +1231,7 @@ SearchCatCacheb(CatCache *cache, Datum v3, Datum v4) { - return SearchCatCacheInternal(cache, cache->cc_nkeys, v1, v2, v3, v4); + return SearchCatCacheInternalb(cache, cache->cc_nkeys, v1, v2, v3, v4); } @@ -1201,7 +1245,7 @@ static HeapTuple SearchCatCache1b(CatCache *cache, Datum v1) { - return SearchCatCacheInternal(cache, 1, v1, 0, 0, 0); + return SearchCatCacheInternalb(cache, 1, v1, 0, 0, 0); } @@ -1209,7 +1253,7 @@ static HeapTuple SearchCatCache2b(CatCache *cache, Datum v1, Datum v2) { - return SearchCatCacheInternal(cache, 2, v1, v2, 0, 0); + return SearchCatCacheInternalb(cache, 2, v1, v2, 0, 0); } @@ -1217,7 +1261,7 @@ static HeapTuple SearchCatCache3b(CatCache *cache, Datum v1, Datum v2, Datum v3) { - return SearchCatCacheInternal(cache, 3, v1, v2, v3, 0); + return SearchCatCacheInternalb(cache, 3, v1, v2, v3, 0); } @@ -1225,19 +1269,19 @@ static HeapTuple SearchCatCache4b(CatCache *cache, Datum v1, Datum v2, Datum v3, Datum v4) { - return SearchCatCacheInternal(cache, 4, v1, v2, v3, v4); + return SearchCatCacheInternalb(cache, 4, v1, v2, v3, v4); } /* - * Work-horse for SearchCatCache/SearchCatCacheN. + * Work-horse for SearchCatCacheb/SearchCatCacheNb. */ static inline HeapTuple -SearchCatCacheInternal(CatCache *cache, - int nkeys, - Datum v1, - Datum v2, - Datum v3, - Datum v4) +SearchCatCacheInternalb(CatCache *cache, + int nkeys, + Datum v1, + Datum v2, + Datum v3, + Datum v4) { Datum arguments[CATCACHE_MAXKEYS]; uint32 hashValue; @@ -1462,6 +1506,269 @@ SearchCatCacheMiss(CatCache *cache, return &ct->tuple; } +/* + * SearchCatCache with entry pruning + * + * These functions works the same way with SearchCatCacheNb() functions except + * that less-used entries are removed following catalog_cache_prune_min_age + * setting. + */ +static HeapTuple +SearchCatCachee(CatCache *cache, + Datum v1, + Datum v2, + Datum v3, + Datum v4) +{ + return SearchCatCacheInternale(cache, cache->cc_nkeys, v1, v2, v3, v4); +} + + +/* + * SearchCatCacheN() are SearchCatCache() versions for a specific number of + * arguments. The compiler can inline the body and unroll loops, making them a + * bit faster than SearchCatCache(). + */ + +static HeapTuple +SearchCatCache1e(CatCache *cache, + Datum v1) +{ + return SearchCatCacheInternale(cache, 1, v1, 0, 0, 0); +} + + +static HeapTuple +SearchCatCache2e(CatCache *cache, + Datum v1, Datum v2) +{ + return SearchCatCacheInternale(cache, 2, v1, v2, 0, 0); +} + + +static HeapTuple +SearchCatCache3e(CatCache *cache, + Datum v1, Datum v2, Datum v3) +{ + return SearchCatCacheInternale(cache, 3, v1, v2, v3, 0); +} + + +static HeapTuple +SearchCatCache4e(CatCache *cache, + Datum v1, Datum v2, Datum v3, Datum v4) +{ + return SearchCatCacheInternale(cache, 4, v1, v2, v3, v4); +} + +/* + * Work-horse for SearchCatCachee/SearchCatCacheNe. + */ +static inline HeapTuple +SearchCatCacheInternale(CatCache *cache, + int nkeys, + Datum v1, + Datum v2, + Datum v3, + Datum v4) +{ + Datum arguments[CATCACHE_MAXKEYS]; + uint32 hashValue; + Index hashIndex; + dlist_iter iter; + dlist_head *bucket; + CatCTup *ct; + + /* Make sure we're in an xact, even if this ends up being a cache hit */ + Assert(IsTransactionState()); + + Assert(cache->cc_nkeys == nkeys); + + /* + * one-time startup overhead for each cache + */ + if (unlikely(cache->cc_tupdesc == NULL)) + CatalogCacheInitializeCache(cache); + +#ifdef CATCACHE_STATS + cache->cc_searches++; +#endif + + /* Initialize local parameter array */ + arguments[0] = v1; + arguments[1] = v2; + arguments[2] = v3; + arguments[3] = v4; + + /* + * find the hash bucket in which to look for the tuple + */ + hashValue = CatalogCacheComputeHashValue(cache, nkeys, v1, v2, v3, v4); + hashIndex = HASH_INDEX(hashValue, cache->cc_nbuckets); + + /* + * scan the hash bucket until we find a match or exhaust our tuples + * + * Note: it's okay to use dlist_foreach here, even though we modify the + * dlist within the loop, because we don't continue the loop afterwards. + */ + bucket = &cache->cc_bucket[hashIndex]; + dlist_foreach(iter, bucket) + { + ct = dlist_container(CatCTup, cache_elem, iter.cur); + + if (ct->dead) + continue; /* ignore dead entries */ + + if (ct->hash_value != hashValue) + continue; /* quickly skip entry if wrong hash val */ + + if (!CatalogCacheCompareTuple(cache, nkeys, ct->keys, arguments)) + continue; + + /* + * We found a match in the cache. Move it to the front of the list + * for its hashbucket, in order to speed subsequent searches. (The + * most frequently accessed elements in any hashbucket will tend to be + * near the front of the hashbucket's list.) + */ + dlist_move_head(bucket, &ct->cache_elem); + + /* + * Prolong life of this entry. Since we want run as less instructions + * as possible and want the branch be stable for performance reasons, + * we don't give a strict cap on the counter. All numbers above 1 will + * be regarded as 2 in CatCacheCleanupOldEntries(). + */ + ct->naccess++; + if (unlikely(ct->naccess == 0)) + ct->naccess = 2; + ct->lastaccess = catcacheclock; + + /* + * If it's a positive entry, bump its refcount and return it. If it's + * negative, we can report failure to the caller. + */ + if (!ct->negative) + { + ResourceOwnerEnlargeCatCacheRefs(CurrentResourceOwner); + ct->refcount++; + ResourceOwnerRememberCatCacheRef(CurrentResourceOwner, &ct->tuple); + + CACHE_elog(DEBUG2, "SearchCatCache(%s): found in bucket %d", + cache->cc_relname, hashIndex); + +#ifdef CATCACHE_STATS + cache->cc_hits++; +#endif + + return &ct->tuple; + } + else + { + CACHE_elog(DEBUG2, "SearchCatCache(%s): found neg entry in bucket %d", + cache->cc_relname, hashIndex); + +#ifdef CATCACHE_STATS + cache->cc_neg_hits++; +#endif + + return NULL; + } + } + + return SearchCatCacheMiss(cache, nkeys, hashValue, hashIndex, v1, v2, v3, v4); +} + +/* + * CatCacheCleanupOldEntries - Remove infrequently-used entries + * + * Catcache entries happen to be left unused for a long time for several + * reasons. Remove such entries to prevent catcache from bloating. It is based + * on the similar algorithm with buffer eviction. Entries that are accessed + * several times in a certain period live longer than those that have had less + * access in the same duration. + */ +static bool +CatCacheCleanupOldEntries(CatCache *cp) +{ + int nremoved = 0; + int i; + long oldest_ts = catcacheclock; + long age; + int us; + + /* Return immediately if disabled */ + if (catalog_cache_prune_min_age < 0) + return false; + + /* Don't scan the hash when we know we don't have prunable entries */ + TimestampDifference(cp->cc_oldest_ts, catcacheclock, &age, &us); + if (age < catalog_cache_prune_min_age) + return false; + + /* Scan over the whole hash to find entries to remove */ + for (i = 0 ; i < cp->cc_nbuckets ; i++) + { + dlist_mutable_iter iter; + + dlist_foreach_modify(iter, &cp->cc_bucket[i]) + { + CatCTup *ct = dlist_container(CatCTup, cache_elem, iter.cur); + + /* Don't remove referenced entries */ + if (ct->refcount == 0 && + (ct->c_list == NULL || ct->c_list->refcount == 0)) + { + /* + * Calculate the duration from the time from the last access + * to the "current" time. catcacheclock is updated + * per-statement basis and additionaly udpated periodically + * during a long running query. + */ + TimestampDifference(ct->lastaccess, catcacheclock, &age, &us); + + if (age > catalog_cache_prune_min_age) + { + /* + * Entries that are not accessed after the last pruning + * are removed in that seconds, and their lives are + * prolonged according to how many times they are accessed + * up to three times of the duration. We don't try shrink + * buckets since pruning effectively caps catcache + * expansion in the long term. + */ + if (ct->naccess > 2) + ct->naccess = 1; + else if (ct->naccess > 0) + ct->naccess--; + else + { + CatCacheRemoveCTup(cp, ct); + nremoved++; + + /* don't update oldest_ts by removed entry */ + continue; + } + } + } + + /* update oldest timestamp if the entry remains alive */ + if (ct->lastaccess < oldest_ts) + oldest_ts = ct->lastaccess; + } + } + + cp->cc_oldest_ts = oldest_ts; + + if (nremoved > 0) + elog(DEBUG1, "pruning catalog cache id=%d for %s: removed %d / %d", + cp->id, cp->cc_relname, nremoved, cp->cc_ntup + nremoved); + + return nremoved > 0; +} + + /* * ReleaseCatCache * @@ -1925,6 +2232,8 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments, ct->dead = false; ct->negative = negative; ct->hash_value = hashValue; + ct->naccess = 0; + ct->lastaccess = catcacheclock; dlist_push_head(&cache->cc_bucket[hashIndex], &ct->cache_elem); -- 2.23.0 From 7a793b13803d5defd6d5154a075d1d4cb6826103 Mon Sep 17 00:00:00 2001 From: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Date: Thu, 14 Nov 2019 19:24:36 +0900 Subject: [PATCH] catcache benchmark extension Provides the function catcachebench(bench_no int), which runs CPU intensive benchmark on catcache search. The test table is created by a script separately provided. catcachebench(0): prewarm catcache with provided test tables. catcachebench(1): fetches all attribute stats of all tables. This benchmark loads a vast number of unique entries. Expriration doesn't work since it runs in a transaction. catcachebench(2): fetches all attribute stats of a tables many times. This benchmark repeatedly accesses already loaded entries. Expriration doesn't work since it runs in a transaction. catcachebench(3): fetches all attribute stats of all tables four times. Different from other modes, this runs expiration by forcibly updates reference clock variable every 1000 entries. At this point, variables needed for the expiration feature is not added so SetCatCacheClock is a dummy macro that just replaces it with its parameter. --- contrib/catcachebench/Makefile | 17 + contrib/catcachebench/catcachebench--0.0.sql | 14 + contrib/catcachebench/catcachebench.c | 330 +++++++++++++++++++ contrib/catcachebench/catcachebench.control | 6 + src/backend/utils/cache/catcache.c | 35 ++ src/backend/utils/cache/syscache.c | 2 +- src/include/utils/catcache.h | 3 + 7 files changed, 406 insertions(+), 1 deletion(-) create mode 100644 contrib/catcachebench/Makefile create mode 100644 contrib/catcachebench/catcachebench--0.0.sql create mode 100644 contrib/catcachebench/catcachebench.c create mode 100644 contrib/catcachebench/catcachebench.control diff --git a/contrib/catcachebench/Makefile b/contrib/catcachebench/Makefile new file mode 100644 index 0000000000..0478818b25 --- /dev/null +++ b/contrib/catcachebench/Makefile @@ -0,0 +1,17 @@ +MODULE_big = catcachebench +OBJS = catcachebench.o + +EXTENSION = catcachebench +DATA = catcachebench--0.0.sql +PGFILEDESC = "catcachebench - benchmark for catcache pruning feature" + +ifdef USE_PGXS +PG_CONFIG = pg_config +PGXS := $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) +else +subdir = contrib/catcachebench +top_builddir = ../.. +include $(top_builddir)/src/Makefile.global +include $(top_srcdir)/contrib/contrib-global.mk +endif diff --git a/contrib/catcachebench/catcachebench--0.0.sql b/contrib/catcachebench/catcachebench--0.0.sql new file mode 100644 index 0000000000..ea9cd62abb --- /dev/null +++ b/contrib/catcachebench/catcachebench--0.0.sql @@ -0,0 +1,14 @@ +/* contrib/catcachebench/catcachebench--0.0.sql */ + +-- complain if script is sourced in psql, rather than via CREATE EXTENSION +\echo Use "CREATE EXTENSION catcachebench" to load this file. \quit + +CREATE FUNCTION catcachebench(IN type int) +RETURNS double precision +AS 'MODULE_PATHNAME', 'catcachebench' +LANGUAGE C STRICT VOLATILE; + +CREATE FUNCTION catcachereadstats(OUT catid int, OUT reloid oid, OUT searches bigint, OUT hits bigint, OUT neg_hits bigint) +RETURNS SETOF record +AS 'MODULE_PATHNAME', 'catcachereadstats' +LANGUAGE C STRICT VOLATILE; diff --git a/contrib/catcachebench/catcachebench.c b/contrib/catcachebench/catcachebench.c new file mode 100644 index 0000000000..b6c2b8f577 --- /dev/null +++ b/contrib/catcachebench/catcachebench.c @@ -0,0 +1,330 @@ +/* + * catcachebench: test code for cache pruning feature + */ +/* #define CATCACHE_STATS */ +#include "postgres.h" +#include "catalog/pg_type.h" +#include "catalog/pg_statistic.h" +#include "executor/spi.h" +#include "funcapi.h" +#include "libpq/pqsignal.h" +#include "utils/catcache.h" +#include "utils/syscache.h" +#include "utils/timestamp.h" + +Oid tableoids[10000]; +int ntables = 0; +int16 attnums[1000]; +int natts = 0; + +PG_MODULE_MAGIC; + +double catcachebench1(void); +double catcachebench2(void); +double catcachebench3(void); +void collectinfo(void); +void catcachewarmup(void); + +PG_FUNCTION_INFO_V1(catcachebench); +PG_FUNCTION_INFO_V1(catcachereadstats); + +extern void CatalogCacheFlushCatalog2(Oid catId); +extern int64 catcache_called; +extern CatCache *SysCache[]; + +typedef struct catcachestatsstate +{ + TupleDesc tupd; + int catId; +} catcachestatsstate; + +Datum +catcachereadstats(PG_FUNCTION_ARGS) +{ + catcachestatsstate *state_data = NULL; + FuncCallContext *fctx; + + if (SRF_IS_FIRSTCALL()) + { + TupleDesc tupdesc; + MemoryContext mctx; + + fctx = SRF_FIRSTCALL_INIT(); + mctx = MemoryContextSwitchTo(fctx->multi_call_memory_ctx); + + state_data = palloc(sizeof(catcachestatsstate)); + + /* Build a tuple descriptor for our result type */ + if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) + elog(ERROR, "return type must be a row type"); + + state_data->tupd = tupdesc; + state_data->catId = 0; + + fctx->user_fctx = state_data; + + MemoryContextSwitchTo(mctx); + } + + fctx = SRF_PERCALL_SETUP(); + state_data = fctx->user_fctx; + + if (state_data->catId < SysCacheSize) + { + Datum values[5]; + bool nulls[5]; + HeapTuple resulttup; + Datum result; + int catId = state_data->catId++; + + memset(nulls, 0, sizeof(nulls)); + memset(values, 0, sizeof(values)); + values[0] = Int16GetDatum(catId); + values[1] = ObjectIdGetDatum(SysCache[catId]->cc_reloid); +#ifdef CATCACHE_STATS + values[2] = Int64GetDatum(SysCache[catId]->cc_searches); + values[3] = Int64GetDatum(SysCache[catId]->cc_hits); + values[4] = Int64GetDatum(SysCache[catId]->cc_neg_hits); +#endif + resulttup = heap_form_tuple(state_data->tupd, values, nulls); + result = HeapTupleGetDatum(resulttup); + + SRF_RETURN_NEXT(fctx, result); + } + + SRF_RETURN_DONE(fctx); +} + +Datum +catcachebench(PG_FUNCTION_ARGS) +{ + int testtype = PG_GETARG_INT32(0); + double ms; + + collectinfo(); + + /* flush the catalog -- safe? don't mind. */ + CatalogCacheFlushCatalog2(StatisticRelationId); + + switch (testtype) + { + case 0: + catcachewarmup(); /* prewarm of syscatalog */ + PG_RETURN_NULL(); + case 1: + ms = catcachebench1(); break; + case 2: + ms = catcachebench2(); break; + case 3: + ms = catcachebench3(); break; + default: + elog(ERROR, "Invalid test type: %d", testtype); + } + + PG_RETURN_DATUM(Float8GetDatum(ms)); +} + +/* + * fetch all attribute entires of all tables. + */ +double +catcachebench1(void) +{ + int t, a; + instr_time start, + duration; + + PG_SETMASK(&BlockSig); + INSTR_TIME_SET_CURRENT(start); + for (t = 0 ; t < ntables ; t++) + { + for (a = 0 ; a < natts ; a++) + { + HeapTuple tup; + + tup = SearchSysCache3(STATRELATTINH, + ObjectIdGetDatum(tableoids[t]), + Int16GetDatum(attnums[a]), + BoolGetDatum(false)); + /* should be null, but.. */ + if (HeapTupleIsValid(tup)) + ReleaseSysCache(tup); + } + } + INSTR_TIME_SET_CURRENT(duration); + INSTR_TIME_SUBTRACT(duration, start); + PG_SETMASK(&UnBlockSig); + + return INSTR_TIME_GET_MILLISEC(duration); +}; + +/* + * fetch all attribute entires of a table many times. + */ +double +catcachebench2(void) +{ + int t, a; + instr_time start, + duration; + + PG_SETMASK(&BlockSig); + INSTR_TIME_SET_CURRENT(start); + for (t = 0 ; t < 240000 ; t++) + { + for (a = 0 ; a < natts ; a++) + { + HeapTuple tup; + + tup = SearchSysCache3(STATRELATTINH, + ObjectIdGetDatum(tableoids[0]), + Int16GetDatum(attnums[a]), + BoolGetDatum(false)); + /* should be null, but.. */ + if (HeapTupleIsValid(tup)) + ReleaseSysCache(tup); + } + } + INSTR_TIME_SET_CURRENT(duration); + INSTR_TIME_SUBTRACT(duration, start); + PG_SETMASK(&UnBlockSig); + + return INSTR_TIME_GET_MILLISEC(duration); +}; + +/* + * fetch all attribute entires of all tables several times with having + * expiration happen. + */ +double +catcachebench3(void) +{ + const int clock_step = 1000; + int i, t, a; + instr_time start, + duration; + + PG_SETMASK(&BlockSig); + INSTR_TIME_SET_CURRENT(start); + for (i = 0 ; i < 4 ; i++) + { + int ct = clock_step; + + for (t = 0 ; t < ntables ; t++) + { + /* + * catcacheclock is updated by transaction timestamp, so needs to + * be updated by other means for this test to work. Here I choosed + * to update the clock every 1000 tables scan. + */ + if (--ct < 0) + { + SetCatCacheClock(GetCurrentTimestamp()); + ct = clock_step; + } + for (a = 0 ; a < natts ; a++) + { + HeapTuple tup; + + tup = SearchSysCache3(STATRELATTINH, + ObjectIdGetDatum(tableoids[t]), + Int16GetDatum(attnums[a]), + BoolGetDatum(false)); + /* should be null, but.. */ + if (HeapTupleIsValid(tup)) + ReleaseSysCache(tup); + } + } + } + INSTR_TIME_SET_CURRENT(duration); + INSTR_TIME_SUBTRACT(duration, start); + PG_SETMASK(&UnBlockSig); + + return INSTR_TIME_GET_MILLISEC(duration); +}; + +void +catcachewarmup(void) +{ + int t, a; + + /* load up catalog tables */ + for (t = 0 ; t < ntables ; t++) + { + for (a = 0 ; a < natts ; a++) + { + HeapTuple tup; + + tup = SearchSysCache3(STATRELATTINH, + ObjectIdGetDatum(tableoids[t]), + Int16GetDatum(attnums[a]), + BoolGetDatum(false)); + /* should be null, but.. */ + if (HeapTupleIsValid(tup)) + ReleaseSysCache(tup); + } + } +} + +void +collectinfo(void) +{ + int ret; + Datum values[10000]; + bool nulls[10000]; + Oid types0[] = {OIDOID}; + int i; + + ntables = 0; + natts = 0; + + SPI_connect(); + /* collect target tables */ + ret = SPI_execute("select oid from pg_class where relnamespace = (select oid from pg_namespace where nspname = \'test\')", + true, 0); + if (ret != SPI_OK_SELECT) + elog(ERROR, "Failed 1"); + if (SPI_processed == 0) + elog(ERROR, "no relation found in schema \"test\""); + if (SPI_processed > 10000) + elog(ERROR, "too many relation found in schema \"test\""); + + for (i = 0 ; i < SPI_processed ; i++) + { + heap_deform_tuple(SPI_tuptable->vals[i], SPI_tuptable->tupdesc, + values, nulls); + if (nulls[0]) + elog(ERROR, "Failed 2"); + + tableoids[ntables++] = DatumGetObjectId(values[0]); + } + SPI_finish(); + elog(DEBUG1, "%d tables found", ntables); + + values[0] = ObjectIdGetDatum(tableoids[0]); + nulls[0] = false; + SPI_connect(); + ret = SPI_execute_with_args("select attnum from pg_attribute where attrelid = (select oid from pg_class where oid =$1)", + 1, types0, values, NULL, true, 0); + if (SPI_processed == 0) + elog(ERROR, "no attribute found in table %d", tableoids[0]); + if (SPI_processed > 10000) + elog(ERROR, "too many relation found in table %d", tableoids[0]); + + /* collect target attributes. assuming all tables have the same attnums */ + for (i = 0 ; i < SPI_processed ; i++) + { + int16 attnum; + + heap_deform_tuple(SPI_tuptable->vals[i], SPI_tuptable->tupdesc, + values, nulls); + if (nulls[0]) + elog(ERROR, "Failed 3"); + attnum = DatumGetInt16(values[0]); + + if (attnum > 0) + attnums[natts++] = attnum; + } + SPI_finish(); + elog(DEBUG1, "%d attributes found", natts); +} diff --git a/contrib/catcachebench/catcachebench.control b/contrib/catcachebench/catcachebench.control new file mode 100644 index 0000000000..3fc9d2e420 --- /dev/null +++ b/contrib/catcachebench/catcachebench.control @@ -0,0 +1,6 @@ +# catcachebench + +comment = 'benchmark for catcache pruning' +default_version = '0.0' +module_pathname = '$libdir/catcachebench' +relocatable = true diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 0c68c04caa..35e1a07e57 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -814,6 +814,41 @@ CatalogCacheFlushCatalog(Oid catId) CACHE_elog(DEBUG2, "end of CatalogCacheFlushCatalog call"); } + +/* FUNCTION FOR BENCHMARKING */ +void +CatalogCacheFlushCatalog2(Oid catId) +{ + slist_iter iter; + + CACHE_elog(DEBUG2, "CatalogCacheFlushCatalog called for %u", catId); + + slist_foreach(iter, &CacheHdr->ch_caches) + { + CatCache *cache = slist_container(CatCache, cc_next, iter.cur); + + /* Does this cache store tuples of the target catalog? */ + if (cache->cc_reloid == catId) + { + /* Yes, so flush all its contents */ + ResetCatalogCache(cache); + + /* Tell inval.c to call syscache callbacks for this cache */ + CallSyscacheCallbacks(cache->id, 0); + + cache->cc_nbuckets = 128; + pfree(cache->cc_bucket); + cache->cc_bucket = palloc0(128 * sizeof(dlist_head)); + ereport(DEBUG1, + (errmsg("Catcache reset"), + errhidestmt(true))); + } + } + + CACHE_elog(DEBUG2, "end of CatalogCacheFlushCatalog call"); +} +/* END: FUNCTION FOR BENCHMARKING */ + /* * InitCatCache * diff --git a/src/backend/utils/cache/syscache.c b/src/backend/utils/cache/syscache.c index 53d9ddf159..1c79a85a8c 100644 --- a/src/backend/utils/cache/syscache.c +++ b/src/backend/utils/cache/syscache.c @@ -983,7 +983,7 @@ static const struct cachedesc cacheinfo[] = { } }; -static CatCache *SysCache[SysCacheSize]; +CatCache *SysCache[SysCacheSize]; static bool CacheInitialized = false; diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h index f9e9889339..dc0ad1a268 100644 --- a/src/include/utils/catcache.h +++ b/src/include/utils/catcache.h @@ -278,4 +278,7 @@ extern void PrepareToInvalidateCacheTuple(Relation relation, extern void PrintCatCacheLeakWarning(HeapTuple tuple); extern void PrintCatCacheListLeakWarning(CatCList *list); +/* tentative change to allow benchmark on master branch */ +#define SetCatCacheClock(ts) (ts) + #endif /* CATCACHE_H */ -- 2.23.0 #! /usr/bin/perl use Expect; $prune_age = -1; $warmup_secs = 10; $interval = 10; my $exp = Expect->spawn("psql", "postgres") or die "cannot execute psql: $!\n"; $exp->log_stdout(0); #print $exp "set track_catalog_cache_usage_interval to 1000;\n"; #$exp->expect(10, "postgres=#"); print $exp "set catalog_cache_prune_min_age to '${prune_age}s';\n"; $exp->expect(10, "postgres=#"); $starttime = time(); $count = 0; $mean = 0; $nexttime = $starttime + $warmup_secs; $firsttime = 1; while (1) { print $exp "begin; create temp table t1 (a int, b int, c int, d int, e int, f int, g int, h int, i int, j int) on commitdrop; insert into t1 values (1, 2, 3, 4, 5, 6, 7, 8, 9, 10); select * from t1; commit;\n"; $exp->expect(10, "postgres=#"); $count++; if (time() > $nexttime) { if ($firsttime) { $count = 0; $firsttime = 0; } elsif ($mean == 0) { $mean = $count; } else { $mean = $mean * 0.9 + $count * 0.1; } printf STDERR "%6d : %9.2f\n", $count, $mean if ($mean > 0); $count = 0; $nexttime += $interval; } }
pgsql-hackers by date: