Re: Protect syscache from bloating with negative cache entries - Mailing list pgsql-hackers
| From | Kyotaro HORIGUCHI |
|---|---|
| Subject | Re: Protect syscache from bloating with negative cache entries |
| Date | |
| Msg-id | 20190207.211845.263152487.horiguchi.kyotaro@lab.ntt.co.jp Whole thread Raw |
| In response to | Re: Protect syscache from bloating with negative cache entries (Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>) |
| Responses |
RE: Protect syscache from bloating with negative cache entries
Re: Protect syscache from bloating with negative cache entries RE: Protect syscache from bloating with negative cache entries |
| List | pgsql-hackers |
At Thu, 07 Feb 2019 15:24:18 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in
<20190207.152418.139132570.horiguchi.kyotaro@lab.ntt.co.jp>
> I'm going to retake numbers with search-only queries.
Yeah, I was stupid.
I made a rerun of benchmark using "-S -T 30" on the server build
with no assertion and -O2. The numbers are the best of three
successive attempts. The patched version is running with
cache_target_memory = 0, cache_prune_min_age = 600 and
cache_entry_limit = 0 but pruning doesn't happen by the workload.
master: 13393 tps
v12 : 12625 tps (-6%)
Significant degradation is found.
Recuded frequency of dlist_move_tail by taking 1ms interval
between two succesive updates on the same entry let the
degradation dissapear.
patched : 13720 tps (+2%)
I think there's still no need of such frequency. It is 100ms in
the attched patch.
# I'm not sure the name LRU_IGNORANCE_INTERVAL makes sens..
The attached
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
From 72a569703662b93fb57c55c337b16107ebccfce3 Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horiguchi.kyotaro@lab.ntt.co.jp>
Date: Thu, 7 Feb 2019 14:56:07 +0900
Subject: [PATCH 1/4] Add dlist_move_tail
We have dlist_push_head/tail and dlist_move_head but not
dlist_move_tail. Add it.
---
src/include/lib/ilist.h | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/src/include/lib/ilist.h b/src/include/lib/ilist.h
index b1a5974ee4..659ab1ac87 100644
--- a/src/include/lib/ilist.h
+++ b/src/include/lib/ilist.h
@@ -394,6 +394,25 @@ dlist_move_head(dlist_head *head, dlist_node *node)
dlist_check(head);
}
+/*
+ * Move element from its current position in the list to the tail position in
+ * the same list.
+ *
+ * Undefined behaviour if 'node' is not already part of the list.
+ */
+static inline void
+dlist_move_tail(dlist_head *head, dlist_node *node)
+{
+ /* fast path if it's already at the tail */
+ if (head->head.prev == node)
+ return;
+
+ dlist_delete(node);
+ dlist_push_tail(head, node);
+
+ dlist_check(head);
+}
+
/*
* Check whether 'node' has a following node.
* Caution: unreliable if 'node' is not in the list.
--
2.16.3
From 429001e7cbbb88710cfc5589bc46e2490f93d216 Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horiguchi.kyotaro@lab.ntt.co.jp>
Date: Tue, 16 Oct 2018 13:04:30 +0900
Subject: [PATCH 2/4] Remove entries that haven't been used for a certain time
Catcache entries can be left alone for several reasons. It is not
desirable that they eat up memory. With this patch, This adds
consideration of removal of entries that haven't been used for a
certain time before enlarging the hash array.
This also can put a hard limit on the number of catcache entries.
---
doc/src/sgml/config.sgml | 38 +++++
src/backend/access/transam/xact.c | 5 +
src/backend/utils/cache/catcache.c | 205 +++++++++++++++++++++++++-
src/backend/utils/misc/guc.c | 63 ++++++++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/utils/catcache.h | 33 ++++-
6 files changed, 338 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 9b7a7388d5..d0d2374944 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1662,6 +1662,44 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-syscache-memory-target" xreflabel="syscache_memory_target">
+ <term><varname>syscache_memory_target</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>syscache_memory_target</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to which syscache is expanded
+ without pruning. The value defaults to 0, indicating that pruning is
+ always considered. After exceeding this size, syscache pruning is
+ considered according to
+ <xref linkend="guc-syscache-prune-min-age"/>. If you need to keep
+ certain amount of syscache entries with intermittent usage, try
+ increase this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="guc-syscache-prune-min-age" xreflabel="syscache_prune_min_age">
+ <term><varname>syscache_prune_min_age</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>syscache_prune_min_age</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the minimum amount of unused time in seconds at which a
+ syscache entry is considered to be removed. -1 indicates that syscache
+ pruning is disabled at all. The value defaults to 600 seconds
+ (<literal>10 minutes</literal>). The syscache entries that are not
+ used for the duration can be removed to prevent syscache bloat. This
+ behavior is suppressed until the size of syscache exceeds
+ <xref linkend="guc-syscache-memory-target"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 92bda87804..ddc433c59e 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -734,7 +734,12 @@ void
SetCurrentStatementStartTimestamp(void)
{
if (!IsParallelWorker())
+ {
stmtStartTimestamp = GetCurrentTimestamp();
+
+ /* Set this timestamp as aproximated current time */
+ SetCatCacheClock(stmtStartTimestamp);
+ }
else
Assert(stmtStartTimestamp != 0);
}
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 258a1d64cc..c70ce3b745 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -71,9 +71,38 @@
#define CACHE6_elog(a,b,c,d,e,f,g)
#endif
+/*
+ * GUC variable to define the minimum size of hash to cosider entry eviction.
+ * This variable is shared among various cache mechanisms.
+ */
+int cache_memory_target = 0;
+
+
+/*
+ * GUC for entry limit. Entries are removed when the number of them goes above
+ * cache_entry_limit by the ratio specified by cache_entry_limit_prune_ratio
+ */
+int cache_entry_limit = 0;
+double cache_entry_limit_prune_ratio = 0.8;
+
+/* GUC variable to define the minimum age of entries that will be cosidered to
+ * be evicted in seconds. This variable is shared among various cache
+ * mechanisms.
+ */
+int cache_prune_min_age = 600;
+
+/*
+ * Ignorance interval between two success move of a cache entry in LRU list,
+ * in microseconds.
+ */
+#define LRU_IGNORANCE_INTERVAL 100000 /* 100ms */
+
/* Cache management header --- pointer is NULL until created */
static CatCacheHeader *CacheHdr = NULL;
+/* Timestamp used for any operation on caches. */
+TimestampTz catcacheclock = 0;
+
static inline HeapTuple SearchCatCacheInternal(CatCache *cache,
int nkeys,
Datum v1, Datum v2,
@@ -481,6 +510,7 @@ CatCacheRemoveCTup(CatCache *cache, CatCTup *ct)
/* delink from linked list */
dlist_delete(&ct->cache_elem);
+ dlist_delete(&ct->lru_node);
/*
* Free keys when we're dealing with a negative entry, normal entries just
@@ -490,6 +520,7 @@ CatCacheRemoveCTup(CatCache *cache, CatCTup *ct)
CatCacheFreeKeys(cache->cc_tupdesc, cache->cc_nkeys,
cache->cc_keyno, ct->keys);
+ cache->cc_tupsize -= ct->size;
pfree(ct);
--cache->cc_ntup;
@@ -841,7 +872,9 @@ InitCatCache(int id,
cp->cc_nkeys = nkeys;
for (i = 0; i < nkeys; ++i)
cp->cc_keyno[i] = key[i];
+ cp->cc_tupsize = 0;
+ dlist_init(&cp->cc_lru_list);
/*
* new cache is initialized as far as we can go for now. print some
* debugging information, if appropriate.
@@ -858,9 +891,133 @@ InitCatCache(int id,
*/
MemoryContextSwitchTo(oldcxt);
+ /* initilize catcache reference clock if haven't done yet */
+ if (catcacheclock == 0)
+ catcacheclock = GetCurrentTimestamp();
+
return cp;
}
+/*
+ * CatCacheCleanupOldEntries - Remove infrequently-used entries
+ *
+ * Catcache entries can be left alone for several reasons. We remove them if
+ * they are not accessed for a certain time to prevent catcache from
+ * bloating. The eviction is performed with the similar algorithm with buffer
+ * eviction using access counter. Entries that are accessed several times can
+ * live longer than those that have had no access in the same duration.
+ */
+#define PRUNE_BY_AGE 0x01
+#define PRUNE_BY_NUMBER 0x02
+
+static bool
+CatCacheCleanupOldEntries(CatCache *cp)
+{
+ int nremoved = 0;
+ size_t hash_size;
+ int nelems_before = cp->cc_ntup;
+ int ndelelems = 0;
+ int action = 0;
+ dlist_mutable_iter iter;
+
+ if (cache_prune_min_age >= 0)
+ {
+ /* prune only if the size of the hash is above the target */
+
+ hash_size = cp->cc_nbuckets * sizeof(dlist_head);
+ if (hash_size + cp->cc_tupsize > (Size) cache_memory_target * 1024L)
+ action |= PRUNE_BY_AGE;
+ }
+
+ if (cache_entry_limit > 0 && nelems_before >= cache_entry_limit)
+ {
+ ndelelems = nelems_before -
+ (int) (cache_entry_limit * cache_entry_limit_prune_ratio);
+
+ if (ndelelems < 256)
+ ndelelems = 256;
+ if (ndelelems > nelems_before)
+ ndelelems = nelems_before;
+
+ action |= PRUNE_BY_NUMBER;
+ }
+
+ /* Return immediately if no pruning is wanted */
+ if (action == 0)
+ return false;
+
+ /* Scan over LRU to find entries to remove */
+ dlist_foreach_modify(iter, &cp->cc_lru_list)
+ {
+ CatCTup *ct = dlist_container(CatCTup, lru_node, iter.cur);
+ bool remove_this = false;
+
+ /* We don't remove referenced entry */
+ if (ct->refcount != 0 ||
+ (ct->c_list && ct->c_list->refcount != 0))
+ continue;
+
+ /* check against age */
+ if (action & PRUNE_BY_AGE)
+ {
+ long entry_age;
+ int us;
+
+ /*
+ * Calculate the duration from the time of the last access to the
+ * "current" time. Since catcacheclock is not advanced within a
+ * transaction, the entries that are accessed within the current
+ * transaction won't be pruned.
+ */
+ TimestampDifference(ct->lastaccess, catcacheclock, &entry_age, &us);
+
+ if (entry_age < cache_prune_min_age)
+ {
+ /* no longer have a business with further entries, exit */
+ action &= ~PRUNE_BY_AGE;
+ break;
+ }
+
+ /*
+ * Entries that are not accessed after last pruning are removed in
+ * that seconds, and that has been accessed several times are
+ * removed after leaving alone for up to three times of the
+ * duration. We don't try shrink buckets since pruning effectively
+ * caps catcache expansion in the long term.
+ */
+ if (ct->naccess > 0)
+ ct->naccess--;
+ else
+ remove_this = true;
+ }
+
+ /* check against entry number */
+ if (action & PRUNE_BY_NUMBER)
+ {
+ if (nremoved < ndelelems)
+ remove_this = true;
+ else
+ action &= ~PRUNE_BY_NUMBER; /* satisfied */
+ }
+
+ /* exit if finished */
+ if (action == 0)
+ break;
+
+ /* do the work */
+ if (remove_this)
+ {
+ CatCacheRemoveCTup(cp, ct);
+ nremoved++;
+ }
+ }
+
+ elog(DEBUG1, "pruning catalog cache id=%d for %s: removed %d / %d",
+ cp->id, cp->cc_relname, nremoved, nelems_before);
+
+ return nremoved > 0;
+}
+
/*
* Enlarge a catcache, doubling the number of buckets.
*/
@@ -1274,6 +1431,21 @@ SearchCatCacheInternal(CatCache *cache,
*/
dlist_move_head(bucket, &ct->cache_elem);
+ /* Update access information for pruning */
+ if (ct->naccess < 2)
+ ct->naccess++;
+
+ /*
+ * We don't want too frequent update of LRU. cache_prune_min_age can
+ * be changed on-session so we need to maintan the LRU regardless of
+ * cache_prune_min_age.
+ */
+ if (catcacheclock - ct->lastaccess > LRU_IGNORANCE_INTERVAL)
+ {
+ ct->lastaccess = catcacheclock;
+ dlist_move_tail(&cache->cc_lru_list, &ct->lru_node);
+ }
+
/*
* If it's a positive entry, bump its refcount and return it. If it's
* negative, we can report failure to the caller.
@@ -1819,11 +1991,13 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
CatCTup *ct;
HeapTuple dtp;
MemoryContext oldcxt;
+ int tupsize = 0;
/* negative entries have no tuple associated */
if (ntp)
{
int i;
+ int tupsize;
Assert(!negative);
@@ -1842,13 +2016,14 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
/* Allocate memory for CatCTup and the cached tuple in one go */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
- ct = (CatCTup *) palloc(sizeof(CatCTup) +
- MAXIMUM_ALIGNOF + dtp->t_len);
+ tupsize = sizeof(CatCTup) + MAXIMUM_ALIGNOF + dtp->t_len;
+ ct = (CatCTup *) palloc(tupsize);
ct->tuple.t_len = dtp->t_len;
ct->tuple.t_self = dtp->t_self;
ct->tuple.t_tableOid = dtp->t_tableOid;
ct->tuple.t_data = (HeapTupleHeader)
MAXALIGN(((char *) ct) + sizeof(CatCTup));
+ ct->size = tupsize;
/* copy tuple contents */
memcpy((char *) ct->tuple.t_data,
(const char *) dtp->t_data,
@@ -1876,8 +2051,8 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
{
Assert(negative);
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
- ct = (CatCTup *) palloc(sizeof(CatCTup));
-
+ tupsize = sizeof(CatCTup);
+ ct = (CatCTup *) palloc(tupsize);
/*
* Store keys - they'll point into separately allocated memory if not
* by-value.
@@ -1898,18 +2073,34 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
ct->dead = false;
ct->negative = negative;
ct->hash_value = hashValue;
+ ct->naccess = 0;
+ ct->lastaccess = catcacheclock;
+ dlist_push_tail(&cache->cc_lru_list, &ct->lru_node);
+ ct->size = tupsize;
dlist_push_head(&cache->cc_bucket[hashIndex], &ct->cache_elem);
cache->cc_ntup++;
CacheHdr->ch_ntup++;
+ cache->cc_tupsize += tupsize;
+
+ /* increase refcount so that this survives pruning */
+ ct->refcount++;
/*
- * If the hash table has become too full, enlarge the buckets array. Quite
- * arbitrarily, we enlarge when fill factor > 2.
+ * If the hash table has become too full, try cleanup by removing
+ * infrequently used entries to make a room for the new entry. If it
+ * failed, enlarge the bucket array instead. Quite arbitrarily, we try
+ * this when fill factor > 2.
*/
- if (cache->cc_ntup > cache->cc_nbuckets * 2)
+ if (cache->cc_ntup > cache->cc_nbuckets * 2 &&
+ !CatCacheCleanupOldEntries(cache))
RehashCatCache(cache);
+ /* we may still want to prune by entry number, check it */
+ else if (cache_entry_limit > 0 && cache->cc_ntup > cache_entry_limit)
+ CatCacheCleanupOldEntries(cache);
+
+ ct->refcount--;
return ct;
}
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 8681ada33a..d4df841982 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -81,6 +81,7 @@
#include "tsearch/ts_cache.h"
#include "utils/builtins.h"
#include "utils/bytea.h"
+#include "utils/catcache.h"
#include "utils/guc_tables.h"
#include "utils/float.h"
#include "utils/memutils.h"
@@ -2204,6 +2205,58 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"cache_memory_target", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the minimum syscache size to keep."),
+ gettext_noop("Cache is not pruned before exceeding this size."),
+ GUC_UNIT_KB
+ },
+ &cache_memory_target,
+ 0, 0, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"cache_prune_min_age", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the minimum unused duration of cache entries before removal."),
+ gettext_noop("Cache entries that live unused for longer than this seconds are considered to be
removed."),
+ GUC_UNIT_S
+ },
+ &cache_prune_min_age,
+ 600, -1, INT_MAX,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"cache_entry_limit", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum entries of catcache."),
+ NULL
+ },
+ &cache_entry_limit,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"cache_entry_limit", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum entries of catcache."),
+ NULL
+ },
+ &cache_entry_limit,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"cache_entry_limit", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum entries of catcache."),
+ NULL
+ },
+ &cache_entry_limit,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
@@ -3368,6 +3421,16 @@ static struct config_real ConfigureNamesReal[] =
NULL, NULL, NULL
},
+ {
+ {"cache_entry_limit_prune_ratio", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum entries of catcache."),
+ NULL
+ },
+ &cache_entry_limit_prune_ratio,
+ 0.8, 0.0, 1.0,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0.0, 0.0, 0.0, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index c7f53470df..108d332f2c 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -128,6 +128,8 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#cache_memory_target = 0kB # in kB
+#cache_prune_min_age = 600s # -1 disables pruning
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index 65d816a583..3c6842e272 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -22,6 +22,7 @@
#include "access/htup.h"
#include "access/skey.h"
+#include "datatype/timestamp.h"
#include "lib/ilist.h"
#include "utils/relcache.h"
@@ -61,6 +62,9 @@ typedef struct catcache
slist_node cc_next; /* list link */
ScanKeyData cc_skey[CATCACHE_MAXKEYS]; /* precomputed key info for heap
* scans */
+ dlist_head cc_lru_list;
+ int cc_tupsize; /* total amount of catcache tuples */
+ int cc_nfreeent; /* # of entries currently not referenced */
/*
* Keep these at the end, so that compiling catcache.c with CATCACHE_STATS
@@ -119,7 +123,10 @@ typedef struct catctup
bool dead; /* dead but not yet removed? */
bool negative; /* negative cache entry? */
HeapTupleData tuple; /* tuple management header */
-
+ int naccess; /* # of access to this entry, up to 2 */
+ TimestampTz lastaccess; /* approx. timestamp of the last usage */
+ dlist_node lru_node; /* LRU node */
+ int size; /* palloc'ed size off this tuple */
/*
* The tuple may also be a member of at most one CatCList. (If a single
* catcache is list-searched with varying numbers of keys, we may have to
@@ -189,6 +196,30 @@ typedef struct catcacheheader
/* this extern duplicates utils/memutils.h... */
extern PGDLLIMPORT MemoryContext CacheMemoryContext;
+/* for guc.c, not PGDLLPMPORT'ed */
+extern int cache_prune_min_age;
+extern int cache_memory_target;
+extern int cache_entry_limit;
+extern double cache_entry_limit_prune_ratio;
+
+/* to use as access timestamp of catcache entries */
+extern TimestampTz catcacheclock;
+
+/*
+ * SetCatCacheClock - set timestamp for catcache access record
+ */
+static inline void
+SetCatCacheClock(TimestampTz ts)
+{
+ catcacheclock = ts;
+}
+
+static inline TimestampTz
+GetCatCacheClock(void)
+{
+ return catcacheclock;
+}
+
extern void CreateCacheMemoryContext(void);
extern CatCache *InitCatCache(int id, Oid reloid, Oid indexoid,
--
2.16.3
From 251607ff21981f840392387a28ca8f012ef18aab Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horiguchi.kyotaro@lab.ntt.co.jp>
Date: Tue, 16 Oct 2018 15:48:28 +0900
Subject: [PATCH 3/4] Syscache usage tracking feature.
Collects syscache usage statictics and show it using the view
pg_stat_syscache. The feature is controlled by the GUC variable
track_syscache_usage_interval.
---
doc/src/sgml/config.sgml | 15 ++
src/backend/catalog/system_views.sql | 17 +++
src/backend/postmaster/pgstat.c | 201 ++++++++++++++++++++++++--
src/backend/tcop/postgres.c | 23 +++
src/backend/utils/adt/pgstatfuncs.c | 134 +++++++++++++++++
src/backend/utils/cache/catcache.c | 89 +++++++++---
src/backend/utils/cache/syscache.c | 24 +++
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 11 ++
src/backend/utils/misc/guc.c | 10 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/catalog/pg_proc.dat | 9 ++
src/include/miscadmin.h | 1 +
src/include/pgstat.h | 6 +-
src/include/utils/catcache.h | 9 +-
src/include/utils/syscache.h | 19 +++
src/include/utils/timeout.h | 1 +
src/test/regress/expected/rules.out | 24 ++-
18 files changed, 559 insertions(+), 36 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d0d2374944..5ff3ebeb4e 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -6687,6 +6687,21 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
</listitem>
</varlistentry>
+ <varlistentry id="guc-track-syscache-usage-interval" xreflabel="track_syscache_usage_interval">
+ <term><varname>track_syscache_usage_interval</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>track_syscache_usage_interval</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the interval to collect system cache usage statistics in
+ milliseconds. This parameter is 0 by default, which means disabled.
+ Only superusers can change this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-io-timing" xreflabel="track_io_timing">
<term><varname>track_io_timing</varname> (<type>boolean</type>)
<indexterm>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 3e229c693c..f5d1aaf96f 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -906,6 +906,22 @@ CREATE VIEW pg_stat_progress_vacuum AS
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
+CREATE VIEW pg_stat_syscache AS
+ SELECT
+ S.pid AS pid,
+ S.relid::regclass AS relname,
+ S.indid::regclass AS cache_name,
+ S.size AS size,
+ S.ntup AS ntuples,
+ S.searches AS searches,
+ S.hits AS hits,
+ S.neg_hits AS neg_hits,
+ S.ageclass AS ageclass,
+ S.last_update AS last_update
+ FROM pg_stat_activity A
+ JOIN LATERAL (SELECT A.pid, * FROM pg_get_syscache_stats(A.pid)) S
+ ON (A.pid = S.pid);
+
CREATE VIEW pg_user_mappings AS
SELECT
U.oid AS umid,
@@ -1185,6 +1201,7 @@ GRANT EXECUTE ON FUNCTION pg_ls_waldir() TO pg_monitor;
GRANT EXECUTE ON FUNCTION pg_ls_archive_statusdir() TO pg_monitor;
GRANT EXECUTE ON FUNCTION pg_ls_tmpdir() TO pg_monitor;
GRANT EXECUTE ON FUNCTION pg_ls_tmpdir(oid) TO pg_monitor;
+GRANT EXECUTE ON FUNCTION pg_get_syscache_stats(int) TO pg_monitor;
GRANT pg_read_all_settings TO pg_monitor;
GRANT pg_read_all_stats TO pg_monitor;
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 81c6499251..a1939958b7 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -66,6 +66,7 @@
#include "utils/ps_status.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
+#include "utils/syscache.h"
#include "utils/timestamp.h"
@@ -124,6 +125,7 @@
bool pgstat_track_activities = false;
bool pgstat_track_counts = false;
int pgstat_track_functions = TRACK_FUNC_OFF;
+int pgstat_track_syscache_usage_interval = 0;
int pgstat_track_activity_query_size = 1024;
/* ----------
@@ -236,6 +238,11 @@ typedef struct TwoPhasePgStatRecord
bool t_truncated; /* was the relation truncated? */
} TwoPhasePgStatRecord;
+/* bitmap symbols to specify target file types remove */
+#define PGSTAT_REMFILE_DBSTAT 1 /* remove only databsae stats files */
+#define PGSTAT_REMFILE_SYSCACHE 2 /* remove only syscache stats files */
+#define PGSTAT_REMFILE_ALL 3 /* remove both type of files */
+
/*
* Info about current "snapshot" of stats file
*/
@@ -335,6 +342,7 @@ static void pgstat_recv_funcpurge(PgStat_MsgFuncpurge *msg, int len);
static void pgstat_recv_recoveryconflict(PgStat_MsgRecoveryConflict *msg, int len);
static void pgstat_recv_deadlock(PgStat_MsgDeadlock *msg, int len);
static void pgstat_recv_tempfile(PgStat_MsgTempFile *msg, int len);
+static void pgstat_remove_syscache_statsfile(void);
/* ------------------------------------------------------------
* Public functions called from postmaster follow
@@ -630,10 +638,13 @@ startup_failed:
}
/*
- * subroutine for pgstat_reset_all
+ * remove stats files
+ *
+ * clean up stats files in specified directory. target is one of
+ * PGSTAT_REFILE_DBSTAT/SYSCACHE/ALL and restricts files to remove.
*/
static void
-pgstat_reset_remove_files(const char *directory)
+pgstat_reset_remove_files(const char *directory, int target)
{
DIR *dir;
struct dirent *entry;
@@ -644,25 +655,39 @@ pgstat_reset_remove_files(const char *directory)
{
int nchars;
Oid tmp_oid;
+ int filetype = 0;
/*
* Skip directory entries that don't match the file names we write.
* See get_dbstat_filename for the database-specific pattern.
*/
if (strncmp(entry->d_name, "global.", 7) == 0)
+ {
+ filetype = PGSTAT_REMFILE_DBSTAT;
nchars = 7;
+ }
else
{
+ char head[2];
+
nchars = 0;
- (void) sscanf(entry->d_name, "db_%u.%n",
- &tmp_oid, &nchars);
- if (nchars <= 0)
- continue;
+ (void) sscanf(entry->d_name, "%c%c_%u.%n",
+ head, head + 1, &tmp_oid, &nchars);
+
/* %u allows leading whitespace, so reject that */
- if (strchr("0123456789", entry->d_name[3]) == NULL)
+ if (nchars < 3 || !isdigit(entry->d_name[3]))
continue;
+
+ if (strncmp(head, "db", 2) == 0)
+ filetype = PGSTAT_REMFILE_DBSTAT;
+ else if (strncmp(head, "cc", 2) == 0)
+ filetype = PGSTAT_REMFILE_SYSCACHE;
}
+ /* skip if this is not a target */
+ if ((filetype & target) == 0)
+ continue;
+
if (strcmp(entry->d_name + nchars, "tmp") != 0 &&
strcmp(entry->d_name + nchars, "stat") != 0)
continue;
@@ -683,8 +708,9 @@ pgstat_reset_remove_files(const char *directory)
void
pgstat_reset_all(void)
{
- pgstat_reset_remove_files(pgstat_stat_directory);
- pgstat_reset_remove_files(PGSTAT_STAT_PERMANENT_DIRECTORY);
+ pgstat_reset_remove_files(pgstat_stat_directory, PGSTAT_REMFILE_ALL);
+ pgstat_reset_remove_files(PGSTAT_STAT_PERMANENT_DIRECTORY,
+ PGSTAT_REMFILE_ALL);
}
#ifdef EXEC_BACKEND
@@ -2963,6 +2989,10 @@ pgstat_beshutdown_hook(int code, Datum arg)
if (OidIsValid(MyDatabaseId))
pgstat_report_stat(true);
+ /* clear syscache statistics files and temprary settings */
+ if (MyBackendId != InvalidBackendId)
+ pgstat_remove_syscache_statsfile();
+
/*
* Clear my status entry, following the protocol of bumping st_changecount
* before and after. We use a volatile pointer here to ensure the
@@ -4287,6 +4317,9 @@ PgstatCollectorMain(int argc, char *argv[])
pgStatRunningInCollector = true;
pgStatDBHash = pgstat_read_statsfiles(InvalidOid, true, true);
+ /* Remove left-over syscache stats files */
+ pgstat_reset_remove_files(pgstat_stat_directory, PGSTAT_REMFILE_SYSCACHE);
+
/*
* Loop to process messages until we get SIGQUIT or detect ungraceful
* death of our parent postmaster.
@@ -6377,3 +6410,153 @@ pgstat_clip_activity(const char *raw_activity)
return activity;
}
+
+/*
+ * return the filename for a syscache stat file; filename is the output
+ * buffer, of length len.
+ */
+void
+pgstat_get_syscachestat_filename(bool permanent, bool tempname, int backendid,
+ char *filename, int len)
+{
+ int printed;
+
+ /* NB -- pgstat_reset_remove_files knows about the pattern this uses */
+ printed = snprintf(filename, len, "%s/cc_%u.%s",
+ permanent ? PGSTAT_STAT_PERMANENT_DIRECTORY :
+ pgstat_stat_directory,
+ backendid,
+ tempname ? "tmp" : "stat");
+ if (printed >= len)
+ elog(ERROR, "overlength pgstat path");
+}
+
+/* removes syscache stats files of this backend */
+static void
+pgstat_remove_syscache_statsfile(void)
+{
+ char fname[MAXPGPATH];
+
+ pgstat_get_syscachestat_filename(false, false, MyBackendId,
+ fname, MAXPGPATH);
+ unlink(fname); /* don't care of the result */
+}
+
+/*
+ * pgstat_write_syscache_stats() -
+ * Write the syscache statistics files.
+ *
+ * If 'force' is false, this function skips writing a file and resturns the
+ * time remaining in the current interval in milliseconds. If'force' is true,
+ * writes a file regardless of the remaining time and reset the interval.
+ */
+long
+pgstat_write_syscache_stats(bool force)
+{
+ static TimestampTz last_report = 0;
+ TimestampTz now;
+ long elapsed;
+ long secs;
+ int usecs;
+ int cacheId;
+ FILE *fpout;
+ char statfile[MAXPGPATH];
+ char tmpfile[MAXPGPATH];
+
+ /* Return if we don't want it */
+ if (!force && pgstat_track_syscache_usage_interval <= 0)
+ {
+ /* disabled. remove the statistics file if any */
+ if (last_report > 0)
+ {
+ last_report = 0;
+ pgstat_remove_syscache_statsfile();
+ }
+ return 0;
+ }
+
+ /* Check aginst the interval */
+ now = GetCurrentTransactionStopTimestamp();
+ TimestampDifference(last_report, now, &secs, &usecs);
+ elapsed = secs * 1000 + usecs / 1000;
+
+ if (!force && elapsed < pgstat_track_syscache_usage_interval)
+ {
+ /* not yet the time, inform the remaining time to the caller */
+ return pgstat_track_syscache_usage_interval - elapsed;
+ }
+
+ /* now update the stats */
+ last_report = now;
+
+ pgstat_get_syscachestat_filename(false, true,
+ MyBackendId, tmpfile, MAXPGPATH);
+ pgstat_get_syscachestat_filename(false, false,
+ MyBackendId, statfile, MAXPGPATH);
+
+ /*
+ * This function can be called from ProcessInterrupts(). Inhibit recursive
+ * interrupts to avoid recursive entry.
+ */
+ HOLD_INTERRUPTS();
+
+ fpout = AllocateFile(tmpfile, PG_BINARY_W);
+ if (fpout == NULL)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not open temporary statistics file \"%s\": %m",
+ tmpfile)));
+ /*
+ * Failure writing this file is not critical. Just skip this time and
+ * tell caller to wait for the next interval.
+ */
+ RESUME_INTERRUPTS();
+ return pgstat_track_syscache_usage_interval;
+ }
+
+ /* write out every catcache stats */
+ for (cacheId = 0 ; cacheId < SysCacheSize ; cacheId++)
+ {
+ SysCacheStats *stats;
+
+ stats = SysCacheGetStats(cacheId);
+ Assert (stats);
+
+ /* write error is checked later using ferror() */
+ fputc('T', fpout);
+ (void)fwrite(&cacheId, sizeof(int), 1, fpout);
+ (void)fwrite(&last_report, sizeof(TimestampTz), 1, fpout);
+ (void)fwrite(stats, sizeof(*stats), 1, fpout);
+ }
+ fputc('E', fpout);
+
+ if (ferror(fpout))
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not write syscache statistics file \"%s\": %m",
+ tmpfile)));
+ FreeFile(fpout);
+ unlink(tmpfile);
+ }
+ else if (FreeFile(fpout) < 0)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not close syscache statistics file \"%s\": %m",
+ tmpfile)));
+ unlink(tmpfile);
+ }
+ else if (rename(tmpfile, statfile) < 0)
+ {
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("could not rename syscache statistics file \"%s\" to \"%s\": %m",
+ tmpfile, statfile)));
+ unlink(tmpfile);
+ }
+
+ RESUME_INTERRUPTS();
+ return 0;
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 36cfd507b2..fb77a0ce4c 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3157,6 +3157,12 @@ ProcessInterrupts(void)
}
+ if (IdleSyscacheStatsUpdateTimeoutPending)
+ {
+ IdleSyscacheStatsUpdateTimeoutPending = false;
+ pgstat_write_syscache_stats(true);
+ }
+
if (ParallelMessagePending)
HandleParallelMessages();
}
@@ -3733,6 +3739,7 @@ PostgresMain(int argc, char *argv[],
sigjmp_buf local_sigjmp_buf;
volatile bool send_ready_for_query = true;
bool disable_idle_in_transaction_timeout = false;
+ bool disable_idle_catcache_update_timeout = false;
/* Initialize startup process environment if necessary. */
if (!IsUnderPostmaster)
@@ -4173,9 +4180,19 @@ PostgresMain(int argc, char *argv[],
}
else
{
+ long timeout;
+
ProcessCompletedNotifies();
pgstat_report_stat(false);
+ timeout = pgstat_write_syscache_stats(false);
+
+ if (timeout > 0)
+ {
+ disable_idle_catcache_update_timeout = true;
+ enable_timeout_after(IDLE_CATCACHE_UPDATE_TIMEOUT,
+ timeout);
+ }
set_ps_display("idle", false);
pgstat_report_activity(STATE_IDLE, NULL);
}
@@ -4218,6 +4235,12 @@ PostgresMain(int argc, char *argv[],
disable_idle_in_transaction_timeout = false;
}
+ if (disable_idle_catcache_update_timeout)
+ {
+ disable_timeout(IDLE_CATCACHE_UPDATE_TIMEOUT, false);
+ disable_idle_catcache_update_timeout = false;
+ }
+
/*
* (6) check for any other interesting events that happened while we
* slept.
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index b6ba856ebe..6526cfefb4 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -14,6 +14,8 @@
*/
#include "postgres.h"
+#include <sys/stat.h>
+
#include "access/htup_details.h"
#include "catalog/pg_authid.h"
#include "catalog/pg_type.h"
@@ -28,6 +30,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/inet.h"
+#include "utils/syscache.h"
#include "utils/timestamp.h"
#define UINT32_ACCESS_ONCE(var) ((uint32)(*((volatile uint32 *)&(var))))
@@ -1899,3 +1902,134 @@ pg_stat_get_archiver(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(HeapTupleGetDatum(
heap_form_tuple(tupdesc, values, nulls)));
}
+
+Datum
+pgstat_get_syscache_stats(PG_FUNCTION_ARGS)
+{
+#define PG_GET_SYSCACHE_SIZE 9
+ int pid = PG_GETARG_INT32(0);
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ TupleDesc tupdesc;
+ Tuplestorestate *tupstore;
+ MemoryContext per_query_ctx;
+ MemoryContext oldcontext;
+ PgBackendStatus *beentry;
+ int beid;
+ char fname[MAXPGPATH];
+ FILE *fpin;
+ char c;
+
+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("set-valued function called in context that cannot accept a set")));
+ if (!(rsinfo->allowedModes & SFRM_Materialize))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("materialize mode required, but it is not " \
+ "allowed in this context")));
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+
+ per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
+
+ oldcontext = MemoryContextSwitchTo(per_query_ctx);
+ tupstore = tuplestore_begin_heap(true, false, work_mem);
+ rsinfo->returnMode = SFRM_Materialize;
+ rsinfo->setResult = tupstore;
+ rsinfo->setDesc = tupdesc;
+
+ MemoryContextSwitchTo(oldcontext);
+
+ /* find beentry for given pid*/
+ beentry = NULL;
+ for (beid = 1;
+ (beentry = pgstat_fetch_stat_beentry(beid)) &&
+ beentry->st_procpid != pid ;
+ beid++);
+
+ /*
+ * we silently return empty result on failure or insufficient privileges
+ */
+ if (!beentry ||
+ (!has_privs_of_role(GetUserId(), beentry->st_userid) &&
+ !is_member_of_role(GetUserId(), DEFAULT_ROLE_READ_ALL_STATS)))
+ goto no_data;
+
+ pgstat_get_syscachestat_filename(false, false, beid, fname, MAXPGPATH);
+
+ if ((fpin = AllocateFile(fname, PG_BINARY_R)) == NULL)
+ {
+ if (errno != ENOENT)
+ ereport(WARNING,
+ (errcode_for_file_access(),
+ errmsg("could not open statistics file \"%s\": %m",
+ fname)));
+ /* also return empty on no statistics file */
+ goto no_data;
+ }
+
+ /* read the statistics file into tuplestore */
+ while ((c = fgetc(fpin)) == 'T')
+ {
+ TimestampTz last_update;
+ SysCacheStats stats;
+ int cacheid;
+ Datum values[PG_GET_SYSCACHE_SIZE];
+ bool nulls[PG_GET_SYSCACHE_SIZE] = {0};
+ Datum datums[SYSCACHE_STATS_NAGECLASSES * 2];
+ bool arrnulls[SYSCACHE_STATS_NAGECLASSES * 2] = {0};
+ int dims[] = {SYSCACHE_STATS_NAGECLASSES, 2};
+ int lbs[] = {1, 1};
+ ArrayType *arr;
+ int i, j;
+
+ fread(&cacheid, sizeof(int), 1, fpin);
+ fread(&last_update, sizeof(TimestampTz), 1, fpin);
+ if (fread(&stats, 1, sizeof(stats), fpin) != sizeof(stats))
+ {
+ ereport(WARNING,
+ (errmsg("corrupted syscache statistics file \"%s\"",
+ fname)));
+ goto no_data;
+ }
+
+ i = 0;
+ values[i++] = ObjectIdGetDatum(stats.reloid);
+ values[i++] = ObjectIdGetDatum(stats.indoid);
+ values[i++] = Int64GetDatum(stats.size);
+ values[i++] = Int64GetDatum(stats.ntuples);
+ values[i++] = Int64GetDatum(stats.nsearches);
+ values[i++] = Int64GetDatum(stats.nhits);
+ values[i++] = Int64GetDatum(stats.nneg_hits);
+
+ for (j = 0 ; j < SYSCACHE_STATS_NAGECLASSES ; j++)
+ {
+ datums[j * 2] = Int32GetDatum((int32) stats.ageclasses[j]);
+ datums[j * 2 + 1] = Int32GetDatum((int32) stats.nclass_entries[j]);
+ }
+
+ arr = construct_md_array(datums, arrnulls, 2, dims, lbs,
+ INT4OID, sizeof(int32), true, 'i');
+ values[i++] = PointerGetDatum(arr);
+
+ values[i++] = TimestampTzGetDatum(last_update);
+
+ Assert (i == PG_GET_SYSCACHE_SIZE);
+
+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+ }
+
+ /* check for the end of file. abandon the result if file is broken */
+ if (c != 'E' || fgetc(fpin) != EOF)
+ tuplestore_clear(tupstore);
+
+ FreeFile(fpin);
+
+no_data:
+ tuplestore_donestoring(tupstore);
+ return (Datum) 0;
+}
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index c70ce3b745..484fe43e09 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -103,6 +103,10 @@ static CatCacheHeader *CacheHdr = NULL;
/* Timestamp used for any operation on caches. */
TimestampTz catcacheclock = 0;
+/* age classes for pruning */
+static double ageclass[SYSCACHE_STATS_NAGECLASSES]
+ = {0.05, 0.1, 1.0, 2.0, 3.0, 0.0};
+
static inline HeapTuple SearchCatCacheInternal(CatCache *cache,
int nkeys,
Datum v1, Datum v2,
@@ -634,9 +638,7 @@ CatCacheInvalidate(CatCache *cache, uint32 hashValue)
else
CatCacheRemoveCTup(cache, ct);
CACHE1_elog(DEBUG2, "CatCacheInvalidate: invalidated");
-#ifdef CATCACHE_STATS
cache->cc_invals++;
-#endif
/* could be multiple matches, so keep looking! */
}
}
@@ -712,9 +714,7 @@ ResetCatalogCache(CatCache *cache)
}
else
CatCacheRemoveCTup(cache, ct);
-#ifdef CATCACHE_STATS
cache->cc_invals++;
-#endif
}
}
}
@@ -964,10 +964,10 @@ CatCacheCleanupOldEntries(CatCache *cp)
int us;
/*
- * Calculate the duration from the time of the last access to the
- * "current" time. Since catcacheclock is not advanced within a
- * transaction, the entries that are accessed within the current
- * transaction won't be pruned.
+ * Calculate the duration from the time from the last access to
+ * the "current" time. Since catcacheclock is not advanced within
+ * a transaction, the entries that are accessed within the current
+ * transaction always get 0 as the result.
*/
TimestampDifference(ct->lastaccess, catcacheclock, &entry_age, &us);
@@ -1387,9 +1387,7 @@ SearchCatCacheInternal(CatCache *cache,
if (unlikely(cache->cc_tupdesc == NULL))
CatalogCacheInitializeCache(cache);
-#ifdef CATCACHE_STATS
cache->cc_searches++;
-#endif
/* Initialize local parameter array */
arguments[0] = v1;
@@ -1459,9 +1457,7 @@ SearchCatCacheInternal(CatCache *cache,
CACHE3_elog(DEBUG2, "SearchCatCache(%s): found in bucket %d",
cache->cc_relname, hashIndex);
-#ifdef CATCACHE_STATS
cache->cc_hits++;
-#endif
return &ct->tuple;
}
@@ -1470,9 +1466,7 @@ SearchCatCacheInternal(CatCache *cache,
CACHE3_elog(DEBUG2, "SearchCatCache(%s): found neg entry in bucket %d",
cache->cc_relname, hashIndex);
-#ifdef CATCACHE_STATS
cache->cc_neg_hits++;
-#endif
return NULL;
}
@@ -1600,9 +1594,7 @@ SearchCatCacheMiss(CatCache *cache,
CACHE3_elog(DEBUG2, "SearchCatCache(%s): put in bucket %d",
cache->cc_relname, hashIndex);
-#ifdef CATCACHE_STATS
cache->cc_newloads++;
-#endif
return &ct->tuple;
}
@@ -1713,9 +1705,7 @@ SearchCatCacheList(CatCache *cache,
Assert(nkeys > 0 && nkeys < cache->cc_nkeys);
-#ifdef CATCACHE_STATS
cache->cc_lsearches++;
-#endif
/* Initialize local parameter array */
arguments[0] = v1;
@@ -1772,9 +1762,7 @@ SearchCatCacheList(CatCache *cache,
CACHE2_elog(DEBUG2, "SearchCatCacheList(%s): found list",
cache->cc_relname);
-#ifdef CATCACHE_STATS
cache->cc_lhits++;
-#endif
return cl;
}
@@ -2291,3 +2279,64 @@ PrintCatCacheListLeakWarning(CatCList *list)
list->my_cache->cc_relname, list->my_cache->id,
list, list->refcount);
}
+
+/*
+ * CatCacheGetStats - fill in SysCacheStats struct.
+ *
+ * This is a support routine for SysCacheGetStats, substantially fills in the
+ * result. The classification here is based on the same criteria to
+ * CatCacheCleanupOldEntries().
+ */
+void
+CatCacheGetStats(CatCache *cache, SysCacheStats *stats)
+{
+ int i, j;
+
+ Assert(ageclass[SYSCACHE_STATS_NAGECLASSES - 1] == 0.0);
+
+ /* fill in the stats struct */
+ stats->size = cache->cc_tupsize + cache->cc_nbuckets * sizeof(dlist_head);
+ stats->ntuples = cache->cc_ntup;
+ stats->nsearches = cache->cc_searches;
+ stats->nhits = cache->cc_hits;
+ stats->nneg_hits = cache->cc_neg_hits;
+
+ /* cache_prune_min_age can be changed on-session, fill it every time */
+ for (i = 0 ; i < SYSCACHE_STATS_NAGECLASSES ; i++)
+ stats->ageclasses[i] = (int) (cache_prune_min_age * ageclass[i]);
+
+ /*
+ * nth element in nclass_entries stores the number of cache entries that
+ * have lived unaccessed for corresponding multiple in ageclass of
+ * cache_prune_min_age.
+ */
+ memset(stats->nclass_entries, 0, sizeof(int) * SYSCACHE_STATS_NAGECLASSES);
+
+ /* Scan the whole hash */
+ for (i = 0; i < cache->cc_nbuckets; i++)
+ {
+ dlist_mutable_iter iter;
+
+ dlist_foreach_modify(iter, &cache->cc_bucket[i])
+ {
+ CatCTup *ct = dlist_container(CatCTup, cache_elem, iter.cur);
+ long entry_age;
+ int us;
+
+ /*
+ * Calculate the duration from the time from the last access to
+ * the "current" time. Since catcacheclock is not advanced within
+ * a transaction, the entries that are accessed within the current
+ * transaction won't be pruned.
+ */
+ TimestampDifference(ct->lastaccess, catcacheclock, &entry_age, &us);
+
+ j = 0;
+ while (j < SYSCACHE_STATS_NAGECLASSES - 1 &&
+ entry_age > stats->ageclasses[j])
+ j++;
+
+ stats->nclass_entries[j]++;
+ }
+ }
+}
diff --git a/src/backend/utils/cache/syscache.c b/src/backend/utils/cache/syscache.c
index ac98c19155..7b38a06708 100644
--- a/src/backend/utils/cache/syscache.c
+++ b/src/backend/utils/cache/syscache.c
@@ -20,6 +20,9 @@
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/htup_details.h"
#include "access/sysattr.h"
#include "catalog/indexing.h"
@@ -1534,6 +1537,27 @@ RelationSupportsSysCache(Oid relid)
return false;
}
+/*
+ * SysCacheGetStats - returns stats of specified syscache
+ *
+ * This routine returns the address of its local static memory.
+ */
+SysCacheStats *
+SysCacheGetStats(int cacheId)
+{
+ static SysCacheStats stats;
+
+ Assert(cacheId >=0 && cacheId < SysCacheSize);
+
+ memset(&stats, 0, sizeof(stats));
+
+ stats.reloid = cacheinfo[cacheId].reloid;
+ stats.indoid = cacheinfo[cacheId].indoid;
+
+ CatCacheGetStats(SysCache[cacheId], &stats);
+
+ return &stats;
+}
/*
* OID comparator for pg_qsort
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index fd51934aaf..f039ecd805 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -32,6 +32,7 @@ volatile sig_atomic_t QueryCancelPending = false;
volatile sig_atomic_t ProcDiePending = false;
volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
+volatile sig_atomic_t IdleSyscacheStatsUpdateTimeoutPending = false;
volatile sig_atomic_t ConfigReloadPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index c0b6231458..dee7f19475 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -72,6 +72,7 @@ static void ShutdownPostgres(int code, Datum arg);
static void StatementTimeoutHandler(void);
static void LockTimeoutHandler(void);
static void IdleInTransactionSessionTimeoutHandler(void);
+static void IdleSyscacheStatsUpdateTimeoutHandler(void);
static bool ThereIsAtLeastOneRole(void);
static void process_startup_options(Port *port, bool am_superuser);
static void process_settings(Oid databaseid, Oid roleid);
@@ -628,6 +629,8 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username,
RegisterTimeout(LOCK_TIMEOUT, LockTimeoutHandler);
RegisterTimeout(IDLE_IN_TRANSACTION_SESSION_TIMEOUT,
IdleInTransactionSessionTimeoutHandler);
+ RegisterTimeout(IDLE_CATCACHE_UPDATE_TIMEOUT,
+ IdleSyscacheStatsUpdateTimeoutHandler);
}
/*
@@ -1239,6 +1242,14 @@ IdleInTransactionSessionTimeoutHandler(void)
SetLatch(MyLatch);
}
+static void
+IdleSyscacheStatsUpdateTimeoutHandler(void)
+{
+ IdleSyscacheStatsUpdateTimeoutPending = true;
+ InterruptPending = true;
+ SetLatch(MyLatch);
+}
+
/*
* Returns true if at least one role is defined in this database cluster.
*/
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index d4df841982..7bb239a07e 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -3198,6 +3198,16 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"track_syscache_usage_interval", PGC_SUSET, STATS_COLLECTOR,
+ gettext_noop("Sets the interval between syscache usage collection, in milliseconds. Zero disables syscache
usagetracking."),
+ NULL
+ },
+ &pgstat_track_syscache_usage_interval,
+ 0, 0, INT_MAX / 2,
+ NULL, NULL, NULL
+ },
+
{
{"gin_pending_list_limit", PGC_USERSET, CLIENT_CONN_STATEMENT,
gettext_noop("Sets the maximum size of the pending list for GIN index."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 108d332f2c..4d4fb42251 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -560,6 +560,7 @@
#track_io_timing = off
#track_functions = none # none, pl, all
#track_activity_query_size = 1024 # (change requires restart)
+#track_syscache_usage_interval = 0 # zero disables tracking
#stats_temp_directory = 'pg_stat_tmp'
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index b8de13f03b..6099a828d2 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9669,6 +9669,15 @@
proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
proargnames =>
'{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn}',
prosrc => 'pg_get_replication_slots' },
+{ oid => '3425',
+ descr => 'syscache statistics',
+ proname => 'pg_get_syscache_stats', prorows => '100', proisstrict => 'f',
+ proretset => 't', provolatile => 'v', prorettype => 'record',
+ proargtypes => 'int4',
+ proallargtypes => '{int4,oid,oid,int8,int8,int8,int8,int8,_int4,timestamptz}',
+ proargmodes => '{i,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,relid,indid,size,ntup,searches,hits,neg_hits,ageclass,last_update}',
+ prosrc => 'pgstat_get_syscache_stats' },
{ oid => '3786', descr => 'set up a logical replication slot',
proname => 'pg_create_logical_replication_slot', provolatile => 'v',
proparallel => 'u', prorettype => 'record', proargtypes => 'name name bool',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index c9e35003a5..69b9a976f0 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -82,6 +82,7 @@ extern PGDLLIMPORT volatile sig_atomic_t InterruptPending;
extern PGDLLIMPORT volatile sig_atomic_t QueryCancelPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t IdleSyscacheStatsUpdateTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ConfigReloadPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 88a75fb798..b6bfd7d644 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -1144,6 +1144,7 @@ extern bool pgstat_track_activities;
extern bool pgstat_track_counts;
extern int pgstat_track_functions;
extern PGDLLIMPORT int pgstat_track_activity_query_size;
+extern int pgstat_track_syscache_usage_interval;
extern char *pgstat_stat_directory;
extern char *pgstat_stat_tmpname;
extern char *pgstat_stat_filename;
@@ -1228,7 +1229,8 @@ extern PgStat_BackendFunctionEntry *find_funcstat_entry(Oid func_id);
extern void pgstat_initstats(Relation rel);
extern char *pgstat_clip_activity(const char *raw_activity);
-
+extern void pgstat_get_syscachestat_filename(bool permanent,
+ bool tempname, int backendid, char *filename, int len);
/* ----------
* pgstat_report_wait_start() -
*
@@ -1363,5 +1365,5 @@ extern PgStat_StatFuncEntry *pgstat_fetch_stat_funcentry(Oid funcid);
extern int pgstat_fetch_stat_numbackends(void);
extern PgStat_ArchiverStats *pgstat_fetch_stat_archiver(void);
extern PgStat_GlobalStats *pgstat_fetch_global(void);
-
+extern long pgstat_write_syscache_stats(bool force);
#endif /* PGSTAT_H */
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index 3c6842e272..9af414b307 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -67,10 +67,8 @@ typedef struct catcache
int cc_nfreeent; /* # of entries currently not referenced */
/*
- * Keep these at the end, so that compiling catcache.c with CATCACHE_STATS
- * doesn't break ABI for other modules
+ * Statistics entries
*/
-#ifdef CATCACHE_STATS
long cc_searches; /* total # searches against this cache */
long cc_hits; /* # of matches against existing entry */
long cc_neg_hits; /* # of matches against negative entry */
@@ -83,7 +81,6 @@ typedef struct catcache
long cc_invals; /* # of entries invalidated from cache */
long cc_lsearches; /* total # list-searches */
long cc_lhits; /* # of matches against existing lists */
-#endif
} CatCache;
@@ -259,4 +256,8 @@ extern void PrepareToInvalidateCacheTuple(Relation relation,
extern void PrintCatCacheLeakWarning(HeapTuple tuple);
extern void PrintCatCacheListLeakWarning(CatCList *list);
+/* defined in syscache.h */
+typedef struct syscachestats SysCacheStats;
+extern void CatCacheGetStats(CatCache *cache, SysCacheStats *syscachestats);
+
#endif /* CATCACHE_H */
diff --git a/src/include/utils/syscache.h b/src/include/utils/syscache.h
index 95ee48954e..71b399c902 100644
--- a/src/include/utils/syscache.h
+++ b/src/include/utils/syscache.h
@@ -112,6 +112,24 @@ enum SysCacheIdentifier
#define SysCacheSize (USERMAPPINGUSERSERVER + 1)
};
+#define SYSCACHE_STATS_NAGECLASSES 6
+/* Struct for catcache tracking information */
+typedef struct syscachestats
+{
+ Oid reloid; /* target relation */
+ Oid indoid; /* index */
+ size_t size; /* size of the catcache */
+ int ntuples; /* number of tuples resides in the catcache */
+ int nsearches; /* number of searches */
+ int nhits; /* number of cache hits */
+ int nneg_hits; /* number of negative cache hits */
+ /* age classes in seconds */
+ int ageclasses[SYSCACHE_STATS_NAGECLASSES];
+ /* number of tuples fall into the corresponding age class */
+ int nclass_entries[SYSCACHE_STATS_NAGECLASSES];
+} SysCacheStats;
+
+
extern void InitCatalogCache(void);
extern void InitCatalogCachePhase2(void);
@@ -164,6 +182,7 @@ extern void SysCacheInvalidate(int cacheId, uint32 hashValue);
extern bool RelationInvalidatesSnapshotsOnly(Oid relid);
extern bool RelationHasSysCache(Oid relid);
extern bool RelationSupportsSysCache(Oid relid);
+extern SysCacheStats *SysCacheGetStats(int cacheId);
/*
* The use of the macros below rather than direct calls to the corresponding
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 9244a2a7b7..0ab441a364 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -31,6 +31,7 @@ typedef enum TimeoutId
STANDBY_TIMEOUT,
STANDBY_LOCK_TIMEOUT,
IDLE_IN_TRANSACTION_SESSION_TIMEOUT,
+ IDLE_CATCACHE_UPDATE_TIMEOUT,
/* First user-definable timeout reason */
USER_TIMEOUT,
/* Maximum number of timeout reasons */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 2c8e21baa7..7bd77e9972 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1921,6 +1921,28 @@ pg_stat_sys_tables| SELECT pg_stat_all_tables.relid,
pg_stat_all_tables.autoanalyze_count
FROM pg_stat_all_tables
WHERE ((pg_stat_all_tables.schemaname = ANY (ARRAY['pg_catalog'::name, 'information_schema'::name])) OR
(pg_stat_all_tables.schemaname~ '^pg_toast'::text));
+pg_stat_syscache| SELECT s.pid,
+ (s.relid)::regclass AS relname,
+ (s.indid)::regclass AS cache_name,
+ s.size,
+ s.ntup AS ntuples,
+ s.searches,
+ s.hits,
+ s.neg_hits,
+ s.ageclass,
+ s.last_update
+ FROM (pg_stat_activity a
+ JOIN LATERAL ( SELECT a.pid,
+ pg_get_syscache_stats.relid,
+ pg_get_syscache_stats.indid,
+ pg_get_syscache_stats.size,
+ pg_get_syscache_stats.ntup,
+ pg_get_syscache_stats.searches,
+ pg_get_syscache_stats.hits,
+ pg_get_syscache_stats.neg_hits,
+ pg_get_syscache_stats.ageclass,
+ pg_get_syscache_stats.last_update
+ FROM pg_get_syscache_stats(a.pid) pg_get_syscache_stats(relid, indid, size, ntup, searches, hits, neg_hits,
ageclass,last_update)) s ON ((a.pid = s.pid)));
pg_stat_user_functions| SELECT p.oid AS funcid,
n.nspname AS schemaname,
p.proname AS funcname,
@@ -2352,7 +2374,7 @@ pg_settings|pg_settings_n|CREATE RULE pg_settings_n AS
ON UPDATE TO pg_catalog.pg_settings DO INSTEAD NOTHING;
pg_settings|pg_settings_u|CREATE RULE pg_settings_u AS
ON UPDATE TO pg_catalog.pg_settings
- WHERE (new.name = old.name) DO SELECT set_config(old.name, new.setting, false) AS set_config;
+ WHERE (new.name = old.name) DO SELECT set_config(old.name, new.setting, false, false) AS set_config;
rtest_emp|rtest_emp_del|CREATE RULE rtest_emp_del AS
ON DELETE TO public.rtest_emp DO INSERT INTO rtest_emplog (ename, who, action, newsal, oldsal)
VALUES (old.ename, CURRENT_USER, 'fired'::bpchar, '$0.00'::money, old.salary);
--
2.16.3
pgsql-hackers by date: