Re: PostgreSQL 14.4 ERROR: out of memory issues - Mailing list pgsql-general

From Aleš Zelený
Subject Re: PostgreSQL 14.4 ERROR: out of memory issues
Date
Msg-id CAODqTUZ+pxQVKwOa-H4vSP+dbpLpQv-XyGp2Kbgp1H8fQ69oCQ@mail.gmail.com
Whole thread Raw
In response to Re: PostgreSQL 14.4 ERROR: out of memory issues  (Aleš Zelený <zeleny.ales@gmail.com>)
Responses Re: PostgreSQL 14.4 ERROR: out of memory issues  (Tom Lane <tgl@sss.pgh.pa.us>)
List pgsql-general
Hello,




čt 14. 7. 2022 v 23:31 odesílatel Aleš Zelený <zeleny.ales@gmail.com> napsal:


Dne čt 14. 7. 2022 23:11 uživatel Tomas Vondra <tomas.vondra@enterprisedb.com> napsal:


On 7/14/22 21:25, Aleš Zelený wrote:
>
> st 13. 7. 2022 v 2:20 odesílatel Michael Paquier <michael@paquier.xyz
> <mailto:michael@paquier.xyz>> napsal:
>
>     On Mon, Jul 11, 2022 at 10:50:23AM +0200, Aleš Zelený wrote:
>     > So far, it has happened three times (during a single week) from
>     the 14.3 ->
>     > 14.4 upgrade, before 14.4 we haven't suffered from such an issue.
>     >
>     > Questions:
>     > 1)  Can we safely downgrade from 14.4 to 14.3 by shutting down the
>     instance
>     > and reinstalling 14.3 PG packages (to prove, that the issue
>     disappear)?
>     > 2) What is the best way to diagnose what is the root cause?
>
>     Hmm.  14.4 has nothing in its release notes that would point to a
>     change in the vacuum or autovacuum's code paths:
>     https://www.postgresql.org/docs/14/release-14-4.html#id-1.11.6.5.4
>     <https://www.postgresql.org/docs/14/release-14-4.html#id-1.11.6.5.4>
>
>     There is nothing specific after a look at the changes as of, and I am
>     not grabbing anything that would imply a change in memory context
>     handling either:
>     `git log --stat REL_14_3..REL_14_4`
>     `git diff REL_14_3..REL_14_4 -- *.c`
>
>     Saying that, you should be able to downgrade safely as there are no
>     changes in WAL format or such that would break things.  Saying that,
>     the corruption issue caused by CONCURRENTLY is something you'd still
>     have to face.
>
>
> Thanks, good to know that, we can use it for a test case, since we
> already hit the CONCURRENTLY bug on 14.3.
>
>     > 2022-07-02 14:48:07 CEST [3930]: [3-1] user=,db=,host=,app=
>     ERROR:  out of
>     > memory
>     > 2022-07-02 14:48:07 CEST [3930]: [4-1] user=,db=,host=,app=
>     DETAIL:  Failed
>     > on request of size 152094068 in memory context
>     "TopTransactionContext".
>     > 2022-07-02 14:48:07 CEST [3930]: [5-1] user=,db=,host=,app= CONTEXT:
>     >  automatic vacuum of table "prematch.replication.tab_queue_tmp"
>
>     This is the interesting part.  Do you happen to use logical
>     replication in a custom C++ plugin?
>
>
> We are using logical replication to other instances (pg_output) and
> decoderbufs
> https://github.com/debezium/postgres-decoderbufs
> <https://github.com/debezium/postgres-decoderbufs> for other applications.
>  

This is probably just a red herring - std:bad_alloc is what the process
that runs into the overcommit limit gets. But the real issue (e.g.
memory leak) is likely somewhere else - different part of the code,
different process ...

> ...
>
> Checking the RssAnon from proc/pid/status I've found some points where
> RssAnon memory usage grew very steep for a minute, but no "suspicious"
> queries/arguments were found in the instance logfile.
>  
> Any hint, on how to get the root cause would be appreciated since so far
> I've failed to isolate the issue reproducible testcase.
> At least I hope that looking for the RssAnon process memory is an
> appropriate metric, if not, let me know and I'll try to update the
> monitoring to get the root cause.
>
> I can imagine a workaround with client application regular reconnect...,
> but u to 14.3 it works, so I'd like to fix the issue either on our
> application side or at PG side if it is a PG problem.
>

I think it's be interesting to get memory context stats from the
processes consuming a lot of memory. If you know which processes are
suspect (and it seems you know, bacause if a reconnect helps it's the
backend handling the connection), you can attach a debugger and do

   $ gdb -p $PID
   call MemoryContextStats(TopMemoryContext)

which will log info about memory contexts, just like autovacuum.
Hopefully that tells us memory context is bloated, and that might point
us to particular part of the code.

after some time, I've found a process consuming over 1GB of memory"
-bash-4.2$ grep RssAnon /proc/*/status | sort -nk2 | tail
/proc/12156/status:RssAnon:  902984 kB
/proc/8303/status:RssAnon:  912848 kB
/proc/31187/status:RssAnon:  920424 kB
/proc/16469/status:RssAnon:  922864 kB
/proc/13769/status:RssAnon:  925824 kB
/proc/15449/status:RssAnon:  932452 kB
/proc/26144/status:RssAnon:  936892 kB
/proc/22570/status:RssAnon:  937920 kB
/proc/2910/status:RssAnon: 1039124 kB
/proc/17048/status:RssAnon: 1053952 kB

Here are memory contexts for PID 17048:

TopMemoryContext: 422592 total in 14 blocks; 42536 free (169 chunks); 380056 used
  pgstat TabStatusArray lookup hash table: 16384 total in 2 blocks; 7520 free (3 chunks); 8864 used
  PL/pgSQL function: 16384 total in 2 blocks; 5832 free (1 chunks); 10552 used: feeds_api.update_match(bigint,date,time without time zone)
  PL/pgSQL function: 16384 total in 2 blocks; 2080 free (1 chunks); 14304 used: opportunities.tr_taboddstopchangeshistory_oddstopchangestime()
  PL/pgSQL function: 16384 total in 2 blocks; 8576 free (1 chunks); 7808 used: opportunities.tr_tabodds_risklimits()
  PL/pgSQL function: 16384 total in 2 blocks; 2072 free (1 chunks); 14312 used: opportunities.tr_taboddshistory_oddshistorytime()
  PL/pgSQL function: 32768 total in 3 blocks; 6488 free (1 chunks); 26280 used: opportunities.fix_last_minute_odds(bigint,bigint)
  PL/pgSQL function: 65536 total in 4 blocks; 26624 free (1 chunks); 38912 used: opportunities.create_odds_tip_longname_lang(bigint,integer,integer,integer)
  PL/pgSQL function: 16384 total in 2 blocks; 3472 free (1 chunks); 12912 used: feeds_api.create_odds(bigint,integer,numeric,numeric,numeric,numeric,numeric,integer,jsonb)
  PL/pgSQL function: 65536 total in 4 blocks; 29648 free (2 chunks); 35888 used: feeds_api.create_opportunity(integer,bigint,bigint,integer,integer,integer,integer,integer,integer,i...
  PL/pgSQL function: 16384 total in 2 blocks; 7576 free (1 chunks); 8808 used: feeds.tr_tabopptysparentdata_ismain()
  PL/pgSQL function: 32768 total in 3 blocks; 11720 free (1 chunks); 21048 used: feeds_api.set_game_data(integer,bigint,bigint,character varying)
  PL/pgSQL function: 32768 total in 3 blocks; 7712 free (1 chunks); 25056 used: opportunities_api.set_oppty_info(bigint,boolean)
  PL/pgSQL function: 8192 total in 1 blocks; 3488 free (1 chunks); 4704 used: feeds_api.set_oppty_info_number(bigint)
  PL/pgSQL function: 16384 total in 2 blocks; 6616 free (1 chunks); 9768 used: b5_offer.tr_tabexternalopportunities_checknotnull()
  PL/pgSQL function: 32768 total in 3 blocks; 17928 free (5 chunks); 14840 used: opportunities.generate_league2oppty_correlations(bigint)
  PL/pgSQL function: 131072 total in 5 blocks; 61648 free (6 chunks); 69424 used: opportunities.generate_league2league_correlations(bigint)
  PL/pgSQL function: 16384 total in 2 blocks; 5616 free (1 chunks); 10768 used: opportunities.tr_tabopptytree_addcorrelations()
  PL/pgSQL function: 65536 total in 4 blocks; 19864 free (1 chunks); 45672 used: opportunities.replication_151_tab_oppty_tree()
  PL/pgSQL function: 16384 total in 2 blocks; 7568 free (1 chunks); 8816 used: greep.tr_shared_changedat()
  PL/pgSQL function: 32768 total in 3 blocks; 2112 free (1 chunks); 30656 used: feeds_api.create_match(integer,integer,integer,jsonb,bigint,integer,integer,integer,integer,jsonb,in...
  HandleParallelMessages: 8192 total in 1 blocks; 7928 free (0 chunks); 264 used
  pgqs_localhash: 21448 total in 2 blocks; 2616 free (0 chunks); 18832 used
  PL/pgSQL function: 16384 total in 2 blocks; 2072 free (1 chunks); 14312 used: opportunities.tr_tabopptyhistory_opptyhistorytime()
  PL/pgSQL function: 16384 total in 2 blocks; 2360 free (1 chunks); 14024 used: opportunities.tr_tabopportunitiesevaluationqueue_validtime()
  PL/pgSQL function: 16384 total in 2 blocks; 2512 free (1 chunks); 13872 used: opportunities.tr_tabopportunitiesevaluationqueue_evaluatedtime()
  PL/pgSQL function: 16384 total in 2 blocks; 2360 free (1 chunks); 14024 used: opportunities.tr_tabopportunitiesevaluationqueue_addedtime()
  PL/pgSQL function: 16384 total in 2 blocks; 7512 free (1 chunks); 8872 used: feeds_api.evaluate_opportunities(bigint[],integer,integer,integer,integer,integer)
  Btree proof lookup cache: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
  PL/pgSQL function: 8192 total in 1 blocks; 1384 free (1 chunks); 6808 used: feeds_api.change_odds_is_winning(bigint[],integer)
  PL/pgSQL function: 32768 total in 3 blocks; 3160 free (1 chunks); 29608 used: opportunities.tr_tabopportunity_resultchange()
  PL/pgSQL function: 16384 total in 2 blocks; 4424 free (1 chunks); 11960 used: opportunities.tr_tabopportunity_opptynamechange()
  PL/pgSQL function: 16384 total in 2 blocks; 7448 free (1 chunks); 8936 used: opportunities.tr_tabopportunity_marketingstatechange()
  PL/pgSQL function: 65536 total in 4 blocks; 27224 free (2 chunks); 38312 used: opportunities.tr_tabopportunity_history()
  PL/pgSQL function: 32768 total in 3 blocks; 16224 free (3 chunks); 16544 used: opportunities.tr_tabopportunity_datetimechange()
  PL/pgSQL function: 32768 total in 3 blocks; 15440 free (1 chunks); 17328 used: opportunities.tr_tabopportunity_addcorrelation()
  PL/pgSQL function: 32768 total in 3 blocks; 15672 free (1 chunks); 17096 used: opportunities.tr_tabopportunities_notifycanchanges()
  Sequence values: 8192 total in 1 blocks; 1576 free (0 chunks); 6616 used
  PL/pgSQL function: 262144 total in 6 blocks; 80744 free (0 chunks); 181400 used: opportunities.replication_151_tab_opportunities()
  PL/pgSQL function: 16384 total in 2 blocks; 8904 free (5 chunks); 7480 used: betsys.lastuser_change()
  PL/pgSQL function: 16384 total in 2 blocks; 2448 free (1 chunks); 13936 used: opportunities.tr_tabopportunities_opptyevaltime()
  PL/pgSQL function: 65536 total in 4 blocks; 29064 free (1 chunks); 36472 used: opportunities.tr_tabopportunities_checkstatechange()
  PL/pgSQL function: 16384 total in 2 blocks; 7568 free (1 chunks); 8816 used: greep.tr_shared_changedat()
  PL/pgSQL function: 32768 total in 3 blocks; 15512 free (1 chunks); 17256 used: opportunities.tr_tabopportunities_opptyenddatetime()
  PL/pgSQL function: 32768 total in 3 blocks; 16008 free (4 chunks); 16760 used: opportunities.tr_tabopportunity_checkstatechange_au()
  PL/pgSQL function: 8192 total in 1 blocks; 1752 free (1 chunks); 6440 used: feeds_api.change_opportunities_state(bigint[],integer)
  PL/pgSQL function: 65536 total in 4 blocks; 27640 free (1 chunks); 37896 used: opportunities.tr_tabodds_aggregs()
  PL/pgSQL function: 16384 total in 2 blocks; 7128 free (1 chunks); 9256 used: opportunities.tr_tabodds_notifyoddsblocked()
  PL/pgSQL function: 65536 total in 4 blocks; 31200 free (1 chunks); 34336 used: opportunities.tr_tabodds_history()
  Record information cache: 8192 total in 1 blocks; 1576 free (0 chunks); 6616 used
  PL/pgSQL function: 131072 total in 5 blocks; 65672 free (2 chunks); 65400 used: opportunities.replication_151_tab_odds()
  RI compare cache: 16384 total in 2 blocks; 6656 free (3 chunks); 9728 used
  RI query cache: 8192 total in 1 blocks; 1576 free (0 chunks); 6616 used
  RI constraint cache: 40648 total in 2 blocks; 2616 free (0 chunks); 38032 used
  PL/pgSQL function: 16384 total in 2 blocks; 7920 free (1 chunks); 8464 used: opportunities.tr_tabodds_fixoddsvalue()
  PLpgSQL cast info: 8192 total in 1 blocks; 7928 free (0 chunks); 264 used
    PLpgSQL cast cache: 8192 total in 1 blocks; 1576 free (0 chunks); 6616 used
  PL/pgSQL function: 16384 total in 2 blocks; 7568 free (5 chunks); 8816 used: greep.tr_shared_changedat()
  CFuncHash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
  TableSpace cache: 8192 total in 1 blocks; 2088 free (0 chunks); 6104 used
  Operator lookup cache: 40960 total in 3 blocks; 18888 free (10 chunks); 22072 used
  Type information cache: 24376 total in 2 blocks; 2616 free (0 chunks); 21760 used
  Function stat entries: 16384 total in 2 blocks; 4576 free (2 chunks); 11808 used
  Prepared Queries: 16384 total in 2 blocks; 6656 free (3 chunks); 9728 used
  RowDescriptionContext: 8192 total in 1 blocks; 6888 free (0 chunks); 1304 used
  MessageContext: 8192 total in 1 blocks; 6888 free (1 chunks); 1304 used
  Operator class cache: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
  smgr relation table: 65536 total in 4 blocks; 20768 free (14 chunks); 44768 used
  TransactionAbortContext: 32768 total in 1 blocks; 32504 free (0 chunks); 264 used
  Portal hash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
  TopPortalContext: 8192 total in 1 blocks; 7928 free (2 chunks); 264 used
  Relcache by OID: 32768 total in 3 blocks; 13608 free (6 chunks); 19160 used
  CacheMemoryContext: 2919800 total in 48 blocks; 218712 free (4 chunks); 2701088 used
    CachedPlanSource: 4096 total in 3 blocks; 1496 free (0 chunks); 2600 used: SELECT * FROM feeds_api.change_odds_is_winning(_id_odds := $1, _odds_is_winning := $2)
      CachedPlanQuery: 4096 total in 3 blocks; 1568 free (3 chunks); 2528 used
    CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used: INSERT INTO replication.tab_queue_tmp (id_queue_data)       VALUES (currval('replication.tab_queue_d...
    CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used: INSERT INTO replication.tab_queue_tmp (id_queue_data)       VALUES (currval('replication.tab_queue_d...
    index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: tab_queue_tmp_pkey
    CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used: INSERT INTO replication.tab_queue_tmp (id_queue_data)                         VALUES (currval('repli...
    CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used: INSERT INTO replication.tab_queue_tmp (id_queue_data)     VALUES (currval('replication.tab_queue_dat...
    CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used: INSERT INTO replication.tab_queue_tmp (id_queue_data)                         VALUES (currval('repli...
    CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used: INSERT INTO replication.tab_queue_tmp (id_queue_data)     VALUES (currval('replication.tab_queue_dat...
    CachedPlan: 4096 total in 3 blocks; 136 free (0 chunks); 3960 used: INSERT INTO replication.tab_queue_tmp (id_queue_data)     VALUES (currval('replication.tab_queue_dat...
    index info: 4096 total in 3 blocks; 2264 free (2 chunks); 1832 used: ix_tabopportunitiesevaluationqueue_idopportunity_pk_active_2
    index info: 4096 total in 3 blocks; 2024 free (2 chunks); 2072 used: ix_tabopportunitiesevaluationqueue_idopportunity_pk_active_1
    index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunitiesevaluationqueue_idodds
    index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: pk_tabopportunitiesevaluationqueue_idopportunitiesevaluationque
    CachedPlan: 16384 total in 5 blocks; 3656 free (0 chunks); 12728 used: INSERT INTO opportunities.tab_opportunities_evaluation_queue (id_odds, evaluation_type, id_opportuni...
    CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used: INSERT INTO replication.tab_queue_data (queue_data, id_action_type, id_table, id_replication_group) ...
    CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used: INSERT INTO replication.tab_queue_data (queue_data, id_action_type, id_table, id_replication_group) ...
    CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used: INSERT INTO replication.tab_queue_data (queue_data, id_action_type, id_table, id_replication_group) ...
    CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used: INSERT INTO replication.tab_queue_data (queue_data, id_action_type, id_table, id_replication_group) ...
    CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used: INSERT INTO replication.tab_queue_data (queue_data, id_action_type, id_table, id_replication_group) ...
    CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used: INSERT INTO replication.tab_queue_data (queue_data, id_action_type, id_table, id_replication_group) ...
    index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: i_replication_tabqueuedata_idreplicationgroup
    index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: tab_queue_data_pkey
    CachedPlan: 8192 total in 4 blocks; 4000 free (0 chunks); 4192 used: INSERT INTO replication.tab_queue_data (queue_data, id_action_type, id_table, id_replication_group) ...
    CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog....
    CachedPlan: 134336 total in 10 blocks; 42112 free (0 chunks); 92224 used: WITH opp AS MATERIALIZED         (             SELECT DISTINCT o2.id_opportunity             FROM  o...
    CachedPlan: 65536 total in 7 blocks; 31968 free (0 chunks); 33568 used: SELECT lc.contestant_name_lang                                             FROM opportunities.tab_op...
    CachedPlan: 65536 total in 7 blocks; 31968 free (0 chunks); 33568 used: SELECT lc.contestant_name_lang                                             FROM opportunities.tab_op...
    CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog....
    CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog....
    CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog....
    CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog....
    CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog....
    CachedPlan: 8192 total in 4 blocks; 3672 free (0 chunks); 4520 used: EXISTS         (             SELECT               FROM opportunities.tab_oppty_tree              WHE...
    CachedPlan: 65536 total in 7 blocks; 32224 free (0 chunks); 33312 used: SELECT CASE        WHEN SUBSTRING(op.marketing_states FROM 1 FOR 1) = '1' AND op.oppty_end_date + op...
    CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog....
      CachedPlanQuery: 8192 total in 4 blocks; 1504 free (0 chunks); 6688 used
    SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used
    CachedPlan: 4096 total in 3 blocks; 392 free (0 chunks); 3704 used: INSERT INTO opportunities.tab_oppty_tree     (         id_opportunity,         id_opportunity_parent...
    CachedPlan: 35840 total in 6 blocks; 15344 free (1 chunks); 20496 used: INSERT INTO opportunities.tab_opportunities     (         id_event,         id_oppty_type,         i...
    CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog....
      CachedPlanQuery: 8192 total in 4 blocks; 1504 free (0 chunks); 6688 used
    SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used
    CachedPlan: 8192 total in 4 blocks; 3664 free (1 chunks); 4528 used: NOT EXISTS         (             SELECT             FROM opportunities.tab_oppty_tree             WH...
    CachedPlan: 16384 total in 5 blocks; 7960 free (2 chunks); 8424 used: UPDATE opportunities.tab_opportunities        SET oppty_info_number = _info_number,            id_op...
    CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog....
      CachedPlanQuery: 8192 total in 4 blocks; 1504 free (3 chunks); 6688 used
    SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used
    CachedPlan: 8192 total in 4 blocks; 616 free (2 chunks); 7576 used: SELECT         oppty_info_number,         id_event,         id_oppty_type                           ...
    CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog....
      CachedPlanQuery: 8192 total in 4 blocks; 1504 free (0 chunks); 6688 used
    SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used
    CachedPlan: 134408 total in 10 blocks; 51528 free (0 chunks); 82880 used: INSERT INTO opportunities.tab_correlations (id_opportunity, id_opportunity_2)     SELECT         crr...
    CachedPlan: 277504 total in 9 blocks; 15680 free (0 chunks); 261824 used: INSERT INTO opportunities.tab_correlations (id_opportunity, id_opportunity_2)     -- id_league from ...
    CachedPlan: 134144 total in 8 blocks; 30024 free (0 chunks); 104120 used: SELECT         nm1.contestant_name,         COALESCE(nm2.contestant_name, nm1.contestant_name),     ...
    CachedPlan: 32768 total in 6 blocks; 2064 free (0 chunks); 30704 used: SELECT evn.id_league     FROM opportunities.tab_opportunities    AS opp     JOIN opportunities.tab_e...
    CachedPlan: 16384 total in 5 blocks; 6880 free (0 chunks); 9504 used: NEW.id_opportunity <> NEW.id_opportunity_parent OR         EXISTS         (             SELECT      ...
    CachedPlan: 16384 total in 5 blocks; 4952 free (1 chunks); 11432 used: (TG_OP='INSERT' AND CASE WHEN get_session_var('id_data_provider') = '12' THEN FALSE ELSE (SELECT CAS...
    CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog....
      CachedPlanQuery: 8192 total in 4 blocks; 1504 free (0 chunks); 6688 used
    SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used
    CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog....
      CachedPlanQuery: 8192 total in 4 blocks; 1504 free (0 chunks); 6688 used
    SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used
    CachedPlan: 4096 total in 3 blocks; 392 free (0 chunks); 3704 used: INSERT INTO opportunities.tab_oppty_tree     (         id_opportunity,         id_opportunity_parent...
    CachedPlan: 33792 total in 6 blocks; 14192 free (0 chunks); 19600 used: INSERT INTO opportunities.tab_opportunities     (         id_event,         id_oppty_type,         o...
    CachedPlan: 16384 total in 5 blocks; 8000 free (2 chunks); 8384 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog....
    CachedPlanSource: 4096 total in 3 blocks; 1384 free (1 chunks); 2712 used: SELECT 1 FROM ONLY "opportunities"."tab_opportunities" x WHERE "id_opportunity" OPERATOR(pg_catalog....
      CachedPlanQuery: 8192 total in 4 blocks; 1504 free (0 chunks); 6688 used
    SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used
    CachedPlan: 32768 total in 6 blocks; 9352 free (0 chunks); 23416 used: EXISTS (         SELECT         FROM  new_items AS ni         JOIN  old_items AS oi ON oi.id_opportu...
    CachedPlan: 8192 total in 4 blocks; 3120 free (1 chunks); 5072 used: OLD.oppty_name IS DISTINCT FROM NEW.oppty_name     AND EXISTS     (         SELECT         FROM oppo...
    CachedPlan: 9216 total in 4 blocks; 1408 free (1 chunks); 7808 used: NEW.oppty_end_date + NEW.oppty_end_time > NOW()         AND (COALESCE(OLD.oppty_end_date + OLD.oppty...
    index info: 2048 total in 2 blocks; 392 free (0 chunks); 1656 used: ix_tabopptytree_idopportunityparent_temporary
    index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: pk_tabopptytree_idopportunity
    index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopptytree_idopportunityparent
    index info: 3072 total in 2 blocks; 808 free (1 chunks); 2264 used: pg_statistic_relid_att_inh_index
    index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used: ix_tabopportunities_opptybetradarid_iddataprovider
    index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_opptyevaltimetz
    index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_opptyendtimestamptz
    index info: 2048 total in 2 blocks; 184 free (0 chunks); 1864 used: ix_tabopportunities_opptyenddatetime
    index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_idconstestant4
    index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_idconstestant3
    index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_opptyexternalcompleteid
    index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_opptybetradarid
    index info: 4096 total in 3 blocks; 2024 free (2 chunks); 2072 used: ix_tabopportunities_opptyenddatetime_idopptystate
    index info: 2048 total in 2 blocks; 392 free (0 chunks); 1656 used: ix_tabopportunities_opptyenddatetime_obsolete
    index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_opptyenddate
    index info: 2048 total in 2 blocks; 264 free (0 chunks); 1784 used: ix_tabopportunities_lastminute
    index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: ix_tabopportunities_info
    index info: 3072 total in 2 blocks; 808 free (1 chunks); 2264 used: ix_tabopportunities_idopptytype_vsportsid_idevent
    index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_idopportunitytype
    index info: 3072 total in 2 blocks; 808 free (1 chunks); 2264 used: ix_tabopportunities_idopportunitystate_opptyenddate_idevent
    index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used: ix_tabopportunities_idopportunitystate_opptyenddate
    index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_idopportunitystate
    index info: 3072 total in 2 blocks; 808 free (1 chunks); 2264 used: ix_tabopportunities_idopportunity_opptyenddate_opptyinfonumber
    index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used: ix_tabopportunities_idevent_opptyenddate
    index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_idconstestant2
    index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_tabopportunities_idconstestant1
    index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used: ix_tabopportunities_enddateinfonumber
    index info: 2048 total in 2 blocks; 360 free (0 chunks); 1688 used: idx_tabopportunities_opptyspecialvalues_season_id
    index info: 2048 total in 2 blocks; 360 free (0 chunks); 1688 used: idx_tabopportunities_opptyspecialvalues_match_id
    index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: pk_tabopportunities_idopportunity
    CachedPlan: 16384 total in 5 blocks; 6712 free (0 chunks); 9672 used: UPDATE opportunities.lst_oppty_numbers     SET id_opportunity = _id_opportunity     WHERE id_oppty_n...
    CachedPlan: 8192 total in 4 blocks; 3704 free (1 chunks); 4488 used: UPDATE opportunities.lst_oppty_numbers                    SET id_opportunity  = NULL                ...
    index info: 2048 total in 2 blocks; 528 free (1 chunks); 1520 used: pg_toast_2619_index
    index info: 2048 total in 2 blocks; 584 free (0 chunks); 1464 used: ix_lstopptynumbers_freeopptynumbers
    index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: ix_lstopptynumbers_idopportunity
    1455 more child contexts containing 6947040 total in 4141 blocks; 2312976 free (668 chunks); 4634064 used
  WAL record construction: 49768 total in 2 blocks; 6360 free (0 chunks); 43408 used
  PrivateRefCount: 8192 total in 1 blocks; 2616 free (0 chunks); 5576 used
  MdSmgr: 16384 total in 2 blocks; 5624 free (20 chunks); 10760 used
  LOCALLOCK hash: 32768 total in 3 blocks; 12712 free (8 chunks); 20056 used
  plpgsql_check - profiler context: 8192 total in 1 blocks; 7928 free (0 chunks); 264 used
    plpgsql_check function execution statistics: 8192 total in 1 blocks; 2488 free (0 chunks); 5704 used
    plpgsql_check function profiler local chunks: 8192 total in 1 blocks; 2488 free (0 chunks); 5704 used
    plpgsql_check function profiler local cache: 8192 total in 1 blocks; 2488 free (0 chunks); 5704 used
  plpgsql_check function cache: 8192 total in 1 blocks; 2616 free (0 chunks); 5576 used
  Rendezvous variable hash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
  PLpgSQL function hash: 40864 total in 3 blocks; 2616 free (0 chunks); 38248 used
  Timezones: 104120 total in 2 blocks; 2616 free (0 chunks); 101504 used
  ErrorContext: 8192 total in 1 blocks; 7928 free (8 chunks); 264 used
Grand total: 14312808 bytes in 4752 blocks; 3920880 free (1043 chunks); 10391928 used


I hope, this will help.

Some additional information:

-bash-4.2$ cat /proc/17048/status
Name: postmaster
Umask: 0077
State: S (sleeping)
Tgid: 17048
Ngid: 0
Pid: 17048
PPid: 4476
TracerPid: 0
Uid: 26 26 26 26
Gid: 26 26 26 26
FDSize: 1024
Groups: 26
VmPeak: 12212748 kB
VmSize: 12212284 kB
VmLck:       0 kB
VmPin:       0 kB
VmHWM: 9421940 kB
VmRSS: 9421940 kB
RssAnon: 1054764 kB
RssFile:   15796 kB
RssShmem: 8351380 kB
VmData: 1055404 kB
VmStk:     132 kB
VmExe:    7816 kB
VmLib:   85312 kB
VmPTE:   23264 kB
VmSwap:       0 kB
Threads: 1
SigQ: 0/160197
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000400000
SigIgn: 0000000001701800
SigCgt: 00000001a0006287
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000001fffffffff
CapAmb: 0000000000000000
NoNewPrivs: 0
Seccomp: 0
Speculation_Store_Bypass: thread vulnerable
Cpus_allowed: ffffff
Cpus_allowed_list: 0-23
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
Mems_allowed_list: 0-1
voluntary_ctxt_switches: 4748002
nonvoluntary_ctxt_switches: 12237

-bash-4.2$ pmap 17048
17048:   postgres: db-prematch: app_feed prematch 172.25.0.231(47890) idle
0000000000400000   7816K r-x-- postgres
0000000000da1000      4K r---- postgres
0000000000da2000     68K rw--- postgres
0000000000db3000    200K rw---   [ anon ]
0000000002d15000    504K rw---   [ anon ]
0000000002d93000 838092K rw---   [ anon ]
00007fd999777000 180232K rw---   [ anon ]
00007fd9a8d75000  32772K rw---   [ anon ]
00007fd9ac5d2000   1024K rw-s- PostgreSQL.380355184
00007fd9ac6d2000    196K rw-s- PostgreSQL.38763112
00007fd9ac703000    148K r-x-- libtinfo.so.5.9
00007fd9ac728000   2048K ----- libtinfo.so.5.9
00007fd9ac928000     16K r---- libtinfo.so.5.9
00007fd9ac92c000      4K rw--- libtinfo.so.5.9
00007fd9ac92d000    212K r-x-- libedit.so.0.0.42
00007fd9ac962000   2044K ----- libedit.so.0.0.42
00007fd9acb61000     12K r---- libedit.so.0.0.42
00007fd9acb64000      4K rw--- libedit.so.0.0.42
00007fd9acb65000     20K rw---   [ anon ]
00007fd9acb6a000     28K r-x-- libffi.so.6.0.1
00007fd9acb71000   2044K ----- libffi.so.6.0.1
00007fd9acd70000      4K r---- libffi.so.6.0.1
00007fd9acd71000      4K rw--- libffi.so.6.0.1
00007fd9acd72000  45952K r-x-- libLLVM-5.0.so
00007fd9afa52000   2048K ----- libLLVM-5.0.so
00007fd9afc52000   3072K r---- libLLVM-5.0.so
00007fd9aff52000    444K rw--- libLLVM-5.0.so
00007fd9affc1000    372K rw---   [ anon ]
00007fd9b001e000    116K r-x-- llvmjit.so
00007fd9b003b000   2048K ----- llvmjit.so
00007fd9b023b000      4K r---- llvmjit.so
00007fd9b023c000      4K rw--- llvmjit.so
00007fd9b023d000   1028K rw---   [ anon ]
00007fd9b033e000     48K r-x-- libnss_files-2.17.so
00007fd9b034a000   2044K ----- libnss_files-2.17.so
00007fd9b0549000      4K r---- libnss_files-2.17.so
00007fd9b054a000      4K rw--- libnss_files-2.17.so
00007fd9b054b000     24K rw---   [ anon ]
00007fd9b0551000 10827040K rw-s- zero (deleted)
00007fdc45299000     32K r-x-- libprotobuf-c.so.1
00007fdc452a1000   2044K ----- libprotobuf-c.so.1
00007fdc454a0000      4K r---- libprotobuf-c.so.1
00007fdc454a1000      4K rw--- libprotobuf-c.so.1
00007fdc454a2000     24K r-x-- decoderbufs.so
00007fdc454a8000   2048K ----- decoderbufs.so
00007fdc456a8000      4K r---- decoderbufs.so
00007fdc456a9000      4K rw--- decoderbufs.so
00007fdc456aa000    184K r-x-- plpgsql.so
00007fdc456d8000   2044K ----- plpgsql.so
00007fdc458d7000      8K r---- plpgsql.so
00007fdc458d9000      4K rw--- plpgsql.so
00007fdc458da000    192K r-x-- plpgsql_check.so
00007fdc4590a000   2044K ----- plpgsql_check.so
00007fdc45b09000      4K r---- plpgsql_check.so
00007fdc45b0a000      4K rw--- plpgsql_check.so
00007fdc45b0b000     24K r-x-- pg_wait_sampling.so
00007fdc45b11000   2044K ----- pg_wait_sampling.so
00007fdc45d10000      4K r---- pg_wait_sampling.so
00007fdc45d11000      4K rw--- pg_wait_sampling.so
00007fdc45d12000     20K r-x-- pg_stat_kcache.so
00007fdc45d17000   2044K ----- pg_stat_kcache.so
00007fdc45f16000      4K r---- pg_stat_kcache.so
00007fdc45f17000      4K rw--- pg_stat_kcache.so
00007fdc45f18000     16K rw---   [ anon ]
00007fdc45f1c000     32K r-x-- pg_qualstats.so
00007fdc45f24000   2044K ----- pg_qualstats.so
00007fdc46123000      4K r---- pg_qualstats.so
00007fdc46124000      4K rw--- pg_qualstats.so
00007fdc46125000     32K r-x-- pg_stat_statements.so
00007fdc4612d000   2044K ----- pg_stat_statements.so
00007fdc4632c000      4K r---- pg_stat_statements.so
00007fdc4632d000      4K rw--- pg_stat_statements.so
00007fdc4632e000 103692K r---- locale-archive
00007fdc4c871000      8K r-x-- libfreebl3.so
00007fdc4c873000   2044K ----- libfreebl3.so
00007fdc4ca72000      4K r---- libfreebl3.so
00007fdc4ca73000      4K rw--- libfreebl3.so
00007fdc4ca74000     60K r-x-- libbz2.so.1.0.6
00007fdc4ca83000   2044K ----- libbz2.so.1.0.6
00007fdc4cc82000      4K r---- libbz2.so.1.0.6
00007fdc4cc83000      4K rw--- libbz2.so.1.0.6
00007fdc4cc84000     92K r-x-- libelf-0.176.so
00007fdc4cc9b000   2044K ----- libelf-0.176.so
00007fdc4ce9a000      4K r---- libelf-0.176.so
00007fdc4ce9b000      4K rw--- libelf-0.176.so
00007fdc4ce9c000    384K r-x-- libpcre.so.1.2.0
00007fdc4cefc000   2048K ----- libpcre.so.1.2.0
00007fdc4d0fc000      4K r---- libpcre.so.1.2.0
00007fdc4d0fd000      4K rw--- libpcre.so.1.2.0
00007fdc4d0fe000     16K r-x-- libattr.so.1.1.0
00007fdc4d102000   2044K ----- libattr.so.1.1.0
00007fdc4d301000      4K r---- libattr.so.1.1.0
00007fdc4d302000      4K rw--- libattr.so.1.1.0
00007fdc4d303000     32K r-x-- libcrypt-2.17.so
00007fdc4d30b000   2044K ----- libcrypt-2.17.so
00007fdc4d50a000      4K r---- libcrypt-2.17.so
00007fdc4d50b000      4K rw--- libcrypt-2.17.so
00007fdc4d50c000    184K rw---   [ anon ]
00007fdc4d53a000     16K r-x-- libcap-ng.so.0.0.0
00007fdc4d53e000   2048K ----- libcap-ng.so.0.0.0
00007fdc4d73e000      4K r---- libcap-ng.so.0.0.0
00007fdc4d73f000      4K rw--- libcap-ng.so.0.0.0
00007fdc4d740000    312K r-x-- libdw-0.176.so
00007fdc4d78e000   2048K ----- libdw-0.176.so
00007fdc4d98e000      8K r---- libdw-0.176.so
00007fdc4d990000      4K rw--- libdw-0.176.so
00007fdc4d991000     16K r-x-- libgpg-error.so.0.10.0
00007fdc4d995000   2044K ----- libgpg-error.so.0.10.0
00007fdc4db94000      4K r---- libgpg-error.so.0.10.0
00007fdc4db95000      4K rw--- libgpg-error.so.0.10.0
00007fdc4db96000    500K r-x-- libgcrypt.so.11.8.2
00007fdc4dc13000   2044K ----- libgcrypt.so.11.8.2
00007fdc4de12000      4K r---- libgcrypt.so.11.8.2
00007fdc4de13000     12K rw--- libgcrypt.so.11.8.2
00007fdc4de16000      4K rw---   [ anon ]
00007fdc4de17000    144K r-x-- libselinux.so.1
00007fdc4de3b000   2044K ----- libselinux.so.1
00007fdc4e03a000      4K r---- libselinux.so.1
00007fdc4e03b000      4K rw--- libselinux.so.1
00007fdc4e03c000      8K rw---   [ anon ]
00007fdc4e03e000     16K r-x-- libcap.so.2.22
00007fdc4e042000   2044K ----- libcap.so.2.22
00007fdc4e241000      4K r---- libcap.so.2.22
00007fdc4e242000      4K rw--- libcap.so.2.22
00007fdc4e243000  20296K r-x-- libicudata.so.50.2
00007fdc4f615000   2044K ----- libicudata.so.50.2
00007fdc4f814000      4K r---- libicudata.so.50.2
00007fdc4f815000      4K rw--- libicudata.so.50.2
00007fdc4f816000     84K r-x-- libgcc_s-4.8.5-20150702.so.1
00007fdc4f82b000   2044K ----- libgcc_s-4.8.5-20150702.so.1
00007fdc4fa2a000      4K r---- libgcc_s-4.8.5-20150702.so.1
00007fdc4fa2b000      4K rw--- libgcc_s-4.8.5-20150702.so.1
00007fdc4fa2c000    932K r-x-- libstdc++.so.6.0.19
00007fdc4fb15000   2044K ----- libstdc++.so.6.0.19
00007fdc4fd14000     32K r---- libstdc++.so.6.0.19
00007fdc4fd1c000      8K rw--- libstdc++.so.6.0.19
00007fdc4fd1e000     84K rw---   [ anon ]
00007fdc4fd33000    232K r-x-- libnspr4.so
00007fdc4fd6d000   2044K ----- libnspr4.so
00007fdc4ff6c000      4K r---- libnspr4.so
00007fdc4ff6d000      8K rw--- libnspr4.so
00007fdc4ff6f000      8K rw---   [ anon ]
00007fdc4ff71000     16K r-x-- libplc4.so
00007fdc4ff75000   2044K ----- libplc4.so
00007fdc50174000      4K r---- libplc4.so
00007fdc50175000      4K rw--- libplc4.so
00007fdc50176000     12K r-x-- libplds4.so
00007fdc50179000   2044K ----- libplds4.so
00007fdc50378000      4K r---- libplds4.so
00007fdc50379000      4K rw--- libplds4.so
00007fdc5037a000    164K r-x-- libnssutil3.so
00007fdc503a3000   2044K ----- libnssutil3.so
00007fdc505a2000     28K r---- libnssutil3.so
00007fdc505a9000      4K rw--- libnssutil3.so
00007fdc505aa000   1176K r-x-- libnss3.so
00007fdc506d0000   2048K ----- libnss3.so
00007fdc508d0000     20K r---- libnss3.so
00007fdc508d5000      8K rw--- libnss3.so
00007fdc508d7000      8K rw---   [ anon ]
00007fdc508d9000    148K r-x-- libsmime3.so
00007fdc508fe000   2044K ----- libsmime3.so
00007fdc50afd000     12K r---- libsmime3.so
00007fdc50b00000      4K rw--- libsmime3.so
00007fdc50b01000    332K r-x-- libssl3.so
00007fdc50b54000   2048K ----- libssl3.so
00007fdc50d54000     16K r---- libssl3.so
00007fdc50d58000      4K rw--- libssl3.so
00007fdc50d59000      4K rw---   [ anon ]
00007fdc50d5a000    112K r-x-- libsasl2.so.3.0.0
00007fdc50d76000   2044K ----- libsasl2.so.3.0.0
00007fdc50f75000      4K r---- libsasl2.so.3.0.0
00007fdc50f76000      4K rw--- libsasl2.so.3.0.0
00007fdc50f77000     56K r-x-- liblber-2.4.so.2.10.7
00007fdc50f85000   2044K ----- liblber-2.4.so.2.10.7
00007fdc51184000      4K r---- liblber-2.4.so.2.10.7
00007fdc51185000      4K rw--- liblber-2.4.so.2.10.7
00007fdc51186000     88K r-x-- libresolv-2.17.so
00007fdc5119c000   2048K ----- libresolv-2.17.so
00007fdc5139c000      4K r---- libresolv-2.17.so
00007fdc5139d000      4K rw--- libresolv-2.17.so
00007fdc5139e000      8K rw---   [ anon ]
00007fdc513a0000     12K r-x-- libkeyutils.so.1.5
00007fdc513a3000   2044K ----- libkeyutils.so.1.5
00007fdc515a2000      4K r---- libkeyutils.so.1.5
00007fdc515a3000      4K rw--- libkeyutils.so.1.5
00007fdc515a4000     56K r-x-- libkrb5support.so.0.1
00007fdc515b2000   2048K ----- libkrb5support.so.0.1
00007fdc517b2000      4K r---- libkrb5support.so.0.1
00007fdc517b3000      4K rw--- libkrb5support.so.0.1
00007fdc517b4000    196K r-x-- libk5crypto.so.3.1
00007fdc517e5000   2044K ----- libk5crypto.so.3.1
00007fdc519e4000      8K r---- libk5crypto.so.3.1
00007fdc519e6000      4K rw--- libk5crypto.so.3.1
00007fdc519e7000     12K r-x-- libcom_err.so.2.1
00007fdc519ea000   2044K ----- libcom_err.so.2.1
00007fdc51be9000      4K r---- libcom_err.so.2.1
00007fdc51bea000      4K rw--- libcom_err.so.2.1
00007fdc51beb000    868K r-x-- libkrb5.so.3.3
00007fdc51cc4000   2044K ----- libkrb5.so.3.3
00007fdc51ec3000     56K r---- libkrb5.so.3.3
00007fdc51ed1000     12K rw--- libkrb5.so.3.3
00007fdc51ed4000    120K r-x-- libaudit.so.1.0.0
00007fdc51ef2000   2044K ----- libaudit.so.1.0.0
00007fdc520f1000      4K r---- libaudit.so.1.0.0
00007fdc520f2000      4K rw--- libaudit.so.1.0.0
00007fdc520f3000     40K rw---   [ anon ]
00007fdc520fd000    148K r-x-- liblzma.so.5.2.2
00007fdc52122000   2044K ----- liblzma.so.5.2.2
00007fdc52321000      4K r---- liblzma.so.5.2.2
00007fdc52322000      4K rw--- liblzma.so.5.2.2
00007fdc52323000     84K r-x-- libz.so.1.2.7
00007fdc52338000   2044K ----- libz.so.1.2.7
00007fdc52537000      4K r---- libz.so.1.2.7
00007fdc52538000      4K rw--- libz.so.1.2.7
00007fdc52539000   1804K r-x-- libc-2.17.so
00007fdc526fc000   2048K ----- libc-2.17.so
00007fdc528fc000     16K r---- libc-2.17.so
00007fdc52900000      8K rw--- libc-2.17.so
00007fdc52902000     20K rw---   [ anon ]
00007fdc52907000    188K r-x-- libsystemd.so.0.6.0
00007fdc52936000   2048K ----- libsystemd.so.0.6.0
00007fdc52b36000      4K r---- libsystemd.so.0.6.0
00007fdc52b37000      4K rw--- libsystemd.so.0.6.0
00007fdc52b38000   1424K r-x-- libicuuc.so.50.2
00007fdc52c9c000   2048K ----- libicuuc.so.50.2
00007fdc52e9c000     64K r---- libicuuc.so.50.2
00007fdc52eac000      4K rw--- libicuuc.so.50.2
00007fdc52ead000     16K rw---   [ anon ]
00007fdc52eb1000   1984K r-x-- libicui18n.so.50.2
00007fdc530a1000   2048K ----- libicui18n.so.50.2
00007fdc532a1000     48K r---- libicui18n.so.50.2
00007fdc532ad000      8K rw--- libicui18n.so.50.2
00007fdc532af000      4K rw---   [ anon ]
00007fdc532b0000    328K r-x-- libldap-2.4.so.2.10.7
00007fdc53302000   2048K ----- libldap-2.4.so.2.10.7
00007fdc53502000      8K r---- libldap-2.4.so.2.10.7
00007fdc53504000      4K rw--- libldap-2.4.so.2.10.7
00007fdc53505000   1028K r-x-- libm-2.17.so
00007fdc53606000   2044K ----- libm-2.17.so
00007fdc53805000      4K r---- libm-2.17.so
00007fdc53806000      4K rw--- libm-2.17.so
00007fdc53807000      8K r-x-- libdl-2.17.so
00007fdc53809000   2048K ----- libdl-2.17.so
00007fdc53a09000      4K r---- libdl-2.17.so
00007fdc53a0a000      4K rw--- libdl-2.17.so
00007fdc53a0b000     28K r-x-- librt-2.17.so
00007fdc53a12000   2044K ----- librt-2.17.so
00007fdc53c11000      4K r---- librt-2.17.so
00007fdc53c12000      4K rw--- librt-2.17.so
00007fdc53c13000     92K r-x-- libpthread-2.17.so
00007fdc53c2a000   2044K ----- libpthread-2.17.so
00007fdc53e29000      4K r---- libpthread-2.17.so
00007fdc53e2a000      4K rw--- libpthread-2.17.so
00007fdc53e2b000     16K rw---   [ anon ]
00007fdc53e2f000    296K r-x-- libgssapi_krb5.so.2.2
00007fdc53e79000   2048K ----- libgssapi_krb5.so.2.2
00007fdc54079000      4K r---- libgssapi_krb5.so.2.2
00007fdc5407a000      8K rw--- libgssapi_krb5.so.2.2
00007fdc5407c000   2264K r-x-- libcrypto.so.1.0.2k
00007fdc542b2000   2048K ----- libcrypto.so.1.0.2k
00007fdc544b2000    112K r---- libcrypto.so.1.0.2k
00007fdc544ce000     52K rw--- libcrypto.so.1.0.2k
00007fdc544db000     16K rw---   [ anon ]
00007fdc544df000    412K r-x-- libssl.so.1.0.2k
00007fdc54546000   2048K ----- libssl.so.1.0.2k
00007fdc54746000     16K r---- libssl.so.1.0.2k
00007fdc5474a000     28K rw--- libssl.so.1.0.2k
00007fdc54751000     52K r-x-- libpam.so.0.83.1
00007fdc5475e000   2048K ----- libpam.so.0.83.1
00007fdc5495e000      4K r---- libpam.so.0.83.1
00007fdc5495f000      4K rw--- libpam.so.0.83.1
00007fdc54960000   1404K r-x-- libxml2.so.2.9.1
00007fdc54abf000   2044K ----- libxml2.so.2.9.1
00007fdc54cbe000     32K r---- libxml2.so.2.9.1
00007fdc54cc6000      8K rw--- libxml2.so.2.9.1
00007fdc54cc8000      8K rw---   [ anon ]
00007fdc54cca000     80K r-x-- liblz4.so.1.7.5
00007fdc54cde000   2044K ----- liblz4.so.1.7.5
00007fdc54edd000      4K r---- liblz4.so.1.7.5
00007fdc54ede000      4K rw--- liblz4.so.1.7.5
00007fdc54edf000    136K r-x-- ld-2.17.so
00007fdc54f1a000   1588K rw---   [ anon ]
00007fdc550a7000    212K rw-s- PostgreSQL.4009629762
00007fdc550dc000    104K rw---   [ anon ]
00007fdc550fc000      8K r-x--   [ anon ]
00007fdc550fe000      4K rw-s-   [ shmid=0x67418000 ]
00007fdc550ff000      4K rw---   [ anon ]
00007fdc55100000      4K r---- ld-2.17.so
00007fdc55101000      4K rw--- ld-2.17.so
00007fdc55102000      4K rw---   [ anon ]
00007fffd382b000    132K rw---   [ stack ]
00007fffd38f0000      8K r-x--   [ anon ]
ffffffffff600000      4K r-x--   [ anon ]
 total         12212288K

-bash-4.2$ smem -u
User     Count     Swap      USS      PSS      RSS
postgres   326        0 10503276 20969859 175615624
-bash-4.2$ smem
^C
  PID User     Command                         Swap      USS      PSS      RSS
27250 postgres postgres: db-prematch: arch        0      224      246     2592
27251 postgres postgres: db-prematch: stat        0      512      538     2716
27253 postgres postgres: db-prematch: logi        0      548      567     3480
27249 postgres postgres: db-prematch: auto        0      664      696     3956
27248 postgres postgres: db-prematch: walw        0      244      878    19296
...27246 postgres postgres: db-prematch: chec        0    88308   851490 10271612
12156 postgres postgres: db-prematch: app_        0   902776  1400234  9066116
 8303 postgres postgres: db-prematch: app_        0   913768  1411286  9067356
16469 postgres postgres: db-prematch: app_        0   923668  1417847  9071168
31187 postgres postgres: db-prematch: app_        0   923024  1425936  9196732
13769 postgres postgres: db-prematch: app_        0   925820  1437147  9283344
26144 postgres postgres: db-prematch: app_        0   937440  1437802  9180720
15449 postgres postgres: db-prematch: app_        0   932776  1440671  9262056
22570 postgres postgres: db-prematch: app_        0   938680  1447851  9292176
 2910 postgres postgres: db-prematch: app_        0  1039452  1536980  9183668
17048 postgres postgres: db-prematch: app_        0  1054244  1564436  9425556

-bash-4.2$ smem -w
Area                           Used      Cache   Noncache
firmware/hardware                 0          0          0
kernel image                      0          0          0
kernel dynamic memory      18242380   16741684    1500696
userspace memory           21358228   10746068   10612160
free memory                 1433148    1433148          0

-bash-4.2$ smem -m
Map                                       PIDs   AVGPSS      PSS
/dev/shm/PostgreSQL.4009629762             316        0        0
/usr/lib64/gconv/gconv-modules.cache        18        0        0
/usr/lib64/libattr.so.1.1.0                318        0        0
...
/usr/pgsql-14/lib/libpq.so.5.14             15       27      415
/usr/lib64/libkrb5.so.3.3                  319        1      445
/usr/pgsql-14/lib/pg_qualstats.so          318        1      572
/usr/pgsql-14/lib/pg_stat_kcache.so        318        1      572
/usr/lib64/libpython2.7.so.1.0               1      945      945
/usr/pgsql-14/lib/pg_stat_statements.so    318        3     1212
/usr/lib64/libpthread-2.17.so              320        4     1304
/usr/lib64/ld-2.17.so                      322        4     1373
/usr/pgsql-14/lib/plpgsql.so               318        4     1376
/usr/lib64/libcrypto.so.1.0.2k             319        4     1499
/usr/lib64/libc-2.17.so                    322        9     3179
[stack]                                    322       26     8498
/usr/pgsql-14/bin/postgres                 318       57    18281
/usr/lib64/llvm5.0/lib/libLLVM-5.0.so       10     4098    40984
<anonymous>                                322     6557  2111438
[heap]                                     322    25093  8080218
/dev/zero                                  316    33834 10691660


Kind Regards Ales Zeleny
 

If the RssAnon memory is a good indicator, i can then determine the backends and dump memory context.
It'll take me some time since I'm out of office for vacation, but I'll manage that somewhat way.

Thanks for all to the hints!

Aleš 

regards

--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

pgsql-general by date:

Previous
From: Ron
Date:
Subject: Re: More than one Cluster on single server (single instance)
Next
From: Tom Lane
Date:
Subject: Re: PostgreSQL 14.4 ERROR: out of memory issues