PostgreSQL 14.4 ERROR: out of memory issues - Mailing list pgsql-general

Hello,

we are experiencing out-of-memory issues after Postygres upgrade from 14.3 to 14.4 on CentOS7.

OS: CenotOS7

No OOM killer messages in the syslog. SWAP is disabled.

-bash-4.2$ sysctl -q vm.overcommit_memory
vm.overcommit_memory = 2

[root@sts-uat-pgsql100 ~]# swapon -vs
[root@sts-uat-pgsql100 ~]#

[root@sts-uat-pgsql100 ~]# sysctl -q vm.overcommit_ratio
vm.overcommit_ratio = 95

-bash-4.2$ uname -r
3.10.0-1160.42.2.el7.x86_64

-bash-4.2$ free -h
              total        used        free      shared  buff/cache   available
Mem:            39G        7,8G        220M         10G         31G         20G
Swap:            0B          0B          0B

postgres=# show shared_buffers ;
 shared_buffers
----------------
 10GB
(1 row)

postgres=# show  work_mem ;
 work_mem
----------
 4MB
(1 row)

postgres=# show maintenance_work_mem ;
 maintenance_work_mem
----------------------
 512MB
(1 row)

postgres=# show max_parallel_workers;
 max_parallel_workers
----------------------
 8
(1 row)

postgres=# show max_parallel_maintenance_workers ;
 max_parallel_maintenance_workers
----------------------------------
 2
(1 row)

postgres=# select count(*) from pg_stat_activity ;
 count
-------
    93
(1 row)

postgres=# show max_connections ;
 max_connections
-----------------
 1000
(1 row)

No changes on the application side were made before the out-of-memory happened.

So far, it has happened three times (during a single week) from the 14.3 -> 14.4 upgrade, before 14.4 we haven't suffered from such an issue.

Questions:

1)  Can we safely downgrade from 14.4 to 14.3 by shutting down the instance and reinstalling 14.3 PG packages (to prove, that the issue disappear)?

2) What is the best way to diagnose what is the root cause?

What we have started to collect:

- vmstat -nwt (per second)

- vmstat -nawt (per second)

~onece a minute

df  for /dev/shm

/proc/meminfo

proc/<pid>/stat for all PG processes

proc/<pid>/cmdline for all PG processes

proc/<pid>/status for all PG processes

proc/<pid>/smaps for all PG processes

list from pg_stat_activcity

pmap -x for all PG processes

ps aux listing

smem -u --abbreviate

smem -m --abbreviate

smem -w -k

Except the "pmpap" all of the metrics were already collected before the last out of meory issue.

Error messages:

----------------------

postgresql-Fri.log:2022-07-01 06:49:53 CEST [4476]: [20-1] user=,db=,host=,app= LOG:  could not fork worker process: Cannot allocate memory
postgresql-Fri.log:2022-07-01 06:49:53 CEST [4476]: [21-1] user=,db=,host=,app= LOG:  could not fork worker process: Cannot allocate memory
postgresql-Fri.log:2022-07-01 06:50:02 CEST [4476]: [22-1] user=,db=,host=,app= LOG:  could not fork new process for connection: Cannot allocate memory
postgresql-Fri.log:2022-07-01 06:50:02 CEST [4476]: [23-1] user=,db=,host=,app= LOG:  could not fork new process for connection: Cannot allocate memory
postgresql-Fri.log:2022-07-01 06:50:02 CEST [4476]: [24-1] user=,db=,host=,app= LOG:  could not fork new process for connection: Cannot allocate memory

...

2022-07-02 14:48:07 CEST [3930]: [2-1] user=,db=,host=,app= LOG:  automatic vacuum of table "prematch.monitoring.tab_replication_clock": index scans: 0
        pages: 0 removed, 36 remain, 0 skipped due to pins, 34 skipped frozen
        tuples: 0 removed, 203 remain, 201 are dead but not yet removable, oldest xmin: 269822444
        index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed
        I/O timings: read: 0.000 ms, write: 0.000 ms
        avg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s
        buffer usage: 67 hits, 0 misses, 0 dirtied
        WAL usage: 1 records, 0 full page images, 245 bytes
        system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
TopMemoryContext: 279360 total in 10 blocks; 15384 free (9 chunks); 263976 used
  TopTransactionContext: 8192 total in 1 blocks; 7280 free (12 chunks); 912 used
  TOAST to main relid map: 65536 total in 4 blocks; 35168 free (15 chunks); 30368 used
  AV worker: 16384 total in 2 blocks; 10080 free (10 chunks); 6304 used
    Autovacuum Portal: 8192 total in 1 blocks; 7720 free (0 chunks); 472 used
      Vacuum: 8192 total in 1 blocks; 7768 free (0 chunks); 424 used
  Operator class cache: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
  smgr relation table: 16384 total in 2 blocks; 4592 free (2 chunks); 11792 used
  pgstat TabStatusArray lookup hash table: 8192 total in 1 blocks; 1448 free (0 chunks); 6744 used
  TransactionAbortContext: 32768 total in 1 blocks; 32504 free (0 chunks); 264 used
  Portal hash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
  TopPortalContext: 8192 total in 1 blocks; 7928 free (0 chunks); 264 used
  Relcache by OID: 16384 total in 2 blocks; 3504 free (2 chunks); 12880 used
  CacheMemoryContext: 524288 total in 7 blocks; 123552 free (4 chunks); 400736 used
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: tab_queue_tmp_pkey
    index info: 2048 total in 2 blocks; 952 free (0 chunks); 1096 used: pk_replicationclock_databasename
    index info: 2048 total in 2 blocks; 528 free (1 chunks); 1520 used: pg_attrdef_adrelid_adnum_index
    index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: pg_index_indrelid_index
    index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used: pg_db_role_setting_databaseid_rol_index
    index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used: pg_opclass_am_name_nsp_index
    index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used: pg_foreign_data_wrapper_name_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_enum_oid_index
    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_class_relname_nsp_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_foreign_server_oid_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_publication_pubname_index
    index info: 3072 total in 2 blocks; 1128 free (1 chunks); 1944 used: pg_statistic_relid_att_inh_index
    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_cast_source_target_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_language_name_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_transform_oid_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_collation_oid_index
    index info: 3072 total in 2 blocks; 1096 free (2 chunks); 1976 used: pg_amop_fam_strat_index
    index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: pg_index_indexrelid_index
    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_ts_template_tmplname_index
    index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used: pg_ts_config_map_index
    index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: pg_opclass_oid_index
    index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used: pg_foreign_data_wrapper_oid_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_event_trigger_evtname_index
    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_statistic_ext_name_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_publication_oid_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_ts_dict_oid_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_event_trigger_oid_index
    index info: 3072 total in 2 blocks; 1096 free (2 chunks); 1976 used: pg_conversion_default_index
    index info: 3072 total in 2 blocks; 1096 free (2 chunks); 1976 used: pg_operator_oprname_l_r_n_index
    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_trigger_tgrelid_tgname_index
    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_enum_typid_label_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_ts_config_oid_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_user_mapping_oid_index
    index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used: pg_opfamily_am_name_nsp_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_foreign_table_relid_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_type_oid_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_aggregate_fnoid_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_constraint_oid_index
    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_rewrite_rel_rulename_index
    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_ts_parser_prsname_index
    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_ts_config_cfgname_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_ts_parser_oid_index
    index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used: pg_publication_rel_prrelid_prpubid_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_operator_oid_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_namespace_nspname_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_ts_template_oid_index
    index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used: pg_amop_opr_fam_index
    index info: 3072 total in 2 blocks; 1128 free (1 chunks); 1944 used: pg_default_acl_role_nsp_obj_index
    index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used: pg_collation_name_enc_nsp_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_publication_rel_oid_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_range_rngtypid_index
    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_ts_dict_dictname_index
    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_type_typname_nsp_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_opfamily_oid_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_statistic_ext_oid_index
    index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used: pg_statistic_ext_data_stxoid_index
    index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: pg_class_oid_index
    index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used: pg_proc_proname_args_nsp_index
    index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used: pg_partitioned_table_partrelid_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_range_rngmultitypid_index
    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_transform_type_lang_index
    index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: pg_attribute_relid_attnum_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_proc_oid_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_language_oid_index
    index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: pg_namespace_oid_index
    index info: 3072 total in 2 blocks; 776 free (1 chunks); 2296 used: pg_amproc_fam_proc_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_foreign_server_name_index
    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_attribute_relid_attnam_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_conversion_oid_index
    index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used: pg_user_mapping_user_server_index
    index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used: pg_subscription_rel_srrelid_srsubid_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_sequence_seqrelid_index
    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_conversion_name_nsp_index
    index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: pg_authid_oid_index
    index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used: pg_auth_members_member_role_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_subscription_oid_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_tablespace_oid_index
    index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used: pg_shseclabel_object_index
    index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used: pg_replication_origin_roname_index
    index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: pg_database_datname_index
    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_subscription_subname_index
    index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used: pg_replication_origin_roiident_index
    index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used: pg_auth_members_role_member_index
    index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: pg_database_oid_index
    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_authid_rolname_index
  WAL record construction: 49768 total in 2 blocks; 6360 free (0 chunks); 43408 used
  PrivateRefCount: 8192 total in 1 blocks; 2616 free (0 chunks); 5576 used
  MdSmgr: 8192 total in 1 blocks; 7808 free (2 chunks); 384 used
  LOCALLOCK hash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
  plpgsql_check - profiler context: 8192 total in 1 blocks; 7928 free (0 chunks); 264 used
    plpgsql_check function execution statistics: 8192 total in 1 blocks; 2488 free (0 chunks); 5704 used
    plpgsql_check function profiler local chunks: 8192 total in 1 blocks; 2488 free (0 chunks); 5704 used
    plpgsql_check function profiler local cache: 8192 total in 1 blocks; 2488 free (0 chunks); 5704 used
  plpgsql_check function cache: 8192 total in 1 blocks; 2616 free (0 chunks); 5576 used
  Rendezvous variable hash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
  PLpgSQL function hash: 8192 total in 1 blocks; 2616 free (0 chunks); 5576 used
  Timezones: 104120 total in 2 blocks; 2616 free (0 chunks); 101504 used
  Postmaster: 24576 total in 2 blocks; 23824 free (18 chunks); 752 used
    ident parser context: 1024 total in 1 blocks; 760 free (0 chunks); 264 used
    hba parser context: 25600 total in 6 blocks; 4464 free (8 chunks); 21136 used
  ErrorContext: 8192 total in 1 blocks; 7928 free (5 chunks); 264 used
Grand total: 1456224 bytes in 195 blocks; 378824 free (165 chunks); 1077400 used
2022-07-02 14:48:07 CEST [3930]: [3-1] user=,db=,host=,app= ERROR:  out of memory
2022-07-02 14:48:07 CEST [3930]: [4-1] user=,db=,host=,app= DETAIL:  Failed on request of size 152094068 in memory context "TopTransactionContext".
2022-07-02 14:48:07 CEST [3930]: [5-1] user=,db=,host=,app= CONTEXT:  automatic vacuum of table "prematch.replication.tab_queue_tmp"

2022-07-02 14:48:47 CEST [4476]: [42-1] user=,db=,host=,app= LOG:  could not fork worker process: Cannot allocate memory
2022-07-02 14:48:47 CEST [4476]: [43-1] user=,db=,host=,app= LOG:  could not fork worker process: Cannot allocate memory
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
2022-07-02 14:48:47 CEST [4476]: [44-1] user=,db=,host=,app= LOG:  background worker "parallel worker" (PID 4303) was terminated by signal 6: Aborted

A day later, the same issue happens again:

2022-07-02 14:48:07 CEST [3930]: [2-1] user=,db=,host=,app= LOG:  automatic vacuum of table "prematch.monitoring.tab_replication_clock": index scans: 0
11        pages: 0 removed, 36 remain, 0 skipped due to pins, 34 skipped frozen
12        tuples: 0 removed, 203 remain, 201 are dead but not yet removable, oldest xmin: 269822444
13        index scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed
14        I/O timings: read: 0.000 ms, write: 0.000 ms
15        avg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s
16        buffer usage: 67 hits, 0 misses, 0 dirtied
17        WAL usage: 1 records, 0 full page images, 245 bytes
18        system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
19TopMemoryContext: 279360 total in 10 blocks; 15384 free (9 chunks); 263976 used
20  TopTransactionContext: 8192 total in 1 blocks; 7280 free (12 chunks); 912 used
21  TOAST to main relid map: 65536 total in 4 blocks; 35168 free (15 chunks); 30368 used
22  AV worker: 16384 total in 2 blocks; 10080 free (10 chunks); 6304 used
23    Autovacuum Portal: 8192 total in 1 blocks; 7720 free (0 chunks); 472 used
24      Vacuum: 8192 total in 1 blocks; 7768 free (0 chunks); 424 used
25  Operator class cache: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
26  smgr relation table: 16384 total in 2 blocks; 4592 free (2 chunks); 11792 used
27  pgstat TabStatusArray lookup hash table: 8192 total in 1 blocks; 1448 free (0 chunks); 6744 used
28  TransactionAbortContext: 32768 total in 1 blocks; 32504 free (0 chunks); 264 used
29  Portal hash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
30  TopPortalContext: 8192 total in 1 blocks; 7928 free (0 chunks); 264 used
31  Relcache by OID: 16384 total in 2 blocks; 3504 free (2 chunks); 12880 used
32  CacheMemoryContext: 524288 total in 7 blocks; 123552 free (4 chunks); 400736 used
33    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: tab_queue_tmp_pkey
34    index info: 2048 total in 2 blocks; 952 free (0 chunks); 1096 used: pk_replicationclock_databasename
35    index info: 2048 total in 2 blocks; 528 free (1 chunks); 1520 used: pg_attrdef_adrelid_adnum_index
36    index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: pg_index_indrelid_index
37    index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used: pg_db_role_setting_databaseid_rol_index
38    index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used: pg_opclass_am_name_nsp_index
39    index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used: pg_foreign_data_wrapper_name_index
40    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_enum_oid_index
41    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_class_relname_nsp_index
42    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_foreign_server_oid_index
43    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_publication_pubname_index
44    index info: 3072 total in 2 blocks; 1128 free (1 chunks); 1944 used: pg_statistic_relid_att_inh_index
45    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_cast_source_target_index
46    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_language_name_index
47    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_transform_oid_index
48    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_collation_oid_index
49    index info: 3072 total in 2 blocks; 1096 free (2 chunks); 1976 used: pg_amop_fam_strat_index
50    index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: pg_index_indexrelid_index
51    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_ts_template_tmplname_index
52    index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used: pg_ts_config_map_index
53    index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: pg_opclass_oid_index
54    index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used: pg_foreign_data_wrapper_oid_index
55    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_event_trigger_evtname_index
56    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_statistic_ext_name_index
57    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_publication_oid_index
58    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_ts_dict_oid_index
59    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_event_trigger_oid_index
60    index info: 3072 total in 2 blocks; 1096 free (2 chunks); 1976 used: pg_conversion_default_index
61    index info: 3072 total in 2 blocks; 1096 free (2 chunks); 1976 used: pg_operator_oprname_l_r_n_index
62    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_trigger_tgrelid_tgname_index
63    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_enum_typid_label_index
64    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_ts_config_oid_index
65    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_user_mapping_oid_index
66    index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used: pg_opfamily_am_name_nsp_index
67    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_foreign_table_relid_index
68    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_type_oid_index
69    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_aggregate_fnoid_index
70    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_constraint_oid_index
71    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_rewrite_rel_rulename_index
72    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_ts_parser_prsname_index
73    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_ts_config_cfgname_index
74    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_ts_parser_oid_index
75    index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used: pg_publication_rel_prrelid_prpubid_index
76    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_operator_oid_index
77    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_namespace_nspname_index
78    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_ts_template_oid_index
79    index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used: pg_amop_opr_fam_index
80    index info: 3072 total in 2 blocks; 1128 free (1 chunks); 1944 used: pg_default_acl_role_nsp_obj_index
81    index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used: pg_collation_name_enc_nsp_index
82    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_publication_rel_oid_index
83    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_range_rngtypid_index
84    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_ts_dict_dictname_index
85    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_type_typname_nsp_index
86    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_opfamily_oid_index
87    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_statistic_ext_oid_index
88    index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used: pg_statistic_ext_data_stxoid_index
89    index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: pg_class_oid_index
90    index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used: pg_proc_proname_args_nsp_index
91    index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used: pg_partitioned_table_partrelid_index
92    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_range_rngmultitypid_index
93    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_transform_type_lang_index
94    index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: pg_attribute_relid_attnum_index
95    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_proc_oid_index
96    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_language_oid_index
97    index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: pg_namespace_oid_index
98    index info: 3072 total in 2 blocks; 776 free (1 chunks); 2296 used: pg_amproc_fam_proc_index
99    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_foreign_server_name_index
100    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_attribute_relid_attnam_index
101    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_conversion_oid_index
102    index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used: pg_user_mapping_user_server_index
103    index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used: pg_subscription_rel_srrelid_srsubid_index
104    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_sequence_seqrelid_index
105    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_conversion_name_nsp_index
106    index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: pg_authid_oid_index
107    index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used: pg_auth_members_member_role_index
108    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_subscription_oid_index
109    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_tablespace_oid_index
110    index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used: pg_shseclabel_object_index
111    index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used: pg_replication_origin_roname_index
112    index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: pg_database_datname_index
113    index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used: pg_subscription_subname_index
114    index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used: pg_replication_origin_roiident_index
115    index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used: pg_auth_members_role_member_index
116    index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: pg_database_oid_index
117    index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: pg_authid_rolname_index
118  WAL record construction: 49768 total in 2 blocks; 6360 free (0 chunks); 43408 used
119  PrivateRefCount: 8192 total in 1 blocks; 2616 free (0 chunks); 5576 used
120  MdSmgr: 8192 total in 1 blocks; 7808 free (2 chunks); 384 used
121  LOCALLOCK hash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
122  plpgsql_check - profiler context: 8192 total in 1 blocks; 7928 free (0 chunks); 264 used
123    plpgsql_check function execution statistics: 8192 total in 1 blocks; 2488 free (0 chunks); 5704 used
124    plpgsql_check function profiler local chunks: 8192 total in 1 blocks; 2488 free (0 chunks); 5704 used
125    plpgsql_check function profiler local cache: 8192 total in 1 blocks; 2488 free (0 chunks); 5704 used
126  plpgsql_check function cache: 8192 total in 1 blocks; 2616 free (0 chunks); 5576 used
127  Rendezvous variable hash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
128  PLpgSQL function hash: 8192 total in 1 blocks; 2616 free (0 chunks); 5576 used
129  Timezones: 104120 total in 2 blocks; 2616 free (0 chunks); 101504 used
130  Postmaster: 24576 total in 2 blocks; 23824 free (18 chunks); 752 used
131    ident parser context: 1024 total in 1 blocks; 760 free (0 chunks); 264 used
132    hba parser context: 25600 total in 6 blocks; 4464 free (8 chunks); 21136 used
133  ErrorContext: 8192 total in 1 blocks; 7928 free (5 chunks); 264 used
134Grand total: 1456224 bytes in 195 blocks; 378824 free (165 chunks); 1077400 used
1352022-07-02 14:48:07 CEST [3930]: [3-1] user=,db=,host=,app= ERROR:  out of memory
1362022-07-02 14:48:07 CEST [3930]: [4-1] user=,db=,host=,app= DETAIL:  Failed on request of size 152094068 in memory context "TopTransactionContext".
1372022-07-02 14:48:07 CEST [3930]: [5-1] user=,db=,host=,app= CONTEXT:  automatic vacuum of table "prematch.replication.tab_queue_tmp"

And a few days later again:

Thanks Ales Zeleny