Re: Properly handle OOM death? - Mailing list pgsql-general

> On Mar 13, 2023, at 10:37 AM, Joe Conway <mail@joeconway.com> wrote:
>
> On 3/13/23 13:55, Israel Brewster wrote:
>> 1) They reference a “Memory cgroup out of memory”, which refers back
>> to the opening comment on Joe Conway’s message - this would imply to
>> me that I *AM* running with a cgroup memory.limit set. Not sure how
>> that changes things?
>
> cgroup memory limit is enforced regardless of the actual host level memory pressure. As an example, if your host VM
has128 GB of memory, but your cgroup memory limit is 512MB, you will get an OOM kill when the sum memory usage of all
ofyour postgres processes (and anything else sharing the same cgroup) exceeds 512 MB, even if the host VM has nothing
elsegoing on consuming memory. 
>
> You can check if a memory is set by reading the corresponding virtual file, e.g:
>
> 8<-------------------
> # cat /sys/fs/cgroup/memory/system.slice/postgresql.service/memory.limit_in_bytes
> 9223372036854710272
> 8<-------------------
>
> A few notes:
> 1/ The specific path to memory.limit_in_bytes might vary, but this example is the default for the RHEL 8 postgresql
10RPM. 

Not finding that file specifically (this is probably too much info, but…):

root@novarupta:~# ls /sys/fs/cgroup/system.slice/
 -.mount                   cgroup.threads          dev-hugepages.mount      memory.events.local   memory.swap.events
       proc-diskstats.mount                   ssh.service                           system-postgresql.slice
systemd-resolved.service
 accounts-daemon.service   cgroup.type             dev-lxc-console.mount    memory.high           memory.swap.high
       proc-loadavg.mount                     sys-devices-system-cpu-online.mount   systemd-initctl.socket
systemd-sysctl.service
 cgroup.controllers        console-getty.service   dev-lxc-tty1.mount       memory.low            memory.swap.max
       proc-meminfo.mount                     sys-devices-virtual-net.mount         systemd-journal-flush.service
systemd-sysusers.service
 cgroup.events             console-setup.service   dev-lxc-tty2.mount       memory.max
networkd-dispatcher.service  proc-stat.mount                        sys-fs-fuse-connections.mount
systemd-journald-audit.socket    systemd-tmpfiles-setup-dev.service 
 cgroup.freeze             cpu.pressure            dev-mqueue.mount         memory.min            pids.current
       proc-swaps.mount                       sys-kernel-debug.mount                systemd-journald-dev-log.socket
systemd-tmpfiles-setup.service
 cgroup.max.depth          cpu.stat                dev-ptmx.mount           memory.numa_stat      pids.events
       proc-sys-kernel-random-boot_id.mount   syslog.socket                         systemd-journald.service
systemd-update-utmp.service
 cgroup.max.descendants    cron.service            io.pressure              memory.oom.group      pids.max
       proc-sys-net.mount                     sysstat.service                       systemd-journald.socket
systemd-user-sessions.service
 cgroup.procs              data.mount              keyboard-setup.service   memory.pressure       pool.mount
      'proc-sysrq\x2dtrigger.mount'          'system-container\x2dgetty.slice'      systemd-logind.service
ufw.service
 cgroup.stat               dbus.service            memory.current           memory.stat           postfix.service
       proc-uptime.mount                      system-modprobe.slice                 systemd-networkd.service
uuidd.socket
 cgroup.subtree_control    dbus.socket             memory.events            memory.swap.current   proc-cpuinfo.mount
       rsyslog.service                        system-postfix.slice                  systemd-remount-fs.service 

root@novarupta:~# ls /sys/fs/cgroup/system.slice/system-postgresql.slice/
cgroup.controllers  cgroup.max.depth        cgroup.stat             cgroup.type   io.pressure     memory.events.local
memory.max       memory.oom.group  memory.swap.current  memory.swap.max  pids.max 
cgroup.events       cgroup.max.descendants  cgroup.subtree_control  cpu.pressure  memory.current  memory.high
memory.min       memory.pressure   memory.swap.events   pids.current     postgresql@13-main.service 
cgroup.freeze       cgroup.procs            cgroup.threads          cpu.stat      memory.events   memory.low
memory.numa_stat memory.stat       memory.swap.high     pids.events 

root@novarupta:~# ls /sys/fs/cgroup/system.slice/system-postgresql.slice/postgresql@13-main.service/
cgroup.controllers  cgroup.max.depth        cgroup.stat             cgroup.type   io.pressure     memory.events.local
memory.max       memory.oom.group  memory.swap.current  memory.swap.max  pids.max 
cgroup.events       cgroup.max.descendants  cgroup.subtree_control  cpu.pressure  memory.current  memory.high
memory.min       memory.pressure   memory.swap.events   pids.current 
cgroup.freeze       cgroup.procs            cgroup.threads          cpu.stat      memory.events   memory.low
memory.numa_stat memory.stat       memory.swap.high     pids.events 

>
> 2/ The value above, 9223372036854710272 basically means "no limit" has been set.
>
> 3/ The example assumes cgroup v1. There are very few distro's that enable cgroup v2 by default, and generally I have
notseen much cgroup v2 usage in the wild (although I strongly recommend it), but if you are using cgroup v2 the names
havechanged. You can check by doing: 
>
> 8<--cgroupv2 enabled-----------------
> # stat -fc %T /sys/fs/cgroup/
> cgroup2fs
> 8<--cgroupv1 enabled-----------------
> # stat -fc %T /sys/fs/cgroup/
> tmpfs
> 8<-------------------

Looks like V2:

root@novarupta:~# stat -fc %T /sys/fs/cgroup/
cgroup2fs
root@novarupta:~# lsb_release -a
No LSB modules are available.
Distributor ID:    Ubuntu
Description:    Ubuntu 20.04.3 LTS
Release:    20.04
Codename:    focal

---
Israel Brewster
Software Engineer
Alaska Volcano Observatory
Geophysical Institute - UAF
2156 Koyukuk Drive
Fairbanks AK 99775-7320
Work: 907-474-5172
cell:  907-328-9145

>
>> 2) All the entries contain the line "oom_score_adj:0”, which would
>> seem to imply that the postmaster, with its -900 score is not being
>> directly targeted by the OOM killer.
>
> Sounds correct
>
> --
> Joe Conway
> PostgreSQL Contributors Team
> RDS Open Source Databases
> Amazon Web Services: https://aws.amazon.com
>