Thread: Running docker in postgres, SHM size of the docker container in postgres 16
Running docker in postgres, SHM size of the docker container in postgres 16
From
Koen De Groote
Date:
Hello all,
Assume a machine with 16 CPU cores and 64GB of RAM. 300 max connections.
Experience teaches that the default 64MB of /dev/shm for the postgres container, is not enough for databases of even 50GB in size. Previously I've increased the /dev/shm setting to 256MB for the postgres container. Issues that made this necessary all came in the form of too many queries running at once, asking too much memory space, I assume.
I recently upgraded a setup from postgres 11 to postgres 16 via logical replication, and found that running "vacuum(analyze, verbose)" fails unless I give the container 1GB of /dev/shm. Where previously 256MB sufficed on the old postgres 11 version.
Why would that be? It's the exact same data. The install is about 50GB in size. Is there something wrong with postgres 16, or did some settings significantly change, that I need to know about? I went over all the changelogs, nothing stood out as relevant to this, but that might be a lack of experience on my end.
Regards,
Koen De Groote
Re: Running docker in postgres, SHM size of the docker container in postgres 16
From
Thomas Munro
Date:
On Wed, Nov 20, 2024 at 11:22 AM Koen De Groote <kdg.dev@gmail.com> wrote: > Why would that be? It's the exact same data. The install is about 50GB in size. Is there something wrong with postgres16, or did some settings significantly change, that I need to know about? I went over all the changelogs, nothingstood out as relevant to this, but that might be a lack of experience on my end. Parallel vacuum arrived in PostgreSQL 13, and that uses "dynamic shared memory" to share state between workers, and assuming dynamic_shared_memory_type=posix, that means shm_open(), which opens files under /dev/shm on Linux.
Re: Running docker in postgres, SHM size of the docker container in postgres 16
From
Koen De Groote
Date:
That explains a lot.
I have the default of 2 max_parallel_maintenance_workers set, should I set this to 0?
I realize this is of course an improvement, but working with docker containers, I'd like to avoid taking /dev/shm away from regular queries.
I assume setting max_parallel_maintenance_workers to 0 is the fix here, is there perhaps something else I should know about, if I want to have control over this?
Regards,
Koen De Groote
On Wed, Nov 20, 2024 at 12:38 AM Thomas Munro <thomas.munro@gmail.com> wrote:
On Wed, Nov 20, 2024 at 11:22 AM Koen De Groote <kdg.dev@gmail.com> wrote:
> Why would that be? It's the exact same data. The install is about 50GB in size. Is there something wrong with postgres 16, or did some settings significantly change, that I need to know about? I went over all the changelogs, nothing stood out as relevant to this, but that might be a lack of experience on my end.
Parallel vacuum arrived in PostgreSQL 13, and that uses "dynamic
shared memory" to share state between workers, and assuming
dynamic_shared_memory_type=posix, that means shm_open(), which opens
files under /dev/shm on Linux.
Re: Running docker in postgres, SHM size of the docker container in postgres 16
From
felix.quintgz@yahoo.com
Date:
Docker databases do not perform well under heavy load. Everything can be configured to improve performance but going through the Docker disk access layer will significantly decreaseperformance. On Wednesday, November 20, 2024 at 06:06:36 AM GMT-5, Koen De Groote <kdg.dev@gmail.com> wrote: That explains a lot. I have the default of 2 max_parallel_maintenance_workers set, should I set this to 0? I realize this is of course an improvement, but working with docker containers, I'd like to avoid taking /dev/shm away fromregular queries. I assume setting max_parallel_maintenance_workers to 0 is the fix here, is there perhaps something else I should know about,if I want to have control over this? Regards,Koen De Groote On Wed, Nov 20, 2024 at 12:38 AM Thomas Munro <thomas.munro@gmail.com> wrote: On Wed, Nov 20, 2024 at 11:22 AM Koen De Groote <kdg.dev@gmail.com> wrote: > Why would that be? It's the exact same data. The install is about 50GB in size. Is there something wrong with postgres16, or did some settings significantly change, that I need to know about? I went over all the changelogs, nothingstood out as relevant to this, but that might be a lack of experience on my end. Parallel vacuum arrived in PostgreSQL 13, and that uses "dynamic shared memory" to share state between workers, and assuming dynamic_shared_memory_type=posix, that means shm_open(), which opens files under /dev/shm on Linux.