On Mon, Mar 11, 2024 at 12:33 AM Avi Weinberg <AviW@gilat.com> wrote:
>
> Hi Experts,
>
>
>
> Your input is most welcome!
>
>
>
> We are using Postgres 13 (and plan to upgrade to 15 soon). We have logical replication with about 40 servers
subscribingto one publisher. 40 Walsender processes are running on the publisher server. When we insert a row into a
tableholding binary data the walsender RAM usage increases by 500MB although the row binary data is only 80MB. We see
thisincrease in all walsender processes. At some point we got OOM and the process was killed.
This sounds like similar reports[1][2] we got before. Were there any
long-running transactions at that time when the 80MB data change was
made? And is it accessible to the core dump of the walsender process
who was killed due to OOM?
> Why does the walsender increases by 500MB when the data change was only 80MB
> Is some of the 500MB increase due to shared memory or each walsender has its own 500MB increase. I assume that if it
wasonly in shared memory we would not have gotten OOM…
> Why when logical_decoding_work_mem = 64MB the RAM is 15 times that size? Shouldn't any additional space be used from
diskand not RAM?
> Will adding streaming = on to publication "PUBLICATION pub WITH (streaming = on)" can alleviate the issue?
> Are there configuration options that can resolve the RAM issue. It can be also in version 15 since we plan to
upgradesoon.
If you're facing a similar issue I shared above, temporarily setting
logical_decoding_work_mem a *lower* value could alleviate the
situation. Setting a lower value would lead to more evictions in
logical decoding, and it probably can avoid using much memory and OOM
(note that the logical decoding gets slower instead). I think there is
a memory accounting issue in logical decoding, which could end up
using memory much more than the logical_decoding_work_mem limit in
some scenarios. This issue is not fixed yet, and setting "streaming =
on" doesn't help.
Regards,
[1] https://www.postgresql.org/message-id/CAMnUB3oYugXCBLSkih%2BqNsWQPciEwos6g_AMbnz_peNoxfHwyw%40mail.gmail.com
[2] https://www.postgresql.org/message-id/17974-f8c9d353a62f414d%40postgresql.org
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
