Thread: Very slow queries followed by checkpointer process killed with signal 9

Jason McLaurin <jason@jcore.io> writes:
> I'm troubleshooting an issue where about once a week, a database appears to
> lock up and then the PostgreSQL process crashes and recovers. When this
> happens, a few queries will be logged, but there is no pattern to which
> queries are executing when the crash happens, and the queries logged don't
> appear to be queries that would consume a lot of resources.

Hmm, is it always the checkpointer that gets the OOM kill?  That seems
quite odd.

What PG version is this exactly?  Do you have any extensions installed?

I recall having seen somebody before reporting odd slowness of trivial
commands like BEGIN.  I failed to find the thread(s) in the archives
though, so I'm not sure if we identified the cause.

            regards, tom lane

Jason McLaurin <jason@jcore.io> writes:
> Is there anywhere you'd suggest we start looking for hints? I'd be
> interested in increasing relevant logging verbosity so that we can see when
> key background processes are running, both in Postgres core and Timescale.

It might be helpful to try to identify which wait events the slow
queries are blocking on (pg_stat_activity.wait_event_type and
.wait_event).  I'm not sure if you're going to be able to extract
useful data, because your query on pg_stat_activity is likely to
be slow too.  But it's a place to start.

Also, given that you're evidently incurring the wrath of the OOM
killer, you should try to understand why the kernel thinks it's
under memory pressure.  Do you have too many processes, or perhaps
you've configured too much shared memory?

            regards, tom lane

On 4/2/23 21:40, Tom Lane wrote:
> Jason McLaurin <jason@jcore.io> writes:
>> Is there anywhere you'd suggest we start looking for hints? I'd be
>> interested in increasing relevant logging verbosity so that we can see when
>> key background processes are running, both in Postgres core and Timescale.
> 
> It might be helpful to try to identify which wait events the slow
> queries are blocking on (pg_stat_activity.wait_event_type and
> .wait_event).  I'm not sure if you're going to be able to extract
> useful data, because your query on pg_stat_activity is likely to
> be slow too.  But it's a place to start.
> 
> Also, given that you're evidently incurring the wrath of the OOM
> killer, you should try to understand why the kernel thinks it's
> under memory pressure.  Do you have too many processes, or perhaps
> you've configured too much shared memory?

Given this:

> This is Postgres 14.5 running in the TimescaleDB Docker image.

Possibly the docker image is running with a cgroup memory.limit set?

The OOM killer will trigger on any cgroup limit even if the host has 
plenty of free memory.


-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

## Tom Lane (tgl@sss.pgh.pa.us):

> Jason McLaurin <jason@jcore.io> writes:
> > I'm troubleshooting an issue where about once a week, a database appears to
> > lock up and then the PostgreSQL process crashes and recovers. When this
> > happens, a few queries will be logged, but there is no pattern to which
> > queries are executing when the crash happens, and the queries logged don't
> > appear to be queries that would consume a lot of resources.
> 
> Hmm, is it always the checkpointer that gets the OOM kill?  That seems
> quite odd.

Are you sure it's OOM? That's a frequent source of SIGKILLs, but
SIGKILL is not exclusive to oomkiller. Maybe something in the
container stack (some cgroup resource limiting? would be in line
with slowdown at least; or a hard ulimit on cpu time?) or maybe even
some "anti virus" thingy ("It's been 0 days since AV ate the database").

Regards,
Christoph

-- 
Spare Space.