Thread: High CPU usage
Hello,
I am using PostgreSQL v14.5 on Linux Debian 11.5. I recently observe very
high CPU usage on my Linux system as below
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
2357756 postgres 20 0 2441032 2,3g 4 S 298,7 67,9 2114:58
Tspjzj2Z
I could not find any file named Tspjzj2Z on the file system. I could not
find PID number using below SQL
SELECT * FROM pg_stat_activity
I also see another suggestion like below to identify long running queries
SELECT max(now() - xact_start) FROM pg_stat_activity
WHERE state IN ('idle in transaction',
'active');
I get no long running query at all.
There is no replication of any kind. This is a single instance server which
alows certification login only.
I appreciate any help to figure this out.
Thanks & Regards,
Ertan
On 2022-10-20 15:59, ertan.kucukoglu@1nar.com.tr wrote: > Hello, > > I am using PostgreSQL v14.5 on Linux Debian 11.5. I recently observe very > high CPU usage on my Linux system as below > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 2357756 postgres 20 0 2441032 2,3g 4 S 298,7 67,9 2114:58 > Tspjzj2Z > > I could not find any file named Tspjzj2Z on the file system. I could not > find PID number using below SQL Hi, I'm not an expert in PostgreSQL but that looks like a rogue app, if you're lucky just a miner running as the prostgres user, likely the result of a postgres RCE exploited successfully... The more worring case would be a program exfiltrating and/or encrypting the database in a ransomware attack. The executable has most likely been removed to hide traces, or cleaned up automatically from ex. /tmp, however if the process is still running you should be able to cat the executable, and any other open files, directly from /proc/<pid>/ (look for exe and fd/*). I strongly recommend you check other postgres servers you have, make a copy of any process file found (for later investigation), then isolate or shutdown these servers and proceed with a proper investigation from a livecd or revovery OS. > There is no replication of any kind. This is a single instance server which > alows certification login only. Is is even available from the outside world? Else you should likely audit any internal hosts that could have accessed your postgresql server. If you have firewall logs looks for unusual connection attempts, any evidence of scanning, etc. Hackers will often spend quite some time once inside to gather as much information as possible before doing any real damage, although if this is effectively a miner it would be less likely to be that kind of attack as they would probably not risk getting discovered with something that will at best make them pennies... If you see the process having any open database files, it's possible it's either compressing them to exfiltrate the data or encrypting them, or both.... Hope this helps... -- Thomas
On 10/20/22 12:59, ertan.kucukoglu@1nar.com.tr wrote:
> Hello,
>
> I am using PostgreSQL v14.5 on Linux Debian 11.5. I recently observe very
> high CPU usage on my Linux system as below
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND
> 2357756 postgres 20 0 2441032 2,3g 4 S 298,7 67,9 2114:58
> Tspjzj2Z
The USER is the system user postgres which could be running any program
not necessarily the Postgres server. In this case that would seem to be
the program Tspjzj2Z.
Try running:
top -U postgres -c
to get the full command line under COMMAND.
>
> I could not find any file named Tspjzj2Z on the file system. I could not
> find PID number using below SQL
I doubt this has anything to do with the Postgres server itself.
Is this machine accessible to the Internet?
There is the possibility you have had an intrusion.
>
> SELECT * FROM pg_stat_activity
>
> I also see another suggestion like below to identify long running queries
>
> SELECT max(now() - xact_start) FROM pg_stat_activity
> WHERE state IN ('idle in transaction',
> 'active');
>
> I get no long running query at all.
>
> There is no replication of any kind. This is a single instance server which
> alows certification login only.
>
> I appreciate any help to figure this out.
>
> Thanks & Regards,
> Ertan
>
>
>
>
--
Adrian Klaver
adrian.klaver@aklaver.com
ertan.kucukoglu@1nar.com.tr schrieb am 20.10.2022 um 21:59: > Hello, > > I am using PostgreSQL v14.5 on Linux Debian 11.5. I recently observe very > high CPU usage on my Linux system as below > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 2357756 postgres 20 0 2441032 2,3g 4 S 298,7 67,9 2114:58 > Tspjzj2Z > The program name looks suspiciously as if you have been hacked. See e.g. https://stackoverflow.com/questions/50149034 https://dba.stackexchange.com/questions/191282/ https://www.imperva.com/blog/deep-dive-database-attacks-scarlett-johanssons-picture-used-for-crypto-mining-on-postgre-database/