Thread: Query out of memory

Query out of memory

From
aditya desai
Date:
Hi,
I am running the below query. Table has 21 million records. I get an Out Of Memory error after a while.(from both pgadmin and psql). Can someone review DB parameters given below.

select t.*,g.column,a.column from
gk_staging g, transaction t,account a
where
g.accountcodeis not null AND
g.accountcode::text <> '' AND
length(g.accountcode)=13 AND
g.closeid::text=t.transactionid::text AND
subsrting(g.accountcode::text,8)=a.mask_code::text

Below are system parameters.
shared_buffers=3GB
work_mem=2GB
effective_cache_size=10GB
maintenance_work_mem=1GB
max_connections=250

I am unable to paste explain plan here due to security concerns.

Regards,
Aditya.

Re: Query out of memory

From
aditya desai
Date:
Database has 20GB RAM.

On Mon, Oct 18, 2021 at 10:12 PM aditya desai <admad123@gmail.com> wrote:
Hi,
I am running the below query. Table has 21 million records. I get an Out Of Memory error after a while.(from both pgadmin and psql). Can someone review DB parameters given below.

select t.*,g.column,a.column from
gk_staging g, transaction t,account a
where
g.accountcodeis not null AND
g.accountcode::text <> '' AND
length(g.accountcode)=13 AND
g.closeid::text=t.transactionid::text AND
subsrting(g.accountcode::text,8)=a.mask_code::text

Below are system parameters.
shared_buffers=3GB
work_mem=2GB
effective_cache_size=10GB
maintenance_work_mem=1GB
max_connections=250

I am unable to paste explain plan here due to security concerns.

Regards,
Aditya.

Re: Query out of memory

From
Vijaykumar Jain
Date:
Work memory 2 GB ?
Is this intentional?
How many max active connections do you see?

if you have too many connections. You can try toning it down to 


On Mon, Oct 18, 2021, 10:13 PM aditya desai <admad123@gmail.com> wrote:
Hi,
I am running the below query. Table has 21 million records. I get an Out Of Memory error after a while.(from both pgadmin and psql). Can someone review DB parameters given below.

select t.*,g.column,a.column from
gk_staging g, transaction t,account a
where
g.accountcodeis not null AND
g.accountcode::text <> '' AND
length(g.accountcode)=13 AND
g.closeid::text=t.transactionid::text AND
subsrting(g.accountcode::text,8)=a.mask_code::text

Below are system parameters.
shared_buffers=3GB
work_mem=2GB
effective_cache_size=10GB
maintenance_work_mem=1GB
max_connections=250

I am unable to paste explain plan here due to security concerns.

Regards,
Aditya.

Re: Query out of memory

From
aditya desai
Date:
Not many active connections. Only 30-40.

On Monday, October 18, 2021, Vijaykumar Jain <vijaykumarjain.github@gmail.com> wrote:
Work memory 2 GB ?
Is this intentional?
How many max active connections do you see?

if you have too many connections. You can try toning it down to 


On Mon, Oct 18, 2021, 10:13 PM aditya desai <admad123@gmail.com> wrote:
Hi,
I am running the below query. Table has 21 million records. I get an Out Of Memory error after a while.(from both pgadmin and psql). Can someone review DB parameters given below.

select t.*,g.column,a.column from
gk_staging g, transaction t,account a
where
g.accountcodeis not null AND
g.accountcode::text <> '' AND
length(g.accountcode)=13 AND
g.closeid::text=t.transactionid::text AND
subsrting(g.accountcode::text,8)=a.mask_code::text

Below are system parameters.
shared_buffers=3GB
work_mem=2GB
effective_cache_size=10GB
maintenance_work_mem=1GB
max_connections=250

I am unable to paste explain plan here due to security concerns.

Regards,
Aditya.

Re: Query out of memory

From
Guillaume Lelarge
Date:
Le lun. 18 oct. 2021 à 21:03, aditya desai <admad123@gmail.com> a écrit :
Not many active connections. Only 30-40.

This means you can consume up to 60-80 GB. Way much more than the available RAM. You should lower your work_mem value.


On Monday, October 18, 2021, Vijaykumar Jain <vijaykumarjain.github@gmail.com> wrote:
Work memory 2 GB ?
Is this intentional?
How many max active connections do you see?

if you have too many connections. You can try toning it down to 


On Mon, Oct 18, 2021, 10:13 PM aditya desai <admad123@gmail.com> wrote:
Hi,
I am running the below query. Table has 21 million records. I get an Out Of Memory error after a while.(from both pgadmin and psql). Can someone review DB parameters given below.

select t.*,g.column,a.column from
gk_staging g, transaction t,account a
where
g.accountcodeis not null AND
g.accountcode::text <> '' AND
length(g.accountcode)=13 AND
g.closeid::text=t.transactionid::text AND
subsrting(g.accountcode::text,8)=a.mask_code::text

Below are system parameters.
shared_buffers=3GB
work_mem=2GB
effective_cache_size=10GB
maintenance_work_mem=1GB
max_connections=250

I am unable to paste explain plan here due to security concerns.

Regards,
Aditya.

Re: Query out of memory

From
aditya desai
Date:
This is how I received a query from the App Team. They have migrated from Oracle to Postgres. I see in Oracle where it is working fine and also has the same joins. Some of the developers increased the work_mem. I will try and tone it down. Will get back to you. Thanks.

On Tue, Oct 19, 2021 at 1:28 AM Geri Wright <geri.w123@gmail.com> wrote:
Hi,
It looks like you have Cartesian joins in the query.   Try updating your where clause to include

And g.columnname = t.columnname 
And t.columnname2 = a.columnname2

On Mon, Oct 18, 2021, 12:43 PM aditya desai <admad123@gmail.com> wrote:
Hi,
I am running the below query. Table has 21 million records. I get an Out Of Memory error after a while.(from both pgadmin and psql). Can someone review DB parameters given below.

select t.*,g.column,a.column from
gk_staging g, transaction t,account a
where
g.accountcodeis not null AND
g.accountcode::text <> '' AND
length(g.accountcode)=13 AND
g.closeid::text=t.transactionid::text AND
subsrting(g.accountcode::text,8)=a.mask_code::text

Below are system parameters.
shared_buffers=3GB
work_mem=2GB
effective_cache_size=10GB
maintenance_work_mem=1GB
max_connections=250

I am unable to paste explain plan here due to security concerns.

Regards,
Aditya.