Thread: High memory retention by QueryExecutorImpl.pgStream.encoding.decoderArray
Hello
I am trying to figure out reasons for high memory utilization by PgConnection object. My application occasionally runs out of memory and the heap dump shows PgConnection as one of the largest consumers of memory.
The heap dump shows PgConnection having a retained size of 640 MB.
We use a connection pool of 8 connections. Each connection object is retaining about 80 MB. If I drill down into the PgConnection object, I see that QueryExecutorImpl taking almost all memory within which it is the pgStream.encoding.decoderArray taking all the memory.
Not all connection objects are 80 MB all the time. Sometimes the heap dump also shows that some connection objects are as low as 3 MB in size.
I would like to understand under what circumstances can pgStream.encoding.decoderArray retain 80MB. I have checked places in the code and have ensured that we are doing resultSet.close, stmt.close and conn.close wherever we are using them. We do have big objects in the database. I have tried to reproduce this problem using a simple program that queries tuples that are large text, some are even more than 500MB. I haven't been able to see such high memory usage of QueryExecutorImpl in my local setup.
We use the following JDBC driver and the server
- JDBC driver build number: 42.2.2
- Server version: 10.4 (Ubuntu 10.4-2.pgdg14.04+1)
Will appreciate if someone can throw light on the reasons for excessive object retention in QueryExecutorImpl.pgStream.encoding.decoderArray which can help me dig this problem further.
Thanks,
Atul
Re: High memory retention by QueryExecutorImpl.pgStream.encoding.decoderArray
From
Dave Cramer
Date:
Hello,
It would be useful if you could try the latest driver as that may have fixed the problem, alternatively it is also what we are developing against.
As you already have a heap dump can you shed any more light on what's going on?
Regards
On Mon, 4 Mar 2019 at 11:01, Atul Deopujari <atuldeopujari@gmail.com> wrote:
HelloI am trying to figure out reasons for high memory utilization by PgConnection object. My application occasionally runs out of memory and the heap dump shows PgConnection as one of the largest consumers of memory.The heap dump shows PgConnection having a retained size of 640 MB.We use a connection pool of 8 connections. Each connection object is retaining about 80 MB. If I drill down into the PgConnection object, I see that QueryExecutorImpl taking almost all memory within which it is the pgStream.encoding.decoderArray taking all the memory.Not all connection objects are 80 MB all the time. Sometimes the heap dump also shows that some connection objects are as low as 3 MB in size.I would like to understand under what circumstances can pgStream.encoding.decoderArray retain 80MB. I have checked places in the code and have ensured that we are doing resultSet.close, stmt.close and conn.close wherever we are using them. We do have big objects in the database. I have tried to reproduce this problem using a simple program that queries tuples that are large text, some are even more than 500MB. I haven't been able to see such high memory usage of QueryExecutorImpl in my local setup.We use the following JDBC driver and the server
- JDBC driver build number: 42.2.2
- Server version: 10.4 (Ubuntu 10.4-2.pgdg14.04+1)
Will appreciate if someone can throw light on the reasons for excessive object retention in QueryExecutorImpl.pgStream.encoding.decoderArray which can help me dig this problem further.Thanks,Atul
Re: High memory retention by QueryExecutorImpl.pgStream.encoding.decoderArray
From
Atul Deopujari
Date:
I have created this issue https://github.com/pgjdbc/pgjdbc/issues/1431 and have also submitted a PR https://github.com/pgjdbc/pgjdbc/pull/1433
--
Atul
On Tue, 5 Mar 2019 at 16:30, Dave Cramer <pg@fastcrypt.com> wrote:
Hello,It would be useful if you could try the latest driver as that may have fixed the problem, alternatively it is also what we are developing against.As you already have a heap dump can you shed any more light on what's going on?RegardsOn Mon, 4 Mar 2019 at 11:01, Atul Deopujari <atuldeopujari@gmail.com> wrote:HelloI am trying to figure out reasons for high memory utilization by PgConnection object. My application occasionally runs out of memory and the heap dump shows PgConnection as one of the largest consumers of memory.The heap dump shows PgConnection having a retained size of 640 MB.We use a connection pool of 8 connections. Each connection object is retaining about 80 MB. If I drill down into the PgConnection object, I see that QueryExecutorImpl taking almost all memory within which it is the pgStream.encoding.decoderArray taking all the memory.Not all connection objects are 80 MB all the time. Sometimes the heap dump also shows that some connection objects are as low as 3 MB in size.I would like to understand under what circumstances can pgStream.encoding.decoderArray retain 80MB. I have checked places in the code and have ensured that we are doing resultSet.close, stmt.close and conn.close wherever we are using them. We do have big objects in the database. I have tried to reproduce this problem using a simple program that queries tuples that are large text, some are even more than 500MB. I haven't been able to see such high memory usage of QueryExecutorImpl in my local setup.We use the following JDBC driver and the server
- JDBC driver build number: 42.2.2
- Server version: 10.4 (Ubuntu 10.4-2.pgdg14.04+1)
Will appreciate if someone can throw light on the reasons for excessive object retention in QueryExecutorImpl.pgStream.encoding.decoderArray which can help me dig this problem further.Thanks,Atul