Thread: BUG #18569: Memory leak in Postgres Enterprise server
The following bug has been logged on the website: Bug reference: 18569 Logged by: Abhisek Sahu Email address: absahu@informatica.com PostgreSQL version: 16.3 Operating system: Redhat Linux8 Description: We are encountering a issue with our PostgreSQL database, which is impacting our operations. Issue Description: Our Informatica repository setup is enabled with PostgreSQL 16.2 database. We are running multiple concurrent operations for extended periods (10 to 12 hours), which is resulting in database connection errors. Error Details: We have observed that memory consumption by PostgreSQL processes increases gradually over time. Eventually, this leads to out-of-memory errors and process termination. The log entries are as follows- We also noticed that until the application either hangs or we manually terminate the connection on our side, the memory used by these processes on the PostgreSQL server continues to increase. This has led to severe performance issues and frequent database connection errors. Additional Information: PostgreSQL Server/Client Version: PostgreSQL 16.2 Connector in Use: LibPQ We have also attached our Repository logs, PostgreSQL server logs and DB logs. Based on our analysis, it seems that processes are being killed due to out-of-memory errors, as indicated by the logs from around 17:02 We would appreciate your assistance in analyzing and resolving this memory leak issue as soon as possible.
> On 6 Aug 2024, at 10:34, PG Bug reporting form <noreply@postgresql.org> wrote: > PostgreSQL Server/Client Version: PostgreSQL 16.2 Your report mentions "Postgres Enterprise Server", are you running a product from a vendor like EDB or are you running postgres installed by yourself via a package manager (or similar)? If you are buying postgres from a vendor then please contact them, if not you are in the right place. Upgrading to the latest 16.3 is also recommended even though there are no references to memory leak fixes in the release notes. > We have also attached our Repository logs, PostgreSQL server logs and DB > logs. No, there are no logs attached (I don't think the bugreporting form support attaching any files). Please provide additional information as there is nothing to go in the report. -- Daniel Gustafsson
Hi Daniel, Thank you for your response. @Sachan, Vishal please attach all the logs, including the ODL tracing. Adding @Yeddula, Madhusudhan reddy [CONTINGENT WORKER] DBA to answer the below queries. Regards Abhisek -----Original Message----- From: Daniel Gustafsson <daniel@yesql.se> Sent: Tuesday, August 6, 2024 4:45 PM To: Sahu, Abhisek Kumar <absahu@informatica.com>; pgsql-bugs@lists.postgresql.org Subject: Re: BUG #18569: Memory leak in Postgres Enterprise server Caution: This email originated from outside of the organization. Review for Phishing! > On 6 Aug 2024, at 10:34, PG Bug reporting form <noreply@postgresql.org> wrote: > PostgreSQL Server/Client Version: PostgreSQL 16.2 Your report mentions "Postgres Enterprise Server", are you running a product from a vendor like EDB or are you running postgresinstalled by yourself via a package manager (or similar)? If you are buying postgres from a vendor then please contactthem, if not you are in the right place. Upgrading to the latest 16.3 is also recommended even though there are no references to memory leak fixes in the releasenotes. > We have also attached our Repository logs, PostgreSQL server logs and > DB logs. No, there are no logs attached (I don't think the bugreporting form support attaching any files). Please provide additionalinformation as there is nothing to go in the report. -- Daniel Gustafsson
Hi @Daniel, The log files are attached for your reference. Please let us know if you have any queries. Thanks, Renci Monthi Mathias | QA Engineer Bagmane Tech Park, C V Raman Nagar, Bangalore - 560093 Email: rmathias@informatica.com -----Original Message----- From: Sahu, Abhisek Kumar <absahu@informatica.com> Sent: Tuesday, August 6, 2024 5:31 PM To: Daniel Gustafsson <daniel@yesql.se>; pgsql-bugs@lists.postgresql.org; Yeddula, Madhusudhan reddy [CONTINGENT WORKER]<myeddula@informatica.com>; Sachan, Vishal <vsachan@informatica.com> Cc: Mathias, Renci <rmathias@informatica.com>; Kumar, Gaurav <gaurkumar@informatica.com> Subject: RE: BUG #18569: Memory leak in Postgres Enterprise server Hi Daniel, Thank you for your response. @Sachan, Vishal please attach all the logs, including the ODL tracing. Adding @Yeddula, Madhusudhan reddy [CONTINGENT WORKER] DBA to answer the below queries. Regards Abhisek -----Original Message----- From: Daniel Gustafsson <daniel@yesql.se> Sent: Tuesday, August 6, 2024 4:45 PM To: Sahu, Abhisek Kumar <absahu@informatica.com>; pgsql-bugs@lists.postgresql.org Subject: Re: BUG #18569: Memory leak in Postgres Enterprise server Caution: This email originated from outside of the organization. Review for Phishing! > On 6 Aug 2024, at 10:34, PG Bug reporting form <noreply@postgresql.org> wrote: > PostgreSQL Server/Client Version: PostgreSQL 16.2 Your report mentions "Postgres Enterprise Server", are you running a product from a vendor like EDB or are you running postgresinstalled by yourself via a package manager (or similar)? If you are buying postgres from a vendor then please contactthem, if not you are in the right place. Upgrading to the latest 16.3 is also recommended even though there are no references to memory leak fixes in the releasenotes. > We have also attached our Repository logs, PostgreSQL server logs and > DB logs. No, there are no logs attached (I don't think the bugreporting form support attaching any files). Please provide additionalinformation as there is nothing to go in the report. -- Daniel Gustafsson
Attachment
RE: BUG #18569: Memory leak in Postgres Enterprise server
From
"Yeddula, Madhusudhan reddy [CONTINGENT WORKER]"
Date:
HI @Daniel Gustafsson Your report mentions "Postgres Enterprise Server", are you running a product from a vendor like EDB ANS : We are not using Postgres Enterprise Server from EDB. Are you running postgres installed by yourself via a package manager (or similar)? Ans : Installed postgres myself through YUM menthod If you are buying postgres from a vendor then please contact them, if not you are in the right place. We did not buy postgres from any vendor . NOTE : We are using opensource postgresql 16.2 server on RHEL9. Madhusudhan | GOC Database Engineer Informatica Global Operation Center Office: Mob: +91 9502034172 Email: DL_GOC_Database_Services@informatica.com -----Original Message----- From: Mathias, Renci <rmathias@informatica.com> Sent: Wednesday, August 7, 2024 11:05 AM To: Daniel Gustafsson <daniel@yesql.se>; pgsql-bugs@lists.postgresql.org; Yeddula, Madhusudhan reddy [CONTINGENT WORKER]<myeddula@informatica.com>; Sahu, Abhisek Kumar <absahu@informatica.com>; Sachan, Vishal <vsachan@informatica.com> Cc: Kumar, Gaurav <gaurkumar@informatica.com> Subject: RE: BUG #18569: Memory leak in Postgres Enterprise server Hi @Daniel, The log files are attached for your reference. Please let us know if you have any queries. Thanks, Renci Monthi Mathias | QA Engineer Bagmane Tech Park, C V Raman Nagar, Bangalore - 560093 Email: rmathias@informatica.com -----Original Message----- From: Sahu, Abhisek Kumar <absahu@informatica.com> Sent: Tuesday, August 6, 2024 5:31 PM To: Daniel Gustafsson <daniel@yesql.se>; pgsql-bugs@lists.postgresql.org; Yeddula, Madhusudhan reddy [CONTINGENT WORKER]<myeddula@informatica.com>; Sachan, Vishal <vsachan@informatica.com> Cc: Mathias, Renci <rmathias@informatica.com>; Kumar, Gaurav <gaurkumar@informatica.com> Subject: RE: BUG #18569: Memory leak in Postgres Enterprise server Hi Daniel, Thank you for your response. @Sachan, Vishal please attach all the logs, including the ODL tracing. Adding @Yeddula, Madhusudhan reddy [CONTINGENT WORKER] DBA to answer the below queries. Regards Abhisek -----Original Message----- From: Daniel Gustafsson <daniel@yesql.se> Sent: Tuesday, August 6, 2024 4:45 PM To: Sahu, Abhisek Kumar <absahu@informatica.com>; pgsql-bugs@lists.postgresql.org Subject: Re: BUG #18569: Memory leak in Postgres Enterprise server Caution: This email originated from outside of the organization. Review for Phishing! > On 6 Aug 2024, at 10:34, PG Bug reporting form <noreply@postgresql.org> wrote: > PostgreSQL Server/Client Version: PostgreSQL 16.2 Your report mentions "Postgres Enterprise Server", are you running a product from a vendor like EDB or are you running postgresinstalled by yourself via a package manager (or similar)? If you are buying postgres from a vendor then please contactthem, if not you are in the right place. Upgrading to the latest 16.3 is also recommended even though there are no references to memory leak fixes in the releasenotes. > We have also attached our Repository logs, PostgreSQL server logs and > DB logs. No, there are no logs attached (I don't think the bugreporting form support attaching any files). Please provide additionalinformation as there is nothing to go in the report. -- Daniel Gustafsson
Hi Daniel, Kindly let us know if you have any updates. Regards Abhisek Kumar Sahu Senior Manager Main: +91 80 40203000 Ext : 3135 Mobile: 9886680490 Email: absahu@informatica.com -----Original Message----- From: Yeddula, Madhusudhan reddy [CONTINGENT WORKER] <myeddula@informatica.com> Sent: Wednesday, August 7, 2024 4:45 PM To: Mathias, Renci <rmathias@informatica.com>; Daniel Gustafsson <daniel@yesql.se>; pgsql-bugs@lists.postgresql.org; Sahu,Abhisek Kumar <absahu@informatica.com>; Sachan, Vishal <vsachan@informatica.com> Cc: Kumar, Gaurav <gaurkumar@informatica.com>; Nayak, Deepak Vaikunta <dnayak@informatica.com> Subject: RE: BUG #18569: Memory leak in Postgres Enterprise server HI @Daniel Gustafsson Your report mentions "Postgres Enterprise Server", are you running a product from a vendor like EDB ANS : We are not using Postgres Enterprise Server from EDB. Are you running postgres installed by yourself via a package manager (or similar)? Ans : Installed postgres myself through YUM menthod If you are buying postgres from a vendor then please contact them, if not you are in the right place. We did not buy postgres from any vendor . NOTE : We are using opensource postgresql 16.2 server on RHEL9. Madhusudhan | GOC Database Engineer Informatica Global Operation Center Office: Mob: +91 9502034172 Email: DL_GOC_Database_Services@informatica.com -----Original Message----- From: Mathias, Renci <rmathias@informatica.com> Sent: Wednesday, August 7, 2024 11:05 AM To: Daniel Gustafsson <daniel@yesql.se>; pgsql-bugs@lists.postgresql.org; Yeddula, Madhusudhan reddy [CONTINGENT WORKER]<myeddula@informatica.com>; Sahu, Abhisek Kumar <absahu@informatica.com>; Sachan, Vishal <vsachan@informatica.com> Cc: Kumar, Gaurav <gaurkumar@informatica.com> Subject: RE: BUG #18569: Memory leak in Postgres Enterprise server Hi @Daniel, The log files are attached for your reference. Please let us know if you have any queries. Thanks, Renci Monthi Mathias | QA Engineer Bagmane Tech Park, C V Raman Nagar, Bangalore - 560093 Email: rmathias@informatica.com -----Original Message----- From: Sahu, Abhisek Kumar <absahu@informatica.com> Sent: Tuesday, August 6, 2024 5:31 PM To: Daniel Gustafsson <daniel@yesql.se>; pgsql-bugs@lists.postgresql.org; Yeddula, Madhusudhan reddy [CONTINGENT WORKER]<myeddula@informatica.com>; Sachan, Vishal <vsachan@informatica.com> Cc: Mathias, Renci <rmathias@informatica.com>; Kumar, Gaurav <gaurkumar@informatica.com> Subject: RE: BUG #18569: Memory leak in Postgres Enterprise server Hi Daniel, Thank you for your response. @Sachan, Vishal please attach all the logs, including the ODL tracing. Adding @Yeddula, Madhusudhan reddy [CONTINGENT WORKER] DBA to answer the below queries. Regards Abhisek -----Original Message----- From: Daniel Gustafsson <daniel@yesql.se> Sent: Tuesday, August 6, 2024 4:45 PM To: Sahu, Abhisek Kumar <absahu@informatica.com>; pgsql-bugs@lists.postgresql.org Subject: Re: BUG #18569: Memory leak in Postgres Enterprise server Caution: This email originated from outside of the organization. Review for Phishing! > On 6 Aug 2024, at 10:34, PG Bug reporting form <noreply@postgresql.org> wrote: > PostgreSQL Server/Client Version: PostgreSQL 16.2 Your report mentions "Postgres Enterprise Server", are you running a product from a vendor like EDB or are you running postgresinstalled by yourself via a package manager (or similar)? If you are buying postgres from a vendor then please contactthem, if not you are in the right place. Upgrading to the latest 16.3 is also recommended even though there are no references to memory leak fixes in the releasenotes. > We have also attached our Repository logs, PostgreSQL server logs and > DB logs. No, there are no logs attached (I don't think the bugreporting form support attaching any files). Please provide additionalinformation as there is nothing to go in the report. -- Daniel Gustafsson
Hi Abhisek, On 8/12/24 07:37, Sahu, Abhisek Kumar wrote: > Hi Daniel, > > Kindly let us know if you have any updates. > It's hard to say what happened, based on the information provided so far. It might be that there really is a bug / memory leak, or maybe the selected query plan is not great, or maybe there's a configuration issue. We'll need more information to find out. 1) What query was the failed backend running? Was it a big long-running query (e.g. in ETL/BI application), or many small OLTP queries? 2) Do you know what execution plan was used? 3) If you rerun the query, does the memory usage grow the same way? This would allow us to collect important data. For example, it would be very useful if you could attach a debugger to a backend consuming a lot of memory, and get memory context stats. 4) Can you share the configuration parameters? I think the memory config is the most interesting one (shared_buffers, work_mem, ...). How much RAM does the system have? 5) Is this a parallel query? In the snippet from /var/log/messages there are 3-5 postgres processes that look like they might be parallel workers (nearly sequential PIDs, same memory pattern). 6) Is this a long-running problem, or did it just start to happen recently? Perhaps after some change/upgrade? regards -- Tomas Vondra