Postgres Pro
Downloads
Home   >   mailing lists

Thread: Memory growth observed with C++ application consuming libpq.dll on Windows

Memory growth observed with C++ application consuming libpq.dll on Windows

From
Rajesh Kokkonda
Date:
Hi,

We are seeing a gradual growth in the memory consumption of our process on Windows. Ours is a C++ application that directly loads libpq.dll and handles the queries and functions. We use setSingleRowMethod to limit the number of rows returned simultaneously to the application. We do not observe any memory increase when the application is run on Linux. There is no code difference between Windows and Linux from the application standpoint. We ran valgrind against our application on Linux and found no memory leaks. Since the same code is being used on Windows as well, we do not suspect any memory leak there.  The question is if there are any known memory leaks with the version of the library we are using on Windows. Kindly let us know.

The version of the library on Linux is libpq.so.5.16

The windows version of the library is 16.0.3.0


image.png

Thanks,
Rajesh
Attachment

Re: Memory growth observed with C++ application consuming libpq.dll on Windows

From
Yasir
Date:
Hi Rajesh, 

Can you please attach a sample code snippet showing libpq's functions being called? It will help to identify the libpq's functions to investigate further for a potential mem leak. 

Regards...

Yasir Hussain

On Thu, Aug 1, 2024 at 4:30 PM Rajesh Kokkonda <rajeshk.kokkonda@gmail.com> wrote:
Hi,

We are seeing a gradual growth in the memory consumption of our process on Windows. Ours is a C++ application that directly loads libpq.dll and handles the queries and functions. We use setSingleRowMethod to limit the number of rows returned simultaneously to the application. We do not observe any memory increase when the application is run on Linux. There is no code difference between Windows and Linux from the application standpoint. We ran valgrind against our application on Linux and found no memory leaks. Since the same code is being used on Windows as well, we do not suspect any memory leak there.  The question is if there are any known memory leaks with the version of the library we are using on Windows. Kindly let us know.

The version of the library on Linux is libpq.so.5.16

The windows version of the library is 16.0.3.0


image.png

Thanks,
Rajesh
Attachment

Re: Memory growth observed with C++ application consuming libpq.dll on Windows

From
Rajesh Kokkonda
Date:
Hi Yasir,

Are you looking for a fully functional sample program or only the APIs from libpq library that our product uses? I am asking this because if the requirement is to have a sample code, then I will have to work on creating one on the same lines as our product.

Rajesh

On Thu, Aug 1, 2024 at 5:27 PM Yasir <yasir.hussain.shah@gmail.com> wrote:
Hi Rajesh, 

Can you please attach a sample code snippet showing libpq's functions being called? It will help to identify the libpq's functions to investigate further for a potential mem leak. 

Regards...

Yasir Hussain

On Thu, Aug 1, 2024 at 4:30 PM Rajesh Kokkonda <rajeshk.kokkonda@gmail.com> wrote:
Hi,

We are seeing a gradual growth in the memory consumption of our process on Windows. Ours is a C++ application that directly loads libpq.dll and handles the queries and functions. We use setSingleRowMethod to limit the number of rows returned simultaneously to the application. We do not observe any memory increase when the application is run on Linux. There is no code difference between Windows and Linux from the application standpoint. We ran valgrind against our application on Linux and found no memory leaks. Since the same code is being used on Windows as well, we do not suspect any memory leak there.  The question is if there are any known memory leaks with the version of the library we are using on Windows. Kindly let us know.

The version of the library on Linux is libpq.so.5.16

The windows version of the library is 16.0.3.0


image.png

Thanks,
Rajesh
Attachment

Re: Memory growth observed with C++ application consuming libpq.dll on Windows

From
Yasir
Date:

On Fri, Aug 2, 2024 at 1:53 PM Rajesh Kokkonda <rajeshk.kokkonda@gmail.com> wrote:
Hi Yasir,

Are you looking for a fully functional sample program or only the APIs from libpq library that our product uses? I am asking this because if the requirement is to have a sample code, then I will have to work on creating one on the same lines as our product.

 
A functional sample is always best and preferred, however, only APIs used by your product would also be sufficient.  

Rajesh

On Thu, Aug 1, 2024 at 5:27 PM Yasir <yasir.hussain.shah@gmail.com> wrote:
Hi Rajesh, 

Can you please attach a sample code snippet showing libpq's functions being called? It will help to identify the libpq's functions to investigate further for a potential mem leak. 

Regards...

Yasir Hussain

On Thu, Aug 1, 2024 at 4:30 PM Rajesh Kokkonda <rajeshk.kokkonda@gmail.com> wrote:
Hi,

We are seeing a gradual growth in the memory consumption of our process on Windows. Ours is a C++ application that directly loads libpq.dll and handles the queries and functions. We use setSingleRowMethod to limit the number of rows returned simultaneously to the application. We do not observe any memory increase when the application is run on Linux. There is no code difference between Windows and Linux from the application standpoint. We ran valgrind against our application on Linux and found no memory leaks. Since the same code is being used on Windows as well, we do not suspect any memory leak there.  The question is if there are any known memory leaks with the version of the library we are using on Windows. Kindly let us know.

The version of the library on Linux is libpq.so.5.16

The windows version of the library is 16.0.3.0


image.png

Thanks,
Rajesh
Attachment

Re: Memory growth observed with C++ application consuming libpq.dll on Windows

From
Rajesh Kokkonda
Date:
Okay. I will try to create one sample program and send it to you sometime next week. In the meantime, I am listing down all the methods we are consuming from libpq.

PQconnectdbParams
PQstatus
PQerrorMessage
PQpingParams
PQfinish
PQresultStatus
PQclear
PQsetSingleRowMode
PQntuples
PQnfields
PQftype
PQgetvalue
PQgetlength
PQgetisnull
PQgetCancel
PQfreeCancel
PQcancel
PQsetErrorVerbosity
PQsendPrepare
PQsendQueryPrepared
PQgetResult
PQconsumeInput
PQisBusy
PQsetnonblocking
PQflush
PQsocket
PQtransactionStatus
PQresultErrorField

Regards,
Rajesh

On Fri, Aug 2, 2024 at 5:06 PM Yasir <yasir.hussain.shah@gmail.com> wrote:

On Fri, Aug 2, 2024 at 1:53 PM Rajesh Kokkonda <rajeshk.kokkonda@gmail.com> wrote:
Hi Yasir,

Are you looking for a fully functional sample program or only the APIs from libpq library that our product uses? I am asking this because if the requirement is to have a sample code, then I will have to work on creating one on the same lines as our product.

 
A functional sample is always best and preferred, however, only APIs used by your product would also be sufficient.  

Rajesh

On Thu, Aug 1, 2024 at 5:27 PM Yasir <yasir.hussain.shah@gmail.com> wrote:
Hi Rajesh, 

Can you please attach a sample code snippet showing libpq's functions being called? It will help to identify the libpq's functions to investigate further for a potential mem leak. 

Regards...

Yasir Hussain

On Thu, Aug 1, 2024 at 4:30 PM Rajesh Kokkonda <rajeshk.kokkonda@gmail.com> wrote:
Hi,

We are seeing a gradual growth in the memory consumption of our process on Windows. Ours is a C++ application that directly loads libpq.dll and handles the queries and functions. We use setSingleRowMethod to limit the number of rows returned simultaneously to the application. We do not observe any memory increase when the application is run on Linux. There is no code difference between Windows and Linux from the application standpoint. We ran valgrind against our application on Linux and found no memory leaks. Since the same code is being used on Windows as well, we do not suspect any memory leak there.  The question is if there are any known memory leaks with the version of the library we are using on Windows. Kindly let us know.

The version of the library on Linux is libpq.so.5.16

The windows version of the library is 16.0.3.0


image.png

Thanks,
Rajesh
Attachment

Re: Memory growth observed with C++ application consuming libpq.dll on Windows

From
Tom Lane
Date:
Rajesh Kokkonda <rajeshk.kokkonda@gmail.com> writes:
> Are you looking for a fully functional sample program or only the APIs from
> libpq library that our product uses? I am asking this because if the
> requirement is to have a sample code, then I will have to work on creating
> one on the same lines as our product.

Just for the record, the last field-reported memory leak in libpq
was found/fixed in 2020, and it occurred only when using GSSAPI
encryption.  Previous reports weren't frequent either.  So while
it may be that you've found one, it seems far more likely that the
fault is in your application.  In any case, nobody is likely to
spend time looking for a bug that may not be there unless you can
produce a self-contained test case demonstrating a leak.

If we had a test case, the first thing we'd likely do would be
to run it under Valgrind, to see if automated analysis is enough
to locate the logic fault.  So an alternative you could consider
before trying to extract a test case is to run your app under
Valgrind for yourself.  As a bonus, that has a decent shot at
locating the fault whether it's ours or yours.  I'm not sure
if Valgrind is available for Windows though --- can you easily
put the app on a different platform?

            regards, tom lane

Re: Memory growth observed with C++ application consuming libpq.dll on Windows

From
Ranier Vilela
Date:
Em sex., 2 de ago. de 2024 às 11:54, Rajesh Kokkonda <rajeshk.kokkonda@gmail.com> escreveu:
Okay. I will try to create one sample program and send it to you sometime next week. In the meantime, I am listing down all the methods we are consuming from libpq.

PQconnectdbParams
PQstatus
PQerrorMessage
PQpingParams
PQfinish
PQresultStatus
PQclear
PQsetSingleRowMode
PQntuples
PQnfields
PQftype
PQgetvalue
PQgetlength
PQgetisnull
PQgetCancel
PQfreeCancel
PQcancel
PQsetErrorVerbosity
PQsendPrepare
PQsendQueryPrepared
PQgetResult
PQconsumeInput
PQisBusy
PQsetnonblocking
PQflush
PQsocket
PQtransactionStatus
PQresultErrorField

It is highly likely that the memory consumption is caused by your application.
Perhaps due to the lack of freeing up the resources used by the library.
You can try using this tool, to find out the root cause.

https://drmemory.org/

best regards,
Ranier Vilela

Re: Memory growth observed with C++ application consuming libpq.dll on Windows

From
Rajesh Kokkonda
Date:
We did run our application under valgrind on Linux. We did not see any leaks. There is no platform dependent code in our application. We are seeing gradual memory growth only on windows.

That is what lead me to believe the leak may be present in postgresql. I will run under available memory tools on windows and get back to you.

I will also try to create a sample and see if I can reproduce the problem.

Thanks,
Rajesh 

On Fri, 2 Aug 2024, 21:45 Ranier Vilela, <ranier.vf@gmail.com> wrote:
Em sex., 2 de ago. de 2024 às 11:54, Rajesh Kokkonda <rajeshk.kokkonda@gmail.com> escreveu:
Okay. I will try to create one sample program and send it to you sometime next week. In the meantime, I am listing down all the methods we are consuming from libpq.

PQconnectdbParams
PQstatus
PQerrorMessage
PQpingParams
PQfinish
PQresultStatus
PQclear
PQsetSingleRowMode
PQntuples
PQnfields
PQftype
PQgetvalue
PQgetlength
PQgetisnull
PQgetCancel
PQfreeCancel
PQcancel
PQsetErrorVerbosity
PQsendPrepare
PQsendQueryPrepared
PQgetResult
PQconsumeInput
PQisBusy
PQsetnonblocking
PQflush
PQsocket
PQtransactionStatus
PQresultErrorField

It is highly likely that the memory consumption is caused by your application.
Perhaps due to the lack of freeing up the resources used by the library.
You can try using this tool, to find out the root cause.

https://drmemory.org/

best regards,
Ranier Vilela

Re: Memory growth observed with C++ application consuming libpq.dll on Windows

From
Rajesh Kokkonda
Date:
I ran a trial version of a memory leak detector called Deleaker on windows and found some modules that are listed as having leaks. I  ran the program on Linux under valgrind and I do not see any leaks reported there. I have attached the reported leaks on windows as windows_leaks.txt and valgrind summary report as valgrind.txt.

I am working on generating a trimmed down version of the sample program to share with you.  Let me know if you have any questions.

Thanks,
Rajesh

On Fri, Aug 2, 2024 at 10:19 PM Rajesh Kokkonda <rajeshk.kokkonda@gmail.com> wrote:
We did run our application under valgrind on Linux. We did not see any leaks. There is no platform dependent code in our application. We are seeing gradual memory growth only on windows.

That is what lead me to believe the leak may be present in postgresql. I will run under available memory tools on windows and get back to you.

I will also try to create a sample and see if I can reproduce the problem.

Thanks,
Rajesh 

On Fri, 2 Aug 2024, 21:45 Ranier Vilela, <ranier.vf@gmail.com> wrote:
Em sex., 2 de ago. de 2024 às 11:54, Rajesh Kokkonda <rajeshk.kokkonda@gmail.com> escreveu:
Okay. I will try to create one sample program and send it to you sometime next week. In the meantime, I am listing down all the methods we are consuming from libpq.

PQconnectdbParams
PQstatus
PQerrorMessage
PQpingParams
PQfinish
PQresultStatus
PQclear
PQsetSingleRowMode
PQntuples
PQnfields
PQftype
PQgetvalue
PQgetlength
PQgetisnull
PQgetCancel
PQfreeCancel
PQcancel
PQsetErrorVerbosity
PQsendPrepare
PQsendQueryPrepared
PQgetResult
PQconsumeInput
PQisBusy
PQsetnonblocking
PQflush
PQsocket
PQtransactionStatus
PQresultErrorField

It is highly likely that the memory consumption is caused by your application.
Perhaps due to the lack of freeing up the resources used by the library.
You can try using this tool, to find out the root cause.

https://drmemory.org/

best regards,
Ranier Vilela
Attachment

Re: Memory growth observed with C++ application consuming libpq.dll on Windows

From
Rajesh Kokkonda
Date:
I attached the image from the utility showing the leaks. 

On Tue, Aug 6, 2024 at 2:03 PM Rajesh Kokkonda <rajeshk.kokkonda@gmail.com> wrote:
I ran a trial version of a memory leak detector called Deleaker on windows and found some modules that are listed as having leaks. I  ran the program on Linux under valgrind and I do not see any leaks reported there. I have attached the reported leaks on windows as windows_leaks.txt and valgrind summary report as valgrind.txt.

I am working on generating a trimmed down version of the sample program to share with you.  Let me know if you have any questions.

Thanks,
Rajesh

On Fri, Aug 2, 2024 at 10:19 PM Rajesh Kokkonda <rajeshk.kokkonda@gmail.com> wrote:
We did run our application under valgrind on Linux. We did not see any leaks. There is no platform dependent code in our application. We are seeing gradual memory growth only on windows.

That is what lead me to believe the leak may be present in postgresql. I will run under available memory tools on windows and get back to you.

I will also try to create a sample and see if I can reproduce the problem.

Thanks,
Rajesh 

On Fri, 2 Aug 2024, 21:45 Ranier Vilela, <ranier.vf@gmail.com> wrote:
Em sex., 2 de ago. de 2024 às 11:54, Rajesh Kokkonda <rajeshk.kokkonda@gmail.com> escreveu:
Okay. I will try to create one sample program and send it to you sometime next week. In the meantime, I am listing down all the methods we are consuming from libpq.

PQconnectdbParams
PQstatus
PQerrorMessage
PQpingParams
PQfinish
PQresultStatus
PQclear
PQsetSingleRowMode
PQntuples
PQnfields
PQftype
PQgetvalue
PQgetlength
PQgetisnull
PQgetCancel
PQfreeCancel
PQcancel
PQsetErrorVerbosity
PQsendPrepare
PQsendQueryPrepared
PQgetResult
PQconsumeInput
PQisBusy
PQsetnonblocking
PQflush
PQsocket
PQtransactionStatus
PQresultErrorField

It is highly likely that the memory consumption is caused by your application.
Perhaps due to the lack of freeing up the resources used by the library.
You can try using this tool, to find out the root cause.

https://drmemory.org/

best regards,
Ranier Vilela
Attachment

Re: Memory growth observed with C++ application consuming libpq.dll on Windows

From
Ranier Vilela
Date:
Em ter., 6 de ago. de 2024 às 05:33, Rajesh Kokkonda <rajeshk.kokkonda@gmail.com> escreveu:
I ran a trial version of a memory leak detector called Deleaker on windows and found some modules that are listed as having leaks. I  ran the program on Linux under valgrind and I do not see any leaks reported there. I have attached the reported leaks on windows as windows_leaks.txt and valgrind summary report as valgrind.txt.
None of these sources are Postgres.


best regards,
Ranier Vilela