Thread: DB connection issue suggestions

On Wed, May 11, 2022 at 12:59:01AM +0800, Sudhir Guna wrote:
>  Dear All,
> 
> We have recently upgraded Postgresql 9.4 standalone server to Postgresql
> 11.2 with High Availability (2 servers : Master and Standby).
> 
> While trying to test using ETL applications and reports, we observe that
> the ETL jobs fails with below error,
> 
> 2022/05/06 16:27:36 - Error occurred while trying to connect to the database
> 2022/05/06 16:27:36 - Error connecting to database: (using class
> org.postgresql.Driver)
> 2022/05/06 16:27:36 - FATAL: Sorry, too many clients already
> 
> We have increased the max_connections = 1000 in postgresql.conf file.
> 
> It worked ok for a day and later we get the same error message.
> 
> Please help to advise on any additional settings required. The prior
> Postgresql 9.4 had the default max_connections = 100 and the applications
> worked fine.

It sounds like at least one thing is still running, perhaps running very
slowly.

You should monitor the number of connections to figure out what.

If you expect to be able to run with only 100 connections, then when
connections>200, there's already over 100 connections which shouldn't still be
there.

You could query pg_stat_activity to determine what they're doing - trying to
run a slow query ?  Are all/most of them stuck doing the same thing ?

You should try to provide the information here for the slow query, and for the
rest of your environment.

https://wiki.postgresql.org/wiki/Slow_Query_Questions

-- 
Justin

On Wed, 2022-05-11 at 00:59 +0800, Sudhir Guna wrote:
> We have recently upgraded Postgresql 9.4 standalone server to Postgresql 11.2 with High Availability (2 servers :
Masterand Standby).
 
> 
> While trying to test using ETL applications and reports, we observe that the ETL jobs fails with below error,
> 
> 2022/05/06 16:27:36 - Error occurred while trying to connect to the database
> 2022/05/06 16:27:36 - Error connecting to database: (using class org.postgresql.Driver)
> 2022/05/06 16:27:36 - FATAL: Sorry, too many clients already
> 
> We have increased the max_connections = 1000 in postgresql.conf file.
> 
> It worked ok for a day and later we get the same error message.
> 
> Please help to advise on any additional settings required. The prior Postgresql 9.4 had the default max_connections =
100andthe applications worked fine.
 

Some application that uses the database has a connection leak: it opens new connections
without closing old ones.  Examine "pg_stat_activity" to find out which application is
at fault, and then go and fix that application.

Yours,
Laurenz Albe
-- 
Cybertec | https://www.cybertec-postgresql.com

Dear team,

Can you confirm whether, post upgrade activity, all the post-upgrade steps including stats update on all the relations
iscomplete. Upgrade doesn’t carry over the stats to the new upgraded cluster. 

Regards,
Anbazhagan M

> On 11-May-2022, at 11:43 AM, Laurenz Albe <laurenz.albe@cybertec.at> wrote:
>
> On Wed, 2022-05-11 at 00:59 +0800, Sudhir Guna wrote:
>> We have recently upgraded Postgresql 9.4 standalone server to Postgresql 11.2 with High Availability (2 servers :
Masterand Standby). 
>>
>> While trying to test using ETL applications and reports, we observe that the ETL jobs fails with below error,
>>
>> 2022/05/06 16:27:36 - Error occurred while trying to connect to the database
>> 2022/05/06 16:27:36 - Error connecting to database: (using class org.postgresql.Driver)
>> 2022/05/06 16:27:36 - FATAL: Sorry, too many clients already
>>
>> We have increased the max_connections = 1000 in postgresql.conf file.
>>
>> It worked ok for a day and later we get the same error message.
>>
>> Please help to advise on any additional settings required. The prior Postgresql 9.4 had the default max_connections
=100and the applications worked fine. 
>
> Some application that uses the database has a connection leak: it opens new connections
> without closing old ones.  Examine "pg_stat_activity" to find out which application is
> at fault, and then go and fix that application.
>
> Yours,
> Laurenz Albe
> --
> Cybertec | https://www.cybertec-postgresql.com
>
>

On Wed, May 11, 2022 at 09:52:10AM +0800, Sudhir Guna wrote:
> Hi Justin,
> 
> Thank you for reviewing.
> 
> I have tried to run the below query and could see only less than 5
> connections active when I get this error. The total rows I see is only 10
> including idle and active sessions for this output.

That doesn't sound right.  Are you sure you're connecting to the correct
instance ?  Are there really only 5 postgres processes on the server, and fewer
than 5 connections to its network port or socket ?

You didn't provide any other info like what OS this is.

-- 
Justin

If the problem occurs gradually (like leaking 20 connections per hour during
ETL), you can check pg_stat_activity every hour or so to try to observe the
problem before all the connection slots are used up, to collect diagnostic
information.

Alternately, leave a connection opened to the DB and wait until all connection
slots *are* used up, and then check pg_stat_activity.  That will take longer,
and you'll have more information to weed through.

What messages are in the server's log ?

v11.2 is years old and hundreds of bugfixes behind.  Since you ran into this
problem anyway, why not run 11.16, which was released today ?

How did you install postgres 11 ?  From source or from packages ?  Which
packages ?  The semi-official PGDG RPM packages are available here:
https://yum.postgresql.org/

-- 
Justin