Thread: PGBOUNCER ISSUE PLEASE HELP(Slowing down the site)
Guys
I have an issue going on with PGBOUNCER which is slowing down the site
PGBOUNCER VERSION: pgbouncer-1.5.4-2.el6 (Hosted on separate machine) (16 cpu) 98GB RAM
DATABASE VERION: postgresql 9.3
When the total client connections to pgbouncer are close to 1000, site application works fine but when the total client connections crosses 1150 site application starts showing slowness.
Here is an example of output
postgres@symds-pg:~ $ netstat -atnp | grep 5432 | wc
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
960 6720 104640
As you can see total connections are like 960 right now my site application is working fine. When connections crosses 1150 and even though I see lot of available connections coz my default_pool_size is set high to 250 but still the application gets slow. Database performance on the other end is great with no slow running queries or anything. So the only place I can think the issue is at PGBOUNCER end.
pgbouncer=# show config;
key | value | changeable
---------------------------+----------------------------------+------------
job_name | pgbouncer | no
conffile | /etc/pgbouncer/pgbouncer.ini | yes
logfile | /var/log/pgbouncer.log | yes
pidfile | /var/run/pgbouncer/pgbouncer.pid | no
listen_addr | * | no
listen_port | 5432 | no
listen_backlog | 128 | no
unix_socket_dir | /tmp | no
unix_socket_mode | 511 | no
unix_socket_group | | no
auth_type | md5 | yes
auth_file | /etc/pgbouncer/userlist.txt | yes
pool_mode | transaction | yes
max_client_conn | 3000 | yes
default_pool_size | 250 | yes
min_pool_size | 0 | yes
reserve_pool_size | 0 | yes
reserve_pool_timeout | 5 | yes
syslog | 0 | yes
syslog_facility | daemon | yes
syslog_ident | pgbouncer | yes
user | | no
autodb_idle_timeout | 3600 | yes
server_reset_query | | yes
server_check_query | select 1 | yes
server_check_delay | 30 | yes
query_timeout | 0 | yes
query_wait_timeout | 0 | yes
client_idle_timeout | 0 | yes
client_login_timeout | 60 | yes
idle_transaction_timeout | 0 | yes
server_lifetime | 3600 | yes
server_idle_timeout | 600 | yes
server_connect_timeout | 15 | yes
server_login_retry | 15 | yes
server_round_robin | 0 | yes
suspend_timeout | 10 | yes
ignore_startup_parameters | extra_float_digits | yes
disable_pqexec | 0 | no
dns_max_ttl | 15 | yes
dns_zone_check_period | 0 | yes
max_packet_size | 2147483647 | yes
pkt_buf | 2048 | no
sbuf_loopcnt | 5 | yes
tcp_defer_accept | 1 | yes
tcp_socket_buffer | 0 | yes
tcp_keepalive | 1 | yes
tcp_keepcnt | 0 | yes
tcp_keepidle | 0 | yes
tcp_keepintvl | 0 | yes
verbose | 0 | yes
admin_users | postgres | yes
stats_users | stats, postgres | yes
stats_period | 60 | yes
log_connections | 1 | yes
log_disconnections | 1 | yes
log_pooler_errors | 1 | yes
Thanks
Prabhjot
On Thu, Jun 18, 2015 at 05:09:10PM +0000, Sheena, Prabhjot wrote: > Guys > I have an issue going on with PGBOUNCER which is slowing down the site > > PGBOUNCER VERSION: pgbouncer-1.5.4-2.el6 (Hosted on separate machine) (16 cpu) 98GB RAM > DATABASE VERION: postgresql 9.3 > > When the total client connections to pgbouncer are close to 1000, site application works fine but when the total clientconnections crosses 1150 site application starts showing slowness. > > Here is an example of output > > postgres@symds-pg:~ $ netstat -atnp | grep 5432 | wc > (Not all processes could be identified, non-owned process info > will not be shown, you would have to be root to see it all.) > 960 6720 104640 > > > As you can see total connections are like 960 right now my site application is working fine. When connections crosses 1150 and even though I see lot of available connections coz my default_pool_size is set high to 250 but still the applicationgets slow. Database performance on the other end is great with no slow running queries or anything. So the onlyplace I can think the issue is at PGBOUNCER end. > Hi Prabhjot, This is classic behavior when you have a 1024 file limit. When you are below that number, it work fine. Above that number, you must wait for a connection to close and exit before you can connect which will cause a delay. See what ulimit has to say? Regards, Ken
Guys
I have an issue going on with PGBOUNCER which is slowing down the site
PGBOUNCER VERSION: pgbouncer-1.5.4-2.el6 (Hosted on separate machine) (16 cpu) 98GB RAM
DATABASE VERION: postgresql 9.3
When the total client connections to pgbouncer are close to 1000, site application works fine but when the total client connections crosses 1150 site application starts showing slowness.
Here is an example of output
postgres@symds-pg:~ $ netstat -atnp | grep 5432 | wc
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
960 6720 104640
As you can see total connections are like 960 right now my site application is working fine. When connections crosses 1150 and even though I see lot of available connections coz my default_pool_size is set high to 250 but still the application gets slow. Database performance on the other end is great with no slow running queries or anything. So the only place I can think the issue is at PGBOUNCER end.
pgbouncer=# show config;
key | value | changeable
---------------------------+----------------------------------+------------
job_name | pgbouncer | no
conffile | /etc/pgbouncer/pgbouncer.ini | yes
logfile | /var/log/pgbouncer.log | yes
pidfile | /var/run/pgbouncer/pgbouncer.pid | no
listen_addr | * | no
listen_port | 5432 | no
listen_backlog | 128 | no
unix_socket_dir | /tmp | no
unix_socket_mode | 511 | no
unix_socket_group | | no
auth_type | md5 | yes
auth_file | /etc/pgbouncer/userlist.txt | yes
pool_mode | transaction | yes
max_client_conn | 3000 | yes
default_pool_size | 250 | yes
min_pool_size | 0 | yes
reserve_pool_size | 0 | yes
reserve_pool_timeout | 5 | yes
syslog | 0 | yes
syslog_facility | daemon | yes
syslog_ident | pgbouncer | yes
user | | no
autodb_idle_timeout | 3600 | yes
server_reset_query | | yes
server_check_query | select 1 | yes
server_check_delay | 30 | yes
query_timeout | 0 | yes
query_wait_timeout | 0 | yes
client_idle_timeout | 0 | yes
client_login_timeout | 60 | yes
idle_transaction_timeout | 0 | yes
server_lifetime | 3600 | yes
server_idle_timeout | 600 | yes
server_connect_timeout | 15 | yes
server_login_retry | 15 | yes
server_round_robin | 0 | yes
suspend_timeout | 10 | yes
ignore_startup_parameters | extra_float_digits | yes
disable_pqexec | 0 | no
dns_max_ttl | 15 | yes
dns_zone_check_period | 0 | yes
max_packet_size | 2147483647 | yes
pkt_buf | 2048 | no
sbuf_loopcnt | 5 | yes
tcp_defer_accept | 1 | yes
tcp_socket_buffer | 0 | yes
tcp_keepalive | 1 | yes
tcp_keepcnt | 0 | yes
tcp_keepidle | 0 | yes
tcp_keepintvl | 0 | yes
verbose | 0 | yes
admin_users | postgres | yes
stats_users | stats, postgres | yes
stats_period | 60 | yes
log_connections | 1 | yes
log_disconnections | 1 | yes
log_pooler_errors | 1 | yes
Thanks
Prabhjot
Here is the output of OS limits postgres@symds-pg:~ $ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 790527 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 4096 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 16384 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Thanks Prabhjot -----Original Message----- From: ktm@rice.edu [mailto:ktm@rice.edu] Sent: Thursday, June 18, 2015 10:16 AM To: Sheena, Prabhjot Cc: pgsql-performance@postgresql.org; pgsql-general@postgresql.org Subject: Re: [PERFORM] PGBOUNCER ISSUE PLEASE HELP(Slowing down the site) On Thu, Jun 18, 2015 at 05:09:10PM +0000, Sheena, Prabhjot wrote: > Guys > I have an issue going on with PGBOUNCER which is slowing down > the site > > PGBOUNCER VERSION: pgbouncer-1.5.4-2.el6 (Hosted on separate machine) (16 cpu) 98GB RAM > DATABASE VERION: postgresql 9.3 > > When the total client connections to pgbouncer are close to 1000, site application works fine but when the total clientconnections crosses 1150 site application starts showing slowness. > > Here is an example of output > > postgres@symds-pg:~ $ netstat -atnp | grep 5432 | wc (Not all > processes could be identified, non-owned process info will not be > shown, you would have to be root to see it all.) > 960 6720 104640 > > > As you can see total connections are like 960 right now my site application is working fine. When connections crosses 1150 and even though I see lot of available connections coz my default_pool_size is set high to 250 but still the applicationgets slow. Database performance on the other end is great with no slow running queries or anything. So the onlyplace I can think the issue is at PGBOUNCER end. > Hi Prabhjot, This is classic behavior when you have a 1024 file limit. When you are below that number, it work fine. Above that number,you must wait for a connection to close and exit before you can connect which will cause a delay. See what ulimithas to say? Regards, Ken
On Thu, Jun 18, 2015 at 05:41:01PM +0000, Sheena, Prabhjot wrote: > Here is the output of OS limits > > postgres@symds-pg:~ $ ulimit -a > > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 790527 > max locked memory (kbytes, -l) 32 > max memory size (kbytes, -m) unlimited > open files (-n) 4096 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 10240 > cpu time (seconds, -t) unlimited > max user processes (-u) 16384 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > > > Thanks > Prabhjot > I would bump your open files as was suggested in your pgbouncer start script. Regards, Ken
Hi Ken/ Will I have checked the ulimit value and we are nowhere hitting the max 4096 that we have currently set. Is there any otherexplanation why we should be thinking of bumping it to like ulimit -n 50000 ( Add ulimit -n 50000 to the start of whateveryou use to start pgbouncer (init script, etc..)) even though we are not reaching 4096 max value Regards Prabhjot Singh -----Original Message----- From: ktm@rice.edu [mailto:ktm@rice.edu] Sent: Thursday, June 18, 2015 11:10 AM To: Sheena, Prabhjot Cc: pgsql-performance@postgresql.org; pgsql-general@postgresql.org Subject: Re: [PERFORM] PGBOUNCER ISSUE PLEASE HELP(Slowing down the site) On Thu, Jun 18, 2015 at 05:41:01PM +0000, Sheena, Prabhjot wrote: > Here is the output of OS limits > > postgres@symds-pg:~ $ ulimit -a > > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 790527 > max locked memory (kbytes, -l) 32 > max memory size (kbytes, -m) unlimited > open files (-n) 4096 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 10240 > cpu time (seconds, -t) unlimited > max user processes (-u) 16384 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > > > Thanks > Prabhjot > I would bump your open files as was suggested in your pgbouncer start script. Regards, Ken
"Sheena, Prabhjot" <Prabhjot.Singh@classmates.com> writes: > Hi Ken/ Will > > I have checked the ulimit value and we are nowhere hitting the max > 4096 that we have currently set. Is there any other explanation why > we should be thinking of bumping it to like ulimit -n 50000 ( Add > ulimit -n 50000 to the start of whatever you use to start pgbouncer > (init script, etc..)) even though we are not reaching 4096 max value If I can assume you're running on linux, best you get limits readout from... /proc/$PID-of-bouncer-process/limits Best not to trust that run time env of interactive shell is same as where bouncer launched from. FWIW > Regards > Prabhjot Singh > > > > -----Original Message----- > From: ktm@rice.edu [mailto:ktm@rice.edu] > Sent: Thursday, June 18, 2015 11:10 AM > To: Sheena, Prabhjot > Cc: pgsql-performance@postgresql.org; pgsql-general@postgresql.org > Subject: Re: [PERFORM] PGBOUNCER ISSUE PLEASE HELP(Slowing down the site) > > On Thu, Jun 18, 2015 at 05:41:01PM +0000, Sheena, Prabhjot wrote: >> Here is the output of OS limits >> >> postgres@symds-pg:~ $ ulimit -a >> >> core file size (blocks, -c) 0 >> data seg size (kbytes, -d) unlimited >> scheduling priority (-e) 0 >> file size (blocks, -f) unlimited >> pending signals (-i) 790527 >> max locked memory (kbytes, -l) 32 >> max memory size (kbytes, -m) unlimited >> open files (-n) 4096 >> pipe size (512 bytes, -p) 8 >> POSIX message queues (bytes, -q) 819200 >> real-time priority (-r) 0 >> stack size (kbytes, -s) 10240 >> cpu time (seconds, -t) unlimited >> max user processes (-u) 16384 >> virtual memory (kbytes, -v) unlimited >> file locks (-x) unlimited >> >> >> Thanks >> Prabhjot >> > > I would bump your open files as was suggested in your pgbouncer start script. > > Regards, > Ken -- Jerry Sievers Postgres DBA/Development Consulting e: postgres.consulting@comcast.net p: 312.241.7800
-----Original Message----- From: pgsql-performance-owner@postgresql.org [mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Sheena, Prabhjot Sent: Thursday, June 18, 2015 3:19 PM To: ktm@rice.edu; Will Platnick Cc: pgsql-performance@postgresql.org; pgsql-general@postgresql.org Subject: Re: [PERFORM] PGBOUNCER ISSUE PLEASE HELP(Slowing down the site) Hi Ken/ Will I have checked the ulimit value and we are nowhere hitting the max 4096 that we have currently set. Is there any otherexplanation why we should be thinking of bumping it to like ulimit -n 50000 ( Add ulimit -n 50000 to the start of whateveryou use to start pgbouncer (init script, etc..)) even though we are not reaching 4096 max value Regards Prabhjot Singh -----Original Message----- From: ktm@rice.edu [mailto:ktm@rice.edu] Sent: Thursday, June 18, 2015 11:10 AM To: Sheena, Prabhjot Cc: pgsql-performance@postgresql.org; pgsql-general@postgresql.org Subject: Re: [PERFORM] PGBOUNCER ISSUE PLEASE HELP(Slowing down the site) On Thu, Jun 18, 2015 at 05:41:01PM +0000, Sheena, Prabhjot wrote: > Here is the output of OS limits > > postgres@symds-pg:~ $ ulimit -a > > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 790527 > max locked memory (kbytes, -l) 32 > max memory size (kbytes, -m) unlimited > open files (-n) 4096 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 10240 > cpu time (seconds, -t) unlimited > max user processes (-u) 16384 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > > > Thanks > Prabhjot > I would bump your open files as was suggested in your pgbouncer start script. Regards, Ken --- Why are you so sure that it is PgBouncer causing slowness? You, said, bouncer pool_size is set to 250. How many cores do you have on your db server? Also, why are you running bouncer on a separate machine? It is very "light-weight", so running it on the db server wouldn'trequire much additional resource, but will eliminate some network traffic that you have with the current configuration. Regards, Igor Neyman
On Thu, Jun 18, 2015 at 07:19:13PM +0000, Sheena, Prabhjot wrote: > Hi Ken/ Will > > I have checked the ulimit value and we are nowhere hitting the max 4096 that we have currently set. Is there any otherexplanation why we should be thinking of bumping it to like ulimit -n 50000 ( Add ulimit -n 50000 to the start of whateveryou use to start pgbouncer (init script, etc..)) even though we are not reaching 4096 max value > > Regards > Prabhjot Singh > Hi, Try attaching to the pgbouncer with strace and see if you are getting any particular errors. Do you have a /etc/security/limits.d directory? And if so, what is in it? We found a nice default ulimit of 1024 for all non-root users. :( Regards, Ken
Here is the output of pid postgres@symds-pg:~ $ cat /proc/15610/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 10485760 unlimited bytes Max core file size 0 0 bytes Max resident set unlimited unlimited bytes Max processes 16384 16384 processes Max open files 4096 4096 files Max locked memory 32768 32768 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 790527 790527 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Thanks Prabhjot Singh -----Original Message----- From: Jerry Sievers [mailto:gsievers19@comcast.net] Sent: Thursday, June 18, 2015 12:47 PM To: Sheena, Prabhjot Cc: ktm@rice.edu; Will Platnick; pgsql-performance@postgresql.org; pgsql-general@postgresql.org Subject: Re: [PERFORM] PGBOUNCER ISSUE PLEASE HELP(Slowing down the site) "Sheena, Prabhjot" <Prabhjot.Singh@classmates.com> writes: > Hi Ken/ Will > > I have checked the ulimit value and we are nowhere hitting the max > 4096 that we have currently set. Is there any other explanation why > we should be thinking of bumping it to like ulimit -n 50000 ( Add > ulimit -n 50000 to the start of whatever you use to start pgbouncer > (init script, etc..)) even though we are not reaching 4096 max value If I can assume you're running on linux, best you get limits readout from... /proc/$PID-of-bouncer-process/limits Best not to trust that run time env of interactive shell is same as where bouncer launched from. FWIW > Regards > Prabhjot Singh > > > > -----Original Message----- > From: ktm@rice.edu [mailto:ktm@rice.edu] > Sent: Thursday, June 18, 2015 11:10 AM > To: Sheena, Prabhjot > Cc: pgsql-performance@postgresql.org; pgsql-general@postgresql.org > Subject: Re: [PERFORM] PGBOUNCER ISSUE PLEASE HELP(Slowing down the > site) > > On Thu, Jun 18, 2015 at 05:41:01PM +0000, Sheena, Prabhjot wrote: >> Here is the output of OS limits >> >> postgres@symds-pg:~ $ ulimit -a >> >> core file size (blocks, -c) 0 >> data seg size (kbytes, -d) unlimited >> scheduling priority (-e) 0 >> file size (blocks, -f) unlimited >> pending signals (-i) 790527 >> max locked memory (kbytes, -l) 32 >> max memory size (kbytes, -m) unlimited >> open files (-n) 4096 >> pipe size (512 bytes, -p) 8 >> POSIX message queues (bytes, -q) 819200 >> real-time priority (-r) 0 >> stack size (kbytes, -s) 10240 >> cpu time (seconds, -t) unlimited >> max user processes (-u) 16384 >> virtual memory (kbytes, -v) unlimited >> file locks (-x) unlimited >> >> >> Thanks >> Prabhjot >> > > I would bump your open files as was suggested in your pgbouncer start script. > > Regards, > Ken -- Jerry Sievers Postgres DBA/Development Consulting e: postgres.consulting@comcast.net p: 312.241.7800