Re: swarm of processes in BIND state? - Mailing list pgsql-general

On Sat, May 28, 2016 at 08:04:43AM +0200, Pavel Stehule wrote:
> > > you should to install debug info - or compile with dubug symbols
> > Installed debug info, and the problem stopped.

OK. ot he problem back.

Ps looked like this:

USER        PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
postgres   9762  0.0  1.3 6816588 424512 ?      S    05:49   0:09 /usr/lib/postgresql/9.3/bin/postgres -D
/var/lib/postgresql/9.3/main-c config_file=/etc/postgresql/9.3/main/postgresql.conf 
postgres   9763  1.6 20.3 6819980 6284520 ?     Ss   05:49  11:49  \_ postgres: startup process   recovering
0000000400007D61000000D2
postgres  10300  0.1 20.0 6827396 6190236 ?     Ss   05:49   1:22  \_ postgres: checkpointer process
postgres  10301  0.1 19.9 6819736 6145144 ?     Ss   05:49   0:55  \_ postgres: writer process
postgres  11111  0.0  0.0 104704  2076 ?        Ss   05:49   0:11  \_ postgres: stats collector process
postgres  11148  0.7  0.0 6832848 4840 ?        Ds   05:49   5:42  \_ postgres: wal receiver process   streaming
7D61/D2CE4D90
postgres  43248  0.0  0.2 6823908 79072 ?       Ss   06:41   0:00  \_ postgres: monitor dbname [local] idle
postgres  14802  4.0 18.5 6898688 5723652 ?     Ss   09:57  19:54  \_ postgres: app_user dbname 10.1.15.177(44508) BIND
postgres  14804  2.9 16.3 6870980 5061020 ?     Ss   09:57  14:25  \_ postgres: app_user dbname 10.1.15.177(44510) BIND
postgres 121044  6.2 16.8 6871756 5189808 ?     Ss   13:19  17:52  \_ postgres: app_user dbname 10.1.14.77(58756) BIND
postgres 130506  5.0 15.6 6871536 4844704 ?     Ss   13:38  13:22  \_ postgres: app_user dbname 10.1.15.177(47734) BIND
postgres   3767  2.8 13.6 6870876 4226928 ?     Ss   13:45   7:20  \_ postgres: app_user dbname 10.1.14.77(59142) BIND
postgres   3768  3.7 16.0 6877232 4943916 ?     Ss   13:45   9:35  \_ postgres: app_user dbname 10.1.14.77(59143) BIND
postgres   7168  0.0  0.0 6821248 8260 ?        Ss   13:52   0:00  \_ postgres: pgbouncer dbname 10.1.16.158(50990)
idle
postgres   7170  0.0  2.0 6863740 637368 ?      Ss   13:52   0:01  \_ postgres: app_user dbname 10.1.16.158(50991) idle
postgres  23083  5.9 14.7 6872584 4552872 ?     Ds   14:22  13:16  \_ postgres: app_user dbname 10.1.15.177(48363) BIND
postgres  70043  0.0  0.0 6821224 7548 ?        Ss   15:52   0:00  \_ postgres: pgbouncer dbname 10.1.14.77(60980) idle
postgres  70719  0.0  0.0 6821248 7448 ?        Ss   15:53   0:00  \_ postgres: pgbouncer dbname 10.1.15.177(49701)
idle
postgres  71257  5.8 14.5 6870940 4479808 ?     Ss   15:54   7:33  \_ postgres: app_user dbname 10.1.14.77(32783) BIND
postgres  77780  5.9 13.3 6871008 4119172 ?     Ss   16:06   6:58  \_ postgres: app_user dbname 10.1.14.77(32951) BIND
postgres  84682  9.2 15.5 6877216 4806100 ?     Ss   16:19   9:40  \_ postgres: app_user dbname 10.1.15.177(50074) BIND
postgres 100924  7.1 13.5 6870804 4176864 ?     Ss   16:51   5:11  \_ postgres: app_user dbname 10.1.15.177(50537) BIND
postgres 118767  7.7 13.7 6870736 4251048 ?     Ss   17:24   3:04  \_ postgres: app_user dbname 10.1.14.77(34090) BIND
postgres 118897  6.2 13.6 6870588 4219824 ?     Ss   17:24   2:27  \_ postgres: app_user dbname 10.1.14.77(34094) BIND
postgres 118916  8.6 12.4 6869524 3852648 ?     Ss   17:25   3:22  \_ postgres: app_user dbname 10.1.15.177(51026) BIND
postgres 118978  3.3 13.0 6868456 4033924 ?     Ss   17:25   1:18  \_ postgres: app_user dbname 10.1.14.77(34100) BIND
postgres 119022  2.3 13.4 6829152 4148088 ?     Ss   17:25   0:53  \_ postgres: app_user dbname 10.1.14.77(34101) BIND
postgres 119060  3.3 13.8 6869172 4265976 ?     Ss   17:25   1:18  \_ postgres: app_user dbname 10.1.15.177(51034) BIND
postgres 119096  1.9 12.1 6828716 3754204 ?     Ss   17:25   0:46  \_ postgres: app_user dbname 10.1.15.177(51037) BIND
postgres 119097  6.4 13.8 6870868 4263168 ?     Ss   17:25   2:29  \_ postgres: app_user dbname 10.1.15.177(51038) BIND
postgres 119111  1.9 12.1 6826656 3763260 ?     Ss   17:25   0:44  \_ postgres: app_user dbname 10.1.14.77(34105) BIND
postgres 119152  3.5 12.7 6869468 3921916 ?     Ss   17:25   1:22  \_ postgres: app_user dbname 10.1.14.77(34107) BIND
postgres 119266  7.3 13.5 6868908 4193496 ?     Ss   17:25   2:49  \_ postgres: app_user dbname 10.1.15.177(51041) BIND
postgres 119298  5.4 13.3 6870552 4107692 ?     Ss   17:25   2:05  \_ postgres: app_user dbname 10.1.14.77(34110) BIND
postgres 119303  6.2 14.7 6870816 4553052 ?     Ss   17:25   2:24  \_ postgres: app_user dbname 10.1.15.177(51043) BIND
postgres 119314  4.9 13.0 6869704 4024040 ?     Ss   17:25   1:54  \_ postgres: app_user dbname 10.1.14.77(34113) BIND
postgres 119315  2.0 12.5 6869208 3881956 ?     Ss   17:25   0:47  \_ postgres: app_user dbname 10.1.15.177(51045) BIND
postgres 119316  7.0 13.3 6870908 4136492 ?     Ss   17:25   2:41  \_ postgres: app_user dbname 10.1.15.177(51046) BIND
postgres 119317  3.2 13.0 6870100 4018860 ?     Ss   17:25   1:14  \_ postgres: app_user dbname 10.1.14.77(34114) BIND
postgres 119318  8.2 14.4 6870868 4457528 ?     Ss   17:25   3:09  \_ postgres: app_user dbname 10.1.14.77(34115) BIND
postgres 119319  2.3 12.0 6827084 3724248 ?     Ss   17:25   0:54  \_ postgres: app_user dbname 10.1.15.177(51047) BIND
postgres 119320  5.0 13.0 6870208 4037396 ?     Ss   17:25   1:56  \_ postgres: app_user dbname 10.1.15.177(51048) BIND
postgres 119368  4.5 13.2 6868480 4085724 ?     Ss   17:25   1:44  \_ postgres: app_user dbname 10.1.15.177(51050) BIND
postgres 119417  4.9 12.8 6869740 3981108 ?     Ss   17:25   1:52  \_ postgres: app_user dbname 10.1.14.77(34117) BIND
postgres 119418  3.4 13.0 6869204 4027344 ?     Ss   17:25   1:18  \_ postgres: app_user dbname 10.1.14.77(34118) BIND
postgres 119441  2.4 12.3 6828944 3826244 ?     Ss   17:25   0:57  \_ postgres: app_user dbname 10.1.15.177(51051) BIND
postgres 119442  3.4 12.6 6869248 3896108 ?     Ss   17:26   1:18  \_ postgres: app_user dbname 10.1.15.177(51052) BIND
postgres 119550 10.7 14.4 6870876 4448704 ?     Ss   17:26   4:04  \_ postgres: app_user dbname 10.1.15.177(51062) BIND
postgres 119560  7.0 14.2 6870744 4402940 ?     Ss   17:26   2:40  \_ postgres: app_user dbname 10.1.14.77(34125) BIND
postgres 119621  3.3 13.1 6870516 4052152 ?     Ss   17:26   1:16  \_ postgres: app_user dbname 10.1.14.77(34128) BIND
postgres 119631  1.9 12.4 6827044 3858696 ?     Ss   17:26   0:43  \_ postgres: app_user dbname 10.1.14.77(34129) BIND
postgres 130410  0.0  1.5 6824028 465040 ?      Ss   17:47   0:00  \_ postgres: app_user dbname 10.1.15.177(51372) idle

I ran perf top and got:

Samples: 620K of event 'cpu-clock', Event count (approx.): 30018858044

       [0/32] 
 23.77%  postgres                [.] s_lock
 13.58%  postgres                [.] 0x000000000027668c
  8.00%  postgres                [.] LWLockAcquire
  6.45%  [kernel]                [k] _raw_spin_unlock_irqrestore
  6.13%  postgres                [.] hash_search_with_hash_value
  4.97%  postgres                [.] LWLockRelease
  4.12%  [kernel]                [k] finish_task_switch
  3.95%  postgres                [.] heap_page_prune_opt
  1.33%  postgres                [.] RecoveryInProgress
  1.19%  [kernel]                [k] copy_user_enhanced_fast_string
  1.09%  postgres                [.] _bt_checkkeys
  1.05%  postgres                [.] heap_hot_search_buffer
  1.02%  libc-2.19.so            [.] 0x0000000000160b96
  0.99%  postgres                [.] hash_any
...

Does that help us in any way?

Best regards,

depesz