Re: Idle processes chewing up CPU? - Mailing list pgsql-general

Hi Craig/Tom,

I've managed to trap the full stack trace this time - 2 processes chewing up
25% each (1 core each on a quad core server), while SELECT * FROM
pg_stat_activity revealed they were <IDLE>. I also confirmed that the two
runaway processes were started by a developer remotely connecting pgAdmin
via SSL. It appears that no actual queries were run, just the connection
established. Unfortunately I couldn't confirm if the connections were dirty
disconnected.


Relevant DLL versions are:

8.3\bin\LIBEAY32.DLL - 0.9.8.5
8.3\bin\SSLEAY32.DLL - 0.9.8.5
Other DLLS - standard for Windows 2003 Server SP2


The juiciest stack traces I captured (in no particular order) were:

ntkrnlpa.exe!KiUnexpectedInterrupt+0x48
ntkrnlpa.exe!KeWaitForSingleObject+0x346
ntkrnlpa.exe!ZwYieldExecution+0x3514
hal.dll!KfRaiseIrql+0xe5
hal.dll!KeRaiseIrqlToSynchLevel+0x8d
hal.dll!HalEndSystemInterrupt+0x67
hal.dll!HalInitializeProcessor+0x856
LIBEAY32.dll!lh_doall_arg+0x1c4


ntkrnlpa.exe!KiUnexpectedInterrupt+0x48
ntkrnlpa.exe!KeWaitForSingleObject+0x346
ntkrnlpa.exe!ZwYieldExecution+0x3514
hal.dll!KfRaiseIrql+0xe5
hal.dll!KeRaiseIrqlToSynchLevel+0x8d
hal.dll!KfLowerIrql+0x62
ntkrnlpa.exe!ZwYieldExecution+0x163a
ntkrnlpa.exe!KeInsertQueueApc+0x57
ntkrnlpa.exe!IoCancelIrp+0x27d
hal.dll!HalEndSystemInterrupt+0x6e
hal.dll!HalInitializeProcessor+0x856
ntkrnlpa.exe!IofCallDriver+0x45
ntkrnlpa.exe!NtWriteFile+0x2943
ntkrnlpa.exe!NtWriteFile+0x36cb
ntkrnlpa.exe!NtDeviceIoControlFile+0x2a
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb64
ntdll.dll!KiFastSystemCallRet
WS2_32.dll!WSARecv+0x65
WSOCK32.dll!recv+0x31
LIBEAY32.dll!BIO_sock_should_retry+0x57
postgres.exe!my_sock_read+0x1b
LIBEAY32.dll!BIO_read+0x6f
SSLEAY32.dll!SSLv3_client_method+0x1ee1
SSLEAY32.dll!SSLv3_client_method+0x22ea
mswsock.dll!StartWsdpService+0x500
SSLEAY32.dll!SSLv3_client_method+0x225a
SSLEAY32.dll!SSLv3_client_method+0x2a15
postgres.exe!pgwin32_waitforsinglesocket+0x1ed
postgres.exe!secure_read+0x26
postgres.exe!pq_recvbuf+0x71
postgres.exe!pq_getbyte+0x15
postgres.exe!SocketBackend+0x6
postgres.exe!PostgresMain+0xbf8
postgres.exe!BackendRun+0x200
postgres.exe!SubPostmasterMain+0x21d
postgres.exe!main+0x177
postgres.exe!__tmainCRTStartup+0x10f
kernel32.dll!ProcessIdToSessionId+0x209


ntkrnlpa.exe!ZwYieldExecution+0x163a
ntkrnlpa.exe!KeSetEvent+0xcc
ntkrnlpa.exe!PsLookupThreadByThreadId+0x54bc
ntkrnlpa.exe!KiDeliverApc+0xbb
ntkrnlpa.exe!ZwYieldExecution+0x3801
ntkrnlpa.exe!KeWaitForSingleObject+0x346
ntkrnlpa.exe!ZwYieldExecution+0x3514
ntkrnlpa.exe!KiCheckForKernelApcDelivery+0x1c
ntkrnlpa.exe!wctomb+0x4229
ntkrnlpa.exe!IofCallDriver+0x45
ntkrnlpa.exe!NtWriteFile+0x2943
ntkrnlpa.exe!NtWriteFile+0x36cb
ntkrnlpa.exe!NtDeviceIoControlFile+0x2a
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb64
ntdll.dll!KiFastSystemCallRet
WS2_32.dll!WSARecv+0x65
WSOCK32.dll!recv+0x31
LIBEAY32.dll!BIO_sock_should_retry+0x57
postgres.exe!my_sock_read+0x1b
LIBEAY32.dll!BIO_read+0x6f
SSLEAY32.dll!SSLv3_client_method+0x1ee1
SSLEAY32.dll!SSLv3_client_method+0x22ea
mswsock.dll!StartWsdpService+0x500
SSLEAY32.dll!SSLv3_client_method+0x225a
SSLEAY32.dll!SSLv3_client_method+0x2a15
postgres.exe!pgwin32_waitforsinglesocket+0x1ed
postgres.exe!secure_read+0x26
postgres.exe!pq_recvbuf+0x71
postgres.exe!pq_getbyte+0x15
postgres.exe!SocketBackend+0x6
postgres.exe!PostgresMain+0xbf8
postgres.exe!BackendRun+0x200
postgres.exe!SubPostmasterMain+0x21d
postgres.exe!main+0x177
postgres.exe!__tmainCRTStartup+0x10f
kernel32.dll!ProcessIdToSessionId+0x209


ntkrnlpa.exe!KiUnexpectedInterrupt+0x48
ntkrnlpa.exe!KeWaitForSingleObject+0x346
ntkrnlpa.exe!ZwYieldExecution+0x3514
ntkrnlpa.exe!KiCheckForKernelApcDelivery+0x1c
ntkrnlpa.exe!NtWriteFile+0x3195
ntkrnlpa.exe!NtDeviceIoControlFile+0x2a
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb64
ntdll.dll!KiFastSystemCallRet
WS2_32.dll!WSARecv+0x65
WSOCK32.dll!recv+0x31
LIBEAY32.dll!BIO_sock_should_retry+0x57
postgres.exe!my_sock_read+0x1b
LIBEAY32.dll!BIO_read+0x6f
SSLEAY32.dll!SSLv3_client_method+0x1ee1
SSLEAY32.dll!SSLv3_client_method+0x22ea
mswsock.dll!StartWsdpService+0x500
SSLEAY32.dll!SSLv3_client_method+0x225a
SSLEAY32.dll!SSLv3_client_method+0x2a15
postgres.exe!pgwin32_waitforsinglesocket+0x1ed
postgres.exe!secure_read+0x26
postgres.exe!pq_recvbuf+0x71
postgres.exe!pq_getbyte+0x15
postgres.exe!SocketBackend+0x6
postgres.exe!PostgresMain+0xbf8
postgres.exe!BackendRun+0x200
postgres.exe!SubPostmasterMain+0x21d
postgres.exe!main+0x177
postgres.exe!__tmainCRTStartup+0x10f
kernel32.dll!ProcessIdToSessionId+0x209


I'd appreciate any help diagnosing this problem - cutting off remote access
via SSL isn't the ideal solution.

Regards,
-Brendan



-----Original Message-----
From: Craig Ringer [mailto:craig@postnewspapers.com.au]
Sent: Wednesday, 5 August 2009 5:44 PM
To: Brendan Hill
Cc: 'Tom Lane'; pgsql-general@postgresql.org
Subject: RE: [GENERAL] Idle processes chewing up CPU?

On Wed, 2009-08-05 at 16:44 +1000, Brendan Hill wrote:
> Hi Craig,
>
> Sorry, I had the stack trace so I thought it was enough. I'll make sure
the
> debug environment is set up and post the full stack traces again.

No worries. Sorry it cost you time.

I've extended the wiki article on win32 debug info to (hopefully)
explain how to identify a useful stack trace, or at least a likely bogus
one. If you feel like a look I'd value your opinion especially when it
comes to clarity/readability.

http://wiki.postgresql.org/wiki/Getting_a_stack_trace_of_a_running_PostgreSQ
L_backend_on_Windows

--
Craig Ringer