Thread: Crash on Windows Server 2019
Hi,
We have a multi-threaded Windows server process (C++) that writes data to a PostgreSQL database server using the 11.1 PostgreSQL ODBC driver.
This works smoothly pretty much everywhere except for one system, where the server process seems to crash on a regular basis inside the PostgreSQL ODBC driver, about every 10 minutes.
We have a crash dump that I've been looking at, but since I don't have access to the symbols (I tried this approach, without success), I can't determine why/how the driver is crashing. I do however see in the stack trace that crash is happening in the ConfigDriver() function:
psqlodbc35w!ConfigDriver
The full stack trace of the crashing thread below:
kernel32!WerpReportFaultInternal+0x51b
kernel32!WerpReportFault+0xac
KERNELBASE!UnhandledExceptionFilter+0x35e
ntdll!RtlUserThreadStart$filt$0+0xaa
ntdll!_C_specific_handler+0x96
ntdll!RtlpExecuteHandlerForException+0xf
ntdll!RtlDispatchException+0x40f
ntdll!KiUserExceptionDispatch+0x2e (TrapFrame @ 000000d9`479fd468)
psqlodbc35w!ConfigDriver+0x9253
psqlodbc35w!PgDtc_isolate+0x8314
psqlodbc35w!PgDtc_isolate+0x6ee3
psqlodbc35w!ConfigDriver+0x67cd
psqlodbc35w!ConfigDriver+0x49d5
psqlodbc35w!dconn_FDriverConnectProc+0x2ac1
psqlodbc35w!dconn_FDriverConnectProc+0x3e5e
psqlodbc35w!dconn_FDriverConnectProc+0x24ab
psqlodbc35w!SQLExecDirectW+0xe3
odbc32!SQLExecDirectCover+0x2e0
odbc32!SQLExecDirectW+0xe1
kernel32!WerpReportFault+0xac
KERNELBASE!UnhandledExceptionFilter+0x35e
ntdll!RtlUserThreadStart$filt$0+0xaa
ntdll!_C_specific_handler+0x96
ntdll!RtlpExecuteHandlerForException+0xf
ntdll!RtlDispatchException+0x40f
ntdll!KiUserExceptionDispatch+0x2e (TrapFrame @ 000000d9`479fd468)
psqlodbc35w!ConfigDriver+0x9253
psqlodbc35w!PgDtc_isolate+0x8314
psqlodbc35w!PgDtc_isolate+0x6ee3
psqlodbc35w!ConfigDriver+0x67cd
psqlodbc35w!ConfigDriver+0x49d5
psqlodbc35w!dconn_FDriverConnectProc+0x2ac1
psqlodbc35w!dconn_FDriverConnectProc+0x3e5e
psqlodbc35w!dconn_FDriverConnectProc+0x24ab
psqlodbc35w!SQLExecDirectW+0xe3
odbc32!SQLExecDirectCover+0x2e0
odbc32!SQLExecDirectW+0xe1
We've also let the application run under the application verifier, and the crash happens again, in the ConfigDriver function. I'm not sure what this function does, and why it's having having to configure the driver after issuing an INSERT statement (which I believe the code does). A few more things to help triage this:
* An earlier version of our application works just fine, even though there have been no changes to the database code
* We tried different versions of the ODBC driver (including older ones, 9.6), it crashes with those too
* It works everywhere else, it only fails on this one system
I realize that this is not enough information to immediately come up with a solution, but I'm hoping that somebody can provide me with some ideas on how to get to the bottom of this.
Thank you,
Ingmar.
Twitter @IngmarKoecher
On Sat, 5 Dec 2020 at 00:46, Ingmar Koecher <ingmar.koecher@gmail.com> wrote:
Hi,We have a multi-threaded Windows server process (C++) that writes data to a PostgreSQL database server using the 11.1 PostgreSQL ODBC driver.This works smoothly pretty much everywhere except for one system, where the server process seems to crash on a regular basis inside the PostgreSQL ODBC driver, about every 10 minutes.We have a crash dump that I've been looking at, but since I don't have access to the symbols (I tried this approach, without success), I can't determine why/how the driver is crashing. I do however see in the stack trace that crash is happening in the ConfigDriver() function:psqlodbc35w!ConfigDriverThe full stack trace of the crashing thread below:
Good on you for getting a stack.
How was that obtained?
Can you please repeat it with debug symbols? The .pdb files for postgres itself are in the PostgreSQL install directory. I don't recall off the top of my head where the psqlODBC ones are, but likely the same. Make sure these are on your NT_SYMBOL_PATH or configured to be searched by your debugger.
If you can't get symbols, export a minidump of the crashed process and upload it somewhere. But you will need to understand that the minidump may contain sensitive information including usernames, passwords, queries, user data, etc. I can't offer you a private 1:1 way to exchange this info when I'm doing public mailing list help. So it'd be much better for you to debug the minidump yourself.
Since the issue is system specific, I suggest checking the linker path to ensure there aren't any issues with the wrong libpq or ssl libraries etc being loaded. The "depends.exe" tool from the Sysinternals suite can help with this, or you can examine the loaded libraries in the debugger session.
Also check your ini files and registry to see if there's anything funky and different in the config on the machine.