Thread: [ODBC] ODBC crash after DB cleanup

[ODBC] ODBC crash after DB cleanup

From

"K S, Sandhya (Nokia - IN/Bangalore)"

Date:

28 August 2017, 16:08:07

Hi,

We are using PostgreSQL release 9.3.14 and psqlODBC release 09.03.0400 configured over linux based OS. We are encountering an issue in production where we are observing ODBC crash after rehoming(cleanup of database) and BTS sites are going down after this activity. This issue is observed 3 times in production environment.

Below the bt of the core file.

(gdb) bt

#0 0x000000555a0323c0 in strcpy () from /srv/Log/bak/mnt/backup/iso/lib64/libc.so.6

#1 0x00000055c53ddfa4 in SC_create_errorinfo (self=Cannot access memory at address 0x55bbfe80e0

) at statement.c:1416

Cannot access memory at address 0x55bbfe8108

(gdb) disassemble SC_create_errorinfo

Dump of assembler code for function SC_create_errorinfo:

0x00000055c53dd910 <+0>: daddiu sp,sp,-4272

0x00000055c53dd914 <+4>: sd ra,4264(sp)

0x00000055c53dd918 <+8>: sd s8,4256(sp)

0x00000055c53dd91c <+12>: sd gp,4248(sp)

0x00000055c53dd920 <+16>: move s8,sp

0x00000055c53dd924 <+20>: lui gp,0x6

0x00000055c53dd928 <+24>: daddu gp,gp,t9

0x00000055c53dd92c <+28>: daddiu gp,gp,-26368

…

0x00000055c53ddf98 <+1672>: move t9,v0

0x00000055c53ddf9c <+1676>: jalr t9

0x00000055c53ddfa0 <+1680>: nop

0x00000055c53ddfa4 <+1684>: ld v0,40(s8)

0x00000055c53ddfa8 <+1688>: move sp,s8

0x00000055c53ddfac <+1692>: ld ra,4264(sp)

0x00000055c53ddfb0 <+1696>: ld s8,4256(sp)

0x00000055c53ddfb4 <+1700>: ld gp,4248(sp)

0x00000055c53ddfb8 <+1704>: daddiu sp,sp,4272

0x00000055c53ddfbc <+1708>: jr ra

---Type <return> to continue, or q <return> to quit---

0x00000055c53ddfc0 <+1712>: nop

End of assembler dump

We are trying to reproduce the issue in local environment, no success yet. Enabling ODBC debug logs on production is not an option as we see 50% decrease in performance when we enable ODBC logs.

Please help us in resolving this issue.

Regards,

Sandhya

Re: [ODBC] ODBC crash after DB cleanup

From

"Inoue, Hiroshi"

Date:

29 August 2017, 05:21:56

Hi,

On 2017/08/28 19:08, K S, Sandhya (Nokia - IN/Bangalore) wrote:

Hi,

We are using PostgreSQL release 9.3.14 and psqlODBC release 09.03.0400 configured over linux based OS. We are encountering an issue in production where we are observing ODBC crash after rehoming(cleanup of database) and BTS sites are going down after this activity. This issue is observed 3 times in production environment.

The current version 09.06.0410 of psqlodbc is pretty different from 09.03.0400.
Could you try newer versions?

regards,
Hiroshi Inoue

Below the bt of the core file.

(gdb) bt
#0 0x000000555a0323c0 in strcpy () from /srv/Log/bak/mnt/backup/iso/lib64/libc.so.6
#1 0x00000055c53ddfa4 in SC_create_errorinfo (self=Cannot access memory at address 0x55bbfe80e0
) at statement.c:1416
Cannot access memory at address 0x55bbfe8108
(gdb) disassemble SC_create_errorinfo
Dump of assembler code for function SC_create_errorinfo:
   0x00000055c53dd910 <+0>:     daddiu sp,sp,-4272
   0x00000055c53dd914 <+4>:     sd      ra,4264(sp)
   0x00000055c53dd918 <+8>:     sd      s8,4256(sp)
   0x00000055c53dd91c <+12>:    sd      gp,4248(sp)
   0x00000055c53dd920 <+16>:    move    s8,sp
   0x00000055c53dd924 <+20>:    lui     gp,0x6
   0x00000055c53dd928 <+24>:    daddu   gp,gp,t9
   0x00000055c53dd92c <+28>:    daddiu gp,gp,-26368
…
   0x00000055c53ddf98 <+1672>: move    t9,v0
   0x00000055c53ddf9c <+1676>: jalr    t9
   0x00000055c53ddfa0 <+1680>: nop
   0x00000055c53ddfa4 <+1684>: ld      v0,40(s8)
   0x00000055c53ddfa8 <+1688>: move    sp,s8
   0x00000055c53ddfac <+1692>: ld      ra,4264(sp)
   0x00000055c53ddfb0 <+1696>: ld      s8,4256(sp)
   0x00000055c53ddfb4 <+1700>: ld      gp,4248(sp)
   0x00000055c53ddfb8 <+1704>: daddiu sp,sp,4272
   0x00000055c53ddfbc <+1708>: jr      ra
---Type <return> to continue, or q <return> to quit---
   0x00000055c53ddfc0 <+1712>: nop
End of assembler dump

We are trying to reproduce the issue in local environment, no success yet. Enabling ODBC debug logs on production is not an option as we see 50% decrease in performance when we enable ODBC logs.
Please help us in resolving this issue.

Regards,
Sandhya

Re: [ODBC] ODBC crash after DB cleanup

From

"K S, Sandhya (Nokia - IN/Bangalore)"

Date:

29 August 2017, 12:58:18

Hi,

We will try upgrade as future plan. We are searching for solution for current setup in Production env.

Please help us debug the issue further.

Regards,

Sandhya

From: Inoue, Hiroshi [mailto:h-inoue@dream.email.ne.jp]
Sent: Tuesday, August 29, 2017 4:52 AM
To: K S, Sandhya (Nokia - IN/Bangalore) <sandhya.k_s@nokia.com>
Cc: pgsql-odbc@postgresql.org; T, Rasna (Nokia - IN/Bangalore) <rasna.t@nokia.com>; Itnal, Prakash (Nokia - IN/Bangalore) <prakash.itnal@nokia.com>
Subject: Re: [ODBC] ODBC crash after DB cleanup

Hi,

On 2017/08/28 19:08, K S, Sandhya (Nokia - IN/Bangalore) wrote:

Hi,

We are using PostgreSQL release 9.3.14 and psqlODBC release 09.03.0400 configured over linux based OS. We are encountering an issue in production where we are observing ODBC crash after rehoming(cleanup of database) and BTS sites are going down after this activity. This issue is observed 3 times in production environment.

The current version 09.06.0410 of psqlodbc is pretty different from 09.03.0400.
Could you try newer versions?

regards,
Hiroshi Inoue

Below the bt of the core file.

(gdb) bt
#0 0x000000555a0323c0 in strcpy () from /srv/Log/bak/mnt/backup/iso/lib64/libc.so.6
#1 0x00000055c53ddfa4 in SC_create_errorinfo (self=Cannot access memory at address 0x55bbfe80e0
) at statement.c:1416
Cannot access memory at address 0x55bbfe8108
(gdb) disassemble SC_create_errorinfo
Dump of assembler code for function SC_create_errorinfo:
   0x00000055c53dd910 <+0>:     daddiu sp,sp,-4272
   0x00000055c53dd914 <+4>:     sd      ra,4264(sp)
   0x00000055c53dd918 <+8>:     sd      s8,4256(sp)
   0x00000055c53dd91c <+12>:    sd      gp,4248(sp)
   0x00000055c53dd920 <+16>:    move    s8,sp
   0x00000055c53dd924 <+20>:    lui     gp,0x6
   0x00000055c53dd928 <+24>:    daddu   gp,gp,t9
   0x00000055c53dd92c <+28>:    daddiu gp,gp,-26368
…
   0x00000055c53ddf98 <+1672>: move    t9,v0
   0x00000055c53ddf9c <+1676>: jalr    t9
   0x00000055c53ddfa0 <+1680>: nop
   0x00000055c53ddfa4 <+1684>: ld      v0,40(s8)
   0x00000055c53ddfa8 <+1688>: move    sp,s8
   0x00000055c53ddfac <+1692>: ld      ra,4264(sp)
   0x00000055c53ddfb0 <+1696>: ld      s8,4256(sp)
   0x00000055c53ddfb4 <+1700>: ld      gp,4248(sp)
   0x00000055c53ddfb8 <+1704>: daddiu sp,sp,4272
   0x00000055c53ddfbc <+1708>: jr      ra
---Type <return> to continue, or q <return> to quit---
   0x00000055c53ddfc0 <+1712>: nop
End of assembler dump

We are trying to reproduce the issue in local environment, no success yet. Enabling ODBC debug logs on production is not an option as we see 50% decrease in performance when we enable ODBC logs.
Please help us in resolving this issue.

Regards,
Sandhya