Thread: BUG #17229: Segmentation Fault after upgrading to version 13

The following bug has been logged on the website:

Bug reference:      17229
Logged by:          Efrain Berdecia
Email address:      ejberdecia@yahoo.com
PostgreSQL version: 13.1
Operating system:   Linux  3.10.0-1160.11.1
Description:

Ever since we upgraded to PostgreSQL version 13 we are sporadically getting
segmentation faults for the following process and all existing connections
get killed and our streaming replication process goes down. The sql below
does establish a dblink to a Greenplum cluster. For reference, this is our
log_line_prefix value;
                                  log_line_prefix
------------------------------------------------------------------------------------
 %t [%p]: [%v]-[%x] [%l-1]
cluster_name=campaign_service,user=%u,db=%d,hostname=%r


2021-10-13 03:45:24 UTC [31961]: []-[0] [1-1]
cluster_name=campaign_service,user=[unknown],db=[unknown],hostname=10.110.181.113(65011)
LOG:  connection received: host=10.110.181.113 port=65011
2021-10-13 03:45:24 UTC [31961]: [37/1745]-[0] [2-1]
cluster_name=campaign_service,user=repmgr,db=[unknown],hostname=10.110.181.113(65011)
FATAL:  no pg_hba.conf entry for replication connection from host
"10.110.181.113", user "repmgr", SSL off
2021-10-13 08:00:08 UTC [31961]: []-[0] [1-1]
cluster_name=campaign_service,user=[unknown],db=[unknown],hostname=10.110.149.59(46724)
LOG:  connection received: host=10.110.149.59 port=46724
2021-10-13 08:00:08 UTC [31961]: [52/9474]-[0] [2-1]
cluster_name=campaign_service,user=generic_toolkit_utility,db=bi_tools,hostname=10.110.149.59(46724)
LOG:  connection authorized: user=generic_toolkit_utility
database=bi_tools
2021-10-13 08:01:12 UTC [31961]: [52/0]-[0] [3-1]
cluster_name=campaign_service,user=generic_toolkit_utility,db=bi_tools,hostname=10.110.149.59(46724)
LOG:  duration: 63946.616 ms  statement:
            SELECT dblink_connect('gp_metadata_gpso','greenplum_gpso');

            SELECT dblink_exec('gp_metadata_gpso', 'SET statement_timeout =
90000');

            SET statement_timeout = 600000;

            SELECT gp_metadata.load_functions('gp_metadata_gpso');
            SELECT gp_metadata.load_tables('gp_metadata_gpso');
            SELECT gp_metadata.load_views('gp_metadata_gpso');
            SELECT gp_metadata.load_columns('gp_metadata_gpso');
            SELECT gp_metadata.load_dependencies('gp_metadata_gpso');

            SELECT
gp_metadata.load_from_tmp_to_main('functions','gp_metadata_gpso');
            SELECT
gp_metadata.load_from_tmp_to_main('tables','gp_metadata_gpso');
            SELECT
gp_metadata.load_from_tmp_to_main('views','gp_metadata_gpso');
            SELECT
gp_metadata.load_from_tmp_to_main('dependencies','gp_metadata_gpso');
            SELECT
gp_metadata.load_from_tmp_to_main('columns','gp_metadata_gpso');

            SELECT dblink_disconnect('gp_metadata_gpso');

            SELECT gp_metadata.load_all_objects('gp_metadata_gpso');
            SELECT
gp_metadata.load_from_tmp_to_main('all_objects','gp_metadata_gpso');

            UPDATE gp_metadata.status SET last_update=NOW() WHERE
enviroment='gp_metadata_gpso';
2021-10-13 08:01:36 UTC [59543]: []-[0] [248-1]
cluster_name=campaign_service,user=,db=,hostname= LOG:  server process (PID
31961) was terminated by signal 11: Segmentation fault

Here are the /var/log/messages corresponding entries;
Oct 13 08:01:36 dtord00pgm39p.dc.dotomi.net kernel: traps: postgres[31961]
general protection ip:7ff03e003140 sp:7fff3d87cd58 error:0 in
libc-2.17.so[7ff03deac000+1c4000]
Oct 13 08:01:36 dtord00pgm39p.dc.dotomi.net kernel: [12628017.578633] traps:
postgres[31961] general protection ip:7ff03e003140 sp:7fff3d87cd58 error:0
in libc-2.17.so[7ff03deac000+1c4000]

In other occasions we do see the following messages in /var/log/messages;
Oct 12 22:01:37 dtord00pgm39p.dc.dotomi.net kernel: postgres[14751]:
segfault at 204 ip 00007ff03e003140 sp 00007fff3d87d038 error 4 in
libc-2.17.so[7ff03deac000+1c4000]
Oct 12 22:01:37 dtord00pgm39p.dc.dotomi.net kernel: [12592102.529977]
postgres[14751]: segfault at 204 ip 00007ff03e003140 sp 00007fff3d87d038
error 4 in libc-2.17.so[7ff03deac000+1c4000]

PG Bug reporting form <noreply@postgresql.org> writes:
> Ever since we upgraded to PostgreSQL version 13 we are sporadically getting
> segmentation faults for the following process and all existing connections
> get killed and our streaming replication process goes down.

Hmm, can you get a stack trace from the crash?  See

https://wiki.postgresql.org/wiki/Generating_a_stack_trace_of_a_PostgreSQL_backend

            regards, tom lane

On Wed, Oct 13, 2021 at 3:55 PM, Tom Lane

<tgl@sss.pgh.pa.us> wrote:

PG Bug reporting form <noreply@postgresql.org> writes:

> Ever since we upgraded to PostgreSQL version 13 we are sporadically getting
> segmentation faults for the following process and all existing connections
> get killed and our streaming replication process goes down.

Hmm, can you get a stack trace from the crash?  See

https://wiki.postgresql.org/wiki/Generating_a_stack_trace_of_a_PostgreSQL_backend

            regards, tom lane

"Efrain J. Berdecia" <ejberdecia@yahoo.com> writes:
> Trying to attach it again...
> core.28085.sig11.zip

A core dump is entirely useless except on the system it was generated
on.  Please inspect the dump with gdb and send the textual stack trace,
per the directions in the wiki page I pointed you to.

            regards, tom lane

On Mon, Oct 18, 2021 at 11:20 AM Efrain J. Berdecia
<ejberdecia@yahoo.com> wrote:
> #0  __memcpy_ssse3_back () at ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:1852
> #1  0x00007f42aee2488b in memcpy (__len=12, __src=0x7023c19c7, __dest=0x291967d) at /usr/include/bits/string3.h:51
> #2  gtrgm_alloc (isalltrue=<optimized out>, siglen=siglen@entry=12, sign=0x7023c19c7 <Address 0x7023c19c7 out of
bounds>)at trgm_gist.c:82
 
> #3  0x00007f42aee25d80 in gtrgm_picksplit (fcinfo=<optimized out>) at trgm_gist.c:852
> #4  0x00000000008afcfa in FunctionCall2Coll (flinfo=flinfo@entry=0x2bf9400, collation=<optimized out>,
arg1=arg1@entry=43095272,arg2=arg2@entry=140737194144480) at fmgr.c:1164
 
> #5  0x00000000004b2d03 in gistUserPicksplit (len=2, giststate=0x2bf75d8, itup=0x29193c8, v=0x7fffee76b2e0, attno=0,
entryvec=0x29194e8,r=0x7f42aecd3668) at gistsplit.c:433
 

Commit 911e702077 ("Implement operator class parameters") seems like
the most likely culprit, based on a quick "git blame". Alexander?

-- 
Peter Geoghegan

On Mon, Oct 18, 2021 at 9:27 PM Peter Geoghegan <pg@bowt.ie> wrote:
>
> On Mon, Oct 18, 2021 at 11:20 AM Efrain J. Berdecia
> <ejberdecia@yahoo.com> wrote:
> > #0  __memcpy_ssse3_back () at ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:1852
> > #1  0x00007f42aee2488b in memcpy (__len=12, __src=0x7023c19c7, __dest=0x291967d) at /usr/include/bits/string3.h:51
> > #2  gtrgm_alloc (isalltrue=<optimized out>, siglen=siglen@entry=12, sign=0x7023c19c7 <Address 0x7023c19c7 out of
bounds>)at trgm_gist.c:82
 
> > #3  0x00007f42aee25d80 in gtrgm_picksplit (fcinfo=<optimized out>) at trgm_gist.c:852
> > #4  0x00000000008afcfa in FunctionCall2Coll (flinfo=flinfo@entry=0x2bf9400, collation=<optimized out>,
arg1=arg1@entry=43095272,arg2=arg2@entry=140737194144480) at fmgr.c:1164
 
> > #5  0x00000000004b2d03 in gistUserPicksplit (len=2, giststate=0x2bf75d8, itup=0x29193c8, v=0x7fffee76b2e0, attno=0,
entryvec=0x29194e8,r=0x7f42aecd3668) at gistsplit.c:433
 
>
> Commit 911e702077 ("Implement operator class parameters") seems like
> the most likely culprit, based on a quick "git blame". Alexander?


As I can see from the bug report, the affected version is 13.1.  I
think this bug is already fixed by 48ab1fa304 in 13.2.

------
Regards,
Alexander Korotkov

On Tue, Oct 19, 2021 at 7:17 AM, Alexander Korotkov

<aekorotkov@gmail.com> wrote:

On Mon, Oct 18, 2021 at 9:27 PM Peter Geoghegan <pg@bowt.ie> wrote:

>
> On Mon, Oct 18, 2021 at 11:20 AM Efrain J. Berdecia
> <ejberdecia@yahoo.com> wrote:
> > #0  __memcpy_ssse3_back () at ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:1852
> > #1  0x00007f42aee2488b in memcpy (__len=12, __src=0x7023c19c7, __dest=0x291967d) at /usr/include/bits/string3.h:51
> > #2  gtrgm_alloc (isalltrue=<optimized out>, siglen=siglen@entry=12, sign=0x7023c19c7 <Address 0x7023c19c7 out of bounds>) at trgm_gist.c:82
> > #3  0x00007f42aee25d80 in gtrgm_picksplit (fcinfo=<optimized out>) at trgm_gist.c:852
> > #4  0x00000000008afcfa in FunctionCall2Coll (flinfo=flinfo@entry=0x2bf9400, collation=<optimized out>, arg1=arg1@entry=43095272, arg2=arg2@entry=140737194144480) at fmgr.c:1164
> > #5  0x00000000004b2d03 in gistUserPicksplit (len=2, giststate=0x2bf75d8, itup=0x29193c8, v=0x7fffee76b2e0, attno=0, entryvec=0x29194e8, r=0x7f42aecd3668) at gistsplit.c:433
>
> Commit 911e702077 ("Implement operator class parameters") seems like
> the most likely culprit, based on a quick "git blame". Alexander?

As I can see from the bug report, the affected version is 13.1.  I
think this bug is already fixed by 48ab1fa304 in 13.2.

------
Regards,
Alexander Korotkov

Hi!

On Mon, Oct 25, 2021 at 4:49 PM Efrain J. Berdecia <ejberdecia@yahoo.com> wrote:
> We've been running 13.4 since last Thursday on our test cluster and haven't seen the dump.
>
> I would say the latest patched addressed the issue.

Cool, thank you for reporting it!

> On a separate note, applying 13.4 on a running cluster caused some issues with;
>
> "could not load library "/usr/pgsql-13/lib/plpgsql.so": /usr/pgsql-13/lib/plpgsql.so: undefined symbol:
EnsurePortalSnapshotExists"

Not sure what is our policy on introducing incompatibilities between
extensions and postgres binary in minor releases...

------
Regards,
Alexander Korotkov

Alexander Korotkov <aekorotkov@gmail.com> writes:
> On Mon, Oct 25, 2021 at 4:49 PM Efrain J. Berdecia <ejberdecia@yahoo.com> wrote:
>> On a separate note, applying 13.4 on a running cluster caused some issues with;
>> "could not load library "/usr/pgsql-13/lib/plpgsql.so": /usr/pgsql-13/lib/plpgsql.so: undefined symbol:
EnsurePortalSnapshotExists"

That sounds a lot like the OP hadn't actually restarted the server,
or at least not right away.  A 13.4 plpgsql.so loading into a 13.3
server would do that.

> Not sure what is our policy on introducing incompatibilities between
> extensions and postgres binary in minor releases...

We've done that before, we'll do it again.  You can't realistically
fix many bugs without adding new functions.

            regards, tom lane

On Tue, Oct 26, 2021 at 2:28 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Alexander Korotkov <aekorotkov@gmail.com> writes:
> > On Mon, Oct 25, 2021 at 4:49 PM Efrain J. Berdecia <ejberdecia@yahoo.com> wrote:
> >> On a separate note, applying 13.4 on a running cluster caused some issues with;
> >> "could not load library "/usr/pgsql-13/lib/plpgsql.so": /usr/pgsql-13/lib/plpgsql.so: undefined symbol:
EnsurePortalSnapshotExists"
>
> That sounds a lot like the OP hadn't actually restarted the server,
> or at least not right away.  A 13.4 plpgsql.so loading into a 13.3
> server would do that.
>
> > Not sure what is our policy on introducing incompatibilities between
> > extensions and postgres binary in minor releases...
>
> We've done that before, we'll do it again.  You can't realistically
> fix many bugs without adding new functions.

OK, thank you for the clarification!

------
Regards,
Alexander Korotkov