Thread: BUG #16184: Segmentation Fault during update

The following bug has been logged on the website:

Bug reference:      16184
Logged by:          Piotr Wlodarczyk
Email address:      piotrwlodarczyk89@gmail.com
PostgreSQL version: 12.1
Operating system:   Linux dev-g1-db-synchro.gnb.adm 3.10.0-1062.7.1.el
Description:

Hello geeks,

We're preparing to upgrade postgresql from 11.5 to 12.1 but still we have
some bugs and issues. Most of them are our fault, but now we have problem we
don't know how to play with it.

Log:
2020-01-02 15:42:26 CET [12729]: [95-1] user=,db=,xid=0,state=00000 LOG:
server process (PID 28879) was terminated by signal 11: Segmentation fault
2020-01-02 15:42:26 CET [12729]: [96-1] user=,db=,xid=0,state=00000 DETAIL:
Failed process was running: UPDATE job_abstract a
          SET job_status = job_status
          WHERE (a.xmax :: TEXT :: BIGINT < (SELECT

txid_snapshot_xmin(txid_current_snapshot()))) AND ((SELECT

                        txid_snapshot_xmin(

                            txid_current_snapshot())

                        <= (SELECT

                              txid_snapshot_xmax(

                                  txid_current_snapshot()))))
                AND job_status = 0 AND NOT xmax = 0;

Please don't ask me why this update is made in this way - I suppose to set
xmax to 0 but why? I don't know. Its legacy code, being part of core of our
application.

Reproducting this bug is not easy and I don't know which conditions has to
be met to replicate this bug. But during application is working and we made
this update, segmentation fault occurs. But sometimes it works. 

Column job_status is telling application that this row should be preceded
one more time. Up to 11.5 this bug doesn't exists.

PG Bug reporting form <noreply@postgresql.org> writes:
> Log:
> 2020-01-02 15:42:26 CET [12729]: [95-1] user=,db=,xid=0,state=00000 LOG:
> server process (PID 28879) was terminated by signal 11: Segmentation fault
> 2020-01-02 15:42:26 CET [12729]: [96-1] user=,db=,xid=0,state=00000 DETAIL:
> Failed process was running: UPDATE job_abstract a
>           SET job_status = job_status
>           WHERE (a.xmax :: TEXT :: BIGINT < (SELECT
> txid_snapshot_xmin(txid_current_snapshot()))) AND ((SELECT

Clearly a bug, but there's not much we can do with this amount of
information.  If you can provide a stack trace, that'd be helpful.

https://wiki.postgresql.org/wiki/Generating_a_stack_trace_of_a_PostgreSQL_backend

> Reproducting this bug is not easy and I don't know which conditions has to
> be met to replicate this bug. But during application is working and we made
> this update, segmentation fault occurs. But sometimes it works.

My gut instinct is that this might be a concurrent-update problem,
which would explain the lack of reproducibility.  Try to get
multiple sessions to update the same table row(s) at the same time.

Also, we had some bugs of that ilk on 12.0, please make sure you
really are updated to 12.1.

            regards, tom lane

=?UTF-8?Q?Piotr_W=C5=82odarczyk?= <piotrwlodarczyk89@gmail.com> writes:
> sorry for the late reply, but it was not so easy to get more logs. So what
> we have now is listed below:

> (gdb) bt
> #0  0x0000000000616912 in EvalPlanQualBegin (epqstate=epqstate@entry=0x2ae9520)
> at execMain.c:2735
> #1  0x0000000000616cfb in EvalPlanQual (epqstate=epqstate@entry=0x2ae9520,
> relation=relation@entry=0x7f694c76f838, rti=14,
> inputslot=inputslot@entry=0x31bc408)
> at execMain.c:2454
> #2  0x0000000000638bdf in ExecUpdate (mtstate=mtstate@entry=0x2ae9428,
> tupleid=0x7ffc94e3cf40, oldtuple=0x0, slot=<optimized out>,
> planSlot=0x312fc28, epqstate=epqstate@entry=0x2ae9520,
> estate=estate@entry=0x2aff450,
> canSetTag=true) at nodeModifyTable.c:1387
> #3  0x000000000063914d in ExecModifyTable (pstate=0x2ae9428) at
> nodeModifyTable.c:2223

OK, so that confirms my suspicion that it was an EvalPlanQual issue
(hence, you need concurrent updates on the same row to trigger it).

I believe this might be the same bug we isolated and fixed just a
couple of days ago:

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=87fed2a197abc1397b63ee74b3fa7eb20471fff5

The stack trace looks the same as what Oleksii showed in that thread
for a non-assert build.  As far as I know, that bug only triggers if
a single ModifyTable plan node has to run EPQ checks for more than
one target table --- so, is this query updating a partitioned table
or inheritance tree, and is it plausible that it had to update rows
in more than one sub-table, and that some other transaction(s) were
concurrently updating those same rows?

            regards, tom lane

=?UTF-8?Q?Piotr_W=C5=82odarczyk?= <piotrwlodarczyk89@gmail.com> writes:
> I don't know what is EvalPlanQual but conditions you wrote about are met.

OK, then I'm pretty confident that patch will fix your problem.
It'll be in 12.2 which is due out in two weeks.

            regards, tom lane