Thread: Server crash with parallel workers with Postgres 14.7

Server crash with parallel workers with Postgres 14.7

From
José Lorenzo Urdaneta Rodriguez
Date:
Hi, 

I've been having intermittent server crashes when executing certain queries. I have narrowed the cases to queries that scan large tables, and the most recent cases when the planner uses parallel workers.

I managed to collect a core dump of the crash, here's the result of `bt` using `gdb`:

```
Reading symbols from /usr/lib/postgresql/14/bin/postgres...
Reading symbols from /usr/lib/debug/.build-id/4a/4ff1b11a45a428e502b992679932bc188f92c1.debug...
[New LWP 3008897]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `postgres: 14/kronor: parallel worker for PID 3008825               '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000fffea2ac7a68 in ?? ()
(gdb) bt
#0  0x0000fffea2ac7a68 in ?? ()
#1  0x0000aaaabb378020 in ExecProcNode (node=0xaaaae311d068) at ./build/../src/include/executor/executor.h:257
#2  ExecAppend (pstate=0xaaaae30dd358) at ./build/../src/backend/executor/nodeAppend.c:360
#3  0x0000aaaabb378020 in ExecProcNode (node=0xaaaae30dd358) at ./build/../src/include/executor/executor.h:257
#4  ExecAppend (pstate=0xaaaae30bf258) at ./build/../src/backend/executor/nodeAppend.c:360
#5  0x0000000000000001 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
```

The query that was running was:

```
SELECT
  coalesce(
    json_agg(
      "root"
      ORDER BY
        "root.pg.sent_at" DESC NULLS FIRST
    ),
    '[]'
  ) AS "root"
FROM
  (
    SELECT
      "_root.base"."sent_at" AS "root.pg.sent_at",
      row_to_json(
        (
          SELECT
            "_e"
          FROM
            (
              SELECT
                "_root.base"."webhook_url" AS "webhookUrl",
                ("_root.base"."id"):: text AS "id",
                "_root.base"."payment_request_id" AS "paymentRequestId",
                ("_root.base"."payment_id"):: text AS "paymentId",
                ("_root.base"."purchase_id"):: text AS "purchaseId",
                "_root.base"."refund_id" AS "refundId",
                "_root.base"."event" AS "event",
                "_root.base"."sent_at" AS "sentAt",
                ("_root.base"."merchant_id"):: text AS "merchantId"
            ) AS "_e"
        )
      ) AS "root"
    FROM
      (
        SELECT
          *
        FROM
          "runtime"."successful_webhook_notification"
        WHERE
          (
            (
              (
                "runtime"."successful_webhook_notification"."merchant_id"
              ) = ANY(
                (
                  ('{1,2}')
                ):: bigint array
              )
            )
            AND (
              (
                (
                  "runtime"."successful_webhook_notification"."sent_at"
                ) >= (
                  (now() - interval '3 months'):: timestamptz
                )
              )
              AND (
                (
                  "runtime"."successful_webhook_notification"."sent_at"
                ) <= (
                  (now()):: timestamptz
                )
              )
            )
          )
        ORDER BY
          "sent_at" DESC
      ) as "_root.base"
  ) as b;
```

The plan for this query was:

```
  QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=312222.17..312222.18 rows=1 width=32)
   ->  Gather Merge  (cost=161103.18..270941.51 rows=917348 width=182)
         Workers Planned: 4
         ->  Sort  (cost=160103.12..160676.46 rows=229337 width=182)
               Sort Key: successful_webhook_notification.sent_at DESC
               ->  Parallel Append  (cost=0.00..127744.78 rows=229337 width=182)
                     Subplans Removed: 3
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_02 successful_webhook_notification_73  (cost=0.00..4326.30 rows=13803 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_18 successful_webhook_notification_59  (cost=0.00..4249.98 rows=13595 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_30 successful_webhook_notification_71  (cost=0.00..4114.79 rows=12749 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_04 successful_webhook_notification_75  (cost=0.00..4081.71 rows=13660 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_15 successful_webhook_notification_86  (cost=0.00..3976.48 rows=12533 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_17 successful_webhook_notification_58  (cost=0.00..3941.69 rows=12385 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_17 successful_webhook_notification_88  (cost=0.00..3940.35 rows=12893 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_03 successful_webhook_notification_74  (cost=0.00..3851.78 rows=12654 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_09 successful_webhook_notification_80  (cost=0.00..3745.25 rows=11796 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_16 successful_webhook_notification_57  (cost=0.00..3630.76 rows=10532 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_08 successful_webhook_notification_79  (cost=0.00..3417.41 rows=14992 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_10 successful_webhook_notification_81  (cost=0.00..3395.96 rows=10877 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_23 successful_webhook_notification_64  (cost=0.00..3374.39 rows=10008 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_25 successful_webhook_notification_66  (cost=0.00..3368.65 rows=10221 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_14 successful_webhook_notification_85  (cost=0.00..3366.52 rows=9804 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_24 successful_webhook_notification_65  (cost=0.00..3365.28 rows=10442 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_07 successful_webhook_notification_78  (cost=0.00..3315.31 rows=13174 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_27 successful_webhook_notification_68  (cost=0.00..3191.01 rows=13831 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_16 successful_webhook_notification_87  (cost=0.00..3178.63 rows=13746 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_11 successful_webhook_notification_82  (cost=0.00..3148.94 rows=14139 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_19 successful_webhook_notification_60  (cost=0.00..3147.04 rows=13399 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_28 successful_webhook_notification_69  (cost=0.00..3069.06 rows=13764 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_15 successful_webhook_notification_56  (cost=0.00..2991.49 rows=12028 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_26 successful_webhook_notification_67  (cost=0.00..2874.35 rows=12159 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_20 successful_webhook_notification_61  (cost=0.00..2684.26 rows=11182 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_29 successful_webhook_notification_70  (cost=0.00..2520.51 rows=10041 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_05 successful_webhook_notification_76  (cost=0.00..2481.01 rows=10982 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_12 successful_webhook_notification_83  (cost=0.00..2387.85 rows=10488 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_14 successful_webhook_notification_55  (cost=0.00..2163.93 rows=8416 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_01 successful_webhook_notification_72  (cost=0.00..2021.22 rows=6673 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_06 successful_webhook_notification_77  (cost=0.00..2015.04 rows=7629 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_13 successful_webhook_notification_84  (cost=0.00..1972.15 rows=7432 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_22 successful_webhook_notification_63  (cost=0.00..1957.99 rows=7752 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_13 successful_webhook_notification_54  (cost=0.00..1868.65 rows=6456 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_21 successful_webhook_notification_62  (cost=0.00..1797.46 rows=7130 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_18 successful_webhook_notification_89  (cost=0.00..1607.06 rows=5164 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_12 successful_webhook_notification_53  (cost=0.00..1093.90 rows=3288 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_11 successful_webhook_notification_52  (cost=0.00..940.12 rows=3169 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_19 successful_webhook_notification_90  (cost=0.00..791.62 rows=3252 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_26 successful_webhook_notification_36  (cost=0.00..612.93 rows=1661 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_10 successful_webhook_notification_51  (cost=0.00..603.54 rows=1714 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_08 successful_webhook_notification_49  (cost=0.00..504.01 rows=1496 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_09 successful_webhook_notification_50  (cost=0.00..485.82 rows=1310 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_02 successful_webhook_notification_43  (cost=0.00..478.82 rows=1214 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_27 successful_webhook_notification_37  (cost=0.00..460.90 rows=1397 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_04 successful_webhook_notification_45  (cost=0.00..412.32 rows=1294 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_06 successful_webhook_notification_47  (cost=0.00..412.12 rows=1325 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_25 successful_webhook_notification_35  (cost=0.00..410.00 rows=1072 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_24 successful_webhook_notification_34  (cost=0.00..397.38 rows=1509 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_03 successful_webhook_notification_44  (cost=0.00..391.09 rows=1203 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_05 successful_webhook_notification_46  (cost=0.00..371.12 rows=1154 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_30 successful_webhook_notification_40  (cost=0.00..335.82 rows=1097 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_28 successful_webhook_notification_38  (cost=0.00..329.26 rows=991 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_29 successful_webhook_notification_39  (cost=0.00..327.76 rows=1001 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_07 successful_webhook_notification_48  (cost=0.00..314.76 rows=800 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_01 successful_webhook_notification_42  (cost=0.00..291.68 rows=726 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_23 successful_webhook_notification_33  (cost=0.00..287.85 rows=905 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_31 successful_webhook_notification_41  (cost=0.00..273.99 rows=827 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_22 successful_webhook_notification_32  (cost=0.00..260.01 rows=1028 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_02_27 successful_webhook_notification_9  (cost=0.00..223.87 rows=1514 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_13 successful_webhook_notification_23  (cost=0.00..206.63 rows=1385 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_02_26 successful_webhook_notification_8  (cost=0.00..203.18 rows=1367 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_14 successful_webhook_notification_24  (cost=0.00..201.91 rows=1356 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_02_28 successful_webhook_notification_10  (cost=0.00..201.78 rows=1351 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_06 successful_webhook_notification_16  (cost=0.00..194.76 rows=1311 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_02_20 successful_webhook_notification_2  (cost=0.00..194.37 rows=1295 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_16 successful_webhook_notification_26  (cost=0.00..192.31 rows=1292 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_21 successful_webhook_notification_31  (cost=0.00..187.38 rows=1075 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_20 successful_webhook_notification_30  (cost=0.00..182.93 rows=961 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_10 successful_webhook_notification_20  (cost=0.00..178.01 rows=1201 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_12 successful_webhook_notification_22  (cost=0.00..177.21 rows=1208 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_09 successful_webhook_notification_19  (cost=0.00..176.91 rows=1196 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_15 successful_webhook_notification_25  (cost=0.00..175.94 rows=1198 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_19 successful_webhook_notification_29  (cost=0.00..174.35 rows=875 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_03 successful_webhook_notification_13  (cost=0.00..168.50 rows=1140 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_07 successful_webhook_notification_17  (cost=0.00..165.03 rows=1121 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_02_24 successful_webhook_notification_6  (cost=0.00..162.69 rows=1108 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_02 successful_webhook_notification_12  (cost=0.00..162.53 rows=1101 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_05 successful_webhook_notification_15  (cost=0.00..162.18 rows=1087 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_02_23 successful_webhook_notification_5  (cost=0.00..161.28 rows=1091 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_17 successful_webhook_notification_27  (cost=0.00..160.22 rows=993 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_01 successful_webhook_notification_11  (cost=0.00..157.82 rows=1073 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_08 successful_webhook_notification_18  (cost=0.00..157.68 rows=1067 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_02_22 successful_webhook_notification_4  (cost=0.00..141.87 rows=955 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_02_19 successful_webhook_notification_1  (cost=0.00..141.84 rows=854 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_04 successful_webhook_notification_14  (cost=0.00..138.49 rows=939 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_02_21 successful_webhook_notification_3  (cost=0.00..135.87 rows=915 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_02_25 successful_webhook_notification_7  (cost=0.00..131.24 rows=889 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_11 successful_webhook_notification_21  (cost=0.00..127.60 rows=864 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_18 successful_webhook_notification_28  (cost=0.00..115.66 rows=605 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
   SubPlan 1
     ->  Result  (cost=0.00..0.03 rows=1 width=32)
 JIT:
   Functions: 375
   Options: Inlining false, Optimization false, Expressions true, Deforming true
```

Operating System: Ubuntu 20
Architecture: aarch64
Server version: 14.7

Is there any other information I can provide?

Re: Server crash with parallel workers with Postgres 14.7

From
José Lorenzo Urdaneta Rodriguez
Date:
I just wanted to confirm this was the right place to report the issue. Can anyone confirm, please?

On Fri, 19 May 2023 at 11:14, José Lorenzo Urdaneta Rodriguez <lorenzo@kronor.io> wrote:
Hi, 

I've been having intermittent server crashes when executing certain queries. I have narrowed the cases to queries that scan large tables, and the most recent cases when the planner uses parallel workers.

I managed to collect a core dump of the crash, here's the result of `bt` using `gdb`:

```
Reading symbols from /usr/lib/postgresql/14/bin/postgres...
Reading symbols from /usr/lib/debug/.build-id/4a/4ff1b11a45a428e502b992679932bc188f92c1.debug...
[New LWP 3008897]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `postgres: 14/kronor: parallel worker for PID 3008825               '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000fffea2ac7a68 in ?? ()
(gdb) bt
#0  0x0000fffea2ac7a68 in ?? ()
#1  0x0000aaaabb378020 in ExecProcNode (node=0xaaaae311d068) at ./build/../src/include/executor/executor.h:257
#2  ExecAppend (pstate=0xaaaae30dd358) at ./build/../src/backend/executor/nodeAppend.c:360
#3  0x0000aaaabb378020 in ExecProcNode (node=0xaaaae30dd358) at ./build/../src/include/executor/executor.h:257
#4  ExecAppend (pstate=0xaaaae30bf258) at ./build/../src/backend/executor/nodeAppend.c:360
#5  0x0000000000000001 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
```

The query that was running was:

```
SELECT
  coalesce(
    json_agg(
      "root"
      ORDER BY
        "root.pg.sent_at" DESC NULLS FIRST
    ),
    '[]'
  ) AS "root"
FROM
  (
    SELECT
      "_root.base"."sent_at" AS "root.pg.sent_at",
      row_to_json(
        (
          SELECT
            "_e"
          FROM
            (
              SELECT
                "_root.base"."webhook_url" AS "webhookUrl",
                ("_root.base"."id"):: text AS "id",
                "_root.base"."payment_request_id" AS "paymentRequestId",
                ("_root.base"."payment_id"):: text AS "paymentId",
                ("_root.base"."purchase_id"):: text AS "purchaseId",
                "_root.base"."refund_id" AS "refundId",
                "_root.base"."event" AS "event",
                "_root.base"."sent_at" AS "sentAt",
                ("_root.base"."merchant_id"):: text AS "merchantId"
            ) AS "_e"
        )
      ) AS "root"
    FROM
      (
        SELECT
          *
        FROM
          "runtime"."successful_webhook_notification"
        WHERE
          (
            (
              (
                "runtime"."successful_webhook_notification"."merchant_id"
              ) = ANY(
                (
                  ('{1,2}')
                ):: bigint array
              )
            )
            AND (
              (
                (
                  "runtime"."successful_webhook_notification"."sent_at"
                ) >= (
                  (now() - interval '3 months'):: timestamptz
                )
              )
              AND (
                (
                  "runtime"."successful_webhook_notification"."sent_at"
                ) <= (
                  (now()):: timestamptz
                )
              )
            )
          )
        ORDER BY
          "sent_at" DESC
      ) as "_root.base"
  ) as b;
```

The plan for this query was:

```
  QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=312222.17..312222.18 rows=1 width=32)
   ->  Gather Merge  (cost=161103.18..270941.51 rows=917348 width=182)
         Workers Planned: 4
         ->  Sort  (cost=160103.12..160676.46 rows=229337 width=182)
               Sort Key: successful_webhook_notification.sent_at DESC
               ->  Parallel Append  (cost=0.00..127744.78 rows=229337 width=182)
                     Subplans Removed: 3
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_02 successful_webhook_notification_73  (cost=0.00..4326.30 rows=13803 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_18 successful_webhook_notification_59  (cost=0.00..4249.98 rows=13595 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_30 successful_webhook_notification_71  (cost=0.00..4114.79 rows=12749 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_04 successful_webhook_notification_75  (cost=0.00..4081.71 rows=13660 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_15 successful_webhook_notification_86  (cost=0.00..3976.48 rows=12533 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_17 successful_webhook_notification_58  (cost=0.00..3941.69 rows=12385 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_17 successful_webhook_notification_88  (cost=0.00..3940.35 rows=12893 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_03 successful_webhook_notification_74  (cost=0.00..3851.78 rows=12654 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_09 successful_webhook_notification_80  (cost=0.00..3745.25 rows=11796 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_16 successful_webhook_notification_57  (cost=0.00..3630.76 rows=10532 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_08 successful_webhook_notification_79  (cost=0.00..3417.41 rows=14992 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_10 successful_webhook_notification_81  (cost=0.00..3395.96 rows=10877 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_23 successful_webhook_notification_64  (cost=0.00..3374.39 rows=10008 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_25 successful_webhook_notification_66  (cost=0.00..3368.65 rows=10221 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_14 successful_webhook_notification_85  (cost=0.00..3366.52 rows=9804 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_24 successful_webhook_notification_65  (cost=0.00..3365.28 rows=10442 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_07 successful_webhook_notification_78  (cost=0.00..3315.31 rows=13174 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_27 successful_webhook_notification_68  (cost=0.00..3191.01 rows=13831 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_16 successful_webhook_notification_87  (cost=0.00..3178.63 rows=13746 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_11 successful_webhook_notification_82  (cost=0.00..3148.94 rows=14139 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_19 successful_webhook_notification_60  (cost=0.00..3147.04 rows=13399 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_28 successful_webhook_notification_69  (cost=0.00..3069.06 rows=13764 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_15 successful_webhook_notification_56  (cost=0.00..2991.49 rows=12028 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_26 successful_webhook_notification_67  (cost=0.00..2874.35 rows=12159 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_20 successful_webhook_notification_61  (cost=0.00..2684.26 rows=11182 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_29 successful_webhook_notification_70  (cost=0.00..2520.51 rows=10041 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_05 successful_webhook_notification_76  (cost=0.00..2481.01 rows=10982 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_12 successful_webhook_notification_83  (cost=0.00..2387.85 rows=10488 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_14 successful_webhook_notification_55  (cost=0.00..2163.93 rows=8416 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_01 successful_webhook_notification_72  (cost=0.00..2021.22 rows=6673 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_06 successful_webhook_notification_77  (cost=0.00..2015.04 rows=7629 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_13 successful_webhook_notification_84  (cost=0.00..1972.15 rows=7432 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_22 successful_webhook_notification_63  (cost=0.00..1957.99 rows=7752 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_13 successful_webhook_notification_54  (cost=0.00..1868.65 rows=6456 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_21 successful_webhook_notification_62  (cost=0.00..1797.46 rows=7130 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_18 successful_webhook_notification_89  (cost=0.00..1607.06 rows=5164 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_12 successful_webhook_notification_53  (cost=0.00..1093.90 rows=3288 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_11 successful_webhook_notification_52  (cost=0.00..940.12 rows=3169 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_05_19 successful_webhook_notification_90  (cost=0.00..791.62 rows=3252 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_26 successful_webhook_notification_36  (cost=0.00..612.93 rows=1661 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_10 successful_webhook_notification_51  (cost=0.00..603.54 rows=1714 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_08 successful_webhook_notification_49  (cost=0.00..504.01 rows=1496 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_09 successful_webhook_notification_50  (cost=0.00..485.82 rows=1310 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_02 successful_webhook_notification_43  (cost=0.00..478.82 rows=1214 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_27 successful_webhook_notification_37  (cost=0.00..460.90 rows=1397 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_04 successful_webhook_notification_45  (cost=0.00..412.32 rows=1294 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_06 successful_webhook_notification_47  (cost=0.00..412.12 rows=1325 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_25 successful_webhook_notification_35  (cost=0.00..410.00 rows=1072 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_24 successful_webhook_notification_34  (cost=0.00..397.38 rows=1509 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_03 successful_webhook_notification_44  (cost=0.00..391.09 rows=1203 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_05 successful_webhook_notification_46  (cost=0.00..371.12 rows=1154 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_30 successful_webhook_notification_40  (cost=0.00..335.82 rows=1097 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_28 successful_webhook_notification_38  (cost=0.00..329.26 rows=991 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_29 successful_webhook_notification_39  (cost=0.00..327.76 rows=1001 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_07 successful_webhook_notification_48  (cost=0.00..314.76 rows=800 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_04_01 successful_webhook_notification_42  (cost=0.00..291.68 rows=726 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_23 successful_webhook_notification_33  (cost=0.00..287.85 rows=905 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_31 successful_webhook_notification_41  (cost=0.00..273.99 rows=827 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_22 successful_webhook_notification_32  (cost=0.00..260.01 rows=1028 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_02_27 successful_webhook_notification_9  (cost=0.00..223.87 rows=1514 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_13 successful_webhook_notification_23  (cost=0.00..206.63 rows=1385 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_02_26 successful_webhook_notification_8  (cost=0.00..203.18 rows=1367 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_14 successful_webhook_notification_24  (cost=0.00..201.91 rows=1356 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_02_28 successful_webhook_notification_10  (cost=0.00..201.78 rows=1351 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_06 successful_webhook_notification_16  (cost=0.00..194.76 rows=1311 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_02_20 successful_webhook_notification_2  (cost=0.00..194.37 rows=1295 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_16 successful_webhook_notification_26  (cost=0.00..192.31 rows=1292 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_21 successful_webhook_notification_31  (cost=0.00..187.38 rows=1075 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_20 successful_webhook_notification_30  (cost=0.00..182.93 rows=961 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_10 successful_webhook_notification_20  (cost=0.00..178.01 rows=1201 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_12 successful_webhook_notification_22  (cost=0.00..177.21 rows=1208 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_09 successful_webhook_notification_19  (cost=0.00..176.91 rows=1196 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_15 successful_webhook_notification_25  (cost=0.00..175.94 rows=1198 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_19 successful_webhook_notification_29  (cost=0.00..174.35 rows=875 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_03 successful_webhook_notification_13  (cost=0.00..168.50 rows=1140 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_07 successful_webhook_notification_17  (cost=0.00..165.03 rows=1121 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_02_24 successful_webhook_notification_6  (cost=0.00..162.69 rows=1108 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_02 successful_webhook_notification_12  (cost=0.00..162.53 rows=1101 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_05 successful_webhook_notification_15  (cost=0.00..162.18 rows=1087 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_02_23 successful_webhook_notification_5  (cost=0.00..161.28 rows=1091 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_17 successful_webhook_notification_27  (cost=0.00..160.22 rows=993 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_01 successful_webhook_notification_11  (cost=0.00..157.82 rows=1073 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_08 successful_webhook_notification_18  (cost=0.00..157.68 rows=1067 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_02_22 successful_webhook_notification_4  (cost=0.00..141.87 rows=955 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_02_19 successful_webhook_notification_1  (cost=0.00..141.84 rows=854 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_04 successful_webhook_notification_14  (cost=0.00..138.49 rows=939 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_02_21 successful_webhook_notification_3  (cost=0.00..135.87 rows=915 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_02_25 successful_webhook_notification_7  (cost=0.00..131.24 rows=889 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_11 successful_webhook_notification_21  (cost=0.00..127.60 rows=864 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
                     ->  Parallel Seq Scan on successful_webhook_notification_p2023_03_18 successful_webhook_notification_28  (cost=0.00..115.66 rows=605 width=182)
                           Filter: ((merchant_id = ANY ('{1,2}'::bigint[])) AND (sent_at <= now()) AND (sent_at >= (now() - '3 mons'::interval)))
   SubPlan 1
     ->  Result  (cost=0.00..0.03 rows=1 width=32)
 JIT:
   Functions: 375
   Options: Inlining false, Optimization false, Expressions true, Deforming true
```

Operating System: Ubuntu 20
Architecture: aarch64
Server version: 14.7

Is there any other information I can provide?

Re: Server crash with parallel workers with Postgres 14.7

From
Jaime Casanova
Date:
On Mon, May 29, 2023 at 10:38 AM José Lorenzo Urdaneta Rodriguez
<lorenzo@kronor.io> wrote:
>
> I just wanted to confirm this was the right place to report the issue. Can anyone confirm, please?
>

yes, this is the right place to report... only there is no guaranted
SLA and because this report is not that useful (read below for
details) that makes a lot of people not follow

> On Fri, 19 May 2023 at 11:14, José Lorenzo Urdaneta Rodriguez <lorenzo@kronor.io> wrote:
>>
>> Hi,
>>
>> I've been having intermittent server crashes when executing certain queries. I have narrowed the cases to queries
thatscan large tables, and the most recent cases when the planner uses parallel workers. 
>>

intermittent means is not reproducible all times? I mean, just
executing this query does not cause the crash?

>> I managed to collect a core dump of the crash, here's the result of `bt` using `gdb`:
>>
>> ```
>> Reading symbols from /usr/lib/postgresql/14/bin/postgres...
>> Reading symbols from /usr/lib/debug/.build-id/4a/4ff1b11a45a428e502b992679932bc188f92c1.debug...
>> [New LWP 3008897]
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
>> Core was generated by `postgres: 14/kronor: parallel worker for PID 3008825               '.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0  0x0000fffea2ac7a68 in ?? ()
>> (gdb) bt
>> #0  0x0000fffea2ac7a68 in ?? ()
>> #1  0x0000aaaabb378020 in ExecProcNode (node=0xaaaae311d068) at ./build/../src/include/executor/executor.h:257
>> #2  ExecAppend (pstate=0xaaaae30dd358) at ./build/../src/backend/executor/nodeAppend.c:360
>> #3  0x0000aaaabb378020 in ExecProcNode (node=0xaaaae30dd358) at ./build/../src/include/executor/executor.h:257
>> #4  ExecAppend (pstate=0xaaaae30bf258) at ./build/../src/backend/executor/nodeAppend.c:360
>> #5  0x0000000000000001 in ?? ()
>> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
>> ```
>>

this backtrace doesn't have all debug symbols, did you install the
postgresql-14-dbgsym module?
without the names of the functions we don't really know what is happening.

also, have you installed any extensions? you can execute "\dx" on psql
to see what extensions are installed (remember that extensions are
installed by database so executing that commando on only one database
is not enough).

>> The query that was running was:

- a big query goes here - the query itself is not useful if you don't
provide the table structues and a minimal amount of data (fake data)
to make the problem appear

>> The plan for this query was:
>>
- A typical plan for a partitioned table -

>>  JIT:
>>    Functions: 375
>>    Options: Inlining false, Optimization false, Expressions true, Deforming true
>> ```
>>

the backtrace says this is a segmentation fault, but anyway I will
suggest deactivate JIT before the query: just "SET jit TO off;" should
be enough
and try to cause the problem again, JIT is known to have a leak memory
problem (which is not consistent with a segmentation fault, but who
knows)


>> Operating System: Ubuntu 20
>> Architecture: aarch64
>> Server version: 14.7
>>

try to update to v14.8 which has some fixes on it



--
Jaime Casanova
Consultores de PostgreSQL
SYSTEMGUARDS S.A.



Re: Server crash with parallel workers with Postgres 14.7

From
Michael Paquier
Date:
On Mon, May 29, 2023 at 11:56:09PM -0500, Jaime Casanova wrote:
> try to update to v14.8 which has some fixes on it

Agreed about re-checking with 14.8.  I'd be happy to look at the
report if there is a way to diagnose it, but there is no such evidence
at hand.  A self-contained test case would be the best thing, of
course, if the problem still persists on your hand.

You may want to be careful in not providing production-sensitive data
on this mailing list as this is a public space.
--
Michael

Attachment