Re: SegFault on 9.6.14 - Mailing list pgsql-hackers

Tomas Vondra <tomas.vondra@2ndquadrant.com> writes:

> On Mon, Jul 15, 2019 at 07:22:55PM -0500, Jerry Sievers wrote:
>
>>Tomas Vondra <tomas.vondra@2ndquadrant.com> writes:
>>
>>> On Mon, Jul 15, 2019 at 06:48:05PM -0500, Jerry Sievers wrote:
>>>
>>>>Greetings Hackers.
>>>>
>>>>We have a reproduceable case of $subject that issues a backtrace such as
>>>>seen below.
>>>>
>>>>The query that I'd prefer to sanitize before sending is <30 lines of at
>>>>a glance, not terribly complex logic.
>>>>
>>>>It nonetheless dies hard after a few seconds of running and as expected,
>>>>results in an automatic all-backend restart.
>>>>
>>>>Please advise on how to proceed.  Thanks!
>>>>
>>>>bt
>>>>#0  initscan (scan=scan@entry=0x55d7a7daa0b0, key=0x0, keep_startblock=keep_startblock@entry=1 '\001')
>>>>    at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/access/heap/heapam.c:233
>>>>#1  0x000055d7a72fa8d0 in heap_rescan (scan=0x55d7a7daa0b0, key=key@entry=0x0) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/access/heap/heapam.c:1529
>>>>#2  0x000055d7a7451fef in ExecReScanSeqScan (node=node@entry=0x55d7a7d85100) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeSeqscan.c:280
>>>>#3  0x000055d7a742d36e in ExecReScan (node=0x55d7a7d85100) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execAmi.c:158
>>>>#4  0x000055d7a7445d38 in ExecReScanGather (node=node@entry=0x55d7a7d84d30) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeGather.c:475
>>>>#5  0x000055d7a742d255 in ExecReScan (node=0x55d7a7d84d30) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execAmi.c:166
>>>>#6  0x000055d7a7448673 in ExecReScanHashJoin (node=node@entry=0x55d7a7d84110) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeHashjoin.c:1019
>>>>#7  0x000055d7a742d29e in ExecReScan (node=node@entry=0x55d7a7d84110) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execAmi.c:226
>>>><about 30 lines omitted>
>>>>
>>>
>>> Hmmm, that means it's crashing here:
>>>
>>>    if (scan->rs_parallel != NULL)
>>>        scan->rs_nblocks = scan->rs_parallel->phs_nblocks;     <--- here
>>>    else
>>>        scan->rs_nblocks = RelationGetNumberOfBlocks(scan->rs_rd);
>>>
>>> But clearly, scan is valid (otherwise it'd crash on the if condition),
>>> and scan->rs_parallel must me non-NULL. Which probably means the pointer
>>> is (no longer) valid.
>>>
>>> Could it be that the rs_parallel DSM disappears on rescan, or something
>>> like that?
>>
>>No clue but something I just tried was to disable parallelism by setting
>>max_parallel_workers_per_gather to 0 and however the query has not
>>finished after a few minutes, there is no crash.
>>
>
> That might be a hint my rough analysis was somewhat correct. The
> question is whether the non-parallel plan does the same thing. Maybe it
> picks a plan that does not require rescans, or something like that.
>
>>Please advise.
>>
>
> It would be useful to see (a) exacution plan of the query, (b) full
> backtrace and (c) a bit of context for the place where it crashed.
>
> Something like (in gdb):
>
>    bt full
>    list
>    p *scan

The p *scan did nothing unless I ran it first however my gdb $foo isn't
strong presently.

I'll need to sanitize the explain output but can do so ASAP and send it
along.

Thx!


$ gdb /usr/lib/postgresql/9.6/bin/postgres core
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/postgresql/9.6/bin/postgres...Reading symbols from
/usr/lib/debug/.build-id/04/6f55a5ce6ce05064edfc8feee61c6cb039d296.debug...done.
done.
[New LWP 31654]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `postgres: foo_eis_segfault: jsievers staging 10.220.22.26(57948) SELECT       '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  initscan (scan=scan@entry=0x55d7a7daa0b0, key=0x0, keep_startblock=keep_startblock@entry=1 '\001')
    at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/access/heap/heapam.c:233
233    /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/access/heap/heapam.c: No such file or
directory.
(gdb) p *scan
$1 = {rs_rd = 0x7fa6c6935a08, rs_snapshot = 0x55d7a7c2e630, rs_nkeys = 0, rs_key = 0x0, rs_bitmapscan = 0 '\000',
rs_samplescan= 0 '\000', rs_pageatatime = 1 '\001', 
 
  rs_allow_strat = 1 '\001', rs_allow_sync = 1 '\001', rs_temp_snap = 1 '\001', rs_nblocks = 198714, rs_startblock =
1920300133,rs_numblocks = 4294967295, rs_strategy = 0x55d7a7daa6a0, 
 
  rs_syncscan = 1 '\001', rs_inited = 0 '\000', rs_ctup = {t_len = 114, t_self = {ip_blkid = {bi_hi = 0, bi_lo =
62879},ip_posid = 77}, t_tableOid = 994804890, t_data = 0x0}, 
 
  rs_cblock = 4294967295, rs_cbuf = 0, rs_parallel = 0x7fa673a54108, rs_cindex = 76, rs_ntuples = 77, rs_vistuples =
{1,2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 
 
    21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49,
50,51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 
 
    67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 28255, 25711, 24421, 25705, 12576, 8247, 29754, 29281,
25959,27764, 29545, 8308, 31528, 16724, 18258, 21573, 20037, 21076, 
 
    8281, 25914, 28792, 8306, 22139, 21057, 14880, 24950, 28274, 8303, 12337, 14880, 24950, 24946, 29812, 28526, 13088,
14880,24950, 29810, 28793, 8293, 13106, 14880, 24950, 29810, 28793, 
 
    28525, 8292, 12589, 14880, 24950, 25458, 27759, 26988, 8292, 8240, 30266, 29281, 25964, 25974, 29548, 28789, 12320,
14880,24950, 28274, 28527, 25708, 12576, 8240, 30266, 29281, 24943, 
 
    29812, 28526, 13088, 14880, 28524, 24931, 26996, 28271, 13344, 13110, 8317, 29242, 29541, 28526, 12576, 14880,
25970,28275, 28001, 8293, 15932, 14880, 25970, 29555, 29295, 26484, 
 
    28530, 28789, 25970, 8294, 8240, 29242, 29541, 29295, 26473, 25204, 8300, 8240, 29242, 29541, 29295, 26473, 28515,
8300,8240, 29242, 29541...}}
 
(gdb) bt full
#0  initscan (scan=scan@entry=0x55d7a7daa0b0, key=0x0, keep_startblock=keep_startblock@entry=1 '\001')
    at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/access/heap/heapam.c:233
        allow_strat = <optimized out>
        allow_sync = <optimized out>
#1  0x000055d7a72fa8d0 in heap_rescan (scan=0x55d7a7daa0b0, key=key@entry=0x0) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/access/heap/heapam.c:1529
        __func__ = "heap_rescan"
#2  0x000055d7a7451fef in ExecReScanSeqScan (node=node@entry=0x55d7a7d85100) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeSeqscan.c:280
        scan = <optimized out>
#3  0x000055d7a742d36e in ExecReScan (node=0x55d7a7d85100) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execAmi.c:158
        __func__ = "ExecReScan"
#4  0x000055d7a7445d38 in ExecReScanGather (node=node@entry=0x55d7a7d84d30) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeGather.c:475
No locals.
#5  0x000055d7a742d255 in ExecReScan (node=0x55d7a7d84d30) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execAmi.c:166
        __func__ = "ExecReScan"
#6  0x000055d7a7448673 in ExecReScanHashJoin (node=node@entry=0x55d7a7d84110) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeHashjoin.c:1019
No locals.
#7  0x000055d7a742d29e in ExecReScan (node=node@entry=0x55d7a7d84110) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execAmi.c:226
        __func__ = "ExecReScan"
#8  0x000055d7a7433ce7 in ExecProcNode (node=node@entry=0x55d7a7d84110) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execProcnode.c:381
        result = <optimized out>
        __func__ = "ExecProcNode"
#9  0x000055d7a7452989 in ExecSort (node=node@entry=0x55d7a7d83ea0) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeSort.c:103
        plannode = <optimized out>
        outerNode = 0x55d7a7d84110
        tupDesc = <optimized out>
        estate = 0x55d7a7d5fee8
        dir = ForwardScanDirection
        tuplesortstate = 0x55d7a7dd2448
        slot = <optimized out>
#10 0x000055d7a7433de8 in ExecProcNode (node=0x55d7a7d83ea0) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execProcnode.c:495
        result = <optimized out>
        __func__ = "ExecProcNode"
#11 0x000055d7a743ffe9 in fetch_input_tuple (aggstate=aggstate@entry=0x55d7a7d83528) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeAgg.c:598
        slot = <optimized out>
#12 0x000055d7a7441bb3 in agg_retrieve_direct (aggstate=0x55d7a7d83528) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeAgg.c:2078
        econtext = 0x55d7a7d838b0
        pergroup = 0x55d7a7d8e758
        firstSlot = 0x55d7a7d83960
        numGroupingSets = 1
        node = 0x7fa6c68a5da8
        tmpcontext = 0x55d7a7d83750
        peragg = 0x55d7a7d8d6b8
        outerslot = <optimized out>
        nextSetSize = <optimized out>
        result = <optimized out>
        hasGroupingSets = 0 '\000'
        currentSet = <optimized out>
        numReset = 1
        i = <optimized out>
---Type <return> to continue, or q <return> to quit---
#13 ExecAgg (node=node@entry=0x55d7a7d83528) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeAgg.c:1903
        result = <optimized out>
#14 0x000055d7a7433dc8 in ExecProcNode (node=node@entry=0x55d7a7d83528) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execProcnode.c:503
        result = <optimized out>
        __func__ = "ExecProcNode"
#15 0x000055d7a744af74 in ExecLimit (node=node@entry=0x55d7a7d83288) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeLimit.c:91
        direction = ForwardScanDirection
        slot = <optimized out>
        outerPlan = 0x55d7a7d83528
        __func__ = "ExecLimit"
#16 0x000055d7a7433d28 in ExecProcNode (node=node@entry=0x55d7a7d83288) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execProcnode.c:531
        result = <optimized out>
        __func__ = "ExecProcNode"
#17 0x000055d7a744ff69 in ExecNestLoop (node=node@entry=0x55d7a7d60cd0) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeNestloop.c:174
        nl = 0x7fa6c68a6048
        innerPlan = 0x55d7a7d83288
        outerPlan = 0x55d7a7d610c0
        outerTupleSlot = <optimized out>
        innerTupleSlot = <optimized out>
        joinqual = 0x0
        otherqual = 0x0
        econtext = 0x55d7a7d60de0
        lc = <optimized out>
#18 0x000055d7a7433e28 in ExecProcNode (node=node@entry=0x55d7a7d60cd0) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execProcnode.c:476
        result = <optimized out>
        __func__ = "ExecProcNode"
#19 0x000055d7a7452989 in ExecSort (node=node@entry=0x55d7a7d60a60) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeSort.c:103
        plannode = <optimized out>
        outerNode = 0x55d7a7d60cd0
        tupDesc = <optimized out>
        estate = 0x55d7a7d5fee8
        dir = ForwardScanDirection
        tuplesortstate = 0x55d7a7d98398
        slot = <optimized out>
#20 0x000055d7a7433de8 in ExecProcNode (node=0x55d7a7d60a60) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execProcnode.c:495
        result = <optimized out>
        __func__ = "ExecProcNode"
#21 0x000055d7a743ffe9 in fetch_input_tuple (aggstate=aggstate@entry=0x55d7a7d60088) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeAgg.c:598
        slot = <optimized out>
#22 0x000055d7a7441bb3 in agg_retrieve_direct (aggstate=0x55d7a7d60088) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeAgg.c:2078
        econtext = 0x55d7a7d60440
        pergroup = 0x55d7a7d91230
        firstSlot = 0x55d7a7d604f0
        numGroupingSets = 1
        node = 0x7fa6c68a6328
        tmpcontext = 0x55d7a7d602b0
        peragg = 0x55d7a7d90190
        outerslot = <optimized out>
        nextSetSize = <optimized out>
---Type <return> to continue, or q <return> to quit---
        result = <optimized out>
        hasGroupingSets = 0 '\000'
        currentSet = <optimized out>
        numReset = 1
        i = <optimized out>
#23 ExecAgg (node=node@entry=0x55d7a7d60088) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeAgg.c:1903
        result = <optimized out>
#24 0x000055d7a7433dc8 in ExecProcNode (node=node@entry=0x55d7a7d60088) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execProcnode.c:503
        result = <optimized out>
        __func__ = "ExecProcNode"
#25 0x000055d7a742ff2e in ExecutePlan (dest=0x7fa673a96308, direction=<optimized out>, numberTuples=0,
sendTuples=<optimizedout>, operation=CMD_SELECT, use_parallel_mode=<optimized out>, 
 
    planstate=0x55d7a7d60088, estate=0x55d7a7d5fee8) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execMain.c:1567
        slot = <optimized out>
        current_tuple_count = 0
#26 standard_ExecutorRun (queryDesc=0x55d7a7d54718, direction=<optimized out>, count=0) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execMain.c:339
        estate = 0x55d7a7d5fee8
        operation = CMD_SELECT
        dest = 0x7fa673a96308
        sendTuples = <optimized out>
#27 0x00007fa6c7027515 in explain_ExecutorRun (queryDesc=0x55d7a7d54718, direction=ForwardScanDirection, count=0)
    at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../contrib/auto_explain/auto_explain.c:281
        save_exception_stack = 0x7fff4aeeaa80
        save_context_stack = 0x0
        local_sigjmp_buf = {{__jmpbuf = {94384722106264, 8229023444991490729, 0, 94384722102040, 0, 1,
8229023444890827433,8250672449167702697}, __mask_was_saved = 0, __saved_mask = {
 
              __val = {94384721739856, 140734450543072, 94384714940022, 140354273004312, 140354273004312,
140734450543104,94384714691234, 2, 2, 140734450543200, 94384711690034, 2, 
 
                3462443396, 8388608, 3547611511646930944, 140734450543200}}}}
#28 0x00007fa6c6e1fdb0 in pgss_ExecutorRun (queryDesc=0x55d7a7d54718, direction=ForwardScanDirection, count=0)
    at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../contrib/pg_stat_statements/pg_stat_statements.c:875
        save_exception_stack = 0x7fff4aeeac20
        save_context_stack = 0x0
        local_sigjmp_buf = {{__jmpbuf = {94384722106264, 8229023444960033449, 0, 94384722102040, 0, 1,
8229023444993587881,8250670555334589097}, __mask_was_saved = 0, __saved_mask = {
 
              __val = {4294967296, 140354272256808, 94384714928429, 16, 94384719269552, 24, 94384720895528,
94384722102040,0, 140734450543408, 94384714928429, 94384722106264, 
 
                94384720895528, 140734450543440, 94384714994982, 94384722106264}}}}
#29 0x000055d7a7553167 in PortalRunSelect (portal=portal@entry=0x55d7a7d55798, forward=forward@entry=1 '\001', count=0,
count@entry=9223372036854775807,dest=dest@entry=0x7fa673a96308) 
    at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/tcop/pquery.c:948
        queryDesc = 0x55d7a7d54718
        direction = <optimized out>
        nprocessed = <optimized out>
        __func__ = "PortalRunSelect"
#30 0x000055d7a75547a0 in PortalRun (portal=portal@entry=0x55d7a7d55798, count=count@entry=9223372036854775807,
isTopLevel=isTopLevel@entry=1'\001', dest=dest@entry=0x7fa673a96308, 
 
    altdest=altdest@entry=0x7fa673a96308, completionTag=completionTag@entry=0x7fff4aeeb050 "") at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/tcop/pquery.c:789
        save_exception_stack = 0x7fff4aeeaf00
        save_context_stack = 0x0
        local_sigjmp_buf = {{__jmpbuf = {94384721085312, 8229023445033433769, 94384722106264, 140352881779464,
94384721085584,2, 8229023444955839145, 2765920793019169449}, 
 
            __mask_was_saved = 0, __saved_mask = {__val = {0, 12099560782865280144, 0, 8, 8, 140734450544226, 1, 88,
94384722106264,94384715935530, 94384721085584, 140734450543840, 
 
                94384714930017, 2, 94384722106264, 140734450543872}}}}
        result = <optimized out>
        nprocessed = <optimized out>
        saveTopTransactionResourceOwner = 0x55d7a7c118e8
---Type <return> to continue, or q <return> to quit---
        saveTopTransactionContext = 0x55d7a7c10eb8
        saveActivePortal = 0x0
        saveResourceOwner = 0x55d7a7c118e8
        savePortalContext = 0x0
        saveMemoryContext = 0x55d7a7c10eb8
        __func__ = "PortalRun"
#31 0x000055d7a75512d6 in exec_simple_query (
    query_string=0x55d7a7ce6b38 "select v.account_id, COUNT(cnt.clicks), te.description,\nl.product_id\nfrom
nbox_nc_ah.tracking_eventste\njoin nbox_nc_ah.page_views pv on pv.page_view_id = te.page_view_id\njoin
nbox_nc_ah.visitsv on v"...) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/tcop/postgres.c:1109
        parsetree = 0x55d7a7c5c380
        portal = 0x55d7a7d55798
        snapshot_set = <optimized out>
        commandTag = <optimized out>
        completionTag =
"\000\370\253\247\327U\000\000\240F\252\247\327U\000\000\200\260\356J\377\177\000\000\215\326g\247\327U\000\000\300\260\356J\377\177\000\000d\261\356J\377\177\000\000\240\260\356J\377\177\000\000v\031F\247\327U\000"
        querytree_list = <optimized out>
        plantree_list = 0x7fa673a962d8
        receiver = 0x7fa673a96308
        format = 0
        dest = DestRemote
        parsetree_list = 0x55d7a7c5c4b0
        save_log_statement_stats = 0 '\000'
        was_logged = 0 '\000'
        msec_str = "\020\261\356J\377\177\000\000(\002", '\000' <repeats 14 times>, "\340?\256\247\327U\000"
        parsetree_item = 0x55d7a7c5c490
        isTopLevel = 1 '\001'
#32 PostgresMain (argc=<optimized out>, argv=argv@entry=0x55d7a7c56830, dbname=0x55d7a7c11b88 "staging",
username=<optimizedout>)
 
    at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/tcop/postgres.c:4101
        query_string = 0x55d7a7ce6b38 "select v.account_id, COUNT(cnt.clicks), te.description,\nl.product_id\nfrom
nbox_nc_ah.tracking_eventste\njoin nbox_nc_ah.page_views pv on pv.page_view_id = te.page_view_id\njoin
nbox_nc_ah.visitsv on v"...
 
        firstchar = -1479190632
        input_message = {
          data = 0x55d7a7ce6b38 "select v.account_id, COUNT(cnt.clicks), te.description,\nl.product_id\nfrom
nbox_nc_ah.tracking_eventste\njoin nbox_nc_ah.page_views pv on pv.page_view_id = te.page_view_id\njoin
nbox_nc_ah.visitsv on v"..., len = 1042, maxlen = 2048, cursor = 1042}
 
        local_sigjmp_buf = {{__jmpbuf = {140734450544288, 8229023445169748649, 94384721061936, 1, 94384721061720,
94384721052928,8229023445035530921, 2765920790734322345}, 
 
            __mask_was_saved = 1, __saved_mask = {__val = {0, 94386201296895, 94384713689589, 18446603339259007057,
140354407146656,0, 1305670059009, 32, 4, 489626271867, 0, 0, 
 
                532575944823, 140734450544608, 0, 140734450544704}}}}
        send_ready_for_query = 0 '\000'
        disable_idle_in_transaction_timeout = <optimized out>
        __func__ = "PostgresMain"
#33 0x000055d7a72c6a1b in BackendRun (port=0x55d7a7c54500) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/postmaster/postmaster.c:4339
        ac = 1
        secs = 616545808
        usecs = 503344
        i = 1
        av = 0x55d7a7c56830
        maxac = <optimized out>
#34 BackendStartup (port=0x55d7a7c54500) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/postmaster/postmaster.c:4013
        bn = <optimized out>
---Type <return> to continue, or q <return> to quit---
        pid = <optimized out>
#35 ServerLoop () at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/postmaster/postmaster.c:1722
        rmask = {fds_bits = {16, 0 <repeats 15 times>}}
        selres = <optimized out>
        now = <optimized out>
        readmask = {fds_bits = {48, 0 <repeats 15 times>}}
        last_lockfile_recheck_time = 1563230588
        last_touch_time = 1563230588
        __func__ = "ServerLoop"
#36 0x000055d7a74ed281 in PostmasterMain (argc=13, argv=<optimized out>) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/postmaster/postmaster.c:1330
        opt = <optimized out>
        status = <optimized out>
        userDoption = <optimized out>
        listen_addr_saved = 1 '\001'
        i = <optimized out>
        output_config_variable = <optimized out>
        __func__ = "PostmasterMain"
#37 0x000055d7a72c7bf1 in main (argc=13, argv=0x55d7a7c0f840) at
/build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/main/main.c:228
No locals.
(gdb) list
228    in /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/access/heap/heapam.c
(gdb) 

>
>
>
> regards

-- 
Jerry Sievers
Postgres DBA/Development Consulting
e: postgres.consulting@comcast.net