Thread: crash 11.5~

crash 11.5~

From
Justin Pryzby
Date:
A daily report crashed repeatedly this morning running pg11.4.
I compiled 11.5 and reproduced it there, too, so I'm including backtrace with
-O0.

I'm trying to dig further into it, but it seems to be crashing under load, but
not when I try to narrow down to a single report, which seem to run to
completion when run individually.

postmaster[6750]: segfault at 7f4a527545bc ip 00000000004834ae sp 00007ffd547e2760 error 4 in postgres[400000+6e6000]
postmaster[29786]: segfault at 7f49a968d000 ip 00007f49b24abb0f sp 00007ffd547e2268 error 4 in libc-2.12.so
(deleted)[7f49b2422000+18b000]

Core was generated by `postgres: telsasoft ts 192.168.122.11(35454) SELECT                    '.
Program terminated with signal 11, Segmentation fault.

#0  0x00000000004877df in slot_deform_tuple (slot=0xe204638, natts=3) at heaptuple.c:1465
#1  0x0000000000487f5b in slot_getsomeattrs (slot=0xe204638, attnum=3) at heaptuple.c:1673
#2  0x00000000006d5aae in ExecInterpExpr (state=0xe257948, econtext=0xe204578, isnull=0x7ffff5f38077) at
execExprInterp.c:443
#3  0x00000000006ed221 in ExecEvalExprSwitchContext (state=0xe257948, econtext=0xe204578, isNull=0x7ffff5f38077) at
../../../src/include/executor/executor.h:313
#4  0x00000000006ed30e in ExecQual (state=0xe257948, econtext=0xe204578) at
../../../src/include/executor/executor.h:382
#5  0x00000000006ed5f4 in ExecScan (node=0x252edc8, accessMtd=0x71c54c <SeqNext>, recheckMtd=0x71c623 <SeqRecheck>) at
execScan.c:190
#6  0x000000000071c670 in ExecSeqScan (pstate=0x252edc8) at nodeSeqscan.c:129
#7  0x00000000006eb855 in ExecProcNodeInstr (node=0x252edc8) at execProcnode.c:461
#8  0x00000000006f6555 in ExecProcNode (node=0x252edc8) at ../../../src/include/executor/executor.h:247
#9  0x00000000006f6a05 in ExecAppend (pstate=0xe20e7b0) at nodeAppend.c:294
#10 0x00000000006eb855 in ExecProcNodeInstr (node=0xe20e7b0) at execProcnode.c:461
#11 0x0000000000708a96 in ExecProcNode (node=0xe20e7b0) at ../../../src/include/executor/executor.h:247
#12 0x0000000000709b0d in ExecHashJoinOuterGetTuple (outerNode=0xe20e7b0, hjstate=0xe20e4d8, hashvalue=0x7ffff5f3827c)
atnodeHashjoin.c:821
 
#13 0x0000000000708f8d in ExecHashJoinImpl (pstate=0xe20e4d8, parallel=false) at nodeHashjoin.c:355
#14 0x00000000007094ca in ExecHashJoin (pstate=0xe20e4d8) at nodeHashjoin.c:565
#15 0x00000000006eb855 in ExecProcNodeInstr (node=0xe20e4d8) at execProcnode.c:461
#16 0x000000000071d9da in ExecProcNode (node=0xe20e4d8) at ../../../src/include/executor/executor.h:247
#17 0x000000000071db26 in ExecSort (pstate=0xe20e3c0) at nodeSort.c:107
#18 0x00000000006eb855 in ExecProcNodeInstr (node=0xe20e3c0) at execProcnode.c:461
#19 0x00000000006eb826 in ExecProcNodeFirst (node=0xe20e3c0) at execProcnode.c:445
#20 0x00000000006f74c5 in ExecProcNode (node=0xe20e3c0) at ../../../src/include/executor/executor.h:247
#21 0x00000000006f79bb in fetch_input_tuple (aggstate=0xe20df70) at nodeAgg.c:406
#22 0x00000000006f9d11 in agg_retrieve_direct (aggstate=0xe20df70) at nodeAgg.c:1755
#23 0x00000000006f98fa in ExecAgg (pstate=0xe20df70) at nodeAgg.c:1570
#24 0x00000000006eb855 in ExecProcNodeInstr (node=0xe20df70) at execProcnode.c:461
#25 0x00000000006eb826 in ExecProcNodeFirst (node=0xe20df70) at execProcnode.c:445
#26 0x0000000000708a96 in ExecProcNode (node=0xe20df70) at ../../../src/include/executor/executor.h:247
#27 0x0000000000709b0d in ExecHashJoinOuterGetTuple (outerNode=0xe20df70, hjstate=0x2778580, hashvalue=0x7ffff5f3865c)
atnodeHashjoin.c:821
 
#28 0x0000000000708f8d in ExecHashJoinImpl (pstate=0x2778580, parallel=false) at nodeHashjoin.c:355
#29 0x00000000007094ca in ExecHashJoin (pstate=0x2778580) at nodeHashjoin.c:565
#30 0x00000000006eb855 in ExecProcNodeInstr (node=0x2778580) at execProcnode.c:461
#31 0x00000000006eb826 in ExecProcNodeFirst (node=0x2778580) at execProcnode.c:445
#32 0x0000000000702a9b in ExecProcNode (node=0x2778580) at ../../../src/include/executor/executor.h:247
#33 0x0000000000702d12 in MultiExecPrivateHash (node=0x27783a8) at nodeHash.c:164
#34 0x0000000000702c8c in MultiExecHash (node=0x27783a8) at nodeHash.c:114
#35 0x00000000006eb8f7 in MultiExecProcNode (node=0x27783a8) at execProcnode.c:501
#36 0x0000000000708e2a in ExecHashJoinImpl (pstate=0x2777210, parallel=false) at nodeHashjoin.c:290
#37 0x00000000007094ca in ExecHashJoin (pstate=0x2777210) at nodeHashjoin.c:565
#38 0x00000000006eb855 in ExecProcNodeInstr (node=0x2777210) at execProcnode.c:461
#39 0x00000000006eb826 in ExecProcNodeFirst (node=0x2777210) at execProcnode.c:445
#40 0x000000000071d9da in ExecProcNode (node=0x2777210) at ../../../src/include/executor/executor.h:247
#41 0x000000000071db26 in ExecSort (pstate=0x2776ce0) at nodeSort.c:107
#42 0x00000000006eb855 in ExecProcNodeInstr (node=0x2776ce0) at execProcnode.c:461
#43 0x00000000006eb826 in ExecProcNodeFirst (node=0x2776ce0) at execProcnode.c:445
#44 0x00000000006f74c5 in ExecProcNode (node=0x2776ce0) at ../../../src/include/executor/executor.h:247
#45 0x00000000006f79bb in fetch_input_tuple (aggstate=0x2776ec8) at nodeAgg.c:406
#46 0x00000000006f9d11 in agg_retrieve_direct (aggstate=0x2776ec8) at nodeAgg.c:1755
#47 0x00000000006f98fa in ExecAgg (pstate=0x2776ec8) at nodeAgg.c:1570
#48 0x00000000006eb855 in ExecProcNodeInstr (node=0x2776ec8) at execProcnode.c:461
#49 0x00000000006eb826 in ExecProcNodeFirst (node=0x2776ec8) at execProcnode.c:445
#50 0x00000000006e0129 in ExecProcNode (node=0x2776ec8) at ../../../src/include/executor/executor.h:247
#51 0x00000000006e2abd in ExecutePlan (estate=0x27769d8, planstate=0x2776ec8, use_parallel_mode=false,
operation=CMD_SELECT,sendTuples=true, numberTuples=0, direction=ForwardScanDirection, dest=0xa66bef0,
execute_once=true)at execMain.c:1723
 
#52 0x00000000006e0710 in standard_ExecutorRun (queryDesc=0xdeb5f38, direction=ForwardScanDirection, count=0,
execute_once=true)at execMain.c:364
 
#53 0x00007fd37e757633 in pgss_ExecutorRun (queryDesc=0xdeb5f38, direction=ForwardScanDirection, count=0,
execute_once=true)at pg_stat_statements.c:892
 
#54 0x00007fd37e2d86ef in explain_ExecutorRun (queryDesc=0xdeb5f38, direction=ForwardScanDirection, count=0,
execute_once=true)at auto_explain.c:281
 
#55 0x00000000006e0511 in ExecutorRun (queryDesc=0xdeb5f38, direction=ForwardScanDirection, count=0, execute_once=true)
atexecMain.c:305
 
#56 0x00000000008cfebe in PortalRunSelect (portal=0x21d4bf8, forward=true, count=0, dest=0xa66bef0) at pquery.c:932
#57 0x00000000008cfb4c in PortalRun (portal=0x21d4bf8, count=9223372036854775807, isTopLevel=true, run_once=true,
dest=0xa66bef0,altdest=0xa66bef0, completionTag=0x7ffff5f39100 "") at pquery.c:773
 
#58 0x00000000008c9b22 in exec_simple_query (query_string=0xe2fbd80 "--BEGIN SQL\nSELECT site_office AS
site_gran,\n\tsite_location||'.'||sect_numAS bs,\n\tgsm_site_name_to_sect_name(site_name, sect_num, sect_name) AS
sitename,\n\tperiodAS start_time,\n\t(maybe_div(sum(rrc_su"...)
 
    at postgres.c:1145
#59 0x00000000008cddf3 in PostgresMain (argc=1, argv=0x218fef8, dbname=0x215da48 "ts", username=0x218fed0 "telsasoft")
atpostgres.c:4182
 
#60 0x000000000082a098 in BackendRun (port=0x21867d0) at postmaster.c:4358
#61 0x0000000000829806 in BackendStartup (port=0x21867d0) at postmaster.c:4030
#62 0x0000000000825cab in ServerLoop () at postmaster.c:1707
#63 0x00000000008255dd in PostmasterMain (argc=3, argv=0x215b9a0) at postmaster.c:1380
#64 0x000000000074ba30 in main (argc=3, argv=0x215b9a0) at main.c:228
(gdb) 

#0  0x00000000004877df in slot_deform_tuple (slot=0xe204638, natts=3) at heaptuple.c:1465
        thisatt = 0x5265db0
        tuple = 0xe838d40
        tupleDesc = 0x5265d20
        values = 0xe204698
        isnull = 0xe2057e8
        tup = 0x7fd3eb1b8058
        hasnulls = false
        attnum = 1
        tp = 0x7fd3eb1b80bd <Address 0x7fd3eb1b80bd out of bounds>
        off = 583369983
        bp = 0x7fd3eb1b806f <Address 0x7fd3eb1b806f out of bounds>
        slow = true
#1  0x0000000000487f5b in slot_getsomeattrs (slot=0xe204638, attnum=3) at heaptuple.c:1673
        tuple = 0xe838d40
        attno = 3
        __func__ = "slot_getsomeattrs"
#2  0x00000000006d5aae in ExecInterpExpr (state=0xe257948, econtext=0xe204578, isnull=0x7ffff5f38077) at
execExprInterp.c:443
        op = 0xe257530
        resultslot = 0x0
        innerslot = 0x0
        outerslot = 0x0
        scanslot = 0xe204638
        dispatch_table = {0x6d5a0d, 0x6d5a32, 0x6d5a61, 0x6d5a93, 0x6d5ac5, 0x6d5b74, 0x6d5c23, 0x6d5cd2, 0x6d5da3,
0x6d5e74,0x6d5f45, 0x6d5f7c, 0x6d6056, 0x6d6130, 0x6d620a, 0x6d627e, 0x6d6347, 0x6d638f, 0x6d640e, 0x6d64ec, 0x6d6523,
0x6d655a,0x6d6568, 0x6d65e9, 0x6d6657, 0x6d6665, 
 
          0x6d66e6, 0x6d6754, 0x6d6790, 0x6d6821, 0x6d6856, 0x6d68b4, 0x6d6915, 0x6d6986, 0x6d69ca, 0x6d6a14, 0x6d6a4b,
0x6d6a82,0x6d6acb, 0x6d6b3b, 0x6d6bab, 0x6d6bf4, 0x6d6c2b, 0x6d6c62, 0x6d6ca1, 0x6d6dc3, 0x6d6e2e, 0x6d704c, 0x6d7161,
0x6d726f,0x6d7376, 0x6d73a6, 0x6d73d6, 0x6d7406, 
 
          0x6d7436, 0x6d746d, 0x6d749d, 0x6d7625, 0x6d7724, 0x6d7754, 0x6d778b, 0x6d77c2, 0x6d77f9, 0x6d7862, 0x6d7892,
0x6d78c2,0x6d6d32, 0x6d7959, 0x6d7989, 0x6d78f2, 0x6d7929, 0x6d79b9, 0x6d79e9, 0x6d7a98, 0x6d7ac8, 0x6d7b77, 0x6d7bae,
0x6d7be5,0x6d7c40, 0x6d7cf5, 0x6d7d9a, 0x6d7ead, 
 
          0x6d7f52, 0x6d80b4, 0x6d8261, 0x6d8298, 0x6d82cf}
#3  0x00000000006ed221 in ExecEvalExprSwitchContext (state=0xe257948, econtext=0xe204578, isNull=0x7ffff5f38077) at
../../../src/include/executor/executor.h:313
        retDatum = 140737319764160
        oldContext = 0x27768c0
#4  0x00000000006ed30e in ExecQual (state=0xe257948, econtext=0xe204578) at
../../../src/include/executor/executor.h:382
        ret = 3018996923
        isnull = false
#5  0x00000000006ed5f4 in ExecScan (node=0x252edc8, accessMtd=0x71c54c <SeqNext>, recheckMtd=0x71c623 <SeqRecheck>) at
execScan.c:190
        slot = 0xe204638
        econtext = 0xe204578
        qual = 0xe257948
        projInfo = 0xe257498
#6  0x000000000071c670 in ExecSeqScan (pstate=0x252edc8) at nodeSeqscan.c:129
---Type <return> to continue, or q <return> to quit---
        node = 0x252edc8
#7  0x00000000006eb855 in ExecProcNodeInstr (node=0x252edc8) at execProcnode.c:461
        result = 0x702ace
#8  0x00000000006f6555 in ExecProcNode (node=0x252edc8) at ../../../src/include/executor/executor.h:247
No locals.
#9  0x00000000006f6a05 in ExecAppend (pstate=0xe20e7b0) at nodeAppend.c:294
        subnode = 0x252edc8
        result = 0xe258508
        node = 0xe20e7b0
#10 0x00000000006eb855 in ExecProcNodeInstr (node=0xe20e7b0) at execProcnode.c:461
        result = 0x20e257058
#11 0x0000000000708a96 in ExecProcNode (node=0xe20e7b0) at ../../../src/include/executor/executor.h:247
No locals.
#12 0x0000000000709b0d in ExecHashJoinOuterGetTuple (outerNode=0xe20e7b0, hjstate=0xe20e4d8, hashvalue=0x7ffff5f3827c)
atnodeHashjoin.c:821
 
        hashtable = 0xe8387c8
        curbatch = 0
        slot = 0x0
#13 0x0000000000708f8d in ExecHashJoinImpl (pstate=0xe20e4d8, parallel=false) at nodeHashjoin.c:355
        node = 0xe20e4d8
        outerNode = 0xe20e7b0
        hashNode = 0xe258720
        joinqual = 0x0
        otherqual = 0x0
        econtext = 0xe20e6f0
        hashtable = 0xe8387c8
        outerTupleSlot = 0xe257058
        hashvalue = 3018996923
        batchno = 0
        parallel_state = 0x0
        __func__ = "ExecHashJoinImpl"
#14 0x00000000007094ca in ExecHashJoin (pstate=0xe20e4d8) at nodeHashjoin.c:565
No locals.
#15 0x00000000006eb855 in ExecProcNodeInstr (node=0xe20e4d8) at execProcnode.c:461
        result = 0x20c4
#16 0x000000000071d9da in ExecProcNode (node=0xe20e4d8) at ../../../src/include/executor/executor.h:247
No locals.
#17 0x000000000071db26 in ExecSort (pstate=0xe20e3c0) at nodeSort.c:107
        plannode = 0xa6309c0
        outerNode = 0xe20e4d8
---Type <return> to continue, or q <return> to quit---
        tupDesc = 0xe2598a0
        node = 0xe20e3c0
        estate = 0x27769d8
        dir = ForwardScanDirection
        tuplesortstate = 0x2766998
        slot = 0xe25b8b8
#18 0x00000000006eb855 in ExecProcNodeInstr (node=0xe20e3c0) at execProcnode.c:461
        result = 0x4785d0
#19 0x00000000006eb826 in ExecProcNodeFirst (node=0xe20e3c0) at execProcnode.c:445
No locals.
#20 0x00000000006f74c5 in ExecProcNode (node=0xe20e3c0) at ../../../src/include/executor/executor.h:247
No locals.
#21 0x00000000006f79bb in fetch_input_tuple (aggstate=0xe20df70) at nodeAgg.c:406
        slot = 0xe20e240
#22 0x00000000006f9d11 in agg_retrieve_direct (aggstate=0xe20df70) at nodeAgg.c:1755
        node = 0xa6315d0
        econtext = 0xe20e300
        tmpcontext = 0xe20e1a8
        peragg = 0x252e198
        pergroups = 0xe7cdd40
        outerslot = 0x7ffff5f38520
        firstSlot = 0xe263e70
        result = 0x0
        hasGroupingSets = false
        numGroupingSets = 1
        currentSet = 222632768
        nextSetSize = 0
        numReset = 1
        i = 1
#23 0x00000000006f98fa in ExecAgg (pstate=0xe20df70) at nodeAgg.c:1570
        node = 0xe20df70
        result = 0x0
#24 0x00000000006eb855 in ExecProcNodeInstr (node=0xe20df70) at execProcnode.c:461
        result = 0x4785d0
#25 0x00000000006eb826 in ExecProcNodeFirst (node=0xe20df70) at execProcnode.c:445
No locals.
#26 0x0000000000708a96 in ExecProcNode (node=0xe20df70) at ../../../src/include/executor/executor.h:247
No locals.
#27 0x0000000000709b0d in ExecHashJoinOuterGetTuple (outerNode=0xe20df70, hjstate=0x2778580, hashvalue=0x7ffff5f3865c)
atnodeHashjoin.c:821
 
#28 0x0000000000708f8d in ExecHashJoinImpl (pstate=0x2778580, parallel=false) at nodeHashjoin.c:355
        node = 0x2778580
        outerNode = 0xe20df70
        hashNode = 0xe7e2560
        joinqual = 0x0
        otherqual = 0x0
        econtext = 0x2778798
        hashtable = 0xe837df0
        outerTupleSlot = 0x4040
        hashvalue = 0
        batchno = 10926240
        parallel_state = 0x0
        __func__ = "ExecHashJoinImpl"
#29 0x00000000007094ca in ExecHashJoin (pstate=0x2778580) at nodeHashjoin.c:565
No locals.
#30 0x00000000006eb855 in ExecProcNodeInstr (node=0x2778580) at execProcnode.c:461
        result = 0x4785d0
#31 0x00000000006eb826 in ExecProcNodeFirst (node=0x2778580) at execProcnode.c:445
No locals.
#32 0x0000000000702a9b in ExecProcNode (node=0x2778580) at ../../../src/include/executor/executor.h:247
No locals.
#33 0x0000000000702d12 in MultiExecPrivateHash (node=0x27783a8) at nodeHash.c:164
        outerNode = 0x2778580
        hashkeys = 0xe7ebf78
        hashtable = 0xe837cd8
        slot = 0x80000000001
        econtext = 0x27784c0
        hashvalue = 0
#34 0x0000000000702c8c in MultiExecHash (node=0x27783a8) at nodeHash.c:114
No locals.
#35 0x00000000006eb8f7 in MultiExecProcNode (node=0x27783a8) at execProcnode.c:501
        result = 0x4
        __func__ = "MultiExecProcNode"
#36 0x0000000000708e2a in ExecHashJoinImpl (pstate=0x2777210, parallel=false) at nodeHashjoin.c:290
        node = 0x2777210
        outerNode = 0x27774e8
---Type <return> to continue, or q <return> to quit---
        hashNode = 0x27783a8
        joinqual = 0x0
        otherqual = 0x0
        econtext = 0x2777428
        hashtable = 0xe837cd8
        outerTupleSlot = 0x7ffff5f388ac
        hashvalue = 0
        batchno = 0
        parallel_state = 0x0
        __func__ = "ExecHashJoinImpl"
#37 0x00000000007094ca in ExecHashJoin (pstate=0x2777210) at nodeHashjoin.c:565
No locals.
#38 0x00000000006eb855 in ExecProcNodeInstr (node=0x2777210) at execProcnode.c:461
        result = 0x4785d0
#39 0x00000000006eb826 in ExecProcNodeFirst (node=0x2777210) at execProcnode.c:445
No locals.
#40 0x000000000071d9da in ExecProcNode (node=0x2777210) at ../../../src/include/executor/executor.h:247
No locals.
#41 0x000000000071db26 in ExecSort (pstate=0x2776ce0) at nodeSort.c:107
        plannode = 0xa632c18
        outerNode = 0x2777210
        tupDesc = 0xe7e6980
        node = 0x2776ce0
        estate = 0x27769d8
        dir = ForwardScanDirection
        tuplesortstate = 0x24feec8
        slot = 0x219bfa0
#42 0x00000000006eb855 in ExecProcNodeInstr (node=0x2776ce0) at execProcnode.c:461
        result = 0x4785d0
#43 0x00000000006eb826 in ExecProcNodeFirst (node=0x2776ce0) at execProcnode.c:445
No locals.
#44 0x00000000006f74c5 in ExecProcNode (node=0x2776ce0) at ../../../src/include/executor/executor.h:247
No locals.
#45 0x00000000006f79bb in fetch_input_tuple (aggstate=0x2776ec8) at nodeAgg.c:406
        slot = 0x2776df8
#46 0x00000000006f9d11 in agg_retrieve_direct (aggstate=0x2776ec8) at nodeAgg.c:1755
        node = 0xa633eb0
        econtext = 0x2777150
        tmpcontext = 0x2776bf0
---Type <return> to continue, or q <return> to quit---
        peragg = 0xe824520
        pergroups = 0xe832e88
        outerslot = 0x7ffff5f38b40
        firstSlot = 0xe81f6b8
        result = 0x7ffff5f38b10
        hasGroupingSets = false
        numGroupingSets = 1
        currentSet = 243398904
        nextSetSize = 0
        numReset = 1
        i = 1
#47 0x00000000006f98fa in ExecAgg (pstate=0x2776ec8) at nodeAgg.c:1570
        node = 0x2776ec8
        result = 0x0
#48 0x00000000006eb855 in ExecProcNodeInstr (node=0x2776ec8) at execProcnode.c:461
        result = 0x4785d0
#49 0x00000000006eb826 in ExecProcNodeFirst (node=0x2776ec8) at execProcnode.c:445
No locals.
#50 0x00000000006e0129 in ExecProcNode (node=0x2776ec8) at ../../../src/include/executor/executor.h:247
No locals.
#51 0x00000000006e2abd in ExecutePlan (estate=0x27769d8, planstate=0x2776ec8, use_parallel_mode=false,
operation=CMD_SELECT,sendTuples=true, numberTuples=0, direction=ForwardScanDirection, dest=0xa66bef0,
execute_once=true)at execMain.c:1723
 
        slot = 0x150
        current_tuple_count = 0
#52 0x00000000006e0710 in standard_ExecutorRun (queryDesc=0xdeb5f38, direction=ForwardScanDirection, count=0,
execute_once=true)at execMain.c:364
 
        estate = 0x27769d8
        operation = CMD_SELECT
        dest = 0xa66bef0
        sendTuples = true
        oldcontext = 0xdeb5e20
        __func__ = "standard_ExecutorRun"
#53 0x00007fd37e757633 in pgss_ExecutorRun (queryDesc=0xdeb5f38, direction=ForwardScanDirection, count=0,
execute_once=true)at pg_stat_statements.c:892
 
        save_exception_stack = 0x7ffff5f38dd0
        save_context_stack = 0x0
        local_sigjmp_buf = {{__jmpbuf = {233529144, 9176610699338068304, 4687312, 140737319770032, 0, 0,
9176610699377914192,9151536278965596496}, __mask_was_saved = 0, __saved_mask = {__val = {4294967302, 243497128,
4295032832,140737319767408, 7255973, 140737319767408, 68726688595, 
 
                41380312, 174276272, 174278840, 41381576, 0, 41381576, 16, 174278840, 140737319767696}}}}
#54 0x00007fd37e2d86ef in explain_ExecutorRun (queryDesc=0xdeb5f38, direction=ForwardScanDirection, count=0,
execute_once=true)at auto_explain.c:281
 
        save_exception_stack = 0x7ffff5f38fa0
        save_context_stack = 0x0
        local_sigjmp_buf = {{__jmpbuf = {233529144, 9176610699451314512, 4687312, 140737319770032, 0, 0,
9176610699340165456,9151535941362266448}, __mask_was_saved = 0, __saved_mask = {__val = {10926707, 24, 41371824,
41371824,238009728, 41372168, 8600905475, 35241888, 336, 243497664, 
 
---Type <return> to continue, or q <return> to quit---
                41380032, 243498000, 233528864, 336, 174505712, 140737319767760}}}}
#55 0x00000000006e0511 in ExecutorRun (queryDesc=0xdeb5f38, direction=ForwardScanDirection, count=0, execute_once=true)
atexecMain.c:305
 
No locals.
#56 0x00000000008cfebe in PortalRunSelect (portal=0x21d4bf8, forward=true, count=0, dest=0xa66bef0) at pquery.c:932
        queryDesc = 0xdeb5f38
        direction = ForwardScanDirection
        nprocessed = 233528864
        __func__ = "PortalRunSelect"
#57 0x00000000008cfb4c in PortalRun (portal=0x21d4bf8, count=9223372036854775807, isTopLevel=true, run_once=true,
dest=0xa66bef0,altdest=0xa66bef0, completionTag=0x7ffff5f39100 "") at pquery.c:773
 
        save_exception_stack = 0x7ffff5f39240
        save_context_stack = 0x0
        local_sigjmp_buf = {{__jmpbuf = {174505712, 9176610698715214160, 4687312, 140737319770032, 0, 0,
9176610699432440144,-9176589804060397232}, __mask_was_saved = 0, __saved_mask = {__val = {233529080, 2405784,
21306251184,128, 13, 616, 174505656, 140737319768192, 10970883, 480, 112, 
 
                174505712, 34999312, 174505824, 233528864, 112}}}}
        result = false
        nprocessed = 4687312
        saveTopTransactionResourceOwner = 0x2197a20
        saveTopTransactionContext = 0x27748b0
        saveActivePortal = 0x0
        saveResourceOwner = 0x2197a20
        savePortalContext = 0x0
        saveMemoryContext = 0x27748b0
        __func__ = "PortalRun"
#58 0x00000000008c9b22 in exec_simple_query (query_string=0xe2fbd80 "--BEGIN SQL\nSELECT site_office AS
site_gran,\n\tsite_location||'.'||sect_numAS bs,\n\tgsm_site_name_to_sect_name(site_name, sect_num, sect_name) AS
sitename,\n\tperiodAS start_time,\n\t(maybe_div(sum(rrc_su"...)
 
    at postgres.c:1145
        parsetree = 0xd4f1c38
        portal = 0x21d4bf8
        snapshot_set = true
        commandTag = 0xc5ed4d "SELECT"
        completionTag =
"\000\000\000\000\002\000\000\000\n\000\000\000\000\000\000\000P\221\363\365\377\177\000\000\017s\245\000\000\000\000\000\036\020\000\000\000\000\000\000\000\f\026\002\030\020\000\000\200\275/\016\000\000\000\000(\r\026\002\006\000\000"
        querytree_list = 0xe230d58
        plantree_list = 0xa66beb8
        receiver = 0xa66bef0
        format = 0
        dest = DestRemote
        oldcontext = 0x27748b0
        parsetree_list = 0xd4f1c98
        parsetree_item = 0xd4f1c70
        save_log_statement_stats = false
        was_logged = false
---Type <return> to continue, or q <return> to quit---
        use_implicit_block = false
        msec_str =
"\220\221\363\365\377\177\000\000&C\245\000\000\000\000\000\006\000\000\000\030\020\000\000\200\275/\016\000\000\000"
        __func__ = "exec_simple_query"
#59 0x00000000008cddf3 in PostgresMain (argc=1, argv=0x218fef8, dbname=0x215da48 "ts", username=0x218fed0 "telsasoft")
atpostgres.c:4182
 
        query_string = 0xe2fbd80 "--BEGIN SQL\nSELECT site_office AS site_gran,\n\tsite_location||'.'||sect_num AS
bs,\n\tgsm_site_name_to_sect_name(site_name,sect_num, sect_name) AS sitename,\n\tperiod AS
start_time,\n\t(maybe_div(sum(rrc_su"...
        firstchar = 81
        input_message = {data = 0xe2fbd80 "--BEGIN SQL\nSELECT site_office AS
site_gran,\n\tsite_location||'.'||sect_numAS bs,\n\tgsm_site_name_to_sect_name(site_name, sect_num, sect_name) AS
sitename,\n\tperiodAS start_time,\n\t(maybe_div(sum(rrc_su"..., len = 4121, maxlen = 8192, 
 
          cursor = 4121}
        local_sigjmp_buf = {{__jmpbuf = {35192568, 9176610698734088528, 4687312, 140737319770032, 0, 0,
9176610698788614480,-9176589805133483696}, __mask_was_saved = 1, __saved_mask = {__val = {0, 0, 4494597, 0, 0, 0,
10923965,1024, 35138464, 140737319768848, 10928134, 35138464, 34999312, 
 
                34999312, 35138440, 8}}}}
        send_ready_for_query = false
        disable_idle_in_transaction_timeout = false
        __func__ = "PostgresMain"
#60 0x000000000082a098 in BackendRun (port=0x21867d0) at postmaster.c:4358
        av = 0x218fef8
        maxac = 2
        ac = 1
        secs = 618513625
        usecs = 637903
        i = 1
        __func__ = "BackendRun"
#61 0x0000000000829806 in BackendStartup (port=0x21867d0) at postmaster.c:4030
        bn = 0x2185ef0
        pid = 0
        __func__ = "BackendStartup"
#62 0x0000000000825cab in ServerLoop () at postmaster.c:1707
        port = 0x21867d0
        i = 0
        rmask = {fds_bits = {8, 0 <repeats 15 times>}}
        selres = 1
        now = 1565198425
        readmask = {fds_bits = {56, 0 <repeats 15 times>}}
        nSockets = 6
        last_lockfile_recheck_time = 1565198368
        last_touch_time = 1565197325
        __func__ = "ServerLoop"
#63 0x00000000008255dd in PostmasterMain (argc=3, argv=0x215b9a0) at postmaster.c:1380
        opt = -1
        status = 0
---Type <return> to continue, or q <return> to quit---
        userDoption = 0x217d500 "/var/lib/pgsql/11/data"
        listen_addr_saved = true
        i = 64
        output_config_variable = 0x0
        __func__ = "PostmasterMain"
#64 0x000000000074ba30 in main (argc=3, argv=0x215b9a0) at main.c:228
        do_check_root = true



Re: crash 11.5~ (and 11.4)

From
Justin Pryzby
Date:
I checked this still happens with max_parallel_workers_per_gather=0.
Now, I just reproduced using SELECT * FROM that table:

(gdb) p thisatt->attrelid
$4 = 2015128626

ts=# SELECT 2015128626::regclass;
regclass | child.huawei_umts_ucell_201908

(gdb) p thisatt->attnum 
$1 = 2
(gdb) p attnum # For earlier crash this is 1....
$2 = 1

ts=# \dt+ child.huawei_umts_ucell_201908
 child  | huawei_umts_ucell_201908 | table | telsasoft | 612 MB | 

\d+ child.huawei_umts_ucell_201908
 bsc6900ucell                               | text                        |           | not null |         | extended |
            | 
 
 ne_name                                    | text                        |           | not null |         | extended |
            | 
 
[...]
Partition of: huawei_umts_ucell_metrics FOR VALUES FROM ('2019-08-01 00:00:00') TO ('2019-09-01 00:00:00')
Partition constraint: ((start_time IS NOT NULL) AND (start_time >= '2019-08-01 00:00:00'::timestamp without time zone)
AND(start_time < '2019-09-01 00:00:00'::timestamp without time zone))
 
Indexes:
    "huawei_umts_ucell_201908_unique_idx" UNIQUE, btree (start_time, device_id, bsc6900ucell, ne_name,
interval_seconds)
    "huawei_umts_ucell_201908_idx" brin (start_time) WITH (autosummarize='true')
    "huawei_umts_ucell_201908_site_idx" btree (site_id)
Check constraints:
    "huawei_umts_ucell_201908_start_time_check" CHECK (start_time >= '2019-08-01 00:00:00'::timestamp without time zone
ANDstart_time < '2019-09-01 00:00:00'::timestamp without time zone)
 
Statistics objects:
    "child"."huawei_umts_ucell_201908" (ndistinct) ON bsc6900ucell, ne_name, start_time, interval_seconds, device_id
FROMchild.huawei_umts_ucell_201908
 
Options: autovacuum_analyze_threshold=2, autovacuum_analyze_scale_factor=0.005

ts=# SELECT COUNT(1) FROM pg_attribute WHERE attrelid='child.huawei_umts_ucell_201908'::regclass;
count | 560

ts=# SELECT * FROM pg_attribute WHERE attrelid='child.huawei_umts_ucell_201908'::regclass AND attnum>=0 ORDER BY
attnum;

attrelid      | 2015128626
attname       | bsc6900ucell
atttypid      | 25
attstattarget | -1
attlen        | -1
attnum        | 1
attndims      | 0
attcacheoff   | -1
atttypmod     | -1
attbyval      | f
attstorage    | x
attalign      | i
attnotnull    | t
atthasdef     | f
atthasmissing | f
attidentity   | 
attisdropped  | f
attislocal    | f
attinhcount   | 1
attcollation  | 100
attacl        | 
attoptions    | 
attfdwoptions | 
attmissingval | 

The only interesting thing about the parent/siblings is that the previous month
was partitioned with "daily" granularity.  I adjusted our threshold for that,
so that August is partitioned monthly:

    [...]
    child.huawei_umts_ucell_20190730 FOR VALUES FROM ('2019-07-30 00:00:00') TO ('2019-07-31 00:00:00'),
    child.huawei_umts_ucell_20190731 FOR VALUES FROM ('2019-07-31 00:00:00') TO ('2019-08-01 00:00:00'),
    child.huawei_umts_ucell_201908 FOR VALUES FROM ('2019-08-01 00:00:00') TO ('2019-09-01 00:00:00')

Program terminated with signal 11, Segmentation fault.
#0  0x00000000004877df in slot_deform_tuple (slot=0x22b5860, natts=554) at heaptuple.c:1465
1465                                    off = att_align_pointer(off, thisatt->attalign, -1,

(gdb) bt
#0  0x00000000004877df in slot_deform_tuple (slot=0x22b5860, natts=554) at heaptuple.c:1465
#1  0x0000000000487e4b in slot_getallattrs (slot=0x22b5860) at heaptuple.c:1626
#2  0x000000000048aab9 in printtup (slot=0x22b5860, self=0x2161b28) at printtup.c:392
#3  0x00000000008d0346 in RunFromStore (portal=0x21d5638, direction=ForwardScanDirection, count=0, dest=0x2161b28) at
pquery.c:1106
#4  0x00000000008cfe88 in PortalRunSelect (portal=0x21d5638, forward=true, count=0, dest=0x2161b28) at pquery.c:928
#5  0x00000000008cfb4c in PortalRun (portal=0x21d5638, count=9223372036854775807, isTopLevel=true, run_once=true,
dest=0x2161b28,altdest=0x2161b28, completionTag=0x7ffff5f39100 "") at pquery.c:773
 
#6  0x00000000008c9b22 in exec_simple_query (query_string=0x2160d28 "FETCH FORWARD 999 FROM _psql_cursor") at
postgres.c:1145
#7  0x00000000008cddf3 in PostgresMain (argc=1, argv=0x218bfb0, dbname=0x218beb0 "ts", username=0x218be90 "pryzbyj") at
postgres.c:4182
#8  0x000000000082a098 in BackendRun (port=0x2181ac0) at postmaster.c:4358
#9  0x0000000000829806 in BackendStartup (port=0x2181ac0) at postmaster.c:4030
#10 0x0000000000825cab in ServerLoop () at postmaster.c:1707
#11 0x00000000008255dd in PostmasterMain (argc=3, argv=0x215b9a0) at postmaster.c:1380
#12 0x000000000074ba30 in main (argc=3, argv=0x215b9a0) at main.c:228

#0  0x00000000004877df in slot_deform_tuple (slot=0x22b5860, natts=554) at heaptuple.c:1465
        thisatt = 0x22dec50
        tuple = 0x22b58a0
        tupleDesc = 0x22debc0
        values = 0x22b58c0
        isnull = 0x22b6a10
        tup = 0x263f9a0
        hasnulls = false
        attnum = 1
        tp = 0x263fa05
"\374\023\026\213^s#\347(\235=\326\321\067\032\245\321B\026}܋FS\375\244\003\065\336\277;\252O\006\065\320\353\211}F\237\373B\243\357J~\270\"\230ƣ\024xǍ\334\377\202\277S\031\375\351\003\220{\004"
        off = 583369983
        bp = 0x263f9b7
"\270\027$U}\232\246\235\004\255\331\033\006Qp\376E\316h\376\363\247\366Նgy7\311E\224~F\274\023ϋ%\216,\221\331@\024\363\233\070\275\004\254L\217t\262X\227\352\346\347\371\070\321ш\221\350fc\316\r\356\351h\275\213\230\360\203\374\023\026\213^s#\347(\235=\326\321\067\032\245\321B\026}܋FS\375\244\003\065\336\277;\252O\006\065\320\353\211}F\237\373B\243\357J~\270\"\230ƣ\024xǍ\334\377\202\277S\031\375\351\003\220{\004"
        slow = true
#1  0x0000000000487e4b in slot_getallattrs (slot=0x22b5860) at heaptuple.c:1626
        tdesc_natts = 554
        attnum = 554
        tuple = 0x22b58a0
        __func__ = "slot_getallattrs"

#2  0x000000000048aab9 in printtup (slot=0x22b5860, self=0x2161b28) at printtup.c:392
        typeinfo = 0x22debc0
        myState = 0x2161b28
        oldcontext = 0xa8e29a
        buf = 0x2161b78
        natts = 554
        i = 0
---Type <return> to continue, or q <return> to quit---
#3  0x00000000008d0346 in RunFromStore (portal=0x21d5638, direction=ForwardScanDirection, count=0, dest=0x2161b28) at
pquery.c:1106
        oldcontext = 0x21931a0
        ok = true
        forward = true
        current_tuple_count = 879
        slot = 0x22b5860
#4  0x00000000008cfe88 in PortalRunSelect (portal=0x21d5638, forward=true, count=0, dest=0x2161b28) at pquery.c:928
        queryDesc = 0x0
        direction = ForwardScanDirection
        nprocessed = 35205816
        __func__ = "PortalRunSelect"
#5  0x00000000008cfb4c in PortalRun (portal=0x21d5638, count=9223372036854775807, isTopLevel=true, run_once=true,
dest=0x2161b28,altdest=0x2161b28, completionTag=0x7ffff5f39100 "") at pquery.c:773
 
        save_exception_stack = 0x7ffff5f39240
        save_context_stack = 0x0
        local_sigjmp_buf = {{__jmpbuf = {35003176, 9176610698715214160, 4687312, 140737319770032, 0, 0,
9176610699432440144,-9176589804060397232}, __mask_was_saved = 0, __saved_mask = {__val = {
 
                36636288, 4352, 35002176, 1112, 1176, 0, 35003120, 140737319768192, 10970883, 35488536, 112, 35003176,
34999312,35003288, 35205536, 112}}}}
 
        result = false
        nprocessed = 4687312
        saveTopTransactionResourceOwner = 0x2187b88
        saveTopTransactionContext = 0x21f5d90
        saveActivePortal = 0x0
        saveResourceOwner = 0x2187b88
        savePortalContext = 0x0
        saveMemoryContext = 0x21f5d90
        __func__ = "PortalRun"
#6  0x00000000008c9b22 in exec_simple_query (query_string=0x2160d28 "FETCH FORWARD 999 FROM _psql_cursor") at
postgres.c:1145
[...]





Re: crash 11.5~ (and 11.4)

From
Justin Pryzby
Date:
Just found this, although I'm not sure what to do about it.  If it's corrupt
table data, I can restore from backup.

ts=# VACUUM FREEZE VERBOSE child.huawei_umts_ucell_201908;
INFO:  00000: aggressively vacuuming "child.huawei_umts_ucell_201908"
LOCATION:  lazy_scan_heap, vacuumlazy.c:502
ERROR:  XX001: found xmin 73850277 from before relfrozenxid 111408920
LOCATION:  heap_prepare_freeze_tuple, heapam.c:6853

I confirmed I updated to 11.4 immediately on its release:
[pryzbyj@database ~]$ ls -ltc /usr/pgsql-11/bin/postgres
-rwxr-xr-x. 1 root root 7291736 Jun 20 07:09 /usr/pgsql-11/bin/postgres

That table would've been created shortly after midnight on Aug 1 when we loaded
first data for the month.  So it was created and processed only on pg11.4,
although the parent has probably been around since pg_upgrade last October.

Here's usable looking bt

#0  heap_prepare_freeze_tuple (tuple=0x7fd3d8fa0058, relfrozenxid=111408920, 
    relminmxid=1846, cutoff_xid=111658731, cutoff_multi=1846, frz=0x223e930, 
    totally_frozen_p=0x7ffff5f3848f) at heapam.c:6832
        changed = 211
        xmax_already_frozen = 127
        xmin_frozen = false
        freeze_xmax = false
        xid = 0
        __func__ = "heap_prepare_freeze_tuple"

#1  0x00000000006c8e6e in lazy_scan_heap (onerel=0x7fd37e4f8fa0, options=13, 
    vacrelstats=0x223e790, Irel=0x223e8d8, nindexes=3, aggressive=true)
    at vacuumlazy.c:1151
        tuple_totally_frozen = false
        itemid = 0x7fd3d8f9e918
        buf = 168313
        page = 0x7fd3d8f9e900 "\020:"
        offnum = 1
        maxoff = 3
        hastup = true
        nfrozen = 0
        freespace = 35907664
        all_frozen = true
        tupgone = false
        prev_dead_count = 0
        all_visible_according_to_vm = true
        all_visible = true
        has_dead_tuples = false
        visibility_cutoff_xid = 73850277
        nblocks = 75494
        blkno = 34915
        tuple = {t_len = 2212, t_self = {ip_blkid = {bi_hi = 0, 
              bi_lo = 34915}, ip_posid = 1}, t_tableOid = 2015128626, 
          t_data = 0x7fd3d8fa0058}
        relname = 0x7fd37e4f6ac8 "huawei_umts_ucell_201908"
        relfrozenxid = 111408920
        relminmxid = 1846
        empty_pages = 0
        vacuumed_pages = 0
        next_fsm_block_to_vacuum = 0
        num_tuples = 1
        live_tuples = 1
        tups_vacuumed = 0
        nkeep = 0
        nunused = 0
        indstats = 0x223e850
        i = 32512
        ru0 = {tv = {tv_sec = 1565204981, tv_usec = 609217}, ru = {ru_utime = {
              tv_sec = 13, tv_usec = 865892}, ru_stime = {tv_sec = 1, 
              tv_usec = 960701}, ru_maxrss = 136988, ru_ixrss = 0, 
            ru_idrss = 0, ru_isrss = 0, ru_minflt = 48841, ru_majflt = 1, 
            ru_nswap = 0, ru_inblock = 196152, ru_oublock = 292656, 
            ru_msgsnd = 0, ru_msgrcv = 0, ru_nsignals = 0, ru_nvcsw = 849, 
            ru_nivcsw = 6002}}
        vmbuffer = 4267
        next_unskippable_block = 34916
        skipping_blocks = false
        frozen = 0x223e930
        buf = {data = 0x0, len = -168590068, maxlen = 32767, 
          cursor = -168589904}
        initprog_index = {0, 1, 5}
        initprog_val = {1, 75494, 21968754}
        __func__ = "lazy_scan_heap"
#2  0x00000000006c77c7 in lazy_vacuum_rel (onerel=0x7fd37e4f8fa0, options=13, 
    params=0x7ffff5f38a70, bstrategy=0x22b8cf8) at vacuumlazy.c:265
        vacrelstats = 0x223e790
        Irel = 0x223e8d8
        nindexes = 3
        ru0 = {tv = {tv_sec = 140733193388036, tv_usec = 281474985219646}, 
          ru = {ru_utime = {tv_sec = 12933152, tv_usec = 35123496}, 
            ru_stime = {tv_sec = 140737319765996, tv_usec = 4687312}, 
            ru_maxrss = 0, ru_ixrss = -1887226286461968708, 
            ru_idrss = 35351240, ru_isrss = 35249392, 
            ru_minflt = 140737319765776, ru_majflt = 35249392, ru_nswap = 20, 
            ru_inblock = 0, ru_oublock = 140737319765936, 
            ru_msgsnd = 10990471, ru_msgrcv = 140546333970336, 
            ru_nsignals = 35249264, ru_nvcsw = 4, ru_nivcsw = 4687312}}
        starttime = 0
        secs = 72057594037927936
        usecs = 2015128626
        read_rate = 6.9533474784178538e-310
        write_rate = 6.9439115263673584e-310
        aggressive = true
        scanned_all_unfrozen = false
        xidFullScanLimit = 111658735
        mxactFullScanLimit = 1846
        new_rel_pages = 0
        new_rel_allvisible = 36424984
        new_live_tuples = 6.9533474784194348e-310
        new_frozen_xid = 9080197
        new_min_multi = 0
        __func__ = "lazy_vacuum_rel"
#3  0x00000000006c72b3 in vacuum_rel (relid=2015128626, relation=0x2161820, 
    options=13, params=0x7ffff5f38a70) at vacuum.c:1557
        lmode = 4
        onerel = 0x7fd37e4f8fa0
        onerelid = {relId = 2015128626, dbId = 16564}
        toast_relid = 2015128673
        save_userid = 18712
        save_sec_context = 0
        save_nestlevel = 2
        rel_lock = true
        __func__ = "vacuum_rel"
#4  0x00000000006c5939 in vacuum (options=13, relations=0x22b8e70, 
    params=0x7ffff5f38a70, bstrategy=0x22b8cf8, isTopLevel=true)
    at vacuum.c:340
        vrel = 0x22b8e10
        cur = 0x22b8e48
        save_exception_stack = 0x7ffff5f38c20
        save_context_stack = 0x0
        local_sigjmp_buf = {{__jmpbuf = {35002584, 9176610699604406608, 
              4687312, 140737319770032, 0, 0, 9176610699520520528, 
              -9176587885370396336}, __mask_was_saved = 0, __saved_mask = {
              __val = {35906192, 140737319766400, 10964897, 6853, 35905936, 
                208, 35890568, 21474836680, 35890776, 140737319766560, 
                10927612, 208, 35890288, 35890288, 35890504, 35890544}}}}
        in_vacuum = true
        stmttype = 0xc03841 "VACUUM"
        in_outer_xact = false
        use_own_xacts = true
        __func__ = "vacuum"
#5  0x00000000006c54e0 in ExecVacuum (vacstmt=0x2161910, isTopLevel=true)
    at vacuum.c:141
        params = {freeze_min_age = 0, freeze_table_age = 0, 
          multixact_freeze_min_age = 0, multixact_freeze_table_age = 0, 
          is_wraparound = false, log_min_duration = -1}
        __func__ = "ExecVacuum"


errfinish one seems to be not useful?

(gdb) bt
#0  errfinish (dummy=0) at elog.c:415
#1  0x000000000054aefd in ShowTransactionStateRec (
    str=0xafed12 "StartTransaction", s=0xf9efa0) at xact.c:5157
#2  0x000000000054ad5c in ShowTransactionState (
    str=0xafed12 "StartTransaction") at xact.c:5130
#3  0x00000000005470c8 in StartTransaction () at xact.c:1961
#4  0x0000000000547db9 in StartTransactionCommand () at xact.c:2734
#5  0x00000000008cbe65 in start_xact_command () at postgres.c:2500
#6  0x00000000008c9772 in exec_simple_query (
    query_string=0x2160d28 "VACUUM FREEZE VERBOSE child.huawei_umts_ucell_201908;") at postgres.c:948
#7  0x00000000008cddf3 in PostgresMain (argc=1, argv=0x218c768, 
    dbname=0x218c5d0 "ts", username=0x218c5b0 "pryzbyj") at postgres.c:4182
#8  0x000000000082a098 in BackendRun (port=0x21818d0) at postmaster.c:4358
#9  0x0000000000829806 in BackendStartup (port=0x21818d0) at postmaster.c:4030
#10 0x0000000000825cab in ServerLoop () at postmaster.c:1707
#11 0x00000000008255dd in PostmasterMain (argc=3, argv=0x215b9a0)
    at postmaster.c:1380
#12 0x000000000074ba30 in main (argc=3, argv=0x215b9a0) at main.c:228



#0  errfinish (dummy=0) at elog.c:415
        edata = 0xfe7b80
        elevel = 0
        oldcontext = 0x215df90
        econtext = 0xa4d11a
        __func__ = "errfinish"
#1  0x000000000054aefd in ShowTransactionStateRec (
    str=0xafed12 "StartTransaction", s=0xf9efa0) at xact.c:5157
        buf = {data = 0x22462d8 "", len = 0, maxlen = 1024, cursor = 0}
        __func__ = "ShowTransactionStateRec"
#2  0x000000000054ad5c in ShowTransactionState (
    str=0xafed12 "StartTransaction") at xact.c:5130
No locals.
#3  0x00000000005470c8 in StartTransaction () at xact.c:1961
        s = 0xf9efa0
        vxid = {backendId = 5, localTransactionId = 16}
#4  0x0000000000547db9 in StartTransactionCommand () at xact.c:2734
        s = 0xf9efa0
        __func__ = "StartTransactionCommand"
#5  0x00000000008cbe65 in start_xact_command () at postgres.c:2500
No locals.

I looked at this briefly:

ts=# SELECT * FROM page_header(get_raw_page('child.huawei_umts_ucell_201908', 0));
lsn       | 3A49/66F44310
checksum  | 0
flags     | 4
lower     | 36
upper     | 1432
special   | 8192
pagesize  | 8192
version   | 4
prune_xid | 0

In case someone wants me to look at bits and pieces, I saved a copy of:
relfilenode         | 2015128626



Re: crash 11.5~ (and 11.4)

From
Andres Freund
Date:
Hi,

On 2019-08-07 14:29:28 -0500, Justin Pryzby wrote:
> Just found this, although I'm not sure what to do about it.  If it's corrupt
> table data, I can restore from backup.
> 
> ts=# VACUUM FREEZE VERBOSE child.huawei_umts_ucell_201908;
> INFO:  00000: aggressively vacuuming "child.huawei_umts_ucell_201908"
> LOCATION:  lazy_scan_heap, vacuumlazy.c:502
> ERROR:  XX001: found xmin 73850277 from before relfrozenxid 111408920
> LOCATION:  heap_prepare_freeze_tuple, heapam.c:6853

Ugh :(

We really need to add a error context to vacuumlazy that shows which
block is being processed.


> I confirmed I updated to 11.4 immediately on its release:
> [pryzbyj@database ~]$ ls -ltc /usr/pgsql-11/bin/postgres
> -rwxr-xr-x. 1 root root 7291736 Jun 20 07:09 /usr/pgsql-11/bin/postgres

Any chance you might not have restarted postgres at that time?


> That table would've been created shortly after midnight on Aug 1 when we loaded
> first data for the month.  So it was created and processed only on pg11.4,
> although the parent has probably been around since pg_upgrade last October.

I don't think there's a way the parent could play a role.


> Here's usable looking bt
> 
> #0  heap_prepare_freeze_tuple (tuple=0x7fd3d8fa0058, relfrozenxid=111408920, 
>     relminmxid=1846, cutoff_xid=111658731, cutoff_multi=1846, frz=0x223e930, 
>     totally_frozen_p=0x7ffff5f3848f) at heapam.c:6832
>         changed = 211
>         xmax_already_frozen = 127
>         xmin_frozen = false
>         freeze_xmax = false
>         xid = 0
>         __func__ = "heap_prepare_freeze_tuple"

Hm. That's a backtrace to what precisely? Are you sure it's the the
erroring call to heap_prepare_freeze_tuple? Because I think that's at
the function's start - which is why some of the stack variables have
bogus contents.

I think you'd need to set the breakpoint to heapam.c:6850 to be sure to
catch the error (while the error message heapam.c:6853, that's because
the line macro in some compilers expands to the end of the statement).


> errfinish one seems to be not useful?
> 
> (gdb) bt
> #0  errfinish (dummy=0) at elog.c:415
> #1  0x000000000054aefd in ShowTransactionStateRec (
>     str=0xafed12 "StartTransaction", s=0xf9efa0) at xact.c:5157
> #2  0x000000000054ad5c in ShowTransactionState (
>     str=0xafed12 "StartTransaction") at xact.c:5130
> #3  0x00000000005470c8 in StartTransaction () at xact.c:1961
> #4  0x0000000000547db9 in StartTransactionCommand () at xact.c:2734
> #5  0x00000000008cbe65 in start_xact_command () at postgres.c:2500
> #6  0x00000000008c9772 in exec_simple_query (
>     query_string=0x2160d28 "VACUUM FREEZE VERBOSE child.huawei_umts_ucell_201908;") at postgres.c:948
> #7  0x00000000008cddf3 in PostgresMain (argc=1, argv=0x218c768, 
>     dbname=0x218c5d0 "ts", username=0x218c5b0 "pryzbyj") at postgres.c:4182
> #8  0x000000000082a098 in BackendRun (port=0x21818d0) at postmaster.c:4358
> #9  0x0000000000829806 in BackendStartup (port=0x21818d0) at postmaster.c:4030
> #10 0x0000000000825cab in ServerLoop () at postmaster.c:1707
> #11 0x00000000008255dd in PostmasterMain (argc=3, argv=0x215b9a0)
>     at postmaster.c:1380
> #12 0x000000000074ba30 in main (argc=3, argv=0x215b9a0) at main.c:228

That's probably just the errfinish for a debug message. Might be easier
to set a breakpoint to pg_re_throw().


> I looked at this briefly:
> 
> ts=# SELECT * FROM page_header(get_raw_page('child.huawei_umts_ucell_201908', 0));
> lsn       | 3A49/66F44310
> checksum  | 0
> flags     | 4
> lower     | 36
> upper     | 1432
> special   | 8192
> pagesize  | 8192
> version   | 4
> prune_xid | 0

That's not necessarily the target block though, right?  It'd be useful
to get this for the block with the corruption, and perhaps one
before/after.  If the backtrace I commented on at the top is correct,
the relevant tuple was at "blkno = 34915" (visible a frame or two
up).

Also heap_page_items() for those pages would be useful - you might want
to not include the t_data column, as it has raw tuple data.


Could you show
* pg_controldata
* SELECT oid, datname, datfrozenxid, datminmxid FROM pg_database;

That should be easy to redact if necessary. It'd be useful to know which
database the corrupted table is in.

It'd also be helpful to get something roughly like
SELECT
    oid, oid::regclass,
    relkind, reltoastrelid,
    relfrozenxid, age(relfrozenxid),
    relminmxid, mxid_age(relminmxid),
    relpages,
    (SELECT txid_current())
FROM pg_class
WHERE relfrozenxid <> 0;

But I'm not sure if including the tablenames would be a problematic for
your case. Possibly a bit hard to infer "usage" correlations without,
but probably still worthwhile.

Greetings,

Andres Freund



Re: crash 11.5~ (and 11.4)

From
Justin Pryzby
Date:
On Wed, Aug 07, 2019 at 04:51:54PM -0700, Andres Freund wrote:
> Hi,
> 
> On 2019-08-07 14:29:28 -0500, Justin Pryzby wrote:
> > Just found this, although I'm not sure what to do about it.  If it's corrupt
> > table data, I can restore from backup.

In the meantime, I've renamed+uninherited the table and restored from backup,
with the broken table preserved.

However, the previous days' backup did this, so I used Monday's backup.

[pryzbyj@database ~]$ time sudo -u postgres pg_restore /srv/cdrperfbackup/ts/2019-08-06/curtables/ -t
huawei_umts_ucell_201908--verbose -d ts --host /tmp
 
pg_restore: connecting to database for restore
pg_restore: creating TABLE "child.huawei_umts_ucell_201908"
pg_restore: INFO:  partition constraint for table "huawei_umts_ucell_201908" is implied by existing constraints
pg_restore: processing data for table "child.huawei_umts_ucell_201908"
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 62710; 0 2015128626 TABLE DATA huawei_umts_ucell_201908 telsasoft
pg_restore: [archiver (db)] COPY failed for table "huawei_umts_ucell_201908": ERROR:  invalid byte sequence for
encoding"UTF8": 0xe7 0x28 0x9d
 
CONTEXT:  COPY huawei_umts_ucell_201908, line 104746

Also, I found core with this BT from my own manual invocation of ANALYZE.

(gdb) bt
#0  0x00000039674324f5 in raise () from /lib64/libc.so.6
#1  0x0000003967433cd5 in abort () from /lib64/libc.so.6
#2  0x0000000000a3c4b2 in ExceptionalCondition (conditionName=0xaad9e7 "!(j > attnum)", errorType=0xaad84c
"FailedAssertion",fileName=0xaad840 "heaptuple.c", lineNumber=582) at assert.c:54
 
#3  0x00000000004856a7 in nocachegetattr (tuple=0x1c116b08, attnum=1, tupleDesc=0x23332b0) at heaptuple.c:582
#4  0x000000000062981d in std_fetch_func (stats=0x22700c8, rownum=54991, isNull=0x7ffff5f3844f) at analyze.c:1718
#5  0x000000000062aa99 in compute_scalar_stats (stats=0x22700c8, fetchfunc=0x62942b <std_fetch_func>,
samplerows=120000,totalrows=227970) at analyze.c:2370
 
#6  0x00000000006270db in do_analyze_rel (onerel=0x7fd37e2afa50, options=2, params=0x7ffff5f38a70, va_cols=0x0,
acquirefunc=0x627e9e<acquire_sample_rows>, relpages=75494, inh=false,
 
    in_outer_xact=false, elevel=13) at analyze.c:579
#7  0x0000000000626616 in analyze_rel (relid=2015128626, relation=0x2161820, options=2, params=0x7ffff5f38a70,
va_cols=0x0,in_outer_xact=false, bstrategy=0x21f7848) at analyze.c:310
 
#8  0x00000000006c59b2 in vacuum (options=2, relations=0x21f79c0, params=0x7ffff5f38a70, bstrategy=0x21f7848,
isTopLevel=true)at vacuum.c:357
 
#9  0x00000000006c54e0 in ExecVacuum (vacstmt=0x2161910, isTopLevel=true) at vacuum.c:141
#10 0x00000000008d1d7e in standard_ProcessUtility (pstmt=0x21619d0, queryString=0x2160d28 "ANALYZE child.
huawei_umts_ucell_201908;",context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0,
 
    dest=0x2161cc0, completionTag=0x7ffff5f39100 "") at utility.c:670

#3  0x00000000004856a7 in nocachegetattr (tuple=0x1c116b08, attnum=1, tupleDesc=0x23332b0) at heaptuple.c:582
        natts = 554
        j = 1
        tup = 0x1c116b20
        tp = 0x1c116b85
"\374\023\026\213^s#\347(\235=\326\321\067\032\245\321B\026}܋FS\375\244\003\065\336\277;\252O\006\065\320\353\211}F\237\373B\243\357J~\270\"\230ƣ\024xǍ\334\377\202\277S\031\375\351\003\220{\004"
        bp = 0x1c116b37
"\270\027$U}\232\246\235\004\255\331\033\006Qp\376E\316h\376\363\247\366Նgy7\311E\224~F\274\023ϋ%\216,\221\331@\024\363\233\070\275\004\254L\217t\262X\227\352\346\347\371\070\321ш\221\350fc\316\r\356\351h\275\213\230\360\203\374\023\026\213^s#\347(\235=\326\321\067\032\245\321B\026}܋FS\375\244\003\065\336\277;\252O\006\065\320\353\211}F\237\373B\243\357J~\270\"\230ƣ\024xǍ\334\377\202\277S\031\375\351\003\220{\004"
        slow = false
        off = -1
#4  0x000000000062981d in std_fetch_func (stats=0x22700c8, rownum=54991, isNull=0x7ffff5f3844f) at analyze.c:1718
        attnum = 2
        tuple = 0x1c116b08
        tupDesc = 0x23332b0
#5  0x000000000062aa99 in compute_scalar_stats (stats=0x22700c8, fetchfunc=0x62942b <std_fetch_func>,
samplerows=120000,totalrows=227970) at analyze.c:2370
 
        value = 546120480
        isnull = false
        i = 54991
        null_cnt = 0
        nonnull_cnt = 54991
        toowide_cnt = 0
---Type <return> to continue, or q <return> to quit---
        total_width = 274955
        is_varlena = true
        is_varwidth = true
        corr_xysum = 3.9525251667299724e-323
        ssup = {ssup_cxt = 0x22ae080, ssup_collation = 100, ssup_reverse = false, ssup_nulls_first = false, ssup_attno
=0, ssup_extra = 0x22ae5b0, comparator = 0xa06720 <varstrfastcmp_locale>,
 
          abbreviate = false, abbrev_converter = 0, abbrev_abort = 0, abbrev_full_comparator = 0}
        values = 0x204f9970
        values_cnt = 54991
        tupnoLink = 0x206ce5c0
        track = 0x22ae198
        track_cnt = 0
        num_mcv = 100
        num_bins = 100
        mystats = 0x2270490

> > ts=# VACUUM FREEZE VERBOSE child.huawei_umts_ucell_201908;
> > INFO:  00000: aggressively vacuuming "child.huawei_umts_ucell_201908"
> > LOCATION:  lazy_scan_heap, vacuumlazy.c:502
> > ERROR:  XX001: found xmin 73850277 from before relfrozenxid 111408920
> > LOCATION:  heap_prepare_freeze_tuple, heapam.c:6853
> 
> Ugh :(
> 
> We really need to add a error context to vacuumlazy that shows which
> block is being processed.

> > I confirmed I updated to 11.4 immediately on its release:
> > [pryzbyj@database ~]$ ls -ltc /usr/pgsql-11/bin/postgres
> > -rwxr-xr-x. 1 root root 7291736 Jun 20 07:09 /usr/pgsql-11/bin/postgres
> 
> Any chance you might not have restarted postgres at that time?

I don't think so, nagios would've been yelling at me.  Also..I found I have a
log of pg_settings, which shows server_version updated 2019-06-20
07:05:01.425645.  Is there a bug in 11.3 which could explain it ?

> > Here's usable looking bt
> > 
> > #0  heap_prepare_freeze_tuple (tuple=0x7fd3d8fa0058, relfrozenxid=111408920, 
> >     relminmxid=1846, cutoff_xid=111658731, cutoff_multi=1846, frz=0x223e930, 
> >     totally_frozen_p=0x7ffff5f3848f) at heapam.c:6832
> >         changed = 211
> >         xmax_already_frozen = 127
> >         xmin_frozen = false
> >         freeze_xmax = false
> >         xid = 0
> >         __func__ = "heap_prepare_freeze_tuple"
> 
> Hm. That's a backtrace to what precisely? Are you sure it's the the
> erroring call to heap_prepare_freeze_tuple? Because I think that's at
> the function's start - which is why some of the stack variables have
> bogus contents.

Yes, I had just done this: b heap_prepare_freeze_tuple.

> I think you'd need to set the breakpoint to heapam.c:6850 to be sure to
> catch the error (while the error message heapam.c:6853, that's because
> the line macro in some compilers expands to the end of the statement).

I did 6848 and got;

#0  heap_prepare_freeze_tuple (tuple=0x7fd403a02058, relfrozenxid=111408920, relminmxid=1846, cutoff_xid=111665057,
cutoff_multi=1846,frz=0x22aa6f8, totally_frozen_p=0x7ffff5f3848f) at heapam.c:6849
 
        changed = false
        xmax_already_frozen = false
        xmin_frozen = false
        freeze_xmax = false
        xid = 73850277
        __func__ = "heap_prepare_freeze_tuple"

> > errfinish one seems to be not useful?
> > 
> > (gdb) bt
> > #0  errfinish (dummy=0) at elog.c:415
> > #1  0x000000000054aefd in ShowTransactionStateRec (
> >     str=0xafed12 "StartTransaction", s=0xf9efa0) at xact.c:5157
> > [...]
>
> That's probably just the errfinish for a debug message. Might be easier
> to set a breakpoint to pg_re_throw().

Right, that works.

(gdb) bt
#0  pg_re_throw () at elog.c:1730
#1  0x0000000000a3cb1e in errfinish (dummy=0) at elog.c:467
#2  0x00000000004e72c4 in heap_prepare_freeze_tuple (tuple=0x7fd403a02058, relfrozenxid=111408920, relminmxid=1846,
cutoff_xid=111665065,cutoff_multi=1846, frz=0x2260558, totally_frozen_p=0x7ffff5f3848f) at heapam.c:6850
 
#3  0x00000000006c8e6e in lazy_scan_heap (onerel=0x7fd37d5f99c8, options=13, vacrelstats=0x22603b8, Irel=0x2260500,
nindexes=3,aggressive=true) at vacuumlazy.c:1151 

> > I looked at this briefly:
> > 
> > ts=# SELECT * FROM page_header(get_raw_page('child.huawei_umts_ucell_201908', 0));
> > lsn       | 3A49/66F44310
> > checksum  | 0
> > flags     | 4
> > lower     | 36
> > upper     | 1432
> > special   | 8192
> > pagesize  | 8192
> > version   | 4
> > prune_xid | 0
> 
> That's not necessarily the target block though, right?  It'd be useful
> to get this for the block with the corruption, and perhaps one
> before/after.  If the backtrace I commented on at the top is correct,
> the relevant tuple was at "blkno = 34915" (visible a frame or two
> up).

Ah, right.

ts=# SELECT * FROM page_header(get_raw_page('child.huawei_umts_ucell_201908', 34914));
      lsn      | checksum | flags | lower | upper | special | pagesize | version | prune_xid 
---------------+----------+-------+-------+-------+---------+----------+---------+-----------
 3A4A/4F2192B8 |        0 |     0 |    36 |  1520 |    8192 |     8192 |       4 |         0

ts=# SELECT * FROM page_header(get_raw_page('child.huawei_umts_ucell_201908', 34915));
 3A4A/4F21C268 |        0 |     0 |    36 |  1520 |    8192 |     8192 |       4 |         0

ts=# SELECT * FROM page_header(get_raw_page('child.huawei_umts_ucell_201908', 34916));
 3A4A/4F21DC90 |        0 |     0 |    36 |  1496 |    8192 |     8192 |       4 |         0

> Also heap_page_items() for those pages would be useful - you might want
> to not include the t_data column, as it has raw tuple data.

ts=# SELECT lp, lp_off, lp_flags, lp_len, t_xmin, t_xmax, t_field3, t_ctid, t_infomask2, t_infomask, t_hoff, t_bits,
t_oidFROM heap_page_items(get_raw_page('child.huawei_umts_ucell_201908', 34914));
 
 lp | lp_off | lp_flags | lp_len |  t_xmin   | t_xmax | t_field3 |  t_ctid   | t_infomask2 | t_infomask | t_hoff |
t_bits... | t_oid 
 

  1 |   5952 |        1 |   2236 | 111659903 |      0 |        0 | (34914,1) |         554 |       2311 |     96 |
11111111111111111111111111111111111111011100111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111000001111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111001111111111111111111111001111111111111111111111011111111111111111011111111101101111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111000000000001111111111111111111111111111111111111111111100111111000000
|     
 
  2 |   3736 |        1 |   2212 | 111659903 |      0 |        0 | (34914,2) |         554 |       2311 |     96 |
11111111111111111111111111111111111111011100111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111100000000011111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111001111111111111111111111000111111111111111111111011111111111111111010111111101101111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111000000000001111111111111111111111111111111111111111111100111111000000
|     
 
  3 |   1520 |        1 |   2216 | 111659903 |      0 |        0 | (34914,3) |         554 |       2311 |     96 |
11111111111111111111111111111111111111011100111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111100000001011111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111001111111111111111111111000111111111111111111111011111111111111111011111111101101111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111000000000001111111111111111111111111111111111111111111100111111000000
|     
 

ts=# SELECT lp, lp_off, lp_flags, lp_len, t_xmin, t_xmax, t_field3, t_ctid, t_infomask2, t_infomask, t_hoff, t_bits,
t_oidFROM heap_page_items(get_raw_page('child.huawei_umts_ucell_201908', 34915));
 
  1 |   5976 |        1 |   2212 | 111659903 |      0 |        0 | (34915,1) |         554 |       2311 |     96 |
11111111111111111111111111111111111111011100111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111100000001011111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111001111111111111111111111000111111111111111111111011111111111111111010111111101101111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111000000000001111111111111111111111111111111111111111111100111111000000
|     
 
  2 |   3744 |        1 |   2232 | 111659903 |      0 |        0 | (34915,2) |         554 |       2311 |     96 |
11111111111111111111111111111111111111011100111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111100001111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111001111111111111111111111001111111111111111111111011111111111111111010111111101101111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111000000000001111111111111111111111111111111111111111111100111111000000
|     
 
  3 |   1520 |        1 |   2220 | 111659903 |      0 |        0 | (34915,3) |         554 |       2311 |     96 |
11111111111111111111111111111111111111011100111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111000001011111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111001111111111111111111111000111111111111111111111011111111111111111010111111101101111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111000000000001111111111111111111111111111111111111111111100111111000000
|     
 

ts=# SELECT lp, lp_off, lp_flags, lp_len, t_xmin, t_xmax, t_field3, t_ctid, t_infomask2, t_infomask, t_hoff, t_bits,
t_oidFROM heap_page_items(get_raw_page('child.huawei_umts_ucell_201908', 34916));
 
  1 |   5952 |        1 |   2236 | 111659903 |      0 |        0 | (34916,1) |         554 |       2311 |     96 |
11111111111111111111111111111111111111011100111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111000001111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111001111111111111111111111001111111111111111111111011111111111111111011111111101101111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111000000000001111111111111111111111111111111111111111111100111111000000
|     
 
  2 |   3736 |        1 |   2212 | 111659903 |      0 |        0 | (34916,2) |         554 |       2311 |     96 |
11111111111111111111111111111111111111011100111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111100000001011111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111001111111111111111111111000111111111111111111111011111111111111111010111111101101111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111000000000001111111111111111111111111111111111111111111100111111000000
|     
 
  3 |   1496 |        1 |   2240 | 111659903 |      0 |        0 | (34916,3) |         554 |       2311 |     96 |
11111111111111111111111111111111111111011100111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111000001111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111001111111111111111111111001111111111111111111111011111111111111111010111111101101111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111001100000001111111111111111111111111111111111111111111100111111000000
|      
> Could you show
> * pg_controldata

[pryzbyj@database ~]$ sudo -u postgres ./src/postgresql.bin/bin/pg_controldata -D /var/lib/pgsql/11/data
[sudo] password for pryzbyj:
pg_control version number:            1100
Catalog version number:               201809051
Database system identifier:           6616072377454496350
Database cluster state:               in production
pg_control last modified:             Wed 07 Aug 2019 06:32:05 PM MDT
Latest checkpoint location:           3A4C/D1BB5BD8
Latest checkpoint's REDO location:    3A4C/CCC1DF90
Latest checkpoint's REDO WAL file:    0000000100003A4C000000CC
Latest checkpoint's TimeLineID:       1
Latest checkpoint's PrevTimeLineID:   1
Latest checkpoint's full_page_writes: on
Latest checkpoint's NextXID:          0:111664949
Latest checkpoint's NextOID:          2069503852
Latest checkpoint's NextMultiXactId:  1846
Latest checkpoint's NextMultiOffset:  3891
Latest checkpoint's oldestXID:        33682470
Latest checkpoint's oldestXID's DB:   16564
Latest checkpoint's oldestActiveXID:  111664945
Latest checkpoint's oldestMultiXid:   1484
Latest checkpoint's oldestMulti's DB: 16564
Latest checkpoint's oldestCommitTsXid:0
Latest checkpoint's newestCommitTsXid:0
Time of latest checkpoint:            Wed 07 Aug 2019 06:31:35 PM MDT
Fake LSN counter for unlogged rels:   0/1
Minimum recovery ending location:     0/0
Min recovery ending loc's timeline:   0
Backup start location:                0/0
Backup end location:                  0/0
End-of-backup record required:        no
wal_level setting:                    replica
wal_log_hints setting:                off
max_connections setting:              200
max_worker_processes setting:         8
max_prepared_xacts setting:           0
max_locks_per_xact setting:           128
track_commit_timestamp setting:       off
Maximum data alignment:               8
Database block size:                  8192
Blocks per segment of large relation: 131072
WAL block size:                       8192
Bytes per WAL segment:                16777216
Maximum length of identifiers:        64
Maximum columns in an index:          32
Maximum size of a TOAST chunk:        1996
Size of a large-object chunk:         2048
Date/time type storage:               64-bit integers
Float4 argument passing:              by value
Float8 argument passing:              by value
Data page checksum version:           0
Mock authentication nonce:            ff863744b99b849579d561c5117157c78a8ced0563df824f3035f70810fec534

> * SELECT oid, datname, datfrozenxid, datminmxid FROM pg_database;
ts=# SELECT oid, datname, datfrozenxid, datminmxid FROM pg_database;
  oid  |  datname  | datfrozenxid | datminmxid 
-------+-----------+--------------+------------
 13529 | template0 |    103564559 |       1846
 16400 | template1 |     52362404 |       1486
 16443 | postgres  |     46060229 |       1485
 16564 | ts        |     33682470 |       1484
(4 rows)

> That should be easy to redact if necessary. It'd be useful to know which
> database the corrupted table is in.
> 
> It'd also be helpful to get something roughly like
> SELECT oid, oid::regclass, relkind, reltoastrelid, relfrozenxid, age(relfrozenxid), relminmxid, mxid_age(relminmxid),
relpages,(SELECT txid_current()) FROM pg_class WHERE relfrozenxid <> 0;
 

> But I'm not sure if including the tablenames would be a problematic for
> your case. Possibly a bit hard to infer "usage" correlations without,
> but probably still worthwhile.

I'll send to you individually, I don't think it's secret, but it's 60k lines
long...  Its OID verifies to me that the table was created on 8/1, and not 7/1,
which was possible.  Week-old backup also verifies that.

Thanks,
Justin