Thread: Re: BUG #16696: Backend crash in llvmjit

* PG Bug reporting form (noreply@postgresql.org) wrote:

> Environment details:
> - FreeBSD 12.1 amd64
> - PostgreSQL 13.0 (built from FreeBSD ports)
> - llvm-10.0.1 (build from FreeBSD ports)

My bad, it's actually llvm-9.0.1. Multiple llvm versions are installed on
the system, and PostgreSQL uses llvm9:

ldd /usr/local/lib/postgresql/llvmjit.so | grep LLVM
    libLLVM-9.so => /usr/local/llvm90/lib/libLLVM-9.so (0x800e00000)

-- 
Dmitry Marakasov   .   55B5 0596 FF1E 8D84 5F56  9510 D35A 80DD F9D2 F77D
amdmi3@amdmi3.ru  ..:              https://github.com/AMDmi3

The following bug has been logged on the website:

Bug reference:      16696
Logged by:          Dmitry Marakasov
Email address:      amdmi3@amdmi3.ru
PostgreSQL version: 13.0
Operating system:   FreeBSD 12.1
Description:

Hi!

I've ran into llvmjit related backend crash with a query involving unnesting
a column which contains array of arrays in jsonb format. I've been able to
isolate a query which triggers the problem:

BEGIN;
    CREATE TEMP TABLE test AS
    SELECT
        (
            SELECT
jsonb_agg('["foofoofoo","foofoofoo","*","*","*","*","*","*","*","*",true,false]'::jsonb)
            FROM generate_series(1,1000)
        ) AS data
    FROM generate_series(1, 10000);

    SELECT COUNT(*) FROM(
        SELECT
            jsonb_array_elements(data)->>0,
            jsonb_array_elements(data)->>1,
            jsonb_array_elements(data)->>2,
            jsonb_array_elements(data)->>3,
            jsonb_array_elements(data)->>4,
            jsonb_array_elements(data)->>5,
            jsonb_array_elements(data)->>6,
            jsonb_array_elements(data)->>7,
            jsonb_array_elements(data)->>8,
            jsonb_array_elements(data)->>9,
            (jsonb_array_elements(data)->>10)::boolean,
            (jsonb_array_elements(data)->>11)::boolean
        FROM test
    ) AS t;
ROLLBACK;

The result is 100% reproduciby on my setup:

% psql < test.sql
BEGIN
SELECT 10000
server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.
connection to server was lost

Syslog: "pid 53852 (postgres), jid 0, uid 770: exited on signal 11 (core
dumped)"

Expected result which is achievable on PostgreSQL compiled without LLVM
support:

% psql < test.sql
BEGIN
SELECT 10000
  count   
----------
 10000000
(1 row)

ROLLBACK

Environment details:
- FreeBSD 12.1 amd64
- PostgreSQL 13.0 (built from FreeBSD ports)
- llvm-10.0.1 (build from FreeBSD ports)
- version(): PostgreSQL 13.0 on amd64-portbld-freebsd12.1, compiled by
FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM
8.0.1), 64-bit
- the config is as far as I can tell default: http://dpaste.com//AJ4RCJ7DG

Note that the problem may be related to some buffer overflow, as using less
rows in test table or less nested arrays in the jsonb field makes the
problem go away.

Backtrace:

(lldb) bt
* thread #1, name = 'postgres', stop reason = signal SIGSEGV
  * frame #0: 0x0000000810f4a70b
    frame #1: 0x000000080bfded8d
llvmjit.so`ExecRunCompiledExpr(state=0x000000080b18a850,
econtext=0x0000000801bc2a80, isNull=0x00007fffffffddd7) at
llvmjit_expr.c:2424:9
    frame #2: 0x00000000007dd70b
postgres`ExecEvalExprSwitchContext(state=0x000000080b18a850,
econtext=0x0000000801bc2a80, isNull=0x00007fffffffddd7) at
executor.h:313:13
    frame #3: 0x00000000007dd669
postgres`ExecProject(projInfo=0x000000080b18a848) at executor.h:347:9
    frame #4: 0x00000000007dd42f
postgres`ExecResult(pstate=0x0000000801bc2970) at nodeResult.c:136:10
    frame #5: 0x00000000007a1035
postgres`ExecProcNodeFirst(node=0x0000000801bc2970) at
execProcnode.c:450:9
    frame #6: 0x00000000007b5e22
postgres`ExecProcNode(node=0x0000000801bc2970) at executor.h:245:9
    frame #7: 0x00000000007b5a4c
postgres`fetch_input_tuple(aggstate=0x0000000801bc2398) at
nodeAgg.c:589:10
    frame #8: 0x00000000007b5678
postgres`agg_retrieve_direct(aggstate=0x0000000801bc2398) at
nodeAgg.c:2356:17
    frame #9: 0x00000000007b250e postgres`ExecAgg(pstate=0x0000000801bc2398)
at nodeAgg.c:2171:14
    frame #10: 0x00000000007a1035
postgres`ExecProcNodeFirst(node=0x0000000801bc2398) at
execProcnode.c:450:9
    frame #11: 0x0000000000799db2
postgres`ExecProcNode(node=0x0000000801bc2398) at executor.h:245:9
    frame #12: 0x0000000000796161
postgres`ExecutePlan(estate=0x0000000801bc2118,
planstate=0x0000000801bc2398, use_parallel_mode=false, operation=CMD_SELECT,
sendTuples=true, numberTuples=0, direction=ForwardScanDirection,
dest=0x000000080b565f00, execute_once=true) at execMain.c:1646:10
    frame #13: 0x000000000079602e
postgres`standard_ExecutorRun(queryDesc=0x000000080220e918,
direction=ForwardScanDirection, count=0, execute_once=true) at
execMain.c:364:3
    frame #14: 0x0000000000795e84
postgres`ExecutorRun(queryDesc=0x000000080220e918,
direction=ForwardScanDirection, count=0, execute_once=true) at
execMain.c:308:3
    frame #15: 0x00000000009f4928
postgres`PortalRunSelect(portal=0x000000080b0c3118, forward=true, count=0,
dest=0x000000080b565f00) at pquery.c:912:4
    frame #16: 0x00000000009f438d
postgres`PortalRun(portal=0x000000080b0c3118, count=9223372036854775807,
isTopLevel=true, run_once=true, dest=0x000000080b565f00,
altdest=0x000000080b565f00, qc=0x00007fffffffe3a0) at pquery.c:756:18
    frame #17: 0x00000000009efc10
postgres`exec_simple_query(query_string="SELECT COUNT(*)

FROM(\n\t\tSELECT\n\t\t\tjsonb_array_elements(data)->>0,\n\t\t\tjsonb_array_elements(data)->>1,\n\t\t\tjsonb_array_elements(data)->>2,\n\t\t\tjsonb_array_elements(data)->>3,\n\t\t\tjsonb_array_elements(data)->>4,\n\t\t\tjsonb_array_elements(data)->>5,\n\t\t\tjsonb_array_elements(data)->>6,\n\t\t\tjsonb_array_elements(data)->>7,\n\t\t\tjsonb_array_elements(data)->>8,\n\t\t\tjsonb_array_elements(data)->>9,\n\t\t\t(jsonb_array_elements(data)->>10)::boolean,\n\t\t\t(jsonb_array_elements(data)->>11)::boolean\n\t\tFROM
test\n\t) AS t;") at postgres.c:1239:10
    frame #18: 0x00000000009eeec9 postgres`PostgresMain(argc=1,
argv=0x000000080b0ac730, dbname="repology", username="repology") at
postgres.c:4315:7
    frame #19: 0x000000000092bcb7
postgres`BackendRun(port=0x000000080b0a5000) at postmaster.c:4536:2
    frame #20: 0x000000000092b090
postgres`BackendStartup(port=0x000000080b0a5000) at postmaster.c:4220:3
    frame #21: 0x000000000092a073 postgres`ServerLoop at
postmaster.c:1739:7
    frame #22: 0x0000000000927d52 postgres`PostmasterMain(argc=3,
argv=0x00007fffffffeb48) at postmaster.c:1412:11
    frame #23: 0x0000000000819bcf postgres`main(argc=3,
argv=0x00007fffffffeb48) at main.c:210:3
    frame #24: 0x00000000004cb10f postgres`_start(ap=<unavailable>,
cleanup=<unavailable>) at crt1.c:76:7

Some locals at #1

(lldb) print *state
(ExprState) $6 = {
  tag = T_ExprState
  flags = '\0'
  resnull = false
  resvalue = 34388814424
  resultslot = 0x0000000801bc3ec0
  steps = 0x000000080b4a7f38
  evalfunc = 0x0000000810f4a120
  expr = 0x000000080b564940
  evalfunc_private = 0x000000080b4a9fe0
  steps_len = 39
  steps_alloc = 64
  parent = 0x0000000801bc2970
  ext_params = 0x0000000000000000
  innermost_caseval = 0x0000000000000000
  innermost_casenull = 0x0000000000000000
  innermost_domainval = 0x0000000000000000
  innermost_domainnull = 0x0000000000000000
}
(lldb) print *econtext
(ExprContext) $7 = {
  type = T_ExprContext
  ecxt_scantuple = 0x0000000000000000
  ecxt_innertuple = 0x0000000000000000
  ecxt_outertuple = 0x0000000801bc31e0
  ecxt_per_query_memory = 0x0000000801bc2000
  ecxt_per_tuple_memory = 0x0000000801bba000
  ecxt_param_exec_vals = 0x0000000000000000
  ecxt_param_list_info = 0x0000000000000000
  ecxt_aggvalues = 0x0000000000000000
  ecxt_aggnulls = 0x0000000000000000
  caseValue_datum = 0
  caseValue_isNull = true
  domainValue_datum = 0
  domainValue_isNull = true
  ecxt_estate = 0x0000000801bc2118
  ecxt_callbacks = 0x0000000000000000
}
(lldb) print *isNull
(bool) $8 = false
(lldb) print *(CompiledExprState*)state->evalfunc_private
(CompiledExprState) $16 = {
  context = 0x000000080b126050
  funcname = 0x000000080b4a90d8 "evalexpr_0_2"
}

Hi

On Mon, Nov 2, 2020, at 13:45, Dmitry Marakasov wrote:
> * PG Bug reporting form (noreply@postgresql.org) wrote:
> 
> > Environment details:
> > - FreeBSD 12.1 amd64
> > - PostgreSQL 13.0 (built from FreeBSD ports)
> > - llvm-10.0.1 (build from FreeBSD ports)
> 
> My bad, it's actually llvm-9.0.1. Multiple llvm versions are installed on
> the system, and PostgreSQL uses llvm9:
> 
> ldd /usr/local/lib/postgresql/llvmjit.so | grep LLVM
>     libLLVM-9.so => /usr/local/llvm90/lib/libLLVM-9.so (0x800e00000)

Could you try generating a backtrace after turning jit_debugging_support on? That might give a bit more information.

I'll check once I'm home whether I can reproduce in my environment.

Regards,

Andres

* Andres Freund (andres@anarazel.de) wrote:

> On Mon, Nov 2, 2020, at 13:45, Dmitry Marakasov wrote:
> > * PG Bug reporting form (noreply@postgresql.org) wrote:
> > 
> > > Environment details:
> > > - FreeBSD 12.1 amd64
> > > - PostgreSQL 13.0 (built from FreeBSD ports)
> > > - llvm-10.0.1 (build from FreeBSD ports)
> > 
> > My bad, it's actually llvm-9.0.1. Multiple llvm versions are installed on
> > the system, and PostgreSQL uses llvm9:
> > 
> > ldd /usr/local/lib/postgresql/llvmjit.so | grep LLVM
> >     libLLVM-9.so => /usr/local/llvm90/lib/libLLVM-9.so (0x800e00000)
> 
> Could you try generating a backtrace after turning jit_debugging_support on? That might give a bit more information.

It doesn't give too much apart from the function name. Here's gdb
output:

#0  0x0000000810f5c70b in evalexpr_0_2 ()
No symbol table info available.
#1  0x000000080bfded8d in ExecRunCompiledExpr (state=0x80b18a850, econtext=0x801bc2a80, isNull=0x7fffffffddd7) at
llvmjit_expr.c:2424
        cstate = 0x80b4a9fe0
        func = 0x810f5c120 <evalexpr_0_2>
#2  0x00000000007dd70b in ExecEvalExprSwitchContext (state=0x80b18a850, econtext=0x801bc2a80, isNull=0x7fffffffddd7) at
../../../src/include/executor/executor.h:313
        retDatum = 140737488346496
        oldContext = 0x801bc2000
#3  0x00000000007dd669 in ExecProject (projInfo=0x80b18a848) at ../../../src/include/executor/executor.h:347
        econtext = 0x801bc2a80
        state = 0x80b18a850
        slot = 0x801bc3ec0
        isnull = false
#4  0x00000000007dd42f in ExecResult (pstate=0x801bc2970) at nodeResult.c:136
        node = 0x801bc2970
        outerTupleSlot = 0x801bc31e0
        outerPlan = 0x801bc2b10
        econtext = 0x801bc2a80
#5  0x00000000007a1035 in ExecProcNodeFirst (node=0x801bc2970) at execProcnode.c:450
No locals.
#6  0x00000000007b5e22 in ExecProcNode (node=0x801bc2970) at ../../../src/include/executor/executor.h:245
No locals.
#7  0x00000000007b5a4c in fetch_input_tuple (aggstate=0x801bc2398) at nodeAgg.c:589
        slot = 0x7a9ef7 <ReScanExprContext+39>
#8  0x00000000007b5678 in agg_retrieve_direct (aggstate=0x801bc2398) at nodeAgg.c:2356
        node = 0x801bb5dd0
        econtext = 0x801bc28e0
        tmpcontext = 0x801bc27c0
        peragg = 0x80b4aa6f0
        pergroups = 0x80b4aa990
        outerslot = 0x1a
        firstSlot = 0x80b4aa000
        result = 0xffff0b4aa12c
        hasGroupingSets = false
        numGroupingSets = 1
        currentSet = 32767
        nextSetSize = 0
        numReset = 1
        i = 1
#9  0x00000000007b250e in ExecAgg (pstate=0x801bc2398) at nodeAgg.c:2171
        node = 0x801bc2398
        result = 0x0
#10 0x00000000007a1035 in ExecProcNodeFirst (node=0x801bc2398) at execProcnode.c:450
No locals.
#11 0x0000000000799db2 in ExecProcNode (node=0x801bc2398) at ../../../src/include/executor/executor.h:245
No locals.
#12 0x0000000000796161 in ExecutePlan (estate=0x801bc2118, planstate=0x801bc2398, use_parallel_mode=false,
operation=CMD_SELECT,sendTuples=true, numberTuples=0, direction=ForwardScanDirection, 
 
    dest=0x80b565f00, execute_once=true) at execMain.c:1646
        slot = 0x80b565f50
        current_tuple_count = 0
#13 0x000000000079602e in standard_ExecutorRun (queryDesc=0x80220e918, direction=ForwardScanDirection, count=0,
execute_once=true)at execMain.c:364
 
        estate = 0x801bc2118
        operation = CMD_SELECT
        dest = 0x80b565f00
        sendTuples = true
        oldcontext = 0x80220e800
#14 0x0000000000795e84 in ExecutorRun (queryDesc=0x80220e918, direction=ForwardScanDirection, count=0,
execute_once=true)at execMain.c:308
 
No locals.
#15 0x00000000009f4928 in PortalRunSelect (portal=0x80b0c3118, forward=true, count=0, dest=0x80b565f00) at
pquery.c:912
        queryDesc = 0x80220e918
        direction = ForwardScanDirection
        nprocessed = 294333
#16 0x00000000009f438d in PortalRun (portal=0x80b0c3118, count=9223372036854775807, isTopLevel=true, run_once=true,
dest=0x80b565f00,altdest=0x80b565f00, qc=0x7fffffffe3a0) at pquery.c:756
 
        _save_exception_stack = 0x7fffffffe580
        _save_context_stack = 0x0
        _local_sigjmp_buf = {{_sjb = {10437281, -4571079317099439115, 140737488347448, 140737488347776,
140737488350016,0, 140737488350024, 140737488350056, -64641, 2, 4294967297, 34359738368}}}
 
        _do_rethrow = false
        result = false
        nprocessed = 34395449568
        saveTopTransactionResourceOwner = 0x80b0ae3a0
        saveTopTransactionContext = 0x80b0cb000
        saveActivePortal = 0x0
        saveResourceOwner = 0x80b0ae3a0
        savePortalContext = 0x0
        saveMemoryContext = 0x80b0cb000
#17 0x00000000009efc10 in exec_simple_query (
    query_string=0x801b87118 "SELECT COUNT(*)
FROM(\n\t\tSELECT\n\t\t\tjsonb_array_elements(data)->>0,\n\t\t\tjsonb_array_elements(data)->>1,\n\t\t\tjsonb_array_elements(data)->>2,\n\t\t\tjsonb_array_elements(data)->>3,\n\t\t\tjsonb_array_elements(data)"...)
atpostgres.c:1239
 
        snapshot_set = true
        per_parsetree_context = 0x0
        plantree_list = 0x80b565eb0
        parsetree = 0x801b8ecd0
        commandTag = CMDTAG_SELECT
        qc = {commandTag = CMDTAG_UNKNOWN, nprocessed = 0}
        querytree_list = 0x801bb9160
        portal = 0x80b0c3118
        receiver = 0x80b565f00
        format = 0
        parsetree_item__state = {l = 0x801b8ed00, i = 0}
        dest = DestRemote
        oldcontext = 0x80b0cb000
        parsetree_list = 0x801b8ed00
        parsetree_item = 0x801b8ed18
        save_log_statement_stats = false
        was_logged = false
        use_implicit_block = false
        msec_str =
"\340\343\377\377\377\177\000\000\022S\201\000\000\000\000\000\377q\t\000\000\000\000\000\365\033'h\356\001\000"
#18 0x00000000009eeec9 in PostgresMain (argc=1, argv=0x80b0ac730, dbname=0x80b0ac6a8 "repology", username=0x80b0ac688
"repology")at postgres.c:4315
 
        query_string = 0x801b87118 "SELECT COUNT(*)
FROM(\n\t\tSELECT\n\t\t\tjsonb_array_elements(data)->>0,\n\t\t\tjsonb_array_elements(data)->>1,\n\t\t\tjsonb_array_elements(data)->>2,\n\t\t\tjsonb_array_elements(data)->>3,\n\t\t\tjsonb_array_elements(data)"...
        firstchar = 81
        input_message = {
          data = 0x801b87118 "SELECT COUNT(*)
FROM(\n\t\tSELECT\n\t\t\tjsonb_array_elements(data)->>0,\n\t\t\tjsonb_array_elements(data)->>1,\n\t\t\tjsonb_array_elements(data)->>2,\n\t\t\tjsonb_array_elements(data)->>3,\n\t\t\tjsonb_array_elements(data)"...,
len= 495, maxlen = 1024, cursor = 495}
 
        local_sigjmp_buf = {{_sjb = {10414768, 3, 140737488348136, 140737488348704, 140737488350016, 0,
140737488350024,140737488350056, 34388575103, 0, 0, 1}}}
 
        send_ready_for_query = false
        disable_idle_in_transaction_timeout = false
#19 0x000000000092bcb7 in BackendRun (port=0x80b0a5000) at postmaster.c:4536
        av = 0x80b0ac730
        maxac = 2
        ac = 1
        i = 1
#20 0x000000000092b090 in BackendStartup (port=0x80b0a5000) at postmaster.c:4220
        bn = 0x801b97510
        pid = 0
#21 0x000000000092a073 in ServerLoop () at postmaster.c:1739
        port = 0x80b0a5000
        i = 2
        rmask = {__fds_bits = {256, 0 <repeats 15 times>}}
        selres = 1
        now = 1604363041
        readmask = {__fds_bits = {448, 0 <repeats 15 times>}}
        nSockets = 9
        last_lockfile_recheck_time = 1604363041
        last_touch_time = 1604363041
#22 0x0000000000927d52 in PostmasterMain (argc=3, argv=0x7fffffffeb48) at postmaster.c:1412
        opt = -1
        status = 0
        userDoption = 0x800c881a0 "/var/db/postgres/data13"
        listen_addr_saved = true
        i = 64
        output_config_variable = 0x0
#23 0x0000000000819bcf in main (argc=3, argv=0x7fffffffeb48) at main.c:210
        do_check_root = true

Disassembly:

Dump of assembler code for function evalexpr_0_2:
   0x0000000810f5c120 <+0>:    push   %rbp
   0x0000000810f5c121 <+1>:    mov    %rsp,%rbp
   0x0000000810f5c124 <+4>:    push   %r15
   0x0000000810f5c126 <+6>:    push   %r14
   0x0000000810f5c128 <+8>:    push   %r13
   0x0000000810f5c12a <+10>:    push   %r12
   0x0000000810f5c12c <+12>:    push   %rbx
   0x0000000810f5c12d <+13>:    sub    $0x48,%rsp
   0x0000000810f5c131 <+17>:    mov    %rdx,-0x68(%rbp)
   0x0000000810f5c135 <+21>:    mov    %rdi,%r12
   0x0000000810f5c138 <+24>:    movabs $0x80b18a855,%r13
   0x0000000810f5c142 <+34>:    mov    0x18(%rsi),%rax
   0x0000000810f5c146 <+38>:    mov    0x10(%rdi),%rcx
   0x0000000810f5c14a <+42>:    mov    0x18(%rax),%r14
   0x0000000810f5c14e <+46>:    mov    0x20(%rax),%r15
   0x0000000810f5c152 <+50>:    mov    0x18(%rcx),%rbx
   0x0000000810f5c156 <+54>:    mov    0x20(%rcx),%rdx
   0x0000000810f5c15a <+58>:    mov    (%r14),%rax
   0x0000000810f5c15d <+61>:    mov    (%r15),%cl
   0x0000000810f5c160 <+64>:    mov    %rax,0xf3(%r13)
   0x0000000810f5c167 <+71>:    mov    %cl,0xfb(%r13)
   0x0000000810f5c16e <+78>:    movb   $0x1,0x0(%r13)
   0x0000000810f5c173 <+83>:    cmp    $0x1,%cl
   0x0000000810f5c176 <+86>:    mov    %rdx,-0x30(%rbp)
   0x0000000810f5c17a <+90>:    je     0x810f5c1b4 <evalexpr_0_2+148>
   0x0000000810f5c17c <+92>:    cmpb   $0x1,0x10b(%r13)
   0x0000000810f5c184 <+100>:    je     0x810f5c1b4 <evalexpr_0_2+148>
   0x0000000810f5c186 <+102>:    movb   $0x0,0xef(%r13)
   0x0000000810f5c18e <+110>:    lea    0xd3(%r13),%rdi
   0x0000000810f5c195 <+117>:    movabs $0xaa0b40,%rax
   0x0000000810f5c19f <+127>:    callq  *%rax
   0x0000000810f5c1a1 <+129>:    mov    -0x30(%rbp),%rdx
   0x0000000810f5c1a5 <+133>:    mov    0xef(%r13),%cl
   0x0000000810f5c1ac <+140>:    mov    %rax,0x3(%r13)
   0x0000000810f5c1b0 <+144>:    mov    %cl,0x0(%r13)
   0x0000000810f5c1b4 <+148>:    mov    0x8(%r12),%rax
   0x0000000810f5c1b9 <+153>:    mov    0x5(%r12),%cl
   0x0000000810f5c1be <+158>:    mov    %cl,(%rdx)
   0x0000000810f5c1c0 <+160>:    test   %cl,%cl
   0x0000000810f5c1c2 <+162>:    jne    0x810f5c1da <evalexpr_0_2+186>
   0x0000000810f5c1c4 <+164>:    cmpb   $0x1,(%rax)
   0x0000000810f5c1c7 <+167>:    jne    0x810f5c1d7 <evalexpr_0_2+183>
   0x0000000810f5c1c9 <+169>:    cmpb   $0x3,0x1(%rax)
   0x0000000810f5c1cd <+173>:    jne    0x810f5c1d7 <evalexpr_0_2+183>
   0x0000000810f5c1cf <+175>:    mov    0x2(%rax),%rax
   0x0000000810f5c1d3 <+179>:    add    $0x22,%rax
   0x0000000810f5c1d7 <+183>:    mov    %rax,(%rbx)
   0x0000000810f5c1da <+186>:    mov    (%r14),%rax
   0x0000000810f5c1dd <+189>:    mov    (%r15),%cl
   0x0000000810f5c1e0 <+192>:    mov    %rax,0x5a3(%r13)
   0x0000000810f5c1e7 <+199>:    mov    %cl,0x5ab(%r13)
   0x0000000810f5c1ee <+206>:    movb   $0x1,0x0(%r13)
   0x0000000810f5c1f3 <+211>:    cmp    $0x1,%cl
   0x0000000810f5c1f6 <+214>:    je     0x810f5c230 <evalexpr_0_2+272>
   0x0000000810f5c1f8 <+216>:    cmpb   $0x1,0x5bb(%r13)
   0x0000000810f5c200 <+224>:    je     0x810f5c230 <evalexpr_0_2+272>
   0x0000000810f5c202 <+226>:    movb   $0x0,0x59f(%r13)
   0x0000000810f5c20a <+234>:    lea    0x583(%r13),%rdi
   0x0000000810f5c211 <+241>:    movabs $0xaa0b40,%rax
   0x0000000810f5c21b <+251>:    callq  *%rax
   0x0000000810f5c21d <+253>:    mov    -0x30(%rbp),%rdx
   0x0000000810f5c221 <+257>:    mov    0x59f(%r13),%cl
   0x0000000810f5c228 <+264>:    mov    %rax,0x3(%r13)
   0x0000000810f5c22c <+268>:    mov    %cl,0x0(%r13)
   0x0000000810f5c230 <+272>:    mov    0x8(%r12),%rax
   0x0000000810f5c235 <+277>:    mov    0x5(%r12),%cl
   0x0000000810f5c23a <+282>:    mov    %cl,0x1(%rdx)
   0x0000000810f5c23d <+285>:    test   %cl,%cl
   0x0000000810f5c23f <+287>:    jne    0x810f5c258 <evalexpr_0_2+312>
   0x0000000810f5c241 <+289>:    cmpb   $0x1,(%rax)
   0x0000000810f5c244 <+292>:    jne    0x810f5c254 <evalexpr_0_2+308>
   0x0000000810f5c246 <+294>:    cmpb   $0x3,0x1(%rax)
   0x0000000810f5c24a <+298>:    jne    0x810f5c254 <evalexpr_0_2+308>
   0x0000000810f5c24c <+300>:    mov    0x2(%rax),%rax
   0x0000000810f5c250 <+304>:    add    $0x22,%rax
   0x0000000810f5c254 <+308>:    mov    %rax,0x8(%rbx)
   0x0000000810f5c258 <+312>:    mov    (%r14),%rax
   0x0000000810f5c25b <+315>:    mov    (%r15),%cl
   0x0000000810f5c25e <+318>:    mov    %rax,0x643(%r13)
   0x0000000810f5c265 <+325>:    mov    %cl,0x64b(%r13)
   0x0000000810f5c26c <+332>:    movb   $0x1,0x0(%r13)
   0x0000000810f5c271 <+337>:    cmp    $0x1,%cl
   0x0000000810f5c274 <+340>:    je     0x810f5c2ae <evalexpr_0_2+398>
   0x0000000810f5c276 <+342>:    cmpb   $0x1,0x65b(%r13)
   0x0000000810f5c27e <+350>:    je     0x810f5c2ae <evalexpr_0_2+398>
   0x0000000810f5c280 <+352>:    movb   $0x0,0x63f(%r13)
   0x0000000810f5c288 <+360>:    lea    0x623(%r13),%rdi
   0x0000000810f5c28f <+367>:    movabs $0xaa0b40,%rax
   0x0000000810f5c299 <+377>:    callq  *%rax
   0x0000000810f5c29b <+379>:    mov    -0x30(%rbp),%rdx
   0x0000000810f5c29f <+383>:    mov    0x63f(%r13),%cl
   0x0000000810f5c2a6 <+390>:    mov    %rax,0x3(%r13)
   0x0000000810f5c2aa <+394>:    mov    %cl,0x0(%r13)
   0x0000000810f5c2ae <+398>:    mov    0x8(%r12),%rax
   0x0000000810f5c2b3 <+403>:    mov    0x5(%r12),%cl
   0x0000000810f5c2b8 <+408>:    mov    %cl,0x2(%rdx)
   0x0000000810f5c2bb <+411>:    test   %cl,%cl
   0x0000000810f5c2bd <+413>:    jne    0x810f5c2d6 <evalexpr_0_2+438>
   0x0000000810f5c2bf <+415>:    cmpb   $0x1,(%rax)
   0x0000000810f5c2c2 <+418>:    jne    0x810f5c2d2 <evalexpr_0_2+434>
   0x0000000810f5c2c4 <+420>:    cmpb   $0x3,0x1(%rax)
   0x0000000810f5c2c8 <+424>:    jne    0x810f5c2d2 <evalexpr_0_2+434>
   0x0000000810f5c2ca <+426>:    mov    0x2(%rax),%rax
   0x0000000810f5c2ce <+430>:    add    $0x22,%rax
   0x0000000810f5c2d2 <+434>:    mov    %rax,0x10(%rbx)
   0x0000000810f5c2d6 <+438>:    mov    (%r14),%rax
   0x0000000810f5c2d9 <+441>:    mov    (%r15),%cl
   0x0000000810f5c2dc <+444>:    mov    %rax,0x6e3(%r13)
   0x0000000810f5c2e3 <+451>:    mov    %cl,0x6eb(%r13)
   0x0000000810f5c2ea <+458>:    movb   $0x1,0x0(%r13)
   0x0000000810f5c2ef <+463>:    cmp    $0x1,%cl
   0x0000000810f5c2f2 <+466>:    je     0x810f5c32c <evalexpr_0_2+524>
   0x0000000810f5c2f4 <+468>:    cmpb   $0x1,0x6fb(%r13)
   0x0000000810f5c2fc <+476>:    je     0x810f5c32c <evalexpr_0_2+524>
   0x0000000810f5c2fe <+478>:    movb   $0x0,0x6df(%r13)
   0x0000000810f5c306 <+486>:    lea    0x6c3(%r13),%rdi
   0x0000000810f5c30d <+493>:    movabs $0xaa0b40,%rax
   0x0000000810f5c317 <+503>:    callq  *%rax
   0x0000000810f5c319 <+505>:    mov    -0x30(%rbp),%rdx
   0x0000000810f5c31d <+509>:    mov    0x6df(%r13),%cl
   0x0000000810f5c324 <+516>:    mov    %rax,0x3(%r13)
   0x0000000810f5c328 <+520>:    mov    %cl,0x0(%r13)
   0x0000000810f5c32c <+524>:    mov    0x8(%r12),%rax
   0x0000000810f5c331 <+529>:    mov    0x5(%r12),%cl
   0x0000000810f5c336 <+534>:    mov    %cl,0x3(%rdx)
   0x0000000810f5c339 <+537>:    test   %cl,%cl
   0x0000000810f5c33b <+539>:    jne    0x810f5c354 <evalexpr_0_2+564>
   0x0000000810f5c33d <+541>:    cmpb   $0x1,(%rax)
   0x0000000810f5c340 <+544>:    jne    0x810f5c350 <evalexpr_0_2+560>
   0x0000000810f5c342 <+546>:    cmpb   $0x3,0x1(%rax)
   0x0000000810f5c346 <+550>:    jne    0x810f5c350 <evalexpr_0_2+560>
   0x0000000810f5c348 <+552>:    mov    0x2(%rax),%rax
   0x0000000810f5c34c <+556>:    add    $0x22,%rax
   0x0000000810f5c350 <+560>:    mov    %rax,0x18(%rbx)
   0x0000000810f5c354 <+564>:    mov    (%r14),%rax
   0x0000000810f5c357 <+567>:    mov    (%r15),%cl
   0x0000000810f5c35a <+570>:    mov    %rax,0x783(%r13)
   0x0000000810f5c361 <+577>:    mov    %cl,0x78b(%r13)
   0x0000000810f5c368 <+584>:    movb   $0x1,0x0(%r13)
   0x0000000810f5c36d <+589>:    cmp    $0x1,%cl
   0x0000000810f5c370 <+592>:    je     0x810f5c3aa <evalexpr_0_2+650>
   0x0000000810f5c372 <+594>:    cmpb   $0x1,0x79b(%r13)
   0x0000000810f5c37a <+602>:    je     0x810f5c3aa <evalexpr_0_2+650>
   0x0000000810f5c37c <+604>:    movb   $0x0,0x77f(%r13)
   0x0000000810f5c384 <+612>:    lea    0x763(%r13),%rdi
   0x0000000810f5c38b <+619>:    movabs $0xaa0b40,%rax
   0x0000000810f5c395 <+629>:    callq  *%rax
   0x0000000810f5c397 <+631>:    mov    -0x30(%rbp),%rdx
   0x0000000810f5c39b <+635>:    mov    0x77f(%r13),%cl
   0x0000000810f5c3a2 <+642>:    mov    %rax,0x3(%r13)
   0x0000000810f5c3a6 <+646>:    mov    %cl,0x0(%r13)
   0x0000000810f5c3aa <+650>:    mov    0x8(%r12),%rax
   0x0000000810f5c3af <+655>:    mov    0x5(%r12),%cl
   0x0000000810f5c3b4 <+660>:    mov    %cl,0x4(%rdx)
   0x0000000810f5c3b7 <+663>:    test   %cl,%cl
   0x0000000810f5c3b9 <+665>:    jne    0x810f5c3d2 <evalexpr_0_2+690>
   0x0000000810f5c3bb <+667>:    cmpb   $0x1,(%rax)
   0x0000000810f5c3be <+670>:    jne    0x810f5c3ce <evalexpr_0_2+686>
   0x0000000810f5c3c0 <+672>:    cmpb   $0x3,0x1(%rax)
   0x0000000810f5c3c4 <+676>:    jne    0x810f5c3ce <evalexpr_0_2+686>
   0x0000000810f5c3c6 <+678>:    mov    0x2(%rax),%rax
   0x0000000810f5c3ca <+682>:    add    $0x22,%rax
   0x0000000810f5c3ce <+686>:    mov    %rax,0x20(%rbx)
   0x0000000810f5c3d2 <+690>:    mov    (%r14),%rax
   0x0000000810f5c3d5 <+693>:    mov    (%r15),%cl
   0x0000000810f5c3d8 <+696>:    mov    %rax,0x823(%r13)
   0x0000000810f5c3df <+703>:    mov    %cl,0x82b(%r13)
   0x0000000810f5c3e6 <+710>:    movb   $0x1,0x0(%r13)
   0x0000000810f5c3eb <+715>:    cmp    $0x1,%cl
   0x0000000810f5c3ee <+718>:    je     0x810f5c428 <evalexpr_0_2+776>
   0x0000000810f5c3f0 <+720>:    cmpb   $0x1,0x83b(%r13)
   0x0000000810f5c3f8 <+728>:    je     0x810f5c428 <evalexpr_0_2+776>
   0x0000000810f5c3fa <+730>:    movb   $0x0,0x81f(%r13)
   0x0000000810f5c402 <+738>:    lea    0x803(%r13),%rdi
   0x0000000810f5c409 <+745>:    movabs $0xaa0b40,%rax
   0x0000000810f5c413 <+755>:    callq  *%rax
   0x0000000810f5c415 <+757>:    mov    -0x30(%rbp),%rdx
   0x0000000810f5c419 <+761>:    mov    0x81f(%r13),%cl
   0x0000000810f5c420 <+768>:    mov    %rax,0x3(%r13)
   0x0000000810f5c424 <+772>:    mov    %cl,0x0(%r13)
   0x0000000810f5c428 <+776>:    mov    0x8(%r12),%rax
   0x0000000810f5c42d <+781>:    mov    0x5(%r12),%cl
   0x0000000810f5c432 <+786>:    mov    %cl,0x5(%rdx)
   0x0000000810f5c435 <+789>:    test   %cl,%cl
   0x0000000810f5c437 <+791>:    jne    0x810f5c450 <evalexpr_0_2+816>
   0x0000000810f5c439 <+793>:    cmpb   $0x1,(%rax)
   0x0000000810f5c43c <+796>:    jne    0x810f5c44c <evalexpr_0_2+812>
   0x0000000810f5c43e <+798>:    cmpb   $0x3,0x1(%rax)
   0x0000000810f5c442 <+802>:    jne    0x810f5c44c <evalexpr_0_2+812>
   0x0000000810f5c444 <+804>:    mov    0x2(%rax),%rax
   0x0000000810f5c448 <+808>:    add    $0x22,%rax
   0x0000000810f5c44c <+812>:    mov    %rax,0x28(%rbx)
   0x0000000810f5c450 <+816>:    mov    (%r14),%rax
   0x0000000810f5c453 <+819>:    mov    (%r15),%cl
   0x0000000810f5c456 <+822>:    mov    %rax,0x10d3(%r13)
   0x0000000810f5c45d <+829>:    mov    %cl,0x10db(%r13)
   0x0000000810f5c464 <+836>:    movb   $0x1,0x0(%r13)
   0x0000000810f5c469 <+841>:    cmp    $0x1,%cl
   0x0000000810f5c46c <+844>:    je     0x810f5c4a6 <evalexpr_0_2+902>
   0x0000000810f5c46e <+846>:    cmpb   $0x1,0x10eb(%r13)
   0x0000000810f5c476 <+854>:    je     0x810f5c4a6 <evalexpr_0_2+902>
   0x0000000810f5c478 <+856>:    movb   $0x0,0x10cf(%r13)
   0x0000000810f5c480 <+864>:    lea    0x10b3(%r13),%rdi
   0x0000000810f5c487 <+871>:    movabs $0xaa0b40,%rax
   0x0000000810f5c491 <+881>:    callq  *%rax
   0x0000000810f5c493 <+883>:    mov    -0x30(%rbp),%rdx
   0x0000000810f5c497 <+887>:    mov    0x10cf(%r13),%cl
   0x0000000810f5c49e <+894>:    mov    %rax,0x3(%r13)
   0x0000000810f5c4a2 <+898>:    mov    %cl,0x0(%r13)
   0x0000000810f5c4a6 <+902>:    mov    0x8(%r12),%rax
   0x0000000810f5c4ab <+907>:    mov    0x5(%r12),%cl
   0x0000000810f5c4b0 <+912>:    mov    %cl,0x6(%rdx)
   0x0000000810f5c4b3 <+915>:    test   %cl,%cl
   0x0000000810f5c4b5 <+917>:    jne    0x810f5c4ce <evalexpr_0_2+942>
   0x0000000810f5c4b7 <+919>:    cmpb   $0x1,(%rax)
   0x0000000810f5c4ba <+922>:    jne    0x810f5c4ca <evalexpr_0_2+938>
   0x0000000810f5c4bc <+924>:    cmpb   $0x3,0x1(%rax)
   0x0000000810f5c4c0 <+928>:    jne    0x810f5c4ca <evalexpr_0_2+938>
   0x0000000810f5c4c2 <+930>:    mov    0x2(%rax),%rax
   0x0000000810f5c4c6 <+934>:    add    $0x22,%rax
   0x0000000810f5c4ca <+938>:    mov    %rax,0x30(%rbx)
   0x0000000810f5c4ce <+942>:    mov    (%r14),%rax
   0x0000000810f5c4d1 <+945>:    mov    (%r15),%cl
   0x0000000810f5c4d4 <+948>:    mov    %rax,0x1173(%r13)
   0x0000000810f5c4db <+955>:    mov    %cl,0x117b(%r13)
   0x0000000810f5c4e2 <+962>:    movb   $0x1,0x0(%r13)
   0x0000000810f5c4e7 <+967>:    cmp    $0x1,%cl
   0x0000000810f5c4ea <+970>:    je     0x810f5c524 <evalexpr_0_2+1028>
   0x0000000810f5c4ec <+972>:    cmpb   $0x1,0x118b(%r13)
   0x0000000810f5c4f4 <+980>:    je     0x810f5c524 <evalexpr_0_2+1028>
   0x0000000810f5c4f6 <+982>:    movb   $0x0,0x116f(%r13)
   0x0000000810f5c4fe <+990>:    lea    0x1153(%r13),%rdi
   0x0000000810f5c505 <+997>:    movabs $0xaa0b40,%rax
   0x0000000810f5c50f <+1007>:    callq  *%rax
   0x0000000810f5c511 <+1009>:    mov    -0x30(%rbp),%rdx
   0x0000000810f5c515 <+1013>:    mov    0x116f(%r13),%cl
   0x0000000810f5c51c <+1020>:    mov    %rax,0x3(%r13)
   0x0000000810f5c520 <+1024>:    mov    %cl,0x0(%r13)
   0x0000000810f5c524 <+1028>:    mov    0x8(%r12),%rax
   0x0000000810f5c529 <+1033>:    mov    0x5(%r12),%cl
   0x0000000810f5c52e <+1038>:    mov    %cl,0x7(%rdx)
   0x0000000810f5c531 <+1041>:    test   %cl,%cl
   0x0000000810f5c533 <+1043>:    jne    0x810f5c54c <evalexpr_0_2+1068>
   0x0000000810f5c535 <+1045>:    cmpb   $0x1,(%rax)
   0x0000000810f5c538 <+1048>:    jne    0x810f5c548 <evalexpr_0_2+1064>
   0x0000000810f5c53a <+1050>:    cmpb   $0x3,0x1(%rax)
   0x0000000810f5c53e <+1054>:    jne    0x810f5c548 <evalexpr_0_2+1064>
   0x0000000810f5c540 <+1056>:    mov    0x2(%rax),%rax
   0x0000000810f5c544 <+1060>:    add    $0x22,%rax
   0x0000000810f5c548 <+1064>:    mov    %rax,0x38(%rbx)
   0x0000000810f5c54c <+1068>:    mov    (%r14),%rax
   0x0000000810f5c54f <+1071>:    mov    (%r15),%cl
   0x0000000810f5c552 <+1074>:    mov    %rax,0x1213(%r13)
   0x0000000810f5c559 <+1081>:    mov    %cl,0x121b(%r13)
   0x0000000810f5c560 <+1088>:    movb   $0x1,0x0(%r13)
   0x0000000810f5c565 <+1093>:    cmp    $0x1,%cl
   0x0000000810f5c568 <+1096>:    je     0x810f5c5a2 <evalexpr_0_2+1154>
   0x0000000810f5c56a <+1098>:    cmpb   $0x1,0x122b(%r13)
   0x0000000810f5c572 <+1106>:    je     0x810f5c5a2 <evalexpr_0_2+1154>
   0x0000000810f5c574 <+1108>:    movb   $0x0,0x120f(%r13)
   0x0000000810f5c57c <+1116>:    lea    0x11f3(%r13),%rdi
   0x0000000810f5c583 <+1123>:    movabs $0xaa0b40,%rax
   0x0000000810f5c58d <+1133>:    callq  *%rax
   0x0000000810f5c58f <+1135>:    mov    -0x30(%rbp),%rdx
   0x0000000810f5c593 <+1139>:    mov    0x120f(%r13),%cl
   0x0000000810f5c59a <+1146>:    mov    %rax,0x3(%r13)
   0x0000000810f5c59e <+1150>:    mov    %cl,0x0(%r13)
   0x0000000810f5c5a2 <+1154>:    mov    0x8(%r12),%rax
   0x0000000810f5c5a7 <+1159>:    mov    0x5(%r12),%cl
   0x0000000810f5c5ac <+1164>:    mov    %cl,0x8(%rdx)
   0x0000000810f5c5af <+1167>:    test   %cl,%cl
   0x0000000810f5c5b1 <+1169>:    jne    0x810f5c5ca <evalexpr_0_2+1194>
   0x0000000810f5c5b3 <+1171>:    cmpb   $0x1,(%rax)
   0x0000000810f5c5b6 <+1174>:    jne    0x810f5c5c6 <evalexpr_0_2+1190>
   0x0000000810f5c5b8 <+1176>:    cmpb   $0x3,0x1(%rax)
   0x0000000810f5c5bc <+1180>:    jne    0x810f5c5c6 <evalexpr_0_2+1190>
   0x0000000810f5c5be <+1182>:    mov    0x2(%rax),%rax
   0x0000000810f5c5c2 <+1186>:    add    $0x22,%rax
   0x0000000810f5c5c6 <+1190>:    mov    %rax,0x40(%rbx)
   0x0000000810f5c5ca <+1194>:    mov    (%r14),%rax
   0x0000000810f5c5cd <+1197>:    mov    (%r15),%cl
   0x0000000810f5c5d0 <+1200>:    mov    %rax,0x12b3(%r13)
   0x0000000810f5c5d7 <+1207>:    mov    %cl,0x12bb(%r13)
   0x0000000810f5c5de <+1214>:    movb   $0x1,0x0(%r13)
   0x0000000810f5c5e3 <+1219>:    cmp    $0x1,%cl
   0x0000000810f5c5e6 <+1222>:    je     0x810f5c620 <evalexpr_0_2+1280>
   0x0000000810f5c5e8 <+1224>:    cmpb   $0x1,0x12cb(%r13)
   0x0000000810f5c5f0 <+1232>:    je     0x810f5c620 <evalexpr_0_2+1280>
   0x0000000810f5c5f2 <+1234>:    movb   $0x0,0x12af(%r13)
   0x0000000810f5c5fa <+1242>:    lea    0x1293(%r13),%rdi
   0x0000000810f5c601 <+1249>:    movabs $0xaa0b40,%rax
   0x0000000810f5c60b <+1259>:    callq  *%rax
   0x0000000810f5c60d <+1261>:    mov    -0x30(%rbp),%rdx
   0x0000000810f5c611 <+1265>:    mov    0x12af(%r13),%cl
   0x0000000810f5c618 <+1272>:    mov    %rax,0x3(%r13)
   0x0000000810f5c61c <+1276>:    mov    %cl,0x0(%r13)
   0x0000000810f5c620 <+1280>:    mov    0x8(%r12),%rax
   0x0000000810f5c625 <+1285>:    mov    0x5(%r12),%cl
   0x0000000810f5c62a <+1290>:    mov    %cl,0x9(%rdx)
   0x0000000810f5c62d <+1293>:    test   %cl,%cl
   0x0000000810f5c62f <+1295>:    jne    0x810f5c648 <evalexpr_0_2+1320>
   0x0000000810f5c631 <+1297>:    cmpb   $0x1,(%rax)
   0x0000000810f5c634 <+1300>:    jne    0x810f5c644 <evalexpr_0_2+1316>
   0x0000000810f5c636 <+1302>:    cmpb   $0x3,0x1(%rax)
   0x0000000810f5c63a <+1306>:    jne    0x810f5c644 <evalexpr_0_2+1316>
   0x0000000810f5c63c <+1308>:    mov    0x2(%rax),%rax
   0x0000000810f5c640 <+1312>:    add    $0x22,%rax
   0x0000000810f5c644 <+1316>:    mov    %rax,0x48(%rbx)
   0x0000000810f5c648 <+1320>:    mov    %rbx,-0x58(%rbp)
   0x0000000810f5c64c <+1324>:    mov    (%r14),%rax
   0x0000000810f5c64f <+1327>:    mov    (%r15),%cl
   0x0000000810f5c652 <+1330>:    mov    %rax,0x1353(%r13)
   0x0000000810f5c659 <+1337>:    mov    %cl,0x135b(%r13)
   0x0000000810f5c660 <+1344>:    movb   $0x1,0x0(%r13)
   0x0000000810f5c665 <+1349>:    cmp    $0x1,%cl
   0x0000000810f5c668 <+1352>:    je     0x810f5c7e3 <evalexpr_0_2+1731>
   0x0000000810f5c66e <+1358>:    cmpb   $0x1,0x136b(%r13)
   0x0000000810f5c676 <+1366>:    je     0x810f5c7e3 <evalexpr_0_2+1731>
   0x0000000810f5c67c <+1372>:    movb   $0x0,0x134f(%r13)
   0x0000000810f5c684 <+1380>:    lea    0x1333(%r13),%rdi
   0x0000000810f5c68b <+1387>:    movabs $0xaa0b40,%rax
   0x0000000810f5c695 <+1397>:    callq  *%rax
   0x0000000810f5c697 <+1399>:    mov    -0x30(%rbp),%rdx
   0x0000000810f5c69b <+1403>:    mov    0x134f(%r13),%bl
   0x0000000810f5c6a2 <+1410>:    mov    %rax,0x3(%r13)
   0x0000000810f5c6a6 <+1414>:    mov    %bl,0x0(%r13)
   0x0000000810f5c6aa <+1418>:    cmp    $0x1,%bl
   0x0000000810f5c6ad <+1421>:    je     0x810f5c7e3 <evalexpr_0_2+1731>
   0x0000000810f5c6b3 <+1427>:    mov    0x3(%r13),%rax
   0x0000000810f5c6b7 <+1431>:    mov    %rax,0x13f3(%r13)
   0x0000000810f5c6be <+1438>:    movb   $0x0,0x13fb(%r13)
   0x0000000810f5c6c6 <+1446>:    lea    0x13d3(%r13),%rdi
   0x0000000810f5c6cd <+1453>:    movabs $0xb7cfc0,%rax
   0x0000000810f5c6d7 <+1463>:    callq  *%rax
   0x0000000810f5c6d9 <+1465>:    mov    -0x30(%rbp),%rdx
   0x0000000810f5c6dd <+1469>:    test   %rax,%rax
   0x0000000810f5c6e0 <+1472>:    je     0x810f5c7e3 <evalexpr_0_2+1731>
   0x0000000810f5c6e6 <+1478>:    mov    %rax,0x1493(%r13)
   0x0000000810f5c6ed <+1485>:    mov    %bl,0x149b(%r13)
   0x0000000810f5c6f4 <+1492>:    movb   $0x0,0x148f(%r13)
   0x0000000810f5c6fc <+1500>:    movzbl (%rax),%esi
   0x0000000810f5c6ff <+1503>:    movabs $0xc85a04,%rcx
   0x0000000810f5c709 <+1513>:    mov    (%rcx),%edi
   0x0000000810f5c70b <+1515>:    mov    0x1b178bc(%rip),%rcx        # 0x812a73fce
   0x0000000810f5c712 <+1522>:    mov    %fs:0x0,%rdx
   0x0000000810f5c71b <+1531>:    mov    (%rdx,%rcx,1),%rbx
   0x0000000810f5c71f <+1535>:    cmp    %esi,%edi
   0x0000000810f5c721 <+1537>:    mov    %rax,-0x60(%rbp)
   0x0000000810f5c725 <+1541>:    mov    %edi,-0x48(%rbp)
   0x0000000810f5c728 <+1544>:    mov    %rbx,-0x40(%rbp)
   0x0000000810f5c72c <+1548>:    jle    0x810f5c769 <evalexpr_0_2+1609>
   0x0000000810f5c72e <+1550>:    test   %rbx,%rbx
   0x0000000810f5c731 <+1553>:    movabs $0xc85a10,%rcx
   0x0000000810f5c73b <+1563>:    mov    (%rcx),%rdx
   0x0000000810f5c73e <+1566>:    mov    %rbx,%rcx
   0x0000000810f5c741 <+1569>:    mov    %rdx,%r8
   0x0000000810f5c744 <+1572>:    cmove  %rdx,%rcx
   0x0000000810f5c748 <+1576>:    mov    %rax,%rbx
   0x0000000810f5c74b <+1579>:    mov    %rdx,-0x50(%rbp)
   0x0000000810f5c74f <+1583>:    nop
   0x0000000810f5c750 <+1584>:    movzbl %sil,%edx
   0x0000000810f5c754 <+1588>:    testb  $0x40,0x41(%rcx,%rdx,8)
   0x0000000810f5c759 <+1593>:    je     0x810f5c77d <evalexpr_0_2+1629>
   0x0000000810f5c75b <+1595>:    movzbl 0x1(%rbx),%esi
   0x0000000810f5c75f <+1599>:    add    $0x1,%rbx
   0x0000000810f5c763 <+1603>:    cmp    %esi,%edi
   0x0000000810f5c765 <+1605>:    jg     0x810f5c750 <evalexpr_0_2+1584>
   0x0000000810f5c767 <+1607>:    jmp    0x810f5c77d <evalexpr_0_2+1629>
   0x0000000810f5c769 <+1609>:    movabs $0xc85a10,%rcx
   0x0000000810f5c773 <+1619>:    mov    (%rcx),%rcx
   0x0000000810f5c776 <+1622>:    mov    %rcx,-0x50(%rbp)
   0x0000000810f5c77a <+1626>:    mov    %rax,%rbx
   0x0000000810f5c77d <+1629>:    movabs $0x801769f90,%rax
   0x0000000810f5c787 <+1639>:    mov    %rbx,%rdi
   0x0000000810f5c78a <+1642>:    callq  *%rax
   0x0000000810f5c78c <+1644>:    mov    -0x40(%rbp),%rcx
   0x0000000810f5c790 <+1648>:    test   %rcx,%rcx
   0x0000000810f5c793 <+1651>:    mov    -0x50(%rbp),%rdi
   0x0000000810f5c797 <+1655>:    cmovne %rcx,%rdi
   0x0000000810f5c79b <+1659>:    mov    -0x48(%rbp),%edx
   0x0000000810f5c79e <+1662>:    xchg   %ax,%ax
   0x0000000810f5c7a0 <+1664>:    mov    %rax,%rsi
   0x0000000810f5c7a3 <+1667>:    test   %rax,%rax
   0x0000000810f5c7a6 <+1670>:    je     0x810f5c7bc <evalexpr_0_2+1692>
   0x0000000810f5c7a8 <+1672>:    movzbl -0x1(%rbx,%rsi,1),%ecx
   0x0000000810f5c7ad <+1677>:    cmp    %ecx,%edx
   0x0000000810f5c7af <+1679>:    jle    0x810f5c7bc <evalexpr_0_2+1692>
   0x0000000810f5c7b1 <+1681>:    lea    -0x1(%rsi),%rax
   0x0000000810f5c7b5 <+1685>:    testb  $0x40,0x41(%rdi,%rcx,8)
   0x0000000810f5c7ba <+1690>:    jne    0x810f5c7a0 <evalexpr_0_2+1664>
   0x0000000810f5c7bc <+1692>:    movabs $0xa3acc0,%rax
   0x0000000810f5c7c6 <+1702>:    lea    -0x32(%rbp),%rdx
   0x0000000810f5c7ca <+1706>:    mov    %rbx,%rdi
   0x0000000810f5c7cd <+1709>:    callq  *%rax
   0x0000000810f5c7cf <+1711>:    test   %al,%al
   0x0000000810f5c7d1 <+1713>:    je     0x810f5c9cc <evalexpr_0_2+2220>
   0x0000000810f5c7d7 <+1719>:    movzbl -0x32(%rbp),%eax
   0x0000000810f5c7db <+1723>:    mov    %rax,0x3(%r13)
   0x0000000810f5c7df <+1727>:    mov    -0x30(%rbp),%rdx
   0x0000000810f5c7e3 <+1731>:    mov    0x8(%r12),%rax
   0x0000000810f5c7e8 <+1736>:    mov    0x5(%r12),%cl
   0x0000000810f5c7ed <+1741>:    mov    %cl,0xa(%rdx)
   0x0000000810f5c7f0 <+1744>:    mov    -0x58(%rbp),%rcx
   0x0000000810f5c7f4 <+1748>:    mov    %rax,0x50(%rcx)
   0x0000000810f5c7f8 <+1752>:    mov    (%r14),%rax
   0x0000000810f5c7fb <+1755>:    mov    (%r15),%cl
   0x0000000810f5c7fe <+1758>:    mov    %rax,0x31e763(%r13)
   0x0000000810f5c805 <+1765>:    mov    %cl,0x31e76b(%r13)
   0x0000000810f5c80c <+1772>:    movb   $0x1,0x0(%r13)
   0x0000000810f5c811 <+1777>:    cmp    $0x1,%cl
   0x0000000810f5c814 <+1780>:    je     0x810f5c995 <evalexpr_0_2+2165>
   0x0000000810f5c81a <+1786>:    cmpb   $0x1,0x31e77b(%r13)
   0x0000000810f5c822 <+1794>:    je     0x810f5c995 <evalexpr_0_2+2165>
   0x0000000810f5c828 <+1800>:    movb   $0x0,0x31e75f(%r13)
   0x0000000810f5c830 <+1808>:    lea    0x31e743(%r13),%rdi
   0x0000000810f5c837 <+1815>:    movabs $0xaa0b40,%rax
   0x0000000810f5c841 <+1825>:    callq  *%rax
   0x0000000810f5c843 <+1827>:    mov    -0x30(%rbp),%rdx
   0x0000000810f5c847 <+1831>:    mov    0x31e75f(%r13),%bl
   0x0000000810f5c84e <+1838>:    mov    %rax,0x3(%r13)
   0x0000000810f5c852 <+1842>:    mov    %bl,0x0(%r13)
   0x0000000810f5c856 <+1846>:    cmp    $0x1,%bl
   0x0000000810f5c859 <+1849>:    je     0x810f5c995 <evalexpr_0_2+2165>
   0x0000000810f5c85f <+1855>:    mov    0x3(%r13),%rax
   0x0000000810f5c863 <+1859>:    mov    %rax,0x31e803(%r13)
   0x0000000810f5c86a <+1866>:    movb   $0x0,0x31e80b(%r13)
   0x0000000810f5c872 <+1874>:    lea    0x31e7e3(%r13),%rdi
   0x0000000810f5c879 <+1881>:    movabs $0xb7cfc0,%rax
   0x0000000810f5c883 <+1891>:    callq  *%rax
   0x0000000810f5c885 <+1893>:    mov    -0x30(%rbp),%rdx
   0x0000000810f5c889 <+1897>:    test   %rax,%rax
   0x0000000810f5c88c <+1900>:    je     0x810f5c995 <evalexpr_0_2+2165>
   0x0000000810f5c892 <+1906>:    mov    %rax,0x1733(%r13)
   0x0000000810f5c899 <+1913>:    mov    %bl,0x173b(%r13)
   0x0000000810f5c8a0 <+1920>:    movb   $0x0,0x172f(%r13)
   0x0000000810f5c8a8 <+1928>:    movzbl (%rax),%esi
   0x0000000810f5c8ab <+1931>:    movabs $0xc85a04,%rcx
   0x0000000810f5c8b5 <+1941>:    mov    (%rcx),%r14d
   0x0000000810f5c8b8 <+1944>:    mov    0x1b178bc(%rip),%rcx        # 0x812a7417b
   0x0000000810f5c8bf <+1951>:    mov    %fs:0x0,%rdx
   0x0000000810f5c8c8 <+1960>:    mov    (%rdx,%rcx,1),%rdi
   0x0000000810f5c8cc <+1964>:    cmp    %esi,%r14d
   0x0000000810f5c8cf <+1967>:    mov    %rax,-0x40(%rbp)
   0x0000000810f5c8d3 <+1971>:    mov    %rdi,-0x48(%rbp)
   0x0000000810f5c8d7 <+1975>:    jle    0x810f5c91a <evalexpr_0_2+2042>
   0x0000000810f5c8d9 <+1977>:    test   %rdi,%rdi
   0x0000000810f5c8dc <+1980>:    movabs $0xc85a10,%rcx
   0x0000000810f5c8e6 <+1990>:    mov    (%rcx),%r15
   0x0000000810f5c8e9 <+1993>:    mov    %rdi,%rcx
   0x0000000810f5c8ec <+1996>:    cmove  %r15,%rcx
   0x0000000810f5c8f0 <+2000>:    mov    %rax,%rbx
   0x0000000810f5c8f3 <+2003>:    nopw   %cs:0x0(%rax,%rax,1)
   0x0000000810f5c8fd <+2013>:    nopl   (%rax)
   0x0000000810f5c900 <+2016>:    movzbl %sil,%edx
   0x0000000810f5c904 <+2020>:    testb  $0x40,0x41(%rcx,%rdx,8)
   0x0000000810f5c909 <+2025>:    je     0x810f5c92a <evalexpr_0_2+2058>
   0x0000000810f5c90b <+2027>:    movzbl 0x1(%rbx),%esi
   0x0000000810f5c90f <+2031>:    add    $0x1,%rbx
   0x0000000810f5c913 <+2035>:    cmp    %esi,%r14d
   0x0000000810f5c916 <+2038>:    jg     0x810f5c900 <evalexpr_0_2+2016>
   0x0000000810f5c918 <+2040>:    jmp    0x810f5c92a <evalexpr_0_2+2058>
   0x0000000810f5c91a <+2042>:    movabs $0xc85a10,%rcx
   0x0000000810f5c924 <+2052>:    mov    (%rcx),%r15
   0x0000000810f5c927 <+2055>:    mov    %rax,%rbx
   0x0000000810f5c92a <+2058>:    movabs $0x801769f90,%rax
   0x0000000810f5c934 <+2068>:    mov    %rbx,%rdi
   0x0000000810f5c937 <+2071>:    callq  *%rax
   0x0000000810f5c939 <+2073>:    mov    -0x48(%rbp),%rcx
   0x0000000810f5c93d <+2077>:    test   %rcx,%rcx
   0x0000000810f5c940 <+2080>:    cmovne %rcx,%r15
   0x0000000810f5c944 <+2084>:    nopw   %cs:0x0(%rax,%rax,1)
   0x0000000810f5c94e <+2094>:    xchg   %ax,%ax
   0x0000000810f5c950 <+2096>:    mov    %rax,%rsi
   0x0000000810f5c953 <+2099>:    test   %rax,%rax
   0x0000000810f5c956 <+2102>:    je     0x810f5c96e <evalexpr_0_2+2126>
   0x0000000810f5c958 <+2104>:    movzbl -0x1(%rbx,%rsi,1),%ecx
   0x0000000810f5c95d <+2109>:    cmp    %ecx,%r14d
   0x0000000810f5c960 <+2112>:    jle    0x810f5c96e <evalexpr_0_2+2126>
   0x0000000810f5c962 <+2114>:    lea    -0x1(%rsi),%rax
   0x0000000810f5c966 <+2118>:    testb  $0x40,0x41(%r15,%rcx,8)
   0x0000000810f5c96c <+2124>:    jne    0x810f5c950 <evalexpr_0_2+2096>
   0x0000000810f5c96e <+2126>:    movabs $0xa3acc0,%rax
   0x0000000810f5c978 <+2136>:    lea    -0x31(%rbp),%rdx
   0x0000000810f5c97c <+2140>:    mov    %rbx,%rdi
   0x0000000810f5c97f <+2143>:    callq  *%rax
   0x0000000810f5c981 <+2145>:    test   %al,%al
   0x0000000810f5c983 <+2147>:    je     0x810f5ca14 <evalexpr_0_2+2292>
   0x0000000810f5c989 <+2153>:    movzbl -0x31(%rbp),%eax
   0x0000000810f5c98d <+2157>:    mov    %rax,0x3(%r13)
   0x0000000810f5c991 <+2161>:    mov    -0x30(%rbp),%rdx
   0x0000000810f5c995 <+2165>:    mov    0x8(%r12),%rax
   0x0000000810f5c99a <+2170>:    mov    0x5(%r12),%cl
   0x0000000810f5c99f <+2175>:    mov    %cl,0xb(%rdx)
   0x0000000810f5c9a2 <+2178>:    mov    -0x58(%rbp),%rcx
   0x0000000810f5c9a6 <+2182>:    mov    %rax,0x58(%rcx)
   0x0000000810f5c9aa <+2186>:    mov    0x8(%r12),%rax
   0x0000000810f5c9af <+2191>:    mov    0x5(%r12),%cl
   0x0000000810f5c9b4 <+2196>:    and    $0x1,%cl
   0x0000000810f5c9b7 <+2199>:    mov    -0x68(%rbp),%rdx
   0x0000000810f5c9bb <+2203>:    mov    %cl,(%rdx)
   0x0000000810f5c9bd <+2205>:    add    $0x48,%rsp
   0x0000000810f5c9c1 <+2209>:    pop    %rbx
   0x0000000810f5c9c2 <+2210>:    pop    %r12
   0x0000000810f5c9c4 <+2212>:    pop    %r13
   0x0000000810f5c9c6 <+2214>:    pop    %r14
   0x0000000810f5c9c8 <+2216>:    pop    %r15
   0x0000000810f5c9ca <+2218>:    pop    %rbp
   0x0000000810f5c9cb <+2219>:    retq   
   0x0000000810f5c9cc <+2220>:    movabs $0xbbf270,%rax
   0x0000000810f5c9d6 <+2230>:    mov    $0x14,%edi
   0x0000000810f5c9db <+2235>:    xor    %esi,%esi
   0x0000000810f5c9dd <+2237>:    callq  *%rax
   0x0000000810f5c9df <+2239>:    movabs $0xbc0510,%rax
   0x0000000810f5c9e9 <+2249>:    mov    $0x2020082,%edi
   0x0000000810f5c9ee <+2254>:    callq  *%rax
   0x0000000810f5c9f0 <+2256>:    movabs $0x810f5d023,%rdi
   0x0000000810f5c9fa <+2266>:    movabs $0x810f5d04a,%rsi
   0x0000000810f5ca04 <+2276>:    movabs $0xbc0800,%rcx
   0x0000000810f5ca0e <+2286>:    mov    -0x60(%rbp),%rdx
   0x0000000810f5ca12 <+2290>:    jmp    0x810f5ca5a <evalexpr_0_2+2362>
   0x0000000810f5ca14 <+2292>:    movabs $0xbbf270,%rax
   0x0000000810f5ca1e <+2302>:    mov    $0x14,%edi
   0x0000000810f5ca23 <+2307>:    xor    %esi,%esi
   0x0000000810f5ca25 <+2309>:    callq  *%rax
   0x0000000810f5ca27 <+2311>:    movabs $0xbc0510,%rax
   0x0000000810f5ca31 <+2321>:    mov    $0x2020082,%edi
   0x0000000810f5ca36 <+2326>:    callq  *%rax
   0x0000000810f5ca38 <+2328>:    movabs $0x810f5d023,%rdi
   0x0000000810f5ca42 <+2338>:    movabs $0x810f5d04a,%rsi
   0x0000000810f5ca4c <+2348>:    movabs $0xbc0800,%rcx
   0x0000000810f5ca56 <+2358>:    mov    -0x40(%rbp),%rdx
   0x0000000810f5ca5a <+2362>:    xor    %eax,%eax
   0x0000000810f5ca5c <+2364>:    callq  *%rcx
   0x0000000810f5ca5e <+2366>:    movabs $0x810f5d052,%rdi
   0x0000000810f5ca68 <+2376>:    movabs $0x810f5d059,%rdx
   0x0000000810f5ca72 <+2386>:    movabs $0xbbfc50,%rax
   0x0000000810f5ca7c <+2396>:    mov    $0x9a,%esi
   0x0000000810f5ca81 <+2401>:    callq  *%rax
End of assembler dump.

-- 
Dmitry Marakasov   .   55B5 0596 FF1E 8D84 5F56  9510 D35A 80DD F9D2 F77D
amdmi3@amdmi3.ru  ..:              https://github.com/AMDmi3

* Andres Freund (andres@anarazel.de) wrote:

> > > Environment details:
> > > - FreeBSD 12.1 amd64
> > > - PostgreSQL 13.0 (built from FreeBSD ports)
> > > - llvm-10.0.1 (build from FreeBSD ports)
> > 
> > My bad, it's actually llvm-9.0.1. Multiple llvm versions are installed on
> > the system, and PostgreSQL uses llvm9:
> > 
> > ldd /usr/local/lib/postgresql/llvmjit.so | grep LLVM
> >     libLLVM-9.so => /usr/local/llvm90/lib/libLLVM-9.so (0x800e00000)
> 
> Could you try generating a backtrace after turning jit_debugging_support on? That might give a bit more information.
> 
> I'll check once I'm home whether I can reproduce in my environment.

I did some digging. First of all, I've discovered that the problem
goes away if llvm bitcode optimization is disabled (by commenting out
llvm_optimize_module call).

I've dumped the opcode and tried compiling it back to match disassembly
of the failing function in gdb disassembly. It didn't match perfectly,
but this place looked similar:

# %bb.84:                               # %op.32.inputcall
    movq    %rax, 5267(%r13)
    movb    %bl, 5275(%r13)
    movb    $0, 5263(%r13)
    movzbl  (%rax), %esi
    movl    __mb_sb_limit(%rip), %edi
    movq    _ThreadRuneLocale@GOTTPOFF(%rip), %rcx
    movq    %fs:0, %rdx
    movq    (%rdx,%rcx), %rcx
    cmpl    %esi, %edi
    movq    %rax, -96(%rbp)         # 8-byte Spill
    movl    %edi, -72(%rbp)         # 4-byte Spill
    movq    %rcx, -64(%rbp)         # 8-byte Spill
jle     .LBB1_85

Here's my hypothesis:

The problem happens when boolin() function is inlined by LLVM.
The named function calls isspace() internally, which on FreeBSD is
locale-specific and involves caching some locale parameters in
thread-local variable defined as

extern _Thread_local const _RuneLocale *_ThreadRuneLocale;

The execution crashes on trying to access the named thread-local varible,
probably because something related to TLS is not set up properly in/for
LLVM.

I've confirmed this hypothesis by disabling isspace() calls in boolin()
which has also fixed the problem.

-- 
Dmitry Marakasov   .   55B5 0596 FF1E 8D84 5F56  9510 D35A 80DD F9D2 F77D
amdmi3@amdmi3.ru  ..:              https://github.com/AMDmi3

* Dmitry Marakasov (amdmi3@amdmi3.ru) wrote:

> > > > Environment details:
> > > > - FreeBSD 12.1 amd64
> > > > - PostgreSQL 13.0 (built from FreeBSD ports)
> > > > - llvm-10.0.1 (build from FreeBSD ports)
> > > 
> > > My bad, it's actually llvm-9.0.1. Multiple llvm versions are installed on
> > > the system, and PostgreSQL uses llvm9:
> > > 
> > > ldd /usr/local/lib/postgresql/llvmjit.so | grep LLVM
> > >     libLLVM-9.so => /usr/local/llvm90/lib/libLLVM-9.so (0x800e00000)
> > 
> > Could you try generating a backtrace after turning jit_debugging_support on? That might give a bit more
information.
> > 
> > I'll check once I'm home whether I can reproduce in my environment.
> 
> I did some digging. First of all, I've discovered that the problem
> goes away if llvm bitcode optimization is disabled (by commenting out
> llvm_optimize_module call).
> 
> I've dumped the opcode and tried compiling it back to match disassembly
> of the failing function in gdb disassembly. It didn't match perfectly,
> but this place looked similar:
> 
> # %bb.84:                               # %op.32.inputcall
>     movq    %rax, 5267(%r13)
>     movb    %bl, 5275(%r13)
>     movb    $0, 5263(%r13)
>     movzbl  (%rax), %esi
>     movl    __mb_sb_limit(%rip), %edi
>     movq    _ThreadRuneLocale@GOTTPOFF(%rip), %rcx
>     movq    %fs:0, %rdx
>     movq    (%rdx,%rcx), %rcx
>     cmpl    %esi, %edi
>     movq    %rax, -96(%rbp)         # 8-byte Spill
>     movl    %edi, -72(%rbp)         # 4-byte Spill
>     movq    %rcx, -64(%rbp)         # 8-byte Spill
> jle     .LBB1_85
> 
> Here's my hypothesis:
> 
> The problem happens when boolin() function is inlined by LLVM.
> The named function calls isspace() internally, which on FreeBSD is
> locale-specific and involves caching some locale parameters in
> thread-local variable defined as
> 
> extern _Thread_local const _RuneLocale *_ThreadRuneLocale;
> 
> The execution crashes on trying to access the named thread-local varible,
> probably because something related to TLS is not set up properly in/for
> LLVM.
> 
> I've confirmed this hypothesis by disabling isspace() calls in boolin()
> which has also fixed the problem.

Long story short, I was able to mitigate the crash with the following patch:

--- disable-inlining-tls-using-functions.patch begins here ---
commit f703544edc406293e39b7a59a245e798d18f458e
Author: Dmitry Marakasov <amdmi3@amdmi3.ru>
Date:   Thu Nov 5 02:56:00 2020 +0300

    Do not inline functions accessing TLS in LLVM JIT

diff --git src/backend/jit/llvm/llvmjit_inline.cpp src/backend/jit/llvm/llvmjit_inline.cpp
index 2617a46..a063edb 100644
--- src/backend/jit/llvm/llvmjit_inline.cpp
+++ src/backend/jit/llvm/llvmjit_inline.cpp
@@ -608,6 +608,16 @@ function_inlinable(llvm::Function &F,
         if (rv->materialize())
             elog(FATAL, "failed to materialize metadata");
 
+        /*
+         * Don't inline functions with thread-local variables until
+         * related crashes are investigated (see BUG #16696)
+         */
+        if (rv->isThreadLocal()) {
+            ilog(DEBUG1, "cannot inline %s due to thread-local variable %s",
+                F.getName().data(), rv->getName().data());
+            return false;
+        }
+
         /*
          * Never want to inline externally visible vars, cheap enough to
          * reference.
--- disable-inlining-tls-using-functions.patch ends here ---

I have no knowledge of LLVM to investigate this further, but the guess
is that something TLS related is not initialized properly.

-- 
Dmitry Marakasov   .   55B5 0596 FF1E 8D84 5F56  9510 D35A 80DD F9D2 F77D
amdmi3@amdmi3.ru  ..:              https://github.com/AMDmi3

Hi,

On 2020-11-05 00:20:15 +0300, Dmitry Marakasov wrote:
> The problem happens when boolin() function is inlined by LLVM.
> The named function calls isspace() internally, which on FreeBSD is
> locale-specific and involves caching some locale parameters in
> thread-local variable defined as
> 
> extern _Thread_local const _RuneLocale *_ThreadRuneLocale;
> 
> The execution crashes on trying to access the named thread-local varible,
> probably because something related to TLS is not set up properly in/for
> LLVM.
> 
> I've confirmed this hypothesis by disabling isspace() calls in boolin()
> which has also fixed the problem.

Nice digging. I suspect your approach of disabling inlining for TLS
might be the most reasonable one for now.

Any chance you're interested in extending your patch with a testcase
that crashes reliably without it? Best on multiple platforms, including
linux? Easiest way would probably be to add a function referencing a
thread local variable in regress.c and set jit_above_cost = 0,
jit_inline_above_cost=0 for a query using that function.

Greetings,

Andres Freund

* Andres Freund (andres@anarazel.de) wrote:

> > The problem happens when boolin() function is inlined by LLVM.
> > The named function calls isspace() internally, which on FreeBSD is
> > locale-specific and involves caching some locale parameters in
> > thread-local variable defined as
> > 
> > extern _Thread_local const _RuneLocale *_ThreadRuneLocale;
> > 
> > The execution crashes on trying to access the named thread-local varible,
> > probably because something related to TLS is not set up properly in/for
> > LLVM.
> > 
> > I've confirmed this hypothesis by disabling isspace() calls in boolin()
> > which has also fixed the problem.
> 
> Nice digging. I suspect your approach of disabling inlining for TLS
> might be the most reasonable one for now.
> 
> Any chance you're interested in extending your patch with a testcase
> that crashes reliably without it? Best on multiple platforms, including
> linux? Easiest way would probably be to add a function referencing a
> thread local variable in regress.c and set jit_above_cost = 0,
> jit_inline_above_cost=0 for a query using that function.

Can try. Here's what I could come with so far:

https://github.com/postgres/postgres/compare/master...AMDmi3:llvm-jit-regress.patch

Unfortunately it doesn't crash with the dedicated function, likely
because there's no llvm bitcode generation performed for regress.c.
It does however reliably crash on

SELECT (jsonb_array_elements('[true]'::jsonb)->>0)::boolean;
or
SELECT (case when random() >= 0.5 then 'true' else 'false' end)::boolean;

but it would likely not reproduce on Linux due to different isspace
implementation. Notably, it does not crash on simple 'true'::boolean,
I guess because it may be evaluated at compile time.

-- 
Dmitry Marakasov   .   55B5 0596 FF1E 8D84 5F56  9510 D35A 80DD F9D2 F77D
amdmi3@amdmi3.ru  ..:              https://github.com/AMDmi3

Hi,

This seems to still not be fixed. Is it planned for 14.0 release?

Palle




--
Sent from: https://www.postgresql-archive.org/PostgreSQL-bugs-f2117394.html

On Thu, May 20, 2021 at 2:49 AM Palle <girgen@pingpong.net> wrote:
> This seems to still not be fixed. Is it planned for 14.0 release?

I don't know enough about LLVM and TLS to have an opinion on the
details, but I tested this and it works, and I think it should
probably be back-patched based on the discussion so far (and hopefully
in the future some way can be figured out to make it actually work?).

When I tested the original repro, I got SIGTRAP (not SIGSEGV as
reported) with LLVM 9 and 10, while 11, 12 and 13 failed gracefully
with:

2021-07-18 15:27:42.785 NZST [78884] FATAL:  fatal llvm error:
Relocation type not implemented yet!

I was reminded of this thread because I happened to see a
relevant-looking commit whiz past in the FreeBSD commit log[1]: libc
is changing to a different TLS model.  I just tried that, and it makes
no difference.

In case it helps someone who wants to see this failure, I recently
XKCD1319'd myself into producing Vagrant files for PostgreSQL hacking
on various OSes[2] after a clash with illumos on the build farm (cd
freebsd12; vagrant up; vagrant ssh; you'll find a fresh build of the
PostgreSQL master branch built against LLVM 9 under ~/install).

[1] https://cgit.freebsd.org/src/commit/?id=9c97062b620137a1f7cad4c6b3fb030a396b3266
[2] https://github.com/macdice/postgresql-dev-vagrant/

Hi!

I’m on holiday and AFK another week or so. Please feel free to commit if it seems correct.

Palle

> 18 juli 2021 kl. 07:14 skrev Thomas Munro <thomas.munro@gmail.com>:
>
> On Thu, May 20, 2021 at 2:49 AM Palle <girgen@pingpong.net> wrote:
>> This seems to still not be fixed. Is it planned for 14.0 release?
>
> I don't know enough about LLVM and TLS to have an opinion on the
> details, but I tested this and it works, and I think it should
> probably be back-patched based on the discussion so far (and hopefully
> in the future some way can be figured out to make it actually work?).
>
> When I tested the original repro, I got SIGTRAP (not SIGSEGV as
> reported) with LLVM 9 and 10, while 11, 12 and 13 failed gracefully
> with:
>
> 2021-07-18 15:27:42.785 NZST [78884] FATAL:  fatal llvm error:
> Relocation type not implemented yet!
>
> I was reminded of this thread because I happened to see a
> relevant-looking commit whiz past in the FreeBSD commit log[1]: libc
> is changing to a different TLS model.  I just tried that, and it makes
> no difference.
>
> In case it helps someone who wants to see this failure, I recently
> XKCD1319'd myself into producing Vagrant files for PostgreSQL hacking
> on various OSes[2] after a clash with illumos on the build farm (cd
> freebsd12; vagrant up; vagrant ssh; you'll find a fresh build of the
> PostgreSQL master branch built against LLVM 9 under ~/install).
>
> [1] https://cgit.freebsd.org/src/commit/?id=9c97062b620137a1f7cad4c6b3fb030a396b3266
> [2] https://github.com/macdice/postgresql-dev-vagrant/

Hi,

On 2021-07-18 17:13:50 +1200, Thomas Munro wrote:
> On Thu, May 20, 2021 at 2:49 AM Palle <girgen@pingpong.net> wrote:
> > This seems to still not be fixed. Is it planned for 14.0 release?
> 
> I don't know enough about LLVM and TLS to have an opinion on the
> details, but I tested this and it works, and I think it should
> probably be back-patched based on the discussion so far (and hopefully
> in the future some way can be figured out to make it actually work?).
> 
> When I tested the original repro, I got SIGTRAP (not SIGSEGV as
> reported) with LLVM 9 and 10, while 11, 12 and 13 failed gracefully
> with:
> 
> 2021-07-18 15:27:42.785 NZST [78884] FATAL:  fatal llvm error:
> Relocation type not implemented yet!
> 
> I was reminded of this thread because I happened to see a
> relevant-looking commit whiz past in the FreeBSD commit log[1]: libc
> is changing to a different TLS model.  I just tried that, and it makes
> no difference.

FWIW, there's ongoing work in llvm to make the jit stuff handle TLS
correctly.

Greetings,

Andres Freund

On Thu, Jul 22, 2021 at 4:59 AM Andres Freund <andres@anarazel.de> wrote:
> On 2021-07-18 17:13:50 +1200, Thomas Munro wrote:
> > On Thu, May 20, 2021 at 2:49 AM Palle <girgen@pingpong.net> wrote:
> > > This seems to still not be fixed. Is it planned for 14.0 release?
> >
> > I don't know enough about LLVM and TLS to have an opinion on the
> > details, but I tested this and it works, and I think it should
> > probably be back-patched based on the discussion so far (and hopefully
> > in the future some way can be figured out to make it actually work?).

Done.

For the record, the original repro now causes PostgreSQL on FreeBSD
(with INLINE_DEBUG defined) to say:

2021-07-22 15:20:51.738 NZST [82982] DEBUG:  checking inlinability of boolin
2021-07-22 15:20:51.738 NZST [82982] DEBUG:  cannot inline boolin due
to thread-local variable _ThreadRuneLocale
2021-07-22 15:20:51.738 NZST [82982] DEBUG:  had to skip inlining boolin

> FWIW, there's ongoing work in llvm to make the jit stuff handle TLS
> correctly.

Cool.

Dmitry: Thanks very much for the report, reproducer, analysis and patch!