Re: GIST and TOAST - Mailing list pgsql-hackers

"Teodor Sigaev" <teodor@sigaev.ru> writes:

>> And it's not clear _int_gist.c has been running with toasted array values for
>> years because it's limited to arrays of 100 integers (or perhaps 200 integers,
>> there's a factor of 2 in the test). That's not enough to trigger toasting
>> unless there are other large columns in the same table.
>
> That's was intended limitation to prevent indexing of huge arrays.
> gist__int_ops compression method is orientated for small and isn't effective on
> big ones.

Right, so it's possible nobody see any toasted arrays with _int_gist.c since
they never get very large. It looks like index_form_tuple will never compress
anything under 512b so I guess it's safe currently.

>> I do know that with packed varlenas I get a crash in g_int_union among other
>> places. I can't tell where the datum came from originally and how it ended up
>> stored in packed format.
> Can you provide your patch (in current state) and test suite? Or backtrace at least.

It doesn't actually crash, it just fails CHECKARRVALID. I added an assertion
in there to cause it to generate a core dump.


You can download the core dump and binary from 
http://community.enterprisedb.com/varlena/core._inthttp://community.enterprisedb.com/varlena/postgres._int

The last patch (without the assertion) is at:
http://community.enterprisedb.com/varlena/patch-varvarlena-14.patch.gz


What I'm seeing is this:

(gdb) f 3
#3  0xb7fd924b in inner_int_union (a=0x84e41f4, b=0xb64220d0)   at _int_tool.c:81
81        CHECKARRVALID(b);

The array is actually garbage:

(gdb) p *b
$2 = {vl_len_ = 141, ndim = 0, dataoffset = 5888, elemtype = 0}

What's going on is that the va_1byte header is 141 which is 0x80 | 13. So it's
actually only 13 bytes with a 1 byte header or a 12 byte array:

(gdb) p *(varattrib*)b
$3 = {va_1byte = {va_header = 141 '\215', va_data = ""}, va_external = {   va_header = 141 '\215', va_padding =
"\000\000",va_rawsize = 0,    va_extsize = 5888, va_valueid = 0, va_toastrelid = 0}, va_compressed = {   va_header =
141,va_rawsize = 0, va_data = ""}, va_4byte = {   va_header = 141, va_data = ""}}
 

(gdb) bt
#0  0xb7e6a947 in raise () from /lib/tls/libc.so.6
#1  0xb7e6c0c9 in abort () from /lib/tls/libc.so.6
#2  0x082fec97 in ExceptionalCondition (   conditionName=0xb7fdd3b9 "!(!((b)->dataoffset != 0))",
errorType=0xb7fdd371"FailedAssertion",    fileName=0xb7fdd347 "_int_tool.c", lineNumber=81) at assert.c:51
 
#3  0xb7fd924b in inner_int_union (a=0x84e41f4, b=0xb64220d0)   at _int_tool.c:81
#4  0xb7fd547d in g_int_picksplit (fcinfo=0xbf9e43f0) at _int_gist.c:403
#5  0x08304d9c in FunctionCall2 (flinfo=0xbf9e5a94, arg1=139342312,    arg2=3214821160) at fmgr.c:1154
#6  0x08094fd3 in gistUserPicksplit (r=0xb6078d4c, entryvec=0x84e31e8,    attno=0, v=0xbf9e4728, itup=0x84e2ddc,
len=142,giststate=0xbf9e4b94)   at gistsplit.c:306
 
#7  0x08095deb in gistSplitByKey (r=0xb6078d4c, page=0xb6420220 "",    itup=0x84e2ddc, len=142, giststate=0xbf9e4b94,
v=0xbf9e4728,   entryvec=0x84e31e8, attno=0) at gistsplit.c:548
 
#8  0x080874bd in gistSplit (r=0xb6078d4c, page=0xb6420220 "",    itup=0x84e2ddc, len=142, giststate=0xbf9e4b94) at
gist.c:943
#9  0x080850fa in gistplacetopage (state=0xbf9e49e0, giststate=0xbf9e4b94)   at gist.c:329
#10 0x080871eb in gistmakedeal (state=0xbf9e49e0, giststate=0xbf9e4b94)   at gist.c:873
#11 0x08084f21 in gistdoinsert (r=0xb6078d4c, itup=0x84e2ce4, freespace=819,    giststate=0xbf9e4b94) at gist.c:278
#12 0x08084cf5 in gistbuildCallback (index=0xb6078d4c, htup=0x84c8c30,    values=0xbf9e4a98, isnull=0xbf9e4a78 "",
tupleIsAlive=1'\001',    state=0xbf9e4b94) at gist.c:201
 
#13 0x080fc81f in IndexBuildHeapScan (heapRelation=0xb60d6860,    indexRelation=0xb6078d4c, indexInfo=0x84cd620,
callback=0x8084c27<gistbuildCallback>, callback_state=0xbf9e4b94)   at index.c:1548
 
#14 0x08084bdd in gistbuild (fcinfo=0xbf9e60e8) at gist.c:150
#15 0x08305630 in OidFunctionCall3 (functionId=782, arg1=3054332000,    arg2=3053948236, arg3=139253280) at
fmgr.c:1460
#16 0x080fc363 in index_build (heapRelation=0xb60d6860,    indexRelation=0xb6078d4c, indexInfo=0x84cd620, isprimary=0
'\0')  at index.c:1296
 
#17 0x080fb86a in index_create (heapRelationId=21361,    indexRelationName=0x84a531c "text_idx", indexRelationId=21366,
  indexInfo=0x84cd620, accessMethodObjectId=783, tableSpaceId=0,    classObjectId=0x84cd60c, coloptions=0x84cd6ac,
reloptions=0,   isprimary=0 '\0', isconstraint=0 '\0', allow_system_table_mods=0 '\0',    skip_build=0 '\0',
concurrent=0'\0') at index.c:794
 
#18 0x0815f3e4 in DefineIndex (heapRelation=0x84a5354,    indexRelationName=0x84a531c "text_idx", indexRelationId=0,
accessMethodName=0x84a5380"gist", tableSpaceName=0x0,    attributeList=0x84a5448, predicate=0x0, rangetable=0x0,
options=0x0,   unique=0 '\0', primary=0 '\0', isconstraint=0 '\0',    is_alter_table=0 '\0', check_rights=1 '\001',
skip_build=0'\0',    quiet=0 '\0', concurrent=0 '\0') at indexcmds.c:452
 
#19 0x0825dcea in ProcessUtility (parsetree=0x84a5464, params=0x0,    dest=0x84a54e0, completionTag=0xbf9e687e "") at
utility.c:797
#20 0x0825bef9 in PortalRunUtility (portal=0x84ca2ec, utilityStmt=0x84a5464,    dest=0x84a54e0,
completionTag=0xbf9e687e"") at pquery.c:1176
 
#21 0x0825c04b in PortalRunMulti (portal=0x84ca2ec, dest=0x84a54e0,    altdest=0x84a54e0, completionTag=0xbf9e687e "")
atpquery.c:1263
 
#22 0x0825b786 in PortalRun (portal=0x84ca2ec, count=2147483647,    dest=0x84a54e0, altdest=0x84a54e0,
completionTag=0xbf9e687e"")   at pquery.c:814
 
#23 0x0825604d in exec_simple_query (   query_string=0x84a4fec "CREATE INDEX text_idx on test__int using gist ( a
gist__int_ops);") at postgres.c:953
 
#24 0x08259c2f in PostgresMain (argc=4, argv=0x844df58,    username=0x844de44 "stark") at postgres.c:3434
#25 0x08223d2c in BackendRun (port=0x8461550) at postmaster.c:2974
#26 0x082232b9 in BackendStartup (port=0x8461550) at postmaster.c:2601
#27 0x08220d82 in ServerLoop () at postmaster.c:1214
#28 0x08220728 in PostmasterMain (argc=3, argv=0x844a340) at postmaster.c:967
#29 0x081c426b in main (argc=3, argv=0x844a340) at main.c:188

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com