Re: I s this a bug of spgist index in a heavy write condition? - Mailing list pgsql-hackers
From | 李海龙 |
---|---|
Subject | Re: I s this a bug of spgist index in a heavy write condition? |
Date | |
Msg-id | 50EF9043.2060009@qunar.com Whole thread Raw |
In response to | Re: I s this a bug of spgist index in a heavy write condition? (Tom Lane <tgl@sss.pgh.pa.us>) |
Responses |
Re: I s this a bug of spgist index in a heavy write condition?
|
List | pgsql-hackers |
<div class="moz-cite-prefix"><b>Hi,dear tom lane && pgsql-hackers</b><br /><br /> In your last e-mail, you want meto think anbout the only spgist-indexed column and give you some more further information.<br /><br /> This time I willgive you the contents of the table route_raw, the download link is <b><a class="moz-txt-link-freetext" href="https://www.box.com/s/yxa4yxo6rcb3dzeaefmz">https://www.box.com/s/yxa4yxo6rcb3dzeaefmz</a></b>or <b> <a class="moz-txt-link-freetext"href="http://dl.dropbox.com/u/203288/route_raw_spgist.sql.tar.gz"> http://dl.dropbox.com/u/203288/route_raw_spgist.sql.tar.gz</a></b>. <br /><br /><br /><pre wrap="">OS Version: CentOS release 5.5 (Final) PostgreSQL Version: postgres=# select version(); version ---------------------------------------------------------------------------------------------------------------PostgreSQL 9.2.2on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-50), 64-bit (1 row) I create a database named test_spgist,and there is only one table named route_raw which only has a id (primary key ) columnand a spgist index column in the database.</pre> <pre wrap="">Some details is shown as following: </pre> test_spgist=# \dt<br /> List of relations<br /> Schema | Name | Type | Owner <br /> --------+-----------+-------+----------<br/> public | route_raw | table | postgres<br /> (1 row)<br /><br /> test_spgist=#\d route_raw<br /> Table "public.route_raw"<br /> Column | Type | Modifiers <br /> -----------+-------------------------+--------------------------------------------------------<br/> sourceurl | charactervarying(1000) | not null default ''::character varying<br /> id | integer | not null defaultnextval('route_raw_id_seq'::regclass)<br /> Indexes:<br /> "route_raw_pkey" PRIMARY KEY, btree (id)<br /> "route_raw_sourceurl_idx" spgist (sourceurl)<br /><br /> --<b>Before the test, this table has 997736 records.</b><br />test_spgist=# select count(1) from route_raw ;<br /> count <br /> --------<br /> 997736<br /> (1 row)<br /><br /><b><big>1.edit a shell script that function is insert data to table route_raw</big></b><br /><br /><b>$ cat /tmp/insert_spgist.sh</b><br /> #/bin/bash<br /><br /> PSQL="/opt/pg92/bin/psql"<br /> DB=test_spgist<br /> USR=postgres<br/><br /> while [ 10 -ne 9 ]<br /> do<br /> SQL="insert into route_raw(sourceurl) values ('<a class="moz-txt-link-freetext"href="http://suz.bacts.com/group/group_33_128972.aspx" moz-do-not-send="true">http://suz.bacts.com/group/group_33_128972.aspx</a>, <a class="moz-txt-link-freetext" href="http://www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=GHX%B0%B2%D0%C4%C0%F6%C0%A520120613%B4%BF%CD%E6&xl=12623" moz-do-not-send="true"> http://www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=GHX%B0%B2%D0%C4%C0%F6%C0%A520120613%B4%BF%CD%E6&xl=12623</a>, <aclass="moz-txt-link-freetext" href="http://www.ctszj.com.cn/route_group_1_147707.html,http://jx.bacts.com/group/group_33_118381.aspx" moz-do-not-send="true"> http://www.ctszj.com.cn/route_group_1_147707.html,http://jx.bacts.com/group/group_33_118381.aspx</a>,<a class="moz-txt-link-freetext" href="http://www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=GHX%C9%F1%C5%A9%CE%D420120723&xl=7096" moz-do-not-send="true"> http://www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=GHX%C9%F1%C5%A9%CE%D420120723&xl=7096</a>, <a class="moz-txt-link-freetext"href="http://www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=TUCCZ120625A&xl=7957" moz-do-not-send="true">http://www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=TUCCZ120625A&xl=7957</a> , <a class="moz-txt-link-freetext"href="http://www.ctszj.com.cn/route_group_1_134782.html" moz-do-not-send="true"> http://www.ctszj.com.cn/route_group_1_134782.html</a>, <a class="moz-txt-link-freetext" href="http://fj.bacts.com/group/group_33_113288.asp"moz-do-not-send="true"> http://fj.bacts.com/group/group_33_113288.asp</a>, <a class="moz-txt-link-freetext" href="http://www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=TUCCZ120625A&xl=7957"moz-do-not-send="true"> http://www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=TUCCZ120625A&xl=7957</a>,<a class="moz-txt-link-freetext" href="http://www.ctszj.com.cn/route_group_1_134782.html,http://fj.bacts.com/group/group_33_113288.aspx" moz-do-not-send="true"> http://www.ctszj.com.cn/route_group_1_134782.html,http://fj.bacts.com/group/group_33_113288.aspx</a>,<a class="moz-txt-link-freetext"href="http://www.sztravel.com.cn/travel/detail.aspx?xlid=19883&ctype=1" moz-do-not-send="true">http://www.sztravel.com.cn/travel/detail.aspx?xlid=19883&ctype=1</a>, <a class="moz-txt-link-freetext"href="http://www.ctszj.com.cn/route_group_1_134695.html" moz-do-not-send="true"> http://www.ctszj.com.cn/route_group_1_134695.html</a>')"<br /> sleep 1<br /> $PSQL -U $USR -d $DB -c "$SQL"<br/> done<br /><br /><b>2.</b> <b>I create 15 clients and execute this shell script in each client </b><br /><br />$ /tmp/insert_spgist.sh >/dev/null<br /> ...<br /> ...<br /> ...<br /> ]$ /tmp/insert_spgist.sh >/dev/null<br /><br/><b>3.</b> <b>only 1 connection to update the table </b><br /><br /><b>$ psql -U postgres</b><br /> psql (9.2.2)<br/> Type "help" for help.<br /><br /> postgres=# \c test_spgist <br /> You are now connected to database "test_spgist"as user "postgres".<br /><br /> test_spgist=# update route_raw set sourceurl = '<a class="moz-txt-link-freetext" href="http://www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=CGZL120701MDG05LA&xl=13935%27where" moz-do-not-send="true">http://www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=CGZL120701MDG05LA&xl=13935'where</a> sourceurllike '%=CGZL120701MDG0%';<br /><br /><br /> 4. <b>I execute the statement "<font color="#ff0000">select * from pg_stat_activity;</font>and <font color="#ff0000">select count(1) from route_raw;</font> " to monitor the connections</b><br/><br /> test_spgist=# select count(1) from route_raw ;<br /> count <br /> --------<br /> 997799<br/> (1 row)<br /><br /> ......<br /> ......<br /><br /> test_spgist=# select count(1) from route_raw;<br /> count <br /> ---------<br /> 1004066<br /> (1 row)<br /><br /> ......<br /> ......<br /><br /> test_spgist=# select count(1)from route_raw;<br /> count <br /> ---------<br /> 1004066<br /> (1 row)<br /><br /> test_spgist=# select * frompg_stat_activity ;<br /> datid | datname | pid | usesysid | usename | application_name | client_addr | client_hostname| client_port | backend_start | xact_start | <br /> query_start | state_change | waiting | state | <br/> &nb sp; <br /> query <br /> -------+-------------+-------+----------+----------+------------------+-------------+-----------------+-------------+-------------------------------+-------------------------------+----------<br /> ---------------------+-------------------------------+---------+--------+----------------------------------------------------------------------------------------------------------------------<br /> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br /> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br /> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br /> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br /> 16384 | test_spgist | 18797 | 10 | postgres | psql | | | -1 | 2013-01-1020:27:05.171787+08 | | 2013-01-1<br /> 0 20:29:05.393515+08 | 2013-01-10 20:29:06.126108+08| f | idle | select * from route_raw where sourceurl like '%CGZL120701MDG05LA&xl%';<br /> 16384| test_spgist | 22272 | 10 | postgres | psql | | | -1 | 2013-01-1018:23:20.035532+08 | 2013-01-10 20:31:57.898985+08 | 2013-01-1<br /> 0 20:31:57.898985+08 | 2013-01-10 20:31:57.898986+08| f | active | select * from pg_stat_activity ; <br /> 16384 | test_spgist | 24391 | 10 |postgres | psql | | | -1 | 2013-01-10 20:29:07.113574+08 | 2013-01-10 20:29:07.1161+08 | 2013-01-1<br /> 0 20:29:07.1161+08 | 2013-01-10 20:29:07.116107+08 | f | active | insert intoroute_raw(sourceurl) values ('<a class="moz-txt-link-freetext" href="http://suz.bacts.com/group/group_33_128972.aspx" moz-do-not-send="true">http://suz.bacts.com/group/group_33_128972.aspx</a>, <a class="moz-txt-link-freetext" href="http://www.gzl.com.cn/Tra"moz-do-not-send="true"> http://www.gzl.com.cn/Tra</a><br /> vel/TeamDetail.aspx?tuanhao=GHX%B0%B2%D0%C4%C0%F6%C0%A520120613%B4%BF%CD%E6&xl=12623, <a class="moz-txt-link-freetext" href="http://www.ctszj.com.cn/route_group_1_147707.html,http://jx.bacts.com/group/group_33_118381.aspx" moz-do-not-send="true"> http://www.ctszj.com.cn/route_group_1_147707.html,http://jx.bacts.com/group/group_33_118381.aspx</a>,<a class="moz-txt-link-freetext"href="http:/" moz-do-not-send="true">http:/</a><br /> /www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=GHX%C9%F1%C5%A9%CE%D420120723&xl=7096, <a class="moz-txt-link-freetext"href="http://www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=TUCCZ120625A&xl=7957" moz-do-not-send="true">http://www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=TUCCZ120625A&xl=7957</a> , <a class="moz-txt-link-freetext"href="http://www.ctszj.com.cn/rout" moz-do-not-send="true"> http://www.ctszj.com.cn/rout</a><br/> e_group_1_134782.html , <a class="moz-txt-link-freetext" href="http://fj.bacts.com/group/group_33_113288.asp"moz-do-not-send="true"> http://fj.bacts.com/group/group_33_113288.asp</a>, <a class="moz-txt-link-freetext" href="http://www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=TUCCZ120625A&xl=7957"moz-do-not-send="true"> http://www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=TUCCZ120625A&xl=7957</a>,<a class="moz-txt-link-freetext" href="http://www.ctszj.com.cn/route_group_1_13478"moz-do-not-send="true"> http://www.ctszj.com.cn/route_group_1_13478</a><br/> 2.html,<a class="moz-txt-link-freetext" href="http://fj.bacts.com/group/group_33_113288.aspx" moz-do-not-send="true">http://fj.bacts.com/group/group_33_113288.aspx</a>,<a class="moz-txt-link-freetext" href="http://www.sztravel.com.cn/travel/detail.aspx?xlid=19883&ctype=1" moz-do-not-send="true">http://www.sztravel.com.cn/travel/detail.aspx?xlid=19883&ctype=1</a>, <a class="moz-txt-link-freetext"href="http://www.ctszj.com.cn/route_group_1_134695.html" moz-do-not-send="true"> http://www.ctszj.com.cn/route_group_1_134695.html</a>')<br /><br /> ......<br /> ......<br /><br /> 16384 | test_spgist| 24437 | 10 | postgres | psql | | | -1 | 2013-01-10 20:29:38.498181+08| 2013-01-10 20:31:17.963092+08 | 2013-01-1<br /> 0 20:31:17.963092+08 | 2013-01-10 20:31:17.963095+08| f | active | update route_raw set sourceurl = '<a class="moz-txt-link-freetext" href="http://www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=CGZL120701MDG05LA&xl=13935%27whe" moz-do-not-send="true">http://www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=CGZL120701MDG05LA&xl=13935'whe</a><br/> resourceurl like '%=CGZL120701MDG0%';<br /> (18 rows)<br /><br /> 5. <b>display process information using gdb</b><br /><br/> [root@XXX ~]# <font color="#ff0000"><b>gdb </b></font><br /> GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.2)<br/> Copyright (C) 2009 Free Software Foundation, Inc.<br /> License GPLv3+: GNU GPL version 3 or later<a class="moz-txt-link-rfc2396E" href="http://gnu.org/licenses/gpl.html" moz-do-not-send="true"> <http://gnu.org/licenses/gpl.html></a><br/> This is free software: you are free to change and redistribute it.<br />There is NO WARRANTY, to the extent permitted by law. Type "show copying"<br /> and "show warranty" for details.<br />This GDB was configured as "x86_64-redhat-linux-gnu".<br /> For bug reporting instructions, please see:<br /><a class="moz-txt-link-rfc2396E"href="http://www.gnu.org/software/gdb/bugs/" moz-do-not-send="true"><http://www.gnu.org/software/gdb/bugs/></a>.<br/> (gdb) attach 24391<br /> Attaching to process24391<br /> Reading symbols from /opt/pg92/bin/postgres...done.<br /> Reading symbols from /usr/lib64/libxslt.so.1...(nodebugging symbols found)...done.<br /> Loaded symbols for /usr/lib64/libxslt.so.1<br /> Readingsymbols from /usr/lib64/libxml2.so.2...(no debugging symbols found)...done.<br /> Loaded symbols for /usr/lib64/libxml2.so.2<br/> Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.<br /> Loadedsymbols for /lib64/libcrypt.so.1<br /> Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.<br/> Loaded symbols for /lib64/libdl.so.2<br /> Reading symbols from /lib64/libm.so.6...(no debugging symbolsfound)...done.<br /> Loaded symbols for /lib64/libm.so.6<br /> Reading symbols from /lib64/libc.so.6...(no debuggingsymbols found)...done.<br /> Loaded symbols for /lib64/libc.so.6<br /> Reading symbols from /usr/lib64/libz.so.1...(nodebugging symbols found)...done.<br /> Loaded symbols for /usr/lib64/libz.so.1<br /> Reading symbolsfrom /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.<br /> Loaded symbols for /lib64/ld-linux-x86-64.so.2<br/> Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.<br/> Loaded symbols for /lib64/libnss_files.so.2<br /> 0x00000039e4ed5337 in semop () from /lib64/libc.so.6<br/><font color="#ff0000"><small><b><big>(gdb) bt</big></b></small></font><br /> #0 0x00000039e4ed5337in semop () from /lib64/libc.so.6<br /> #1 0x00000000006985f3 in PGSemaphoreLock (sema=0x2ac6df58b850,interruptOK=0 '\000') at pg_sema.c:415<br /> #2 0x00000000006ea043 in LWLockAcquire (lockid=375058,mode=LW_EXCLUSIVE) at lwlock.c:474<br /> #3 0x00000000006d0739 in LockBuffer (buffer=187445, mode=2) at bufmgr.c:2533<br/> #4 0x00000000004d9ab0 in spgdoinsert (index=0x2ac6dfe42730, state=0x7fff015dad90, heapPtr=0x2a491c4,datum=44339704, isnull=0 '\000') at spgdoinsert.c:1940<br /> #5 0x00000000004d1880 in spginsert (fcinfo=0x7fff015dae40)at spginsert.c:220<br /> #6 0x000000000080c921 in FunctionCall6Coll (flinfo=0x296b360, collation=0,arg1=47033648162608, arg2=140733216305856, arg3=140733216305824, arg4=44339652, arg5=47033648129744, arg6=0)<br/> at fmgr.c:1439<br /> #7 0x0000000000489526 in index_insert (indexRelation=0x2ac6dfe42730, values=0x7fff015db2c0,isnull=0x7fff015db2a0 "", heap_t_ctid=0x2a491c4, heapRelation=0x2ac6dfe3a6d0, <br /> checkUnique=UNIQUE_CHECK_NO)at indexam.c:216<br /> #8 0x00000000005eaf3c in ExecInsertIndexTuples (slot=0x2a42420, tupleid=0x2a491c4,estate=0x2a42190) at execUtils.c:1087<br /> #9 0x00000000005fcfb5 in ExecInsert (slot=0x2a42420, planSlot=0x2a42420,estate=0x2a42190, canSetTag=1 '\001') at nodeModifyTable.c:248<br /> #10 0x00000000005fde2e in ExecModifyTable(node=0x2a424b0) at nodeModifyTable.c:848<br /> #11 0x00000000005ddde3 in ExecProcNode (node=0x2a424b0) atexecProcnode.c:376<br /> #12 0x00000000005dc24e in ExecutePlan (estate=0x2a42190, planstate=0x2a424b0, operation=CMD_INSERT,sendTuples=0 '\000', numberTuples=0, direction=ForwardScanDirection, dest=0x2a38f98)<br /> at execMain.c:1393<br/> #13 0x00000000005daa2a in standard_ExecutorRun (queryDesc=0x296a740, direction=ForwardScanDirection,count=0) at execMain.c:301<br /> #14 0x00000000005da922 in ExecutorRun (queryDesc=0x296a740,direction=ForwardScanDirection, count=0) at execMain.c:249<br /> #15 0x00000000006fb344 in ProcessQuery(plan=0x29f0d90, <br /> sourceText=0x29ef410 "insert into route_raw(sourceurl) values ('<a class="moz-txt-link-freetext"href="http://suz.bacts.com/group/group_33_128972.aspx" moz-do-not-send="true">http://suz.bacts.com/group/group_33_128972.aspx</a>, <a class="moz-txt-link-freetext" href="http://www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=GHX%B0%B2%D0%C4%C0%F6%C0%A520120613%B4%BF%CD%E6&xl=1262" moz-do-not-send="true"> http://www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=GHX%B0%B2%D0%C4%C0%F6%C0%A520120613%B4%BF%CD%E6&xl=1262</a>"..., params=0x0,dest=0x2a38f98, completionTag=0x7fff015db940 "") at pquery.c:185<br /> #16 0x00000000006fcaa9 in PortalRunMulti(portal=0x2968730, isTopLevel=1 '\001', dest=0x2a38f98, altdest=0x2a38f98, completionTag=0x7fff015db940 "")at pquery.c:1277<br /> #17 0x00000000006fc183 in PortalRun (portal=0x2968730, count=9223372036854775807, isTopLevel=1'\001', dest=0x2a38f98, altdest=0x2a38f98, completionTag=0x7fff015db940 "") at pquery.c:814<br /> #18 0x00000000006f66dain exec_simple_query (<br /> query_string=0x29ef410 "insert into route_raw(sourceurl) values ('<a class="moz-txt-link-freetext"href="http://suz.bacts.com/group/group_33_128972.aspx" moz-do-not-send="true">http://suz.bacts.com/group/group_33_128972.aspx</a>, <a class="moz-txt-link-freetext" href="http://www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=GHX%B0%B2%D0%C4%C0%F6%C0%A520120613%B4%BF%CD%E6&xl=1262" moz-do-not-send="true"> http://www.gzl.com.cn/Travel/TeamDetail.aspx?tuanhao=GHX%B0%B2%D0%C4%C0%F6%C0%A520120613%B4%BF%CD%E6&xl=1262</a>"...) atpostgres.c:1046<br /> #19 0x00000000006fa5bd in PostgresMain (argc=2, argv=0x2949640, username=0x29494d0 "postgres") atpostgres.c:3958<br /> #20 0x00000000006a9305 in BackendRun (port=0x296cd70) at postmaster.c:3619<br /> #21 0x00000000006a8a02in BackendStartup (port=0x296cd70) at postmaster.c:3304<br /> #22 0x00000000006a591a in ServerLoop () atpostmaster.c:1367<br /> #23 0x00000000006a52a8 in PostmasterMain (argc=3, argv=0x2947820) at postmaster.c:1127<br /> #240x000000000061cfcb in main (argc=3, argv=0x2947820) at main.c:199<br /><br /><font color="#ff0000"><b>(gdb) frame 4</b></font><br/> #4 0x00000000004d9ab0 in spgdoinsert (index=0x2ac6dfe42730, state=0x7fff015dad90, heapPtr=0x2a491c4, datum=44339704,isnull=0 '\000') at spgdoinsert.c:1940<br /> 1940 spgdoinsert.c: No such file or directory.<br /> inspgdoinsert.c<br /><br /><font color="#ff0000"><b>(gdb) info locals</b></font><br /> isNew = 0 '\000'<br /> level = 0<br/> leafDatum = 44339704<br /> leafSize = 844<br /> current = {blkno = 1, buffer = 187445, page = 0x0, offnum = 1, node= -1}<br /> parent = {blkno = 4294967295, buffer = 0, page = 0x0, offnum = 0, node = -1}<br /> __func__ = "spgdoinsert"<br/><font color="#ff0000"><b>(gdb) p *(SpGistCache *) index->rd_amcache</b></font><br /> $1 = {config ={prefixType = 25, labelType = 18, canReturnData = 1 '\001', longValuesOK = 1 '\001'}, attType = {type = 1043, attbyval =0 '\000', attlen = -1}, attPrefixType = {type = 25, <br /> attbyval = 0 '\000', attlen = -1}, attLabelType = {type =18, attbyval = 1 '\001', attlen = 1}, lastUsedPages = {cachedPage = {{blkno = 6174, freeSpace = 4308}, {blkno = 6172, <br/> freeSpace = 2024}, {blkno = 6173, freeSpace = 5244}, {blkno = 6217, freeSpace = 6228}, {blkno = 4294967295,freeSpace = 0}, {blkno = 4294967295, freeSpace = 0}, {blkno = 4294967295, <br /> freeSpace = 0}, {blkno= 4294967295, freeSpace = 0}}}}<br /><br /><pre wrap="">Thanks again Best Regards!</pre><br /><br /> 于 2013年01月09日 12:28, Tom Lane 写道:<br /></div><blockquote cite="mid:11459.1357705723@sss.pgh.pa.us"type="cite"><pre wrap="">I wrote: </pre><blockquote type="cite"><pre wrap="">The control flow in spgdoinsert.c is flat enough that the stack trace alone isn't much help in understanding the bug, I'm afraid. </pre></blockquote><pre wrap=""> BTW, something that possibly *would* help, since you seem to be able to reproduce the bug easily, is to do that and then capture the values of the local variables in spgdoinsert() -- especially the contents of the "current" and "parent" structs --- from each of the processes that are stuck. Also interesting would be to print the SpGistCache structs. It'd go something likeframe 4info localsp *(SpGistCache *) index->rd_amcache regards, tom lane </pre></blockquote><br />
pgsql-hackers by date: