Re: Parallel Sort - Mailing list pgsql-hackers

<div dir="ltr">2013/5/13 Noah Misch <span dir="ltr"><<a href="mailto:noah@leadboat.com"
target="_blank">noah@leadboat.com</a>></span><br/><div class="gmail_extra"><div class="gmail_quote"><blockquote
class="gmail_quote"style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">* Planner
&Similar Issues<br /><br /> We should decide whether to actually sort in parallel based on the comparator<br />
costand the data size.  The system currently has no information on comparator<br /> cost: bt*cmp (and indeed almost all
built-infunctions) all have procost=1,<br /> but bttextcmp is at least 1000x slower than btint4cmp.  Let's improve
the<br/> procost estimates of all core B-tree and hash operators.  This will have other<br /> benefits, but we will
needto be cognizant of the risk of upsetting setups<br /> that have tuned cpu_operator_cost based on the present
situation.<br/><br /> The choice of whether to parallelize can probably be made a manner similar to<br /> the choice to
doan external sort: the planner guesses the outcome for costing<br /> purposes, but the actual decision is made at
executiontime.  The planner<br /> would determine a tuple count cutoff at which parallelism becomes favorable,<br />
andtuplesort would check that to establish its actual decision.<br /><br /></blockquote><div style="style">It probably
crossoversmy <span style="color:rgb(0,0,0);font-family:Arial,'Arial New','\00ff2d\00ff33 \00ff30
\0030b4\0030b7\0030c3\0030af',sans-serif;font-size:13px">problemconsciousness to off-load CPU bounds</span></div><div
style="style"><spanstyle="color:rgb(0,0,0);font-family:Arial,'Arial New','\00ff2d\00ff33 \00ff30
\0030b4\0030b7\0030c3\0030af',sans-serif;font-size:13px">workloads;that I partially tried to implement on writable
foreigntable feature.</span></div><div style="style"><font color="#000000" face="Arial, Arial New, ＭＳ Ｐ ゴシック,
sans-serif">Notonly sorting stuff, I think it may be worthful to have capability to push</font></div><div
style="style"><fontcolor="#000000" face="Arial, Arial New, ＭＳ Ｐ ゴシック, sans-serif">heavy workload (like sort, aggregate
orcomplex target-list) out external</font></div><div style="style"><font color="#000000" face="Arial, Arial New, ＭＳ Ｐ
ゴシック,sans-serif">computing resources.</font></div><div style="style">However, I doubt whether the decision to
parallelizeshould be done in</div><div style="style">execution time, rather than plan stage. For example, in case when
we</div><divstyle="style">have enough number of records and 10-core multiprocessor, the wise</div><div
style="style">planmay take parallel data load by 10-processors, partial-sort by 10-</div><div style="style">processors
individually,then merge-sort. It needs fundamental different</div><div style="style">tree structure from the
traditionalsingle-processors based plan-tree.</div><div style="style">So, it seems to me we should take an enhancement
toallow to inject</div><div style="style">plan-tree special purpose parallel processing plan node.</div><div
style="style">Howabout your opinion?</div><div style="style"><br /></div><div
style="style">Thanks,</div></div></div></div>