I have started reviewing the patch and doing some testing, and I have pretty quickly ran into a segfault. Attached is a simple reproducer and an backtrace. AFAICS the bug seems to be somewhere in the tuplesort changes, likely resetting a memory context too soon or something like that. I haven't investigated it further, but it matches my hunch that tuplesort is likely where the bugs will be.
Right. Incremental sort patch introduces maincontext of memory which
is persistent between incremental sort groups. But mergeruns()
reallocates memtuples in sortcontext which is cleared by tuplesort_reset().
Fixed in the revised patch.
Otherwise the patch seems fairly complete. A couple of minor things that I noticed while eyeballing the changes in a diff editor.
1) On a couple of places the new code has this comment
/* even when not parallel-aware */
while all the immediately preceding blocks use
/* even when not parallel-aware, for EXPLAIN ANALYZE */
I suggest using the same comment, otherwise it kinda suggests it's not because of EXPLAIN ANALYZE.
Right, fixed. I also found that incremental sort shoudn't support
DSM reinitialization similarly to regular sort. Fixes in the revised patch.
2) I think the purpose of sampleSlot should be explicitly documented (and I'm not sure "sample" is a good term here, as is suggest some sort of sampling (for example nodeAgg uses grp_firstTuple).
Yes, "sample" isn't a good term here. However, "first" isn't really correct,
because we can skip some tuples from beginning of the group in
order to not form groups too frequently. I'd rather name it "pivot" tuple
slot.
3) skipCols/SkipKeyData seems a bit strange too, I think. I'd use PresortedKeyData or something like that.
Good point, renamed.
4) In cmpSortSkipCols, when checking if the columns changed, the patch does this: n = ((IncrementalSort *) node->ss.ps.plan)->skipCols;
for (i = 0; i < n; i++) { ... check i-th key ... }
My hunch is that checking the keys from the last one, i.e.
for (i = (n-1); i >= 0; i--) { .... }
would be faster. The reasoning is that with "ORDER BY a,b" the column "b" changes more often. But I've been unable to test this because of the segfault crashes.
Agreed.
5) The changes from
if (pathkeys_contained_in(...))
to
n = pathkeys_common(pathkeys, subpath->pathkeys);
if (n == 0)
seem rather inconvenient to me, as it makes the code unnecessarily verbose. I wonder if there's a better way to deal with this.
I would rather say, that it changes from
if (pathkeys_contained_in(...))
to
n = pathkeys_common(pathkeys, subpath->pathkeys);
if (n == list_length(pathkeys))
I've introduced pathkeys_common_contained_in() which returns the same
result as pathkeys_contained_in(), but sets number of common pathkeys
to the last argument. It simplifies code a little bit. The name, probably,