Thread: Question: test "aggregates" failed in 32-bit machine
Hi hackers, While running `make check LANC=C` with 32-bit virtual machine, I found that it was failed at "aggregates". PSA the a1b3bca1_regression.diffs. IIUC that part has been added by db0d67db. I checked out the source, tested, and got same result. PSA the db0d67db_regression.diffs I'm not sure about it, but is it an expected behavior? I know that we do not have to consider about "row" ordering, Followings show the environment. Please tell me if another information is needed. OS: RHEL 6.10 server Arch: i686 Gcc: 4.4.7 $ uname -a Linux VMXXXXX 2.6.32-754.41.2.el6.i686 #1 SMP Sat Jul 10 04:21:20 EDT 2021 i686 i686 i386 GNU/Linux Configure option: --enable-cassert --enable-debug --enable-tap-tests Best Regards, Hayato Kuroda FUJITSU LIMITED
Attachment
"kuroda.hayato@fujitsu.com" <kuroda.hayato@fujitsu.com> writes:
> While running `make check LANC=C` with 32-bit virtual machine,
> I found that it was failed at "aggregates".
Hmm, we're not seeing any such failures in the buildfarm's 32-bit
animals, so there must be some additional condition needed to make
it happen. Can you be more specific?
regards, tom lane
Dear Tom, > Hmm, we're not seeing any such failures in the buildfarm's 32-bit > animals, so there must be some additional condition needed to make > it happen. Can you be more specific? Hmm, I was not sure about additional conditions, sorry. I could reproduce with followings steps: $ git clone https://github.com/postgres/postgres.git $ cd postgres $ ./configure --enable-cassert --enable-debug $ make -j2 $ make check LANG=C -> aggregates ... FAILED 3562 ms The hypervisor of the virtual machine is " VMware vSphere 7.0" And I picked another information related with the machine. Could you find something? ``` pg_config]$ ./pg_config ... CONFIGURE = '--enable-cassert' '--enable-debug' CC = gcc -std=gnu99 CPPFLAGS = -D_GNU_SOURCE CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute-Wformat-security -fno-strict-aliasing -fwrapv -g -O2 CFLAGS_SL = -fPIC LDFLAGS = -Wl,--as-needed -Wl,-rpath,'/usr/local/pgsql/lib',--enable-new-dtags LDFLAGS_EX = LDFLAGS_SL = LIBS = -lpgcommon -lpgport -lz -lreadline -lrt -ldl -lm VERSION = PostgreSQL 16devel $ locale LANG=C ... $ arch i686 $cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 85 model name : Intel(R) Xeon(R) Platinum 8260 CPU @ 2.40GHz stepping : 7 microcode : 83898371 cpu MHz : 2394.374 cache size : 36608 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 22 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss nxpdpe1gb rdtscp lm constant_tsc arch_perfmon xtopology tsc_reliable nonstop_tsc unfair_spinlock eagerfpu pni pclmulqdq ssse3fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm3dnowprefetch arat xsaveopt ssbd ibrs ibpb stibp fsgsbase bmi1 avx2 smep bmi2 invpcid avx512f rdseed adx avx512cd md_clearflush_l1d arch_capabilities bogomips : 4788.74 clflush size : 64 cache_alignment : 64 address sizes : 43 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 85 model name : Intel(R) Xeon(R) Platinum 8260 CPU @ 2.40GHz stepping : 7 microcode : 83898371 cpu MHz : 2394.374 cache size : 36608 KB physical id : 2 siblings : 1 core id : 0 cpu cores : 1 apicid : 2 initial apicid : 2 fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 22 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss nxpdpe1gb rdtscp lm constant_tsc arch_perfmon xtopology tsc_reliable nonstop_tsc unfair_spinlock eagerfpu pni pclmulqdq ssse3fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm3dnowprefetch arat xsaveopt ssbd ibrs ibpb stibp fsgsbase bmi1 avx2 smep bmi2 invpcid avx512f rdseed adx avx512cd md_clearflush_l1d arch_capabilities bogomips : 4788.74 clflush size : 64 cache_alignment : 64 address sizes : 43 bits physical, 48 bits virtual power management ``` Best Regards, Hayato Kuroda FUJITSU LIMITED
"kuroda.hayato@fujitsu.com" <kuroda.hayato@fujitsu.com> writes:
> Hmm, I was not sure about additional conditions, sorry.
> I could reproduce with followings steps:
I tried this on a 32-bit VM with gcc 11.3, but couldn't reproduce.
You said earlier
>> OS: RHEL 6.10 server
>> Arch: i686
>> Gcc: 4.4.7
That is an awfully old compiler; I fear I no longer have anything
comparable on a working platform.
The most likely theory, I think, is that that compiler is generating
slightly different floating-point code causing different plans to
be costed slightly differently than what the test case is expecting.
Probably, the different orderings of the keys in this test case have
exactly the same cost, or almost exactly, so that different roundoff
error could be enough to change the selected plan.
This probably doesn't have a lot of real-world impact, but it's
still annoying on a couple of grounds. Failing regression isn't
nice, and also this suggests that db0d67db2 is causing us to waste
time considering multiple plans with effectively equal costs.
Maybe that code needs to filter a little harder.
regards, tom lane
Hi, On 2022-09-30 12:13:11 -0400, Tom Lane wrote: > "kuroda.hayato@fujitsu.com" <kuroda.hayato@fujitsu.com> writes: > > Hmm, I was not sure about additional conditions, sorry. > > I could reproduce with followings steps: > > I tried this on a 32-bit VM with gcc 11.3, but couldn't reproduce. > You said earlier > > >> OS: RHEL 6.10 server > >> Arch: i686 > >> Gcc: 4.4.7 > > That is an awfully old compiler; I fear I no longer have anything > comparable on a working platform. > > The most likely theory, I think, is that that compiler is generating > slightly different floating-point code causing different plans to > be costed slightly differently than what the test case is expecting. > Probably, the different orderings of the keys in this test case have > exactly the same cost, or almost exactly, so that different roundoff > error could be enough to change the selected plan. Yea. I suspect that's because that compiler version doesn't have -fexcess-precision=standard: > CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute-Wformat-security -fno-strict-aliasing -fwrapv -g -O2 It's possible one could work around the issue with -msse -mfpmath=sse instead of -fexcess-precision=standard. Greetings, Andres Freund
I wrote:
> The most likely theory, I think, is that that compiler is generating
> slightly different floating-point code causing different plans to
> be costed slightly differently than what the test case is expecting.
> Probably, the different orderings of the keys in this test case have
> exactly the same cost, or almost exactly, so that different roundoff
> error could be enough to change the selected plan.
I added some debug printouts to get_cheapest_group_keys_order()
and verified that in the two problematic queries, there are two
different orderings that have (on my machine) exactly equal lowest
cost. So the code picks the first of those and ignores the second.
Different roundoff error would be enough to make it do something
else.
I find this problematic because "exactly equal" costs are not going
to be unusual. That's because the values that cost_sort_estimate
relies on are, sadly, just about completely fictional. It's expecting
that it can get a good cost estimate based on:
* procost. In case you hadn't noticed, this is going to be 1 for
just about every function we might be considering here.
* column width. This is either going to be a constant (e.g. 4
for integers) or, again, largely fictional. The logic for
converting widths to cost multipliers adds yet another layer
of debatability.
* numdistinct estimates. Sometimes we know what we're talking
about there, but often we don't.
So what I'm afraid we are dealing with here is usually going to
be garbage in, garbage out. And we're expending an awful lot
of code and cycles to arrive at these highly questionable choices.
Given the previous complaints about db0d67db2, I wonder if it's not
most prudent to revert it. I doubt we are going to get satisfactory
behavior out of it until there's fairly substantial improvements in
all these underlying estimates.
regards, tom lane
I wrote:
> Given the previous complaints about db0d67db2, I wonder if it's not
> most prudent to revert it. I doubt we are going to get satisfactory
> behavior out of it until there's fairly substantial improvements in
> all these underlying estimates.
After spending some more time looking at the code, I think that that
is something we absolutely have to discuss. I already complained at
[1] about how db0d67db2 made very significant changes in sort cost
estimation behavior, which seem likely to result in significant
user-visible plan changes that might or might not be for the better.
But I hadn't read any of the code at that point. Now I have, and
frankly it's not ready for prime time. Beyond the question of
whether we have sufficiently accurate input values, I see these
issues in and around compute_cpu_sort_cost():
1. The algorithm is said to be based on Sedgewick & Bentley 2002 [2].
I have the highest regard for those two gentlemen, so I'm quite
prepared to believe that their estimate of the number of comparisons
used by Quicksort is good. However, the expression given in our
comments:
* log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
doesn't look much like anything they wrote. More, what we're actually
doing is
* We assume all Xi the same because now we don't have any estimation of
* group sizes, we have only know the estimate of number of groups (distinct
* values). In that case, formula becomes:
* N * log(NumberOfGroups)
That's a pretty drastic simplification. No argument is given as to why
that's still reliable enough to be useful for the purposes to which this
code tries to put it --- especially when you consider that real-world
data is more likely to follow Zipf's law than have uniform group sizes.
If you're going to go as far as doing this:
* For multi-column sorts we need to estimate the number of comparisons for
* each individual column - for example with columns (c1, c2, ..., ck) we
* can estimate that number of comparisons on ck is roughly
* ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
you'd better pray that your number-of-comparisons estimates are pretty
darn good, or what you're going to get out is going to be mostly
fiction.
2. Sedgewick & Bentley analyzed a specific version of Quicksort,
which is ... um ... not the version we are using. It doesn't look
to me like the choice of partitioning element is the same. Maybe
that doesn't matter much in the end, but there's sure no discussion
of the point in this patch.
So at this point I've lost all faith in the estimates being meaningful
at all. And that's assuming that the simplified algorithm is
implemented accurately, which it is not:
3. totalFuncCost is started off at 1.0. Surely that should be zero?
If not, at least a comment to justify it would be nice.
4. The code around the add_function_cost call evidently wants to carry
the procost lookup result from one column to the next, because it
skips the lookup when prev_datatype == em->em_datatype. However, the
value of funcCost isn't carried across columns, because it's local to
the loop. The effect of this is that anyplace where adjacent GROUP BY
columns are of the same datatype, we'll use the fixed 1.0 value of
funcCost instead of looking up the real procost. Admittedly, since
the real procost is probably also 1.0, this might not mean much in
practice. Nonetheless it's broken code. (Oh, btw: I doubt that
using add_function_cost rather than raw procost is of any value
whatsoever if you're just going to pass it a NULL node tree.)
5. I'm pretty dubious about the idea that we can use the rather-random
first element of the EquivalenceClass to determine the datatype that
will be compared, much less the average widths of the columns. It's
entirely possible for an EC to contain both int4 and int8 vars, or
text vars of substantially different average widths. I think we
really need to be going back to the original GroupClauses and looking
at the variables named there.
6. Worse than that, we're also using the first element of the
EquivalenceClass to calculate the number of groups of this sort key.
This is FLAT OUT WRONG, as certainly different EC members can have
very different stats.
7. The code considers that presorted-key columns do not add to the
comparison costs, yet the comment about it claims the opposite:
/*
* Presorted keys are not considered in the cost above, but we still
* do have to compare them in the qsort comparator. So make sure to
* factor in the cost in that case.
*/
if (i >= nPresortedKeys)
{
I'm not entirely sure whether the code is broken or the comment is,
but at least one of them is. I'm also pretty confused about why
we still add such columns' comparison functions to the running
totalFuncCost if we think they're not sorted on.
8. In the case complained of to start this thread, we're unable
to perceive any sort-cost difference between "p, d, c, v" and
"p, c, d, v", which is a little surprising because that test case
sets up c with twice as many distinct values as d. Other things
being equal (which they are, because both columns are int4), surely
the latter key ordering should be favored in hopes of reducing the
number of times we have to compare the third column. But it's not.
I think that this can probably be blamed on the early-exit condition
at the bottom of the loop:
/*
* Once we get single-row group, it means tuples in the group are
* unique and we can skip all remaining columns.
*/
if (tuplesPerPrevGroup <= 1.0)
break;
Ordering on p already gets us down to 2 tuples per group, so pretty
much any of the other columns as second grouping column will compute
a next group size of 1, and then we don't consider columns beyond that.
9. The is_fake_var() hackery is pretty sad. We should have found a
better solution than that. Maybe estimate_num_groups() needs more
work.
10. As I already mentioned, get_width_cost_multiplier() doesn't appear
to have any foundation in reality; or if it does, the comments sure
provide no justification for these particular equations rather than
some other ones. The shakiness of the logic can be inferred
immediately from the fact that the header comment is fundamentally
confused about what it's doing:
* Return value is in cpu_operator_cost units.
No it isn't, it's a pure ratio.
In short, I think the claim that this code provides better sort cost
estimates than we had before is quite unjustified. Maybe it could
get there eventually, but I do not want to ship v15 with this.
I think we ought to revert all the changes around cost_sort.
Perhaps we could salvage the GROUP BY changes by just ordering the
columns by decreasing number of groups, which is the only component of
the current cost estimation that I think has any detectable connection
to reality. But I suspect the RMT will favor just reverting the
whole thing for v15.
regards, tom lane
[1] https://www.postgresql.org/message-id/3242058.1659563057%40sss.pgh.pa.us
[2] The URL given in the code doesn't work anymore, but this does:
https://sedgewick.io/wp-content/uploads/2022/03/2002QuicksortIsOptimal.pdf
I wrote:
> So at this point I've lost all faith in the estimates being meaningful
> at all.
I spent some time today looking into the question of what our qsort
code actually does. I wrote a quick-n-dirty little test module
(attached) to measure the number of comparisons qsort really uses
for assorted sample inputs. The results will move a bit from run
to run because of randomization, but the average counts should be
pretty stable I think. I got results like these:
regression=# create temp table data as
select * from qsort_comparisons(100000);
SELECT 10
regression=# select n * log(groups)/log(2) as est, 100*(n * log(groups)/log(2) - avg_cmps)/avg_cmps as pct_err, * from
data;
est | pct_err | n | groups | avg_cmps | min_cmps | max_cmps | note
--------------------+--------------------+--------+--------+----------+----------+----------+------------------------
0 | -100 | 100000 | 1 | 99999 | 99999 | 99999 | all values the same
1660964.0474436812 | -5.419880052975057 | 100000 | 100000 | 1756145 | 1722569 | 1835627 | all values distinct
100000 | -33.33911061041376 | 100000 | 2 | 150013 | 150008 | 150024 | 2 distinct values
400000 | 11.075628618635713 | 100000 | 16 | 360115 | 337586 | 431376 | 16 distinct values
600000 | 8.369757612975473 | 100000 | 64 | 553660 | 523858 | 639492 | 64 distinct values
800000 | 4.770461016221087 | 100000 | 256 | 763574 | 733898 | 844450 | 256 distinct values
1000000 | 1.5540821186618827 | 100000 | 1024 | 984697 | 953830 | 1111384 | 1024 distinct values
1457116.0087927429 | 41.97897366170798 | 100000 | 24342 | 1026290 | 994694 | 1089503 | Zipfian, parameter 1.1
1150828.9986140348 | 158.28880094758154 | 100000 | 2913 | 445559 | 426575 | 511214 | Zipfian, parameter 1.5
578135.9713524659 | 327.6090378488971 | 100000 | 55 | 135202 | 132541 | 213467 | Zipfian, parameter 3.0
(10 rows)
So "N * log(NumberOfGroups)" is a pretty solid estimate for
uniformly-sized groups ... except when NumberOfGroups = 1 ... but it
is a significant overestimate if the groups aren't uniformly sized.
Now a factor of 2X or 3X isn't awful --- we're very happy to accept
estimates only that good in other contexts --- but I still wonder
whether this is reliable enough to justify the calculations being
done in compute_cpu_sort_cost. I'm still very afraid that the
conclusions we're drawing about the sort costs for different column
orders are mostly junk.
In any case, something's got to be done about the failure at
NumberOfGroups = 1. Instead of this:
correctedNGroups = Max(1.0, ceil(correctedNGroups));
per_tuple_cost += totalFuncCost * LOG2(correctedNGroups);
I suggest
if (correctedNGroups > 1.0)
per_tuple_cost += totalFuncCost * LOG2(correctedNGroups);
else /* Sorting N all-alike tuples takes only N-1 comparisons */
per_tuple_cost += totalFuncCost;
(Note that the ceil() here is a complete waste, because all paths leading
to this produced integral estimates already. Even if they didn't, I see
no good argument why ceil() makes the result better.)
I'm still of the opinion that we need to revert this code for now.
regards, tom lane
/*
create function qsort_comparisons(N int)
returns table (N int, groups int,
avg_cmps int8, min_cmps int8, max_cmps int8,
note text)
strict volatile language c as '/path/to/count_comparisons.so';
select * from qsort_comparisons(10000);
*/
#include "postgres.h"
#include <math.h>
#include "common/pg_prng.h"
#include "fmgr.h"
#include "funcapi.h"
#include "miscadmin.h"
#include "utils/tuplestore.h"
PG_MODULE_MAGIC;
/*
* qsort_arg comparator function for integers.
* The number of calls is tracked in *arg.
*/
static int
cmp_func(const void *a, const void *b, void *arg)
{
int aa = *(const int *) a;
int bb = *(const int *) b;
size_t *count_ptr = (size_t *) arg;
(*count_ptr)++;
if (aa < bb)
return -1;
if (aa > bb)
return 1;
return 0;
}
/*
* Randomly shuffle an array of integers.
*/
static void
shuffle(int *x, int n)
{
/*
* Shuffle array using Fisher-Yates algorithm. Iterate array and swap head
* (i) with a randomly chosen same-or-later item (j) at each iteration.
*/
for (int i = 0; i < n; i++)
{
int j = (int) pg_prng_uint64_range(&pg_global_prng_state,
i, n - 1);
int tmp = x[i];
x[i] = x[j];
x[j] = tmp;
}
}
/*
* Test qsort for the given test case.
* Shuffle the given array, sort it using qsort_arg,
* and report the numbers of comparisons made.
*/
static void
qsort_test_case(int *array, int n, char *note, int trials,
Tuplestorestate *tupstore, AttInMetadata *attinmeta)
{
size_t tot_count = 0;
size_t min_count = SIZE_MAX;
size_t max_count = 0;
int groups;
HeapTuple tuple;
char *values[6];
char v0[32];
char v1[32];
char v2[32];
char v3[32];
char v4[32];
for (int t = 0; t < trials; t++)
{
size_t count = 0;
CHECK_FOR_INTERRUPTS();
shuffle(array, n);
qsort_arg(array, n, sizeof(int), cmp_func, &count);
tot_count += count;
min_count = Min(min_count, count);
max_count = Max(max_count, count);
}
/* count groups in sorted array */
groups = 1;
for (int i = 1; i < n; i++)
{
if (array[i - 1] != array[i])
groups++;
}
snprintf(v0, sizeof(v0), "%d", n);
snprintf(v1, sizeof(v1), "%d", groups);
snprintf(v2, sizeof(v2), "%zd", tot_count / trials);
snprintf(v3, sizeof(v3), "%zd", min_count);
snprintf(v4, sizeof(v4), "%zd", max_count);
values[0] = v0;
values[1] = v1;
values[2] = v2;
values[3] = v3;
values[4] = v4;
values[5] = note;
tuple = BuildTupleFromCStrings(attinmeta, values);
tuplestore_puttuple(tupstore, tuple);
heap_freetuple(tuple);
}
/* zipfian code borrowed from pgbench */
/*
* Computing zipfian using rejection method, based on
* "Non-Uniform Random Variate Generation",
* Luc Devroye, p. 550-551, Springer 1986.
*
* This works for s > 1.0, but may perform badly for s very close to 1.0.
*/
static int64
computeIterativeZipfian(pg_prng_state *state, int64 n, double s)
{
double b = pow(2.0, s - 1.0);
double x,
t,
u,
v;
/* Ensure n is sane */
if (n <= 1)
return 1;
while (true)
{
/* random variates */
u = pg_prng_double(state);
v = pg_prng_double(state);
x = floor(pow(u, -1.0 / (s - 1.0)));
t = pow(1.0 + 1.0 / x, s - 1.0);
/* reject if too large or out of bound */
if (v * x * (t - 1.0) / (b - 1.0) <= t / b && x <= n)
break;
}
return (int64) x;
}
/* random number generator: zipfian distribution from min to max inclusive */
static int64
getZipfianRand(pg_prng_state *state, int64 min, int64 max, double s)
{
int64 n = max - min + 1;
return min - 1 + computeIterativeZipfian(state, n, s);
}
/*
* qsort_comparisons(N int) returns table
*/
PG_FUNCTION_INFO_V1(qsort_comparisons);
Datum
qsort_comparisons(PG_FUNCTION_ARGS)
{
int32 N = PG_GETARG_INT32(0);
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
Tuplestorestate *tupstore;
TupleDesc tupdesc;
AttInMetadata *attinmeta;
int *array;
MemoryContext per_query_ctx;
MemoryContext oldcontext;
/* check to see if caller supports us returning a tuplestore */
if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("set-valued function called in context that cannot accept a set")));
if (!(rsinfo->allowedModes & SFRM_Materialize))
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("materialize mode required, but it is not allowed in this context")));
per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
/* get a tuple descriptor for our result type */
switch (get_call_result_type(fcinfo, NULL, &tupdesc))
{
case TYPEFUNC_COMPOSITE:
/* success */
break;
case TYPEFUNC_RECORD:
/* failed to determine actual type of RECORD */
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("function returning record called in context "
"that cannot accept type record")));
break;
default:
/* result type isn't composite */
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("return type must be a row type")));
break;
}
/* make sure we have a persistent copy of the result tupdesc */
oldcontext = MemoryContextSwitchTo(per_query_ctx);
tupdesc = CreateTupleDescCopy(tupdesc);
/* initialize our tuplestore in long-lived context */
tupstore =
tuplestore_begin_heap(rsinfo->allowedModes & SFRM_Materialize_Random,
false, work_mem);
MemoryContextSwitchTo(oldcontext);
/* Set up to construct tuples */
attinmeta = TupleDescGetAttInMetadata(tupdesc);
/* Allocate test array */
array = (int *) palloc(N * sizeof(int));
/* All-the-same */
for (int i = 0; i < N; i++)
array[i] = 1;
qsort_test_case(array, N, "all values the same", 10,
tupstore, attinmeta);
/* All different */
for (int i = 0; i < N; i++)
array[i] = i;
qsort_test_case(array, N, "all values distinct", 1000,
tupstore, attinmeta);
/* Various uniform group sizes */
for (int i = 0; i < N; i++)
array[i] = i % 2;
qsort_test_case(array, N, "2 distinct values", 1000,
tupstore, attinmeta);
for (int i = 0; i < N; i++)
array[i] = i % 16;
qsort_test_case(array, N, "16 distinct values", 1000,
tupstore, attinmeta);
for (int i = 0; i < N; i++)
array[i] = i % 64;
qsort_test_case(array, N, "64 distinct values", 1000,
tupstore, attinmeta);
for (int i = 0; i < N; i++)
array[i] = i % 256;
qsort_test_case(array, N, "256 distinct values", 1000,
tupstore, attinmeta);
for (int i = 0; i < N; i++)
array[i] = i % 1024;
qsort_test_case(array, N, "1024 distinct values", 1000,
tupstore, attinmeta);
/* Zipfian */
for (int i = 0; i < N; i++)
array[i] = (int) getZipfianRand(&pg_global_prng_state, 0, 1000000,
1.1);
qsort_test_case(array, N, "Zipfian, parameter 1.1", 1000,
tupstore, attinmeta);
for (int i = 0; i < N; i++)
array[i] = (int) getZipfianRand(&pg_global_prng_state, 0, 1000000,
1.5);
qsort_test_case(array, N, "Zipfian, parameter 1.5", 1000,
tupstore, attinmeta);
for (int i = 0; i < N; i++)
array[i] = (int) getZipfianRand(&pg_global_prng_state, 0, 1000000,
3.0);
qsort_test_case(array, N, "Zipfian, parameter 3.0", 1000,
tupstore, attinmeta);
/* let the caller know we're sending back a tuplestore */
rsinfo->returnMode = SFRM_Materialize;
rsinfo->setResult = tupstore;
rsinfo->setDesc = tupdesc;
return (Datum) 0;
}
On Sat, Oct 1, 2022 at 12:14 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > I spent some time today looking into the question of what our qsort > code actually does. I wrote a quick-n-dirty little test module > (attached) to measure the number of comparisons qsort really uses > for assorted sample inputs. Reminds me of the other sort testing program that you wrote when the B&M code first went in: https://www.postgresql.org/message-id/18732.1142967137@sss.pgh.pa.us This was notable for recreating the tests from the original B&M paper. The paper uses various types of test inputs with characteristics that were challenging to the implementation and worth specifically getting right. For example, "saw tooth" input. -- Peter Geoghegan
Peter Geoghegan <pg@bowt.ie> writes: > Reminds me of the other sort testing program that you wrote when the > B&M code first went in: > https://www.postgresql.org/message-id/18732.1142967137@sss.pgh.pa.us Ha, I'd totally forgotten about that ... regards, tom lane
On 10/1/22 3:13 PM, Tom Lane wrote: > I'm still of the opinion that we need to revert this code for now. [RMT hat, but speaking just for me] reading through Tom's analysis, this seems to be the safest path forward. I have a few questions to better understand: 1. How invasive would the revert be? 2. Are the other user-visible items that would be impacted? 3. Is there an option of disabling the feature by default viable? Thanks, Jonathan
Attachment
"Jonathan S. Katz" <jkatz@postgresql.org> writes:
> On 10/1/22 3:13 PM, Tom Lane wrote:
>> I'm still of the opinion that we need to revert this code for now.
> [RMT hat, but speaking just for me] reading through Tom's analysis, this
> seems to be the safest path forward. I have a few questions to better
> understand:
> 1. How invasive would the revert be?
I've just finished constructing a draft full-reversion patch. I'm not
confident in this yet; in particular, teasing it apart from 1349d2790
("Improve performance of ORDER BY / DISTINCT aggregates") was fairly
messy. I need to look through the regression test changes and make
sure that none are surprising. But this is approximately the right
scope if we rip it out entirely.
I plan to have a look tomorrow at the idea of reverting only the cost_sort
changes, and rewriting get_cheapest_group_keys_order() to just sort the
keys by decreasing numgroups estimates as I suggested upthread. That
might be substantially less messy, because of fewer interactions with
1349d2790.
> 2. Are the other user-visible items that would be impacted?
See above. (But note that 1349d2790 is HEAD-only, not in v15.)
> 3. Is there an option of disabling the feature by default viable?
Not one that usefully addresses my concerns. The patch did add an
enable_group_by_reordering GUC which we could change to default-off,
but it does nothing about the cost_sort behavioral changes. I would
be a little inclined to rip out that GUC in either case, because
I doubt that we need it with the more restricted change.
regards, tom lane
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 2e4e82a94f..cc9e39c4a5 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -2862,13 +2862,16 @@ select c2 * (random() <= 1)::int as c2 from ft2 group by c2 * (random() <= 1)::i
-- GROUP BY clause in various forms, cardinal, alias and constant expression
explain (verbose, costs off)
select count(c2) w, c2 x, 5 y, 7.0 z from ft1 group by 2, y, 9.0::int order by 2;
- QUERY PLAN
-------------------------------------------------------------------------------------------------------------
- Foreign Scan
+ QUERY PLAN
+---------------------------------------------------------------------------------------
+ Sort
Output: (count(c2)), c2, 5, 7.0, 9
- Relations: Aggregate on (public.ft1)
- Remote SQL: SELECT count(c2), c2, 5, 7.0, 9 FROM "S 1"."T 1" GROUP BY 2, 3, 5 ORDER BY c2 ASC NULLS LAST
-(4 rows)
+ Sort Key: ft1.c2
+ -> Foreign Scan
+ Output: (count(c2)), c2, 5, 7.0, 9
+ Relations: Aggregate on (public.ft1)
+ Remote SQL: SELECT count(c2), c2, 5, 7.0, 9 FROM "S 1"."T 1" GROUP BY 2, 3, 5
+(7 rows)
select count(c2) w, c2 x, 5 y, 7.0 z from ft1 group by 2, y, 9.0::int order by 2;
w | x | y | z
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d8848bc774..d750290f13 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -5062,20 +5062,6 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
</listitem>
</varlistentry>
- <varlistentry id="guc-enable-groupby-reordering" xreflabel="enable_group_by_reordering">
- <term><varname>enable_group_by_reordering</varname> (<type>boolean</type>)
- <indexterm>
- <primary><varname>enable_group_by_reordering</varname> configuration parameter</primary>
- </indexterm>
- </term>
- <listitem>
- <para>
- Enables or disables reordering of keys in a <literal>GROUP BY</literal>
- clause. The default is <literal>on</literal>.
- </para>
- </listitem>
- </varlistentry>
-
<varlistentry id="guc-enable-hashagg" xreflabel="enable_hashagg">
<term><varname>enable_hashagg</varname> (<type>boolean</type>)
<indexterm>
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index f486d42441..5ef29eea69 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1814,327 +1814,6 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
rterm->pathtarget->width);
}
-/*
- * is_fake_var
- * Workaround for generate_append_tlist() which generates fake Vars with
- * varno == 0, that will cause a fail of estimate_num_group() call
- *
- * XXX Ummm, why would estimate_num_group fail with this?
- */
-static bool
-is_fake_var(Expr *expr)
-{
- if (IsA(expr, RelabelType))
- expr = (Expr *) ((RelabelType *) expr)->arg;
-
- return (IsA(expr, Var) && ((Var *) expr)->varno == 0);
-}
-
-/*
- * get_width_cost_multiplier
- * Returns relative complexity of comparing two values based on its width.
- * The idea behind is that the comparison becomes more expensive the longer the
- * value is. Return value is in cpu_operator_cost units.
- */
-static double
-get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
-{
- double width = -1.0; /* fake value */
-
- if (IsA(expr, RelabelType))
- expr = (Expr *) ((RelabelType *) expr)->arg;
-
- /* Try to find actual stat in corresponding relation */
- if (IsA(expr, Var))
- {
- Var *var = (Var *) expr;
-
- if (var->varno > 0 && var->varno < root->simple_rel_array_size)
- {
- RelOptInfo *rel = root->simple_rel_array[var->varno];
-
- if (rel != NULL &&
- var->varattno >= rel->min_attr &&
- var->varattno <= rel->max_attr)
- {
- int ndx = var->varattno - rel->min_attr;
-
- if (rel->attr_widths[ndx] > 0)
- width = rel->attr_widths[ndx];
- }
- }
- }
-
- /* Didn't find any actual stats, try using type width instead. */
- if (width < 0.0)
- {
- Node *node = (Node *) expr;
-
- width = get_typavgwidth(exprType(node), exprTypmod(node));
- }
-
- /*
- * Values are passed as Datum type, so comparisons can't be cheaper than
- * comparing a Datum value.
- *
- * FIXME I find this reasoning questionable. We may pass int2, and
- * comparing it is probably a bit cheaper than comparing a bigint.
- */
- if (width <= sizeof(Datum))
- return 1.0;
-
- /*
- * We consider the cost of a comparison not to be directly proportional to
- * width of the argument, because widths of the arguments could be
- * slightly different (we only know the average width for the whole
- * column). So we use log16(width) as an estimate.
- */
- return 1.0 + 0.125 * LOG2(width / sizeof(Datum));
-}
-
-/*
- * compute_cpu_sort_cost
- * compute CPU cost of sort (i.e. in-memory)
- *
- * The main thing we need to calculate to estimate sort CPU costs is the number
- * of calls to the comparator functions. The difficulty is that for multi-column
- * sorts there may be different data types involved (for some of which the calls
- * may be much more expensive). Furthermore, columns may have a very different
- * number of distinct values - the higher the number, the fewer comparisons will
- * be needed for the following columns.
- *
- * The algorithm is incremental - we add pathkeys one by one, and at each step we
- * estimate the number of necessary comparisons (based on the number of distinct
- * groups in the current pathkey prefix and the new pathkey), and the comparison
- * costs (which is data type specific).
- *
- * Estimation of the number of comparisons is based on ideas from:
- *
- * "Quicksort Is Optimal", Robert Sedgewick, Jon Bentley, 2002
- * [https://www.cs.princeton.edu/~rs/talks/QuicksortIsOptimal.pdf]
- *
- * In term of that paper, let N - number of tuples, Xi - number of identical
- * tuples with value Ki, then the estimate of number of comparisons is:
- *
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- *
- * We assume all Xi the same because now we don't have any estimation of
- * group sizes, we have only know the estimate of number of groups (distinct
- * values). In that case, formula becomes:
- *
- * N * log(NumberOfGroups)
- *
- * For multi-column sorts we need to estimate the number of comparisons for
- * each individual column - for example with columns (c1, c2, ..., ck) we
- * can estimate that number of comparisons on ck is roughly
- *
- * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
- *
- * Let k be a column number, Gk - number of groups defined by k columns, and Fk
- * the cost of the comparison is
- *
- * N * sum( Fk * log(Gk) )
- *
- * Note: We also consider column width, not just the comparator cost.
- *
- * NOTE: some callers currently pass NIL for pathkeys because they
- * can't conveniently supply the sort keys. In this case, it will fallback to
- * simple comparison cost estimate.
- */
-static Cost
-compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
- Cost comparison_cost, double tuples, double output_tuples,
- bool heapSort)
-{
- Cost per_tuple_cost = 0.0;
- ListCell *lc;
- List *pathkeyExprs = NIL;
- double tuplesPerPrevGroup = tuples;
- double totalFuncCost = 1.0;
- bool has_fake_var = false;
- int i = 0;
- Oid prev_datatype = InvalidOid;
- List *cache_varinfos = NIL;
-
- /* fallback if pathkeys is unknown */
- if (pathkeys == NIL)
- {
- /*
- * If we'll use a bounded heap-sort keeping just K tuples in memory,
- * for a total number of tuple comparisons of N log2 K; but the
- * constant factor is a bit higher than for quicksort. Tweak it so
- * that the cost curve is continuous at the crossover point.
- */
- output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
- per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
-
- /* add cost provided by caller */
- per_tuple_cost += comparison_cost;
-
- return per_tuple_cost * tuples;
- }
-
- /*
- * Computing total cost of sorting takes into account the per-column
- * comparison function cost. We try to compute the needed number of
- * comparisons per column.
- */
- foreach(lc, pathkeys)
- {
- PathKey *pathkey = (PathKey *) lfirst(lc);
- EquivalenceMember *em;
- double nGroups,
- correctedNGroups;
- Cost funcCost = 1.0;
-
- /*
- * We believe that equivalence members aren't very different, so, to
- * estimate cost we consider just the first member.
- */
- em = (EquivalenceMember *) linitial(pathkey->pk_eclass->ec_members);
-
- if (em->em_datatype != InvalidOid)
- {
- /* do not lookup funcCost if the data type is the same */
- if (prev_datatype != em->em_datatype)
- {
- Oid sortop;
- QualCost cost;
-
- sortop = get_opfamily_member(pathkey->pk_opfamily,
- em->em_datatype, em->em_datatype,
- pathkey->pk_strategy);
-
- cost.startup = 0;
- cost.per_tuple = 0;
- add_function_cost(root, get_opcode(sortop), NULL, &cost);
-
- /*
- * add_function_cost returns the product of cpu_operator_cost
- * and procost, but we need just procost, co undo that.
- */
- funcCost = cost.per_tuple / cpu_operator_cost;
-
- prev_datatype = em->em_datatype;
- }
- }
-
- /* factor in the width of the values in this column */
- funcCost *= get_width_cost_multiplier(root, em->em_expr);
-
- /* now we have per-key cost, so add to the running total */
- totalFuncCost += funcCost;
-
- /* remember if we have found a fake Var in pathkeys */
- has_fake_var |= is_fake_var(em->em_expr);
- pathkeyExprs = lappend(pathkeyExprs, em->em_expr);
-
- /*
- * We need to calculate the number of comparisons for this column,
- * which requires knowing the group size. So we estimate the number of
- * groups by calling estimate_num_groups_incremental(), which
- * estimates the group size for "new" pathkeys.
- *
- * Note: estimate_num_groups_incremental does not handle fake Vars, so
- * use a default estimate otherwise.
- */
- if (!has_fake_var)
- nGroups = estimate_num_groups_incremental(root, pathkeyExprs,
- tuplesPerPrevGroup, NULL, NULL,
- &cache_varinfos,
- list_length(pathkeyExprs) - 1);
- else if (tuples > 4.0)
-
- /*
- * Use geometric mean as estimation if there are no stats.
- *
- * We don't use DEFAULT_NUM_DISTINCT here, because that's used for
- * a single column, but here we're dealing with multiple columns.
- */
- nGroups = ceil(2.0 + sqrt(tuples) * (i + 1) / list_length(pathkeys));
- else
- nGroups = tuples;
-
- /*
- * Presorted keys are not considered in the cost above, but we still
- * do have to compare them in the qsort comparator. So make sure to
- * factor in the cost in that case.
- */
- if (i >= nPresortedKeys)
- {
- if (heapSort)
- {
- /*
- * have to keep at least one group, and a multiple of group
- * size
- */
- correctedNGroups = ceil(output_tuples / tuplesPerPrevGroup);
- }
- else
- /* all groups in the input */
- correctedNGroups = nGroups;
-
- correctedNGroups = Max(1.0, ceil(correctedNGroups));
-
- per_tuple_cost += totalFuncCost * LOG2(correctedNGroups);
- }
-
- i++;
-
- /*
- * Uniform distributions with all groups being of the same size are
- * the best case, with nice smooth behavior. Real-world distributions
- * tend not to be uniform, though, and we don't have any reliable
- * easy-to-use information. As a basic defense against skewed
- * distributions, we use a 1.5 factor to make the expected group a bit
- * larger, but we need to be careful not to make the group larger than
- * in the preceding step.
- */
- tuplesPerPrevGroup = Min(tuplesPerPrevGroup,
- ceil(1.5 * tuplesPerPrevGroup / nGroups));
-
- /*
- * Once we get single-row group, it means tuples in the group are
- * unique and we can skip all remaining columns.
- */
- if (tuplesPerPrevGroup <= 1.0)
- break;
- }
-
- list_free(pathkeyExprs);
-
- /* per_tuple_cost is in cpu_operator_cost units */
- per_tuple_cost *= cpu_operator_cost;
-
- /*
- * Accordingly to "Introduction to algorithms", Thomas H. Cormen, Charles
- * E. Leiserson, Ronald L. Rivest, ISBN 0-07-013143-0, quicksort
- * estimation formula has additional term proportional to number of tuples
- * (see Chapter 8.2 and Theorem 4.1). That affects cases with a low number
- * of tuples, approximately less than 1e4. We could implement it as an
- * additional multiplier under the logarithm, but we use a bit more
- * complex formula which takes into account the number of unique tuples
- * and it's not clear how to combine the multiplier with the number of
- * groups. Estimate it as 10 cpu_operator_cost units.
- */
- per_tuple_cost += 10 * cpu_operator_cost;
-
- per_tuple_cost += comparison_cost;
-
- return tuples * per_tuple_cost;
-}
-
-/*
- * simple wrapper just to estimate best sort path
- */
-Cost
-cost_sort_estimate(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
- double tuples)
-{
- return compute_cpu_sort_cost(root, pathkeys, nPresortedKeys,
- 0, tuples, tuples, false);
-}
-
/*
* cost_tuplesort
* Determines and returns the cost of sorting a relation using tuplesort,
@@ -2151,7 +1830,7 @@ cost_sort_estimate(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* number of initial runs formed and M is the merge order used by tuplesort.c.
* Since the average initial run should be about sort_mem, we have
* disk traffic = 2 * relsize * ceil(logM(p / sort_mem))
- * and cpu cost (computed by compute_cpu_sort_cost()).
+ * cpu = comparison_cost * t * log2(t)
*
* If the sort is bounded (i.e., only the first k result tuples are needed)
* and k tuples can fit into sort_mem, we use a heap method that keeps only
@@ -2170,11 +1849,9 @@ cost_sort_estimate(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* 'comparison_cost' is the extra cost per comparison, if any
* 'sort_mem' is the number of kilobytes of work memory allowed for the sort
* 'limit_tuples' is the bound on the number of output tuples; -1 if no bound
- * 'startup_cost' is expected to be 0 at input. If there is "input cost" it should
- * be added by caller later
*/
static void
-cost_tuplesort(PlannerInfo *root, List *pathkeys, Cost *startup_cost, Cost *run_cost,
+cost_tuplesort(Cost *startup_cost, Cost *run_cost,
double tuples, int width,
Cost comparison_cost, int sort_mem,
double limit_tuples)
@@ -2191,6 +1868,9 @@ cost_tuplesort(PlannerInfo *root, List *pathkeys, Cost *startup_cost, Cost *run_
if (tuples < 2.0)
tuples = 2.0;
+ /* Include the default cost-per-comparison */
+ comparison_cost += 2.0 * cpu_operator_cost;
+
/* Do we have a useful LIMIT? */
if (limit_tuples > 0 && limit_tuples < tuples)
{
@@ -2214,10 +1894,12 @@ cost_tuplesort(PlannerInfo *root, List *pathkeys, Cost *startup_cost, Cost *run_
double log_runs;
double npageaccesses;
- /* CPU costs */
- *startup_cost = compute_cpu_sort_cost(root, pathkeys, 0,
- comparison_cost, tuples,
- tuples, false);
+ /*
+ * CPU costs
+ *
+ * Assume about N log2 N comparisons
+ */
+ *startup_cost = comparison_cost * tuples * LOG2(tuples);
/* Disk costs */
@@ -2233,17 +1915,18 @@ cost_tuplesort(PlannerInfo *root, List *pathkeys, Cost *startup_cost, Cost *run_
}
else if (tuples > 2 * output_tuples || input_bytes > sort_mem_bytes)
{
- /* We'll use a bounded heap-sort keeping just K tuples in memory. */
- *startup_cost = compute_cpu_sort_cost(root, pathkeys, 0,
- comparison_cost, tuples,
- output_tuples, true);
+ /*
+ * We'll use a bounded heap-sort keeping just K tuples in memory, for
+ * a total number of tuple comparisons of N log2 K; but the constant
+ * factor is a bit higher than for quicksort. Tweak it so that the
+ * cost curve is continuous at the crossover point.
+ */
+ *startup_cost = comparison_cost * tuples * LOG2(2.0 * output_tuples);
}
else
{
/* We'll use plain quicksort on all the input tuples */
- *startup_cost = compute_cpu_sort_cost(root, pathkeys, 0,
- comparison_cost, tuples,
- tuples, false);
+ *startup_cost = comparison_cost * tuples * LOG2(tuples);
}
/*
@@ -2276,8 +1959,8 @@ cost_incremental_sort(Path *path,
double input_tuples, int width, Cost comparison_cost, int sort_mem,
double limit_tuples)
{
- Cost startup_cost,
- run_cost,
+ Cost startup_cost = 0,
+ run_cost = 0,
input_run_cost = input_total_cost - input_startup_cost;
double group_tuples,
input_groups;
@@ -2362,7 +2045,7 @@ cost_incremental_sort(Path *path,
* pessimistic about incremental sort performance and increase its average
* group size by half.
*/
- cost_tuplesort(root, pathkeys, &group_startup_cost, &group_run_cost,
+ cost_tuplesort(&group_startup_cost, &group_run_cost,
1.5 * group_tuples, width, comparison_cost, sort_mem,
limit_tuples);
@@ -2370,7 +2053,7 @@ cost_incremental_sort(Path *path,
* Startup cost of incremental sort is the startup cost of its first group
* plus the cost of its input.
*/
- startup_cost = group_startup_cost
+ startup_cost += group_startup_cost
+ input_startup_cost + group_input_run_cost;
/*
@@ -2379,7 +2062,7 @@ cost_incremental_sort(Path *path,
* group, plus the total cost to process the remaining groups, plus the
* remaining cost of input.
*/
- run_cost = group_run_cost
+ run_cost += group_run_cost
+ (group_run_cost + group_startup_cost) * (input_groups - 1)
+ group_input_run_cost * (input_groups - 1);
@@ -2419,7 +2102,7 @@ cost_sort(Path *path, PlannerInfo *root,
Cost startup_cost;
Cost run_cost;
- cost_tuplesort(root, pathkeys, &startup_cost, &run_cost,
+ cost_tuplesort(&startup_cost, &run_cost,
tuples, width,
comparison_cost, sort_mem,
limit_tuples);
@@ -2517,7 +2200,7 @@ append_nonpartial_cost(List *subpaths, int numpaths, int parallel_workers)
* Determines and returns the cost of an Append node.
*/
void
-cost_append(AppendPath *apath, PlannerInfo *root)
+cost_append(AppendPath *apath)
{
ListCell *l;
@@ -2585,7 +2268,7 @@ cost_append(AppendPath *apath, PlannerInfo *root)
* any child.
*/
cost_sort(&sort_path,
- root,
+ NULL, /* doesn't currently need root */
pathkeys,
subpath->total_cost,
subpath->rows,
diff --git a/src/backend/optimizer/path/equivclass.c b/src/backend/optimizer/path/equivclass.c
index 799bdc91d0..f962ff82ad 100644
--- a/src/backend/optimizer/path/equivclass.c
+++ b/src/backend/optimizer/path/equivclass.c
@@ -681,18 +681,7 @@ get_eclass_for_sort_expr(PlannerInfo *root,
if (opcintype == cur_em->em_datatype &&
equal(expr, cur_em->em_expr))
- {
- /*
- * Match!
- *
- * Copy the sortref if it wasn't set yet. That may happen if
- * the ec was constructed from WHERE clause, i.e. it doesn't
- * have a target reference at all.
- */
- if (cur_ec->ec_sortref == 0 && sortref > 0)
- cur_ec->ec_sortref = sortref;
- return cur_ec;
- }
+ return cur_ec; /* Match! */
}
}
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index e2fdcd3163..a9943cd6e0 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -17,24 +17,17 @@
*/
#include "postgres.h"
-#include <float.h>
-
-#include "miscadmin.h"
#include "access/stratnum.h"
#include "catalog/pg_opfamily.h"
#include "nodes/makefuncs.h"
#include "nodes/nodeFuncs.h"
#include "nodes/plannodes.h"
-#include "optimizer/cost.h"
#include "optimizer/optimizer.h"
#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
#include "partitioning/partbounds.h"
#include "utils/lsyscache.h"
-#include "utils/selfuncs.h"
-/* Consider reordering of GROUP BY keys? */
-bool enable_group_by_reordering = true;
static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys);
static bool matches_boolean_partition_clause(RestrictInfo *rinfo,
@@ -363,540 +356,6 @@ pathkeys_contained_in(List *keys1, List *keys2)
return false;
}
-/*
- * group_keys_reorder_by_pathkeys
- * Reorder GROUP BY keys to match pathkeys of input path.
- *
- * Function returns new lists (pathkeys and clauses), original GROUP BY lists
- * stay untouched.
- *
- * Returns the number of GROUP BY keys with a matching pathkey.
- */
-int
-group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
- List **group_clauses)
-{
- List *new_group_pathkeys = NIL,
- *new_group_clauses = NIL;
- ListCell *lc;
- int n;
-
- if (pathkeys == NIL || *group_pathkeys == NIL)
- return 0;
-
- /*
- * Walk the pathkeys (determining ordering of the input path) and see if
- * there's a matching GROUP BY key. If we find one, we append it to the
- * list, and do the same for the clauses.
- *
- * Once we find the first pathkey without a matching GROUP BY key, the
- * rest of the pathkeys are useless and can't be used to evaluate the
- * grouping, so we abort the loop and ignore the remaining pathkeys.
- *
- * XXX Pathkeys are built in a way to allow simply comparing pointers.
- */
- foreach(lc, pathkeys)
- {
- PathKey *pathkey = (PathKey *) lfirst(lc);
- SortGroupClause *sgc;
-
- /* abort on first mismatch */
- if (!list_member_ptr(*group_pathkeys, pathkey))
- break;
-
- new_group_pathkeys = lappend(new_group_pathkeys, pathkey);
-
- sgc = get_sortgroupref_clause(pathkey->pk_eclass->ec_sortref,
- *group_clauses);
-
- new_group_clauses = lappend(new_group_clauses, sgc);
- }
-
- /* remember the number of pathkeys with a matching GROUP BY key */
- n = list_length(new_group_pathkeys);
-
- /* append the remaining group pathkeys (will be treated as not sorted) */
- *group_pathkeys = list_concat_unique_ptr(new_group_pathkeys,
- *group_pathkeys);
- *group_clauses = list_concat_unique_ptr(new_group_clauses,
- *group_clauses);
-
- return n;
-}
-
-/*
- * Used to generate all permutations of a pathkey list.
- */
-typedef struct PathkeyMutatorState
-{
- List *elemsList;
- ListCell **elemCells;
- void **elems;
- int *positions;
- int mutatorNColumns;
- int count;
-} PathkeyMutatorState;
-
-
-/*
- * PathkeyMutatorInit
- * Initialize state of the permutation generator.
- *
- * We want to generate permutations of elements in the "elems" list. We may want
- * to skip some number of elements at the beginning (when treating as presorted)
- * or at the end (we only permute a limited number of group keys).
- *
- * The list is decomposed into elements, and we also keep pointers to individual
- * cells. This allows us to build the permuted list quickly and cheaply, without
- * creating any copies.
- */
-static void
-PathkeyMutatorInit(PathkeyMutatorState *state, List *elems, int start, int end)
-{
- int i;
- int n = end - start;
- ListCell *lc;
-
- memset(state, 0, sizeof(*state));
-
- state->mutatorNColumns = n;
-
- state->elemsList = list_copy(elems);
-
- state->elems = palloc(sizeof(void *) * n);
- state->elemCells = palloc(sizeof(ListCell *) * n);
- state->positions = palloc(sizeof(int) * n);
-
- i = 0;
- for_each_cell(lc, state->elemsList, list_nth_cell(state->elemsList, start))
- {
- state->elemCells[i] = lc;
- state->elems[i] = lfirst(lc);
- state->positions[i] = i + 1;
- i++;
-
- if (i >= n)
- break;
- }
-}
-
-/* Swap two elements of an array. */
-static void
-PathkeyMutatorSwap(int *a, int i, int j)
-{
- int s = a[i];
-
- a[i] = a[j];
- a[j] = s;
-}
-
-/*
- * Generate the next permutation of elements.
- */
-static bool
-PathkeyMutatorNextSet(int *a, int n)
-{
- int j,
- k,
- l,
- r;
-
- j = n - 2;
-
- while (j >= 0 && a[j] >= a[j + 1])
- j--;
-
- if (j < 0)
- return false;
-
- k = n - 1;
-
- while (k >= 0 && a[j] >= a[k])
- k--;
-
- PathkeyMutatorSwap(a, j, k);
-
- l = j + 1;
- r = n - 1;
-
- while (l < r)
- PathkeyMutatorSwap(a, l++, r--);
-
- return true;
-}
-
-/*
- * PathkeyMutatorNext
- * Generate the next permutation of list of elements.
- *
- * Returns the next permutation (as a list of elements) or NIL if there are no
- * more permutations.
- */
-static List *
-PathkeyMutatorNext(PathkeyMutatorState *state)
-{
- int i;
-
- state->count++;
-
- /* first permutation is original list */
- if (state->count == 1)
- return state->elemsList;
-
- /* when there are no more permutations, return NIL */
- if (!PathkeyMutatorNextSet(state->positions, state->mutatorNColumns))
- {
- pfree(state->elems);
- pfree(state->elemCells);
- pfree(state->positions);
-
- list_free(state->elemsList);
-
- return NIL;
- }
-
- /* update the list cells to point to the right elements */
- for (i = 0; i < state->mutatorNColumns; i++)
- lfirst(state->elemCells[i]) =
- (void *) state->elems[state->positions[i] - 1];
-
- return state->elemsList;
-}
-
-/*
- * Cost of comparing pathkeys.
- */
-typedef struct PathkeySortCost
-{
- Cost cost;
- PathKey *pathkey;
-} PathkeySortCost;
-
-static int
-pathkey_sort_cost_comparator(const void *_a, const void *_b)
-{
- const PathkeySortCost *a = (PathkeySortCost *) _a;
- const PathkeySortCost *b = (PathkeySortCost *) _b;
-
- if (a->cost < b->cost)
- return -1;
- else if (a->cost == b->cost)
- return 0;
- return 1;
-}
-
-/*
- * get_cheapest_group_keys_order
- * Reorders the group pathkeys / clauses to minimize the comparison cost.
- *
- * Given the list of pathkeys in '*group_pathkeys', we try to arrange these
- * in an order that minimizes the sort costs that will be incurred by the
- * GROUP BY. The costs mainly depend on the cost of the sort comparator
- * function(s) and the number of distinct values in each column of the GROUP
- * BY clause (*group_clauses). Sorting on subsequent columns is only required
- * for tiebreak situations where two values sort equally.
- *
- * In case the input is partially sorted, only the remaining pathkeys are
- * considered. 'n_preordered' denotes how many of the leading *group_pathkeys
- * the input is presorted by.
- *
- * Returns true and sets *group_pathkeys and *group_clauses to the newly
- * ordered versions of the lists that were passed in via these parameters.
- * If no reordering was deemed necessary then we return false, in which case
- * the *group_pathkeys and *group_clauses lists are left untouched. The
- * original *group_pathkeys and *group_clauses parameter values are never
- * destructively modified in place.
- */
-static bool
-get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
- List **group_pathkeys, List **group_clauses,
- int n_preordered)
-{
- List *new_group_pathkeys = NIL,
- *new_group_clauses = NIL,
- *var_group_pathkeys;
-
- ListCell *cell;
- PathkeyMutatorState mstate;
- double cheapest_sort_cost = DBL_MAX;
-
- int nFreeKeys;
- int nToPermute;
-
- /* If there are less than 2 unsorted pathkeys, we're done. */
- if (list_length(*group_pathkeys) - n_preordered < 2)
- return false;
-
- /*
- * We could exhaustively cost all possible orderings of the pathkeys, but
- * for a large number of pathkeys it might be prohibitively expensive. So
- * we try to apply simple cheap heuristics first - we sort the pathkeys by
- * sort cost (as if the pathkey was sorted independently) and then check
- * only the four cheapest pathkeys. The remaining pathkeys are kept
- * ordered by cost.
- *
- * XXX This is a very simple heuristics, but likely to work fine for most
- * cases (because the number of GROUP BY clauses tends to be lower than
- * 4). But it ignores how the number of distinct values in each pathkey
- * affects the following steps. It might be better to use "more expensive"
- * pathkey first if it has many distinct values, because it then limits
- * the number of comparisons for the remaining pathkeys. But evaluating
- * that is likely quite the expensive.
- */
- nFreeKeys = list_length(*group_pathkeys) - n_preordered;
- nToPermute = 4;
- if (nFreeKeys > nToPermute)
- {
- PathkeySortCost *costs = palloc(sizeof(PathkeySortCost) * nFreeKeys);
- PathkeySortCost *cost = costs;
-
- /*
- * Estimate cost for sorting individual pathkeys skipping the
- * pre-ordered pathkeys.
- */
- for_each_from(cell, *group_pathkeys, n_preordered)
- {
- PathKey *pathkey = (PathKey *) lfirst(cell);
- List *to_cost = list_make1(pathkey);
-
- cost->pathkey = pathkey;
- cost->cost = cost_sort_estimate(root, to_cost, 0, nrows);
- cost++;
-
- list_free(to_cost);
- }
-
- /* sort the pathkeys by sort cost in ascending order */
- qsort(costs, nFreeKeys, sizeof(*costs), pathkey_sort_cost_comparator);
-
- /*
- * Rebuild the list of pathkeys - first the preordered ones, then the
- * rest ordered by cost.
- */
- new_group_pathkeys = list_copy_head(*group_pathkeys, n_preordered);
-
- for (int i = 0; i < nFreeKeys; i++)
- new_group_pathkeys = lappend(new_group_pathkeys, costs[i].pathkey);
-
- pfree(costs);
- }
- else
- {
- /* Copy the list, so that we can free the new list by list_free. */
- new_group_pathkeys = list_copy(*group_pathkeys);
- nToPermute = nFreeKeys;
- }
-
- Assert(list_length(new_group_pathkeys) == list_length(*group_pathkeys));
-
- /*
- * Generate pathkey lists with permutations of the first nToPermute
- * pathkeys.
- *
- * XXX We simply calculate sort cost for each individual pathkey list, but
- * there's room for two dynamic programming optimizations here. Firstly,
- * we may pass the current "best" cost to cost_sort_estimate so that it
- * can "abort" if the estimated pathkeys list exceeds it. Secondly, it
- * could pass the return information about the position when it exceeded
- * the cost, and we could skip all permutations with the same prefix.
- *
- * Imagine we've already found ordering with cost C1, and we're evaluating
- * another ordering - cost_sort_estimate() calculates cost by adding the
- * pathkeys one by one (more or less), and the cost only grows. If at any
- * point it exceeds C1, it can't possibly be "better" so we can discard
- * it. But we also know that we can discard all ordering with the same
- * prefix, because if we're estimating (a,b,c,d) and we exceed C1 at (a,b)
- * then the same thing will happen for any ordering with this prefix.
- */
- PathkeyMutatorInit(&mstate, new_group_pathkeys, n_preordered, n_preordered + nToPermute);
-
- while ((var_group_pathkeys = PathkeyMutatorNext(&mstate)) != NIL)
- {
- Cost cost;
-
- cost = cost_sort_estimate(root, var_group_pathkeys, n_preordered, nrows);
-
- if (cost < cheapest_sort_cost)
- {
- list_free(new_group_pathkeys);
- new_group_pathkeys = list_copy(var_group_pathkeys);
- cheapest_sort_cost = cost;
- }
- }
-
- /* Reorder the group clauses according to the reordered pathkeys. */
- foreach(cell, new_group_pathkeys)
- {
- PathKey *pathkey = (PathKey *) lfirst(cell);
-
- new_group_clauses = lappend(new_group_clauses,
- get_sortgroupref_clause(pathkey->pk_eclass->ec_sortref,
- *group_clauses));
- }
-
- /* Just append the rest GROUP BY clauses */
- new_group_clauses = list_concat_unique_ptr(new_group_clauses,
- *group_clauses);
-
- *group_pathkeys = new_group_pathkeys;
- *group_clauses = new_group_clauses;
-
- return true;
-}
-
-/*
- * get_useful_group_keys_orderings
- * Determine which orderings of GROUP BY keys are potentially interesting.
- *
- * Returns list of PathKeyInfo items, each representing an interesting ordering
- * of GROUP BY keys. Each item stores pathkeys and clauses in matching order.
- *
- * The function considers (and keeps) multiple group by orderings:
- *
- * - the original ordering, as specified by the GROUP BY clause
- *
- * - GROUP BY keys reordered to minimize the sort cost
- *
- * - GROUP BY keys reordered to match path ordering (as much as possible), with
- * the tail reordered to minimize the sort cost
- *
- * - GROUP BY keys to match target ORDER BY clause (as much as possible), with
- * the tail reordered to minimize the sort cost
- *
- * There are other potentially interesting orderings (e.g. it might be best to
- * match the first ORDER BY key, order the remaining keys differently and then
- * rely on the incremental sort to fix this), but we ignore those for now. To
- * make this work we'd have to pretty much generate all possible permutations.
- */
-List *
-get_useful_group_keys_orderings(PlannerInfo *root, double nrows,
- List *path_pathkeys,
- List *group_pathkeys, List *group_clauses,
- List *aggregate_pathkeys)
-{
- Query *parse = root->parse;
- List *infos = NIL;
- PathKeyInfo *info;
- int n_preordered = 0;
-
- List *pathkeys = group_pathkeys;
- List *clauses = group_clauses;
-
- /* always return at least the original pathkeys/clauses */
- info = makeNode(PathKeyInfo);
- if (aggregate_pathkeys != NIL)
- info->pathkeys = list_concat_copy(pathkeys, aggregate_pathkeys);
- else
- info->pathkeys = pathkeys;
- info->clauses = clauses;
-
- infos = lappend(infos, info);
-
- /*
- * Should we try generating alternative orderings of the group keys? If
- * not, we produce only the order specified in the query, i.e. the
- * optimization is effectively disabled.
- */
- if (!enable_group_by_reordering)
- return infos;
-
- /* for grouping sets we can't do any reordering */
- if (parse->groupingSets)
- return infos;
-
- /*
- * Try reordering pathkeys to minimize the sort cost, ignoring both the
- * target ordering (ORDER BY) and ordering of the input path.
- */
- if (get_cheapest_group_keys_order(root, nrows, &pathkeys, &clauses,
- n_preordered))
- {
- info = makeNode(PathKeyInfo);
- if (aggregate_pathkeys != NIL)
- info->pathkeys = list_concat_copy(pathkeys, aggregate_pathkeys);
- else
- info->pathkeys = pathkeys;
- info->clauses = clauses;
-
- infos = lappend(infos, info);
- }
-
- /*
- * If the path is sorted in some way, try reordering the group keys to
- * match as much of the ordering as possible - we get this sort for free
- * (mostly).
- *
- * We must not do this when there are no grouping sets, because those use
- * more complex logic to decide the ordering.
- *
- * XXX Isn't this somewhat redundant with presorted_keys? Actually, it's
- * more a complement, because it allows benefiting from incremental sort
- * as much as possible.
- *
- * XXX This does nothing if (n_preordered == 0). We shouldn't create the
- * info in this case.
- */
- if (path_pathkeys)
- {
- n_preordered = group_keys_reorder_by_pathkeys(path_pathkeys,
- &pathkeys,
- &clauses);
-
- /* reorder the tail to minimize sort cost */
- get_cheapest_group_keys_order(root, nrows, &pathkeys, &clauses,
- n_preordered);
-
- /*
- * reorder the tail to minimize sort cost
- *
- * XXX Ignore the return value - there may be nothing to reorder, in
- * which case get_cheapest_group_keys_order returns false. But we
- * still want to keep the keys reordered to path_pathkeys.
- */
- info = makeNode(PathKeyInfo);
- if (aggregate_pathkeys != NIL)
- info->pathkeys = list_concat_copy(pathkeys, aggregate_pathkeys);
- else
- info->pathkeys = pathkeys;
- info->clauses = clauses;
-
- infos = lappend(infos, info);
- }
-
- /*
- * Try reordering pathkeys to minimize the sort cost (this time consider
- * the ORDER BY clause, but only if set debug_group_by_match_order_by).
- */
- if (root->sort_pathkeys)
- {
- n_preordered = group_keys_reorder_by_pathkeys(root->sort_pathkeys,
- &pathkeys,
- &clauses);
-
- /*
- * reorder the tail to minimize sort cost
- *
- * XXX Ignore the return value - there may be nothing to reorder, in
- * which case get_cheapest_group_keys_order returns false. But we
- * still want to keep the keys reordered to sort_pathkeys.
- */
- get_cheapest_group_keys_order(root, nrows, &pathkeys, &clauses,
- n_preordered);
-
- /* keep the group keys reordered to match ordering of input path */
- info = makeNode(PathKeyInfo);
- if (aggregate_pathkeys != NIL)
- info->pathkeys = list_concat_copy(pathkeys, aggregate_pathkeys);
- else
- info->pathkeys = pathkeys;
- info->clauses = clauses;
-
- infos = lappend(infos, info);
- }
-
- return infos;
-}
-
/*
* pathkeys_count_contained_in
* Same as pathkeys_contained_in, but also sets length of longest
@@ -2456,54 +1915,6 @@ pathkeys_useful_for_ordering(PlannerInfo *root, List *pathkeys)
return n_common_pathkeys;
}
-/*
- * pathkeys_useful_for_grouping
- * Count the number of pathkeys that are useful for grouping (instead of
- * explicit sort)
- *
- * Group pathkeys could be reordered to benefit from the ordering. The
- * ordering may not be "complete" and may require incremental sort, but that's
- * fine. So we simply count prefix pathkeys with a matching group key, and
- * stop once we find the first pathkey without a match.
- *
- * So e.g. with pathkeys (a,b,c) and group keys (a,b,e) this determines (a,b)
- * pathkeys are useful for grouping, and we might do incremental sort to get
- * path ordered by (a,b,e).
- *
- * This logic is necessary to retain paths with ordering not matching grouping
- * keys directly, without the reordering.
- *
- * Returns the length of pathkey prefix with matching group keys.
- */
-static int
-pathkeys_useful_for_grouping(PlannerInfo *root, List *pathkeys)
-{
- ListCell *key;
- int n = 0;
-
- /* no special ordering requested for grouping */
- if (root->group_pathkeys == NIL)
- return 0;
-
- /* unordered path */
- if (pathkeys == NIL)
- return 0;
-
- /* walk the pathkeys and search for matching group key */
- foreach(key, pathkeys)
- {
- PathKey *pathkey = (PathKey *) lfirst(key);
-
- /* no matching group key, we're done */
- if (!list_member_ptr(root->group_pathkeys, pathkey))
- break;
-
- n++;
- }
-
- return n;
-}
-
/*
* truncate_useless_pathkeys
* Shorten the given pathkey list to just the useful pathkeys.
@@ -2518,9 +1929,6 @@ truncate_useless_pathkeys(PlannerInfo *root,
nuseful = pathkeys_useful_for_merging(root, rel, pathkeys);
nuseful2 = pathkeys_useful_for_ordering(root, pathkeys);
- if (nuseful2 > nuseful)
- nuseful = nuseful2;
- nuseful2 = pathkeys_useful_for_grouping(root, pathkeys);
if (nuseful2 > nuseful)
nuseful = nuseful2;
@@ -2556,8 +1964,6 @@ has_useful_pathkeys(PlannerInfo *root, RelOptInfo *rel)
{
if (rel->joininfo != NIL || rel->has_eclass_joins)
return true; /* might be able to use pathkeys for merging */
- if (root->group_pathkeys != NIL)
- return true; /* might be able to use pathkeys for grouping */
if (root->query_pathkeys != NIL)
return true; /* might be able to use them for ordering */
return false; /* definitely useless */
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 8014d1fd25..19f9cd66cc 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -6433,148 +6433,30 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
if (can_sort)
{
- List *group_pathkeys;
- List *orderAggPathkeys;
- int numAggPathkeys;
-
- numAggPathkeys = list_length(root->group_pathkeys) -
- root->num_groupby_pathkeys;
-
- if (numAggPathkeys > 0)
- {
- group_pathkeys = list_copy_head(root->group_pathkeys,
- root->num_groupby_pathkeys);
- orderAggPathkeys = list_copy_tail(root->group_pathkeys,
- root->num_groupby_pathkeys);
- }
- else
- {
- group_pathkeys = root->group_pathkeys;
- orderAggPathkeys = NIL;
- }
-
/*
* Use any available suitably-sorted path as input, and also consider
* sorting the cheapest-total path.
*/
foreach(lc, input_rel->pathlist)
{
- ListCell *lc2;
Path *path = (Path *) lfirst(lc);
Path *path_original = path;
+ bool is_sorted;
+ int presorted_keys;
- List *pathkey_orderings = NIL;
-
- List *group_clauses = parse->groupClause;
-
- /* generate alternative group orderings that might be useful */
- pathkey_orderings = get_useful_group_keys_orderings(root,
- path->rows,
- path->pathkeys,
- group_pathkeys,
- group_clauses,
- orderAggPathkeys);
-
- Assert(pathkey_orderings != NIL);
+ is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
+ path->pathkeys,
+ &presorted_keys);
- /* process all potentially interesting grouping reorderings */
- foreach(lc2, pathkey_orderings)
+ if (path == cheapest_path || is_sorted)
{
- bool is_sorted;
- int presorted_keys = 0;
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
-
- /* restore the path (we replace it in the loop) */
- path = path_original;
-
- is_sorted = pathkeys_count_contained_in(info->pathkeys,
- path->pathkeys,
- &presorted_keys);
-
- if (path == cheapest_path || is_sorted)
- {
- /* Sort the cheapest-total path if it isn't already sorted */
- if (!is_sorted)
- path = (Path *) create_sort_path(root,
- grouped_rel,
- path,
- info->pathkeys,
- -1.0);
-
- /* Now decide what to stick atop it */
- if (parse->groupingSets)
- {
- consider_groupingsets_paths(root, grouped_rel,
- path, true, can_hash,
- gd, agg_costs, dNumGroups);
- }
- else if (parse->hasAggs)
- {
- /*
- * We have aggregation, possibly with plain GROUP BY.
- * Make an AggPath.
- */
- add_path(grouped_rel, (Path *)
- create_agg_path(root,
- grouped_rel,
- path,
- grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_SIMPLE,
- info->clauses,
- havingQual,
- agg_costs,
- dNumGroups));
- }
- else if (group_clauses)
- {
- /*
- * We have GROUP BY without aggregation or grouping
- * sets. Make a GroupPath.
- */
- add_path(grouped_rel, (Path *)
- create_group_path(root,
- grouped_rel,
- path,
- info->clauses,
- havingQual,
- dNumGroups));
- }
- else
- {
- /* Other cases should have been handled above */
- Assert(false);
- }
- }
-
- /*
- * Now we may consider incremental sort on this path, but only
- * when the path is not already sorted and when incremental
- * sort is enabled.
- */
- if (is_sorted || !enable_incremental_sort)
- continue;
-
- /* Restore the input path (we might have added Sort on top). */
- path = path_original;
-
- /* no shared prefix, no point in building incremental sort */
- if (presorted_keys == 0)
- continue;
-
- /*
- * We should have already excluded pathkeys of length 1
- * because then presorted_keys > 0 would imply is_sorted was
- * true.
- */
- Assert(list_length(root->group_pathkeys) != 1);
-
- path = (Path *) create_incremental_sort_path(root,
- grouped_rel,
- path,
- info->pathkeys,
- presorted_keys,
- -1.0);
+ /* Sort the cheapest-total path if it isn't already sorted */
+ if (!is_sorted)
+ path = (Path *) create_sort_path(root,
+ grouped_rel,
+ path,
+ root->group_pathkeys,
+ -1.0);
/* Now decide what to stick atop it */
if (parse->groupingSets)
@@ -6594,9 +6476,9 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
grouped_rel,
path,
grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
AGGSPLIT_SIMPLE,
- info->clauses,
+ parse->groupClause,
havingQual,
agg_costs,
dNumGroups));
@@ -6611,7 +6493,7 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
create_group_path(root,
grouped_rel,
path,
- info->clauses,
+ parse->groupClause,
havingQual,
dNumGroups));
}
@@ -6621,6 +6503,79 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
Assert(false);
}
}
+
+ /*
+ * Now we may consider incremental sort on this path, but only
+ * when the path is not already sorted and when incremental sort
+ * is enabled.
+ */
+ if (is_sorted || !enable_incremental_sort)
+ continue;
+
+ /* Restore the input path (we might have added Sort on top). */
+ path = path_original;
+
+ /* no shared prefix, no point in building incremental sort */
+ if (presorted_keys == 0)
+ continue;
+
+ /*
+ * We should have already excluded pathkeys of length 1 because
+ * then presorted_keys > 0 would imply is_sorted was true.
+ */
+ Assert(list_length(root->group_pathkeys) != 1);
+
+ path = (Path *) create_incremental_sort_path(root,
+ grouped_rel,
+ path,
+ root->group_pathkeys,
+ presorted_keys,
+ -1.0);
+
+ /* Now decide what to stick atop it */
+ if (parse->groupingSets)
+ {
+ consider_groupingsets_paths(root, grouped_rel,
+ path, true, can_hash,
+ gd, agg_costs, dNumGroups);
+ }
+ else if (parse->hasAggs)
+ {
+ /*
+ * We have aggregation, possibly with plain GROUP BY. Make an
+ * AggPath.
+ */
+ add_path(grouped_rel, (Path *)
+ create_agg_path(root,
+ grouped_rel,
+ path,
+ grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_SIMPLE,
+ parse->groupClause,
+ havingQual,
+ agg_costs,
+ dNumGroups));
+ }
+ else if (parse->groupClause)
+ {
+ /*
+ * We have GROUP BY without aggregation or grouping sets. Make
+ * a GroupPath.
+ */
+ add_path(grouped_rel, (Path *)
+ create_group_path(root,
+ grouped_rel,
+ path,
+ parse->groupClause,
+ havingQual,
+ dNumGroups));
+ }
+ else
+ {
+ /* Other cases should have been handled above */
+ Assert(false);
+ }
}
/*
@@ -6631,128 +6586,100 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
{
foreach(lc, partially_grouped_rel->pathlist)
{
- ListCell *lc2;
Path *path = (Path *) lfirst(lc);
Path *path_original = path;
- List *pathkey_orderings = NIL;
- List *group_clauses = parse->groupClause;
-
- /* generate alternative group orderings that might be useful */
- pathkey_orderings = get_useful_group_keys_orderings(root,
- path->rows,
- path->pathkeys,
- group_pathkeys,
- group_clauses,
- orderAggPathkeys);
+ bool is_sorted;
+ int presorted_keys;
- Assert(pathkey_orderings != NIL);
+ is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
+ path->pathkeys,
+ &presorted_keys);
- /* process all potentially interesting grouping reorderings */
- foreach(lc2, pathkey_orderings)
+ /*
+ * Insert a Sort node, if required. But there's no point in
+ * sorting anything but the cheapest path.
+ */
+ if (!is_sorted)
{
- bool is_sorted;
- int presorted_keys = 0;
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
-
- /* restore the path (we replace it in the loop) */
- path = path_original;
-
- is_sorted = pathkeys_count_contained_in(info->pathkeys,
- path->pathkeys,
- &presorted_keys);
-
- /*
- * Insert a Sort node, if required. But there's no point
- * in sorting anything but the cheapest path.
- */
- if (!is_sorted)
- {
- if (path != partially_grouped_rel->cheapest_total_path)
- continue;
- path = (Path *) create_sort_path(root,
- grouped_rel,
- path,
- info->pathkeys,
- -1.0);
- }
+ if (path != partially_grouped_rel->cheapest_total_path)
+ continue;
+ path = (Path *) create_sort_path(root,
+ grouped_rel,
+ path,
+ root->group_pathkeys,
+ -1.0);
+ }
- if (parse->hasAggs)
- add_path(grouped_rel, (Path *)
- create_agg_path(root,
- grouped_rel,
- path,
- grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_FINAL_DESERIAL,
- info->clauses,
- havingQual,
- agg_final_costs,
- dNumGroups));
- else
- add_path(grouped_rel, (Path *)
- create_group_path(root,
- grouped_rel,
- path,
- info->clauses,
- havingQual,
- dNumGroups));
+ if (parse->hasAggs)
+ add_path(grouped_rel, (Path *)
+ create_agg_path(root,
+ grouped_rel,
+ path,
+ grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_FINAL_DESERIAL,
+ parse->groupClause,
+ havingQual,
+ agg_final_costs,
+ dNumGroups));
+ else
+ add_path(grouped_rel, (Path *)
+ create_group_path(root,
+ grouped_rel,
+ path,
+ parse->groupClause,
+ havingQual,
+ dNumGroups));
- /*
- * Now we may consider incremental sort on this path, but
- * only when the path is not already sorted and when
- * incremental sort is enabled.
- */
- if (is_sorted || !enable_incremental_sort)
- continue;
+ /*
+ * Now we may consider incremental sort on this path, but only
+ * when the path is not already sorted and when incremental
+ * sort is enabled.
+ */
+ if (is_sorted || !enable_incremental_sort)
+ continue;
- /*
- * Restore the input path (we might have added Sort on
- * top).
- */
- path = path_original;
+ /* Restore the input path (we might have added Sort on top). */
+ path = path_original;
- /*
- * no shared prefix, not point in building incremental
- * sort
- */
- if (presorted_keys == 0)
- continue;
+ /* no shared prefix, not point in building incremental sort */
+ if (presorted_keys == 0)
+ continue;
- /*
- * We should have already excluded pathkeys of length 1
- * because then presorted_keys > 0 would imply is_sorted
- * was true.
- */
- Assert(list_length(root->group_pathkeys) != 1);
+ /*
+ * We should have already excluded pathkeys of length 1
+ * because then presorted_keys > 0 would imply is_sorted was
+ * true.
+ */
+ Assert(list_length(root->group_pathkeys) != 1);
- path = (Path *) create_incremental_sort_path(root,
- grouped_rel,
- path,
- info->pathkeys,
- presorted_keys,
- -1.0);
+ path = (Path *) create_incremental_sort_path(root,
+ grouped_rel,
+ path,
+ root->group_pathkeys,
+ presorted_keys,
+ -1.0);
- if (parse->hasAggs)
- add_path(grouped_rel, (Path *)
- create_agg_path(root,
- grouped_rel,
- path,
- grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_FINAL_DESERIAL,
- info->clauses,
- havingQual,
- agg_final_costs,
- dNumGroups));
- else
- add_path(grouped_rel, (Path *)
- create_group_path(root,
- grouped_rel,
- path,
- info->clauses,
- havingQual,
- dNumGroups));
- }
+ if (parse->hasAggs)
+ add_path(grouped_rel, (Path *)
+ create_agg_path(root,
+ grouped_rel,
+ path,
+ grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_FINAL_DESERIAL,
+ parse->groupClause,
+ havingQual,
+ agg_final_costs,
+ dNumGroups));
+ else
+ add_path(grouped_rel, (Path *)
+ create_group_path(root,
+ grouped_rel,
+ path,
+ parse->groupClause,
+ havingQual,
+ dNumGroups));
}
}
}
@@ -6946,26 +6873,6 @@ create_partial_grouping_paths(PlannerInfo *root,
if (can_sort && cheapest_total_path != NULL)
{
- List *group_pathkeys;
- List *orderAggPathkeys;
- int numAggPathkeys;
-
- numAggPathkeys = list_length(root->group_pathkeys) -
- root->num_groupby_pathkeys;
-
- if (numAggPathkeys > 0)
- {
- group_pathkeys = list_copy_head(root->group_pathkeys,
- root->num_groupby_pathkeys);
- orderAggPathkeys = list_copy_tail(root->group_pathkeys,
- root->num_groupby_pathkeys);
- }
- else
- {
- group_pathkeys = root->group_pathkeys;
- orderAggPathkeys = NIL;
- }
-
/* This should have been checked previously */
Assert(parse->hasAggs || parse->groupClause);
@@ -6975,69 +6882,41 @@ create_partial_grouping_paths(PlannerInfo *root,
*/
foreach(lc, input_rel->pathlist)
{
- ListCell *lc2;
Path *path = (Path *) lfirst(lc);
- Path *path_save = path;
- List *pathkey_orderings = NIL;
- List *group_clauses = parse->groupClause;
-
- /* generate alternative group orderings that might be useful */
- pathkey_orderings = get_useful_group_keys_orderings(root,
- path->rows,
- path->pathkeys,
- group_pathkeys,
- group_clauses,
- orderAggPathkeys);
-
- Assert(pathkey_orderings != NIL);
-
- /* process all potentially interesting grouping reorderings */
- foreach(lc2, pathkey_orderings)
- {
- bool is_sorted;
- int presorted_keys = 0;
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
-
- /* restore the path (we replace it in the loop) */
- path = path_save;
-
- is_sorted = pathkeys_count_contained_in(info->pathkeys,
- path->pathkeys,
- &presorted_keys);
+ bool is_sorted;
- if (path == cheapest_total_path || is_sorted)
- {
- /* Sort the cheapest partial path, if it isn't already */
- if (!is_sorted)
- {
- path = (Path *) create_sort_path(root,
- partially_grouped_rel,
- path,
- info->pathkeys,
- -1.0);
- }
+ is_sorted = pathkeys_contained_in(root->group_pathkeys,
+ path->pathkeys);
+ if (path == cheapest_total_path || is_sorted)
+ {
+ /* Sort the cheapest partial path, if it isn't already */
+ if (!is_sorted)
+ path = (Path *) create_sort_path(root,
+ partially_grouped_rel,
+ path,
+ root->group_pathkeys,
+ -1.0);
- if (parse->hasAggs)
- add_path(partially_grouped_rel, (Path *)
- create_agg_path(root,
- partially_grouped_rel,
- path,
- partially_grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_INITIAL_SERIAL,
- info->clauses,
- NIL,
- agg_partial_costs,
- dNumPartialGroups));
- else
- add_path(partially_grouped_rel, (Path *)
- create_group_path(root,
- partially_grouped_rel,
- path,
- info->clauses,
- NIL,
- dNumPartialGroups));
- }
+ if (parse->hasAggs)
+ add_path(partially_grouped_rel, (Path *)
+ create_agg_path(root,
+ partially_grouped_rel,
+ path,
+ partially_grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_INITIAL_SERIAL,
+ parse->groupClause,
+ NIL,
+ agg_partial_costs,
+ dNumPartialGroups));
+ else
+ add_path(partially_grouped_rel, (Path *)
+ create_group_path(root,
+ partially_grouped_rel,
+ path,
+ parse->groupClause,
+ NIL,
+ dNumPartialGroups));
}
}
@@ -7047,8 +6926,6 @@ create_partial_grouping_paths(PlannerInfo *root,
* We can also skip the entire loop when we only have a single-item
* group_pathkeys because then we can't possibly have a presorted
* prefix of the list without having the list be fully sorted.
- *
- * XXX Shouldn't this also consider the group-key-reordering?
*/
if (enable_incremental_sort && list_length(root->group_pathkeys) > 1)
{
@@ -7103,122 +6980,27 @@ create_partial_grouping_paths(PlannerInfo *root,
if (can_sort && cheapest_partial_path != NULL)
{
- List *group_pathkeys;
- List *orderAggPathkeys;
- int numAggPathkeys;
-
- numAggPathkeys = list_length(root->group_pathkeys) -
- root->num_groupby_pathkeys;
-
- if (numAggPathkeys > 0)
- {
- group_pathkeys = list_copy_head(root->group_pathkeys,
- root->num_groupby_pathkeys);
- orderAggPathkeys = list_copy_tail(root->group_pathkeys,
- root->num_groupby_pathkeys);
- }
- else
- {
- group_pathkeys = root->group_pathkeys;
- orderAggPathkeys = NIL;
- }
-
/* Similar to above logic, but for partial paths. */
foreach(lc, input_rel->partial_pathlist)
{
- ListCell *lc2;
Path *path = (Path *) lfirst(lc);
Path *path_original = path;
- List *pathkey_orderings = NIL;
- List *group_clauses = parse->groupClause;
-
- /* generate alternative group orderings that might be useful */
- pathkey_orderings = get_useful_group_keys_orderings(root,
- path->rows,
- path->pathkeys,
- group_pathkeys,
- group_clauses,
- orderAggPathkeys);
+ bool is_sorted;
+ int presorted_keys;
- Assert(pathkey_orderings != NIL);
+ is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
+ path->pathkeys,
+ &presorted_keys);
- /* process all potentially interesting grouping reorderings */
- foreach(lc2, pathkey_orderings)
+ if (path == cheapest_partial_path || is_sorted)
{
- bool is_sorted;
- int presorted_keys = 0;
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
-
- /* restore the path (we replace it in the loop) */
- path = path_original;
-
- is_sorted = pathkeys_count_contained_in(info->pathkeys,
- path->pathkeys,
- &presorted_keys);
-
- if (path == cheapest_partial_path || is_sorted)
- {
-
- /* Sort the cheapest partial path, if it isn't already */
- if (!is_sorted)
- {
- path = (Path *) create_sort_path(root,
- partially_grouped_rel,
- path,
- info->pathkeys,
- -1.0);
- }
-
- if (parse->hasAggs)
- add_partial_path(partially_grouped_rel, (Path *)
- create_agg_path(root,
- partially_grouped_rel,
- path,
- partially_grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_INITIAL_SERIAL,
- info->clauses,
- NIL,
- agg_partial_costs,
- dNumPartialPartialGroups));
- else
- add_partial_path(partially_grouped_rel, (Path *)
- create_group_path(root,
- partially_grouped_rel,
- path,
- info->clauses,
- NIL,
- dNumPartialPartialGroups));
- }
-
- /*
- * Now we may consider incremental sort on this path, but only
- * when the path is not already sorted and when incremental
- * sort is enabled.
- */
- if (is_sorted || !enable_incremental_sort)
- continue;
-
- /* Restore the input path (we might have added Sort on top). */
- path = path_original;
-
- /* no shared prefix, not point in building incremental sort */
- if (presorted_keys == 0)
- continue;
-
- /*
- * We should have already excluded pathkeys of length 1
- * because then presorted_keys > 0 would imply is_sorted was
- * true.
- */
- Assert(list_length(root->group_pathkeys) != 1);
-
- path = (Path *) create_incremental_sort_path(root,
- partially_grouped_rel,
- path,
- info->pathkeys,
- presorted_keys,
- -1.0);
+ /* Sort the cheapest partial path, if it isn't already */
+ if (!is_sorted)
+ path = (Path *) create_sort_path(root,
+ partially_grouped_rel,
+ path,
+ root->group_pathkeys,
+ -1.0);
if (parse->hasAggs)
add_partial_path(partially_grouped_rel, (Path *)
@@ -7226,9 +7008,9 @@ create_partial_grouping_paths(PlannerInfo *root,
partially_grouped_rel,
path,
partially_grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
AGGSPLIT_INITIAL_SERIAL,
- info->clauses,
+ parse->groupClause,
NIL,
agg_partial_costs,
dNumPartialPartialGroups));
@@ -7237,10 +7019,59 @@ create_partial_grouping_paths(PlannerInfo *root,
create_group_path(root,
partially_grouped_rel,
path,
- info->clauses,
+ parse->groupClause,
NIL,
dNumPartialPartialGroups));
}
+
+ /*
+ * Now we may consider incremental sort on this path, but only
+ * when the path is not already sorted and when incremental sort
+ * is enabled.
+ */
+ if (is_sorted || !enable_incremental_sort)
+ continue;
+
+ /* Restore the input path (we might have added Sort on top). */
+ path = path_original;
+
+ /* no shared prefix, not point in building incremental sort */
+ if (presorted_keys == 0)
+ continue;
+
+ /*
+ * We should have already excluded pathkeys of length 1 because
+ * then presorted_keys > 0 would imply is_sorted was true.
+ */
+ Assert(list_length(root->group_pathkeys) != 1);
+
+ path = (Path *) create_incremental_sort_path(root,
+ partially_grouped_rel,
+ path,
+ root->group_pathkeys,
+ presorted_keys,
+ -1.0);
+
+ if (parse->hasAggs)
+ add_partial_path(partially_grouped_rel, (Path *)
+ create_agg_path(root,
+ partially_grouped_rel,
+ path,
+ partially_grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_INITIAL_SERIAL,
+ parse->groupClause,
+ NIL,
+ agg_partial_costs,
+ dNumPartialPartialGroups));
+ else
+ add_partial_path(partially_grouped_rel, (Path *)
+ create_group_path(root,
+ partially_grouped_rel,
+ path,
+ parse->groupClause,
+ NIL,
+ dNumPartialPartialGroups));
}
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index e10561d843..70f61ae7b1 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -1346,12 +1346,12 @@ create_append_path(PlannerInfo *root,
pathnode->path.total_cost = child->total_cost;
}
else
- cost_append(pathnode, root);
+ cost_append(pathnode);
/* Must do this last, else cost_append complains */
pathnode->path.pathkeys = child->pathkeys;
}
else
- cost_append(pathnode, root);
+ cost_append(pathnode);
/* If the caller provided a row estimate, override the computed value. */
if (rows >= 0)
diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c
index 1808388397..234fb66580 100644
--- a/src/backend/utils/adt/selfuncs.c
+++ b/src/backend/utils/adt/selfuncs.c
@@ -3369,28 +3369,11 @@ double
estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows,
List **pgset, EstimationInfo *estinfo)
{
- return estimate_num_groups_incremental(root, groupExprs,
- input_rows, pgset, estinfo,
- NULL, 0);
-}
-
-/*
- * estimate_num_groups_incremental
- * An estimate_num_groups variant, optimized for cases that are adding the
- * expressions incrementally (e.g. one by one).
- */
-double
-estimate_num_groups_incremental(PlannerInfo *root, List *groupExprs,
- double input_rows,
- List **pgset, EstimationInfo *estinfo,
- List **cache_varinfos, int prevNExprs)
-{
- List *varinfos = (cache_varinfos) ? *cache_varinfos : NIL;
+ List *varinfos = NIL;
double srf_multiplier = 1.0;
double numdistinct;
ListCell *l;
- int i,
- j;
+ int i;
/* Zero the estinfo output parameter, if non-NULL */
if (estinfo != NULL)
@@ -3421,7 +3404,7 @@ estimate_num_groups_incremental(PlannerInfo *root, List *groupExprs,
*/
numdistinct = 1.0;
- i = j = 0;
+ i = 0;
foreach(l, groupExprs)
{
Node *groupexpr = (Node *) lfirst(l);
@@ -3430,14 +3413,6 @@ estimate_num_groups_incremental(PlannerInfo *root, List *groupExprs,
List *varshere;
ListCell *l2;
- /* was done on previous call */
- if (cache_varinfos && j++ < prevNExprs)
- {
- if (pgset)
- i++; /* to keep in sync with lines below */
- continue;
- }
-
/* is expression in this grouping set? */
if (pgset && !list_member_int(*pgset, i++))
continue;
@@ -3507,11 +3482,7 @@ estimate_num_groups_incremental(PlannerInfo *root, List *groupExprs,
if (varshere == NIL)
{
if (contain_volatile_functions(groupexpr))
- {
- if (cache_varinfos)
- *cache_varinfos = varinfos;
return input_rows;
- }
continue;
}
@@ -3528,9 +3499,6 @@ estimate_num_groups_incremental(PlannerInfo *root, List *groupExprs,
}
}
- if (cache_varinfos)
- *cache_varinfos = varinfos;
-
/*
* If now no Vars, we must have an all-constant or all-boolean GROUP BY
* list.
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index fda3f9befb..7ff653b517 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -967,16 +967,6 @@ struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
- {
- {"enable_group_by_reordering", PGC_USERSET, QUERY_TUNING_METHOD,
- gettext_noop("Enables reordering of GROUP BY keys."),
- NULL,
- GUC_EXPLAIN
- },
- &enable_group_by_reordering,
- true,
- NULL, NULL, NULL
- },
{
{"geqo", PGC_USERSET, QUERY_TUNING_GEQO,
gettext_noop("Enables genetic query optimization."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 2ae76e5cfb..868d21c351 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -388,7 +388,6 @@
#enable_seqscan = on
#enable_sort = on
#enable_tidscan = on
-#enable_group_by_reordering = on
# - Planner Cost Constants -
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 294cfe9c47..6bda383bea 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1372,18 +1372,6 @@ typedef struct PathKey
bool pk_nulls_first; /* do NULLs come before normal values? */
} PathKey;
-/*
- * Combines information about pathkeys and the associated clauses.
- */
-typedef struct PathKeyInfo
-{
- pg_node_attr(no_read)
-
- NodeTag type;
- List *pathkeys;
- List *clauses;
-} PathKeyInfo;
-
/*
* VolatileFunctionStatus -- allows nodes to cache their
* contain_volatile_functions properties. VOLATILITY_UNKNOWN means not yet
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index f27d11eaa9..204e94b6d1 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -115,9 +115,7 @@ extern void cost_incremental_sort(Path *path,
Cost input_startup_cost, Cost input_total_cost,
double input_tuples, int width, Cost comparison_cost, int sort_mem,
double limit_tuples);
-extern Cost cost_sort_estimate(PlannerInfo *root, List *pathkeys,
- int nPresortedKeys, double tuples);
-extern void cost_append(AppendPath *apath, PlannerInfo *root);
+extern void cost_append(AppendPath *apath);
extern void cost_merge_append(Path *path, PlannerInfo *root,
List *pathkeys, int n_streams,
Cost input_startup_cost, Cost input_total_cost,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 881386997c..41f765d342 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -24,7 +24,6 @@ extern PGDLLIMPORT bool enable_geqo;
extern PGDLLIMPORT int geqo_threshold;
extern PGDLLIMPORT int min_parallel_table_scan_size;
extern PGDLLIMPORT int min_parallel_index_scan_size;
-extern PGDLLIMPORT bool enable_group_by_reordering;
/* Hook for plugins to get control in set_rel_pathlist() */
typedef void (*set_rel_pathlist_hook_type) (PlannerInfo *root,
@@ -203,13 +202,6 @@ typedef enum
extern PathKeysComparison compare_pathkeys(List *keys1, List *keys2);
extern bool pathkeys_contained_in(List *keys1, List *keys2);
extern bool pathkeys_count_contained_in(List *keys1, List *keys2, int *n_common);
-extern int group_keys_reorder_by_pathkeys(List *pathkeys,
- List **group_pathkeys,
- List **group_clauses);
-extern List *get_useful_group_keys_orderings(PlannerInfo *root, double nrows,
- List *path_pathkeys,
- List *group_pathkeys, List *group_clauses,
- List *aggregate_pathkeys);
extern Path *get_cheapest_path_for_pathkeys(List *paths, List *pathkeys,
Relids required_outer,
CostSelector cost_criterion,
diff --git a/src/include/utils/selfuncs.h b/src/include/utils/selfuncs.h
index d485b9bfcd..8f3d73edfb 100644
--- a/src/include/utils/selfuncs.h
+++ b/src/include/utils/selfuncs.h
@@ -214,11 +214,6 @@ extern double estimate_num_groups(PlannerInfo *root, List *groupExprs,
double input_rows, List **pgset,
EstimationInfo *estinfo);
-extern double estimate_num_groups_incremental(PlannerInfo *root, List *groupExprs,
- double input_rows, List **pgset,
- EstimationInfo *estinfo,
- List **cache_varinfos, int prevNExprs);
-
extern void estimate_hash_bucket_stats(PlannerInfo *root,
Node *hashkey, double nbuckets,
Selectivity *mcv_freq,
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index b2198724e3..fc2bd40be2 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -1210,8 +1210,7 @@ explain (costs off)
select distinct min(f1), max(f1) from minmaxtest;
QUERY PLAN
---------------------------------------------------------------------------------------------
- HashAggregate
- Group Key: $0, $1
+ Unique
InitPlan 1 (returns $0)
-> Limit
-> Merge Append
@@ -1234,8 +1233,10 @@ explain (costs off)
-> Index Only Scan using minmaxtest2i on minmaxtest2 minmaxtest_8
Index Cond: (f1 IS NOT NULL)
-> Index Only Scan Backward using minmaxtest3i on minmaxtest3 minmaxtest_9
- -> Result
-(25 rows)
+ -> Sort
+ Sort Key: ($0), ($1)
+ -> Result
+(26 rows)
select distinct min(f1), max(f1) from minmaxtest;
min | max
@@ -2525,241 +2526,6 @@ SELECT balk(hundred) FROM tenk1;
(1 row)
ROLLBACK;
--- GROUP BY optimization by reorder columns
-SELECT
- i AS id,
- i/2 AS p,
- format('%60s', i%2) AS v,
- i/4 AS c,
- i/8 AS d,
- (random() * (10000/8))::int as e --the same as d but no correlation with p
- INTO btg
-FROM
- generate_series(1, 10000) i;
-VACUUM btg;
-ANALYZE btg;
--- GROUP BY optimization by reorder columns by frequency
-SET enable_hashagg=off;
-SET max_parallel_workers= 0;
-SET max_parallel_workers_per_gather = 0;
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, v;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, v
- -> Sort
- Sort Key: p, v
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, v
- -> Sort
- Sort Key: p, v
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, c, v
- -> Sort
- Sort Key: p, c, v
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c ORDER BY v, p, c;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: v, p, c
- -> Sort
- Sort Key: v, p, c
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, d, c;
- QUERY PLAN
-------------------------------
- GroupAggregate
- Group Key: p, d, c, v
- -> Sort
- Sort Key: p, d, c, v
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, d, c ORDER BY v, p, d ,c;
- QUERY PLAN
-------------------------------
- GroupAggregate
- Group Key: v, p, d, c
- -> Sort
- Sort Key: v, p, d, c
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, d, c ORDER BY p, v, d ,c;
- QUERY PLAN
-------------------------------
- GroupAggregate
- Group Key: p, v, d, c
- -> Sort
- Sort Key: p, v, d, c
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, d, e;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, d, e
- -> Sort
- Sort Key: p, d, e
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, e, d;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, e, d
- -> Sort
- Sort Key: p, e, d
- -> Seq Scan on btg
-(5 rows)
-
-CREATE STATISTICS btg_dep ON d, e, p FROM btg;
-ANALYZE btg;
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, d, e;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, d, e
- -> Sort
- Sort Key: p, d, e
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, e, d;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, e, d
- -> Sort
- Sort Key: p, e, d
- -> Seq Scan on btg
-(5 rows)
-
--- GROUP BY optimization by reorder columns by index scan
-CREATE INDEX ON btg(p, v);
-SET enable_seqscan=off;
-SET enable_bitmapscan=off;
-VACUUM btg;
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, v;
- QUERY PLAN
-------------------------------------------------
- GroupAggregate
- Group Key: p, v
- -> Index Only Scan using btg_p_v_idx on btg
-(3 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, v ORDER BY p, v;
- QUERY PLAN
-------------------------------------------------
- GroupAggregate
- Group Key: p, v
- -> Index Only Scan using btg_p_v_idx on btg
-(3 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p;
- QUERY PLAN
-------------------------------------------------
- GroupAggregate
- Group Key: p, v
- -> Index Only Scan using btg_p_v_idx on btg
-(3 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p ORDER BY p, v;
- QUERY PLAN
-------------------------------------------------
- GroupAggregate
- Group Key: p, v
- -> Index Only Scan using btg_p_v_idx on btg
-(3 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c;
- QUERY PLAN
--------------------------------------------------
- GroupAggregate
- Group Key: p, c, v
- -> Incremental Sort
- Sort Key: p, c, v
- Presorted Key: p
- -> Index Scan using btg_p_v_idx on btg
-(6 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c ORDER BY p, v;
- QUERY PLAN
--------------------------------------------------
- GroupAggregate
- Group Key: p, v, c
- -> Incremental Sort
- Sort Key: p, v, c
- Presorted Key: p, v
- -> Index Scan using btg_p_v_idx on btg
-(6 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, c, p, d;
- QUERY PLAN
--------------------------------------------------
- GroupAggregate
- Group Key: p, c, d, v
- -> Incremental Sort
- Sort Key: p, c, d, v
- Presorted Key: p
- -> Index Scan using btg_p_v_idx on btg
-(6 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, c, p, d ORDER BY p, v;
- QUERY PLAN
--------------------------------------------------
- GroupAggregate
- Group Key: p, v, c, d
- -> Incremental Sort
- Sort Key: p, v, c, d
- Presorted Key: p, v
- -> Index Scan using btg_p_v_idx on btg
-(6 rows)
-
-DROP TABLE btg;
-RESET enable_hashagg;
-RESET max_parallel_workers;
-RESET max_parallel_workers_per_gather;
-RESET enable_seqscan;
-RESET enable_bitmapscan;
-- Secondly test the case of a parallel aggregate combiner function
-- returning NULL. For that use normal transition function, but a
-- combiner function returning NULL.
diff --git a/src/test/regress/expected/incremental_sort.out b/src/test/regress/expected/incremental_sort.out
index 49953eaade..0a631124c2 100644
--- a/src/test/regress/expected/incremental_sort.out
+++ b/src/test/regress/expected/incremental_sort.out
@@ -1439,7 +1439,7 @@ set parallel_setup_cost = 0;
set parallel_tuple_cost = 0;
set max_parallel_workers_per_gather = 2;
create table t (a int, b int, c int);
-insert into t select mod(i,10),mod(i,10),i from generate_series(1,60000) s(i);
+insert into t select mod(i,10),mod(i,10),i from generate_series(1,10000) s(i);
create index on t (a);
analyze t;
set enable_incremental_sort = off;
diff --git a/src/test/regress/expected/join.out b/src/test/regress/expected/join.out
index 2ed2e542a4..08334761ae 100644
--- a/src/test/regress/expected/join.out
+++ b/src/test/regress/expected/join.out
@@ -1984,8 +1984,8 @@ USING (name);
------+----+----
bb | 12 | 13
cc | 22 | 23
- ee | 42 |
dd | | 33
+ ee | 42 |
(4 rows)
-- Cases with non-nullable expressions in subquery results;
@@ -2019,8 +2019,8 @@ NATURAL FULL JOIN
------+------+------+------+------
bb | 12 | 2 | 13 | 3
cc | 22 | 2 | 23 | 3
- ee | 42 | 2 | |
dd | | | 33 | 3
+ ee | 42 | 2 | |
(4 rows)
SELECT * FROM
@@ -4676,20 +4676,18 @@ select d.* from d left join (select * from b group by b.id, b.c_id) s
explain (costs off)
select d.* from d left join (select distinct * from b) s
on d.a = s.id;
- QUERY PLAN
----------------------------------------------
- Merge Left Join
- Merge Cond: (d.a = s.id)
+ QUERY PLAN
+--------------------------------------
+ Merge Right Join
+ Merge Cond: (b.id = d.a)
+ -> Unique
+ -> Sort
+ Sort Key: b.id, b.c_id
+ -> Seq Scan on b
-> Sort
Sort Key: d.a
-> Seq Scan on d
- -> Sort
- Sort Key: s.id
- -> Subquery Scan on s
- -> HashAggregate
- Group Key: b.id, b.c_id
- -> Seq Scan on b
-(11 rows)
+(9 rows)
-- check join removal works when uniqueness of the join condition is enforced
-- by a UNION
@@ -6399,39 +6397,44 @@ select * from j1 natural join j2;
explain (verbose, costs off)
select * from j1
inner join (select distinct id from j3) j3 on j1.id = j3.id;
- QUERY PLAN
------------------------------------
+ QUERY PLAN
+-----------------------------------------
Nested Loop
Output: j1.id, j3.id
Inner Unique: true
Join Filter: (j1.id = j3.id)
- -> HashAggregate
+ -> Unique
Output: j3.id
- Group Key: j3.id
- -> Seq Scan on public.j3
+ -> Sort
Output: j3.id
+ Sort Key: j3.id
+ -> Seq Scan on public.j3
+ Output: j3.id
-> Seq Scan on public.j1
Output: j1.id
-(11 rows)
+(13 rows)
-- ensure group by clause allows the inner to become unique
explain (verbose, costs off)
select * from j1
inner join (select id from j3 group by id) j3 on j1.id = j3.id;
- QUERY PLAN
------------------------------------
+ QUERY PLAN
+-----------------------------------------
Nested Loop
Output: j1.id, j3.id
Inner Unique: true
Join Filter: (j1.id = j3.id)
- -> HashAggregate
+ -> Group
Output: j3.id
Group Key: j3.id
- -> Seq Scan on public.j3
+ -> Sort
Output: j3.id
+ Sort Key: j3.id
+ -> Seq Scan on public.j3
+ Output: j3.id
-> Seq Scan on public.j1
Output: j1.id
-(11 rows)
+(14 rows)
drop table j1;
drop table j2;
diff --git a/src/test/regress/expected/merge.out b/src/test/regress/expected/merge.out
index 4047c3e761..787af41dfe 100644
--- a/src/test/regress/expected/merge.out
+++ b/src/test/regress/expected/merge.out
@@ -1460,15 +1460,18 @@ WHEN MATCHED AND t.a < 10 THEN
explain_merge
--------------------------------------------------------------------
Merge on ex_mtarget t (actual rows=0 loops=1)
- -> Hash Join (actual rows=0 loops=1)
- Hash Cond: (s.a = t.a)
- -> Seq Scan on ex_msource s (actual rows=1 loops=1)
- -> Hash (actual rows=0 loops=1)
- Buckets: xxx Batches: xxx Memory Usage: xxx
+ -> Merge Join (actual rows=0 loops=1)
+ Merge Cond: (t.a = s.a)
+ -> Sort (actual rows=0 loops=1)
+ Sort Key: t.a
+ Sort Method: quicksort Memory: xxx
-> Seq Scan on ex_mtarget t (actual rows=0 loops=1)
Filter: (a < '-1000'::integer)
Rows Removed by Filter: 54
-(9 rows)
+ -> Sort (never executed)
+ Sort Key: s.a
+ -> Seq Scan on ex_msource s (never executed)
+(12 rows)
DROP TABLE ex_msource, ex_mtarget;
DROP FUNCTION explain_merge(text);
diff --git a/src/test/regress/expected/partition_aggregate.out b/src/test/regress/expected/partition_aggregate.out
index db36e3a150..a82b8fb8fb 100644
--- a/src/test/regress/expected/partition_aggregate.out
+++ b/src/test/regress/expected/partition_aggregate.out
@@ -949,12 +949,12 @@ SET parallel_setup_cost = 0;
-- is not partial agg safe.
EXPLAIN (COSTS OFF)
SELECT a, sum(b), array_agg(distinct c), count(*) FROM pagg_tab_ml GROUP BY a HAVING avg(b) < 3 ORDER BY 1, 2, 3;
- QUERY PLAN
---------------------------------------------------------------------------------------------
- Gather Merge
- Workers Planned: 2
- -> Sort
- Sort Key: pagg_tab_ml.a, (sum(pagg_tab_ml.b)), (array_agg(DISTINCT pagg_tab_ml.c))
+ QUERY PLAN
+--------------------------------------------------------------------------------------
+ Sort
+ Sort Key: pagg_tab_ml.a, (sum(pagg_tab_ml.b)), (array_agg(DISTINCT pagg_tab_ml.c))
+ -> Gather
+ Workers Planned: 2
-> Parallel Append
-> GroupAggregate
Group Key: pagg_tab_ml.a
@@ -1381,26 +1381,28 @@ SELECT x, sum(y), avg(y), count(*) FROM pagg_tab_para GROUP BY x HAVING avg(y) <
-- When GROUP BY clause does not match; partial aggregation is performed for each partition.
EXPLAIN (COSTS OFF)
SELECT y, sum(x), avg(x), count(*) FROM pagg_tab_para GROUP BY y HAVING avg(x) < 12 ORDER BY 1, 2, 3;
- QUERY PLAN
--------------------------------------------------------------------------------------
+ QUERY PLAN
+-------------------------------------------------------------------------------------------
Sort
Sort Key: pagg_tab_para.y, (sum(pagg_tab_para.x)), (avg(pagg_tab_para.x))
- -> Finalize HashAggregate
+ -> Finalize GroupAggregate
Group Key: pagg_tab_para.y
Filter: (avg(pagg_tab_para.x) < '12'::numeric)
- -> Gather
+ -> Gather Merge
Workers Planned: 2
- -> Parallel Append
- -> Partial HashAggregate
- Group Key: pagg_tab_para.y
- -> Parallel Seq Scan on pagg_tab_para_p1 pagg_tab_para
- -> Partial HashAggregate
- Group Key: pagg_tab_para_1.y
- -> Parallel Seq Scan on pagg_tab_para_p2 pagg_tab_para_1
- -> Partial HashAggregate
- Group Key: pagg_tab_para_2.y
- -> Parallel Seq Scan on pagg_tab_para_p3 pagg_tab_para_2
-(17 rows)
+ -> Sort
+ Sort Key: pagg_tab_para.y
+ -> Parallel Append
+ -> Partial HashAggregate
+ Group Key: pagg_tab_para.y
+ -> Parallel Seq Scan on pagg_tab_para_p1 pagg_tab_para
+ -> Partial HashAggregate
+ Group Key: pagg_tab_para_1.y
+ -> Parallel Seq Scan on pagg_tab_para_p2 pagg_tab_para_1
+ -> Partial HashAggregate
+ Group Key: pagg_tab_para_2.y
+ -> Parallel Seq Scan on pagg_tab_para_p3 pagg_tab_para_2
+(19 rows)
SELECT y, sum(x), avg(x), count(*) FROM pagg_tab_para GROUP BY y HAVING avg(x) < 12 ORDER BY 1, 2, 3;
y | sum | avg | count
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index 03926a8413..bb5b7c47a4 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -466,41 +466,52 @@ EXPLAIN (COSTS OFF)
SELECT a, b FROM prt1 FULL JOIN prt2 p2(b,a,c) USING(a,b)
WHERE a BETWEEN 490 AND 510
GROUP BY 1, 2 ORDER BY 1, 2;
- QUERY PLAN
------------------------------------------------------------------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------------------------------------------------------------------
Group
Group Key: (COALESCE(prt1.a, p2.a)), (COALESCE(prt1.b, p2.b))
- -> Sort
+ -> Merge Append
Sort Key: (COALESCE(prt1.a, p2.a)), (COALESCE(prt1.b, p2.b))
- -> Append
- -> Merge Full Join
- Merge Cond: ((prt1_1.a = p2_1.a) AND (prt1_1.b = p2_1.b))
- Filter: ((COALESCE(prt1_1.a, p2_1.a) >= 490) AND (COALESCE(prt1_1.a, p2_1.a) <= 510))
- -> Sort
- Sort Key: prt1_1.a, prt1_1.b
- -> Seq Scan on prt1_p1 prt1_1
- -> Sort
- Sort Key: p2_1.a, p2_1.b
- -> Seq Scan on prt2_p1 p2_1
- -> Merge Full Join
- Merge Cond: ((prt1_2.a = p2_2.a) AND (prt1_2.b = p2_2.b))
- Filter: ((COALESCE(prt1_2.a, p2_2.a) >= 490) AND (COALESCE(prt1_2.a, p2_2.a) <= 510))
- -> Sort
- Sort Key: prt1_2.a, prt1_2.b
- -> Seq Scan on prt1_p2 prt1_2
- -> Sort
- Sort Key: p2_2.a, p2_2.b
- -> Seq Scan on prt2_p2 p2_2
- -> Merge Full Join
- Merge Cond: ((prt1_3.b = p2_3.b) AND (prt1_3.a = p2_3.a))
- Filter: ((COALESCE(prt1_3.a, p2_3.a) >= 490) AND (COALESCE(prt1_3.a, p2_3.a) <= 510))
- -> Sort
- Sort Key: prt1_3.b, prt1_3.a
- -> Seq Scan on prt1_p3 prt1_3
- -> Sort
- Sort Key: p2_3.b, p2_3.a
- -> Seq Scan on prt2_p3 p2_3
-(32 rows)
+ -> Group
+ Group Key: (COALESCE(prt1.a, p2.a)), (COALESCE(prt1.b, p2.b))
+ -> Sort
+ Sort Key: (COALESCE(prt1.a, p2.a)), (COALESCE(prt1.b, p2.b))
+ -> Merge Full Join
+ Merge Cond: ((prt1.a = p2.a) AND (prt1.b = p2.b))
+ Filter: ((COALESCE(prt1.a, p2.a) >= 490) AND (COALESCE(prt1.a, p2.a) <= 510))
+ -> Sort
+ Sort Key: prt1.a, prt1.b
+ -> Seq Scan on prt1_p1 prt1
+ -> Sort
+ Sort Key: p2.a, p2.b
+ -> Seq Scan on prt2_p1 p2
+ -> Group
+ Group Key: (COALESCE(prt1_1.a, p2_1.a)), (COALESCE(prt1_1.b, p2_1.b))
+ -> Sort
+ Sort Key: (COALESCE(prt1_1.a, p2_1.a)), (COALESCE(prt1_1.b, p2_1.b))
+ -> Merge Full Join
+ Merge Cond: ((prt1_1.a = p2_1.a) AND (prt1_1.b = p2_1.b))
+ Filter: ((COALESCE(prt1_1.a, p2_1.a) >= 490) AND (COALESCE(prt1_1.a, p2_1.a) <= 510))
+ -> Sort
+ Sort Key: prt1_1.a, prt1_1.b
+ -> Seq Scan on prt1_p2 prt1_1
+ -> Sort
+ Sort Key: p2_1.a, p2_1.b
+ -> Seq Scan on prt2_p2 p2_1
+ -> Group
+ Group Key: (COALESCE(prt1_2.a, p2_2.a)), (COALESCE(prt1_2.b, p2_2.b))
+ -> Sort
+ Sort Key: (COALESCE(prt1_2.a, p2_2.a)), (COALESCE(prt1_2.b, p2_2.b))
+ -> Merge Full Join
+ Merge Cond: ((prt1_2.a = p2_2.a) AND (prt1_2.b = p2_2.b))
+ Filter: ((COALESCE(prt1_2.a, p2_2.a) >= 490) AND (COALESCE(prt1_2.a, p2_2.a) <= 510))
+ -> Sort
+ Sort Key: prt1_2.a, prt1_2.b
+ -> Seq Scan on prt1_p3 prt1_2
+ -> Sort
+ Sort Key: p2_2.a, p2_2.b
+ -> Seq Scan on prt2_p3 p2_2
+(43 rows)
SELECT a, b FROM prt1 FULL JOIN prt2 p2(b,a,c) USING(a,b)
WHERE a BETWEEN 490 AND 510
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 4e775af175..579b861d84 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -114,7 +114,6 @@ select name, setting from pg_settings where name like 'enable%';
enable_async_append | on
enable_bitmapscan | on
enable_gathermerge | on
- enable_group_by_reordering | on
enable_hashagg | on
enable_hashjoin | on
enable_incremental_sort | on
@@ -132,7 +131,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(21 rows)
+(20 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
diff --git a/src/test/regress/expected/union.out b/src/test/regress/expected/union.out
index 7ac4a9380e..dece7310cf 100644
--- a/src/test/regress/expected/union.out
+++ b/src/test/regress/expected/union.out
@@ -1303,22 +1303,24 @@ select distinct q1 from
union all
select distinct * from int8_tbl i82) ss
where q2 = q2;
- QUERY PLAN
-----------------------------------------------------
- HashAggregate
- Group Key: "*SELECT* 1".q1
- -> Append
+ QUERY PLAN
+----------------------------------------------------------
+ Unique
+ -> Merge Append
+ Sort Key: "*SELECT* 1".q1
-> Subquery Scan on "*SELECT* 1"
- -> HashAggregate
- Group Key: i81.q1, i81.q2
- -> Seq Scan on int8_tbl i81
- Filter: (q2 IS NOT NULL)
+ -> Unique
+ -> Sort
+ Sort Key: i81.q1, i81.q2
+ -> Seq Scan on int8_tbl i81
+ Filter: (q2 IS NOT NULL)
-> Subquery Scan on "*SELECT* 2"
- -> HashAggregate
- Group Key: i82.q1, i82.q2
- -> Seq Scan on int8_tbl i82
- Filter: (q2 IS NOT NULL)
-(13 rows)
+ -> Unique
+ -> Sort
+ Sort Key: i82.q1, i82.q2
+ -> Seq Scan on int8_tbl i82
+ Filter: (q2 IS NOT NULL)
+(15 rows)
select distinct q1 from
(select distinct * from int8_tbl i81
@@ -1337,22 +1339,24 @@ select distinct q1 from
union all
select distinct * from int8_tbl i82) ss
where -q1 = q2;
- QUERY PLAN
---------------------------------------------------
- HashAggregate
- Group Key: "*SELECT* 1".q1
- -> Append
+ QUERY PLAN
+--------------------------------------------------------
+ Unique
+ -> Merge Append
+ Sort Key: "*SELECT* 1".q1
-> Subquery Scan on "*SELECT* 1"
- -> HashAggregate
- Group Key: i81.q1, i81.q2
- -> Seq Scan on int8_tbl i81
- Filter: ((- q1) = q2)
+ -> Unique
+ -> Sort
+ Sort Key: i81.q1, i81.q2
+ -> Seq Scan on int8_tbl i81
+ Filter: ((- q1) = q2)
-> Subquery Scan on "*SELECT* 2"
- -> HashAggregate
- Group Key: i82.q1, i82.q2
- -> Seq Scan on int8_tbl i82
- Filter: ((- q1) = q2)
-(13 rows)
+ -> Unique
+ -> Sort
+ Sort Key: i82.q1, i82.q2
+ -> Seq Scan on int8_tbl i82
+ Filter: ((- q1) = q2)
+(15 rows)
select distinct q1 from
(select distinct * from int8_tbl i81
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index 4540a06f45..a4c00ff7a9 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1068,105 +1068,6 @@ SELECT balk(hundred) FROM tenk1;
ROLLBACK;
--- GROUP BY optimization by reorder columns
-
-SELECT
- i AS id,
- i/2 AS p,
- format('%60s', i%2) AS v,
- i/4 AS c,
- i/8 AS d,
- (random() * (10000/8))::int as e --the same as d but no correlation with p
- INTO btg
-FROM
- generate_series(1, 10000) i;
-
-VACUUM btg;
-ANALYZE btg;
-
--- GROUP BY optimization by reorder columns by frequency
-
-SET enable_hashagg=off;
-SET max_parallel_workers= 0;
-SET max_parallel_workers_per_gather = 0;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, v;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c ORDER BY v, p, c;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, d, c;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, d, c ORDER BY v, p, d ,c;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, d, c ORDER BY p, v, d ,c;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, d, e;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, e, d;
-
-CREATE STATISTICS btg_dep ON d, e, p FROM btg;
-ANALYZE btg;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, d, e;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, e, d;
-
-
--- GROUP BY optimization by reorder columns by index scan
-
-CREATE INDEX ON btg(p, v);
-SET enable_seqscan=off;
-SET enable_bitmapscan=off;
-VACUUM btg;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, v;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, v ORDER BY p, v;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p ORDER BY p, v;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c ORDER BY p, v;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, c, p, d;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, c, p, d ORDER BY p, v;
-
-DROP TABLE btg;
-
-RESET enable_hashagg;
-RESET max_parallel_workers;
-RESET max_parallel_workers_per_gather;
-RESET enable_seqscan;
-RESET enable_bitmapscan;
-
-
-- Secondly test the case of a parallel aggregate combiner function
-- returning NULL. For that use normal transition function, but a
-- combiner function returning NULL.
diff --git a/src/test/regress/sql/incremental_sort.sql b/src/test/regress/sql/incremental_sort.sql
index 6a0e87c7f6..284a354dbb 100644
--- a/src/test/regress/sql/incremental_sort.sql
+++ b/src/test/regress/sql/incremental_sort.sql
@@ -213,7 +213,7 @@ set parallel_tuple_cost = 0;
set max_parallel_workers_per_gather = 2;
create table t (a int, b int, c int);
-insert into t select mod(i,10),mod(i,10),i from generate_series(1,60000) s(i);
+insert into t select mod(i,10),mod(i,10),i from generate_series(1,10000) s(i);
create index on t (a);
analyze t;
On 10/1/22 6:57 PM, Tom Lane wrote:
> "Jonathan S. Katz" <jkatz@postgresql.org> writes:
>> On 10/1/22 3:13 PM, Tom Lane wrote:
>>> I'm still of the opinion that we need to revert this code for now.
>
>> [RMT hat, but speaking just for me] reading through Tom's analysis, this
>> seems to be the safest path forward. I have a few questions to better
>> understand:
>
>> 1. How invasive would the revert be?
>
> I've just finished constructing a draft full-reversion patch. I'm not
> confident in this yet; in particular, teasing it apart from 1349d2790
> ("Improve performance of ORDER BY / DISTINCT aggregates") was fairly
> messy. I need to look through the regression test changes and make
> sure that none are surprising. But this is approximately the right
> scope if we rip it out entirely.
>
> I plan to have a look tomorrow at the idea of reverting only the cost_sort
> changes, and rewriting get_cheapest_group_keys_order() to just sort the
> keys by decreasing numgroups estimates as I suggested upthread. That
> might be substantially less messy, because of fewer interactions with
> 1349d2790.
Maybe this leads to a follow-up question of do we continue to improve
what is in HEAD while reverting the code in v15 (particularly if it's
easier to do it that way)?
I know we're generally not in favor of that approach, but wanted to ask.
>> 2. Are the other user-visible items that would be impacted?
>
> See above. (But note that 1349d2790 is HEAD-only, not in v15.)
With the RMT hat, I'm hyperfocused on PG15 stability. We have plenty of
time time to stabilize head for v16 :)
>
>> 3. Is there an option of disabling the feature by default viable?
>
> Not one that usefully addresses my concerns. The patch did add an
> enable_group_by_reordering GUC which we could change to default-off,
> but it does nothing about the cost_sort behavioral changes. I would
> be a little inclined to rip out that GUC in either case, because
> I doubt that we need it with the more restricted change.
Understood.
I'll wait for your analysis of reverting only the cost_sort changes etc.
mentioned above.
Thanks,
Jonathan
Attachment
"Jonathan S. Katz" <jkatz@postgresql.org> writes:
> On 10/1/22 6:57 PM, Tom Lane wrote:
>> I plan to have a look tomorrow at the idea of reverting only the cost_sort
>> changes, and rewriting get_cheapest_group_keys_order() to just sort the
>> keys by decreasing numgroups estimates as I suggested upthread. That
>> might be substantially less messy, because of fewer interactions with
>> 1349d2790.
> Maybe this leads to a follow-up question of do we continue to improve
> what is in HEAD while reverting the code in v15 (particularly if it's
> easier to do it that way)?
No. I see no prospect that the cost_sort code currently in HEAD is going
to become shippable in the near future. Quite aside from the plain bugs,
I think it's based on untenable assumptions about how accurately we can
estimate the CPU costs associated with different sort-column orders.
Having said that, it's certainly possible that we should do something
different in HEAD than in v15. We could do the rewrite I suggest above
in HEAD while doing a straight-up revert in v15. I've been finding that
1349d2790 is sufficiently entwined with this code that the patches would
look significantly different in any case, so that might be the most
reliable way to proceed in v15.
regards, tom lane
> On Oct 2, 2022, at 1:12 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > "Jonathan S. Katz" <jkatz@postgresql.org> writes: >>> On 10/1/22 6:57 PM, Tom Lane wrote: >>> I plan to have a look tomorrow at the idea of reverting only the cost_sort >>> changes, and rewriting get_cheapest_group_keys_order() to just sort the >>> keys by decreasing numgroups estimates as I suggested upthread. That >>> might be substantially less messy, because of fewer interactions with >>> 1349d2790. > >> Maybe this leads to a follow-up question of do we continue to improve >> what is in HEAD while reverting the code in v15 (particularly if it's >> easier to do it that way)? > > No. I see no prospect that the cost_sort code currently in HEAD is going > to become shippable in the near future. Quite aside from the plain bugs, > I think it's based on untenable assumptions about how accurately we can > estimate the CPU costs associated with different sort-column orders. OK. > Having said that, it's certainly possible that we should do something > different in HEAD than in v15. We could do the rewrite I suggest above > in HEAD while doing a straight-up revert in v15. I've been finding that > 1349d2790 is sufficiently entwined with this code that the patches would > look significantly different in any case, so that might be the most > reliable way to proceed in v15. OK. For v15 I am heavily in favor for the least risky approach given the point we are at in the release cycle. The RMT hasn’t met yet to discuss, but from re-reading this thread again, I would recommend to revert (i.e. the “straight up revert”). I’m less opinionated on the approach for what’s in HEAD, but the rewrite you suggest sounds promising. Thanks, Jonathan
"Jonathan S. Katz" <jkatz@postgresql.org> writes:
> OK. For v15 I am heavily in favor for the least risky approach given the
> point we are at in the release cycle. The RMT hasn’t met yet to discuss,
> but from re-reading this thread again, I would recommend to revert
> (i.e. the “straight up revert”).
OK by me.
> I’m less opinionated on the approach for what’s in HEAD, but the rewrite
> you suggest sounds promising.
I'm just about to throw up my hands and go for reversion in both branches,
because I'm now discovering that the code I'd hoped to salvage in
pathkeys.c (get_useful_group_keys_orderings and related) has its very own
bugs. It's imagining that it can rearrange a PathKeys list arbitrarily
and then rearrange the GROUP BY SortGroupClause list to match, but that's
easier said than done, for a couple of different reasons. (I now
understand why db0d67db2 made a cowboy hack in get_eclass_for_sort_expr ...
but it's still a cowboy hack with difficult-to-foresee side effects.)
There are other things in there that make it painfully obvious that
this code wasn't very carefully reviewed, eg XXX comments that should
have been followed up and were not, or a reference to a nonexistent
"debug_group_by_match_order_by" flag (maybe that was a GUC at some point?).
On top of that, it's producing several distinct pathkey orderings for
the caller to try, but it's completely unclear to me that the subsequent
choice of cheapest path isn't going to largely reduce to the question
of whether we can accurately estimate the relative costs of different
sort-column orders. Which is exactly what we're finding we can't do.
So that end of it seems to need a good deal of rethinking as well.
In short, this needs a whole lotta work, and I'm not volunteering.
regards, tom lane
I wrote:
> I'm just about to throw up my hands and go for reversion in both branches,
As attached.
regards, tom lane
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 2e4e82a94f..cc9e39c4a5 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -2862,13 +2862,16 @@ select c2 * (random() <= 1)::int as c2 from ft2 group by c2 * (random() <= 1)::i
-- GROUP BY clause in various forms, cardinal, alias and constant expression
explain (verbose, costs off)
select count(c2) w, c2 x, 5 y, 7.0 z from ft1 group by 2, y, 9.0::int order by 2;
- QUERY PLAN
-------------------------------------------------------------------------------------------------------------
- Foreign Scan
+ QUERY PLAN
+---------------------------------------------------------------------------------------
+ Sort
Output: (count(c2)), c2, 5, 7.0, 9
- Relations: Aggregate on (public.ft1)
- Remote SQL: SELECT count(c2), c2, 5, 7.0, 9 FROM "S 1"."T 1" GROUP BY 2, 3, 5 ORDER BY c2 ASC NULLS LAST
-(4 rows)
+ Sort Key: ft1.c2
+ -> Foreign Scan
+ Output: (count(c2)), c2, 5, 7.0, 9
+ Relations: Aggregate on (public.ft1)
+ Remote SQL: SELECT count(c2), c2, 5, 7.0, 9 FROM "S 1"."T 1" GROUP BY 2, 3, 5
+(7 rows)
select count(c2) w, c2 x, 5 y, 7.0 z from ft1 group by 2, y, 9.0::int order by 2;
w | x | y | z
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d8848bc774..d750290f13 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -5062,20 +5062,6 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
</listitem>
</varlistentry>
- <varlistentry id="guc-enable-groupby-reordering" xreflabel="enable_group_by_reordering">
- <term><varname>enable_group_by_reordering</varname> (<type>boolean</type>)
- <indexterm>
- <primary><varname>enable_group_by_reordering</varname> configuration parameter</primary>
- </indexterm>
- </term>
- <listitem>
- <para>
- Enables or disables reordering of keys in a <literal>GROUP BY</literal>
- clause. The default is <literal>on</literal>.
- </para>
- </listitem>
- </varlistentry>
-
<varlistentry id="guc-enable-hashagg" xreflabel="enable_hashagg">
<term><varname>enable_hashagg</varname> (<type>boolean</type>)
<indexterm>
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index f486d42441..5ef29eea69 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1814,327 +1814,6 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
rterm->pathtarget->width);
}
-/*
- * is_fake_var
- * Workaround for generate_append_tlist() which generates fake Vars with
- * varno == 0, that will cause a fail of estimate_num_group() call
- *
- * XXX Ummm, why would estimate_num_group fail with this?
- */
-static bool
-is_fake_var(Expr *expr)
-{
- if (IsA(expr, RelabelType))
- expr = (Expr *) ((RelabelType *) expr)->arg;
-
- return (IsA(expr, Var) && ((Var *) expr)->varno == 0);
-}
-
-/*
- * get_width_cost_multiplier
- * Returns relative complexity of comparing two values based on its width.
- * The idea behind is that the comparison becomes more expensive the longer the
- * value is. Return value is in cpu_operator_cost units.
- */
-static double
-get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
-{
- double width = -1.0; /* fake value */
-
- if (IsA(expr, RelabelType))
- expr = (Expr *) ((RelabelType *) expr)->arg;
-
- /* Try to find actual stat in corresponding relation */
- if (IsA(expr, Var))
- {
- Var *var = (Var *) expr;
-
- if (var->varno > 0 && var->varno < root->simple_rel_array_size)
- {
- RelOptInfo *rel = root->simple_rel_array[var->varno];
-
- if (rel != NULL &&
- var->varattno >= rel->min_attr &&
- var->varattno <= rel->max_attr)
- {
- int ndx = var->varattno - rel->min_attr;
-
- if (rel->attr_widths[ndx] > 0)
- width = rel->attr_widths[ndx];
- }
- }
- }
-
- /* Didn't find any actual stats, try using type width instead. */
- if (width < 0.0)
- {
- Node *node = (Node *) expr;
-
- width = get_typavgwidth(exprType(node), exprTypmod(node));
- }
-
- /*
- * Values are passed as Datum type, so comparisons can't be cheaper than
- * comparing a Datum value.
- *
- * FIXME I find this reasoning questionable. We may pass int2, and
- * comparing it is probably a bit cheaper than comparing a bigint.
- */
- if (width <= sizeof(Datum))
- return 1.0;
-
- /*
- * We consider the cost of a comparison not to be directly proportional to
- * width of the argument, because widths of the arguments could be
- * slightly different (we only know the average width for the whole
- * column). So we use log16(width) as an estimate.
- */
- return 1.0 + 0.125 * LOG2(width / sizeof(Datum));
-}
-
-/*
- * compute_cpu_sort_cost
- * compute CPU cost of sort (i.e. in-memory)
- *
- * The main thing we need to calculate to estimate sort CPU costs is the number
- * of calls to the comparator functions. The difficulty is that for multi-column
- * sorts there may be different data types involved (for some of which the calls
- * may be much more expensive). Furthermore, columns may have a very different
- * number of distinct values - the higher the number, the fewer comparisons will
- * be needed for the following columns.
- *
- * The algorithm is incremental - we add pathkeys one by one, and at each step we
- * estimate the number of necessary comparisons (based on the number of distinct
- * groups in the current pathkey prefix and the new pathkey), and the comparison
- * costs (which is data type specific).
- *
- * Estimation of the number of comparisons is based on ideas from:
- *
- * "Quicksort Is Optimal", Robert Sedgewick, Jon Bentley, 2002
- * [https://www.cs.princeton.edu/~rs/talks/QuicksortIsOptimal.pdf]
- *
- * In term of that paper, let N - number of tuples, Xi - number of identical
- * tuples with value Ki, then the estimate of number of comparisons is:
- *
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- *
- * We assume all Xi the same because now we don't have any estimation of
- * group sizes, we have only know the estimate of number of groups (distinct
- * values). In that case, formula becomes:
- *
- * N * log(NumberOfGroups)
- *
- * For multi-column sorts we need to estimate the number of comparisons for
- * each individual column - for example with columns (c1, c2, ..., ck) we
- * can estimate that number of comparisons on ck is roughly
- *
- * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
- *
- * Let k be a column number, Gk - number of groups defined by k columns, and Fk
- * the cost of the comparison is
- *
- * N * sum( Fk * log(Gk) )
- *
- * Note: We also consider column width, not just the comparator cost.
- *
- * NOTE: some callers currently pass NIL for pathkeys because they
- * can't conveniently supply the sort keys. In this case, it will fallback to
- * simple comparison cost estimate.
- */
-static Cost
-compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
- Cost comparison_cost, double tuples, double output_tuples,
- bool heapSort)
-{
- Cost per_tuple_cost = 0.0;
- ListCell *lc;
- List *pathkeyExprs = NIL;
- double tuplesPerPrevGroup = tuples;
- double totalFuncCost = 1.0;
- bool has_fake_var = false;
- int i = 0;
- Oid prev_datatype = InvalidOid;
- List *cache_varinfos = NIL;
-
- /* fallback if pathkeys is unknown */
- if (pathkeys == NIL)
- {
- /*
- * If we'll use a bounded heap-sort keeping just K tuples in memory,
- * for a total number of tuple comparisons of N log2 K; but the
- * constant factor is a bit higher than for quicksort. Tweak it so
- * that the cost curve is continuous at the crossover point.
- */
- output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
- per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
-
- /* add cost provided by caller */
- per_tuple_cost += comparison_cost;
-
- return per_tuple_cost * tuples;
- }
-
- /*
- * Computing total cost of sorting takes into account the per-column
- * comparison function cost. We try to compute the needed number of
- * comparisons per column.
- */
- foreach(lc, pathkeys)
- {
- PathKey *pathkey = (PathKey *) lfirst(lc);
- EquivalenceMember *em;
- double nGroups,
- correctedNGroups;
- Cost funcCost = 1.0;
-
- /*
- * We believe that equivalence members aren't very different, so, to
- * estimate cost we consider just the first member.
- */
- em = (EquivalenceMember *) linitial(pathkey->pk_eclass->ec_members);
-
- if (em->em_datatype != InvalidOid)
- {
- /* do not lookup funcCost if the data type is the same */
- if (prev_datatype != em->em_datatype)
- {
- Oid sortop;
- QualCost cost;
-
- sortop = get_opfamily_member(pathkey->pk_opfamily,
- em->em_datatype, em->em_datatype,
- pathkey->pk_strategy);
-
- cost.startup = 0;
- cost.per_tuple = 0;
- add_function_cost(root, get_opcode(sortop), NULL, &cost);
-
- /*
- * add_function_cost returns the product of cpu_operator_cost
- * and procost, but we need just procost, co undo that.
- */
- funcCost = cost.per_tuple / cpu_operator_cost;
-
- prev_datatype = em->em_datatype;
- }
- }
-
- /* factor in the width of the values in this column */
- funcCost *= get_width_cost_multiplier(root, em->em_expr);
-
- /* now we have per-key cost, so add to the running total */
- totalFuncCost += funcCost;
-
- /* remember if we have found a fake Var in pathkeys */
- has_fake_var |= is_fake_var(em->em_expr);
- pathkeyExprs = lappend(pathkeyExprs, em->em_expr);
-
- /*
- * We need to calculate the number of comparisons for this column,
- * which requires knowing the group size. So we estimate the number of
- * groups by calling estimate_num_groups_incremental(), which
- * estimates the group size for "new" pathkeys.
- *
- * Note: estimate_num_groups_incremental does not handle fake Vars, so
- * use a default estimate otherwise.
- */
- if (!has_fake_var)
- nGroups = estimate_num_groups_incremental(root, pathkeyExprs,
- tuplesPerPrevGroup, NULL, NULL,
- &cache_varinfos,
- list_length(pathkeyExprs) - 1);
- else if (tuples > 4.0)
-
- /*
- * Use geometric mean as estimation if there are no stats.
- *
- * We don't use DEFAULT_NUM_DISTINCT here, because that's used for
- * a single column, but here we're dealing with multiple columns.
- */
- nGroups = ceil(2.0 + sqrt(tuples) * (i + 1) / list_length(pathkeys));
- else
- nGroups = tuples;
-
- /*
- * Presorted keys are not considered in the cost above, but we still
- * do have to compare them in the qsort comparator. So make sure to
- * factor in the cost in that case.
- */
- if (i >= nPresortedKeys)
- {
- if (heapSort)
- {
- /*
- * have to keep at least one group, and a multiple of group
- * size
- */
- correctedNGroups = ceil(output_tuples / tuplesPerPrevGroup);
- }
- else
- /* all groups in the input */
- correctedNGroups = nGroups;
-
- correctedNGroups = Max(1.0, ceil(correctedNGroups));
-
- per_tuple_cost += totalFuncCost * LOG2(correctedNGroups);
- }
-
- i++;
-
- /*
- * Uniform distributions with all groups being of the same size are
- * the best case, with nice smooth behavior. Real-world distributions
- * tend not to be uniform, though, and we don't have any reliable
- * easy-to-use information. As a basic defense against skewed
- * distributions, we use a 1.5 factor to make the expected group a bit
- * larger, but we need to be careful not to make the group larger than
- * in the preceding step.
- */
- tuplesPerPrevGroup = Min(tuplesPerPrevGroup,
- ceil(1.5 * tuplesPerPrevGroup / nGroups));
-
- /*
- * Once we get single-row group, it means tuples in the group are
- * unique and we can skip all remaining columns.
- */
- if (tuplesPerPrevGroup <= 1.0)
- break;
- }
-
- list_free(pathkeyExprs);
-
- /* per_tuple_cost is in cpu_operator_cost units */
- per_tuple_cost *= cpu_operator_cost;
-
- /*
- * Accordingly to "Introduction to algorithms", Thomas H. Cormen, Charles
- * E. Leiserson, Ronald L. Rivest, ISBN 0-07-013143-0, quicksort
- * estimation formula has additional term proportional to number of tuples
- * (see Chapter 8.2 and Theorem 4.1). That affects cases with a low number
- * of tuples, approximately less than 1e4. We could implement it as an
- * additional multiplier under the logarithm, but we use a bit more
- * complex formula which takes into account the number of unique tuples
- * and it's not clear how to combine the multiplier with the number of
- * groups. Estimate it as 10 cpu_operator_cost units.
- */
- per_tuple_cost += 10 * cpu_operator_cost;
-
- per_tuple_cost += comparison_cost;
-
- return tuples * per_tuple_cost;
-}
-
-/*
- * simple wrapper just to estimate best sort path
- */
-Cost
-cost_sort_estimate(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
- double tuples)
-{
- return compute_cpu_sort_cost(root, pathkeys, nPresortedKeys,
- 0, tuples, tuples, false);
-}
-
/*
* cost_tuplesort
* Determines and returns the cost of sorting a relation using tuplesort,
@@ -2151,7 +1830,7 @@ cost_sort_estimate(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* number of initial runs formed and M is the merge order used by tuplesort.c.
* Since the average initial run should be about sort_mem, we have
* disk traffic = 2 * relsize * ceil(logM(p / sort_mem))
- * and cpu cost (computed by compute_cpu_sort_cost()).
+ * cpu = comparison_cost * t * log2(t)
*
* If the sort is bounded (i.e., only the first k result tuples are needed)
* and k tuples can fit into sort_mem, we use a heap method that keeps only
@@ -2170,11 +1849,9 @@ cost_sort_estimate(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* 'comparison_cost' is the extra cost per comparison, if any
* 'sort_mem' is the number of kilobytes of work memory allowed for the sort
* 'limit_tuples' is the bound on the number of output tuples; -1 if no bound
- * 'startup_cost' is expected to be 0 at input. If there is "input cost" it should
- * be added by caller later
*/
static void
-cost_tuplesort(PlannerInfo *root, List *pathkeys, Cost *startup_cost, Cost *run_cost,
+cost_tuplesort(Cost *startup_cost, Cost *run_cost,
double tuples, int width,
Cost comparison_cost, int sort_mem,
double limit_tuples)
@@ -2191,6 +1868,9 @@ cost_tuplesort(PlannerInfo *root, List *pathkeys, Cost *startup_cost, Cost *run_
if (tuples < 2.0)
tuples = 2.0;
+ /* Include the default cost-per-comparison */
+ comparison_cost += 2.0 * cpu_operator_cost;
+
/* Do we have a useful LIMIT? */
if (limit_tuples > 0 && limit_tuples < tuples)
{
@@ -2214,10 +1894,12 @@ cost_tuplesort(PlannerInfo *root, List *pathkeys, Cost *startup_cost, Cost *run_
double log_runs;
double npageaccesses;
- /* CPU costs */
- *startup_cost = compute_cpu_sort_cost(root, pathkeys, 0,
- comparison_cost, tuples,
- tuples, false);
+ /*
+ * CPU costs
+ *
+ * Assume about N log2 N comparisons
+ */
+ *startup_cost = comparison_cost * tuples * LOG2(tuples);
/* Disk costs */
@@ -2233,17 +1915,18 @@ cost_tuplesort(PlannerInfo *root, List *pathkeys, Cost *startup_cost, Cost *run_
}
else if (tuples > 2 * output_tuples || input_bytes > sort_mem_bytes)
{
- /* We'll use a bounded heap-sort keeping just K tuples in memory. */
- *startup_cost = compute_cpu_sort_cost(root, pathkeys, 0,
- comparison_cost, tuples,
- output_tuples, true);
+ /*
+ * We'll use a bounded heap-sort keeping just K tuples in memory, for
+ * a total number of tuple comparisons of N log2 K; but the constant
+ * factor is a bit higher than for quicksort. Tweak it so that the
+ * cost curve is continuous at the crossover point.
+ */
+ *startup_cost = comparison_cost * tuples * LOG2(2.0 * output_tuples);
}
else
{
/* We'll use plain quicksort on all the input tuples */
- *startup_cost = compute_cpu_sort_cost(root, pathkeys, 0,
- comparison_cost, tuples,
- tuples, false);
+ *startup_cost = comparison_cost * tuples * LOG2(tuples);
}
/*
@@ -2276,8 +1959,8 @@ cost_incremental_sort(Path *path,
double input_tuples, int width, Cost comparison_cost, int sort_mem,
double limit_tuples)
{
- Cost startup_cost,
- run_cost,
+ Cost startup_cost = 0,
+ run_cost = 0,
input_run_cost = input_total_cost - input_startup_cost;
double group_tuples,
input_groups;
@@ -2362,7 +2045,7 @@ cost_incremental_sort(Path *path,
* pessimistic about incremental sort performance and increase its average
* group size by half.
*/
- cost_tuplesort(root, pathkeys, &group_startup_cost, &group_run_cost,
+ cost_tuplesort(&group_startup_cost, &group_run_cost,
1.5 * group_tuples, width, comparison_cost, sort_mem,
limit_tuples);
@@ -2370,7 +2053,7 @@ cost_incremental_sort(Path *path,
* Startup cost of incremental sort is the startup cost of its first group
* plus the cost of its input.
*/
- startup_cost = group_startup_cost
+ startup_cost += group_startup_cost
+ input_startup_cost + group_input_run_cost;
/*
@@ -2379,7 +2062,7 @@ cost_incremental_sort(Path *path,
* group, plus the total cost to process the remaining groups, plus the
* remaining cost of input.
*/
- run_cost = group_run_cost
+ run_cost += group_run_cost
+ (group_run_cost + group_startup_cost) * (input_groups - 1)
+ group_input_run_cost * (input_groups - 1);
@@ -2419,7 +2102,7 @@ cost_sort(Path *path, PlannerInfo *root,
Cost startup_cost;
Cost run_cost;
- cost_tuplesort(root, pathkeys, &startup_cost, &run_cost,
+ cost_tuplesort(&startup_cost, &run_cost,
tuples, width,
comparison_cost, sort_mem,
limit_tuples);
@@ -2517,7 +2200,7 @@ append_nonpartial_cost(List *subpaths, int numpaths, int parallel_workers)
* Determines and returns the cost of an Append node.
*/
void
-cost_append(AppendPath *apath, PlannerInfo *root)
+cost_append(AppendPath *apath)
{
ListCell *l;
@@ -2585,7 +2268,7 @@ cost_append(AppendPath *apath, PlannerInfo *root)
* any child.
*/
cost_sort(&sort_path,
- root,
+ NULL, /* doesn't currently need root */
pathkeys,
subpath->total_cost,
subpath->rows,
diff --git a/src/backend/optimizer/path/equivclass.c b/src/backend/optimizer/path/equivclass.c
index 799bdc91d0..f962ff82ad 100644
--- a/src/backend/optimizer/path/equivclass.c
+++ b/src/backend/optimizer/path/equivclass.c
@@ -681,18 +681,7 @@ get_eclass_for_sort_expr(PlannerInfo *root,
if (opcintype == cur_em->em_datatype &&
equal(expr, cur_em->em_expr))
- {
- /*
- * Match!
- *
- * Copy the sortref if it wasn't set yet. That may happen if
- * the ec was constructed from WHERE clause, i.e. it doesn't
- * have a target reference at all.
- */
- if (cur_ec->ec_sortref == 0 && sortref > 0)
- cur_ec->ec_sortref = sortref;
- return cur_ec;
- }
+ return cur_ec; /* Match! */
}
}
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index e2fdcd3163..a9943cd6e0 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -17,24 +17,17 @@
*/
#include "postgres.h"
-#include <float.h>
-
-#include "miscadmin.h"
#include "access/stratnum.h"
#include "catalog/pg_opfamily.h"
#include "nodes/makefuncs.h"
#include "nodes/nodeFuncs.h"
#include "nodes/plannodes.h"
-#include "optimizer/cost.h"
#include "optimizer/optimizer.h"
#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
#include "partitioning/partbounds.h"
#include "utils/lsyscache.h"
-#include "utils/selfuncs.h"
-/* Consider reordering of GROUP BY keys? */
-bool enable_group_by_reordering = true;
static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys);
static bool matches_boolean_partition_clause(RestrictInfo *rinfo,
@@ -363,540 +356,6 @@ pathkeys_contained_in(List *keys1, List *keys2)
return false;
}
-/*
- * group_keys_reorder_by_pathkeys
- * Reorder GROUP BY keys to match pathkeys of input path.
- *
- * Function returns new lists (pathkeys and clauses), original GROUP BY lists
- * stay untouched.
- *
- * Returns the number of GROUP BY keys with a matching pathkey.
- */
-int
-group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
- List **group_clauses)
-{
- List *new_group_pathkeys = NIL,
- *new_group_clauses = NIL;
- ListCell *lc;
- int n;
-
- if (pathkeys == NIL || *group_pathkeys == NIL)
- return 0;
-
- /*
- * Walk the pathkeys (determining ordering of the input path) and see if
- * there's a matching GROUP BY key. If we find one, we append it to the
- * list, and do the same for the clauses.
- *
- * Once we find the first pathkey without a matching GROUP BY key, the
- * rest of the pathkeys are useless and can't be used to evaluate the
- * grouping, so we abort the loop and ignore the remaining pathkeys.
- *
- * XXX Pathkeys are built in a way to allow simply comparing pointers.
- */
- foreach(lc, pathkeys)
- {
- PathKey *pathkey = (PathKey *) lfirst(lc);
- SortGroupClause *sgc;
-
- /* abort on first mismatch */
- if (!list_member_ptr(*group_pathkeys, pathkey))
- break;
-
- new_group_pathkeys = lappend(new_group_pathkeys, pathkey);
-
- sgc = get_sortgroupref_clause(pathkey->pk_eclass->ec_sortref,
- *group_clauses);
-
- new_group_clauses = lappend(new_group_clauses, sgc);
- }
-
- /* remember the number of pathkeys with a matching GROUP BY key */
- n = list_length(new_group_pathkeys);
-
- /* append the remaining group pathkeys (will be treated as not sorted) */
- *group_pathkeys = list_concat_unique_ptr(new_group_pathkeys,
- *group_pathkeys);
- *group_clauses = list_concat_unique_ptr(new_group_clauses,
- *group_clauses);
-
- return n;
-}
-
-/*
- * Used to generate all permutations of a pathkey list.
- */
-typedef struct PathkeyMutatorState
-{
- List *elemsList;
- ListCell **elemCells;
- void **elems;
- int *positions;
- int mutatorNColumns;
- int count;
-} PathkeyMutatorState;
-
-
-/*
- * PathkeyMutatorInit
- * Initialize state of the permutation generator.
- *
- * We want to generate permutations of elements in the "elems" list. We may want
- * to skip some number of elements at the beginning (when treating as presorted)
- * or at the end (we only permute a limited number of group keys).
- *
- * The list is decomposed into elements, and we also keep pointers to individual
- * cells. This allows us to build the permuted list quickly and cheaply, without
- * creating any copies.
- */
-static void
-PathkeyMutatorInit(PathkeyMutatorState *state, List *elems, int start, int end)
-{
- int i;
- int n = end - start;
- ListCell *lc;
-
- memset(state, 0, sizeof(*state));
-
- state->mutatorNColumns = n;
-
- state->elemsList = list_copy(elems);
-
- state->elems = palloc(sizeof(void *) * n);
- state->elemCells = palloc(sizeof(ListCell *) * n);
- state->positions = palloc(sizeof(int) * n);
-
- i = 0;
- for_each_cell(lc, state->elemsList, list_nth_cell(state->elemsList, start))
- {
- state->elemCells[i] = lc;
- state->elems[i] = lfirst(lc);
- state->positions[i] = i + 1;
- i++;
-
- if (i >= n)
- break;
- }
-}
-
-/* Swap two elements of an array. */
-static void
-PathkeyMutatorSwap(int *a, int i, int j)
-{
- int s = a[i];
-
- a[i] = a[j];
- a[j] = s;
-}
-
-/*
- * Generate the next permutation of elements.
- */
-static bool
-PathkeyMutatorNextSet(int *a, int n)
-{
- int j,
- k,
- l,
- r;
-
- j = n - 2;
-
- while (j >= 0 && a[j] >= a[j + 1])
- j--;
-
- if (j < 0)
- return false;
-
- k = n - 1;
-
- while (k >= 0 && a[j] >= a[k])
- k--;
-
- PathkeyMutatorSwap(a, j, k);
-
- l = j + 1;
- r = n - 1;
-
- while (l < r)
- PathkeyMutatorSwap(a, l++, r--);
-
- return true;
-}
-
-/*
- * PathkeyMutatorNext
- * Generate the next permutation of list of elements.
- *
- * Returns the next permutation (as a list of elements) or NIL if there are no
- * more permutations.
- */
-static List *
-PathkeyMutatorNext(PathkeyMutatorState *state)
-{
- int i;
-
- state->count++;
-
- /* first permutation is original list */
- if (state->count == 1)
- return state->elemsList;
-
- /* when there are no more permutations, return NIL */
- if (!PathkeyMutatorNextSet(state->positions, state->mutatorNColumns))
- {
- pfree(state->elems);
- pfree(state->elemCells);
- pfree(state->positions);
-
- list_free(state->elemsList);
-
- return NIL;
- }
-
- /* update the list cells to point to the right elements */
- for (i = 0; i < state->mutatorNColumns; i++)
- lfirst(state->elemCells[i]) =
- (void *) state->elems[state->positions[i] - 1];
-
- return state->elemsList;
-}
-
-/*
- * Cost of comparing pathkeys.
- */
-typedef struct PathkeySortCost
-{
- Cost cost;
- PathKey *pathkey;
-} PathkeySortCost;
-
-static int
-pathkey_sort_cost_comparator(const void *_a, const void *_b)
-{
- const PathkeySortCost *a = (PathkeySortCost *) _a;
- const PathkeySortCost *b = (PathkeySortCost *) _b;
-
- if (a->cost < b->cost)
- return -1;
- else if (a->cost == b->cost)
- return 0;
- return 1;
-}
-
-/*
- * get_cheapest_group_keys_order
- * Reorders the group pathkeys / clauses to minimize the comparison cost.
- *
- * Given the list of pathkeys in '*group_pathkeys', we try to arrange these
- * in an order that minimizes the sort costs that will be incurred by the
- * GROUP BY. The costs mainly depend on the cost of the sort comparator
- * function(s) and the number of distinct values in each column of the GROUP
- * BY clause (*group_clauses). Sorting on subsequent columns is only required
- * for tiebreak situations where two values sort equally.
- *
- * In case the input is partially sorted, only the remaining pathkeys are
- * considered. 'n_preordered' denotes how many of the leading *group_pathkeys
- * the input is presorted by.
- *
- * Returns true and sets *group_pathkeys and *group_clauses to the newly
- * ordered versions of the lists that were passed in via these parameters.
- * If no reordering was deemed necessary then we return false, in which case
- * the *group_pathkeys and *group_clauses lists are left untouched. The
- * original *group_pathkeys and *group_clauses parameter values are never
- * destructively modified in place.
- */
-static bool
-get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
- List **group_pathkeys, List **group_clauses,
- int n_preordered)
-{
- List *new_group_pathkeys = NIL,
- *new_group_clauses = NIL,
- *var_group_pathkeys;
-
- ListCell *cell;
- PathkeyMutatorState mstate;
- double cheapest_sort_cost = DBL_MAX;
-
- int nFreeKeys;
- int nToPermute;
-
- /* If there are less than 2 unsorted pathkeys, we're done. */
- if (list_length(*group_pathkeys) - n_preordered < 2)
- return false;
-
- /*
- * We could exhaustively cost all possible orderings of the pathkeys, but
- * for a large number of pathkeys it might be prohibitively expensive. So
- * we try to apply simple cheap heuristics first - we sort the pathkeys by
- * sort cost (as if the pathkey was sorted independently) and then check
- * only the four cheapest pathkeys. The remaining pathkeys are kept
- * ordered by cost.
- *
- * XXX This is a very simple heuristics, but likely to work fine for most
- * cases (because the number of GROUP BY clauses tends to be lower than
- * 4). But it ignores how the number of distinct values in each pathkey
- * affects the following steps. It might be better to use "more expensive"
- * pathkey first if it has many distinct values, because it then limits
- * the number of comparisons for the remaining pathkeys. But evaluating
- * that is likely quite the expensive.
- */
- nFreeKeys = list_length(*group_pathkeys) - n_preordered;
- nToPermute = 4;
- if (nFreeKeys > nToPermute)
- {
- PathkeySortCost *costs = palloc(sizeof(PathkeySortCost) * nFreeKeys);
- PathkeySortCost *cost = costs;
-
- /*
- * Estimate cost for sorting individual pathkeys skipping the
- * pre-ordered pathkeys.
- */
- for_each_from(cell, *group_pathkeys, n_preordered)
- {
- PathKey *pathkey = (PathKey *) lfirst(cell);
- List *to_cost = list_make1(pathkey);
-
- cost->pathkey = pathkey;
- cost->cost = cost_sort_estimate(root, to_cost, 0, nrows);
- cost++;
-
- list_free(to_cost);
- }
-
- /* sort the pathkeys by sort cost in ascending order */
- qsort(costs, nFreeKeys, sizeof(*costs), pathkey_sort_cost_comparator);
-
- /*
- * Rebuild the list of pathkeys - first the preordered ones, then the
- * rest ordered by cost.
- */
- new_group_pathkeys = list_copy_head(*group_pathkeys, n_preordered);
-
- for (int i = 0; i < nFreeKeys; i++)
- new_group_pathkeys = lappend(new_group_pathkeys, costs[i].pathkey);
-
- pfree(costs);
- }
- else
- {
- /* Copy the list, so that we can free the new list by list_free. */
- new_group_pathkeys = list_copy(*group_pathkeys);
- nToPermute = nFreeKeys;
- }
-
- Assert(list_length(new_group_pathkeys) == list_length(*group_pathkeys));
-
- /*
- * Generate pathkey lists with permutations of the first nToPermute
- * pathkeys.
- *
- * XXX We simply calculate sort cost for each individual pathkey list, but
- * there's room for two dynamic programming optimizations here. Firstly,
- * we may pass the current "best" cost to cost_sort_estimate so that it
- * can "abort" if the estimated pathkeys list exceeds it. Secondly, it
- * could pass the return information about the position when it exceeded
- * the cost, and we could skip all permutations with the same prefix.
- *
- * Imagine we've already found ordering with cost C1, and we're evaluating
- * another ordering - cost_sort_estimate() calculates cost by adding the
- * pathkeys one by one (more or less), and the cost only grows. If at any
- * point it exceeds C1, it can't possibly be "better" so we can discard
- * it. But we also know that we can discard all ordering with the same
- * prefix, because if we're estimating (a,b,c,d) and we exceed C1 at (a,b)
- * then the same thing will happen for any ordering with this prefix.
- */
- PathkeyMutatorInit(&mstate, new_group_pathkeys, n_preordered, n_preordered + nToPermute);
-
- while ((var_group_pathkeys = PathkeyMutatorNext(&mstate)) != NIL)
- {
- Cost cost;
-
- cost = cost_sort_estimate(root, var_group_pathkeys, n_preordered, nrows);
-
- if (cost < cheapest_sort_cost)
- {
- list_free(new_group_pathkeys);
- new_group_pathkeys = list_copy(var_group_pathkeys);
- cheapest_sort_cost = cost;
- }
- }
-
- /* Reorder the group clauses according to the reordered pathkeys. */
- foreach(cell, new_group_pathkeys)
- {
- PathKey *pathkey = (PathKey *) lfirst(cell);
-
- new_group_clauses = lappend(new_group_clauses,
- get_sortgroupref_clause(pathkey->pk_eclass->ec_sortref,
- *group_clauses));
- }
-
- /* Just append the rest GROUP BY clauses */
- new_group_clauses = list_concat_unique_ptr(new_group_clauses,
- *group_clauses);
-
- *group_pathkeys = new_group_pathkeys;
- *group_clauses = new_group_clauses;
-
- return true;
-}
-
-/*
- * get_useful_group_keys_orderings
- * Determine which orderings of GROUP BY keys are potentially interesting.
- *
- * Returns list of PathKeyInfo items, each representing an interesting ordering
- * of GROUP BY keys. Each item stores pathkeys and clauses in matching order.
- *
- * The function considers (and keeps) multiple group by orderings:
- *
- * - the original ordering, as specified by the GROUP BY clause
- *
- * - GROUP BY keys reordered to minimize the sort cost
- *
- * - GROUP BY keys reordered to match path ordering (as much as possible), with
- * the tail reordered to minimize the sort cost
- *
- * - GROUP BY keys to match target ORDER BY clause (as much as possible), with
- * the tail reordered to minimize the sort cost
- *
- * There are other potentially interesting orderings (e.g. it might be best to
- * match the first ORDER BY key, order the remaining keys differently and then
- * rely on the incremental sort to fix this), but we ignore those for now. To
- * make this work we'd have to pretty much generate all possible permutations.
- */
-List *
-get_useful_group_keys_orderings(PlannerInfo *root, double nrows,
- List *path_pathkeys,
- List *group_pathkeys, List *group_clauses,
- List *aggregate_pathkeys)
-{
- Query *parse = root->parse;
- List *infos = NIL;
- PathKeyInfo *info;
- int n_preordered = 0;
-
- List *pathkeys = group_pathkeys;
- List *clauses = group_clauses;
-
- /* always return at least the original pathkeys/clauses */
- info = makeNode(PathKeyInfo);
- if (aggregate_pathkeys != NIL)
- info->pathkeys = list_concat_copy(pathkeys, aggregate_pathkeys);
- else
- info->pathkeys = pathkeys;
- info->clauses = clauses;
-
- infos = lappend(infos, info);
-
- /*
- * Should we try generating alternative orderings of the group keys? If
- * not, we produce only the order specified in the query, i.e. the
- * optimization is effectively disabled.
- */
- if (!enable_group_by_reordering)
- return infos;
-
- /* for grouping sets we can't do any reordering */
- if (parse->groupingSets)
- return infos;
-
- /*
- * Try reordering pathkeys to minimize the sort cost, ignoring both the
- * target ordering (ORDER BY) and ordering of the input path.
- */
- if (get_cheapest_group_keys_order(root, nrows, &pathkeys, &clauses,
- n_preordered))
- {
- info = makeNode(PathKeyInfo);
- if (aggregate_pathkeys != NIL)
- info->pathkeys = list_concat_copy(pathkeys, aggregate_pathkeys);
- else
- info->pathkeys = pathkeys;
- info->clauses = clauses;
-
- infos = lappend(infos, info);
- }
-
- /*
- * If the path is sorted in some way, try reordering the group keys to
- * match as much of the ordering as possible - we get this sort for free
- * (mostly).
- *
- * We must not do this when there are no grouping sets, because those use
- * more complex logic to decide the ordering.
- *
- * XXX Isn't this somewhat redundant with presorted_keys? Actually, it's
- * more a complement, because it allows benefiting from incremental sort
- * as much as possible.
- *
- * XXX This does nothing if (n_preordered == 0). We shouldn't create the
- * info in this case.
- */
- if (path_pathkeys)
- {
- n_preordered = group_keys_reorder_by_pathkeys(path_pathkeys,
- &pathkeys,
- &clauses);
-
- /* reorder the tail to minimize sort cost */
- get_cheapest_group_keys_order(root, nrows, &pathkeys, &clauses,
- n_preordered);
-
- /*
- * reorder the tail to minimize sort cost
- *
- * XXX Ignore the return value - there may be nothing to reorder, in
- * which case get_cheapest_group_keys_order returns false. But we
- * still want to keep the keys reordered to path_pathkeys.
- */
- info = makeNode(PathKeyInfo);
- if (aggregate_pathkeys != NIL)
- info->pathkeys = list_concat_copy(pathkeys, aggregate_pathkeys);
- else
- info->pathkeys = pathkeys;
- info->clauses = clauses;
-
- infos = lappend(infos, info);
- }
-
- /*
- * Try reordering pathkeys to minimize the sort cost (this time consider
- * the ORDER BY clause, but only if set debug_group_by_match_order_by).
- */
- if (root->sort_pathkeys)
- {
- n_preordered = group_keys_reorder_by_pathkeys(root->sort_pathkeys,
- &pathkeys,
- &clauses);
-
- /*
- * reorder the tail to minimize sort cost
- *
- * XXX Ignore the return value - there may be nothing to reorder, in
- * which case get_cheapest_group_keys_order returns false. But we
- * still want to keep the keys reordered to sort_pathkeys.
- */
- get_cheapest_group_keys_order(root, nrows, &pathkeys, &clauses,
- n_preordered);
-
- /* keep the group keys reordered to match ordering of input path */
- info = makeNode(PathKeyInfo);
- if (aggregate_pathkeys != NIL)
- info->pathkeys = list_concat_copy(pathkeys, aggregate_pathkeys);
- else
- info->pathkeys = pathkeys;
- info->clauses = clauses;
-
- infos = lappend(infos, info);
- }
-
- return infos;
-}
-
/*
* pathkeys_count_contained_in
* Same as pathkeys_contained_in, but also sets length of longest
@@ -2456,54 +1915,6 @@ pathkeys_useful_for_ordering(PlannerInfo *root, List *pathkeys)
return n_common_pathkeys;
}
-/*
- * pathkeys_useful_for_grouping
- * Count the number of pathkeys that are useful for grouping (instead of
- * explicit sort)
- *
- * Group pathkeys could be reordered to benefit from the ordering. The
- * ordering may not be "complete" and may require incremental sort, but that's
- * fine. So we simply count prefix pathkeys with a matching group key, and
- * stop once we find the first pathkey without a match.
- *
- * So e.g. with pathkeys (a,b,c) and group keys (a,b,e) this determines (a,b)
- * pathkeys are useful for grouping, and we might do incremental sort to get
- * path ordered by (a,b,e).
- *
- * This logic is necessary to retain paths with ordering not matching grouping
- * keys directly, without the reordering.
- *
- * Returns the length of pathkey prefix with matching group keys.
- */
-static int
-pathkeys_useful_for_grouping(PlannerInfo *root, List *pathkeys)
-{
- ListCell *key;
- int n = 0;
-
- /* no special ordering requested for grouping */
- if (root->group_pathkeys == NIL)
- return 0;
-
- /* unordered path */
- if (pathkeys == NIL)
- return 0;
-
- /* walk the pathkeys and search for matching group key */
- foreach(key, pathkeys)
- {
- PathKey *pathkey = (PathKey *) lfirst(key);
-
- /* no matching group key, we're done */
- if (!list_member_ptr(root->group_pathkeys, pathkey))
- break;
-
- n++;
- }
-
- return n;
-}
-
/*
* truncate_useless_pathkeys
* Shorten the given pathkey list to just the useful pathkeys.
@@ -2518,9 +1929,6 @@ truncate_useless_pathkeys(PlannerInfo *root,
nuseful = pathkeys_useful_for_merging(root, rel, pathkeys);
nuseful2 = pathkeys_useful_for_ordering(root, pathkeys);
- if (nuseful2 > nuseful)
- nuseful = nuseful2;
- nuseful2 = pathkeys_useful_for_grouping(root, pathkeys);
if (nuseful2 > nuseful)
nuseful = nuseful2;
@@ -2556,8 +1964,6 @@ has_useful_pathkeys(PlannerInfo *root, RelOptInfo *rel)
{
if (rel->joininfo != NIL || rel->has_eclass_joins)
return true; /* might be able to use pathkeys for merging */
- if (root->group_pathkeys != NIL)
- return true; /* might be able to use pathkeys for grouping */
if (root->query_pathkeys != NIL)
return true; /* might be able to use them for ordering */
return false; /* definitely useless */
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index 8014d1fd25..9c5836683c 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -2758,9 +2758,8 @@ remove_useless_groupby_columns(PlannerInfo *root)
*
* In principle it might be interesting to consider other orderings of the
* GROUP BY elements, which could match the sort ordering of other
- * possible plans (eg an indexscan) and thereby reduce cost. However, we
- * don't yet have sufficient information to do that here, so that's left until
- * later in planning. See get_useful_group_keys_orderings().
+ * possible plans (eg an indexscan) and thereby reduce cost. We don't
+ * bother with that, though. Hashed grouping will frequently win anyway.
*
* Note: we need no comparable processing of the distinctClause because
* the parser already enforced that that matches ORDER BY.
@@ -6433,148 +6432,30 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
if (can_sort)
{
- List *group_pathkeys;
- List *orderAggPathkeys;
- int numAggPathkeys;
-
- numAggPathkeys = list_length(root->group_pathkeys) -
- root->num_groupby_pathkeys;
-
- if (numAggPathkeys > 0)
- {
- group_pathkeys = list_copy_head(root->group_pathkeys,
- root->num_groupby_pathkeys);
- orderAggPathkeys = list_copy_tail(root->group_pathkeys,
- root->num_groupby_pathkeys);
- }
- else
- {
- group_pathkeys = root->group_pathkeys;
- orderAggPathkeys = NIL;
- }
-
/*
* Use any available suitably-sorted path as input, and also consider
* sorting the cheapest-total path.
*/
foreach(lc, input_rel->pathlist)
{
- ListCell *lc2;
Path *path = (Path *) lfirst(lc);
Path *path_original = path;
+ bool is_sorted;
+ int presorted_keys;
- List *pathkey_orderings = NIL;
-
- List *group_clauses = parse->groupClause;
-
- /* generate alternative group orderings that might be useful */
- pathkey_orderings = get_useful_group_keys_orderings(root,
- path->rows,
- path->pathkeys,
- group_pathkeys,
- group_clauses,
- orderAggPathkeys);
-
- Assert(pathkey_orderings != NIL);
+ is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
+ path->pathkeys,
+ &presorted_keys);
- /* process all potentially interesting grouping reorderings */
- foreach(lc2, pathkey_orderings)
+ if (path == cheapest_path || is_sorted)
{
- bool is_sorted;
- int presorted_keys = 0;
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
-
- /* restore the path (we replace it in the loop) */
- path = path_original;
-
- is_sorted = pathkeys_count_contained_in(info->pathkeys,
- path->pathkeys,
- &presorted_keys);
-
- if (path == cheapest_path || is_sorted)
- {
- /* Sort the cheapest-total path if it isn't already sorted */
- if (!is_sorted)
- path = (Path *) create_sort_path(root,
- grouped_rel,
- path,
- info->pathkeys,
- -1.0);
-
- /* Now decide what to stick atop it */
- if (parse->groupingSets)
- {
- consider_groupingsets_paths(root, grouped_rel,
- path, true, can_hash,
- gd, agg_costs, dNumGroups);
- }
- else if (parse->hasAggs)
- {
- /*
- * We have aggregation, possibly with plain GROUP BY.
- * Make an AggPath.
- */
- add_path(grouped_rel, (Path *)
- create_agg_path(root,
- grouped_rel,
- path,
- grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_SIMPLE,
- info->clauses,
- havingQual,
- agg_costs,
- dNumGroups));
- }
- else if (group_clauses)
- {
- /*
- * We have GROUP BY without aggregation or grouping
- * sets. Make a GroupPath.
- */
- add_path(grouped_rel, (Path *)
- create_group_path(root,
- grouped_rel,
- path,
- info->clauses,
- havingQual,
- dNumGroups));
- }
- else
- {
- /* Other cases should have been handled above */
- Assert(false);
- }
- }
-
- /*
- * Now we may consider incremental sort on this path, but only
- * when the path is not already sorted and when incremental
- * sort is enabled.
- */
- if (is_sorted || !enable_incremental_sort)
- continue;
-
- /* Restore the input path (we might have added Sort on top). */
- path = path_original;
-
- /* no shared prefix, no point in building incremental sort */
- if (presorted_keys == 0)
- continue;
-
- /*
- * We should have already excluded pathkeys of length 1
- * because then presorted_keys > 0 would imply is_sorted was
- * true.
- */
- Assert(list_length(root->group_pathkeys) != 1);
-
- path = (Path *) create_incremental_sort_path(root,
- grouped_rel,
- path,
- info->pathkeys,
- presorted_keys,
- -1.0);
+ /* Sort the cheapest-total path if it isn't already sorted */
+ if (!is_sorted)
+ path = (Path *) create_sort_path(root,
+ grouped_rel,
+ path,
+ root->group_pathkeys,
+ -1.0);
/* Now decide what to stick atop it */
if (parse->groupingSets)
@@ -6594,9 +6475,9 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
grouped_rel,
path,
grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
AGGSPLIT_SIMPLE,
- info->clauses,
+ parse->groupClause,
havingQual,
agg_costs,
dNumGroups));
@@ -6611,7 +6492,7 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
create_group_path(root,
grouped_rel,
path,
- info->clauses,
+ parse->groupClause,
havingQual,
dNumGroups));
}
@@ -6621,6 +6502,79 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
Assert(false);
}
}
+
+ /*
+ * Now we may consider incremental sort on this path, but only
+ * when the path is not already sorted and when incremental sort
+ * is enabled.
+ */
+ if (is_sorted || !enable_incremental_sort)
+ continue;
+
+ /* Restore the input path (we might have added Sort on top). */
+ path = path_original;
+
+ /* no shared prefix, no point in building incremental sort */
+ if (presorted_keys == 0)
+ continue;
+
+ /*
+ * We should have already excluded pathkeys of length 1 because
+ * then presorted_keys > 0 would imply is_sorted was true.
+ */
+ Assert(list_length(root->group_pathkeys) != 1);
+
+ path = (Path *) create_incremental_sort_path(root,
+ grouped_rel,
+ path,
+ root->group_pathkeys,
+ presorted_keys,
+ -1.0);
+
+ /* Now decide what to stick atop it */
+ if (parse->groupingSets)
+ {
+ consider_groupingsets_paths(root, grouped_rel,
+ path, true, can_hash,
+ gd, agg_costs, dNumGroups);
+ }
+ else if (parse->hasAggs)
+ {
+ /*
+ * We have aggregation, possibly with plain GROUP BY. Make an
+ * AggPath.
+ */
+ add_path(grouped_rel, (Path *)
+ create_agg_path(root,
+ grouped_rel,
+ path,
+ grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_SIMPLE,
+ parse->groupClause,
+ havingQual,
+ agg_costs,
+ dNumGroups));
+ }
+ else if (parse->groupClause)
+ {
+ /*
+ * We have GROUP BY without aggregation or grouping sets. Make
+ * a GroupPath.
+ */
+ add_path(grouped_rel, (Path *)
+ create_group_path(root,
+ grouped_rel,
+ path,
+ parse->groupClause,
+ havingQual,
+ dNumGroups));
+ }
+ else
+ {
+ /* Other cases should have been handled above */
+ Assert(false);
+ }
}
/*
@@ -6631,128 +6585,100 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
{
foreach(lc, partially_grouped_rel->pathlist)
{
- ListCell *lc2;
Path *path = (Path *) lfirst(lc);
Path *path_original = path;
- List *pathkey_orderings = NIL;
- List *group_clauses = parse->groupClause;
-
- /* generate alternative group orderings that might be useful */
- pathkey_orderings = get_useful_group_keys_orderings(root,
- path->rows,
- path->pathkeys,
- group_pathkeys,
- group_clauses,
- orderAggPathkeys);
+ bool is_sorted;
+ int presorted_keys;
- Assert(pathkey_orderings != NIL);
+ is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
+ path->pathkeys,
+ &presorted_keys);
- /* process all potentially interesting grouping reorderings */
- foreach(lc2, pathkey_orderings)
+ /*
+ * Insert a Sort node, if required. But there's no point in
+ * sorting anything but the cheapest path.
+ */
+ if (!is_sorted)
{
- bool is_sorted;
- int presorted_keys = 0;
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
-
- /* restore the path (we replace it in the loop) */
- path = path_original;
-
- is_sorted = pathkeys_count_contained_in(info->pathkeys,
- path->pathkeys,
- &presorted_keys);
-
- /*
- * Insert a Sort node, if required. But there's no point
- * in sorting anything but the cheapest path.
- */
- if (!is_sorted)
- {
- if (path != partially_grouped_rel->cheapest_total_path)
- continue;
- path = (Path *) create_sort_path(root,
- grouped_rel,
- path,
- info->pathkeys,
- -1.0);
- }
+ if (path != partially_grouped_rel->cheapest_total_path)
+ continue;
+ path = (Path *) create_sort_path(root,
+ grouped_rel,
+ path,
+ root->group_pathkeys,
+ -1.0);
+ }
- if (parse->hasAggs)
- add_path(grouped_rel, (Path *)
- create_agg_path(root,
- grouped_rel,
- path,
- grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_FINAL_DESERIAL,
- info->clauses,
- havingQual,
- agg_final_costs,
- dNumGroups));
- else
- add_path(grouped_rel, (Path *)
- create_group_path(root,
- grouped_rel,
- path,
- info->clauses,
- havingQual,
- dNumGroups));
+ if (parse->hasAggs)
+ add_path(grouped_rel, (Path *)
+ create_agg_path(root,
+ grouped_rel,
+ path,
+ grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_FINAL_DESERIAL,
+ parse->groupClause,
+ havingQual,
+ agg_final_costs,
+ dNumGroups));
+ else
+ add_path(grouped_rel, (Path *)
+ create_group_path(root,
+ grouped_rel,
+ path,
+ parse->groupClause,
+ havingQual,
+ dNumGroups));
- /*
- * Now we may consider incremental sort on this path, but
- * only when the path is not already sorted and when
- * incremental sort is enabled.
- */
- if (is_sorted || !enable_incremental_sort)
- continue;
+ /*
+ * Now we may consider incremental sort on this path, but only
+ * when the path is not already sorted and when incremental
+ * sort is enabled.
+ */
+ if (is_sorted || !enable_incremental_sort)
+ continue;
- /*
- * Restore the input path (we might have added Sort on
- * top).
- */
- path = path_original;
+ /* Restore the input path (we might have added Sort on top). */
+ path = path_original;
- /*
- * no shared prefix, not point in building incremental
- * sort
- */
- if (presorted_keys == 0)
- continue;
+ /* no shared prefix, not point in building incremental sort */
+ if (presorted_keys == 0)
+ continue;
- /*
- * We should have already excluded pathkeys of length 1
- * because then presorted_keys > 0 would imply is_sorted
- * was true.
- */
- Assert(list_length(root->group_pathkeys) != 1);
+ /*
+ * We should have already excluded pathkeys of length 1
+ * because then presorted_keys > 0 would imply is_sorted was
+ * true.
+ */
+ Assert(list_length(root->group_pathkeys) != 1);
- path = (Path *) create_incremental_sort_path(root,
- grouped_rel,
- path,
- info->pathkeys,
- presorted_keys,
- -1.0);
+ path = (Path *) create_incremental_sort_path(root,
+ grouped_rel,
+ path,
+ root->group_pathkeys,
+ presorted_keys,
+ -1.0);
- if (parse->hasAggs)
- add_path(grouped_rel, (Path *)
- create_agg_path(root,
- grouped_rel,
- path,
- grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_FINAL_DESERIAL,
- info->clauses,
- havingQual,
- agg_final_costs,
- dNumGroups));
- else
- add_path(grouped_rel, (Path *)
- create_group_path(root,
- grouped_rel,
- path,
- info->clauses,
- havingQual,
- dNumGroups));
- }
+ if (parse->hasAggs)
+ add_path(grouped_rel, (Path *)
+ create_agg_path(root,
+ grouped_rel,
+ path,
+ grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_FINAL_DESERIAL,
+ parse->groupClause,
+ havingQual,
+ agg_final_costs,
+ dNumGroups));
+ else
+ add_path(grouped_rel, (Path *)
+ create_group_path(root,
+ grouped_rel,
+ path,
+ parse->groupClause,
+ havingQual,
+ dNumGroups));
}
}
}
@@ -6946,26 +6872,6 @@ create_partial_grouping_paths(PlannerInfo *root,
if (can_sort && cheapest_total_path != NULL)
{
- List *group_pathkeys;
- List *orderAggPathkeys;
- int numAggPathkeys;
-
- numAggPathkeys = list_length(root->group_pathkeys) -
- root->num_groupby_pathkeys;
-
- if (numAggPathkeys > 0)
- {
- group_pathkeys = list_copy_head(root->group_pathkeys,
- root->num_groupby_pathkeys);
- orderAggPathkeys = list_copy_tail(root->group_pathkeys,
- root->num_groupby_pathkeys);
- }
- else
- {
- group_pathkeys = root->group_pathkeys;
- orderAggPathkeys = NIL;
- }
-
/* This should have been checked previously */
Assert(parse->hasAggs || parse->groupClause);
@@ -6975,69 +6881,41 @@ create_partial_grouping_paths(PlannerInfo *root,
*/
foreach(lc, input_rel->pathlist)
{
- ListCell *lc2;
Path *path = (Path *) lfirst(lc);
- Path *path_save = path;
- List *pathkey_orderings = NIL;
- List *group_clauses = parse->groupClause;
-
- /* generate alternative group orderings that might be useful */
- pathkey_orderings = get_useful_group_keys_orderings(root,
- path->rows,
- path->pathkeys,
- group_pathkeys,
- group_clauses,
- orderAggPathkeys);
-
- Assert(pathkey_orderings != NIL);
-
- /* process all potentially interesting grouping reorderings */
- foreach(lc2, pathkey_orderings)
- {
- bool is_sorted;
- int presorted_keys = 0;
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
-
- /* restore the path (we replace it in the loop) */
- path = path_save;
-
- is_sorted = pathkeys_count_contained_in(info->pathkeys,
- path->pathkeys,
- &presorted_keys);
+ bool is_sorted;
- if (path == cheapest_total_path || is_sorted)
- {
- /* Sort the cheapest partial path, if it isn't already */
- if (!is_sorted)
- {
- path = (Path *) create_sort_path(root,
- partially_grouped_rel,
- path,
- info->pathkeys,
- -1.0);
- }
+ is_sorted = pathkeys_contained_in(root->group_pathkeys,
+ path->pathkeys);
+ if (path == cheapest_total_path || is_sorted)
+ {
+ /* Sort the cheapest partial path, if it isn't already */
+ if (!is_sorted)
+ path = (Path *) create_sort_path(root,
+ partially_grouped_rel,
+ path,
+ root->group_pathkeys,
+ -1.0);
- if (parse->hasAggs)
- add_path(partially_grouped_rel, (Path *)
- create_agg_path(root,
- partially_grouped_rel,
- path,
- partially_grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_INITIAL_SERIAL,
- info->clauses,
- NIL,
- agg_partial_costs,
- dNumPartialGroups));
- else
- add_path(partially_grouped_rel, (Path *)
- create_group_path(root,
- partially_grouped_rel,
- path,
- info->clauses,
- NIL,
- dNumPartialGroups));
- }
+ if (parse->hasAggs)
+ add_path(partially_grouped_rel, (Path *)
+ create_agg_path(root,
+ partially_grouped_rel,
+ path,
+ partially_grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_INITIAL_SERIAL,
+ parse->groupClause,
+ NIL,
+ agg_partial_costs,
+ dNumPartialGroups));
+ else
+ add_path(partially_grouped_rel, (Path *)
+ create_group_path(root,
+ partially_grouped_rel,
+ path,
+ parse->groupClause,
+ NIL,
+ dNumPartialGroups));
}
}
@@ -7047,8 +6925,6 @@ create_partial_grouping_paths(PlannerInfo *root,
* We can also skip the entire loop when we only have a single-item
* group_pathkeys because then we can't possibly have a presorted
* prefix of the list without having the list be fully sorted.
- *
- * XXX Shouldn't this also consider the group-key-reordering?
*/
if (enable_incremental_sort && list_length(root->group_pathkeys) > 1)
{
@@ -7103,122 +6979,27 @@ create_partial_grouping_paths(PlannerInfo *root,
if (can_sort && cheapest_partial_path != NULL)
{
- List *group_pathkeys;
- List *orderAggPathkeys;
- int numAggPathkeys;
-
- numAggPathkeys = list_length(root->group_pathkeys) -
- root->num_groupby_pathkeys;
-
- if (numAggPathkeys > 0)
- {
- group_pathkeys = list_copy_head(root->group_pathkeys,
- root->num_groupby_pathkeys);
- orderAggPathkeys = list_copy_tail(root->group_pathkeys,
- root->num_groupby_pathkeys);
- }
- else
- {
- group_pathkeys = root->group_pathkeys;
- orderAggPathkeys = NIL;
- }
-
/* Similar to above logic, but for partial paths. */
foreach(lc, input_rel->partial_pathlist)
{
- ListCell *lc2;
Path *path = (Path *) lfirst(lc);
Path *path_original = path;
- List *pathkey_orderings = NIL;
- List *group_clauses = parse->groupClause;
-
- /* generate alternative group orderings that might be useful */
- pathkey_orderings = get_useful_group_keys_orderings(root,
- path->rows,
- path->pathkeys,
- group_pathkeys,
- group_clauses,
- orderAggPathkeys);
+ bool is_sorted;
+ int presorted_keys;
- Assert(pathkey_orderings != NIL);
+ is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
+ path->pathkeys,
+ &presorted_keys);
- /* process all potentially interesting grouping reorderings */
- foreach(lc2, pathkey_orderings)
+ if (path == cheapest_partial_path || is_sorted)
{
- bool is_sorted;
- int presorted_keys = 0;
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
-
- /* restore the path (we replace it in the loop) */
- path = path_original;
-
- is_sorted = pathkeys_count_contained_in(info->pathkeys,
- path->pathkeys,
- &presorted_keys);
-
- if (path == cheapest_partial_path || is_sorted)
- {
-
- /* Sort the cheapest partial path, if it isn't already */
- if (!is_sorted)
- {
- path = (Path *) create_sort_path(root,
- partially_grouped_rel,
- path,
- info->pathkeys,
- -1.0);
- }
-
- if (parse->hasAggs)
- add_partial_path(partially_grouped_rel, (Path *)
- create_agg_path(root,
- partially_grouped_rel,
- path,
- partially_grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_INITIAL_SERIAL,
- info->clauses,
- NIL,
- agg_partial_costs,
- dNumPartialPartialGroups));
- else
- add_partial_path(partially_grouped_rel, (Path *)
- create_group_path(root,
- partially_grouped_rel,
- path,
- info->clauses,
- NIL,
- dNumPartialPartialGroups));
- }
-
- /*
- * Now we may consider incremental sort on this path, but only
- * when the path is not already sorted and when incremental
- * sort is enabled.
- */
- if (is_sorted || !enable_incremental_sort)
- continue;
-
- /* Restore the input path (we might have added Sort on top). */
- path = path_original;
-
- /* no shared prefix, not point in building incremental sort */
- if (presorted_keys == 0)
- continue;
-
- /*
- * We should have already excluded pathkeys of length 1
- * because then presorted_keys > 0 would imply is_sorted was
- * true.
- */
- Assert(list_length(root->group_pathkeys) != 1);
-
- path = (Path *) create_incremental_sort_path(root,
- partially_grouped_rel,
- path,
- info->pathkeys,
- presorted_keys,
- -1.0);
+ /* Sort the cheapest partial path, if it isn't already */
+ if (!is_sorted)
+ path = (Path *) create_sort_path(root,
+ partially_grouped_rel,
+ path,
+ root->group_pathkeys,
+ -1.0);
if (parse->hasAggs)
add_partial_path(partially_grouped_rel, (Path *)
@@ -7226,9 +7007,9 @@ create_partial_grouping_paths(PlannerInfo *root,
partially_grouped_rel,
path,
partially_grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
AGGSPLIT_INITIAL_SERIAL,
- info->clauses,
+ parse->groupClause,
NIL,
agg_partial_costs,
dNumPartialPartialGroups));
@@ -7237,10 +7018,59 @@ create_partial_grouping_paths(PlannerInfo *root,
create_group_path(root,
partially_grouped_rel,
path,
- info->clauses,
+ parse->groupClause,
NIL,
dNumPartialPartialGroups));
}
+
+ /*
+ * Now we may consider incremental sort on this path, but only
+ * when the path is not already sorted and when incremental sort
+ * is enabled.
+ */
+ if (is_sorted || !enable_incremental_sort)
+ continue;
+
+ /* Restore the input path (we might have added Sort on top). */
+ path = path_original;
+
+ /* no shared prefix, not point in building incremental sort */
+ if (presorted_keys == 0)
+ continue;
+
+ /*
+ * We should have already excluded pathkeys of length 1 because
+ * then presorted_keys > 0 would imply is_sorted was true.
+ */
+ Assert(list_length(root->group_pathkeys) != 1);
+
+ path = (Path *) create_incremental_sort_path(root,
+ partially_grouped_rel,
+ path,
+ root->group_pathkeys,
+ presorted_keys,
+ -1.0);
+
+ if (parse->hasAggs)
+ add_partial_path(partially_grouped_rel, (Path *)
+ create_agg_path(root,
+ partially_grouped_rel,
+ path,
+ partially_grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_INITIAL_SERIAL,
+ parse->groupClause,
+ NIL,
+ agg_partial_costs,
+ dNumPartialPartialGroups));
+ else
+ add_partial_path(partially_grouped_rel, (Path *)
+ create_group_path(root,
+ partially_grouped_rel,
+ path,
+ parse->groupClause,
+ NIL,
+ dNumPartialPartialGroups));
}
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index e10561d843..70f61ae7b1 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -1346,12 +1346,12 @@ create_append_path(PlannerInfo *root,
pathnode->path.total_cost = child->total_cost;
}
else
- cost_append(pathnode, root);
+ cost_append(pathnode);
/* Must do this last, else cost_append complains */
pathnode->path.pathkeys = child->pathkeys;
}
else
- cost_append(pathnode, root);
+ cost_append(pathnode);
/* If the caller provided a row estimate, override the computed value. */
if (rows >= 0)
diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c
index 1808388397..234fb66580 100644
--- a/src/backend/utils/adt/selfuncs.c
+++ b/src/backend/utils/adt/selfuncs.c
@@ -3369,28 +3369,11 @@ double
estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows,
List **pgset, EstimationInfo *estinfo)
{
- return estimate_num_groups_incremental(root, groupExprs,
- input_rows, pgset, estinfo,
- NULL, 0);
-}
-
-/*
- * estimate_num_groups_incremental
- * An estimate_num_groups variant, optimized for cases that are adding the
- * expressions incrementally (e.g. one by one).
- */
-double
-estimate_num_groups_incremental(PlannerInfo *root, List *groupExprs,
- double input_rows,
- List **pgset, EstimationInfo *estinfo,
- List **cache_varinfos, int prevNExprs)
-{
- List *varinfos = (cache_varinfos) ? *cache_varinfos : NIL;
+ List *varinfos = NIL;
double srf_multiplier = 1.0;
double numdistinct;
ListCell *l;
- int i,
- j;
+ int i;
/* Zero the estinfo output parameter, if non-NULL */
if (estinfo != NULL)
@@ -3421,7 +3404,7 @@ estimate_num_groups_incremental(PlannerInfo *root, List *groupExprs,
*/
numdistinct = 1.0;
- i = j = 0;
+ i = 0;
foreach(l, groupExprs)
{
Node *groupexpr = (Node *) lfirst(l);
@@ -3430,14 +3413,6 @@ estimate_num_groups_incremental(PlannerInfo *root, List *groupExprs,
List *varshere;
ListCell *l2;
- /* was done on previous call */
- if (cache_varinfos && j++ < prevNExprs)
- {
- if (pgset)
- i++; /* to keep in sync with lines below */
- continue;
- }
-
/* is expression in this grouping set? */
if (pgset && !list_member_int(*pgset, i++))
continue;
@@ -3507,11 +3482,7 @@ estimate_num_groups_incremental(PlannerInfo *root, List *groupExprs,
if (varshere == NIL)
{
if (contain_volatile_functions(groupexpr))
- {
- if (cache_varinfos)
- *cache_varinfos = varinfos;
return input_rows;
- }
continue;
}
@@ -3528,9 +3499,6 @@ estimate_num_groups_incremental(PlannerInfo *root, List *groupExprs,
}
}
- if (cache_varinfos)
- *cache_varinfos = varinfos;
-
/*
* If now no Vars, we must have an all-constant or all-boolean GROUP BY
* list.
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index fda3f9befb..7ff653b517 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -967,16 +967,6 @@ struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
- {
- {"enable_group_by_reordering", PGC_USERSET, QUERY_TUNING_METHOD,
- gettext_noop("Enables reordering of GROUP BY keys."),
- NULL,
- GUC_EXPLAIN
- },
- &enable_group_by_reordering,
- true,
- NULL, NULL, NULL
- },
{
{"geqo", PGC_USERSET, QUERY_TUNING_GEQO,
gettext_noop("Enables genetic query optimization."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 2ae76e5cfb..868d21c351 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -388,7 +388,6 @@
#enable_seqscan = on
#enable_sort = on
#enable_tidscan = on
-#enable_group_by_reordering = on
# - Planner Cost Constants -
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 294cfe9c47..6bda383bea 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1372,18 +1372,6 @@ typedef struct PathKey
bool pk_nulls_first; /* do NULLs come before normal values? */
} PathKey;
-/*
- * Combines information about pathkeys and the associated clauses.
- */
-typedef struct PathKeyInfo
-{
- pg_node_attr(no_read)
-
- NodeTag type;
- List *pathkeys;
- List *clauses;
-} PathKeyInfo;
-
/*
* VolatileFunctionStatus -- allows nodes to cache their
* contain_volatile_functions properties. VOLATILITY_UNKNOWN means not yet
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index f27d11eaa9..204e94b6d1 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -115,9 +115,7 @@ extern void cost_incremental_sort(Path *path,
Cost input_startup_cost, Cost input_total_cost,
double input_tuples, int width, Cost comparison_cost, int sort_mem,
double limit_tuples);
-extern Cost cost_sort_estimate(PlannerInfo *root, List *pathkeys,
- int nPresortedKeys, double tuples);
-extern void cost_append(AppendPath *apath, PlannerInfo *root);
+extern void cost_append(AppendPath *apath);
extern void cost_merge_append(Path *path, PlannerInfo *root,
List *pathkeys, int n_streams,
Cost input_startup_cost, Cost input_total_cost,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 881386997c..41f765d342 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -24,7 +24,6 @@ extern PGDLLIMPORT bool enable_geqo;
extern PGDLLIMPORT int geqo_threshold;
extern PGDLLIMPORT int min_parallel_table_scan_size;
extern PGDLLIMPORT int min_parallel_index_scan_size;
-extern PGDLLIMPORT bool enable_group_by_reordering;
/* Hook for plugins to get control in set_rel_pathlist() */
typedef void (*set_rel_pathlist_hook_type) (PlannerInfo *root,
@@ -203,13 +202,6 @@ typedef enum
extern PathKeysComparison compare_pathkeys(List *keys1, List *keys2);
extern bool pathkeys_contained_in(List *keys1, List *keys2);
extern bool pathkeys_count_contained_in(List *keys1, List *keys2, int *n_common);
-extern int group_keys_reorder_by_pathkeys(List *pathkeys,
- List **group_pathkeys,
- List **group_clauses);
-extern List *get_useful_group_keys_orderings(PlannerInfo *root, double nrows,
- List *path_pathkeys,
- List *group_pathkeys, List *group_clauses,
- List *aggregate_pathkeys);
extern Path *get_cheapest_path_for_pathkeys(List *paths, List *pathkeys,
Relids required_outer,
CostSelector cost_criterion,
diff --git a/src/include/utils/selfuncs.h b/src/include/utils/selfuncs.h
index d485b9bfcd..8f3d73edfb 100644
--- a/src/include/utils/selfuncs.h
+++ b/src/include/utils/selfuncs.h
@@ -214,11 +214,6 @@ extern double estimate_num_groups(PlannerInfo *root, List *groupExprs,
double input_rows, List **pgset,
EstimationInfo *estinfo);
-extern double estimate_num_groups_incremental(PlannerInfo *root, List *groupExprs,
- double input_rows, List **pgset,
- EstimationInfo *estinfo,
- List **cache_varinfos, int prevNExprs);
-
extern void estimate_hash_bucket_stats(PlannerInfo *root,
Node *hashkey, double nbuckets,
Selectivity *mcv_freq,
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index b2198724e3..fc2bd40be2 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -1210,8 +1210,7 @@ explain (costs off)
select distinct min(f1), max(f1) from minmaxtest;
QUERY PLAN
---------------------------------------------------------------------------------------------
- HashAggregate
- Group Key: $0, $1
+ Unique
InitPlan 1 (returns $0)
-> Limit
-> Merge Append
@@ -1234,8 +1233,10 @@ explain (costs off)
-> Index Only Scan using minmaxtest2i on minmaxtest2 minmaxtest_8
Index Cond: (f1 IS NOT NULL)
-> Index Only Scan Backward using minmaxtest3i on minmaxtest3 minmaxtest_9
- -> Result
-(25 rows)
+ -> Sort
+ Sort Key: ($0), ($1)
+ -> Result
+(26 rows)
select distinct min(f1), max(f1) from minmaxtest;
min | max
@@ -2525,241 +2526,6 @@ SELECT balk(hundred) FROM tenk1;
(1 row)
ROLLBACK;
--- GROUP BY optimization by reorder columns
-SELECT
- i AS id,
- i/2 AS p,
- format('%60s', i%2) AS v,
- i/4 AS c,
- i/8 AS d,
- (random() * (10000/8))::int as e --the same as d but no correlation with p
- INTO btg
-FROM
- generate_series(1, 10000) i;
-VACUUM btg;
-ANALYZE btg;
--- GROUP BY optimization by reorder columns by frequency
-SET enable_hashagg=off;
-SET max_parallel_workers= 0;
-SET max_parallel_workers_per_gather = 0;
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, v;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, v
- -> Sort
- Sort Key: p, v
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, v
- -> Sort
- Sort Key: p, v
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, c, v
- -> Sort
- Sort Key: p, c, v
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c ORDER BY v, p, c;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: v, p, c
- -> Sort
- Sort Key: v, p, c
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, d, c;
- QUERY PLAN
-------------------------------
- GroupAggregate
- Group Key: p, d, c, v
- -> Sort
- Sort Key: p, d, c, v
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, d, c ORDER BY v, p, d ,c;
- QUERY PLAN
-------------------------------
- GroupAggregate
- Group Key: v, p, d, c
- -> Sort
- Sort Key: v, p, d, c
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, d, c ORDER BY p, v, d ,c;
- QUERY PLAN
-------------------------------
- GroupAggregate
- Group Key: p, v, d, c
- -> Sort
- Sort Key: p, v, d, c
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, d, e;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, d, e
- -> Sort
- Sort Key: p, d, e
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, e, d;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, e, d
- -> Sort
- Sort Key: p, e, d
- -> Seq Scan on btg
-(5 rows)
-
-CREATE STATISTICS btg_dep ON d, e, p FROM btg;
-ANALYZE btg;
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, d, e;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, d, e
- -> Sort
- Sort Key: p, d, e
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, e, d;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, e, d
- -> Sort
- Sort Key: p, e, d
- -> Seq Scan on btg
-(5 rows)
-
--- GROUP BY optimization by reorder columns by index scan
-CREATE INDEX ON btg(p, v);
-SET enable_seqscan=off;
-SET enable_bitmapscan=off;
-VACUUM btg;
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, v;
- QUERY PLAN
-------------------------------------------------
- GroupAggregate
- Group Key: p, v
- -> Index Only Scan using btg_p_v_idx on btg
-(3 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, v ORDER BY p, v;
- QUERY PLAN
-------------------------------------------------
- GroupAggregate
- Group Key: p, v
- -> Index Only Scan using btg_p_v_idx on btg
-(3 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p;
- QUERY PLAN
-------------------------------------------------
- GroupAggregate
- Group Key: p, v
- -> Index Only Scan using btg_p_v_idx on btg
-(3 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p ORDER BY p, v;
- QUERY PLAN
-------------------------------------------------
- GroupAggregate
- Group Key: p, v
- -> Index Only Scan using btg_p_v_idx on btg
-(3 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c;
- QUERY PLAN
--------------------------------------------------
- GroupAggregate
- Group Key: p, c, v
- -> Incremental Sort
- Sort Key: p, c, v
- Presorted Key: p
- -> Index Scan using btg_p_v_idx on btg
-(6 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c ORDER BY p, v;
- QUERY PLAN
--------------------------------------------------
- GroupAggregate
- Group Key: p, v, c
- -> Incremental Sort
- Sort Key: p, v, c
- Presorted Key: p, v
- -> Index Scan using btg_p_v_idx on btg
-(6 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, c, p, d;
- QUERY PLAN
--------------------------------------------------
- GroupAggregate
- Group Key: p, c, d, v
- -> Incremental Sort
- Sort Key: p, c, d, v
- Presorted Key: p
- -> Index Scan using btg_p_v_idx on btg
-(6 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, c, p, d ORDER BY p, v;
- QUERY PLAN
--------------------------------------------------
- GroupAggregate
- Group Key: p, v, c, d
- -> Incremental Sort
- Sort Key: p, v, c, d
- Presorted Key: p, v
- -> Index Scan using btg_p_v_idx on btg
-(6 rows)
-
-DROP TABLE btg;
-RESET enable_hashagg;
-RESET max_parallel_workers;
-RESET max_parallel_workers_per_gather;
-RESET enable_seqscan;
-RESET enable_bitmapscan;
-- Secondly test the case of a parallel aggregate combiner function
-- returning NULL. For that use normal transition function, but a
-- combiner function returning NULL.
diff --git a/src/test/regress/expected/incremental_sort.out b/src/test/regress/expected/incremental_sort.out
index 49953eaade..0a631124c2 100644
--- a/src/test/regress/expected/incremental_sort.out
+++ b/src/test/regress/expected/incremental_sort.out
@@ -1439,7 +1439,7 @@ set parallel_setup_cost = 0;
set parallel_tuple_cost = 0;
set max_parallel_workers_per_gather = 2;
create table t (a int, b int, c int);
-insert into t select mod(i,10),mod(i,10),i from generate_series(1,60000) s(i);
+insert into t select mod(i,10),mod(i,10),i from generate_series(1,10000) s(i);
create index on t (a);
analyze t;
set enable_incremental_sort = off;
diff --git a/src/test/regress/expected/join.out b/src/test/regress/expected/join.out
index 2ed2e542a4..08334761ae 100644
--- a/src/test/regress/expected/join.out
+++ b/src/test/regress/expected/join.out
@@ -1984,8 +1984,8 @@ USING (name);
------+----+----
bb | 12 | 13
cc | 22 | 23
- ee | 42 |
dd | | 33
+ ee | 42 |
(4 rows)
-- Cases with non-nullable expressions in subquery results;
@@ -2019,8 +2019,8 @@ NATURAL FULL JOIN
------+------+------+------+------
bb | 12 | 2 | 13 | 3
cc | 22 | 2 | 23 | 3
- ee | 42 | 2 | |
dd | | | 33 | 3
+ ee | 42 | 2 | |
(4 rows)
SELECT * FROM
@@ -4676,20 +4676,18 @@ select d.* from d left join (select * from b group by b.id, b.c_id) s
explain (costs off)
select d.* from d left join (select distinct * from b) s
on d.a = s.id;
- QUERY PLAN
----------------------------------------------
- Merge Left Join
- Merge Cond: (d.a = s.id)
+ QUERY PLAN
+--------------------------------------
+ Merge Right Join
+ Merge Cond: (b.id = d.a)
+ -> Unique
+ -> Sort
+ Sort Key: b.id, b.c_id
+ -> Seq Scan on b
-> Sort
Sort Key: d.a
-> Seq Scan on d
- -> Sort
- Sort Key: s.id
- -> Subquery Scan on s
- -> HashAggregate
- Group Key: b.id, b.c_id
- -> Seq Scan on b
-(11 rows)
+(9 rows)
-- check join removal works when uniqueness of the join condition is enforced
-- by a UNION
@@ -6399,39 +6397,44 @@ select * from j1 natural join j2;
explain (verbose, costs off)
select * from j1
inner join (select distinct id from j3) j3 on j1.id = j3.id;
- QUERY PLAN
------------------------------------
+ QUERY PLAN
+-----------------------------------------
Nested Loop
Output: j1.id, j3.id
Inner Unique: true
Join Filter: (j1.id = j3.id)
- -> HashAggregate
+ -> Unique
Output: j3.id
- Group Key: j3.id
- -> Seq Scan on public.j3
+ -> Sort
Output: j3.id
+ Sort Key: j3.id
+ -> Seq Scan on public.j3
+ Output: j3.id
-> Seq Scan on public.j1
Output: j1.id
-(11 rows)
+(13 rows)
-- ensure group by clause allows the inner to become unique
explain (verbose, costs off)
select * from j1
inner join (select id from j3 group by id) j3 on j1.id = j3.id;
- QUERY PLAN
------------------------------------
+ QUERY PLAN
+-----------------------------------------
Nested Loop
Output: j1.id, j3.id
Inner Unique: true
Join Filter: (j1.id = j3.id)
- -> HashAggregate
+ -> Group
Output: j3.id
Group Key: j3.id
- -> Seq Scan on public.j3
+ -> Sort
Output: j3.id
+ Sort Key: j3.id
+ -> Seq Scan on public.j3
+ Output: j3.id
-> Seq Scan on public.j1
Output: j1.id
-(11 rows)
+(14 rows)
drop table j1;
drop table j2;
diff --git a/src/test/regress/expected/merge.out b/src/test/regress/expected/merge.out
index 4047c3e761..787af41dfe 100644
--- a/src/test/regress/expected/merge.out
+++ b/src/test/regress/expected/merge.out
@@ -1460,15 +1460,18 @@ WHEN MATCHED AND t.a < 10 THEN
explain_merge
--------------------------------------------------------------------
Merge on ex_mtarget t (actual rows=0 loops=1)
- -> Hash Join (actual rows=0 loops=1)
- Hash Cond: (s.a = t.a)
- -> Seq Scan on ex_msource s (actual rows=1 loops=1)
- -> Hash (actual rows=0 loops=1)
- Buckets: xxx Batches: xxx Memory Usage: xxx
+ -> Merge Join (actual rows=0 loops=1)
+ Merge Cond: (t.a = s.a)
+ -> Sort (actual rows=0 loops=1)
+ Sort Key: t.a
+ Sort Method: quicksort Memory: xxx
-> Seq Scan on ex_mtarget t (actual rows=0 loops=1)
Filter: (a < '-1000'::integer)
Rows Removed by Filter: 54
-(9 rows)
+ -> Sort (never executed)
+ Sort Key: s.a
+ -> Seq Scan on ex_msource s (never executed)
+(12 rows)
DROP TABLE ex_msource, ex_mtarget;
DROP FUNCTION explain_merge(text);
diff --git a/src/test/regress/expected/partition_aggregate.out b/src/test/regress/expected/partition_aggregate.out
index db36e3a150..a82b8fb8fb 100644
--- a/src/test/regress/expected/partition_aggregate.out
+++ b/src/test/regress/expected/partition_aggregate.out
@@ -949,12 +949,12 @@ SET parallel_setup_cost = 0;
-- is not partial agg safe.
EXPLAIN (COSTS OFF)
SELECT a, sum(b), array_agg(distinct c), count(*) FROM pagg_tab_ml GROUP BY a HAVING avg(b) < 3 ORDER BY 1, 2, 3;
- QUERY PLAN
---------------------------------------------------------------------------------------------
- Gather Merge
- Workers Planned: 2
- -> Sort
- Sort Key: pagg_tab_ml.a, (sum(pagg_tab_ml.b)), (array_agg(DISTINCT pagg_tab_ml.c))
+ QUERY PLAN
+--------------------------------------------------------------------------------------
+ Sort
+ Sort Key: pagg_tab_ml.a, (sum(pagg_tab_ml.b)), (array_agg(DISTINCT pagg_tab_ml.c))
+ -> Gather
+ Workers Planned: 2
-> Parallel Append
-> GroupAggregate
Group Key: pagg_tab_ml.a
@@ -1381,26 +1381,28 @@ SELECT x, sum(y), avg(y), count(*) FROM pagg_tab_para GROUP BY x HAVING avg(y) <
-- When GROUP BY clause does not match; partial aggregation is performed for each partition.
EXPLAIN (COSTS OFF)
SELECT y, sum(x), avg(x), count(*) FROM pagg_tab_para GROUP BY y HAVING avg(x) < 12 ORDER BY 1, 2, 3;
- QUERY PLAN
--------------------------------------------------------------------------------------
+ QUERY PLAN
+-------------------------------------------------------------------------------------------
Sort
Sort Key: pagg_tab_para.y, (sum(pagg_tab_para.x)), (avg(pagg_tab_para.x))
- -> Finalize HashAggregate
+ -> Finalize GroupAggregate
Group Key: pagg_tab_para.y
Filter: (avg(pagg_tab_para.x) < '12'::numeric)
- -> Gather
+ -> Gather Merge
Workers Planned: 2
- -> Parallel Append
- -> Partial HashAggregate
- Group Key: pagg_tab_para.y
- -> Parallel Seq Scan on pagg_tab_para_p1 pagg_tab_para
- -> Partial HashAggregate
- Group Key: pagg_tab_para_1.y
- -> Parallel Seq Scan on pagg_tab_para_p2 pagg_tab_para_1
- -> Partial HashAggregate
- Group Key: pagg_tab_para_2.y
- -> Parallel Seq Scan on pagg_tab_para_p3 pagg_tab_para_2
-(17 rows)
+ -> Sort
+ Sort Key: pagg_tab_para.y
+ -> Parallel Append
+ -> Partial HashAggregate
+ Group Key: pagg_tab_para.y
+ -> Parallel Seq Scan on pagg_tab_para_p1 pagg_tab_para
+ -> Partial HashAggregate
+ Group Key: pagg_tab_para_1.y
+ -> Parallel Seq Scan on pagg_tab_para_p2 pagg_tab_para_1
+ -> Partial HashAggregate
+ Group Key: pagg_tab_para_2.y
+ -> Parallel Seq Scan on pagg_tab_para_p3 pagg_tab_para_2
+(19 rows)
SELECT y, sum(x), avg(x), count(*) FROM pagg_tab_para GROUP BY y HAVING avg(x) < 12 ORDER BY 1, 2, 3;
y | sum | avg | count
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index 03926a8413..bb5b7c47a4 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -466,41 +466,52 @@ EXPLAIN (COSTS OFF)
SELECT a, b FROM prt1 FULL JOIN prt2 p2(b,a,c) USING(a,b)
WHERE a BETWEEN 490 AND 510
GROUP BY 1, 2 ORDER BY 1, 2;
- QUERY PLAN
------------------------------------------------------------------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------------------------------------------------------------------
Group
Group Key: (COALESCE(prt1.a, p2.a)), (COALESCE(prt1.b, p2.b))
- -> Sort
+ -> Merge Append
Sort Key: (COALESCE(prt1.a, p2.a)), (COALESCE(prt1.b, p2.b))
- -> Append
- -> Merge Full Join
- Merge Cond: ((prt1_1.a = p2_1.a) AND (prt1_1.b = p2_1.b))
- Filter: ((COALESCE(prt1_1.a, p2_1.a) >= 490) AND (COALESCE(prt1_1.a, p2_1.a) <= 510))
- -> Sort
- Sort Key: prt1_1.a, prt1_1.b
- -> Seq Scan on prt1_p1 prt1_1
- -> Sort
- Sort Key: p2_1.a, p2_1.b
- -> Seq Scan on prt2_p1 p2_1
- -> Merge Full Join
- Merge Cond: ((prt1_2.a = p2_2.a) AND (prt1_2.b = p2_2.b))
- Filter: ((COALESCE(prt1_2.a, p2_2.a) >= 490) AND (COALESCE(prt1_2.a, p2_2.a) <= 510))
- -> Sort
- Sort Key: prt1_2.a, prt1_2.b
- -> Seq Scan on prt1_p2 prt1_2
- -> Sort
- Sort Key: p2_2.a, p2_2.b
- -> Seq Scan on prt2_p2 p2_2
- -> Merge Full Join
- Merge Cond: ((prt1_3.b = p2_3.b) AND (prt1_3.a = p2_3.a))
- Filter: ((COALESCE(prt1_3.a, p2_3.a) >= 490) AND (COALESCE(prt1_3.a, p2_3.a) <= 510))
- -> Sort
- Sort Key: prt1_3.b, prt1_3.a
- -> Seq Scan on prt1_p3 prt1_3
- -> Sort
- Sort Key: p2_3.b, p2_3.a
- -> Seq Scan on prt2_p3 p2_3
-(32 rows)
+ -> Group
+ Group Key: (COALESCE(prt1.a, p2.a)), (COALESCE(prt1.b, p2.b))
+ -> Sort
+ Sort Key: (COALESCE(prt1.a, p2.a)), (COALESCE(prt1.b, p2.b))
+ -> Merge Full Join
+ Merge Cond: ((prt1.a = p2.a) AND (prt1.b = p2.b))
+ Filter: ((COALESCE(prt1.a, p2.a) >= 490) AND (COALESCE(prt1.a, p2.a) <= 510))
+ -> Sort
+ Sort Key: prt1.a, prt1.b
+ -> Seq Scan on prt1_p1 prt1
+ -> Sort
+ Sort Key: p2.a, p2.b
+ -> Seq Scan on prt2_p1 p2
+ -> Group
+ Group Key: (COALESCE(prt1_1.a, p2_1.a)), (COALESCE(prt1_1.b, p2_1.b))
+ -> Sort
+ Sort Key: (COALESCE(prt1_1.a, p2_1.a)), (COALESCE(prt1_1.b, p2_1.b))
+ -> Merge Full Join
+ Merge Cond: ((prt1_1.a = p2_1.a) AND (prt1_1.b = p2_1.b))
+ Filter: ((COALESCE(prt1_1.a, p2_1.a) >= 490) AND (COALESCE(prt1_1.a, p2_1.a) <= 510))
+ -> Sort
+ Sort Key: prt1_1.a, prt1_1.b
+ -> Seq Scan on prt1_p2 prt1_1
+ -> Sort
+ Sort Key: p2_1.a, p2_1.b
+ -> Seq Scan on prt2_p2 p2_1
+ -> Group
+ Group Key: (COALESCE(prt1_2.a, p2_2.a)), (COALESCE(prt1_2.b, p2_2.b))
+ -> Sort
+ Sort Key: (COALESCE(prt1_2.a, p2_2.a)), (COALESCE(prt1_2.b, p2_2.b))
+ -> Merge Full Join
+ Merge Cond: ((prt1_2.a = p2_2.a) AND (prt1_2.b = p2_2.b))
+ Filter: ((COALESCE(prt1_2.a, p2_2.a) >= 490) AND (COALESCE(prt1_2.a, p2_2.a) <= 510))
+ -> Sort
+ Sort Key: prt1_2.a, prt1_2.b
+ -> Seq Scan on prt1_p3 prt1_2
+ -> Sort
+ Sort Key: p2_2.a, p2_2.b
+ -> Seq Scan on prt2_p3 p2_2
+(43 rows)
SELECT a, b FROM prt1 FULL JOIN prt2 p2(b,a,c) USING(a,b)
WHERE a BETWEEN 490 AND 510
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 4e775af175..579b861d84 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -114,7 +114,6 @@ select name, setting from pg_settings where name like 'enable%';
enable_async_append | on
enable_bitmapscan | on
enable_gathermerge | on
- enable_group_by_reordering | on
enable_hashagg | on
enable_hashjoin | on
enable_incremental_sort | on
@@ -132,7 +131,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(21 rows)
+(20 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
diff --git a/src/test/regress/expected/union.out b/src/test/regress/expected/union.out
index 7ac4a9380e..dece7310cf 100644
--- a/src/test/regress/expected/union.out
+++ b/src/test/regress/expected/union.out
@@ -1303,22 +1303,24 @@ select distinct q1 from
union all
select distinct * from int8_tbl i82) ss
where q2 = q2;
- QUERY PLAN
-----------------------------------------------------
- HashAggregate
- Group Key: "*SELECT* 1".q1
- -> Append
+ QUERY PLAN
+----------------------------------------------------------
+ Unique
+ -> Merge Append
+ Sort Key: "*SELECT* 1".q1
-> Subquery Scan on "*SELECT* 1"
- -> HashAggregate
- Group Key: i81.q1, i81.q2
- -> Seq Scan on int8_tbl i81
- Filter: (q2 IS NOT NULL)
+ -> Unique
+ -> Sort
+ Sort Key: i81.q1, i81.q2
+ -> Seq Scan on int8_tbl i81
+ Filter: (q2 IS NOT NULL)
-> Subquery Scan on "*SELECT* 2"
- -> HashAggregate
- Group Key: i82.q1, i82.q2
- -> Seq Scan on int8_tbl i82
- Filter: (q2 IS NOT NULL)
-(13 rows)
+ -> Unique
+ -> Sort
+ Sort Key: i82.q1, i82.q2
+ -> Seq Scan on int8_tbl i82
+ Filter: (q2 IS NOT NULL)
+(15 rows)
select distinct q1 from
(select distinct * from int8_tbl i81
@@ -1337,22 +1339,24 @@ select distinct q1 from
union all
select distinct * from int8_tbl i82) ss
where -q1 = q2;
- QUERY PLAN
---------------------------------------------------
- HashAggregate
- Group Key: "*SELECT* 1".q1
- -> Append
+ QUERY PLAN
+--------------------------------------------------------
+ Unique
+ -> Merge Append
+ Sort Key: "*SELECT* 1".q1
-> Subquery Scan on "*SELECT* 1"
- -> HashAggregate
- Group Key: i81.q1, i81.q2
- -> Seq Scan on int8_tbl i81
- Filter: ((- q1) = q2)
+ -> Unique
+ -> Sort
+ Sort Key: i81.q1, i81.q2
+ -> Seq Scan on int8_tbl i81
+ Filter: ((- q1) = q2)
-> Subquery Scan on "*SELECT* 2"
- -> HashAggregate
- Group Key: i82.q1, i82.q2
- -> Seq Scan on int8_tbl i82
- Filter: ((- q1) = q2)
-(13 rows)
+ -> Unique
+ -> Sort
+ Sort Key: i82.q1, i82.q2
+ -> Seq Scan on int8_tbl i82
+ Filter: ((- q1) = q2)
+(15 rows)
select distinct q1 from
(select distinct * from int8_tbl i81
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index 4540a06f45..a4c00ff7a9 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1068,105 +1068,6 @@ SELECT balk(hundred) FROM tenk1;
ROLLBACK;
--- GROUP BY optimization by reorder columns
-
-SELECT
- i AS id,
- i/2 AS p,
- format('%60s', i%2) AS v,
- i/4 AS c,
- i/8 AS d,
- (random() * (10000/8))::int as e --the same as d but no correlation with p
- INTO btg
-FROM
- generate_series(1, 10000) i;
-
-VACUUM btg;
-ANALYZE btg;
-
--- GROUP BY optimization by reorder columns by frequency
-
-SET enable_hashagg=off;
-SET max_parallel_workers= 0;
-SET max_parallel_workers_per_gather = 0;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, v;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c ORDER BY v, p, c;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, d, c;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, d, c ORDER BY v, p, d ,c;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, d, c ORDER BY p, v, d ,c;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, d, e;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, e, d;
-
-CREATE STATISTICS btg_dep ON d, e, p FROM btg;
-ANALYZE btg;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, d, e;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, e, d;
-
-
--- GROUP BY optimization by reorder columns by index scan
-
-CREATE INDEX ON btg(p, v);
-SET enable_seqscan=off;
-SET enable_bitmapscan=off;
-VACUUM btg;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, v;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, v ORDER BY p, v;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p ORDER BY p, v;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c ORDER BY p, v;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, c, p, d;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, c, p, d ORDER BY p, v;
-
-DROP TABLE btg;
-
-RESET enable_hashagg;
-RESET max_parallel_workers;
-RESET max_parallel_workers_per_gather;
-RESET enable_seqscan;
-RESET enable_bitmapscan;
-
-
-- Secondly test the case of a parallel aggregate combiner function
-- returning NULL. For that use normal transition function, but a
-- combiner function returning NULL.
diff --git a/src/test/regress/sql/incremental_sort.sql b/src/test/regress/sql/incremental_sort.sql
index 6a0e87c7f6..284a354dbb 100644
--- a/src/test/regress/sql/incremental_sort.sql
+++ b/src/test/regress/sql/incremental_sort.sql
@@ -213,7 +213,7 @@ set parallel_tuple_cost = 0;
set max_parallel_workers_per_gather = 2;
create table t (a int, b int, c int);
-insert into t select mod(i,10),mod(i,10),i from generate_series(1,60000) s(i);
+insert into t select mod(i,10),mod(i,10),i from generate_series(1,10000) s(i);
create index on t (a);
analyze t;
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 6e581899ed..1ca248f337 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -2840,13 +2840,16 @@ select c2 * (random() <= 1)::int as c2 from ft2 group by c2 * (random() <= 1)::i
-- GROUP BY clause in various forms, cardinal, alias and constant expression
explain (verbose, costs off)
select count(c2) w, c2 x, 5 y, 7.0 z from ft1 group by 2, y, 9.0::int order by 2;
- QUERY PLAN
-------------------------------------------------------------------------------------------------------------
- Foreign Scan
+ QUERY PLAN
+---------------------------------------------------------------------------------------
+ Sort
Output: (count(c2)), c2, 5, 7.0, 9
- Relations: Aggregate on (public.ft1)
- Remote SQL: SELECT count(c2), c2, 5, 7.0, 9 FROM "S 1"."T 1" GROUP BY 2, 3, 5 ORDER BY c2 ASC NULLS LAST
-(4 rows)
+ Sort Key: ft1.c2
+ -> Foreign Scan
+ Output: (count(c2)), c2, 5, 7.0, 9
+ Relations: Aggregate on (public.ft1)
+ Remote SQL: SELECT count(c2), c2, 5, 7.0, 9 FROM "S 1"."T 1" GROUP BY 2, 3, 5
+(7 rows)
select count(c2) w, c2 x, 5 y, 7.0 z from ft1 group by 2, y, 9.0::int order by 2;
w | x | y | z
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 2fe902eed2..331836ba5d 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -5055,20 +5055,6 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class="
</listitem>
</varlistentry>
- <varlistentry id="guc-enable-groupby-reordering" xreflabel="enable_group_by_reordering">
- <term><varname>enable_group_by_reordering</varname> (<type>boolean</type>)
- <indexterm>
- <primary><varname>enable_group_by_reordering</varname> configuration parameter</primary>
- </indexterm>
- </term>
- <listitem>
- <para>
- Enables or disables reordering of keys in a <literal>GROUP BY</literal>
- clause. The default is <literal>on</literal>.
- </para>
- </listitem>
- </varlistentry>
-
<varlistentry id="guc-enable-hashagg" xreflabel="enable_hashagg">
<term><varname>enable_hashagg</varname> (<type>boolean</type>)
<indexterm>
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 8a7f61b0ae..0ba26b207b 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1796,327 +1796,6 @@ cost_recursive_union(Path *runion, Path *nrterm, Path *rterm)
rterm->pathtarget->width);
}
-/*
- * is_fake_var
- * Workaround for generate_append_tlist() which generates fake Vars with
- * varno == 0, that will cause a fail of estimate_num_group() call
- *
- * XXX Ummm, why would estimate_num_group fail with this?
- */
-static bool
-is_fake_var(Expr *expr)
-{
- if (IsA(expr, RelabelType))
- expr = (Expr *) ((RelabelType *) expr)->arg;
-
- return (IsA(expr, Var) && ((Var *) expr)->varno == 0);
-}
-
-/*
- * get_width_cost_multiplier
- * Returns relative complexity of comparing two values based on its width.
- * The idea behind is that the comparison becomes more expensive the longer the
- * value is. Return value is in cpu_operator_cost units.
- */
-static double
-get_width_cost_multiplier(PlannerInfo *root, Expr *expr)
-{
- double width = -1.0; /* fake value */
-
- if (IsA(expr, RelabelType))
- expr = (Expr *) ((RelabelType *) expr)->arg;
-
- /* Try to find actual stat in corresponding relation */
- if (IsA(expr, Var))
- {
- Var *var = (Var *) expr;
-
- if (var->varno > 0 && var->varno < root->simple_rel_array_size)
- {
- RelOptInfo *rel = root->simple_rel_array[var->varno];
-
- if (rel != NULL &&
- var->varattno >= rel->min_attr &&
- var->varattno <= rel->max_attr)
- {
- int ndx = var->varattno - rel->min_attr;
-
- if (rel->attr_widths[ndx] > 0)
- width = rel->attr_widths[ndx];
- }
- }
- }
-
- /* Didn't find any actual stats, try using type width instead. */
- if (width < 0.0)
- {
- Node *node = (Node *) expr;
-
- width = get_typavgwidth(exprType(node), exprTypmod(node));
- }
-
- /*
- * Values are passed as Datum type, so comparisons can't be cheaper than
- * comparing a Datum value.
- *
- * FIXME I find this reasoning questionable. We may pass int2, and
- * comparing it is probably a bit cheaper than comparing a bigint.
- */
- if (width <= sizeof(Datum))
- return 1.0;
-
- /*
- * We consider the cost of a comparison not to be directly proportional to
- * width of the argument, because widths of the arguments could be
- * slightly different (we only know the average width for the whole
- * column). So we use log16(width) as an estimate.
- */
- return 1.0 + 0.125 * LOG2(width / sizeof(Datum));
-}
-
-/*
- * compute_cpu_sort_cost
- * compute CPU cost of sort (i.e. in-memory)
- *
- * The main thing we need to calculate to estimate sort CPU costs is the number
- * of calls to the comparator functions. The difficulty is that for multi-column
- * sorts there may be different data types involved (for some of which the calls
- * may be much more expensive). Furthermore, columns may have a very different
- * number of distinct values - the higher the number, the fewer comparisons will
- * be needed for the following columns.
- *
- * The algorithm is incremental - we add pathkeys one by one, and at each step we
- * estimate the number of necessary comparisons (based on the number of distinct
- * groups in the current pathkey prefix and the new pathkey), and the comparison
- * costs (which is data type specific).
- *
- * Estimation of the number of comparisons is based on ideas from:
- *
- * "Quicksort Is Optimal", Robert Sedgewick, Jon Bentley, 2002
- * [https://www.cs.princeton.edu/~rs/talks/QuicksortIsOptimal.pdf]
- *
- * In term of that paper, let N - number of tuples, Xi - number of identical
- * tuples with value Ki, then the estimate of number of comparisons is:
- *
- * log(N! / (X1! * X2! * ..)) ~ sum(Xi * log(N/Xi))
- *
- * We assume all Xi the same because now we don't have any estimation of
- * group sizes, we have only know the estimate of number of groups (distinct
- * values). In that case, formula becomes:
- *
- * N * log(NumberOfGroups)
- *
- * For multi-column sorts we need to estimate the number of comparisons for
- * each individual column - for example with columns (c1, c2, ..., ck) we
- * can estimate that number of comparisons on ck is roughly
- *
- * ncomparisons(c1, c2, ..., ck) / ncomparisons(c1, c2, ..., c(k-1))
- *
- * Let k be a column number, Gk - number of groups defined by k columns, and Fk
- * the cost of the comparison is
- *
- * N * sum( Fk * log(Gk) )
- *
- * Note: We also consider column width, not just the comparator cost.
- *
- * NOTE: some callers currently pass NIL for pathkeys because they
- * can't conveniently supply the sort keys. In this case, it will fallback to
- * simple comparison cost estimate.
- */
-static Cost
-compute_cpu_sort_cost(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
- Cost comparison_cost, double tuples, double output_tuples,
- bool heapSort)
-{
- Cost per_tuple_cost = 0.0;
- ListCell *lc;
- List *pathkeyExprs = NIL;
- double tuplesPerPrevGroup = tuples;
- double totalFuncCost = 1.0;
- bool has_fake_var = false;
- int i = 0;
- Oid prev_datatype = InvalidOid;
- List *cache_varinfos = NIL;
-
- /* fallback if pathkeys is unknown */
- if (list_length(pathkeys) == 0)
- {
- /*
- * If we'll use a bounded heap-sort keeping just K tuples in memory,
- * for a total number of tuple comparisons of N log2 K; but the
- * constant factor is a bit higher than for quicksort. Tweak it so
- * that the cost curve is continuous at the crossover point.
- */
- output_tuples = (heapSort) ? 2.0 * output_tuples : tuples;
- per_tuple_cost += 2.0 * cpu_operator_cost * LOG2(output_tuples);
-
- /* add cost provided by caller */
- per_tuple_cost += comparison_cost;
-
- return per_tuple_cost * tuples;
- }
-
- /*
- * Computing total cost of sorting takes into account the per-column
- * comparison function cost. We try to compute the needed number of
- * comparisons per column.
- */
- foreach(lc, pathkeys)
- {
- PathKey *pathkey = (PathKey *) lfirst(lc);
- EquivalenceMember *em;
- double nGroups,
- correctedNGroups;
- Cost funcCost = 1.0;
-
- /*
- * We believe that equivalence members aren't very different, so, to
- * estimate cost we consider just the first member.
- */
- em = (EquivalenceMember *) linitial(pathkey->pk_eclass->ec_members);
-
- if (em->em_datatype != InvalidOid)
- {
- /* do not lookup funcCost if the data type is the same */
- if (prev_datatype != em->em_datatype)
- {
- Oid sortop;
- QualCost cost;
-
- sortop = get_opfamily_member(pathkey->pk_opfamily,
- em->em_datatype, em->em_datatype,
- pathkey->pk_strategy);
-
- cost.startup = 0;
- cost.per_tuple = 0;
- add_function_cost(root, get_opcode(sortop), NULL, &cost);
-
- /*
- * add_function_cost returns the product of cpu_operator_cost
- * and procost, but we need just procost, co undo that.
- */
- funcCost = cost.per_tuple / cpu_operator_cost;
-
- prev_datatype = em->em_datatype;
- }
- }
-
- /* factor in the width of the values in this column */
- funcCost *= get_width_cost_multiplier(root, em->em_expr);
-
- /* now we have per-key cost, so add to the running total */
- totalFuncCost += funcCost;
-
- /* remember if we have found a fake Var in pathkeys */
- has_fake_var |= is_fake_var(em->em_expr);
- pathkeyExprs = lappend(pathkeyExprs, em->em_expr);
-
- /*
- * We need to calculate the number of comparisons for this column,
- * which requires knowing the group size. So we estimate the number of
- * groups by calling estimate_num_groups_incremental(), which
- * estimates the group size for "new" pathkeys.
- *
- * Note: estimate_num_groups_incremental does not handle fake Vars, so
- * use a default estimate otherwise.
- */
- if (!has_fake_var)
- nGroups = estimate_num_groups_incremental(root, pathkeyExprs,
- tuplesPerPrevGroup, NULL, NULL,
- &cache_varinfos,
- list_length(pathkeyExprs) - 1);
- else if (tuples > 4.0)
-
- /*
- * Use geometric mean as estimation if there are no stats.
- *
- * We don't use DEFAULT_NUM_DISTINCT here, because that's used for
- * a single column, but here we're dealing with multiple columns.
- */
- nGroups = ceil(2.0 + sqrt(tuples) * (i + 1) / list_length(pathkeys));
- else
- nGroups = tuples;
-
- /*
- * Presorted keys are not considered in the cost above, but we still
- * do have to compare them in the qsort comparator. So make sure to
- * factor in the cost in that case.
- */
- if (i >= nPresortedKeys)
- {
- if (heapSort)
- {
- /*
- * have to keep at least one group, and a multiple of group
- * size
- */
- correctedNGroups = ceil(output_tuples / tuplesPerPrevGroup);
- }
- else
- /* all groups in the input */
- correctedNGroups = nGroups;
-
- correctedNGroups = Max(1.0, ceil(correctedNGroups));
-
- per_tuple_cost += totalFuncCost * LOG2(correctedNGroups);
- }
-
- i++;
-
- /*
- * Uniform distributions with all groups being of the same size are
- * the best case, with nice smooth behavior. Real-world distributions
- * tend not to be uniform, though, and we don't have any reliable
- * easy-to-use information. As a basic defense against skewed
- * distributions, we use a 1.5 factor to make the expected group a bit
- * larger, but we need to be careful not to make the group larger than
- * in the preceding step.
- */
- tuplesPerPrevGroup = Min(tuplesPerPrevGroup,
- ceil(1.5 * tuplesPerPrevGroup / nGroups));
-
- /*
- * Once we get single-row group, it means tuples in the group are
- * unique and we can skip all remaining columns.
- */
- if (tuplesPerPrevGroup <= 1.0)
- break;
- }
-
- list_free(pathkeyExprs);
-
- /* per_tuple_cost is in cpu_operator_cost units */
- per_tuple_cost *= cpu_operator_cost;
-
- /*
- * Accordingly to "Introduction to algorithms", Thomas H. Cormen, Charles
- * E. Leiserson, Ronald L. Rivest, ISBN 0-07-013143-0, quicksort
- * estimation formula has additional term proportional to number of tuples
- * (see Chapter 8.2 and Theorem 4.1). That affects cases with a low number
- * of tuples, approximately less than 1e4. We could implement it as an
- * additional multiplier under the logarithm, but we use a bit more
- * complex formula which takes into account the number of unique tuples
- * and it's not clear how to combine the multiplier with the number of
- * groups. Estimate it as 10 cpu_operator_cost units.
- */
- per_tuple_cost += 10 * cpu_operator_cost;
-
- per_tuple_cost += comparison_cost;
-
- return tuples * per_tuple_cost;
-}
-
-/*
- * simple wrapper just to estimate best sort path
- */
-Cost
-cost_sort_estimate(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
- double tuples)
-{
- return compute_cpu_sort_cost(root, pathkeys, nPresortedKeys,
- 0, tuples, tuples, false);
-}
-
/*
* cost_tuplesort
* Determines and returns the cost of sorting a relation using tuplesort,
@@ -2133,7 +1812,7 @@ cost_sort_estimate(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* number of initial runs formed and M is the merge order used by tuplesort.c.
* Since the average initial run should be about sort_mem, we have
* disk traffic = 2 * relsize * ceil(logM(p / sort_mem))
- * and cpu cost (computed by compute_cpu_sort_cost()).
+ * cpu = comparison_cost * t * log2(t)
*
* If the sort is bounded (i.e., only the first k result tuples are needed)
* and k tuples can fit into sort_mem, we use a heap method that keeps only
@@ -2152,11 +1831,9 @@ cost_sort_estimate(PlannerInfo *root, List *pathkeys, int nPresortedKeys,
* 'comparison_cost' is the extra cost per comparison, if any
* 'sort_mem' is the number of kilobytes of work memory allowed for the sort
* 'limit_tuples' is the bound on the number of output tuples; -1 if no bound
- * 'startup_cost' is expected to be 0 at input. If there is "input cost" it should
- * be added by caller later
*/
static void
-cost_tuplesort(PlannerInfo *root, List *pathkeys, Cost *startup_cost, Cost *run_cost,
+cost_tuplesort(Cost *startup_cost, Cost *run_cost,
double tuples, int width,
Cost comparison_cost, int sort_mem,
double limit_tuples)
@@ -2173,6 +1850,9 @@ cost_tuplesort(PlannerInfo *root, List *pathkeys, Cost *startup_cost, Cost *run_
if (tuples < 2.0)
tuples = 2.0;
+ /* Include the default cost-per-comparison */
+ comparison_cost += 2.0 * cpu_operator_cost;
+
/* Do we have a useful LIMIT? */
if (limit_tuples > 0 && limit_tuples < tuples)
{
@@ -2196,10 +1876,12 @@ cost_tuplesort(PlannerInfo *root, List *pathkeys, Cost *startup_cost, Cost *run_
double log_runs;
double npageaccesses;
- /* CPU costs */
- *startup_cost = compute_cpu_sort_cost(root, pathkeys, 0,
- comparison_cost, tuples,
- tuples, false);
+ /*
+ * CPU costs
+ *
+ * Assume about N log2 N comparisons
+ */
+ *startup_cost = comparison_cost * tuples * LOG2(tuples);
/* Disk costs */
@@ -2215,17 +1897,18 @@ cost_tuplesort(PlannerInfo *root, List *pathkeys, Cost *startup_cost, Cost *run_
}
else if (tuples > 2 * output_tuples || input_bytes > sort_mem_bytes)
{
- /* We'll use a bounded heap-sort keeping just K tuples in memory. */
- *startup_cost = compute_cpu_sort_cost(root, pathkeys, 0,
- comparison_cost, tuples,
- output_tuples, true);
+ /*
+ * We'll use a bounded heap-sort keeping just K tuples in memory, for
+ * a total number of tuple comparisons of N log2 K; but the constant
+ * factor is a bit higher than for quicksort. Tweak it so that the
+ * cost curve is continuous at the crossover point.
+ */
+ *startup_cost = comparison_cost * tuples * LOG2(2.0 * output_tuples);
}
else
{
/* We'll use plain quicksort on all the input tuples */
- *startup_cost = compute_cpu_sort_cost(root, pathkeys, 0,
- comparison_cost, tuples,
- tuples, false);
+ *startup_cost = comparison_cost * tuples * LOG2(tuples);
}
/*
@@ -2258,8 +1941,8 @@ cost_incremental_sort(Path *path,
double input_tuples, int width, Cost comparison_cost, int sort_mem,
double limit_tuples)
{
- Cost startup_cost,
- run_cost,
+ Cost startup_cost = 0,
+ run_cost = 0,
input_run_cost = input_total_cost - input_startup_cost;
double group_tuples,
input_groups;
@@ -2344,7 +2027,7 @@ cost_incremental_sort(Path *path,
* pessimistic about incremental sort performance and increase its average
* group size by half.
*/
- cost_tuplesort(root, pathkeys, &group_startup_cost, &group_run_cost,
+ cost_tuplesort(&group_startup_cost, &group_run_cost,
1.5 * group_tuples, width, comparison_cost, sort_mem,
limit_tuples);
@@ -2352,7 +2035,7 @@ cost_incremental_sort(Path *path,
* Startup cost of incremental sort is the startup cost of its first group
* plus the cost of its input.
*/
- startup_cost = group_startup_cost
+ startup_cost += group_startup_cost
+ input_startup_cost + group_input_run_cost;
/*
@@ -2361,7 +2044,7 @@ cost_incremental_sort(Path *path,
* group, plus the total cost to process the remaining groups, plus the
* remaining cost of input.
*/
- run_cost = group_run_cost
+ run_cost += group_run_cost
+ (group_run_cost + group_startup_cost) * (input_groups - 1)
+ group_input_run_cost * (input_groups - 1);
@@ -2401,7 +2084,7 @@ cost_sort(Path *path, PlannerInfo *root,
Cost startup_cost;
Cost run_cost;
- cost_tuplesort(root, pathkeys, &startup_cost, &run_cost,
+ cost_tuplesort(&startup_cost, &run_cost,
tuples, width,
comparison_cost, sort_mem,
limit_tuples);
@@ -2499,7 +2182,7 @@ append_nonpartial_cost(List *subpaths, int numpaths, int parallel_workers)
* Determines and returns the cost of an Append node.
*/
void
-cost_append(AppendPath *apath, PlannerInfo *root)
+cost_append(AppendPath *apath)
{
ListCell *l;
@@ -2567,7 +2250,7 @@ cost_append(AppendPath *apath, PlannerInfo *root)
* any child.
*/
cost_sort(&sort_path,
- root,
+ NULL, /* doesn't currently need root */
pathkeys,
subpath->total_cost,
subpath->rows,
diff --git a/src/backend/optimizer/path/equivclass.c b/src/backend/optimizer/path/equivclass.c
index 7991295548..9f39f4661a 100644
--- a/src/backend/optimizer/path/equivclass.c
+++ b/src/backend/optimizer/path/equivclass.c
@@ -681,18 +681,7 @@ get_eclass_for_sort_expr(PlannerInfo *root,
if (opcintype == cur_em->em_datatype &&
equal(expr, cur_em->em_expr))
- {
- /*
- * Match!
- *
- * Copy the sortref if it wasn't set yet. That may happen if
- * the ec was constructed from WHERE clause, i.e. it doesn't
- * have a target reference at all.
- */
- if (cur_ec->ec_sortref == 0 && sortref > 0)
- cur_ec->ec_sortref = sortref;
- return cur_ec;
- }
+ return cur_ec; /* Match! */
}
}
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 1d21215f85..86a35cdef1 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -17,24 +17,17 @@
*/
#include "postgres.h"
-#include <float.h>
-
-#include "miscadmin.h"
#include "access/stratnum.h"
#include "catalog/pg_opfamily.h"
#include "nodes/makefuncs.h"
#include "nodes/nodeFuncs.h"
#include "nodes/plannodes.h"
-#include "optimizer/cost.h"
#include "optimizer/optimizer.h"
#include "optimizer/pathnode.h"
#include "optimizer/paths.h"
#include "partitioning/partbounds.h"
#include "utils/lsyscache.h"
-#include "utils/selfuncs.h"
-/* Consider reordering of GROUP BY keys? */
-bool enable_group_by_reordering = true;
static bool pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys);
static bool matches_boolean_partition_clause(RestrictInfo *rinfo,
@@ -341,527 +334,6 @@ pathkeys_contained_in(List *keys1, List *keys2)
return false;
}
-/*
- * group_keys_reorder_by_pathkeys
- * Reorder GROUP BY keys to match pathkeys of input path.
- *
- * Function returns new lists (pathkeys and clauses), original GROUP BY lists
- * stay untouched.
- *
- * Returns the number of GROUP BY keys with a matching pathkey.
- */
-int
-group_keys_reorder_by_pathkeys(List *pathkeys, List **group_pathkeys,
- List **group_clauses)
-{
- List *new_group_pathkeys = NIL,
- *new_group_clauses = NIL;
- ListCell *lc;
- int n;
-
- if (pathkeys == NIL || *group_pathkeys == NIL)
- return 0;
-
- /*
- * Walk the pathkeys (determining ordering of the input path) and see if
- * there's a matching GROUP BY key. If we find one, we append it to the
- * list, and do the same for the clauses.
- *
- * Once we find the first pathkey without a matching GROUP BY key, the
- * rest of the pathkeys are useless and can't be used to evaluate the
- * grouping, so we abort the loop and ignore the remaining pathkeys.
- *
- * XXX Pathkeys are built in a way to allow simply comparing pointers.
- */
- foreach(lc, pathkeys)
- {
- PathKey *pathkey = (PathKey *) lfirst(lc);
- SortGroupClause *sgc;
-
- /* abort on first mismatch */
- if (!list_member_ptr(*group_pathkeys, pathkey))
- break;
-
- new_group_pathkeys = lappend(new_group_pathkeys, pathkey);
-
- sgc = get_sortgroupref_clause(pathkey->pk_eclass->ec_sortref,
- *group_clauses);
-
- new_group_clauses = lappend(new_group_clauses, sgc);
- }
-
- /* remember the number of pathkeys with a matching GROUP BY key */
- n = list_length(new_group_pathkeys);
-
- /* append the remaining group pathkeys (will be treated as not sorted) */
- *group_pathkeys = list_concat_unique_ptr(new_group_pathkeys,
- *group_pathkeys);
- *group_clauses = list_concat_unique_ptr(new_group_clauses,
- *group_clauses);
-
- return n;
-}
-
-/*
- * Used to generate all permutations of a pathkey list.
- */
-typedef struct PathkeyMutatorState
-{
- List *elemsList;
- ListCell **elemCells;
- void **elems;
- int *positions;
- int mutatorNColumns;
- int count;
-} PathkeyMutatorState;
-
-
-/*
- * PathkeyMutatorInit
- * Initialize state of the permutation generator.
- *
- * We want to generate permutations of elements in the "elems" list. We may want
- * to skip some number of elements at the beginning (when treating as presorted)
- * or at the end (we only permute a limited number of group keys).
- *
- * The list is decomposed into elements, and we also keep pointers to individual
- * cells. This allows us to build the permuted list quickly and cheaply, without
- * creating any copies.
- */
-static void
-PathkeyMutatorInit(PathkeyMutatorState *state, List *elems, int start, int end)
-{
- int i;
- int n = end - start;
- ListCell *lc;
-
- memset(state, 0, sizeof(*state));
-
- state->mutatorNColumns = n;
-
- state->elemsList = list_copy(elems);
-
- state->elems = palloc(sizeof(void *) * n);
- state->elemCells = palloc(sizeof(ListCell *) * n);
- state->positions = palloc(sizeof(int) * n);
-
- i = 0;
- for_each_cell(lc, state->elemsList, list_nth_cell(state->elemsList, start))
- {
- state->elemCells[i] = lc;
- state->elems[i] = lfirst(lc);
- state->positions[i] = i + 1;
- i++;
-
- if (i >= n)
- break;
- }
-}
-
-/* Swap two elements of an array. */
-static void
-PathkeyMutatorSwap(int *a, int i, int j)
-{
- int s = a[i];
-
- a[i] = a[j];
- a[j] = s;
-}
-
-/*
- * Generate the next permutation of elements.
- */
-static bool
-PathkeyMutatorNextSet(int *a, int n)
-{
- int j,
- k,
- l,
- r;
-
- j = n - 2;
-
- while (j >= 0 && a[j] >= a[j + 1])
- j--;
-
- if (j < 0)
- return false;
-
- k = n - 1;
-
- while (k >= 0 && a[j] >= a[k])
- k--;
-
- PathkeyMutatorSwap(a, j, k);
-
- l = j + 1;
- r = n - 1;
-
- while (l < r)
- PathkeyMutatorSwap(a, l++, r--);
-
- return true;
-}
-
-/*
- * PathkeyMutatorNext
- * Generate the next permutation of list of elements.
- *
- * Returns the next permutation (as a list of elements) or NIL if there are no
- * more permutations.
- */
-static List *
-PathkeyMutatorNext(PathkeyMutatorState *state)
-{
- int i;
-
- state->count++;
-
- /* first permutation is original list */
- if (state->count == 1)
- return state->elemsList;
-
- /* when there are no more permutations, return NIL */
- if (!PathkeyMutatorNextSet(state->positions, state->mutatorNColumns))
- {
- pfree(state->elems);
- pfree(state->elemCells);
- pfree(state->positions);
-
- list_free(state->elemsList);
-
- return NIL;
- }
-
- /* update the list cells to point to the right elements */
- for (i = 0; i < state->mutatorNColumns; i++)
- lfirst(state->elemCells[i]) =
- (void *) state->elems[state->positions[i] - 1];
-
- return state->elemsList;
-}
-
-/*
- * Cost of comparing pathkeys.
- */
-typedef struct PathkeySortCost
-{
- Cost cost;
- PathKey *pathkey;
-} PathkeySortCost;
-
-static int
-pathkey_sort_cost_comparator(const void *_a, const void *_b)
-{
- const PathkeySortCost *a = (PathkeySortCost *) _a;
- const PathkeySortCost *b = (PathkeySortCost *) _b;
-
- if (a->cost < b->cost)
- return -1;
- else if (a->cost == b->cost)
- return 0;
- return 1;
-}
-
-/*
- * get_cheapest_group_keys_order
- * Reorders the group pathkeys / clauses to minimize the comparison cost.
- *
- * Given the list of pathkeys in '*group_pathkeys', we try to arrange these
- * in an order that minimizes the sort costs that will be incurred by the
- * GROUP BY. The costs mainly depend on the cost of the sort comparator
- * function(s) and the number of distinct values in each column of the GROUP
- * BY clause (*group_clauses). Sorting on subsequent columns is only required
- * for tiebreak situations where two values sort equally.
- *
- * In case the input is partially sorted, only the remaining pathkeys are
- * considered. 'n_preordered' denotes how many of the leading *group_pathkeys
- * the input is presorted by.
- *
- * Returns true and sets *group_pathkeys and *group_clauses to the newly
- * ordered versions of the lists that were passed in via these parameters.
- * If no reordering was deemed necessary then we return false, in which case
- * the *group_pathkeys and *group_clauses lists are left untouched. The
- * original *group_pathkeys and *group_clauses parameter values are never
- * destructively modified in place.
- */
-static bool
-get_cheapest_group_keys_order(PlannerInfo *root, double nrows,
- List **group_pathkeys, List **group_clauses,
- int n_preordered)
-{
- List *new_group_pathkeys = NIL,
- *new_group_clauses = NIL,
- *var_group_pathkeys;
-
- ListCell *cell;
- PathkeyMutatorState mstate;
- double cheapest_sort_cost = DBL_MAX;
-
- int nFreeKeys;
- int nToPermute;
-
- /* If there are less than 2 unsorted pathkeys, we're done. */
- if (list_length(*group_pathkeys) - n_preordered < 2)
- return false;
-
- /*
- * We could exhaustively cost all possible orderings of the pathkeys, but
- * for a large number of pathkeys it might be prohibitively expensive. So
- * we try to apply simple cheap heuristics first - we sort the pathkeys by
- * sort cost (as if the pathkey was sorted independently) and then check
- * only the four cheapest pathkeys. The remaining pathkeys are kept
- * ordered by cost.
- *
- * XXX This is a very simple heuristics, but likely to work fine for most
- * cases (because the number of GROUP BY clauses tends to be lower than
- * 4). But it ignores how the number of distinct values in each pathkey
- * affects the following steps. It might be better to use "more expensive"
- * pathkey first if it has many distinct values, because it then limits
- * the number of comparisons for the remaining pathkeys. But evaluating
- * that is likely quite the expensive.
- */
- nFreeKeys = list_length(*group_pathkeys) - n_preordered;
- nToPermute = 4;
- if (nFreeKeys > nToPermute)
- {
- PathkeySortCost *costs = palloc(sizeof(PathkeySortCost) * nFreeKeys);
- PathkeySortCost *cost = costs;
-
- /*
- * Estimate cost for sorting individual pathkeys skipping the
- * pre-ordered pathkeys.
- */
- for_each_from(cell, *group_pathkeys, n_preordered)
- {
- PathKey *pathkey = (PathKey *) lfirst(cell);
- List *to_cost = list_make1(pathkey);
-
- cost->pathkey = pathkey;
- cost->cost = cost_sort_estimate(root, to_cost, 0, nrows);
- cost++;
-
- list_free(to_cost);
- }
-
- /* sort the pathkeys by sort cost in ascending order */
- qsort(costs, nFreeKeys, sizeof(*costs), pathkey_sort_cost_comparator);
-
- /*
- * Rebuild the list of pathkeys - first the preordered ones, then the
- * rest ordered by cost.
- */
- new_group_pathkeys = list_copy_head(*group_pathkeys, n_preordered);
-
- for (int i = 0; i < nFreeKeys; i++)
- new_group_pathkeys = lappend(new_group_pathkeys, costs[i].pathkey);
-
- pfree(costs);
- }
- else
- {
- /* Copy the list, so that we can free the new list by list_free. */
- new_group_pathkeys = list_copy(*group_pathkeys);
- nToPermute = nFreeKeys;
- }
-
- Assert(list_length(new_group_pathkeys) == list_length(*group_pathkeys));
-
- /*
- * Generate pathkey lists with permutations of the first nToPermute
- * pathkeys.
- *
- * XXX We simply calculate sort cost for each individual pathkey list, but
- * there's room for two dynamic programming optimizations here. Firstly,
- * we may pass the current "best" cost to cost_sort_estimate so that it
- * can "abort" if the estimated pathkeys list exceeds it. Secondly, it
- * could pass the return information about the position when it exceeded
- * the cost, and we could skip all permutations with the same prefix.
- *
- * Imagine we've already found ordering with cost C1, and we're evaluating
- * another ordering - cost_sort_estimate() calculates cost by adding the
- * pathkeys one by one (more or less), and the cost only grows. If at any
- * point it exceeds C1, it can't possibly be "better" so we can discard
- * it. But we also know that we can discard all ordering with the same
- * prefix, because if we're estimating (a,b,c,d) and we exceed C1 at (a,b)
- * then the same thing will happen for any ordering with this prefix.
- */
- PathkeyMutatorInit(&mstate, new_group_pathkeys, n_preordered, n_preordered + nToPermute);
-
- while ((var_group_pathkeys = PathkeyMutatorNext(&mstate)) != NIL)
- {
- Cost cost;
-
- cost = cost_sort_estimate(root, var_group_pathkeys, n_preordered, nrows);
-
- if (cost < cheapest_sort_cost)
- {
- list_free(new_group_pathkeys);
- new_group_pathkeys = list_copy(var_group_pathkeys);
- cheapest_sort_cost = cost;
- }
- }
-
- /* Reorder the group clauses according to the reordered pathkeys. */
- foreach(cell, new_group_pathkeys)
- {
- PathKey *pathkey = (PathKey *) lfirst(cell);
-
- new_group_clauses = lappend(new_group_clauses,
- get_sortgroupref_clause(pathkey->pk_eclass->ec_sortref,
- *group_clauses));
- }
-
- /* Just append the rest GROUP BY clauses */
- new_group_clauses = list_concat_unique_ptr(new_group_clauses,
- *group_clauses);
-
- *group_pathkeys = new_group_pathkeys;
- *group_clauses = new_group_clauses;
-
- return true;
-}
-
-/*
- * get_useful_group_keys_orderings
- * Determine which orderings of GROUP BY keys are potentially interesting.
- *
- * Returns list of PathKeyInfo items, each representing an interesting ordering
- * of GROUP BY keys. Each item stores pathkeys and clauses in matching order.
- *
- * The function considers (and keeps) multiple group by orderings:
- *
- * - the original ordering, as specified by the GROUP BY clause
- *
- * - GROUP BY keys reordered to minimize the sort cost
- *
- * - GROUP BY keys reordered to match path ordering (as much as possible), with
- * the tail reordered to minimize the sort cost
- *
- * - GROUP BY keys to match target ORDER BY clause (as much as possible), with
- * the tail reordered to minimize the sort cost
- *
- * There are other potentially interesting orderings (e.g. it might be best to
- * match the first ORDER BY key, order the remaining keys differently and then
- * rely on the incremental sort to fix this), but we ignore those for now. To
- * make this work we'd have to pretty much generate all possible permutations.
- */
-List *
-get_useful_group_keys_orderings(PlannerInfo *root, double nrows,
- List *path_pathkeys,
- List *group_pathkeys, List *group_clauses)
-{
- Query *parse = root->parse;
- List *infos = NIL;
- PathKeyInfo *info;
- int n_preordered = 0;
-
- List *pathkeys = group_pathkeys;
- List *clauses = group_clauses;
-
- /* always return at least the original pathkeys/clauses */
- info = makeNode(PathKeyInfo);
- info->pathkeys = pathkeys;
- info->clauses = clauses;
-
- infos = lappend(infos, info);
-
- /*
- * Should we try generating alternative orderings of the group keys? If
- * not, we produce only the order specified in the query, i.e. the
- * optimization is effectively disabled.
- */
- if (!enable_group_by_reordering)
- return infos;
-
- /* for grouping sets we can't do any reordering */
- if (parse->groupingSets)
- return infos;
-
- /*
- * Try reordering pathkeys to minimize the sort cost, ignoring both the
- * target ordering (ORDER BY) and ordering of the input path.
- */
- if (get_cheapest_group_keys_order(root, nrows, &pathkeys, &clauses,
- n_preordered))
- {
- info = makeNode(PathKeyInfo);
- info->pathkeys = pathkeys;
- info->clauses = clauses;
-
- infos = lappend(infos, info);
- }
-
- /*
- * If the path is sorted in some way, try reordering the group keys to
- * match as much of the ordering as possible - we get this sort for free
- * (mostly).
- *
- * We must not do this when there are no grouping sets, because those use
- * more complex logic to decide the ordering.
- *
- * XXX Isn't this somewhat redundant with presorted_keys? Actually, it's
- * more a complement, because it allows benefiting from incremental sort
- * as much as possible.
- *
- * XXX This does nothing if (n_preordered == 0). We shouldn't create the
- * info in this case.
- */
- if (path_pathkeys)
- {
- n_preordered = group_keys_reorder_by_pathkeys(path_pathkeys,
- &pathkeys,
- &clauses);
-
- /* reorder the tail to minimize sort cost */
- get_cheapest_group_keys_order(root, nrows, &pathkeys, &clauses,
- n_preordered);
-
- /*
- * reorder the tail to minimize sort cost
- *
- * XXX Ignore the return value - there may be nothing to reorder, in
- * which case get_cheapest_group_keys_order returns false. But we
- * still want to keep the keys reordered to path_pathkeys.
- */
- info = makeNode(PathKeyInfo);
- info->pathkeys = pathkeys;
- info->clauses = clauses;
-
- infos = lappend(infos, info);
- }
-
- /*
- * Try reordering pathkeys to minimize the sort cost (this time consider
- * the ORDER BY clause, but only if set debug_group_by_match_order_by).
- */
- if (root->sort_pathkeys)
- {
- n_preordered = group_keys_reorder_by_pathkeys(root->sort_pathkeys,
- &pathkeys,
- &clauses);
-
- /*
- * reorder the tail to minimize sort cost
- *
- * XXX Ignore the return value - there may be nothing to reorder, in
- * which case get_cheapest_group_keys_order returns false. But we
- * still want to keep the keys reordered to sort_pathkeys.
- */
- get_cheapest_group_keys_order(root, nrows, &pathkeys, &clauses,
- n_preordered);
-
- /* keep the group keys reordered to match ordering of input path */
- info = makeNode(PathKeyInfo);
- info->pathkeys = pathkeys;
- info->clauses = clauses;
-
- infos = lappend(infos, info);
- }
-
- return infos;
-}
-
/*
* pathkeys_count_contained_in
* Same as pathkeys_contained_in, but also sets length of longest
@@ -2390,54 +1862,6 @@ pathkeys_useful_for_ordering(PlannerInfo *root, List *pathkeys)
return n_common_pathkeys;
}
-/*
- * pathkeys_useful_for_grouping
- * Count the number of pathkeys that are useful for grouping (instead of
- * explicit sort)
- *
- * Group pathkeys could be reordered to benefit from the ordering. The
- * ordering may not be "complete" and may require incremental sort, but that's
- * fine. So we simply count prefix pathkeys with a matching group key, and
- * stop once we find the first pathkey without a match.
- *
- * So e.g. with pathkeys (a,b,c) and group keys (a,b,e) this determines (a,b)
- * pathkeys are useful for grouping, and we might do incremental sort to get
- * path ordered by (a,b,e).
- *
- * This logic is necessary to retain paths with ordering not matching grouping
- * keys directly, without the reordering.
- *
- * Returns the length of pathkey prefix with matching group keys.
- */
-static int
-pathkeys_useful_for_grouping(PlannerInfo *root, List *pathkeys)
-{
- ListCell *key;
- int n = 0;
-
- /* no special ordering requested for grouping */
- if (root->group_pathkeys == NIL)
- return 0;
-
- /* unordered path */
- if (pathkeys == NIL)
- return 0;
-
- /* walk the pathkeys and search for matching group key */
- foreach(key, pathkeys)
- {
- PathKey *pathkey = (PathKey *) lfirst(key);
-
- /* no matching group key, we're done */
- if (!list_member_ptr(root->group_pathkeys, pathkey))
- break;
-
- n++;
- }
-
- return n;
-}
-
/*
* truncate_useless_pathkeys
* Shorten the given pathkey list to just the useful pathkeys.
@@ -2452,9 +1876,6 @@ truncate_useless_pathkeys(PlannerInfo *root,
nuseful = pathkeys_useful_for_merging(root, rel, pathkeys);
nuseful2 = pathkeys_useful_for_ordering(root, pathkeys);
- if (nuseful2 > nuseful)
- nuseful = nuseful2;
- nuseful2 = pathkeys_useful_for_grouping(root, pathkeys);
if (nuseful2 > nuseful)
nuseful = nuseful2;
@@ -2490,8 +1911,6 @@ has_useful_pathkeys(PlannerInfo *root, RelOptInfo *rel)
{
if (rel->joininfo != NIL || rel->has_eclass_joins)
return true; /* might be able to use pathkeys for merging */
- if (root->group_pathkeys != NIL)
- return true; /* might be able to use pathkeys for grouping */
if (root->query_pathkeys != NIL)
return true; /* might be able to use them for ordering */
return false; /* definitely useless */
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index d8e8f607b2..468105d91e 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -2756,9 +2756,8 @@ remove_useless_groupby_columns(PlannerInfo *root)
*
* In principle it might be interesting to consider other orderings of the
* GROUP BY elements, which could match the sort ordering of other
- * possible plans (eg an indexscan) and thereby reduce cost. However, we
- * don't yet have sufficient information to do that here, so that's left until
- * later in planning. See get_useful_group_keys_orderings().
+ * possible plans (eg an indexscan) and thereby reduce cost. We don't
+ * bother with that, though. Hashed grouping will frequently win anyway.
*
* Note: we need no comparable processing of the distinctClause because
* the parser already enforced that that matches ORDER BY.
@@ -6232,122 +6231,24 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
*/
foreach(lc, input_rel->pathlist)
{
- ListCell *lc2;
Path *path = (Path *) lfirst(lc);
Path *path_original = path;
+ bool is_sorted;
+ int presorted_keys;
- List *pathkey_orderings = NIL;
-
- List *group_pathkeys = root->group_pathkeys;
- List *group_clauses = parse->groupClause;
-
- /* generate alternative group orderings that might be useful */
- pathkey_orderings = get_useful_group_keys_orderings(root,
- path->rows,
- path->pathkeys,
- group_pathkeys,
- group_clauses);
+ is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
+ path->pathkeys,
+ &presorted_keys);
- Assert(list_length(pathkey_orderings) > 0);
-
- /* process all potentially interesting grouping reorderings */
- foreach(lc2, pathkey_orderings)
+ if (path == cheapest_path || is_sorted)
{
- bool is_sorted;
- int presorted_keys = 0;
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
-
- /* restore the path (we replace it in the loop) */
- path = path_original;
-
- is_sorted = pathkeys_count_contained_in(info->pathkeys,
- path->pathkeys,
- &presorted_keys);
-
- if (path == cheapest_path || is_sorted)
- {
- /* Sort the cheapest-total path if it isn't already sorted */
- if (!is_sorted)
- path = (Path *) create_sort_path(root,
- grouped_rel,
- path,
- info->pathkeys,
- -1.0);
-
- /* Now decide what to stick atop it */
- if (parse->groupingSets)
- {
- consider_groupingsets_paths(root, grouped_rel,
- path, true, can_hash,
- gd, agg_costs, dNumGroups);
- }
- else if (parse->hasAggs)
- {
- /*
- * We have aggregation, possibly with plain GROUP BY.
- * Make an AggPath.
- */
- add_path(grouped_rel, (Path *)
- create_agg_path(root,
- grouped_rel,
- path,
- grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_SIMPLE,
- info->clauses,
- havingQual,
- agg_costs,
- dNumGroups));
- }
- else if (group_clauses)
- {
- /*
- * We have GROUP BY without aggregation or grouping
- * sets. Make a GroupPath.
- */
- add_path(grouped_rel, (Path *)
- create_group_path(root,
- grouped_rel,
- path,
- info->clauses,
- havingQual,
- dNumGroups));
- }
- else
- {
- /* Other cases should have been handled above */
- Assert(false);
- }
- }
-
- /*
- * Now we may consider incremental sort on this path, but only
- * when the path is not already sorted and when incremental
- * sort is enabled.
- */
- if (is_sorted || !enable_incremental_sort)
- continue;
-
- /* Restore the input path (we might have added Sort on top). */
- path = path_original;
-
- /* no shared prefix, no point in building incremental sort */
- if (presorted_keys == 0)
- continue;
-
- /*
- * We should have already excluded pathkeys of length 1
- * because then presorted_keys > 0 would imply is_sorted was
- * true.
- */
- Assert(list_length(root->group_pathkeys) != 1);
-
- path = (Path *) create_incremental_sort_path(root,
- grouped_rel,
- path,
- info->pathkeys,
- presorted_keys,
- -1.0);
+ /* Sort the cheapest-total path if it isn't already sorted */
+ if (!is_sorted)
+ path = (Path *) create_sort_path(root,
+ grouped_rel,
+ path,
+ root->group_pathkeys,
+ -1.0);
/* Now decide what to stick atop it */
if (parse->groupingSets)
@@ -6367,9 +6268,9 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
grouped_rel,
path,
grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
AGGSPLIT_SIMPLE,
- info->clauses,
+ parse->groupClause,
havingQual,
agg_costs,
dNumGroups));
@@ -6384,7 +6285,7 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
create_group_path(root,
grouped_rel,
path,
- info->clauses,
+ parse->groupClause,
havingQual,
dNumGroups));
}
@@ -6394,6 +6295,79 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
Assert(false);
}
}
+
+ /*
+ * Now we may consider incremental sort on this path, but only
+ * when the path is not already sorted and when incremental sort
+ * is enabled.
+ */
+ if (is_sorted || !enable_incremental_sort)
+ continue;
+
+ /* Restore the input path (we might have added Sort on top). */
+ path = path_original;
+
+ /* no shared prefix, no point in building incremental sort */
+ if (presorted_keys == 0)
+ continue;
+
+ /*
+ * We should have already excluded pathkeys of length 1 because
+ * then presorted_keys > 0 would imply is_sorted was true.
+ */
+ Assert(list_length(root->group_pathkeys) != 1);
+
+ path = (Path *) create_incremental_sort_path(root,
+ grouped_rel,
+ path,
+ root->group_pathkeys,
+ presorted_keys,
+ -1.0);
+
+ /* Now decide what to stick atop it */
+ if (parse->groupingSets)
+ {
+ consider_groupingsets_paths(root, grouped_rel,
+ path, true, can_hash,
+ gd, agg_costs, dNumGroups);
+ }
+ else if (parse->hasAggs)
+ {
+ /*
+ * We have aggregation, possibly with plain GROUP BY. Make an
+ * AggPath.
+ */
+ add_path(grouped_rel, (Path *)
+ create_agg_path(root,
+ grouped_rel,
+ path,
+ grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_SIMPLE,
+ parse->groupClause,
+ havingQual,
+ agg_costs,
+ dNumGroups));
+ }
+ else if (parse->groupClause)
+ {
+ /*
+ * We have GROUP BY without aggregation or grouping sets. Make
+ * a GroupPath.
+ */
+ add_path(grouped_rel, (Path *)
+ create_group_path(root,
+ grouped_rel,
+ path,
+ parse->groupClause,
+ havingQual,
+ dNumGroups));
+ }
+ else
+ {
+ /* Other cases should have been handled above */
+ Assert(false);
+ }
}
/*
@@ -6404,130 +6378,100 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel,
{
foreach(lc, partially_grouped_rel->pathlist)
{
- ListCell *lc2;
Path *path = (Path *) lfirst(lc);
Path *path_original = path;
+ bool is_sorted;
+ int presorted_keys;
- List *pathkey_orderings = NIL;
-
- List *group_pathkeys = root->group_pathkeys;
- List *group_clauses = parse->groupClause;
-
- /* generate alternative group orderings that might be useful */
- pathkey_orderings = get_useful_group_keys_orderings(root,
- path->rows,
- path->pathkeys,
- group_pathkeys,
- group_clauses);
-
- Assert(list_length(pathkey_orderings) > 0);
+ is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
+ path->pathkeys,
+ &presorted_keys);
- /* process all potentially interesting grouping reorderings */
- foreach(lc2, pathkey_orderings)
+ /*
+ * Insert a Sort node, if required. But there's no point in
+ * sorting anything but the cheapest path.
+ */
+ if (!is_sorted)
{
- bool is_sorted;
- int presorted_keys = 0;
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
-
- /* restore the path (we replace it in the loop) */
- path = path_original;
-
- is_sorted = pathkeys_count_contained_in(info->pathkeys,
- path->pathkeys,
- &presorted_keys);
-
- /*
- * Insert a Sort node, if required. But there's no point
- * in sorting anything but the cheapest path.
- */
- if (!is_sorted)
- {
- if (path != partially_grouped_rel->cheapest_total_path)
- continue;
- path = (Path *) create_sort_path(root,
- grouped_rel,
- path,
- info->pathkeys,
- -1.0);
- }
+ if (path != partially_grouped_rel->cheapest_total_path)
+ continue;
+ path = (Path *) create_sort_path(root,
+ grouped_rel,
+ path,
+ root->group_pathkeys,
+ -1.0);
+ }
- if (parse->hasAggs)
- add_path(grouped_rel, (Path *)
- create_agg_path(root,
- grouped_rel,
- path,
- grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_FINAL_DESERIAL,
- info->clauses,
- havingQual,
- agg_final_costs,
- dNumGroups));
- else
- add_path(grouped_rel, (Path *)
- create_group_path(root,
- grouped_rel,
- path,
- info->clauses,
- havingQual,
- dNumGroups));
+ if (parse->hasAggs)
+ add_path(grouped_rel, (Path *)
+ create_agg_path(root,
+ grouped_rel,
+ path,
+ grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_FINAL_DESERIAL,
+ parse->groupClause,
+ havingQual,
+ agg_final_costs,
+ dNumGroups));
+ else
+ add_path(grouped_rel, (Path *)
+ create_group_path(root,
+ grouped_rel,
+ path,
+ parse->groupClause,
+ havingQual,
+ dNumGroups));
- /*
- * Now we may consider incremental sort on this path, but
- * only when the path is not already sorted and when
- * incremental sort is enabled.
- */
- if (is_sorted || !enable_incremental_sort)
- continue;
+ /*
+ * Now we may consider incremental sort on this path, but only
+ * when the path is not already sorted and when incremental
+ * sort is enabled.
+ */
+ if (is_sorted || !enable_incremental_sort)
+ continue;
- /*
- * Restore the input path (we might have added Sort on
- * top).
- */
- path = path_original;
+ /* Restore the input path (we might have added Sort on top). */
+ path = path_original;
- /*
- * no shared prefix, not point in building incremental
- * sort
- */
- if (presorted_keys == 0)
- continue;
+ /* no shared prefix, not point in building incremental sort */
+ if (presorted_keys == 0)
+ continue;
- /*
- * We should have already excluded pathkeys of length 1
- * because then presorted_keys > 0 would imply is_sorted
- * was true.
- */
- Assert(list_length(root->group_pathkeys) != 1);
+ /*
+ * We should have already excluded pathkeys of length 1
+ * because then presorted_keys > 0 would imply is_sorted was
+ * true.
+ */
+ Assert(list_length(root->group_pathkeys) != 1);
- path = (Path *) create_incremental_sort_path(root,
- grouped_rel,
- path,
- info->pathkeys,
- presorted_keys,
- -1.0);
+ path = (Path *) create_incremental_sort_path(root,
+ grouped_rel,
+ path,
+ root->group_pathkeys,
+ presorted_keys,
+ -1.0);
- if (parse->hasAggs)
- add_path(grouped_rel, (Path *)
- create_agg_path(root,
- grouped_rel,
- path,
- grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_FINAL_DESERIAL,
- info->clauses,
- havingQual,
- agg_final_costs,
- dNumGroups));
- else
- add_path(grouped_rel, (Path *)
- create_group_path(root,
- grouped_rel,
- path,
- info->clauses,
- havingQual,
- dNumGroups));
- }
+ if (parse->hasAggs)
+ add_path(grouped_rel, (Path *)
+ create_agg_path(root,
+ grouped_rel,
+ path,
+ grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_FINAL_DESERIAL,
+ parse->groupClause,
+ havingQual,
+ agg_final_costs,
+ dNumGroups));
+ else
+ add_path(grouped_rel, (Path *)
+ create_group_path(root,
+ grouped_rel,
+ path,
+ parse->groupClause,
+ havingQual,
+ dNumGroups));
}
}
}
@@ -6730,71 +6674,41 @@ create_partial_grouping_paths(PlannerInfo *root,
*/
foreach(lc, input_rel->pathlist)
{
- ListCell *lc2;
Path *path = (Path *) lfirst(lc);
- Path *path_save = path;
-
- List *pathkey_orderings = NIL;
-
- List *group_pathkeys = root->group_pathkeys;
- List *group_clauses = parse->groupClause;
+ bool is_sorted;
- /* generate alternative group orderings that might be useful */
- pathkey_orderings = get_useful_group_keys_orderings(root,
- path->rows,
- path->pathkeys,
- group_pathkeys,
- group_clauses);
-
- Assert(list_length(pathkey_orderings) > 0);
-
- /* process all potentially interesting grouping reorderings */
- foreach(lc2, pathkey_orderings)
+ is_sorted = pathkeys_contained_in(root->group_pathkeys,
+ path->pathkeys);
+ if (path == cheapest_total_path || is_sorted)
{
- bool is_sorted;
- int presorted_keys = 0;
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
-
- /* restore the path (we replace it in the loop) */
- path = path_save;
-
- is_sorted = pathkeys_count_contained_in(info->pathkeys,
- path->pathkeys,
- &presorted_keys);
-
- if (path == cheapest_total_path || is_sorted)
- {
- /* Sort the cheapest partial path, if it isn't already */
- if (!is_sorted)
- {
- path = (Path *) create_sort_path(root,
- partially_grouped_rel,
- path,
- info->pathkeys,
- -1.0);
- }
+ /* Sort the cheapest partial path, if it isn't already */
+ if (!is_sorted)
+ path = (Path *) create_sort_path(root,
+ partially_grouped_rel,
+ path,
+ root->group_pathkeys,
+ -1.0);
- if (parse->hasAggs)
- add_path(partially_grouped_rel, (Path *)
- create_agg_path(root,
- partially_grouped_rel,
- path,
- partially_grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_INITIAL_SERIAL,
- info->clauses,
- NIL,
- agg_partial_costs,
- dNumPartialGroups));
- else
- add_path(partially_grouped_rel, (Path *)
- create_group_path(root,
- partially_grouped_rel,
- path,
- info->clauses,
- NIL,
- dNumPartialGroups));
- }
+ if (parse->hasAggs)
+ add_path(partially_grouped_rel, (Path *)
+ create_agg_path(root,
+ partially_grouped_rel,
+ path,
+ partially_grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_INITIAL_SERIAL,
+ parse->groupClause,
+ NIL,
+ agg_partial_costs,
+ dNumPartialGroups));
+ else
+ add_path(partially_grouped_rel, (Path *)
+ create_group_path(root,
+ partially_grouped_rel,
+ path,
+ parse->groupClause,
+ NIL,
+ dNumPartialGroups));
}
}
@@ -6804,8 +6718,6 @@ create_partial_grouping_paths(PlannerInfo *root,
* We can also skip the entire loop when we only have a single-item
* group_pathkeys because then we can't possibly have a presorted
* prefix of the list without having the list be fully sorted.
- *
- * XXX Shouldn't this also consider the group-key-reordering?
*/
if (enable_incremental_sort && list_length(root->group_pathkeys) > 1)
{
@@ -6863,101 +6775,24 @@ create_partial_grouping_paths(PlannerInfo *root,
/* Similar to above logic, but for partial paths. */
foreach(lc, input_rel->partial_pathlist)
{
- ListCell *lc2;
Path *path = (Path *) lfirst(lc);
Path *path_original = path;
+ bool is_sorted;
+ int presorted_keys;
- List *pathkey_orderings = NIL;
-
- List *group_pathkeys = root->group_pathkeys;
- List *group_clauses = parse->groupClause;
+ is_sorted = pathkeys_count_contained_in(root->group_pathkeys,
+ path->pathkeys,
+ &presorted_keys);
- /* generate alternative group orderings that might be useful */
- pathkey_orderings = get_useful_group_keys_orderings(root,
- path->rows,
- path->pathkeys,
- group_pathkeys,
- group_clauses);
-
- Assert(list_length(pathkey_orderings) > 0);
-
- /* process all potentially interesting grouping reorderings */
- foreach(lc2, pathkey_orderings)
+ if (path == cheapest_partial_path || is_sorted)
{
- bool is_sorted;
- int presorted_keys = 0;
- PathKeyInfo *info = (PathKeyInfo *) lfirst(lc2);
-
- /* restore the path (we replace it in the loop) */
- path = path_original;
-
- is_sorted = pathkeys_count_contained_in(info->pathkeys,
- path->pathkeys,
- &presorted_keys);
-
- if (path == cheapest_partial_path || is_sorted)
- {
-
- /* Sort the cheapest partial path, if it isn't already */
- if (!is_sorted)
- {
- path = (Path *) create_sort_path(root,
- partially_grouped_rel,
- path,
- info->pathkeys,
- -1.0);
- }
-
- if (parse->hasAggs)
- add_partial_path(partially_grouped_rel, (Path *)
- create_agg_path(root,
- partially_grouped_rel,
- path,
- partially_grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
- AGGSPLIT_INITIAL_SERIAL,
- info->clauses,
- NIL,
- agg_partial_costs,
- dNumPartialPartialGroups));
- else
- add_partial_path(partially_grouped_rel, (Path *)
- create_group_path(root,
- partially_grouped_rel,
- path,
- info->clauses,
- NIL,
- dNumPartialPartialGroups));
- }
-
- /*
- * Now we may consider incremental sort on this path, but only
- * when the path is not already sorted and when incremental
- * sort is enabled.
- */
- if (is_sorted || !enable_incremental_sort)
- continue;
-
- /* Restore the input path (we might have added Sort on top). */
- path = path_original;
-
- /* no shared prefix, not point in building incremental sort */
- if (presorted_keys == 0)
- continue;
-
- /*
- * We should have already excluded pathkeys of length 1
- * because then presorted_keys > 0 would imply is_sorted was
- * true.
- */
- Assert(list_length(root->group_pathkeys) != 1);
-
- path = (Path *) create_incremental_sort_path(root,
- partially_grouped_rel,
- path,
- info->pathkeys,
- presorted_keys,
- -1.0);
+ /* Sort the cheapest partial path, if it isn't already */
+ if (!is_sorted)
+ path = (Path *) create_sort_path(root,
+ partially_grouped_rel,
+ path,
+ root->group_pathkeys,
+ -1.0);
if (parse->hasAggs)
add_partial_path(partially_grouped_rel, (Path *)
@@ -6965,9 +6800,9 @@ create_partial_grouping_paths(PlannerInfo *root,
partially_grouped_rel,
path,
partially_grouped_rel->reltarget,
- info->clauses ? AGG_SORTED : AGG_PLAIN,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
AGGSPLIT_INITIAL_SERIAL,
- info->clauses,
+ parse->groupClause,
NIL,
agg_partial_costs,
dNumPartialPartialGroups));
@@ -6976,10 +6811,59 @@ create_partial_grouping_paths(PlannerInfo *root,
create_group_path(root,
partially_grouped_rel,
path,
- info->clauses,
+ parse->groupClause,
NIL,
dNumPartialPartialGroups));
}
+
+ /*
+ * Now we may consider incremental sort on this path, but only
+ * when the path is not already sorted and when incremental sort
+ * is enabled.
+ */
+ if (is_sorted || !enable_incremental_sort)
+ continue;
+
+ /* Restore the input path (we might have added Sort on top). */
+ path = path_original;
+
+ /* no shared prefix, not point in building incremental sort */
+ if (presorted_keys == 0)
+ continue;
+
+ /*
+ * We should have already excluded pathkeys of length 1 because
+ * then presorted_keys > 0 would imply is_sorted was true.
+ */
+ Assert(list_length(root->group_pathkeys) != 1);
+
+ path = (Path *) create_incremental_sort_path(root,
+ partially_grouped_rel,
+ path,
+ root->group_pathkeys,
+ presorted_keys,
+ -1.0);
+
+ if (parse->hasAggs)
+ add_partial_path(partially_grouped_rel, (Path *)
+ create_agg_path(root,
+ partially_grouped_rel,
+ path,
+ partially_grouped_rel->reltarget,
+ parse->groupClause ? AGG_SORTED : AGG_PLAIN,
+ AGGSPLIT_INITIAL_SERIAL,
+ parse->groupClause,
+ NIL,
+ agg_partial_costs,
+ dNumPartialPartialGroups));
+ else
+ add_partial_path(partially_grouped_rel, (Path *)
+ create_group_path(root,
+ partially_grouped_rel,
+ path,
+ parse->groupClause,
+ NIL,
+ dNumPartialPartialGroups));
}
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index e2a3c110ce..2de5b0c836 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -1342,7 +1342,7 @@ create_append_path(PlannerInfo *root,
pathnode->path.pathkeys = child->pathkeys;
}
else
- cost_append(pathnode, root);
+ cost_append(pathnode);
/* If the caller provided a row estimate, override the computed value. */
if (rows >= 0)
diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c
index 1884918318..8d1b374bdf 100644
--- a/src/backend/utils/adt/selfuncs.c
+++ b/src/backend/utils/adt/selfuncs.c
@@ -3368,28 +3368,11 @@ double
estimate_num_groups(PlannerInfo *root, List *groupExprs, double input_rows,
List **pgset, EstimationInfo *estinfo)
{
- return estimate_num_groups_incremental(root, groupExprs,
- input_rows, pgset, estinfo,
- NULL, 0);
-}
-
-/*
- * estimate_num_groups_incremental
- * An estimate_num_groups variant, optimized for cases that are adding the
- * expressions incrementally (e.g. one by one).
- */
-double
-estimate_num_groups_incremental(PlannerInfo *root, List *groupExprs,
- double input_rows,
- List **pgset, EstimationInfo *estinfo,
- List **cache_varinfos, int prevNExprs)
-{
- List *varinfos = (cache_varinfos) ? *cache_varinfos : NIL;
+ List *varinfos = NIL;
double srf_multiplier = 1.0;
double numdistinct;
ListCell *l;
- int i,
- j;
+ int i;
/* Zero the estinfo output parameter, if non-NULL */
if (estinfo != NULL)
@@ -3420,7 +3403,7 @@ estimate_num_groups_incremental(PlannerInfo *root, List *groupExprs,
*/
numdistinct = 1.0;
- i = j = 0;
+ i = 0;
foreach(l, groupExprs)
{
Node *groupexpr = (Node *) lfirst(l);
@@ -3429,14 +3412,6 @@ estimate_num_groups_incremental(PlannerInfo *root, List *groupExprs,
List *varshere;
ListCell *l2;
- /* was done on previous call */
- if (cache_varinfos && j++ < prevNExprs)
- {
- if (pgset)
- i++; /* to keep in sync with lines below */
- continue;
- }
-
/* is expression in this grouping set? */
if (pgset && !list_member_int(*pgset, i++))
continue;
@@ -3506,11 +3481,7 @@ estimate_num_groups_incremental(PlannerInfo *root, List *groupExprs,
if (varshere == NIL)
{
if (contain_volatile_functions(groupexpr))
- {
- if (cache_varinfos)
- *cache_varinfos = varinfos;
return input_rows;
- }
continue;
}
@@ -3527,9 +3498,6 @@ estimate_num_groups_incremental(PlannerInfo *root, List *groupExprs,
}
}
- if (cache_varinfos)
- *cache_varinfos = varinfos;
-
/*
* If now no Vars, we must have an all-constant or all-boolean GROUP BY
* list.
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index be7bf9d218..328aab0771 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -1214,16 +1214,6 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
- {
- {"enable_group_by_reordering", PGC_USERSET, QUERY_TUNING_METHOD,
- gettext_noop("Enables reordering of GROUP BY keys."),
- NULL,
- GUC_EXPLAIN
- },
- &enable_group_by_reordering,
- true,
- NULL, NULL, NULL
- },
{
{"geqo", PGC_USERSET, QUERY_TUNING_GEQO,
gettext_noop("Enables genetic query optimization."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index b4bc06e5f5..f92ff4cc29 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -388,7 +388,6 @@
#enable_seqscan = on
#enable_sort = on
#enable_tidscan = on
-#enable_group_by_reordering = on
# - Planner Cost Constants -
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a6e5db4eec..8556b2ffe7 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1071,16 +1071,6 @@ typedef struct PathKey
bool pk_nulls_first; /* do NULLs come before normal values? */
} PathKey;
-/*
- * Combines information about pathkeys and the associated clauses.
- */
-typedef struct PathKeyInfo
-{
- NodeTag type;
- List *pathkeys;
- List *clauses;
-} PathKeyInfo;
-
/*
* VolatileFunctionStatus -- allows nodes to cache their
* contain_volatile_functions properties. VOLATILITY_UNKNOWN means not yet
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index dc7fc17411..bc12071af6 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -114,9 +114,7 @@ extern void cost_incremental_sort(Path *path,
Cost input_startup_cost, Cost input_total_cost,
double input_tuples, int width, Cost comparison_cost, int sort_mem,
double limit_tuples);
-extern Cost cost_sort_estimate(PlannerInfo *root, List *pathkeys,
- int nPresortedKeys, double tuples);
-extern void cost_append(AppendPath *path, PlannerInfo *root);
+extern void cost_append(AppendPath *path);
extern void cost_merge_append(Path *path, PlannerInfo *root,
List *pathkeys, int n_streams,
Cost input_startup_cost, Cost input_total_cost,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 54ab709c67..e313eb2138 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -24,7 +24,6 @@ extern PGDLLIMPORT bool enable_geqo;
extern PGDLLIMPORT int geqo_threshold;
extern PGDLLIMPORT int min_parallel_table_scan_size;
extern PGDLLIMPORT int min_parallel_index_scan_size;
-extern PGDLLIMPORT bool enable_group_by_reordering;
/* Hook for plugins to get control in set_rel_pathlist() */
typedef void (*set_rel_pathlist_hook_type) (PlannerInfo *root,
@@ -203,12 +202,6 @@ typedef enum
extern PathKeysComparison compare_pathkeys(List *keys1, List *keys2);
extern bool pathkeys_contained_in(List *keys1, List *keys2);
extern bool pathkeys_count_contained_in(List *keys1, List *keys2, int *n_common);
-extern int group_keys_reorder_by_pathkeys(List *pathkeys,
- List **group_pathkeys,
- List **group_clauses);
-extern List *get_useful_group_keys_orderings(PlannerInfo *root, double nrows,
- List *path_pathkeys,
- List *group_pathkeys, List *group_clauses);
extern Path *get_cheapest_path_for_pathkeys(List *paths, List *pathkeys,
Relids required_outer,
CostSelector cost_criterion,
diff --git a/src/include/utils/selfuncs.h b/src/include/utils/selfuncs.h
index d485b9bfcd..8f3d73edfb 100644
--- a/src/include/utils/selfuncs.h
+++ b/src/include/utils/selfuncs.h
@@ -214,11 +214,6 @@ extern double estimate_num_groups(PlannerInfo *root, List *groupExprs,
double input_rows, List **pgset,
EstimationInfo *estinfo);
-extern double estimate_num_groups_incremental(PlannerInfo *root, List *groupExprs,
- double input_rows, List **pgset,
- EstimationInfo *estinfo,
- List **cache_varinfos, int prevNExprs);
-
extern void estimate_hash_bucket_stats(PlannerInfo *root,
Node *hashkey, double nbuckets,
Selectivity *mcv_freq,
diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out
index 601047fa3d..0a23a39aa2 100644
--- a/src/test/regress/expected/aggregates.out
+++ b/src/test/regress/expected/aggregates.out
@@ -1210,8 +1210,7 @@ explain (costs off)
select distinct min(f1), max(f1) from minmaxtest;
QUERY PLAN
---------------------------------------------------------------------------------------------
- HashAggregate
- Group Key: $0, $1
+ Unique
InitPlan 1 (returns $0)
-> Limit
-> Merge Append
@@ -1234,8 +1233,10 @@ explain (costs off)
-> Index Only Scan using minmaxtest2i on minmaxtest2 minmaxtest_8
Index Cond: (f1 IS NOT NULL)
-> Index Only Scan Backward using minmaxtest3i on minmaxtest3 minmaxtest_9
- -> Result
-(25 rows)
+ -> Sort
+ Sort Key: ($0), ($1)
+ -> Result
+(26 rows)
select distinct min(f1), max(f1) from minmaxtest;
min | max
@@ -2447,241 +2448,6 @@ SELECT balk(hundred) FROM tenk1;
(1 row)
ROLLBACK;
--- GROUP BY optimization by reorder columns
-SELECT
- i AS id,
- i/2 AS p,
- format('%60s', i%2) AS v,
- i/4 AS c,
- i/8 AS d,
- (random() * (10000/8))::int as e --the same as d but no correlation with p
- INTO btg
-FROM
- generate_series(1, 10000) i;
-VACUUM btg;
-ANALYZE btg;
--- GROUP BY optimization by reorder columns by frequency
-SET enable_hashagg=off;
-SET max_parallel_workers= 0;
-SET max_parallel_workers_per_gather = 0;
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, v;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, v
- -> Sort
- Sort Key: p, v
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, v
- -> Sort
- Sort Key: p, v
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, c, v
- -> Sort
- Sort Key: p, c, v
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c ORDER BY v, p, c;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: v, p, c
- -> Sort
- Sort Key: v, p, c
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, d, c;
- QUERY PLAN
-------------------------------
- GroupAggregate
- Group Key: p, d, c, v
- -> Sort
- Sort Key: p, d, c, v
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, d, c ORDER BY v, p, d ,c;
- QUERY PLAN
-------------------------------
- GroupAggregate
- Group Key: v, p, d, c
- -> Sort
- Sort Key: v, p, d, c
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, d, c ORDER BY p, v, d ,c;
- QUERY PLAN
-------------------------------
- GroupAggregate
- Group Key: p, v, d, c
- -> Sort
- Sort Key: p, v, d, c
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, d, e;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, d, e
- -> Sort
- Sort Key: p, d, e
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, e, d;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, e, d
- -> Sort
- Sort Key: p, e, d
- -> Seq Scan on btg
-(5 rows)
-
-CREATE STATISTICS btg_dep ON d, e, p FROM btg;
-ANALYZE btg;
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, d, e;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, d, e
- -> Sort
- Sort Key: p, d, e
- -> Seq Scan on btg
-(5 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, e, d;
- QUERY PLAN
------------------------------
- GroupAggregate
- Group Key: p, e, d
- -> Sort
- Sort Key: p, e, d
- -> Seq Scan on btg
-(5 rows)
-
--- GROUP BY optimization by reorder columns by index scan
-CREATE INDEX ON btg(p, v);
-SET enable_seqscan=off;
-SET enable_bitmapscan=off;
-VACUUM btg;
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, v;
- QUERY PLAN
-------------------------------------------------
- GroupAggregate
- Group Key: p, v
- -> Index Only Scan using btg_p_v_idx on btg
-(3 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, v ORDER BY p, v;
- QUERY PLAN
-------------------------------------------------
- GroupAggregate
- Group Key: p, v
- -> Index Only Scan using btg_p_v_idx on btg
-(3 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p;
- QUERY PLAN
-------------------------------------------------
- GroupAggregate
- Group Key: p, v
- -> Index Only Scan using btg_p_v_idx on btg
-(3 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p ORDER BY p, v;
- QUERY PLAN
-------------------------------------------------
- GroupAggregate
- Group Key: p, v
- -> Index Only Scan using btg_p_v_idx on btg
-(3 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c;
- QUERY PLAN
--------------------------------------------------
- GroupAggregate
- Group Key: p, c, v
- -> Incremental Sort
- Sort Key: p, c, v
- Presorted Key: p
- -> Index Scan using btg_p_v_idx on btg
-(6 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c ORDER BY p, v;
- QUERY PLAN
--------------------------------------------------
- GroupAggregate
- Group Key: p, v, c
- -> Incremental Sort
- Sort Key: p, v, c
- Presorted Key: p, v
- -> Index Scan using btg_p_v_idx on btg
-(6 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, c, p, d;
- QUERY PLAN
--------------------------------------------------
- GroupAggregate
- Group Key: p, c, d, v
- -> Incremental Sort
- Sort Key: p, c, d, v
- Presorted Key: p
- -> Index Scan using btg_p_v_idx on btg
-(6 rows)
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, c, p, d ORDER BY p, v;
- QUERY PLAN
--------------------------------------------------
- GroupAggregate
- Group Key: p, v, c, d
- -> Incremental Sort
- Sort Key: p, v, c, d
- Presorted Key: p, v
- -> Index Scan using btg_p_v_idx on btg
-(6 rows)
-
-DROP TABLE btg;
-RESET enable_hashagg;
-RESET max_parallel_workers;
-RESET max_parallel_workers_per_gather;
-RESET enable_seqscan;
-RESET enable_bitmapscan;
-- Secondly test the case of a parallel aggregate combiner function
-- returning NULL. For that use normal transition function, but a
-- combiner function returning NULL.
diff --git a/src/test/regress/expected/incremental_sort.out b/src/test/regress/expected/incremental_sort.out
index 49953eaade..0a631124c2 100644
--- a/src/test/regress/expected/incremental_sort.out
+++ b/src/test/regress/expected/incremental_sort.out
@@ -1439,7 +1439,7 @@ set parallel_setup_cost = 0;
set parallel_tuple_cost = 0;
set max_parallel_workers_per_gather = 2;
create table t (a int, b int, c int);
-insert into t select mod(i,10),mod(i,10),i from generate_series(1,60000) s(i);
+insert into t select mod(i,10),mod(i,10),i from generate_series(1,10000) s(i);
create index on t (a);
analyze t;
set enable_incremental_sort = off;
diff --git a/src/test/regress/expected/join.out b/src/test/regress/expected/join.out
index 1f0df6b7d9..517c7b2ee2 100644
--- a/src/test/regress/expected/join.out
+++ b/src/test/regress/expected/join.out
@@ -1984,8 +1984,8 @@ USING (name);
------+----+----
bb | 12 | 13
cc | 22 | 23
- ee | 42 |
dd | | 33
+ ee | 42 |
(4 rows)
-- Cases with non-nullable expressions in subquery results;
@@ -2019,8 +2019,8 @@ NATURAL FULL JOIN
------+------+------+------+------
bb | 12 | 2 | 13 | 3
cc | 22 | 2 | 23 | 3
- ee | 42 | 2 | |
dd | | | 33 | 3
+ ee | 42 | 2 | |
(4 rows)
SELECT * FROM
@@ -4645,20 +4645,18 @@ select d.* from d left join (select * from b group by b.id, b.c_id) s
explain (costs off)
select d.* from d left join (select distinct * from b) s
on d.a = s.id;
- QUERY PLAN
----------------------------------------------
- Merge Left Join
- Merge Cond: (d.a = s.id)
+ QUERY PLAN
+--------------------------------------
+ Merge Right Join
+ Merge Cond: (b.id = d.a)
+ -> Unique
+ -> Sort
+ Sort Key: b.id, b.c_id
+ -> Seq Scan on b
-> Sort
Sort Key: d.a
-> Seq Scan on d
- -> Sort
- Sort Key: s.id
- -> Subquery Scan on s
- -> HashAggregate
- Group Key: b.id, b.c_id
- -> Seq Scan on b
-(11 rows)
+(9 rows)
-- check join removal works when uniqueness of the join condition is enforced
-- by a UNION
@@ -6368,39 +6366,44 @@ select * from j1 natural join j2;
explain (verbose, costs off)
select * from j1
inner join (select distinct id from j3) j3 on j1.id = j3.id;
- QUERY PLAN
------------------------------------
+ QUERY PLAN
+-----------------------------------------
Nested Loop
Output: j1.id, j3.id
Inner Unique: true
Join Filter: (j1.id = j3.id)
- -> HashAggregate
+ -> Unique
Output: j3.id
- Group Key: j3.id
- -> Seq Scan on public.j3
+ -> Sort
Output: j3.id
+ Sort Key: j3.id
+ -> Seq Scan on public.j3
+ Output: j3.id
-> Seq Scan on public.j1
Output: j1.id
-(11 rows)
+(13 rows)
-- ensure group by clause allows the inner to become unique
explain (verbose, costs off)
select * from j1
inner join (select id from j3 group by id) j3 on j1.id = j3.id;
- QUERY PLAN
------------------------------------
+ QUERY PLAN
+-----------------------------------------
Nested Loop
Output: j1.id, j3.id
Inner Unique: true
Join Filter: (j1.id = j3.id)
- -> HashAggregate
+ -> Group
Output: j3.id
Group Key: j3.id
- -> Seq Scan on public.j3
+ -> Sort
Output: j3.id
+ Sort Key: j3.id
+ -> Seq Scan on public.j3
+ Output: j3.id
-> Seq Scan on public.j1
Output: j1.id
-(11 rows)
+(14 rows)
drop table j1;
drop table j2;
diff --git a/src/test/regress/expected/merge.out b/src/test/regress/expected/merge.out
index 4047c3e761..787af41dfe 100644
--- a/src/test/regress/expected/merge.out
+++ b/src/test/regress/expected/merge.out
@@ -1460,15 +1460,18 @@ WHEN MATCHED AND t.a < 10 THEN
explain_merge
--------------------------------------------------------------------
Merge on ex_mtarget t (actual rows=0 loops=1)
- -> Hash Join (actual rows=0 loops=1)
- Hash Cond: (s.a = t.a)
- -> Seq Scan on ex_msource s (actual rows=1 loops=1)
- -> Hash (actual rows=0 loops=1)
- Buckets: xxx Batches: xxx Memory Usage: xxx
+ -> Merge Join (actual rows=0 loops=1)
+ Merge Cond: (t.a = s.a)
+ -> Sort (actual rows=0 loops=1)
+ Sort Key: t.a
+ Sort Method: quicksort Memory: xxx
-> Seq Scan on ex_mtarget t (actual rows=0 loops=1)
Filter: (a < '-1000'::integer)
Rows Removed by Filter: 54
-(9 rows)
+ -> Sort (never executed)
+ Sort Key: s.a
+ -> Seq Scan on ex_msource s (never executed)
+(12 rows)
DROP TABLE ex_msource, ex_mtarget;
DROP FUNCTION explain_merge(text);
diff --git a/src/test/regress/expected/partition_aggregate.out b/src/test/regress/expected/partition_aggregate.out
index 0dc6d63347..0d69619a2f 100644
--- a/src/test/regress/expected/partition_aggregate.out
+++ b/src/test/regress/expected/partition_aggregate.out
@@ -949,12 +949,12 @@ SET parallel_setup_cost = 0;
-- is not partial agg safe.
EXPLAIN (COSTS OFF)
SELECT a, sum(b), array_agg(distinct c), count(*) FROM pagg_tab_ml GROUP BY a HAVING avg(b) < 3 ORDER BY 1, 2, 3;
- QUERY PLAN
---------------------------------------------------------------------------------------------
- Gather Merge
- Workers Planned: 2
- -> Sort
- Sort Key: pagg_tab_ml.a, (sum(pagg_tab_ml.b)), (array_agg(DISTINCT pagg_tab_ml.c))
+ QUERY PLAN
+--------------------------------------------------------------------------------------
+ Sort
+ Sort Key: pagg_tab_ml.a, (sum(pagg_tab_ml.b)), (array_agg(DISTINCT pagg_tab_ml.c))
+ -> Gather
+ Workers Planned: 2
-> Parallel Append
-> GroupAggregate
Group Key: pagg_tab_ml.a
@@ -1381,26 +1381,28 @@ SELECT x, sum(y), avg(y), count(*) FROM pagg_tab_para GROUP BY x HAVING avg(y) <
-- When GROUP BY clause does not match; partial aggregation is performed for each partition.
EXPLAIN (COSTS OFF)
SELECT y, sum(x), avg(x), count(*) FROM pagg_tab_para GROUP BY y HAVING avg(x) < 12 ORDER BY 1, 2, 3;
- QUERY PLAN
--------------------------------------------------------------------------------------
+ QUERY PLAN
+-------------------------------------------------------------------------------------------
Sort
Sort Key: pagg_tab_para.y, (sum(pagg_tab_para.x)), (avg(pagg_tab_para.x))
- -> Finalize HashAggregate
+ -> Finalize GroupAggregate
Group Key: pagg_tab_para.y
Filter: (avg(pagg_tab_para.x) < '12'::numeric)
- -> Gather
+ -> Gather Merge
Workers Planned: 2
- -> Parallel Append
- -> Partial HashAggregate
- Group Key: pagg_tab_para.y
- -> Parallel Seq Scan on pagg_tab_para_p1 pagg_tab_para
- -> Partial HashAggregate
- Group Key: pagg_tab_para_1.y
- -> Parallel Seq Scan on pagg_tab_para_p2 pagg_tab_para_1
- -> Partial HashAggregate
- Group Key: pagg_tab_para_2.y
- -> Parallel Seq Scan on pagg_tab_para_p3 pagg_tab_para_2
-(17 rows)
+ -> Sort
+ Sort Key: pagg_tab_para.y
+ -> Parallel Append
+ -> Partial HashAggregate
+ Group Key: pagg_tab_para.y
+ -> Parallel Seq Scan on pagg_tab_para_p1 pagg_tab_para
+ -> Partial HashAggregate
+ Group Key: pagg_tab_para_1.y
+ -> Parallel Seq Scan on pagg_tab_para_p2 pagg_tab_para_1
+ -> Partial HashAggregate
+ Group Key: pagg_tab_para_2.y
+ -> Parallel Seq Scan on pagg_tab_para_p3 pagg_tab_para_2
+(19 rows)
SELECT y, sum(x), avg(x), count(*) FROM pagg_tab_para GROUP BY y HAVING avg(x) < 12 ORDER BY 1, 2, 3;
y | sum | avg | count
diff --git a/src/test/regress/expected/partition_join.out b/src/test/regress/expected/partition_join.out
index 03926a8413..bb5b7c47a4 100644
--- a/src/test/regress/expected/partition_join.out
+++ b/src/test/regress/expected/partition_join.out
@@ -466,41 +466,52 @@ EXPLAIN (COSTS OFF)
SELECT a, b FROM prt1 FULL JOIN prt2 p2(b,a,c) USING(a,b)
WHERE a BETWEEN 490 AND 510
GROUP BY 1, 2 ORDER BY 1, 2;
- QUERY PLAN
------------------------------------------------------------------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------------------------------------------------------------------
Group
Group Key: (COALESCE(prt1.a, p2.a)), (COALESCE(prt1.b, p2.b))
- -> Sort
+ -> Merge Append
Sort Key: (COALESCE(prt1.a, p2.a)), (COALESCE(prt1.b, p2.b))
- -> Append
- -> Merge Full Join
- Merge Cond: ((prt1_1.a = p2_1.a) AND (prt1_1.b = p2_1.b))
- Filter: ((COALESCE(prt1_1.a, p2_1.a) >= 490) AND (COALESCE(prt1_1.a, p2_1.a) <= 510))
- -> Sort
- Sort Key: prt1_1.a, prt1_1.b
- -> Seq Scan on prt1_p1 prt1_1
- -> Sort
- Sort Key: p2_1.a, p2_1.b
- -> Seq Scan on prt2_p1 p2_1
- -> Merge Full Join
- Merge Cond: ((prt1_2.a = p2_2.a) AND (prt1_2.b = p2_2.b))
- Filter: ((COALESCE(prt1_2.a, p2_2.a) >= 490) AND (COALESCE(prt1_2.a, p2_2.a) <= 510))
- -> Sort
- Sort Key: prt1_2.a, prt1_2.b
- -> Seq Scan on prt1_p2 prt1_2
- -> Sort
- Sort Key: p2_2.a, p2_2.b
- -> Seq Scan on prt2_p2 p2_2
- -> Merge Full Join
- Merge Cond: ((prt1_3.b = p2_3.b) AND (prt1_3.a = p2_3.a))
- Filter: ((COALESCE(prt1_3.a, p2_3.a) >= 490) AND (COALESCE(prt1_3.a, p2_3.a) <= 510))
- -> Sort
- Sort Key: prt1_3.b, prt1_3.a
- -> Seq Scan on prt1_p3 prt1_3
- -> Sort
- Sort Key: p2_3.b, p2_3.a
- -> Seq Scan on prt2_p3 p2_3
-(32 rows)
+ -> Group
+ Group Key: (COALESCE(prt1.a, p2.a)), (COALESCE(prt1.b, p2.b))
+ -> Sort
+ Sort Key: (COALESCE(prt1.a, p2.a)), (COALESCE(prt1.b, p2.b))
+ -> Merge Full Join
+ Merge Cond: ((prt1.a = p2.a) AND (prt1.b = p2.b))
+ Filter: ((COALESCE(prt1.a, p2.a) >= 490) AND (COALESCE(prt1.a, p2.a) <= 510))
+ -> Sort
+ Sort Key: prt1.a, prt1.b
+ -> Seq Scan on prt1_p1 prt1
+ -> Sort
+ Sort Key: p2.a, p2.b
+ -> Seq Scan on prt2_p1 p2
+ -> Group
+ Group Key: (COALESCE(prt1_1.a, p2_1.a)), (COALESCE(prt1_1.b, p2_1.b))
+ -> Sort
+ Sort Key: (COALESCE(prt1_1.a, p2_1.a)), (COALESCE(prt1_1.b, p2_1.b))
+ -> Merge Full Join
+ Merge Cond: ((prt1_1.a = p2_1.a) AND (prt1_1.b = p2_1.b))
+ Filter: ((COALESCE(prt1_1.a, p2_1.a) >= 490) AND (COALESCE(prt1_1.a, p2_1.a) <= 510))
+ -> Sort
+ Sort Key: prt1_1.a, prt1_1.b
+ -> Seq Scan on prt1_p2 prt1_1
+ -> Sort
+ Sort Key: p2_1.a, p2_1.b
+ -> Seq Scan on prt2_p2 p2_1
+ -> Group
+ Group Key: (COALESCE(prt1_2.a, p2_2.a)), (COALESCE(prt1_2.b, p2_2.b))
+ -> Sort
+ Sort Key: (COALESCE(prt1_2.a, p2_2.a)), (COALESCE(prt1_2.b, p2_2.b))
+ -> Merge Full Join
+ Merge Cond: ((prt1_2.a = p2_2.a) AND (prt1_2.b = p2_2.b))
+ Filter: ((COALESCE(prt1_2.a, p2_2.a) >= 490) AND (COALESCE(prt1_2.a, p2_2.a) <= 510))
+ -> Sort
+ Sort Key: prt1_2.a, prt1_2.b
+ -> Seq Scan on prt1_p3 prt1_2
+ -> Sort
+ Sort Key: p2_2.a, p2_2.b
+ -> Seq Scan on prt2_p3 p2_2
+(43 rows)
SELECT a, b FROM prt1 FULL JOIN prt2 p2(b,a,c) USING(a,b)
WHERE a BETWEEN 490 AND 510
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 4e775af175..579b861d84 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -114,7 +114,6 @@ select name, setting from pg_settings where name like 'enable%';
enable_async_append | on
enable_bitmapscan | on
enable_gathermerge | on
- enable_group_by_reordering | on
enable_hashagg | on
enable_hashjoin | on
enable_incremental_sort | on
@@ -132,7 +131,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(21 rows)
+(20 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
diff --git a/src/test/regress/expected/union.out b/src/test/regress/expected/union.out
index 7ac4a9380e..dece7310cf 100644
--- a/src/test/regress/expected/union.out
+++ b/src/test/regress/expected/union.out
@@ -1303,22 +1303,24 @@ select distinct q1 from
union all
select distinct * from int8_tbl i82) ss
where q2 = q2;
- QUERY PLAN
-----------------------------------------------------
- HashAggregate
- Group Key: "*SELECT* 1".q1
- -> Append
+ QUERY PLAN
+----------------------------------------------------------
+ Unique
+ -> Merge Append
+ Sort Key: "*SELECT* 1".q1
-> Subquery Scan on "*SELECT* 1"
- -> HashAggregate
- Group Key: i81.q1, i81.q2
- -> Seq Scan on int8_tbl i81
- Filter: (q2 IS NOT NULL)
+ -> Unique
+ -> Sort
+ Sort Key: i81.q1, i81.q2
+ -> Seq Scan on int8_tbl i81
+ Filter: (q2 IS NOT NULL)
-> Subquery Scan on "*SELECT* 2"
- -> HashAggregate
- Group Key: i82.q1, i82.q2
- -> Seq Scan on int8_tbl i82
- Filter: (q2 IS NOT NULL)
-(13 rows)
+ -> Unique
+ -> Sort
+ Sort Key: i82.q1, i82.q2
+ -> Seq Scan on int8_tbl i82
+ Filter: (q2 IS NOT NULL)
+(15 rows)
select distinct q1 from
(select distinct * from int8_tbl i81
@@ -1337,22 +1339,24 @@ select distinct q1 from
union all
select distinct * from int8_tbl i82) ss
where -q1 = q2;
- QUERY PLAN
---------------------------------------------------
- HashAggregate
- Group Key: "*SELECT* 1".q1
- -> Append
+ QUERY PLAN
+--------------------------------------------------------
+ Unique
+ -> Merge Append
+ Sort Key: "*SELECT* 1".q1
-> Subquery Scan on "*SELECT* 1"
- -> HashAggregate
- Group Key: i81.q1, i81.q2
- -> Seq Scan on int8_tbl i81
- Filter: ((- q1) = q2)
+ -> Unique
+ -> Sort
+ Sort Key: i81.q1, i81.q2
+ -> Seq Scan on int8_tbl i81
+ Filter: ((- q1) = q2)
-> Subquery Scan on "*SELECT* 2"
- -> HashAggregate
- Group Key: i82.q1, i82.q2
- -> Seq Scan on int8_tbl i82
- Filter: ((- q1) = q2)
-(13 rows)
+ -> Unique
+ -> Sort
+ Sort Key: i82.q1, i82.q2
+ -> Seq Scan on int8_tbl i82
+ Filter: ((- q1) = q2)
+(15 rows)
select distinct q1 from
(select distinct * from int8_tbl i81
diff --git a/src/test/regress/sql/aggregates.sql b/src/test/regress/sql/aggregates.sql
index c6e0d7ba2b..2f5d0e00f3 100644
--- a/src/test/regress/sql/aggregates.sql
+++ b/src/test/regress/sql/aggregates.sql
@@ -1025,105 +1025,6 @@ SELECT balk(hundred) FROM tenk1;
ROLLBACK;
--- GROUP BY optimization by reorder columns
-
-SELECT
- i AS id,
- i/2 AS p,
- format('%60s', i%2) AS v,
- i/4 AS c,
- i/8 AS d,
- (random() * (10000/8))::int as e --the same as d but no correlation with p
- INTO btg
-FROM
- generate_series(1, 10000) i;
-
-VACUUM btg;
-ANALYZE btg;
-
--- GROUP BY optimization by reorder columns by frequency
-
-SET enable_hashagg=off;
-SET max_parallel_workers= 0;
-SET max_parallel_workers_per_gather = 0;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, v;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c ORDER BY v, p, c;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, d, c;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, d, c ORDER BY v, p, d ,c;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, d, c ORDER BY p, v, d ,c;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, d, e;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, e, d;
-
-CREATE STATISTICS btg_dep ON d, e, p FROM btg;
-ANALYZE btg;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, d, e;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, e, d;
-
-
--- GROUP BY optimization by reorder columns by index scan
-
-CREATE INDEX ON btg(p, v);
-SET enable_seqscan=off;
-SET enable_bitmapscan=off;
-VACUUM btg;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, v;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY p, v ORDER BY p, v;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p ORDER BY p, v;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, p, c ORDER BY p, v;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, c, p, d;
-
-EXPLAIN (COSTS off)
-SELECT count(*) FROM btg GROUP BY v, c, p, d ORDER BY p, v;
-
-DROP TABLE btg;
-
-RESET enable_hashagg;
-RESET max_parallel_workers;
-RESET max_parallel_workers_per_gather;
-RESET enable_seqscan;
-RESET enable_bitmapscan;
-
-
-- Secondly test the case of a parallel aggregate combiner function
-- returning NULL. For that use normal transition function, but a
-- combiner function returning NULL.
diff --git a/src/test/regress/sql/incremental_sort.sql b/src/test/regress/sql/incremental_sort.sql
index 6a0e87c7f6..284a354dbb 100644
--- a/src/test/regress/sql/incremental_sort.sql
+++ b/src/test/regress/sql/incremental_sort.sql
@@ -213,7 +213,7 @@ set parallel_tuple_cost = 0;
set max_parallel_workers_per_gather = 2;
create table t (a int, b int, c int);
-insert into t select mod(i,10),mod(i,10),i from generate_series(1,60000) s(i);
+insert into t select mod(i,10),mod(i,10),i from generate_series(1,10000) s(i);
create index on t (a);
analyze t;
On Mon, 3 Oct 2022 at 08:10, Tom Lane <tgl@sss.pgh.pa.us> wrote: > As attached. For the master version, I think it's safe just to get rid of PlannerInfo.num_groupby_pathkeys now. I only added that so I could strip off the ORDER BY / DISTINCT aggregate PathKeys from the group by pathkeys before passing to the functions that rearranged the GROUP BY clause. David
David Rowley <dgrowleyml@gmail.com> writes:
> For the master version, I think it's safe just to get rid of
> PlannerInfo.num_groupby_pathkeys now. I only added that so I could
> strip off the ORDER BY / DISTINCT aggregate PathKeys from the group by
> pathkeys before passing to the functions that rearranged the GROUP BY
> clause.
I was kind of unhappy with that data structure too, but from the
other direction: I didn't like that you were folding aggregate-derived
pathkeys into root->group_pathkeys in the first place. That seems like
a kluge that might work all right for the moment but will cause problems
down the road. (Despite the issues with the patch at hand, I don't
think it's unreasonable to suppose that somebody will have a more
successful go at optimizing GROUP BY sorting later.) If we keep the
data structure like this, I think we absolutely need num_groupby_pathkeys,
or some other way of recording which pathkeys came from what source.
One way to manage that would be to insist that the length of
root->group_clauses should indicate the number of associated grouping
pathkeys. Right now they might not be the same because we might discover
some of the pathkeys to be redundant --- but if we do, ISTM that the
corresponding GROUP BY clauses are also redundant and could get dropped.
That ties into the stuff I was worried about in [1], though. I'll keep
this in mind when I get back to messing with that.
regards, tom lane
[1] https://www.postgresql.org/message-id/flat/1657885.1657647073%40sss.pgh.pa.us
On Mon, 3 Oct 2022 at 09:59, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > David Rowley <dgrowleyml@gmail.com> writes: > > For the master version, I think it's safe just to get rid of > > PlannerInfo.num_groupby_pathkeys now. I only added that so I could > > strip off the ORDER BY / DISTINCT aggregate PathKeys from the group by > > pathkeys before passing to the functions that rearranged the GROUP BY > > clause. > > I was kind of unhappy with that data structure too, but from the > other direction: I didn't like that you were folding aggregate-derived > pathkeys into root->group_pathkeys in the first place. That seems like > a kluge that might work all right for the moment but will cause problems > down the road. (Despite the issues with the patch at hand, I don't > think it's unreasonable to suppose that somebody will have a more > successful go at optimizing GROUP BY sorting later.) If we keep the > data structure like this, I think we absolutely need num_groupby_pathkeys, > or some other way of recording which pathkeys came from what source. Ok, I don't feel too strongly about removing num_groupby_pathkeys. I'm fine to leave it there. However, I'll reserve slight concerns that we'll likely receive sporadic submissions of cleanup patches that remove the unused field over the course of the next few years and that dealing with those might take up more time than just removing it now and putting it back when we need it. We have been receiving quite a few patches along those lines lately. As for the slight misuse of group_pathkeys, I guess since there are no users that require just the plain pathkeys belonging to the GROUP BY, then likely the best thing would be just to rename that field to something like groupagg_pathkeys. Maintaining two separate fields and concatenating them every time we want group_pathkeys does not seem that appealing to me. Seems like a waste of memory and effort. I don't want to hi-jack this thread to discuss that, but if you have a preferred course of action, then I'm happy to kick off a discussion on a new thread. David
David Rowley <dgrowleyml@gmail.com> writes:
> As for the slight misuse of group_pathkeys, I guess since there are no
> users that require just the plain pathkeys belonging to the GROUP BY,
> then likely the best thing would be just to rename that field to
> something like groupagg_pathkeys. Maintaining two separate fields and
> concatenating them every time we want group_pathkeys does not seem
> that appealing to me. Seems like a waste of memory and effort. I don't
> want to hi-jack this thread to discuss that, but if you have a
> preferred course of action, then I'm happy to kick off a discussion on
> a new thread.
I don't feel any great urgency to resolve this. Let's wait and see
what comes out of the other thread.
regards, tom lane
On Sun, Oct 02, 2022 at 02:11:12PM -0400, Tom Lane wrote: > "Jonathan S. Katz" <jkatz@postgresql.org> writes: >> OK. For v15 I am heavily in favor for the least risky approach given the >> point we are at in the release cycle. The RMT hasn’t met yet to discuss, >> but from re-reading this thread again, I would recommend to revert >> (i.e. the “straight up revert”). > > OK by me. I don't quite see why it would be to let this code live on HEAD if it is not ready to be merged as there is a risk of creating side issues with things tied to the costing still ready to be merged, so I agree that the reversion done on both branches is the way to go for now. This could always be reworked and reproposed in the future. > I'm just about to throw up my hands and go for reversion in both branches, > because I'm now discovering that the code I'd hoped to salvage in > pathkeys.c (get_useful_group_keys_orderings and related) has its very own > bugs. It's imagining that it can rearrange a PathKeys list arbitrarily > and then rearrange the GROUP BY SortGroupClause list to match, but that's > easier said than done, for a couple of different reasons. (I now > understand why db0d67db2 made a cowboy hack in get_eclass_for_sort_expr ... > but it's still a cowboy hack with difficult-to-foresee side effects.) > There are other things in there that make it painfully obvious that > this code wasn't very carefully reviewed, eg XXX comments that should > have been followed up and were not, or a reference to a nonexistent > "debug_group_by_match_order_by" flag (maybe that was a GUC at some point?). Okay. Ugh. -- Michael
Attachment
On 10/2/22 8:45 PM, Michael Paquier wrote: > On Sun, Oct 02, 2022 at 02:11:12PM -0400, Tom Lane wrote: >> "Jonathan S. Katz" <jkatz@postgresql.org> writes: >>> OK. For v15 I am heavily in favor for the least risky approach given the >>> point we are at in the release cycle. The RMT hasn’t met yet to discuss, >>> but from re-reading this thread again, I would recommend to revert >>> (i.e. the “straight up revert”). >> >> OK by me. > > I don't quite see why it would be to let this code live on HEAD if it > is not ready to be merged as there is a risk of creating side issues > with things tied to the costing still ready to be merged, so I agree > that the reversion done on both branches is the way to go for now. > This could always be reworked and reproposed in the future. [RMT-hat] Just to follow things procedure-wise[1], while there do not seem to be any objections to reverting through regular community processes, I do think the RMT has to make this ask as Tomas (patch committer) has not commented and we are up against release deadlines. Based on the above discussion, the RMT asks for a revert of db0d67db2 in the v15 release. The RMT also recommends a revert in HEAD but does not have the power to request that. We do hope to see continued work and inclusion of this feature for a future release. We understand that the work on this optimization is complicated and appreciate all of the efforts on it. Thanks, Jonathan [1] https://wiki.postgresql.org/wiki/Release_Management_Team
Attachment
"Jonathan S. Katz" <jkatz@postgresql.org> writes:
> Based on the above discussion, the RMT asks for a revert of db0d67db2 in
> the v15 release. The RMT also recommends a revert in HEAD but does not
> have the power to request that.
Roger, I'll push these shortly.
regards, tom lane
[ Just for the archives' sake at this point, in case somebody has
another go at this feature. ]
I wrote:
> ... I'm now discovering that the code I'd hoped to salvage in
> pathkeys.c (get_useful_group_keys_orderings and related) has its very own
> bugs. It's imagining that it can rearrange a PathKeys list arbitrarily
> and then rearrange the GROUP BY SortGroupClause list to match, but that's
> easier said than done, for a couple of different reasons.
It strikes me that the easy solution here is to *not* rearrange the
SortGroupClause list at all. What that would be used for later is
to generate a Unique node's list of columns to compare, but since
Unique only cares about equality-or-not, there's no strong reason
why it has to compare the columns in the same order they're sorted
in. (Indeed, if anything we should prefer to compare them in the
opposite order, since the least-significant column should be the
most likely to be different from the previous row.)
I'm fairly sure that the just-reverted code is buggy on its
own terms, in that it might sometimes produce a clause list
that's not ordered the same as the pathkeys; but there's no
visible misbehavior, because that does not in fact matter.
So this'd let us simplify the APIs here, in particular PathKeyInfo
seems unnecessary, because we don't have to pass the SortGroupClause
list into or out of the pathkey-reordering logic.
regards, tom lane
On 10/3/22 16:05, Tom Lane wrote: > "Jonathan S. Katz" <jkatz@postgresql.org> writes: >> Based on the above discussion, the RMT asks for a revert of db0d67db2 in >> the v15 release. The RMT also recommends a revert in HEAD but does not >> have the power to request that. > > Roger, I'll push these shortly. > Thanks for resolving this, and apologies for not noticing this thread earlier (and for the bugs in the code, ofc). regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company