Thread: Wrong cost estimation for foreign tables join withuse_remote_estimate disabled
Wrong cost estimation for foreign tables join withuse_remote_estimate disabled
From
Konstantin Knizhnik
Date:
Hi hackers, I hope that somebody understand postgres_fdw cost calculation magic better than I;) The following very simple test reduce the problem with wrong cost estimation: create table t1(x integer primary key, y integer); create index on t1(y); insert into t1 values (generate_series(1,1000000), generate_series(1000001,2000000)); create table t2(x integer primary key); insert into t2 values (generate_series(1,1000000)); create server pg_fdw FOREIGN DATA WRAPPER postgres_fdw options(host 'localhost', dbname 'postgres'); create foreign table t1_fdw(x integer, y integer) server pg_fdw options (table_name 't1', use_remote_estimate 'false'); create foreign table t2_fdw(x integer) server pg_fdw options (table_name 't2', use_remote_estimate 'false'); analyze t1; analyze t2; analyze t2_fdw; analyze t1_fdw; explain select * from t1_fdw join t2_fdw on t1_fdw.x=t2_fdw.x where y in (1234567,1234577,1234667,1235567,1244567,1334567); ------------------------------------------------------------------------------------------------------------------------------- Hash Join (cost=22125.20..60300.26 rows=6 width=12) (actual time=439.187..1849.459 rows=6 loops=1) Hash Cond: (t2_fdw.x = t1_fdw.x) -> Foreign Scan on t2_fdw (cost=100.00..34525.00 rows=1000000 width=4) (actual time=0.526..1711.671 rows=1000000 loops=1) -> Hash (cost=22025.12..22025.12 rows=6 width=8) (actual time=0.511..0.511 rows=6 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 9kB -> Foreign Scan on t1_fdw (cost=100.00..22025.12 rows=6 width=8) (actual time=0.506..0.507 rows=6 loops=1) Planning Time: 0.173 ms Execution Time: 1849.871 ms (8 rows) So instead of pushing join to the remote server, optimizer decides that it is more efficient to perform join locally. If IN lis contains less alternatives (<= 2), then correct plan is used: postgres=# explain select * from t1_fdw join t2_fdw on t1_fdw.x=t2_fdw.x where y in (1234567,1234577); QUERY PLAN --------------------------------------------------------- Foreign Scan (cost=100.00..41450.04 rows=2 width=12) Relations: (public.t1_fdw) INNER JOIN (public.t2_fdw) (2 rows) It is possible to force Postgres to use correct plan by setting "fdw_startup_cost" to some very large value (100000000 for example). Also correct plan is used when use_remote_estimate is true. But in this case query optimization time is too large (not at this dummy example, but on real database and query with join of many large tables it takes about 10 seconds to perform remote estimation of all joined tables). Please notice that optimizer correctly estimates number of retrieved rows: 6. But it overestimates cost of remote join. Looks like it is because of the following code in estimate_path_cost_size: /* * Run time cost includes: * * 1. Run time cost (total_cost - startup_cost) of relations being * joined * * 2. Run time cost of applying join clauses on the cross product * of the joining relations. * * 3. Run time cost of applying pushed down other clauses on the * result of join * * 4. Run time cost of applying nonpushable other clauses locally * on the result fetched from the foreign server. */ run_cost = fpinfo_i->rel_total_cost - fpinfo_i->rel_startup_cost; run_cost += fpinfo_o->rel_total_cost - fpinfo_o->rel_startup_cost; run_cost += nrows * join_cost.per_tuple; nrows = clamp_row_est(nrows * fpinfo->joinclause_sel); run_cost += nrows * remote_conds_cost.per_tuple; run_cost += fpinfo->local_conds_cost.per_tuple * retrieved_rows; 815 run_cost = fpinfo_i->rel_total_cost - fpinfo_i->rel_startup_cost; (gdb) p fpinfo_i->rel_total_cost $23 = 14425 2816 run_cost += fpinfo_o->rel_total_cost - fpinfo_o->rel_startup_cost; (gdb) p fpinfo_o->rel_total_cost $25 = 21925 2817 run_cost += nrows * join_cost.per_tuple; (gdb) p run_cost $26 = 36350 I wonder if it is possible to make estimation of foreign join cost more precise. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
Re: Wrong cost estimation for foreign tables join with use_remote_estimate disabled
From
Tom Lane
Date:
Konstantin Knizhnik <k.knizhnik@postgrespro.ru> writes:
> The following very simple test reduce the problem with wrong cost
> estimation:
> create foreign table t1_fdw(x integer, y integer) server pg_fdw options
> (table_name 't1', use_remote_estimate 'false');
> create foreign table t2_fdw(x integer) server pg_fdw options (table_name
> 't2', use_remote_estimate 'false');
> It is possible to force Postgres to use correct plan by setting
> "fdw_startup_cost" to some very large value (100000000 for example).
> ...
> Also correct plan is used when use_remote_estimate is true.
If you are unhappy about the results with use_remote_estimate off, don't
run it that way. The optimizer does not have a crystal ball.
regards, tom lane
Re: Wrong cost estimation for foreign tables join withuse_remote_estimate disabled
From
Konstantin Knizhnik
Date:
On 21.06.2018 20:08, Tom Lane wrote:
> Konstantin Knizhnik <k.knizhnik@postgrespro.ru> writes:
>> The following very simple test reduce the problem with wrong cost
>> estimation:
>> create foreign table t1_fdw(x integer, y integer) server pg_fdw options
>> (table_name 't1', use_remote_estimate 'false');
>> create foreign table t2_fdw(x integer) server pg_fdw options (table_name
>> 't2', use_remote_estimate 'false');
>> It is possible to force Postgres to use correct plan by setting
>> "fdw_startup_cost" to some very large value (100000000 for example).
>> ...
>> Also correct plan is used when use_remote_estimate is true.
> If you are unhappy about the results with use_remote_estimate off, don't
> run it that way. The optimizer does not have a crystal ball.
As I wrote, use_remote_estimate can not be used because in this case
query compilation time is unacceptable (10 seconds, while time of query
execution itself is ~200msec).
So the problem can be addressed in two ways:
1. Try to reduce time of remote estimation. I wonder why postgres_fdw
sends so much queries to remote server. For join of two tables there are
7 queries.
I suspect that for ~20 joined tables in the original query number of
calls is more than hundred, so on wonder that it takes so much time.
2. Try to make optimizer make better estimation of join cost based on
local statistic (please notice that ANALYZE is explicitly called for all
foreign tables and number of rows in the result was correctly calculated).
What do you think: which of this two direction is more perspective? Or
it is better to address both of them?
By the way, below is list of remote EXPLAIN statements performed by
postgres_fdw for the mentioned above query when use_remote_estimate is on:
Breakpoint 1, get_remote_estimate (
sql=0x1940008 "EXPLAIN SELECT x, y FROM public.t1 WHERE ((y = ANY
('{1234567,1234577,1234667,1235567,1244567,1334567}'::integer[])))",
conn=0x190e0d0,
rows=0x7ffdd9e93388, width=0x7ffdd9e9337c,
startup_cost=0x7ffdd9e93390, total_cost=0x7ffdd9e93398) at
postgres_fdw.c:2984
2984 {
(gdb) cont
Continuing.
Breakpoint 1, get_remote_estimate (sql=0x196fa68 "EXPLAIN SELECT x FROM
public.t2", conn=0x190e0d0, rows=0x7ffdd9e93388, width=0x7ffdd9e9337c,
startup_cost=0x7ffdd9e93390, total_cost=0x7ffdd9e93398) at
postgres_fdw.c:2984
2984 {
(gdb)
Continuing.
Breakpoint 1, get_remote_estimate (
sql=0x19208f8 "EXPLAIN SELECT x, y FROM public.t1 WHERE ((y = ANY
('{1234567,1234577,1234667,1235567,1244567,1334567}'::integer[]))) ORDER
BY x ASC NULLS LAST", conn=0x190e0d0, rows=0x7ffdd9e932c8,
width=0x7ffdd9e932bc, startup_cost=0x7ffdd9e932d0,
total_cost=0x7ffdd9e932d8) at postgres_fdw.c:2984
2984 {
(gdb)
Continuing.
Breakpoint 1, get_remote_estimate (
sql=0x19227b0 "EXPLAIN SELECT x, y FROM public.t1 WHERE ((((SELECT
null::integer)::integer) = x)) AND ((y = ANY
('{1234567,1234577,1234667,1235567,1244567,1334567}'::integer[])))",
conn=0x190e0d0, rows=0x7ffdd9e93348, width=0x7ffdd9e9333c,
startup_cost=0x7ffdd9e93350, total_cost=0x7ffdd9e93358)
at postgres_fdw.c:2984
2984 {
(gdb)
Continuing.
Breakpoint 1, get_remote_estimate (sql=0x19236c0 "EXPLAIN SELECT x FROM
public.t2 ORDER BY x ASC NULLS LAST", conn=0x190e0d0, rows=0x7ffdd9e932c8,
width=0x7ffdd9e932bc, startup_cost=0x7ffdd9e932d0,
total_cost=0x7ffdd9e932d8) at postgres_fdw.c:2984
2984 {
(gdb)
Continuing.
Breakpoint 1, get_remote_estimate (sql=0x19247c0 "EXPLAIN SELECT x FROM
public.t2 WHERE ((((SELECT null::integer)::integer) = x))", conn=0x190e0d0,
rows=0x7ffdd9e93348, width=0x7ffdd9e9333c,
startup_cost=0x7ffdd9e93350, total_cost=0x7ffdd9e93358) at
postgres_fdw.c:2984
2984 {
(gdb)
Continuing.
Breakpoint 1, get_remote_estimate (
sql=0x19267d0 "EXPLAIN SELECT r1.x, r1.y, r2.x FROM (public.t1 r1
INNER JOIN public.t2 r2 ON (((r1.x = r2.x)) AND ((r1.y = ANY
('{1234567,1234577,1234667,1235567,1244567,1334567}'::integer[])))))",
conn=0x190e0d0, rows=0x7ffdd9e93108, width=0x7ffdd9e930fc,
startup_cost=0x7ffdd9e93110, total_cost=0x7ffdd9e93118)
at postgres_fdw.c:2984
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Re: Wrong cost estimation for foreign tables join with use_remote_estimatedisabled
From
Etsuro Fujita
Date:
Hi Konstantin, (2018/06/22 15:26), Konstantin Knizhnik wrote: > On 21.06.2018 20:08, Tom Lane wrote: >> Konstantin Knizhnik <k.knizhnik@postgrespro.ru> writes: >>> The following very simple test reduce the problem with wrong cost >>> estimation: >>> create foreign table t1_fdw(x integer, y integer) server pg_fdw options >>> (table_name 't1', use_remote_estimate 'false'); >>> create foreign table t2_fdw(x integer) server pg_fdw options (table_name >>> 't2', use_remote_estimate 'false'); >>> It is possible to force Postgres to use correct plan by setting >>> "fdw_startup_cost" to some very large value (100000000 for example). >>> ... >>> Also correct plan is used when use_remote_estimate is true. >> If you are unhappy about the results with use_remote_estimate off, don't >> run it that way. The optimizer does not have a crystal ball. > > As I wrote, use_remote_estimate can not be used because in this case > query compilation time is unacceptable (10 seconds, while time of query > execution itself is ~200msec). > So the problem can be addressed in two ways: > > 1. Try to reduce time of remote estimation. I wonder why postgres_fdw > sends so much queries to remote server. For join of two tables there are > 7 queries. > I suspect that for ~20 joined tables in the original query number of > calls is more than hundred, so on wonder that it takes so much time. > 2. Try to make optimizer make better estimation of join cost based on > local statistic (please notice that ANALYZE is explicitly called for all > foreign tables and number of rows in the result was correctly calculated). To make local estimates more accurate, I think we need other information on remote tables such as remote indexes. > What do you think: which of this two direction is more perspective? Or > it is better to address both of them? I'd vote for #2. One idea for that is to introduce CREATE FOREIGN INDEX to have information on remote indexes on the local side, which I proposed before. I have been putting it on hold since then, though. Best regards, Etsuro Fujita
Re: Wrong cost estimation for foreign tables join withuse_remote_estimate disabled
From
Amit Langote
Date:
On 2018/06/22 18:15, Etsuro Fujita wrote: > I'd vote for #2. One idea for that is to introduce CREATE FOREIGN INDEX > to have information on remote indexes on the local side, which I proposed > before. I have been putting it on hold since then, though. Sorry to hijack this thread, but I'd like to say that CREATE FOREIGN INDEX would be nice, as that would also let us lift certain restrictions on partitioned table indexes [1]. Thanks, Amit [1] https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=4eaa5372754
Re: Wrong cost estimation for foreign tables join with use_remote_estimatedisabled
From
Etsuro Fujita
Date:
(2018/06/22 18:49), Amit Langote wrote: > On 2018/06/22 18:15, Etsuro Fujita wrote: >> I'd vote for #2. One idea for that is to introduce CREATE FOREIGN INDEX >> to have information on remote indexes on the local side, which I proposed >> before. I have been putting it on hold since then, though. > > Sorry to hijack this thread, but I'd like to say that CREATE FOREIGN INDEX > would be nice, as that would also let us lift certain restrictions on > partitioned table indexes [1]. Agreed. I think that would be useful to support INSERT ... ON CONFLICT fully not only on single foreign tables but partitioned tables containing foreign partitions. Best regards, Etsuro Fujita
Re: Wrong cost estimation for foreign tables join withuse_remote_estimate disabled
From
Ashutosh Bapat
Date:
On Fri, Jun 22, 2018 at 11:56 AM, Konstantin Knizhnik
<k.knizhnik@postgrespro.ru> wrote:
>
>
> On 21.06.2018 20:08, Tom Lane wrote:
>>
>> Konstantin Knizhnik <k.knizhnik@postgrespro.ru> writes:
>>>
>>> The following very simple test reduce the problem with wrong cost
>>> estimation:
>>> create foreign table t1_fdw(x integer, y integer) server pg_fdw options
>>> (table_name 't1', use_remote_estimate 'false');
>>> create foreign table t2_fdw(x integer) server pg_fdw options (table_name
>>> 't2', use_remote_estimate 'false');
>>> It is possible to force Postgres to use correct plan by setting
>>> "fdw_startup_cost" to some very large value (100000000 for example).
>>> ...
>>> Also correct plan is used when use_remote_estimate is true.
>>
>> If you are unhappy about the results with use_remote_estimate off, don't
>> run it that way. The optimizer does not have a crystal ball.
>
>
> As I wrote, use_remote_estimate can not be used because in this case query
> compilation time is unacceptable (10 seconds, while time of query execution
> itself is ~200msec).
> So the problem can be addressed in two ways:
>
> 1. Try to reduce time of remote estimation. I wonder why postgres_fdw sends
> so much queries to remote server. For join of two tables there are 7
> queries.
> I suspect that for ~20 joined tables in the original query number of calls
> is more than hundred, so on wonder that it takes so much time.
> 2. Try to make optimizer make better estimation of join cost based on local
> statistic (please notice that ANALYZE is explicitly called for all foreign
> tables and number of rows in the result was correctly calculated).
>
I think estimate_path_cost_size() is too pessimistic about how many
times the join conditions are evaluated (Sorry, I have written that
code when I was worked on join pushdown for postgres_fdw.)
/* Estimate of number of rows in cross product */
nrows = fpinfo_i->rows * fpinfo_o->rows;
and somewhere down in the code
run_cost += nrows * join_cost.per_tuple;
It assumes that the join conditions are run on the cross-product of
the joining tables. In reality that never happens for large tables. In
such cases the optimizer will choose either hash or merge join, which
will apply join conditions only on a small portion of cross-product.
But the reason it was written that way was the local server can not
estimate the fraction of cross product on which the join conditions
will be applied. May be we could assume that the join conditions will
be applied to only 1% of the cross product, i.e. run_cost +=
clamp_rows(nrows/100) * join_cost.per_tuple. With this change I think
the cost of remote plan will be less than local plan.
Here's a preview of blog, I am planning to publish soon, about this
issue at [1]. It has a bit more details.
[1]
https://www.blogger.com/blogger.g?blogID=5253679863234367862#editor/target=post;postID=4019325618679658571;onPublishedMenu=allposts;onClosedMenu=allposts;postNum=0;src=postname
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Re: Wrong cost estimation for foreign tables join withuse_remote_estimate disabled
From
Ashutosh Bapat
Date:
Sorry here's preview link [1] [1] https://ashutoshpg.blogspot.com/b/post-preview?token=TCTIKGQBAAA.2iKpIUItkwZLkXiujvs0zad-DtDdKbwIdRFCGbac9--XbqcA-xnCdz4wmbD4hIaEHuyg5Xrz8eZq8ZNmw83yfQ.HXi__guM-7SzdIWi27QkjA&postId=4019325618679658571&type=POST On Fri, Jun 22, 2018 at 4:00 PM, Ashutosh Bapat <ashutosh.bapat@enterprisedb.com> wrote: > On Fri, Jun 22, 2018 at 11:56 AM, Konstantin Knizhnik > <k.knizhnik@postgrespro.ru> wrote: >> >> >> On 21.06.2018 20:08, Tom Lane wrote: >>> >>> Konstantin Knizhnik <k.knizhnik@postgrespro.ru> writes: >>>> >>>> The following very simple test reduce the problem with wrong cost >>>> estimation: >>>> create foreign table t1_fdw(x integer, y integer) server pg_fdw options >>>> (table_name 't1', use_remote_estimate 'false'); >>>> create foreign table t2_fdw(x integer) server pg_fdw options (table_name >>>> 't2', use_remote_estimate 'false'); >>>> It is possible to force Postgres to use correct plan by setting >>>> "fdw_startup_cost" to some very large value (100000000 for example). >>>> ... >>>> Also correct plan is used when use_remote_estimate is true. >>> >>> If you are unhappy about the results with use_remote_estimate off, don't >>> run it that way. The optimizer does not have a crystal ball. >> >> >> As I wrote, use_remote_estimate can not be used because in this case query >> compilation time is unacceptable (10 seconds, while time of query execution >> itself is ~200msec). >> So the problem can be addressed in two ways: >> >> 1. Try to reduce time of remote estimation. I wonder why postgres_fdw sends >> so much queries to remote server. For join of two tables there are 7 >> queries. >> I suspect that for ~20 joined tables in the original query number of calls >> is more than hundred, so on wonder that it takes so much time. >> 2. Try to make optimizer make better estimation of join cost based on local >> statistic (please notice that ANALYZE is explicitly called for all foreign >> tables and number of rows in the result was correctly calculated). >> > > I think estimate_path_cost_size() is too pessimistic about how many > times the join conditions are evaluated (Sorry, I have written that > code when I was worked on join pushdown for postgres_fdw.) > > /* Estimate of number of rows in cross product */ > nrows = fpinfo_i->rows * fpinfo_o->rows; > > and somewhere down in the code > run_cost += nrows * join_cost.per_tuple; > > It assumes that the join conditions are run on the cross-product of > the joining tables. In reality that never happens for large tables. In > such cases the optimizer will choose either hash or merge join, which > will apply join conditions only on a small portion of cross-product. > But the reason it was written that way was the local server can not > estimate the fraction of cross product on which the join conditions > will be applied. May be we could assume that the join conditions will > be applied to only 1% of the cross product, i.e. run_cost += > clamp_rows(nrows/100) * join_cost.per_tuple. With this change I think > the cost of remote plan will be less than local plan. > > Here's a preview of blog, I am planning to publish soon, about this > issue at [1]. It has a bit more details. > > [1] https://www.blogger.com/blogger.g?blogID=5253679863234367862#editor/target=post;postID=4019325618679658571;onPublishedMenu=allposts;onClosedMenu=allposts;postNum=0;src=postname > > -- > Best Wishes, > Ashutosh Bapat > EnterpriseDB Corporation > The Postgres Database Company -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company
Re: Wrong cost estimation for foreign tables join withuse_remote_estimate disabled
From
Konstantin Knizhnik
Date:
On 22.06.2018 13:30, Ashutosh Bapat wrote: > On Fri, Jun 22, 2018 at 11:56 AM, Konstantin Knizhnik > <k.knizhnik@postgrespro.ru> wrote: >> >> On 21.06.2018 20:08, Tom Lane wrote: >>> Konstantin Knizhnik <k.knizhnik@postgrespro.ru> writes: >>>> The following very simple test reduce the problem with wrong cost >>>> estimation: >>>> create foreign table t1_fdw(x integer, y integer) server pg_fdw options >>>> (table_name 't1', use_remote_estimate 'false'); >>>> create foreign table t2_fdw(x integer) server pg_fdw options (table_name >>>> 't2', use_remote_estimate 'false'); >>>> It is possible to force Postgres to use correct plan by setting >>>> "fdw_startup_cost" to some very large value (100000000 for example). >>>> ... >>>> Also correct plan is used when use_remote_estimate is true. >>> If you are unhappy about the results with use_remote_estimate off, don't >>> run it that way. The optimizer does not have a crystal ball. >> >> As I wrote, use_remote_estimate can not be used because in this case query >> compilation time is unacceptable (10 seconds, while time of query execution >> itself is ~200msec). >> So the problem can be addressed in two ways: >> >> 1. Try to reduce time of remote estimation. I wonder why postgres_fdw sends >> so much queries to remote server. For join of two tables there are 7 >> queries. >> I suspect that for ~20 joined tables in the original query number of calls >> is more than hundred, so on wonder that it takes so much time. >> 2. Try to make optimizer make better estimation of join cost based on local >> statistic (please notice that ANALYZE is explicitly called for all foreign >> tables and number of rows in the result was correctly calculated). >> > I think estimate_path_cost_size() is too pessimistic about how many > times the join conditions are evaluated (Sorry, I have written that > code when I was worked on join pushdown for postgres_fdw.) > > /* Estimate of number of rows in cross product */ > nrows = fpinfo_i->rows * fpinfo_o->rows; > > and somewhere down in the code > run_cost += nrows * join_cost.per_tuple; > > It assumes that the join conditions are run on the cross-product of > the joining tables. In reality that never happens for large tables. In > such cases the optimizer will choose either hash or merge join, which > will apply join conditions only on a small portion of cross-product. > But the reason it was written that way was the local server can not > estimate the fraction of cross product on which the join conditions > will be applied. May be we could assume that the join conditions will > be applied to only 1% of the cross product, i.e. run_cost += > clamp_rows(nrows/100) * join_cost.per_tuple. With this change I think > the cost of remote plan will be less than local plan. > > Here's a preview of blog, I am planning to publish soon, about this > issue at [1]. It has a bit more details. > > [1] https://www.blogger.com/blogger.g?blogID=5253679863234367862#editor/target=post;postID=4019325618679658571;onPublishedMenu=allposts;onClosedMenu=allposts;postNum=0;src=postname > Yes, postgres_fdw is very conservative or event pessimistic regarding cost of joins. It really assumes that there will be cross join with applied filter, which is not true in most cases. It's a pity. especially taken in account that based on local statistic it is able to correctly predict number of rows in the result of join. So if w take in account this estimated number of retrieved rows in calculation of join cost, then estimation is more correct and right plan (with remote joins) is chosen: diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c index 78b0f43..84b30ce 100644 --- a/contrib/postgres_fdw/postgres_fdw.c +++ b/contrib/postgres_fdw/postgres_fdw.c @@ -2812,9 +2812,8 @@ estimate_path_cost_size(PlannerInfo *root, * 4. Run time cost of applying nonpushable other clauses locally * on the result fetched from the foreign server. */ - run_cost = fpinfo_i->rel_total_cost - fpinfo_i->rel_startup_cost; - run_cost += fpinfo_o->rel_total_cost - fpinfo_o->rel_startup_cost; - run_cost += nrows * join_cost.per_tuple; + run_cost = (fpinfo_i->rel_total_cost - fpinfo_i->rel_startup_cost) * retrieved_rows / fpinfo_i->rows ; + run_cost += (fpinfo_o->rel_total_cost - fpinfo_o->rel_startup_cost) * retrieved_rows / fpinfo_o->rows; nrows = clamp_row_est(nrows * fpinfo->joinclause_sel); run_cost += nrows * remote_conds_cost.per_tuple; run_cost += fpinfo->local_conds_cost.per_tuple * retrieved_rows; I also tried to do something with first approach: speed up remote estimation. My idea was to use local estimation whenever possible and use remote estimation only for joins. In case of joining two tables it cause sending only one EXPLAIN request to remote server. But for larger number of joined table amount of considered pathes and so number of remote EXPLAIN requests is growing very fast. For 4 joins 22 explains are issued. So the approach with restricting number of foreign estimations seems to be more difficult to implement and less promising. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company