Re: Foreign join pushdown vs EvalPlanQual - Mailing list pgsql-hackers

> -----Original Message-----
> From: Etsuro Fujita [mailto:fujita.etsuro@lab.ntt.co.jp]
> Sent: Thursday, October 15, 2015 7:00 PM
> To: Kaigai Kouhei(海外 浩平); Kyotaro HORIGUCHI
> Cc: pgsql-hackers@postgresql.org; shigeru.hanada@gmail.com;
> robertmhaas@gmail.com
> Subject: Re: [HACKERS] Foreign join pushdown vs EvalPlanQual
>
> On 2015/10/15 11:36, Kouhei Kaigai wrote:
> >>> Once again, if FDW driver is responsible to construct join-tuple from
> >>> the base relation's tuple cached in EPQ slot, this case don't need to
> >>> kick remote query again, because all the materials to construct join-
> >>> tuple are already held locally. Right?
>
> I now understand clearly what you mean.  Sorry for my misunderstanding.
>
> > In case of scanrelid==0, expectation to ForeignScan/CustomScan is to
> > behave as if local join exists here. It requires ForeignScan to generate
> > joined-tuple as a result of remote join, that may contains multiple junk
> > TLEs to carry whole-var references of base foreign tables.
> > According to the criteria, the desirable behavior is clear as below:
> >
> > 1. FDW/CSP picks up base relation's tuple from the EPQ slots.
> >     It shall be setup by whole-row reference if earlier row-lock semantics,
> >     or by RefetchForeignRow if later row-lock semantics.
> >
> > 2. Fill up ss_ScanTupleSlot according to the xxx_scan_tlist.
> >     We may be able to provide a common support function here, because this
> >     list keeps relation between a particular attribute of the joined-tuple
> >     and its source column.
> >
> > 3. Apply join-clause and base-restrict that were pushed down.
> >     setrefs.c initializes expressions kept in fdw_exprs/custom_exprs to run
> >     on the ss_ScanTupleSlot. It is the easiest way to check here.
> >
> > 4. If joined-tuple is still visible after the step 3, FDW/CSP returns
> >     joined-tuple. Elsewhere, returns an empty slot.
> >
> > It is entirely compatible behavior even if local join is located on
> > the point of ForeignScan/CustomScan with scanrelid==0.
> >
> > Even if remote join is parametalized by other relation, we can simply
> > use param-info delivered from the corresponding outer scan at the step-3.
> > EState should have the parameters already updated, FDW driver needs to
> > care about nothing.
> >
> > It is quite less invasive approach towards the existing EPQ recheck
> > mechanism.
>
> I see.  That's an idea, but I guess that step 2 and 3 would need to add
> a lot of code to the core.  Why don't you use a local join execution
> plan that we discussed?  I think that that would make the series of
> processing much simpler.  I'm now revising the patch that I created for
> that.  If it's okay, I'd like to propose an updated version of the patch
> in a few days.
>
I have to introduce why above idea is simpler and suitable for v9.5
timeline.
As I've consistently proposed for this two months, the step-2 and 3
are assumed to be handled in the callback routine to be kicked from
ForeignRecheck().

Even if core backend eventually provides utility routines to support
above tasks, it is not mandatory requirement from the beginning; v9.5
timeline at least.
As long as the callback is provided, FDW driver "can" implement above
features by itself, with their comfortable way.
Note that alternative local join plan is one way to implement the above
step-2 and -3, however, I never enforce people to use a particular way.
People can choose.

Regarding to scale of the code in the core backend, it is pretty small
because all we need to add is just a callback in v9.5. We can implement
the remaining support routine in v9.6 timeline, but not now.

> > I cannot understand why Fujita-san never "try" this approach.
>
> Maybe my explanation was not correct, but I didn't say such a thing.
> What I rather objected against was to add a new FDW callback routine for
> rechecking pushed-down quals or pushed-down joins, which I think you
> insisted on.
>
My proposition has been consistent.
The interface contract (that is the job of callback implementation in
other words) in the series of sequence is above 4-steps I introduced.
We can use alternative local join plan, or own implementation to fill
up ss_ScanTupleSlot, or something common support routine provided by
core.
Regardless of the implementation choice, the callback approach minimizes
the impact towards existing EPQ recheck mechanism and release schedule of
v9.5. Also, it can cover the case handling when scanrelid==0.

Thanks,
--
NEC Business Creation Division / PG-Strom Project
KaiGai Kohei <kaigai@ak.jp.nec.com>