Re: Custom Scan APIs (Re: Custom Plan node) - Mailing list pgsql-hackers

> * Kouhei Kaigai (kaigai@ak.jp.nec.com) wrote:
> > I (plan to) use custom-scan of course. Once a relation is referenced
> > and optimizer decided GPU acceleration is cheaper, associated custom-
> > scan node read the data from underlying relation (or in-memory cache
> > if exists) then move to the shared memory buffer to deliver GPU
> > management background worker that launches asynchronous DMA one by one.
> > After that, custom-scan node receives filtered records via shared-
> > memory buffer, so it can construct tuples to be returned to the upper
> > node.
>
> Alright- but have you discussed this with Robert?  We're going to be
> whacking things around for parallel support with new nodes and more built-in
> helper functionality for doing this work and I'm not anxious to have
> CustomScan end up being a legacy interface that we're required to pull
> forward because we accepted it before things had settled.
>
I had briefly introduced him my idea using GPU at Ottawa last year,
even though I'm not certain he remembered it.
At least, idea of custom-scan node came from the discussion at that
time.

> > The custom-scan API is thin abstraction towards the plan node
> > interface, not tightly convinced with a particular use case, like GPU,
> > remote-join and so on. So, I'm quite optimistic for the future
> maintainability.
>
> I don't see how you can be when there hasn't been any discussion that I've
> seen about how parallel query execution is going to change things for us.
>
If parallel query execution changes whole of the structure of plan nodes,
it will also affect to the interface of custom-scan because it is a thin-
abstraction of plan-node. However, if parallel execution feature is
implemented as one of new plan node in addition to existing one, I cannot
imagine a scenario that affects to the structure of another plan node.

> > Also, please remind the discussion at the last developer meeting.
> > The purpose of custom-scan (we didn't name it at that time) is to
> > avoid unnecessary project branch for people who want to implement
> > their own special feature but no facilities to enhance
> > optimizer/executor are supported.
> > Even though we have in-core parallel execution feature by CPU, it also
> > makes sense to provide some unique implementation that may be suitable
> > for a specific region.
>
> The issue here is that we're going to be expected to maintain an interface
> once we provide it and so that isn't something we should be doing lightly.
> Particularly when it's as involved as this kind of change is with what's
> going on in the backend where we are nearly 100% sure to be changing things
> in the next release or two.
>
FDW APIs are also revised several times in the recent releases. If we can
design "perfect" interface from the beginning, it's best but usually hard
to design.
Also, custom-scan interface is almost symmetric with existing plan node
structures, so its stability is relatively high, I think.

Thanks,
--
NEC OSS Promotion Center / PG-Strom Project
KaiGai Kohei <kaigai@ak.jp.nec.com>