Re: Custom Scan APIs (Re: Custom Plan node) - Mailing list pgsql-hackers

* Kouhei Kaigai (kaigai@ak.jp.nec.com) wrote:
> * Stephen Frost (sfrost@snowman.net) wrote:
> > I don't see how you can be when there hasn't been any discussion that I've
> > seen about how parallel query execution is going to change things for us.
> >
> If parallel query execution changes whole of the structure of plan nodes,
> it will also affect to the interface of custom-scan because it is a thin-
> abstraction of plan-node. However, if parallel execution feature is
> implemented as one of new plan node in addition to existing one, I cannot
> imagine a scenario that affects to the structure of another plan node.

Let's just say that I have doubts that we'll be able to implement
parallel execution *without* changing the plan node interface in some
way which will require, hopefully minor, changes to all of the nodes.
The issue is that even a minor change would break the custom-scan API
and we'd immediately be in the boat of dealing with complaints regarding
backwards compatibility.  Perhaps we can hand-wave that, and we've had
some success changing hook APIs between major releases, but such changes
may also be in ways which wouldn't break in obvious ways or even
possibly be changes which have to be introduced into back-branches.
Parallel query is going to be brand-new real soon and it's reasonable to
think we'll need to make bug-fix changes to it after it's out which
might even involve changes to the API which is developed for it.

> > The issue here is that we're going to be expected to maintain an interface
> > once we provide it and so that isn't something we should be doing lightly.
> > Particularly when it's as involved as this kind of change is with what's
> > going on in the backend where we are nearly 100% sure to be changing things
> > in the next release or two.
> >
> FDW APIs are also revised several times in the recent releases. If we can
> design "perfect" interface from the beginning, it's best but usually hard
> to design.

Sure, but FDWs also have a *much* broader set of use-cases, in my view,
which is also why I was pushing to work on join-push-down to happen
there instead of having this kind of a hook interface, which I don't
think we'd want to directly expose as part of the 'official FDW API' as
it ends up putting all the work on the FDW with little aide, making it
terribly likely to end up with a bunch of duplciated code in the FDWs
from the backend to deal with it, particularly for individuals writing
FDWs who aren't familiar with what PG already has.

> Also, custom-scan interface is almost symmetric with existing plan node
> structures, so its stability is relatively high, I think.

Perhaps it will come to pass that parallel query execution doesn't
require any changes to the plan node structure, but that's not the horse
that I'd bet on at this point.
Thanks,
    Stephen