Re: POC: postgres_fdw insert batching - Mailing list pgsql-hackers

From Craig Ringer
Subject Re: POC: postgres_fdw insert batching
Date
Msg-id CAGRY4ny3Qqd9MXzbKFBp842So+L3a3YF+X1tUorxu4RcPxPs1Q@mail.gmail.com
Whole thread Raw
In response to Re: POC: postgres_fdw insert batching  (Tomas Vondra <tomas.vondra@enterprisedb.com>)
List pgsql-hackers
On Fri, Nov 27, 2020 at 3:34 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
 
Not sure how is this related to app developers? I think the idea was
that the libpq features might be useful between the two PostgreSQL
instances. I.e. the postgres_fdw would use the libpq batching to send
chunks of data to the other side.

Right. Or at least, when designing the FDW API, do so in a way that doesn't strictly enforce Request/Response alternation without interleaving, so you can benefit from it in the future.

It's hardly just libpq after all. A *lot* of client libraries and drivers will be capable of non-blocking reads or writes with multiple ones in flight at once. Any REST-like API generally can, for example. So for performance reasons we should if possible avoid baking the assumption that a request cannot be made until the response from the previous request is received, and instead have a wait interface to use for when a new request requires the prior response's result before it can proceed.

Well, my point was that we could keep the API, but maybe it should be
implemented using the proposed libpq batching. They could still use the
postgres_fdw example how to use the API, but the internals would need to
be different, of course.

Sure. Or just allow room for it in the FDW API, though using the pipelining support natively would be nice.

If the FDW interface allows Pg to

Insert A
Insert B
Insert C
Wait for outcome of insert A
...

then that'll be useful for using libpq pipelining, but also FDWs for all sorts of other DBs, especially cloud-y ones where latency is a big concern.


pgsql-hackers by date:

Previous
From: Fujii Masao
Date:
Subject: Re: Allow some recovery parameters to be changed with reload
Next
From: Bharath Rupireddy
Date:
Subject: Re: [PATCH] postgres_fdw connection caching - cause remote sessions linger till the local session exit