Re: Resource cleanup callbacks for foreign data wrappers - Mailing list pgsql-hackers

From Haribabu Kommi
Subject Re: Resource cleanup callbacks for foreign data wrappers
Date
Msg-id CAJrrPGcTWp8tLnRrBK+3-E0Onj+CBPy=MVkvSkMg84m1AqNgnw@mail.gmail.com
Whole thread Raw
In response to Resource cleanup callbacks for foreign data wrappers  (Mikhail Bautin <mbautinpgsql@gmail.com>)
List pgsql-hackers

On Tue, Oct 30, 2018 at 11:01 AM Mikhail Bautin <mbautinpgsql@gmail.com> wrote:
Hello All,

We are using the foreign data wrapper API to integrate YugaByte DB (https://github.com/yugabyte/yugabyte-db) with PostgreSQL, so scanning tables stored in YugaByte DB is being done through Foreign Scan callbacks. These callbacks in turn read data from the underlying YugaByte DB cluster and maintain reference counts on RPC-related resources needed for that. Currently we are using PostgreSQL 10.4 codebase for this integration. I am trying to figure out what is the right way for foreign data wrappers to ensure resources used in a query are freed in all cases. It looks like the EndForeignScan callback is not being called in case an error happens during expression evaluation, such as during this query from int2.sql:

SELECT '' AS five, i.f1, i.f1 * int2 '2' AS x FROM INT2_TBL i;
ERROR:  smallint out of range

For the error-free case EndForeignScan appears on this codepath:

ExecEndForeignScan (postgres/src/backend/executor/nodeForeignscan.c:293)
ExecEndNode (postgres/src/backend/executor/execProcnode.c:649)
ExecEndPlan (postgres/src/backend/executor/execMain.c:1607)
standard_ExecutorEnd (postgres/src/backend/executor/execMain.c:495)
ExecutorEnd (postgres/src/backend/executor/execMain.c:466)
PortalCleanup (postgres/src/backend/commands/portalcmds.c:302)
PortalDrop (postgres/src/backend/utils/mmgr/portalmem.c:489)
exec_simple_query (postgres/src/backend/tcop/postgres.c:1118)
PostgresMain (postgres/src/backend/tcop/postgres.c:4126)
BackendRun (postgres/src/backend/postmaster/postmaster.c:4411)
BackendStartup (postgres/src/backend/postmaster/postmaster.c:4083)
ServerLoop (postgres/src/backend/postmaster/postmaster.c:1758)
PostmasterMain (postgres/src/backend/postmaster/postmaster.c:1366)
PostgresServerProcessMain (postgres/src/backend/main/main.c:228)

However, I see that ExecutorEnd is not being executed at all in PortalCleanup in portalcmds.c in case portal->status is PORTAL_FAILED. If this is by design, what would be the right callback mechanism for foreign data wrappers to ensure resources are freed?Would RegisterResourceReleaseCallback from resowner.h be more appropriate for this use case? I only see that being used for OpenSSL-related resources at the moment.

Calling EndForeignScan in error scenario is not possible because of the
PostgreSQL design, as it aborts all the processing further do the cleanup.

You can register the transaction callback function and try to close the remote
connection as it is done in postgres_fdw module GetConnection() function.

Instead of inventing new resource owner callback, transaction callback may
solve your problem.

Regards,
Haribabu Kommi
Fujitsu Australia

pgsql-hackers by date:

Previous
From: Darafei "Komяpa" Praliaskouski
Date:
Subject: Re: INSTALL file
Next
From: Andres Freund
Date:
Subject: global / super barriers (for checksums)