Thread: inherit support for foreign tables

inherit support for foreign tables

From
Shigeru Hanada
Date:
Hi hackers,

I'd like to propose adding inheritance support for foriegn tables.
David Fetter mentioned this feature last July, but it seems stalled.

    http://www.postgresql.org/message-id/20130719005601.GA5760@fetter.org

Supporting inheritance by foreign tables allows us to distribute query
to remote servers by using foreign tables as partition table of a
(perhaps ordinary) table.  For this purpose, I think that constraint
exclusion is necessary.

As result of extending Devid's patch for PoC, and AFAIS we need these changes:

1) Add INHERITS(rel, ...) clause to CREATE/ALTER FOREIGN TABLE
Apperantly we need to add new syntax to define parent table(s) of a
foreign table.  We have options about the position of INHERIT clause,
but I'd prefer before SERVER clause because having options specific to
foreign tables at the tail would be most extensible.

    a) CREATE FOREIGN TABLE child (...) INHERITS(p1, p2) SERVER server;
    b) CREATE FOREIGN TABLE child (...) SERVER server INHERITS(p1, p2);

2) Allow foreign tables to have CHECK constraints
Like NOT NULL, I think we don't need to enforce the check duroing
INSERT/UPDATE against foreign table.

3) Allow foreign table as a child node of Append
Currently prepunion.c assumes that children of Append have
RELKIND_RELATION as relkind always, so we need to set relkind of child
RTE explicitly.

Please see attached PoC patch.  I'll enhance implementation, tests and
document and submit the patch for the next CF.

Regards,
--
Shigeru HANADA

Attachment

Re: inherit support for foreign tables

From
Tom Lane
Date:
Shigeru Hanada <shigeru.hanada@gmail.com> writes:
> I'd like to propose adding inheritance support for foriegn tables.
> David Fetter mentioned this feature last July, but it seems stalled.
>   http://www.postgresql.org/message-id/20130719005601.GA5760@fetter.org

The discussion there pointed out that not enough consideration had been
given to interactions with other commands.  I'm not really satisfied
with your analysis here.  In particular:

> 2) Allow foreign tables to have CHECK constraints
> Like NOT NULL, I think we don't need to enforce the check duroing
> INSERT/UPDATE against foreign table.

Really?  It's one thing to say that somebody who adds a CHECK constraint
to a foreign table is responsible to make sure that the foreign data will
satisfy the constraint.  It feels like a different thing to say that ALTER
TABLE ADD CONSTRAINT applied to a parent table will silently assume that
some child table that happens to be foreign doesn't need any enforcement.

Perhaps more to the point, inheritance trees are the main place where the
planner depends on the assumption that CHECK constraints represent
reality.  Are we really prepared to say that it's the user's fault if the
planner generates an incorrect plan on the strength of a CHECK constraint
that's not actually satisfied by the foreign data?  If so, that had better
be documented by this patch.  But for a project that refuses to let people
create a local CHECK or FOREIGN KEY constraint without mechanically
checking it, it seems pretty darn weird to be so laissez-faire about
constraints on foreign data.
        regards, tom lane



Re: inherit support for foreign tables

From
Robert Haas
Date:
On Thu, Nov 14, 2013 at 12:28 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> 2) Allow foreign tables to have CHECK constraints
>> Like NOT NULL, I think we don't need to enforce the check duroing
>> INSERT/UPDATE against foreign table.
>
> Really?  It's one thing to say that somebody who adds a CHECK constraint
> to a foreign table is responsible to make sure that the foreign data will
> satisfy the constraint.  It feels like a different thing to say that ALTER
> TABLE ADD CONSTRAINT applied to a parent table will silently assume that
> some child table that happens to be foreign doesn't need any enforcement.
>
> Perhaps more to the point, inheritance trees are the main place where the
> planner depends on the assumption that CHECK constraints represent
> reality.  Are we really prepared to say that it's the user's fault if the
> planner generates an incorrect plan on the strength of a CHECK constraint
> that's not actually satisfied by the foreign data?  If so, that had better
> be documented by this patch.  But for a project that refuses to let people
> create a local CHECK or FOREIGN KEY constraint without mechanically
> checking it, it seems pretty darn weird to be so laissez-faire about
> constraints on foreign data.

I can see both sides of this issue.  We certainly have no way to force
the remote side to enforce CHECK constraints defined on the local
side, because the remote side could also be accepting writes from
other sources that don't have any matching constraint. But having said
that, I can't see any particularly principled reason why we shouldn't
at least check the new rows we insert ourselves.  After all, we could
be in the situation proposed by KaiGai Kohei, where the foreign data
wrapper API is being used as a surrogate storage engine API - i.e.
there are no writers to the foreign side except ourselves.  In that
situation, it would seem odd to randomly fail to enforce the
constraints.

On the other hand, the performance costs of checking every row bound
for the remote table could be quite steep.  Consider an update on an
inheritance hierarchy that sets a = a + 1 for every row.  If we don't
worry about verifying that the resulting rows satisfy all local-side
constraints, we can potentially ship a single update statement to the
remote server and let it do all the work there.  But if we DO have to
worry about that, then we're going to have to ship every updated row
over the wire in at least one direction, if not both.  If the purpose
of adding CHECK constraints was to enable constraint exclusion, that's
a mighty steep price to pay for it.

I think it's been previously proposed that we have some version of a
CHECK constraint that effectively acts as an assertion for query
optimization purposes, but isn't actually enforced by the system.  I
can see that being useful in a variety of situations, including this
one.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: inherit support for foreign tables

From
Tom Lane
Date:
Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, Nov 14, 2013 at 12:28 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> 2) Allow foreign tables to have CHECK constraints
>>> Like NOT NULL, I think we don't need to enforce the check duroing
>>> INSERT/UPDATE against foreign table.

>> Really?

> I think it's been previously proposed that we have some version of a
> CHECK constraint that effectively acts as an assertion for query
> optimization purposes, but isn't actually enforced by the system.  I
> can see that being useful in a variety of situations, including this
> one.

Yeah, I think it would be much smarter to provide a different syntax
to explicitly represent the notion that we're only assuming the condition
is true, and not trying to enforce it.
        regards, tom lane



Re: inherit support for foreign tables

From
Shigeru Hanada
Date:
2013/11/18 Tom Lane <tgl@sss.pgh.pa.us>:
> Robert Haas <robertmhaas@gmail.com> writes:
>> I think it's been previously proposed that we have some version of a
>> CHECK constraint that effectively acts as an assertion for query
>> optimization purposes, but isn't actually enforced by the system.  I
>> can see that being useful in a variety of situations, including this
>> one.
>
> Yeah, I think it would be much smarter to provide a different syntax
> to explicitly represent the notion that we're only assuming the condition
> is true, and not trying to enforce it.

I'd like to revisit this feature.

Explicit notation to represent not-enforcing (assertive?) is an
interesting idea.  I'm not sure which word is appropriate, but for
example, let's use the word ASSERTIVE as a new constraint attribute.

CREATE TABLE parent (   id int NOT NULL,   group int NOT NULL,   name text
);

CREATE FOREIGN TABLE child_grp1 ( /* no additional column */
) INHERITS (parent) SERVER server1;
ALTER  TABLE child_grp1 ADD CONSTRAINT chk_group1 CHECK (group = 1) ASSERTIVE;

If ASSERTIVE is specified, it's not guaranteed that the constraint is
enforced completely, so it should be treated as a hint for planner.
As Robert mentioned, enforcing as much as we can during INSERT/UPDATE
is one option about this issue.

In addition, an idea which I can't throw away is to assume that all
constraints defined on foreign tables as ASSERTIVE.  Foreign tables
potentially have dangers to have "wrong" data by updating source data
not through foreign tables.  This is not specific to an FDW, so IMO
constraints defined on foreign tables are basically ASSERTIVE.  Of
course PG can try to maintain data correct, but always somebody might
break it.

Besides CHECK constraints, currently NOT NULL constraints are
virtually ASSERTIVE (not enforcing).  Should it also be noted
explicitly?

Thoughts?

--
Shigeru HANADA



Inheritance and indexes

From
knizhnik
Date:
From PostgreSQL manual:
"A serious limitation of the inheritance feature is that indexes 
(including unique constraints) and foreign key constraints only apply to 
single tables, not to their inheritance children."

But is it possible to use index for derived table at all?
Why sequential search is used for derived table in the example below:

create table base_table (x integer primary key);
create table derived_table (y integer) inherits (base_table);
insert into base_table values (1);
insert into derived_table values (2,2);
create index derived_index on derived_table(x);
explain select * from base_table where x>=0;                                          QUERY PLAN
---------------------------------------------------------------------------------------------- Append  (cost=0.14..4.56
rows=81width=4)   ->  Index Only Scan using base_table_pkey on base_table 
 
(cost=0.14..3.55 rows=80 width=4)         Index Cond: (x >= 0)   ->  Seq Scan on derived_table  (cost=0.00..1.01 rows=1
width=4)        Filter: (x >= 0)
 
(5 rows)




Re: Inheritance and indexes

From
Marti Raudsepp
Date:
On Tue, Jan 14, 2014 at 12:07 PM, knizhnik <knizhnik@garret.ru> wrote:
> But is it possible to use index for derived table at all?

Yes, the planner will do an index scan when it makes sense.

> Why sequential search is used for derived table in the example below:

> insert into derived_table values (2,2);
> create index derived_index on derived_table(x);
> explain select * from base_table where x>=0;

With only 1 row in the table, the planner decides there's no point in
scanning the index. Try with more realistic data.

Regards,
Marti



Re: inherit support for foreign tables

From
Shigeru Hanada
Date:
Hi all,

2014/1/14 Shigeru Hanada <shigeru.hanada@gmail.com>:
> I'd like to revisit this feature.

Attached patch allows a foreign table to be a child of a table.  It
also allows foreign tables to have CHECK constraints.  These changes
provide us a chance to propagate query load to multiple servers via
constraint exclusion.  If FDW supports async operation against remote
server, parallel processing (not stable but read-only case would be
find) can be achieved, though overhead of FDW mechanism is still
there.

Though this would be debatable, in current implementation, constraints
defined on a foreign table (now only NOT NULL and CHECK are supported)
are not enforced during INSERT or UPDATE executed against foreign
tables.  This means that retrieved data might violates the constraints
defined on local side.  This is debatable issue because integrity of
data is important for DBMS, but in the first cut, it is just
documented as a note.

Because I don't see practical case to have a foreign table as a
parent, and it avoid a problem about recursive ALTER TABLE operation,
foreign tables can't be a parent.  An example of such problem is
adding constraint which is not unsupported for foreign tables to the
parent of foreign table.  Propagated operation can be applied to
ordinary tables in the inheritance tree, but can't be to foreign
tables.  If we allow foreign tables to be parent, it's difficult to
process ordinary tables below foreign tables in current traffic cop
mechanism.

For other commands recursively processed such as ANALYZE, foreign
tables in the leaf of inheritance tree are ignored.

Any comments or questions are welcome.
--
Shigeru HANADA

Attachment

Re: inherit support for foreign tables

From
Jim Nasby
Date:
On 11/18/13, 8:36 AM, Robert Haas wrote:
> On the other hand, the performance costs of checking every row bound
> for the remote table could be quite steep.  Consider an update on an
> inheritance hierarchy that sets a = a + 1 for every row.  If we don't
> worry about verifying that the resulting rows satisfy all local-side
> constraints, we can potentially ship a single update statement to the
> remote server and let it do all the work there.  But if we DO have to
> worry about that, then we're going to have to ship every updated row
> over the wire in at least one direction, if not both.  If the purpose
> of adding CHECK constraints was to enable constraint exclusion, that's
> a mighty steep price to pay for it.

A sophisticated enough FDW could verify that the appropriate check already existed in tho foreign side, or it could do
somethinglike:
 

BEGIN;
UPDATE SET ... WHERE <where>
SELECT EXISTS( SELECT 1 WHERE <where> AND NOT (<check condition>) );

And then rollback if the SELECT returns true.

But obviously you can't always do that, so I think there's a place for both true constraints and "suggested
constraints".
-- 
Jim C. Nasby, Data Architect                       jim@nasby.net
512.569.9461 (cell)                         http://jim.nasby.net



Re: inherit support for foreign tables

From
KaiGai Kohei
Date:
(2014/01/14 18:24), Shigeru Hanada wrote:
> 2013/11/18 Tom Lane <tgl@sss.pgh.pa.us>:
>> Robert Haas <robertmhaas@gmail.com> writes:
>>> I think it's been previously proposed that we have some version of a
>>> CHECK constraint that effectively acts as an assertion for query
>>> optimization purposes, but isn't actually enforced by the system.  I
>>> can see that being useful in a variety of situations, including this
>>> one.
>>
>> Yeah, I think it would be much smarter to provide a different syntax
>> to explicitly represent the notion that we're only assuming the condition
>> is true, and not trying to enforce it.
>
> I'd like to revisit this feature.
>
> Explicit notation to represent not-enforcing (assertive?) is an
> interesting idea.  I'm not sure which word is appropriate, but for
> example, let's use the word ASSERTIVE as a new constraint attribute.
>
> CREATE TABLE parent (
>      id int NOT NULL,
>      group int NOT NULL,
>      name text
> );
>
> CREATE FOREIGN TABLE child_grp1 (
>    /* no additional column */
> ) INHERITS (parent) SERVER server1;
> ALTER  TABLE child_grp1 ADD CONSTRAINT chk_group1 CHECK (group = 1) ASSERTIVE;
>
> If ASSERTIVE is specified, it's not guaranteed that the constraint is
> enforced completely, so it should be treated as a hint for planner.
> As Robert mentioned, enforcing as much as we can during INSERT/UPDATE
> is one option about this issue.
>
> In addition, an idea which I can't throw away is to assume that all
> constraints defined on foreign tables as ASSERTIVE.  Foreign tables
> potentially have dangers to have "wrong" data by updating source data
> not through foreign tables.  This is not specific to an FDW, so IMO
> constraints defined on foreign tables are basically ASSERTIVE.  Of
> course PG can try to maintain data correct, but always somebody might
> break it.
> qu
>
Does it make sense to apply "assertive" CHECK constraint on the qual
of ForeignScan to filter out tuples with violated values at the local
side, as if row-level security feature doing.
It enables to handle a situation that planner expects only "clean"
tuples are returned but FDW driver is unavailable to anomalies.

Probably, this additional check can be turned on/off on the fly,
if FDW driver has a way to inform the core system its capability,
like FDW_CAN_ENFORCE_CHECK_CONSTRAINT that informs planner to skip
local checks.

> Besides CHECK constraints, currently NOT NULL constraints are
> virtually ASSERTIVE (not enforcing).  Should it also be noted
> explicitly?
>
Backward compatibility....

NOT NULL [ASSERTIVE] might be an option.

Thanks,
-- 
OSS Promotion Center / The PG-Strom Project
KaiGai Kohei <kaigai@ak.jp.nec.com>



Re: inherit support for foreign tables

From
Shigeru Hanada
Date:
Thanks for the comments.

2014/1/21 KaiGai Kohei <kaigai@ak.jp.nec.com>:
>> In addition, an idea which I can't throw away is to assume that all
>> constraints defined on foreign tables as ASSERTIVE.  Foreign tables
>> potentially have dangers to have "wrong" data by updating source data
>> not through foreign tables.  This is not specific to an FDW, so IMO
>> constraints defined on foreign tables are basically ASSERTIVE.  Of
>> course PG can try to maintain data correct, but always somebody might
>> break it.
>> qu
>>
> Does it make sense to apply "assertive" CHECK constraint on the qual
> of ForeignScan to filter out tuples with violated values at the local
> side, as if row-level security feature doing.
> It enables to handle a situation that planner expects only "clean"
> tuples are returned but FDW driver is unavailable to anomalies.
>
> Probably, this additional check can be turned on/off on the fly,
> if FDW driver has a way to inform the core system its capability,
> like FDW_CAN_ENFORCE_CHECK_CONSTRAINT that informs planner to skip
> local checks.

Hmm, IIUC you mean that local users can't (or don't need to) know that
data which violates the local constraints exist on remote side.
Applying constraints to the data which is modified through FDW would
be necessary as well.  In that design, FDW is a bidirectional filter
which provides these features:

1) Don't push wrong data into remote data source, by applying local
constraints to the result of the modifying query executed on local PG.This is not perfect filter, because remote
constraintsdon't mapped 
automatically or perfectly (imagine constraints which is available on
remote but is not supported in PG).
2) Don't retrieve wrong data from remote to local PG, by applying
local constraints

I have a concern about consistency.  It has not been supported, but
let's think of Aggregate push-down invoked by a query below.

SELECT count(*) FROM remote_table;

If this query was fully pushed down, the result is the # of records
exist on remote side, but the result would be # of valid records when
we don't push down the aggregate.  This would confuse users.

>> Besides CHECK constraints, currently NOT NULL constraints are
>> virtually ASSERTIVE (not enforcing).  Should it also be noted
>> explicitly?
>>
> Backward compatibility….

Yep, backward compatibility (especially visible ones to users) should
be minimal, ideally zero.

> NOT NULL [ASSERTIVE] might be an option.

Treating [ASSERTIVE | NOT ASSERTIVE] like DEFERRABLE, and allow
ingASSERTIVE for only foreign tables?  It makes sense, though we need
consider exclusiveness .  But It needs to default to ASSERTIVE on
foreign tables, and NOT ASSERTIVE (means "forced") on others.  Isn't
is too complicated?

CREATE FOREIGN TABLE foo (   id int NOT NULL ASSERTIVE CHECK (id > 1) ASSERTIVE,   …   CONSTRAINT chk_foo_name_upper
CHECK(upper(name) = name) ASSERTIVE 
) SERVER server;

BTW, I noticed that this is like push-down-able expressions in
JOIN/WHERE.  We need to check a CHECK constraint defined on a foreign
tables contains only expressions which have same semantics as remote
side (in practice, built-in and immutable)?
--
Shigeru HANADA



Re: inherit support for foreign tables

From
KaiGai Kohei
Date:
(2014/01/21 11:44), Shigeru Hanada wrote:
> Thanks for the comments.
>
> 2014/1/21 KaiGai Kohei <kaigai@ak.jp.nec.com>:
>>> In addition, an idea which I can't throw away is to assume that all
>>> constraints defined on foreign tables as ASSERTIVE.  Foreign tables
>>> potentially have dangers to have "wrong" data by updating source data
>>> not through foreign tables.  This is not specific to an FDW, so IMO
>>> constraints defined on foreign tables are basically ASSERTIVE.  Of
>>> course PG can try to maintain data correct, but always somebody might
>>> break it.
>>> qu
>>>
>> Does it make sense to apply "assertive" CHECK constraint on the qual
>> of ForeignScan to filter out tuples with violated values at the local
>> side, as if row-level security feature doing.
>> It enables to handle a situation that planner expects only "clean"
>> tuples are returned but FDW driver is unavailable to anomalies.
>>
>> Probably, this additional check can be turned on/off on the fly,
>> if FDW driver has a way to inform the core system its capability,
>> like FDW_CAN_ENFORCE_CHECK_CONSTRAINT that informs planner to skip
>> local checks.
>
> Hmm, IIUC you mean that local users can't (or don't need to) know that
> data which violates the local constraints exist on remote side.
> Applying constraints to the data which is modified through FDW would
> be necessary as well.  In that design, FDW is a bidirectional filter
> which provides these features:
>
> 1) Don't push wrong data into remote data source, by applying local
> constraints to the result of the modifying query executed on local PG.
>   This is not perfect filter, because remote constraints don't mapped
> automatically or perfectly (imagine constraints which is available on
> remote but is not supported in PG).
> 2) Don't retrieve wrong data from remote to local PG, by applying
> local constraints
>
Yes. (1) can be done with ExecConstraints prior to FDW callback on
UPDATE or INSERT, even not a perfect solution because of side-channel
on the remote data source. For (2), my proposition tries to drop
retrieved violated tuples, however, the result is same.

> I have a concern about consistency.  It has not been supported, but
> let's think of Aggregate push-down invoked by a query below.
>
> SELECT count(*) FROM remote_table;
>
> If this query was fully pushed down, the result is the # of records
> exist on remote side, but the result would be # of valid records when
> we don't push down the aggregate.  This would confuse users.
>
Hmm. In this case, FDW driver needs to be responsible to push-down
the additional check quals into remote side, so it does not work
transparently towards the ForeignScan.
It might be a little bit complicated suggestion for the beginning
of the efforts.

>>> Besides CHECK constraints, currently NOT NULL constraints are
>>> virtually ASSERTIVE (not enforcing).  Should it also be noted
>>> explicitly?
>>>
>> Backward compatibility….
>
> Yep, backward compatibility (especially visible ones to users) should
> be minimal, ideally zero.
>
>> NOT NULL [ASSERTIVE] might be an option.
>
> Treating [ASSERTIVE | NOT ASSERTIVE] like DEFERRABLE, and allow
> ingASSERTIVE for only foreign tables?  It makes sense, though we need
> consider exclusiveness .  But It needs to default to ASSERTIVE on
> foreign tables, and NOT ASSERTIVE (means "forced") on others.  Isn't
> is too complicated?
>
I think it is not easy to implement assertive checks, except for
foreign tables, because all the write stuff to regular tables are
managed by PostgreSQL itself.
So, it is a good first step to add support "ASSERTIVE" CHECK on
foreign table only, and to enforce FDW drivers nothing special
from my personal sense.

How about committer's opinion?

Thanks,
-- 
OSS Promotion Center / The PG-Strom Project
KaiGai Kohei <kaigai@ak.jp.nec.com>



Re: inherit support for foreign tables

From
Robert Haas
Date:
On Mon, Jan 20, 2014 at 9:44 PM, Shigeru Hanada
<shigeru.hanada@gmail.com> wrote:
> Thanks for the comments.
>
> 2014/1/21 KaiGai Kohei <kaigai@ak.jp.nec.com>:
>>> In addition, an idea which I can't throw away is to assume that all
>>> constraints defined on foreign tables as ASSERTIVE.  Foreign tables
>>> potentially have dangers to have "wrong" data by updating source data
>>> not through foreign tables.  This is not specific to an FDW, so IMO
>>> constraints defined on foreign tables are basically ASSERTIVE.  Of
>>> course PG can try to maintain data correct, but always somebody might
>>> break it.
>>> qu
>>>
>> Does it make sense to apply "assertive" CHECK constraint on the qual
>> of ForeignScan to filter out tuples with violated values at the local
>> side, as if row-level security feature doing.
>> It enables to handle a situation that planner expects only "clean"
>> tuples are returned but FDW driver is unavailable to anomalies.
>>
>> Probably, this additional check can be turned on/off on the fly,
>> if FDW driver has a way to inform the core system its capability,
>> like FDW_CAN_ENFORCE_CHECK_CONSTRAINT that informs planner to skip
>> local checks.
>
> Hmm, IIUC you mean that local users can't (or don't need to) know that
> data which violates the local constraints exist on remote side.
> Applying constraints to the data which is modified through FDW would
> be necessary as well.  In that design, FDW is a bidirectional filter
> which provides these features:
>
> 1) Don't push wrong data into remote data source, by applying local
> constraints to the result of the modifying query executed on local PG.
>  This is not perfect filter, because remote constraints don't mapped
> automatically or perfectly (imagine constraints which is available on
> remote but is not supported in PG).
> 2) Don't retrieve wrong data from remote to local PG, by applying
> local constraints
>
> I have a concern about consistency.  It has not been supported, but
> let's think of Aggregate push-down invoked by a query below.
>
> SELECT count(*) FROM remote_table;
>
> If this query was fully pushed down, the result is the # of records
> exist on remote side, but the result would be # of valid records when
> we don't push down the aggregate.  This would confuse users.
>
>>> Besides CHECK constraints, currently NOT NULL constraints are
>>> virtually ASSERTIVE (not enforcing).  Should it also be noted
>>> explicitly?
>>>
>> Backward compatibility….
>
> Yep, backward compatibility (especially visible ones to users) should
> be minimal, ideally zero.
>
>> NOT NULL [ASSERTIVE] might be an option.
>
> Treating [ASSERTIVE | NOT ASSERTIVE] like DEFERRABLE, and allow
> ingASSERTIVE for only foreign tables?  It makes sense, though we need
> consider exclusiveness .  But It needs to default to ASSERTIVE on
> foreign tables, and NOT ASSERTIVE (means "forced") on others.  Isn't
> is too complicated?
>
> CREATE FOREIGN TABLE foo (
>     id int NOT NULL ASSERTIVE CHECK (id > 1) ASSERTIVE,
>     …
>     CONSTRAINT chk_foo_name_upper CHECK (upper(name) = name) ASSERTIVE
> ) SERVER server;
>
> BTW, I noticed that this is like push-down-able expressions in
> JOIN/WHERE.  We need to check a CHECK constraint defined on a foreign
> tables contains only expressions which have same semantics as remote
> side (in practice, built-in and immutable)?

I don't think that that ASSERTIVE is going to fly, because "assertive"
means (sayeth the Google) "having or showing a confident and forceful
personality", which is not what we mean here.  It's tempting to do
something like try to replace the keyword "check" with "assume" or
"assert" or (stretching) "assertion", but that would require whichever
one we picked to be a fully-reserved keyword, which I can't think is
going to get much support here, for entirely understandable reasons.
So I think we should look for another option.

Currently, constraints can be marked NO INHERIT (though this seems to
have not been fully documented, as the ALTER TABLE page doesn't
mention it anywhere) or NOT VALID, so I'm thinking maybe we should go
with something along those lines.  Some ideas:

- NO CHECK.  The idea of writing CHECK (id > 1) NO CHECK is pretty
hilarious, though.
- NO VALIDATE.  But then people need to understand that NOT VALID
means "we didn't validate it yet" while "no validate" means "we don't
ever intend to validate it", which could be confusing.
- NO ENFORCE.  Requires a new (probably unreserved) keyword.
- NOT VALIDATED or NOT CHECKED.  Same problems as NO CHECK and NO
VALIDATE, respectively, plus now we have to create a new keyword.

Another idea is to apply an extensible-options syntax to constraints,
like we do for EXPLAIN, VACUUM, etc.  Like maybe:

CHECK (id > 1) OPTIONS (enforced false, valid true)

Yet another idea is to consider validity a three-state property:
either the constraint is valid (because we have checked it and are
enforcing it), or it is not valid (because we are enforcing it but
have not checked the pre-existing data), or it is assumed true
(because we are not checking or enforcing it but are believing it
anyway).  So then we could have a syntax like this:

CHECK (id > 1) VALIDATE { ON | OFF | ASSERTION }

Other ideas?

One thing that's bugging me a bit about this whole line of attack is
that, in the first instance, the whole goal here is to support
inheritance hierarchies that mix ordinary tables with foreign tables.
If you have a table with children some of which are inherited and
others of which are not inherited, you're very likely going to want
your constraints enforced for real on the children that are tables and
assumed true on the children that are foreign tables, and none of what
we're talking about here gets us to that, because we normally want the
constraints to be identical throughout the inheritance hierarchy.
Maybe there's some way around that, but I'm back to wondering if it
wouldn't be better to simply silently force any constraints on a
foreign-table into assertion mode.  That could be done without any new
syntax at all, and frankly I think it's what people are going to want
more often than not.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: inherit support for foreign tables

From
Tom Lane
Date:
Robert Haas <robertmhaas@gmail.com> writes:
> One thing that's bugging me a bit about this whole line of attack is
> that, in the first instance, the whole goal here is to support
> inheritance hierarchies that mix ordinary tables with foreign tables.
> If you have a table with children some of which are inherited and
> others of which are not inherited, you're very likely going to want
> your constraints enforced for real on the children that are tables and
> assumed true on the children that are foreign tables, and none of what
> we're talking about here gets us to that, because we normally want the
> constraints to be identical throughout the inheritance hierarchy.

There's a nearby thread that's addressing this same question, in which
I make the case (again) that the right thing for postgres_fdw constraints
is that they're just assumed true.  So I'm not sure why this conversation
is proposing to implement a lot of mechanism to do something different
from that.
        regards, tom lane



Re: inherit support for foreign tables

From
Robert Haas
Date:
On Tue, Jan 21, 2014 at 3:00 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> One thing that's bugging me a bit about this whole line of attack is
>> that, in the first instance, the whole goal here is to support
>> inheritance hierarchies that mix ordinary tables with foreign tables.
>> If you have a table with children some of which are inherited and
>> others of which are not inherited, you're very likely going to want
>> your constraints enforced for real on the children that are tables and
>> assumed true on the children that are foreign tables, and none of what
>> we're talking about here gets us to that, because we normally want the
>> constraints to be identical throughout the inheritance hierarchy.
>
> There's a nearby thread that's addressing this same question, in which
> I make the case (again) that the right thing for postgres_fdw constraints
> is that they're just assumed true.  So I'm not sure why this conversation
> is proposing to implement a lot of mechanism to do something different
> from that.

/me scratches head.

Because the other guy named Tom Lane took the opposite position on the
second message on this thread, dated 11/14/13?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/01/22 4:09), Robert Haas wrote:
> On Mon, Jan 20, 2014 at 9:44 PM, Shigeru Hanada
> <shigeru.hanada@gmail.com> wrote:
>> Thanks for the comments.
>>
>> 2014/1/21 KaiGai Kohei <kaigai@ak.jp.nec.com>:
>>>> In addition, an idea which I can't throw away is to assume that all
>>>> constraints defined on foreign tables as ASSERTIVE.  Foreign tables
>>>> potentially have dangers to have "wrong" data by updating source data
>>>> not through foreign tables.  This is not specific to an FDW, so IMO
>>>> constraints defined on foreign tables are basically ASSERTIVE.  Of
>>>> course PG can try to maintain data correct, but always somebody might
>>>> break it.
>>>> qu
>>>>
>>> Does it make sense to apply "assertive" CHECK constraint on the qual
>>> of ForeignScan to filter out tuples with violated values at the local
>>> side, as if row-level security feature doing.
>>> It enables to handle a situation that planner expects only "clean"
>>> tuples are returned but FDW driver is unavailable to anomalies.
>>>
>>> Probably, this additional check can be turned on/off on the fly,
>>> if FDW driver has a way to inform the core system its capability,
>>> like FDW_CAN_ENFORCE_CHECK_CONSTRAINT that informs planner to skip
>>> local checks.
>>
>> Hmm, IIUC you mean that local users can't (or don't need to) know that
>> data which violates the local constraints exist on remote side.
>> Applying constraints to the data which is modified through FDW would
>> be necessary as well.  In that design, FDW is a bidirectional filter
>> which provides these features:
>>
>> 1) Don't push wrong data into remote data source, by applying local
>> constraints to the result of the modifying query executed on local PG.
>>   This is not perfect filter, because remote constraints don't mapped
>> automatically or perfectly (imagine constraints which is available on
>> remote but is not supported in PG).
>> 2) Don't retrieve wrong data from remote to local PG, by applying
>> local constraints
>>
>> I have a concern about consistency.  It has not been supported, but
>> let's think of Aggregate push-down invoked by a query below.
>>
>> SELECT count(*) FROM remote_table;
>>
>> If this query was fully pushed down, the result is the # of records
>> exist on remote side, but the result would be # of valid records when
>> we don't push down the aggregate.  This would confuse users.
>>
>>>> Besides CHECK constraints, currently NOT NULL constraints are
>>>> virtually ASSERTIVE (not enforcing).  Should it also be noted
>>>> explicitly?
>>>>
>>> Backward compatibility….
>>
>> Yep, backward compatibility (especially visible ones to users) should
>> be minimal, ideally zero.
>>
>>> NOT NULL [ASSERTIVE] might be an option.
>>
>> Treating [ASSERTIVE | NOT ASSERTIVE] like DEFERRABLE, and allow
>> ingASSERTIVE for only foreign tables?  It makes sense, though we need
>> consider exclusiveness .  But It needs to default to ASSERTIVE on
>> foreign tables, and NOT ASSERTIVE (means "forced") on others.  Isn't
>> is too complicated?
>>
>> CREATE FOREIGN TABLE foo (
>>      id int NOT NULL ASSERTIVE CHECK (id > 1) ASSERTIVE,
>>      …
>>      CONSTRAINT chk_foo_name_upper CHECK (upper(name) = name) ASSERTIVE
>> ) SERVER server;
>>
>> BTW, I noticed that this is like push-down-able expressions in
>> JOIN/WHERE.  We need to check a CHECK constraint defined on a foreign
>> tables contains only expressions which have same semantics as remote
>> side (in practice, built-in and immutable)?
>
> I don't think that that ASSERTIVE is going to fly, because "assertive"
> means (sayeth the Google) "having or showing a confident and forceful
> personality", which is not what we mean here.  It's tempting to do
> something like try to replace the keyword "check" with "assume" or
> "assert" or (stretching) "assertion", but that would require whichever
> one we picked to be a fully-reserved keyword, which I can't think is
> going to get much support here, for entirely understandable reasons.
> So I think we should look for another option.
>
> Currently, constraints can be marked NO INHERIT (though this seems to
> have not been fully documented, as the ALTER TABLE page doesn't
> mention it anywhere) or NOT VALID, so I'm thinking maybe we should go
> with something along those lines.  Some ideas:
>
> - NO CHECK.  The idea of writing CHECK (id > 1) NO CHECK is pretty
> hilarious, though.
> - NO VALIDATE.  But then people need to understand that NOT VALID
> means "we didn't validate it yet" while "no validate" means "we don't
> ever intend to validate it", which could be confusing.
> - NO ENFORCE.  Requires a new (probably unreserved) keyword.
> - NOT VALIDATED or NOT CHECKED.  Same problems as NO CHECK and NO
> VALIDATE, respectively, plus now we have to create a new keyword.
>
> Another idea is to apply an extensible-options syntax to constraints,
> like we do for EXPLAIN, VACUUM, etc.  Like maybe:
>
> CHECK (id > 1) OPTIONS (enforced false, valid true)
>
> Yet another idea is to consider validity a three-state property:
> either the constraint is valid (because we have checked it and are
> enforcing it), or it is not valid (because we are enforcing it but
> have not checked the pre-existing data), or it is assumed true
> (because we are not checking or enforcing it but are believing it
> anyway).  So then we could have a syntax like this:
>
> CHECK (id > 1) VALIDATE { ON | OFF | ASSERTION }
>
> Other ideas?
>
> One thing that's bugging me a bit about this whole line of attack is
> that, in the first instance, the whole goal here is to support
> inheritance hierarchies that mix ordinary tables with foreign tables.
> If you have a table with children some of which are inherited and
> others of which are not inherited, you're very likely going to want
> your constraints enforced for real on the children that are tables and
> assumed true on the children that are foreign tables, and none of what
> we're talking about here gets us to that, because we normally want the
> constraints to be identical throughout the inheritance hierarchy.
> Maybe there's some way around that, but I'm back to wondering if it
> wouldn't be better to simply silently force any constraints on a
> foreign-table into assertion mode.  That could be done without any new
> syntax at all, and frankly I think it's what people are going to want
> more often than not.

I'd like to vote for the idea of silently forcing any constraints on a 
foreign-table into assertion mode.  No new syntax and better documentation.

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
Hi Hanada-san,

While still reviwing this patch, I feel this patch has given enough 
consideration to interactions with other commands, but found the 
following incorrect? behabior:

postgres=# CREATE TABLE product (id INTEGER, description TEXT);
CREATE TABLE
postgres=# CREATE FOREIGN TABLE product1 () INHERITS (product) SERVER fs 
OPTIONS (filename '/home/foo/product1.csv', format 'csv');
CREATE FOREIGN TABLE
postgres=# ALTER TABLE product ALTER COLUMN description SET STORAGE 
EXTERNAL;
ERROR:  "product1" is not a table or materialized view

ISTN the ALTER TABLE simple recursion mechanism (ie ATSimpleRecursion()) 
should be modified for the ALTER COLUMN SET STORAGE case.

I just wanted to quickly tell you this for you to take time to consider.

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Shigeru Hanada
Date:
2014-01-27 Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>:
> While still reviwing this patch, I feel this patch has given enough
> consideration to interactions with other commands, but found the following
> incorrect? behabior:
>
> postgres=# CREATE TABLE product (id INTEGER, description TEXT);
> CREATE TABLE
> postgres=# CREATE FOREIGN TABLE product1 () INHERITS (product) SERVER fs
> OPTIONS (filename '/home/foo/product1.csv', format 'csv');
> CREATE FOREIGN TABLE
> postgres=# ALTER TABLE product ALTER COLUMN description SET STORAGE
> EXTERNAL;
> ERROR:  "product1" is not a table or materialized view
>
> ISTN the ALTER TABLE simple recursion mechanism (ie ATSimpleRecursion())
> should be modified for the ALTER COLUMN SET STORAGE case.
>
> I just wanted to quickly tell you this for you to take time to consider.

Thanks for the review.  It must be an oversight, so I'll fix it up soon.

-- 
Shigeru HANADA



Re: inherit support for foreign tables

From
David Fetter
Date:
On Mon, Jan 27, 2014 at 05:06:19PM +0900, Etsuro Fujita wrote:
> Hi Hanada-san,
> 
> While still reviwing this patch, I feel this patch has given enough
> consideration to interactions with other commands, but found the
> following incorrect? behabior:
> 
> postgres=# CREATE TABLE product (id INTEGER, description TEXT);
> CREATE TABLE
> postgres=# CREATE FOREIGN TABLE product1 () INHERITS (product)
> SERVER fs OPTIONS (filename '/home/foo/product1.csv', format 'csv');
> CREATE FOREIGN TABLE
> postgres=# ALTER TABLE product ALTER COLUMN description SET STORAGE
> EXTERNAL;
> ERROR:  "product1" is not a table or materialized view
> 
> ISTN the ALTER TABLE simple recursion mechanism (ie
> ATSimpleRecursion()) should be modified for the ALTER COLUMN SET
> STORAGE case.

This points to a larger discussion about what precisely foreign tables
can and cannot inherit from local ones.  I don't think that a generic
solution will be satisfactory, as the PostgreSQL FDW could, at least
in principle, support many more than the CSV FDW, as shown above.

In my estimation, the outcome of discussion above is not a blocker for
this patch.

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate



Re: inherit support for foreign tables

From
Atri Sharma
Date:

Sent from my iPad

> On 27-Jan-2014, at 21:03, David Fetter <david@fetter.org> wrote:
>
>> On Mon, Jan 27, 2014 at 05:06:19PM +0900, Etsuro Fujita wrote:
>> Hi Hanada-san,
>>
>> While still reviwing this patch, I feel this patch has given enough
>> consideration to interactions with other commands, but found the
>> following incorrect? behabior:
>>
>> postgres=# CREATE TABLE product (id INTEGER, description TEXT);
>> CREATE TABLE
>> postgres=# CREATE FOREIGN TABLE product1 () INHERITS (product)
>> SERVER fs OPTIONS (filename '/home/foo/product1.csv', format 'csv');
>> CREATE FOREIGN TABLE
>> postgres=# ALTER TABLE product ALTER COLUMN description SET STORAGE
>> EXTERNAL;
>> ERROR:  "product1" is not a table or materialized view
>>
>> ISTN the ALTER TABLE simple recursion mechanism (ie
>> ATSimpleRecursion()) should be modified for the ALTER COLUMN SET
>> STORAGE case.
>
> This points to a larger discussion about what precisely foreign tables
> can and cannot inherit from local ones.  I don't think that a generic
> solution will be satisfactory, as the PostgreSQL FDW could, at least
> in principle, support many more than the CSV FDW, as shown above.
>
> In my estimation, the outcome of discussion above is not a blocker for
> this

I wonder what shall be the cases when foreign table is on a server which does not support *all* SQL features.

Does a FDW need to have the possible inherit options mentioned in its documentation for this patch?

Regards,

Atri


Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/01/28 0:55), Atri Sharma wrote:
>> On 27-Jan-2014, at 21:03, David Fetter <david@fetter.org> wrote:
>>> On Mon, Jan 27, 2014 at 05:06:19PM +0900, Etsuro Fujita wrote:
>>> Hi Hanada-san,
>>>
>>> While still reviwing this patch, I feel this patch has given enough
>>> consideration to interactions with other commands, but found the
>>> following incorrect? behabior:
>>>
>>> postgres=# CREATE TABLE product (id INTEGER, description TEXT);
>>> CREATE TABLE
>>> postgres=# CREATE FOREIGN TABLE product1 () INHERITS (product)
>>> SERVER fs OPTIONS (filename '/home/foo/product1.csv', format 'csv');
>>> CREATE FOREIGN TABLE
>>> postgres=# ALTER TABLE product ALTER COLUMN description SET STORAGE
>>> EXTERNAL;
>>> ERROR:  "product1" is not a table or materialized view
>>>
>>> ISTN the ALTER TABLE simple recursion mechanism (ie
>>> ATSimpleRecursion()) should be modified for the ALTER COLUMN SET
>>> STORAGE case.
>>
>> This points to a larger discussion about what precisely foreign tables
>> can and cannot inherit from local ones.  I don't think that a generic
>> solution will be satisfactory, as the PostgreSQL FDW could, at least
>> in principle, support many more than the CSV FDW, as shown above.
>>
>> In my estimation, the outcome of discussion above is not a blocker for
>> this

I just thought that among the structures that local tables can alter, 
the ones that foreign tables also can by ALTER FOREIGN TABLE are 
inherited, and the others are not inherited.  So for the case as shown 
above, I thought that we silently ignore executing the ALTER COLUMN SET 
STORAGE command for the foreign table.  I wonder that would be the first 
step.

> I wonder what shall be the cases when foreign table is on a server which does not support *all* SQL features.
>
> Does a FDW need to have the possible inherit options mentioned in its documentation for this patch?

The answer is no, in my understanding.  The altering operation simply 
declares some chages for foreign tables in the inheritance tree and does 
nothing to the underlying storages.

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Kouhei Kaigai
Date:
> > I wonder what shall be the cases when foreign table is on a server which
> does not support *all* SQL features.
> >
> > Does a FDW need to have the possible inherit options mentioned in its
> documentation for this patch?
>
> The answer is no, in my understanding.  The altering operation simply
> declares some chages for foreign tables in the inheritance tree and does
> nothing to the underlying storages.
>
It seems to me Atri mention about the idea to enforce constraint when
partitioned foreign table was referenced...

BTW, isn't it feasible to consign FDW drivers its decision?
If a foreign table has a check constraint (X BETWEEN 101 AND 200),
postgres_fdw will be capable to run this check on the remote server,
however, file_fdw will not be capable. It is usual job of them when
qualifiers are attached on Path node.
Probably, things to do is simple. If the backend appends the configured
check constraint as if a user-given qualifier, FDW driver will handle it
appropriately.

Thanks,
--
NEC OSS Promotion Center / PG-Strom Project
KaiGai Kohei <kaigai@ak.jp.nec.com>


> -----Original Message-----
> From: pgsql-hackers-owner@postgresql.org
> [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Etsuro Fujita
> Sent: Tuesday, January 28, 2014 1:08 PM
> To: Atri Sharma; David Fetter
> Cc: pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] inherit support for foreign tables
>
> (2014/01/28 0:55), Atri Sharma wrote:
> >> On 27-Jan-2014, at 21:03, David Fetter <david@fetter.org> wrote:
> >>> On Mon, Jan 27, 2014 at 05:06:19PM +0900, Etsuro Fujita wrote:
> >>> Hi Hanada-san,
> >>>
> >>> While still reviwing this patch, I feel this patch has given enough
> >>> consideration to interactions with other commands, but found the
> >>> following incorrect? behabior:
> >>>
> >>> postgres=# CREATE TABLE product (id INTEGER, description TEXT);
> >>> CREATE TABLE postgres=# CREATE FOREIGN TABLE product1 () INHERITS
> >>> (product) SERVER fs OPTIONS (filename '/home/foo/product1.csv',
> >>> format 'csv'); CREATE FOREIGN TABLE postgres=# ALTER TABLE product
> >>> ALTER COLUMN description SET STORAGE EXTERNAL;
> >>> ERROR:  "product1" is not a table or materialized view
> >>>
> >>> ISTN the ALTER TABLE simple recursion mechanism (ie
> >>> ATSimpleRecursion()) should be modified for the ALTER COLUMN SET
> >>> STORAGE case.
> >>
> >> This points to a larger discussion about what precisely foreign
> >> tables can and cannot inherit from local ones.  I don't think that a
> >> generic solution will be satisfactory, as the PostgreSQL FDW could,
> >> at least in principle, support many more than the CSV FDW, as shown above.
> >>
> >> In my estimation, the outcome of discussion above is not a blocker
> >> for this
>
> I just thought that among the structures that local tables can alter, the
> ones that foreign tables also can by ALTER FOREIGN TABLE are inherited,
> and the others are not inherited.  So for the case as shown above, I thought
> that we silently ignore executing the ALTER COLUMN SET STORAGE command for
> the foreign table.  I wonder that would be the first step.
>
> > I wonder what shall be the cases when foreign table is on a server which
> does not support *all* SQL features.
> >
> > Does a FDW need to have the possible inherit options mentioned in its
> documentation for this patch?
>
> The answer is no, in my understanding.  The altering operation simply
> declares some chages for foreign tables in the inheritance tree and does
> nothing to the underlying storages.
>
> Thanks,
>
> Best regards,
> Etsuro Fujita
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make
> changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers



Re: inherit support for foreign tables

From
Shigeru Hanada
Date:
(2014/01/27 21:52), Shigeru Hanada wrote:
> 2014-01-27 Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>:
>> While still reviwing this patch, I feel this patch has given enough
>> consideration to interactions with other commands, but found the following
>> incorrect? behabior:
>>
>> postgres=# CREATE TABLE product (id INTEGER, description TEXT);
>> CREATE TABLE
>> postgres=# CREATE FOREIGN TABLE product1 () INHERITS (product) SERVER fs
>> OPTIONS (filename '/home/foo/product1.csv', format 'csv');
>> CREATE FOREIGN TABLE
>> postgres=# ALTER TABLE product ALTER COLUMN description SET STORAGE
>> EXTERNAL;
>> ERROR:  "product1" is not a table or materialized view
>>
>> ISTN the ALTER TABLE simple recursion mechanism (ie ATSimpleRecursion())
>> should be modified for the ALTER COLUMN SET STORAGE case.
>>
>> I just wanted to quickly tell you this for you to take time to consider.
>
> Thanks for the review.  It must be an oversight, so I'll fix it up soon.
>

It seems little bit complex than I expected.  Currently foreign tables
deny ALTER TABLE SET STORAGE with message like below, because foreign
tables don't have storage in the meaning of PG heap tables.

     ERROR:  "pgbench1_accounts_c1" is not a table or materialized view

At the moment we don't use attstorage for foreign tables, so allowing
SET STORAGE against foreign tables never introduce visible change
except \d+ output of foreign tables.  But IMO such operation should
not allowed because users would be confused.  So I changed
ATExecSetStorage() to skip on foreign tables.  This allows us to emit
ALTER TABLE SET STORAGE against ordinary tables in upper level of
inheritance tree, but it have effect on only ordinary tables in the
tree.

This also allows direct ALTER FOREIGN TABLE SET STORAGE against
foreign table but the command is silently ignored.  SET STORAGE
support for foreign tables is not documented because it may confuse
users.

With attached patch, SET STORAGE against wrong relations produces
message like this.  Is it confusing to mention foreign table here?

ERROR:  "pgbench1_accounts_pkey" is not a table, materialized view, or
foreign table

One other idea is to support SET STORAGE against foreign tables though
it has no effect.  This makes all tables and foreign tables to have
same storage option in same column.  It might be more understandable
behavior for users.

Thoughts?

FYI, here are lists of ALTER TABLE features which is allowed/denied
for foreign tables.

Allowed
   - ADD|DROP COLUMN
   - SET DATA TYPE
   - SET|DROP NOT NULL
   - SET STATISTICS
   - SET (attribute_option = value)
   - RESET (atrribute_option)
   - SET|DROP DEFAULT
   - ADD table_constraint
   - ALTER CONSTRAINT
   - DROP CONSTRAINT
   - [NO] INHERIT parent_table
   - OWNER
   - RENAME
   - SET SCHEMA
   - SET STORAGE
     - moved to here by attached patch

Denied
   - ADD table_constraint_using_index
   - VALIDATE CONSTRAINT
   - DISABLE|ENABLE TRIGGER
   - DISABLE|ENABLE RULE
   - CLUSTER ON
   - SET WITHOUT CLUSTER
   - SET WITH|WITHOUT OIDS
   - SET (storage_parameter = value)
   - RESET (storage_parameter)
   - OF type
   - NOT OF
   - SET TABLESPACE
   - REPLICA IDENTITY

--
Shigeru HANADA


--
Shigeru HANADA

Attachment

Re: inherit support for foreign tables

From
Atri Sharma
Date:

Sent from my iPad

On 28-Jan-2014, at 10:04, Kouhei Kaigai <kaigai@ak.jp.nec.com> wrote:

>>> I wonder what shall be the cases when foreign table is on a server which
>> does not support *all* SQL features.
>>>
>>> Does a FDW need to have the possible inherit options mentioned in its
>> documentation for this patch?
>>
>> The answer is no, in my understanding.  The altering operation simply
>> declares some chages for foreign tables in the inheritance tree and does
>> nothing to the underlying storages.
> It seems to me Atri mention about the idea to enforce constraint when
> partitioned foreign table was referenced...
>
> BTW, isn't it feasible to consign FDW drivers its decision?
> If a foreign table has a check constraint (X BETWEEN 101 AND 200),
> postgres_fdw will be capable to run this check on the remote server,
> however, file_fdw will not be capable. It is usual job of them when
> qualifiers are attached on Path node.
> Probably, things to do is simple. If the backend appends the configured
> check constraint as if a user-given qualifier, FDW driver will handle

Hi,

I think that pushing it to FDW driver is the best approach. The FDW driver knows whether or not the foreign server
supportsthe said feature,hence,the user should be abstracted from that. 

I agree that the foreign constraint can be added as a user defined qualifier and dealt with accordingly.

Regards,

Atri


Re: inherit support for foreign tables

From
Tom Lane
Date:
Shigeru Hanada <shigeru.hanada@gmail.com> writes:
> At the moment we don't use attstorage for foreign tables, so allowing
> SET STORAGE against foreign tables never introduce visible change
> except \d+ output of foreign tables.  But IMO such operation should
> not allowed because users would be confused.  So I changed
> ATExecSetStorage() to skip on foreign tables.

I think this is totally misguided.  Who's to say that some weird FDW
might not pay attention to attstorage?  I could imagine a file-based
FDW using that to decide whether to compress columns, for instance.
Admittedly, the chances of that aren't large, but it's pretty hard
to argue that going out of our way to prevent it is a useful activity.
        regards, tom lane



Re: inherit support for foreign tables

From
Robert Haas
Date:
On Thu, Jan 30, 2014 at 11:04 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Shigeru Hanada <shigeru.hanada@gmail.com> writes:
>> At the moment we don't use attstorage for foreign tables, so allowing
>> SET STORAGE against foreign tables never introduce visible change
>> except \d+ output of foreign tables.  But IMO such operation should
>> not allowed because users would be confused.  So I changed
>> ATExecSetStorage() to skip on foreign tables.
>
> I think this is totally misguided.  Who's to say that some weird FDW
> might not pay attention to attstorage?  I could imagine a file-based
> FDW using that to decide whether to compress columns, for instance.
> Admittedly, the chances of that aren't large, but it's pretty hard
> to argue that going out of our way to prevent it is a useful activity.

I think that's a pretty tenuous position.  There are already
FDW-specific options sufficient to let a particular FDW store whatever
kinds of options it likes; letting the user set options that were only
ever intended to be applied to tables just because we can seems sort
of dubious.  I'm tempted by the idea of continuing to disallow SET
STORAGE on an unvarnished foreign table, but allowing it on an
inheritance hierarchy that contains at least one real table, with the
semantics that we quietly ignore the foreign tables and apply the
operation to the plain tables.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: inherit support for foreign tables

From
Tom Lane
Date:
Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, Jan 30, 2014 at 11:04 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I think this is totally misguided.  Who's to say that some weird FDW
>> might not pay attention to attstorage?  I could imagine a file-based
>> FDW using that to decide whether to compress columns, for instance.
>> Admittedly, the chances of that aren't large, but it's pretty hard
>> to argue that going out of our way to prevent it is a useful activity.

> I think that's a pretty tenuous position.  There are already
> FDW-specific options sufficient to let a particular FDW store whatever
> kinds of options it likes; letting the user set options that were only
> ever intended to be applied to tables just because we can seems sort
> of dubious.  I'm tempted by the idea of continuing to disallow SET
> STORAGE on an unvarnished foreign table, but allowing it on an
> inheritance hierarchy that contains at least one real table, with the
> semantics that we quietly ignore the foreign tables and apply the
> operation to the plain tables.

[ shrug... ] By far the easiest implementation of that is just to apply
the catalog change to all of them.  According to your assumptions, it'll
be a no-op on the foreign tables anyway.
        regards, tom lane



Re: inherit support for foreign tables

From
Robert Haas
Date:
On Thu, Jan 30, 2014 at 5:05 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Thu, Jan 30, 2014 at 11:04 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> I think this is totally misguided.  Who's to say that some weird FDW
>>> might not pay attention to attstorage?  I could imagine a file-based
>>> FDW using that to decide whether to compress columns, for instance.
>>> Admittedly, the chances of that aren't large, but it's pretty hard
>>> to argue that going out of our way to prevent it is a useful activity.
>
>> I think that's a pretty tenuous position.  There are already
>> FDW-specific options sufficient to let a particular FDW store whatever
>> kinds of options it likes; letting the user set options that were only
>> ever intended to be applied to tables just because we can seems sort
>> of dubious.  I'm tempted by the idea of continuing to disallow SET
>> STORAGE on an unvarnished foreign table, but allowing it on an
>> inheritance hierarchy that contains at least one real table, with the
>> semantics that we quietly ignore the foreign tables and apply the
>> operation to the plain tables.
>
> [ shrug... ] By far the easiest implementation of that is just to apply
> the catalog change to all of them.  According to your assumptions, it'll
> be a no-op on the foreign tables anyway.

Well, there's some point to that, too, I suppose.  What do others think?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: inherit support for foreign tables

From
Stephen Frost
Date:
* Robert Haas (robertmhaas@gmail.com) wrote:
> On Thu, Jan 30, 2014 at 5:05 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > Robert Haas <robertmhaas@gmail.com> writes:
> >> On Thu, Jan 30, 2014 at 11:04 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >>> I think this is totally misguided.  Who's to say that some weird FDW
> >>> might not pay attention to attstorage?  I could imagine a file-based
> >>> FDW using that to decide whether to compress columns, for instance.
> >>> Admittedly, the chances of that aren't large, but it's pretty hard
> >>> to argue that going out of our way to prevent it is a useful activity.
> >
> >> I think that's a pretty tenuous position.  There are already
> >> FDW-specific options sufficient to let a particular FDW store whatever
> >> kinds of options it likes; letting the user set options that were only
> >> ever intended to be applied to tables just because we can seems sort
> >> of dubious.  I'm tempted by the idea of continuing to disallow SET
> >> STORAGE on an unvarnished foreign table, but allowing it on an
> >> inheritance hierarchy that contains at least one real table, with the
> >> semantics that we quietly ignore the foreign tables and apply the
> >> operation to the plain tables.
> >
> > [ shrug... ] By far the easiest implementation of that is just to apply
> > the catalog change to all of them.  According to your assumptions, it'll
> > be a no-op on the foreign tables anyway.
>
> Well, there's some point to that, too, I suppose.  What do others think?

I agree that using the FDW-specific options is the right approach and
disallowing those to be set on foreign tables makes sense.  I don't
particularly like the idea of applying changes during inheiritance
which we wouldn't allow the user to do directly.  While it might be a
no-op and no FDW might use it, it'd still be confusing.
Thanks,
    Stephen

Re: inherit support for foreign tables

From
Tom Lane
Date:
Stephen Frost <sfrost@snowman.net> writes:
> I agree that using the FDW-specific options is the right approach and
> disallowing those to be set on foreign tables makes sense.  I don't
> particularly like the idea of applying changes during inheiritance
> which we wouldn't allow the user to do directly.  While it might be a
> no-op and no FDW might use it, it'd still be confusing.

If we don't allow it directly, why not?  Seems rather arbitrary.
        regards, tom lane



Re: inherit support for foreign tables

From
Stephen Frost
Date:
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > I agree that using the FDW-specific options is the right approach and
> > disallowing those to be set on foreign tables makes sense.  I don't
> > particularly like the idea of applying changes during inheiritance
> > which we wouldn't allow the user to do directly.  While it might be a
> > no-op and no FDW might use it, it'd still be confusing.
>
> If we don't allow it directly, why not?  Seems rather arbitrary.

I'm apparently missing something here.  From my perspective, they're
either allowed or they're not and if they aren't allowed then they
shouldn't be allowed to happen.  I'd view this like a CHECK constraint-
if it's a foreign table then it can't have a value for SET STORAGE.  I
don't see why it would be 'ok' to allow it to be set if we're setting it
as part of inheiritance but otherwise not allow it to be set.
Thanks,
    Stephen

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/01/31 9:56), Robert Haas wrote:
> On Thu, Jan 30, 2014 at 5:05 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Robert Haas <robertmhaas@gmail.com> writes:
>>> On Thu, Jan 30, 2014 at 11:04 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>>> I think this is totally misguided.  Who's to say that some weird FDW
>>>> might not pay attention to attstorage?  I could imagine a file-based
>>>> FDW using that to decide whether to compress columns, for instance.
>>>> Admittedly, the chances of that aren't large, but it's pretty hard
>>>> to argue that going out of our way to prevent it is a useful activity.
>>
>>> I think that's a pretty tenuous position.  There are already
>>> FDW-specific options sufficient to let a particular FDW store whatever
>>> kinds of options it likes; letting the user set options that were only
>>> ever intended to be applied to tables just because we can seems sort
>>> of dubious.  I'm tempted by the idea of continuing to disallow SET
>>> STORAGE on an unvarnished foreign table, but allowing it on an
>>> inheritance hierarchy that contains at least one real table, with the
>>> semantics that we quietly ignore the foreign tables and apply the
>>> operation to the plain tables.
>>
>> [ shrug... ] By far the easiest implementation of that is just to apply
>> the catalog change to all of them.  According to your assumptions, it'll
>> be a no-op on the foreign tables anyway.
>
> Well, there's some point to that, too, I suppose.  What do others think?

Allowing ALTER COLUMN SET STORAGE on foreign tables would make sense if 
for example, "SELECT * INTO local_table FROM foreign_table" did create a 
new local table of columns having the storage types associated with 
those of a foreign table?

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Robert Haas
Date:
On Sun, Feb 2, 2014 at 10:15 PM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp> wrote:
> Allowing ALTER COLUMN SET STORAGE on foreign tables would make sense if for
> example, "SELECT * INTO local_table FROM foreign_table" did create a new
> local table of columns having the storage types associated with those of a
> foreign table?

Seems like a pretty weak argument.  It's not that we can't find
strange corner cases where applying SET STORAGE to a foreign table
doesn't do something; it's that they *are* strange corner cases.  The
options as we normally don't understand them just aren't sensible in
this context, and a good deal of work has been put into an alternative
options framework, which is what authors of FDWs ought to be using.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/02/04 20:56), Robert Haas wrote:
> On Sun, Feb 2, 2014 at 10:15 PM, Etsuro Fujita
> <fujita.etsuro@lab.ntt.co.jp> wrote:
>> Allowing ALTER COLUMN SET STORAGE on foreign tables would make sense if for
>> example, "SELECT * INTO local_table FROM foreign_table" did create a new
>> local table of columns having the storage types associated with those of a
>> foreign table?
>
> Seems like a pretty weak argument.  It's not that we can't find
> strange corner cases where applying SET STORAGE to a foreign table
> doesn't do something; it's that they *are* strange corner cases.  The
> options as we normally don't understand them just aren't sensible in
> this context, and a good deal of work has been put into an alternative
> options framework, which is what authors of FDWs ought to be using.

I just wanted to discuss the possiblity of allowing SET STORAGE on a 
foreign table, but I've got the point.  I'll resume the patch review.

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
Hi Hanada-san,

Sorry for the delay.

(2014/01/30 14:01), Shigeru Hanada wrote:
>> 2014-01-27 Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>:
>>> While still reviwing this patch, I feel this patch has given enough
>>> consideration to interactions with other commands, but found the following
>>> incorrect? behabior:
>>>
>>> postgres=# CREATE TABLE product (id INTEGER, description TEXT);
>>> CREATE TABLE
>>> postgres=# CREATE FOREIGN TABLE product1 () INHERITS (product) SERVER fs
>>> OPTIONS (filename '/home/foo/product1.csv', format 'csv');
>>> CREATE FOREIGN TABLE
>>> postgres=# ALTER TABLE product ALTER COLUMN description SET STORAGE
>>> EXTERNAL;
>>> ERROR:  "product1" is not a table or materialized view
>>>
>>> ISTN the ALTER TABLE simple recursion mechanism (ie ATSimpleRecursion())
>>> should be modified for the ALTER COLUMN SET STORAGE case.

> It seems little bit complex than I expected.  Currently foreign tables
> deny ALTER TABLE SET STORAGE with message like below, because foreign
> tables don't have storage in the meaning of PG heap tables.
>
>       ERROR:  "pgbench1_accounts_c1" is not a table or materialized view
>
> At the moment we don't use attstorage for foreign tables, so allowing
> SET STORAGE against foreign tables never introduce visible change
> except \d+ output of foreign tables.  But IMO such operation should
> not allowed because users would be confused.  So I changed
> ATExecSetStorage() to skip on foreign tables.  This allows us to emit
> ALTER TABLE SET STORAGE against ordinary tables in upper level of
> inheritance tree, but it have effect on only ordinary tables in the
> tree.
>
> This also allows direct ALTER FOREIGN TABLE SET STORAGE against
> foreign table but the command is silently ignored.  SET STORAGE
> support for foreign tables is not documented because it may confuse
> users.
>
> With attached patch, SET STORAGE against wrong relations produces
> message like this.  Is it confusing to mention foreign table here?
>
> ERROR:  "pgbench1_accounts_pkey" is not a table, materialized view, or
> foreign table

ITSM that would be confusing to users.  So, I've modified the patch so
that we continue to disallow SET STORAGE on a foreign table *in the same
manner as before*, but, as your patch does, allow it on an inheritance
hierarchy that contains foreign tables, with the semantics that we
quietly ignore the foreign tables and apply the operation to the plain
tables, by modifying the ALTER TABLE simple recursion mechanism.
Attached is the updated version of the patch.

I'll continue the patch review a bit more, though the patch looks good
generally to me except for the abobe issue and the way that the ANALYZE
command works.  For the latter, if there are no objections, I'll merge
the ANALYZE patch [1] with your patch.

Thanks,

[1] http://www.postgresql.org/message-id/52EB10AC.4070307@lab.ntt.co.jp

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/02/07 21:31), Etsuro Fujita wrote:
> So, I've modified the patch so 
> that we continue to disallow SET STORAGE on a foreign table *in the same 
> manner as before*, but, as your patch does, allow it on an inheritance 
> hierarchy that contains foreign tables, with the semantics that we 
> quietly ignore the foreign tables and apply the operation to the plain 
> tables, by modifying the ALTER TABLE simple recursion mechanism. 
> Attached is the updated version of the patch.
> 
> I'll continue the patch review a bit more, though the patch looks good 
> generally to me except for the abobe issue and the way that the ANALYZE 
> command works.

While reviwing the patch, I've found some issues on interactions with
other commands, other than the SET STORAGE command.

* ADD table_constraint NOT VALID: the patch allows ADD table_constraint
*NOT VALID* to be set on a foreign table as well as an inheritance
hierarchy that contains foreign tables.  But I think it would be better
to disallow ADD table_constraint *NOT VALID* on a foreign table, but
allow it on an inheritance hierarchy that contains foreign tables, with
the semantics that we apply the operation to the plain tables and apply
the transformed operation *ADD table_constraint* to the foreign tables.

* VALIDATE CONSTRAINT constraint_name: though the patch disallow the
direct operation on foreign tables, it allows the operation on an
inheritance hierarchy that contains foreign tables, and the operation
fails if there is at least one foreign table that has at least one NOT
VALID constraint.  I think it would be better to modify the patch so
that we allow the operation on an inheritance hierarchy that contains
foreign tables, with the semantics that we quietly ignore the foreign
tables and apply the operation on the plain tables just like the
semantics of SET STORAGE.  (Note that the foreign tables wouldn't have
NOT VALID constraints by diallowing ADD table_constraint *NOT VALID* on
foreign tables as mentioned above.)

* SET WITH OIDS: though the patch disallow the direct operation on
foreign tables, it allows the operation on an inheritance hierarchy that
contains foreign tables, and that operation is successfully done on
foreign tables.  I think it would be better to modify the patch so that
we allow it on an inheritance hierarchy that contains foreign tables,
with the semantics that we quietly ignore the foreign tables and apply
the operation to the plain tables just like the semantics of SET STORAGE.

Comments are wellcome!

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/02/10 21:00), Etsuro Fujita wrote:
> (2014/02/07 21:31), Etsuro Fujita wrote:
>> So, I've modified the patch so
>> that we continue to disallow SET STORAGE on a foreign table *in the same
>> manner as before*, but, as your patch does, allow it on an inheritance
>> hierarchy that contains foreign tables, with the semantics that we
>> quietly ignore the foreign tables and apply the operation to the plain
>> tables, by modifying the ALTER TABLE simple recursion mechanism.

> While reviwing the patch, I've found some issues on interactions with
> other commands, other than the SET STORAGE command.

> * ADD table_constraint NOT VALID: the patch allows ADD table_constraint
> *NOT VALID* to be set on a foreign table as well as an inheritance
> hierarchy that contains foreign tables.  But I think it would be better
> to disallow ADD table_constraint *NOT VALID* on a foreign table, but
> allow it on an inheritance hierarchy that contains foreign tables, with
> the semantics that we apply the operation to the plain tables and apply
> the transformed operation *ADD table_constraint* to the foreign tables.

Done.

> * VALIDATE CONSTRAINT constraint_name:

I've modified the patch so that though we continue to disallow the
operation on a foreign table, we allow it on an inheritance hierarchy
that contains foreign tables.  In that case, the to-be-validated
constraints on the plain tables are validated by the operation, as
before, and the constraints on the foreign tables are ignored.  Note
that the constraints on the foreign tables are not NOT VALID due to the
spec mentioned above.

> * SET WITH OIDS: though the patch disallow the direct operation on
> foreign tables, it allows the operation on an inheritance hierarchy that
> contains foreign tables, and that operation is successfully done on
> foreign tables.  I think it would be better to modify the patch so that
> we allow it on an inheritance hierarchy that contains foreign tables,
> with the semantics that we quietly ignore the foreign tables and apply
> the operation to the plain tables just like the semantics of SET STORAGE.

I noticed this breaks inheritance hierarchy.  To avoid this, I've
modified the patch to disallow SET WITH OIDS on an inheritance hierarchy
that contains at least one foreign table.

The other changes:

-       if (stmt->constraints != NIL && relkind == RELKIND_FOREIGN_TABLE)
-               ereport(ERROR,
-                               (errcode(ERRCODE_WRONG_OBJECT_TYPE),
-                                errmsg("constraints are not supported
on foreign tables")));
+/*
+ * Shouldn't this have been checked in parser?
+ */
+       if (relkind == RELKIND_FOREIGN_TABLE)
+       {
+               ListCell   *lc;
+               foreach(lc, stmt->constraints)
+               {
+                       NewConstraint      *nc = lfirst(lc);
+
+                       if (nc->contype != CONSTR_CHECK &&
+                               nc->contype != CONSTR_DEFAULT &&
+                               nc->contype != CONSTR_NULL &&
+                               nc->contype != CONSTR_NOTNULL)
+                               ereport(ERROR,
+
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
+                                                errmsg("only check
constraints are supported on foreign tables")));
+               }
+       }

ISTM we wouldn't need the above check in DefineRelation(), so I've
removed the check from the patch.  And I've modified the vague error
messages in parse_utilcmd.c.

There seems to be no objections, I've merged the ANALYZE patch [1] with
your patch.  I noticed that the patch in [1] isn't efficient.  (To
ANALYZE one inheritance tree that contains foreign tables, the patch in
[1] calls the AnalyzeForeignTable() routine two times for each such
foreign table.)  So, the updated version has been merged that calls the
routine only once for each such foreign table.

Todo:

ISTM the documentation would need to be updated further.

Thanks,

[1] http://www.postgresql.org/message-id/52EB10AC.4070307@lab.ntt.co.jp

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Shigeru Hanada
Date:
Hi Fujita-san,

Thanks for the reviewing.

2014-02-10 21:00 GMT+09:00 Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>:
> (2014/02/07 21:31), Etsuro Fujita wrote:
>> So, I've modified the patch so
>> that we continue to disallow SET STORAGE on a foreign table *in the same
>> manner as before*, but, as your patch does, allow it on an inheritance
>> hierarchy that contains foreign tables, with the semantics that we
>> quietly ignore the foreign tables and apply the operation to the plain
>> tables, by modifying the ALTER TABLE simple recursion mechanism.
>> Attached is the updated version of the patch.

I'm not sure that allowing ALTER TABLE against parent table affects
descendants even some of them are foreign table.  I think the rule
should be simple enough to understand for users, of course it should
be also consistent and have backward compatibility.

If foreign table can be modified through inheritance tree, this kind
of change can be done.

1) create foreign table as a child of a ordinary table
2) run ALTER TABLE parent, the foreign table is also changed
3) remove foreign table from the inheritance tree by ALTER TABLE child
NO INHERIT parent
4) here we can't do same thing as 2), because it is not a child anymore.

So IMO we should determine which ALTER TABLE features are allowed to
foreign tables, and allow them regardless of the recursivity.

Comments?
-- 
Shigeru HANADA



Re: inherit support for foreign tables

From
"Etsuro Fujita"
Date:
> From: Shigeru Hanada [mailto:shigeru.hanada@gmail.com]

> 2014-02-10 21:00 GMT+09:00 Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>:
> > (2014/02/07 21:31), Etsuro Fujita wrote:
> >> So, I've modified the patch so
> >> that we continue to disallow SET STORAGE on a foreign table *in the
> >> same manner as before*, but, as your patch does, allow it on an
> >> inheritance hierarchy that contains foreign tables, with the
> >> semantics that we quietly ignore the foreign tables and apply the
> >> operation to the plain tables, by modifying the ALTER TABLE simple
> recursion mechanism.
> >> Attached is the updated version of the patch.
>
> I'm not sure that allowing ALTER TABLE against parent table affects
> descendants even some of them are foreign table.  I think the rule should
> be simple enough to understand for users, of course it should be also
> consistent and have backward compatibility.

Yeah, the understandability is important.  But I think the flexibility is also important.  In other words, I think it
isa bit too inflexible that we disallow e.g., SET STORAGE to be set on an inheritance tree that contains foreign
table(s)because we disallow SET STORAGE to be set on foreign tables directly. 

> If foreign table can be modified through inheritance tree, this kind of
> change can be done.
>
> 1) create foreign table as a child of a ordinary table
> 2) run ALTER TABLE parent, the foreign table is also changed
> 3) remove foreign table from the inheritance tree by ALTER TABLE child NO
> INHERIT parent
> 4) here we can't do same thing as 2), because it is not a child anymore.
>
> So IMO we should determine which ALTER TABLE features are allowed to foreign
> tables, and allow them regardless of the recursivity.

What I think should be newly allowed to be set on foreign tables is

* ADD table_constraint
* DROP CONSTRAINT
* [NO] INHERIT parent_table

Thanks,

Best regards,
Etsuro Fujita




Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hello,

> 2014-02-10 21:00 GMT+09:00 Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>:
> > (2014/02/07 21:31), Etsuro Fujita wrote:
> >> So, I've modified the patch so
> >> that we continue to disallow SET STORAGE on a foreign table *in the same
> >> manner as before*, but, as your patch does, allow it on an inheritance
> >> hierarchy that contains foreign tables, with the semantics that we
> >> quietly ignore the foreign tables and apply the operation to the plain
> >> tables, by modifying the ALTER TABLE simple recursion mechanism.
> >> Attached is the updated version of the patch.
> 
> I'm not sure that allowing ALTER TABLE against parent table affects
> descendants even some of them are foreign table.  I think the rule
> should be simple enough to understand for users, of course it should
> be also consistent and have backward compatibility.

Could you guess any use cases in which we are happy with ALTER
TABLE's inheritance tree walking?  IMHO, ALTER FOREIGN TABLE
always comes with some changes of the data source so implicitly
invoking of such commands should be defaultly turned off. If the
ALTER'ing the whole familiy is deadfully useful for certain
cases, it might be explicitly turned on by some syntax added to
CREATE FOREIGN TABLE or ALTER TABLE.

It would looks like following,

CREATE FOREIGN TABLE ft1 () INHERITS (pt1 ALLOW_INHERITED_ALTER, pt2);

ALTER TABLE INCLUDE FOREIGN CHILDREN parent ADD COLUMN add1 integer;

These looks quite bad :-( but also seem quite better than
accidentially ALTER'ing foreign children.

If foreign tables were allowed to ALTER'ed with 'ALTER TABLE',
some reconstruction between 'ALTER TABLE' and 'ALTER FOREIGN
TABLE' would be needed.

> If foreign table can be modified through inheritance tree, this kind
> of change can be done.
> 
> 1) create foreign table as a child of a ordinary table
> 2) run ALTER TABLE parent, the foreign table is also changed
> 3) remove foreign table from the inheritance tree by ALTER TABLE child
> NO INHERIT parent
> 4) here we can't do same thing as 2), because it is not a child anymore.
> 
> So IMO we should determine which ALTER TABLE features are allowed to
> foreign tables, and allow them regardless of the recursivity.
> 
> Comments?

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



Re: inherit support for foreign tables

From
Tom Lane
Date:
Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> writes:
> Could you guess any use cases in which we are happy with ALTER
> TABLE's inheritance tree walking?  IMHO, ALTER FOREIGN TABLE
> always comes with some changes of the data source so implicitly
> invoking of such commands should be defaultly turned off. If the
> ALTER'ing the whole familiy is deadfully useful for certain
> cases, it might be explicitly turned on by some syntax added to
> CREATE FOREIGN TABLE or ALTER TABLE.

> It would looks like following,

> CREATE FOREIGN TABLE ft1 () INHERITS (pt1 ALLOW_INHERITED_ALTER, pt2);

> ALTER TABLE INCLUDE FOREIGN CHILDREN parent ADD COLUMN add1 integer;

Ugh.  This adds a lot of logical complication without much real use,
IMO.  The problems that have been discussed in this thread are that
certain types of ALTER are sensible to apply to foreign tables and
others are not --- so how does a one-size-fits-all switch help that?

Also, there are some types of ALTER for which recursion to children
*must* occur or things become inconsistent --- ADD COLUMN itself is
an example, and ADD CONSTRAINT is another.  It would not be acceptable
for an ALLOW_INHERITED_ALTER switch to suppress that, but if the
switch is leaky then it's even less consistent and harder to explain.
        regards, tom lane



Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hello,

At Tue, 18 Feb 2014 09:49:23 -0500, Tom Lane wrote
> Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> writes:
> > Could you guess any use cases in which we are happy with ALTER
> > TABLE's inheritance tree walking?  IMHO, ALTER FOREIGN TABLE
> > always comes with some changes of the data source so implicitly
> > invoking of such commands should be defaultly turned off. If the
> > ALTER'ing the whole familiy is deadfully useful for certain
> > cases, it might be explicitly turned on by some syntax added to
> > CREATE FOREIGN TABLE or ALTER TABLE.
> 
> > It would looks like following,
> 
> > CREATE FOREIGN TABLE ft1 () INHERITS (pt1 ALLOW_INHERITED_ALTER, pt2);
> 
> > ALTER TABLE INCLUDE FOREIGN CHILDREN parent ADD COLUMN add1 integer;
> 
> Ugh.  This adds a lot of logical complication without much real use,
> IMO.

Yeah! That's exactry the words I wanted for the expamples. Sorry
for bothering you. I also don't see any use case driving us to
overcome the extra complexity.

>  The problems that have been discussed in this thread are that
> certain types of ALTER are sensible to apply to foreign tables and
> others are not --- so how does a one-size-fits-all switch help that?

None, I think. I intended only to display what this ALTER's
inheritance walkthrough brings in.

> Also, there are some types of ALTER for which recursion to children
> *must* occur or things become inconsistent --- ADD COLUMN itself is
> an example, and ADD CONSTRAINT is another.  It would not be acceptable
> for an ALLOW_INHERITED_ALTER switch to suppress that, but if the
> switch is leaky then it's even less consistent and harder to explain.

Exactly. It is the problem came with allowing to implicit ALTER
on foreign children, Shigeru's last message seems to referred to.

At Tue, 18 Feb 2014 14:47:51 +0900, Shigeru Hanada wrote
> I'm not sure that allowing ALTER TABLE against parent table affects
> descendants even some of them are foreign table.

Avoiding misunderstandings, my opinion on whether ALTER may
applie to foreign chidlren is 'no'.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hello,

At Tue, 18 Feb 2014 19:24:50 +0900, "Etsuro Fujita" wrote
> > From: Shigeru Hanada [mailto:shigeru.hanada@gmail.com]
> > 2014-02-10 21:00 GMT+09:00 Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>:
> > > (2014/02/07 21:31), Etsuro Fujita wrote:
> > >> So, I've modified the patch so
> > >> that we continue to disallow SET STORAGE on a foreign table *in the
> > >> same manner as before*, but, as your patch does, allow it on an
> > >> inheritance hierarchy that contains foreign tables, with the
> > >> semantics that we quietly ignore the foreign tables and apply the
> > >> operation to the plain tables, by modifying the ALTER TABLE simple
> > recursion mechanism.
> > >> Attached is the updated version of the patch.
> > 
> > I'm not sure that allowing ALTER TABLE against parent table affects
> > descendants even some of them are foreign table.  I think the rule should
> > be simple enough to understand for users, of course it should be also
> > consistent and have backward compatibility.
> 
> Yeah, the understandability is important.  But I think the
> flexibility is also important.  In other words, I think it is a
> bit too inflexible that we disallow e.g., SET STORAGE to be set
> on an inheritance tree that contains foreign table(s) because
> we disallow SET STORAGE to be set on foreign tables directly.

What use case needs such a flexibility precedes the lost behavior
predictivity of ALTER TABLE and/or code "maintenancibility"(more
ordinally words must be...) ? I don't agree with the idea that
ALTER TABLE implicitly affects foreign children for the reason in
the upthread. Also turning on/off feature implemented as special
syntax seems little hope.

If you still want that, I suppose ALTER FOREIGN TABLE is usable
chainging so as to walk through the inheritance tree and affects
only foreign tables along the way. It can naturally affects
foregin tables with no unanticipated behavor.

Thoughts?

> > If foreign table can be modified through inheritance tree, this kind of
> > change can be done.
> > 
> > 1) create foreign table as a child of a ordinary table
> > 2) run ALTER TABLE parent, the foreign table is also changed
> > 3) remove foreign table from the inheritance tree by ALTER TABLE child NO
> > INHERIT parent
> > 4) here we can't do same thing as 2), because it is not a child anymore.
> > 
> > So IMO we should determine which ALTER TABLE features are allowed to foreign
> > tables, and allow them regardless of the recursivity.
> 
> What I think should be newly allowed to be set on foreign tables is
> 
> * ADD table_constraint
> * DROP CONSTRAINT

As of 9.3

http://www.postgresql.org/docs/9.3/static/sql-alterforeigntable.html

> Consistency with the foreign server is not checked when a
> column is added or removed with ADD COLUMN or DROP COLUMN, a
> NOT NULL constraint is added, or a column type is changed with
> SET DATA TYPE. It is the user's responsibility to ensure that
> the table definition matches the remote side.

So I belive implicit and automatic application of any constraint
on foreign childs are considerably danger.


> * [NO] INHERIT parent_table

Is this usable for inheritance foreign children? NO INHERIT
removes all foreign children but INHERIT is nop.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



Re: inherit support for foreign tables

From
Shigeru Hanada
Date:
Hi Horiguchi-san,

2014-02-18 19:29 GMT+09:00 Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>:
> Could you guess any use cases in which we are happy with ALTER
> TABLE's inheritance tree walking?  IMHO, ALTER FOREIGN TABLE
> always comes with some changes of the data source so implicitly
> invoking of such commands should be defaultly turned off.

Imagine a case that foreign data source have attributes (A, B, C, D)
but foreign tables and their parent ware defined as (A, B, C).  If
user wants to use D as well, ALTER TABLE parent ADD COLUMN D type
would be useful (rather necessary?) to keep consistency.

Changing data type from compatible one (i.e., int to numeric,
varchar(n) to text), adding CHECK/NOT NULL constraint would be also
possible.

-- 
Shigeru HANADA



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/02/19 12:12), Kyotaro HORIGUCHI wrote:
> At Tue, 18 Feb 2014 19:24:50 +0900, "Etsuro Fujita" wrote
>>> From: Shigeru Hanada [mailto:shigeru.hanada@gmail.com]

>>> I'm not sure that allowing ALTER TABLE against parent table affects
>>> descendants even some of them are foreign table.  I think the rule should
>>> be simple enough to understand for users, of course it should be also
>>> consistent and have backward compatibility.
>>
>> Yeah, the understandability is important.  But I think the
>> flexibility is also important.  In other words, I think it is a
>> bit too inflexible that we disallow e.g., SET STORAGE to be set
>> on an inheritance tree that contains foreign table(s) because
>> we disallow SET STORAGE to be set on foreign tables directly.
>
> What use case needs such a flexibility precedes the lost behavior
> predictivity of ALTER TABLE and/or code "maintenancibility"(more
> ordinally words must be...) ? I don't agree with the idea that
> ALTER TABLE implicitly affects foreign children for the reason in
> the upthread. Also turning on/off feature implemented as special
> syntax seems little hope.

It is just my personal opinion, but I think it would be convenient for 
users to alter inheritance trees that contain foreign tables the same 
way as inheritance trees that don't contain any foreign tables, without 
making user conscious of the inheritance trees contains foreign tables 
or not.  Imagine we have an inheritance tree that contains only plain 
tables and then add a foreign table as a child of the inheritance tree.  Without the flexiblity, we would need to
changethe way of altering 
 
the structure of the inheritance tree (e.g., ADD CONSTRAINT) to a 
totally different one, immediately when adding the foreign table.  I 
don't think that would be easy to use.

>> What I think should be newly allowed to be set on foreign tables is
>>
>> * ADD table_constraint
>> * DROP CONSTRAINT
>
> As of 9.3
>
> http://www.postgresql.org/docs/9.3/static/sql-alterforeigntable.html
>
>> Consistency with the foreign server is not checked when a
>> column is added or removed with ADD COLUMN or DROP COLUMN, a
>> NOT NULL constraint is added, or a column type is changed with
>> SET DATA TYPE. It is the user's responsibility to ensure that
>> the table definition matches the remote side.
>
> So I belive implicit and automatic application of any constraint
> on foreign childs are considerably danger.

We spent a lot of time discussing this issue, and the consensus is that 
it's users' fault if there are some tuples on the remote side violating 
a given constraint, as mentioned in the documentation.

>> * [NO] INHERIT parent_table
>
> Is this usable for inheritance foreign children? NO INHERIT
> removes all foreign children but INHERIT is nop.

I didn't express clearly.  Sorry for that.  Let me explain about it.

* ALTER FOREIGN TABLE target_table *INHERIT* parent_table: Add the 
target table as a new child of the parent table.
* ALTER FOREIGN TABLE target_table *NO INHERIT* parent_table: Remove the 
target table from the list of children of the parent table.

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hello, 

> It is just my personal opinion, but I think it would be convenient for
> users to alter inheritance trees that contain foreign tables the same
> way as inheritance trees that don't contain any foreign tables,
> without making user conscious of the inheritance trees contains
> foreign tables or not.  Imagine we have an inheritance tree that
> contains only plain tables and then add a foreign table as a child of
> the inheritance tree.  Without the flexiblity, we would need to change
> the way of altering the structure of the inheritance tree (e.g., ADD
> CONSTRAINT) to a totally different one, immediately when adding the
> foreign table.  I don't think that would be easy to use.

I personally don't see significant advantages such a
flexibility. Although my concerns here are only two points,
unanticipated application and "maintenancibility".  I gave a
consideration on these issues again.

Then, I think it could be enough by giving feedback to operators
for the first issue.

=# ALTER TABLE parent ADD CHECK (tempmin < tempmax),                     ALTER tempmin SET NOT NULL,
 ALTER tempmin SET DEFAULT 0;
 
NOTICE: Child foregn table child01 is affected.
NOTICE: Child foregn table child02 is affected
NOTICE: Child foregn table child03 rejected 'alter tempmin set default'

What do you think about this? It looks a bit too loud for me
though...

Then the second issue, however I don't have enough idea of how
ALTER TABLE works, the complexity would be reduced if acceptance
chek for alter "action"s would done on foreign server or data
wrapper side, not on the core of ALTER TABLE. It would also be a
help to output error messages like above. However, (NO)INHERIT
looks a little different..


> > http://www.postgresql.org/docs/9.3/static/sql-alterforeigntable.html
> >
> >> Consistency with the foreign server is not checked when a
> >> column is added or removed with ADD COLUMN or DROP COLUMN, a
> >> NOT NULL constraint is added, or a column type is changed with
> >> SET DATA TYPE. It is the user's responsibility to ensure that
> >> the table definition matches the remote side.
> >
> > So I belive implicit and automatic application of any constraint
> > on foreign childs are considerably danger.
> 
> We spent a lot of time discussing this issue, and the consensus is
> that it's users' fault if there are some tuples on the remote side
> violating a given constraint, as mentioned in the documentation.

I'm worried about not that but the changes and possible
inconsistency would take place behind operators' backs. And this
looks to cause such inconsistencies for me.

> >> * [NO] INHERIT parent_table
> >
> > Is this usable for inheritance foreign children? NO INHERIT
> > removes all foreign children but INHERIT is nop.
> 
> I didn't express clearly.  Sorry for that.  Let me explain about it.
> 
> * ALTER FOREIGN TABLE target_table *INHERIT* parent_table: Add the
> * target table as a new child of the parent table.
> * ALTER FOREIGN TABLE target_table *NO INHERIT* parent_table: Remove the
> * target table from the list of children of the parent table.

I got it, thank you. It alone seems no probmen but also doesn't
seem to be a matter of 'ALTER TABLE'. Could you tell me how this
is related to 'ALTER TABLE'?

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hi,

At Wed, 19 Feb 2014 16:17:05 +0900, Shigeru Hanada wrote
> 2014-02-18 19:29 GMT+09:00 Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>:
> > Could you guess any use cases in which we are happy with ALTER
> > TABLE's inheritance tree walking?  IMHO, ALTER FOREIGN TABLE
> > always comes with some changes of the data source so implicitly
> > invoking of such commands should be defaultly turned off.
> 
> Imagine a case that foreign data source have attributes (A, B, C, D)
> but foreign tables and their parent ware defined as (A, B, C).  If
> user wants to use D as well, ALTER TABLE parent ADD COLUMN D type
> would be useful (rather necessary?) to keep consistency.

Hmm. I seems to me an issue of mis-configuration at first
step. However, my anxiety is - as in my message just before -
ALTER'ing foreign table definitions without any notice to
operatos and irregular or random logic on check applicability(?)
of ALTER actions.

> Changing data type from compatible one (i.e., int to numeric,
> varchar(n) to text), adding CHECK/NOT NULL constraint would be also
> possible.

I see, thank you. Changing data types are surely valuable but
also seems difficult to check validity:(

Anyway, I gave a second thought on this issue. Please have a look
on that.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/02/20 15:47), Kyotaro HORIGUCHI wrote:
> Although my concerns here are only two points,
> unanticipated application and "maintenancibility".  I gave a
> consideration on these issues again.

Sorry, I misunderstood what you mean by "unanticipated application".

> Then, I think it could be enough by giving feedback to operators
> for the first issue.
>
> =# ALTER TABLE parent ADD CHECK (tempmin < tempmax),
>                        ALTER tempmin SET NOT NULL,
>                        ALTER tempmin SET DEFAULT 0;
> NOTICE: Child foregn table child01 is affected.
> NOTICE: Child foregn table child02 is affected
> NOTICE: Child foregn table child03 rejected 'alter tempmin set default'
>
> What do you think about this? It looks a bit too loud for me
> though...

I think that's a good idea.  What do others think?

> Then the second issue, however I don't have enough idea of how
> ALTER TABLE works, the complexity would be reduced if acceptance
> chek for alter "action"s would done on foreign server or data
> wrapper side, not on the core of ALTER TABLE. It would also be a
> help to output error messages like above.

I'm not sure it's worth having such an mechanism inside/outside the PG 
core.  I might misunderstand your concern here, but is it the risk of 
constraint violation?  If so, I'd like to vote for an idea of avoiding 
that violation by making the FDW in itself perform ExecQual() for each 
tuple retrived from the remote server at the query time.

>> We spent a lot of time discussing this issue, and the consensus is
>> that it's users' fault if there are some tuples on the remote side
>> violating a given constraint, as mentioned in the documentation.
>
> I'm worried about not that but the changes and possible
> inconsistency would take place behind operators' backs. And this
> looks to cause such inconsistencies for me.

That is what you mean by "unanticipated application", right?

>>>> * [NO] INHERIT parent_table
>>>
>>> Is this usable for inheritance foreign children? NO INHERIT
>>> removes all foreign children but INHERIT is nop.
>>
>> I didn't express clearly.  Sorry for that.  Let me explain about it.
>>
>> * ALTER FOREIGN TABLE target_table *INHERIT* parent_table: Add the
>> * target table as a new child of the parent table.
>> * ALTER FOREIGN TABLE target_table *NO INHERIT* parent_table: Remove the
>> * target table from the list of children of the parent table.
>
> I got it, thank you. It alone seems no probmen but also doesn't
> seem to be a matter of 'ALTER TABLE'. Could you tell me how this
> is related to 'ALTER TABLE'?

These are not related to ALTER TABLE.  [NO] INHERIT parent_table (and 
ADD/DROP CONSTRAINT) are what I think should be newly allowed to apply 
to foreign tables directly.

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/02/20 19:55), Etsuro Fujita wrote:
> (2014/02/20 15:47), Kyotaro HORIGUCHI wrote:
>> Although my concerns here are only two points,
>> unanticipated application and "maintenancibility".  I gave a
>> consideration on these issues again.
>
> Sorry, I misunderstood what you mean by "unanticipated application".
>
>> Then, I think it could be enough by giving feedback to operators
>> for the first issue.
>>
>> =# ALTER TABLE parent ADD CHECK (tempmin < tempmax),
>>                        ALTER tempmin SET NOT NULL,
>>                        ALTER tempmin SET DEFAULT 0;
>> NOTICE: Child foregn table child01 is affected.
>> NOTICE: Child foregn table child02 is affected
>> NOTICE: Child foregn table child03 rejected 'alter tempmin set default'
>>
>> What do you think about this? It looks a bit too loud for me
>> though...
>
> I think that's a good idea.

I just thought those messages would be shown for the user to readily 
notice the changes of the structures of child tables that are foreign, 
done by the recursive altering operation.  But I overlooked the third line:

NOTICE: Child foregn table child03 rejected 'alter tempmin set default'

What does "rejected" in this message mean?

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hi,

> >> NOTICE: Child foregn table child01 is affected.
> >> NOTICE: Child foregn table child02 is affected
> >> NOTICE: Child foregn table child03 rejected 'alter tempmin set
> >> default'
> >>
> >> What do you think about this? It looks a bit too loud for me
> >> though...
> >
> > I think that's a good idea.
> 
> I just thought those messages would be shown for the user to readily
> notice the changes of the structures of child tables that are foreign,
> done by the recursive altering operation.  But I overlooked the third
> line:
> 
> NOTICE: Child foregn table child03 rejected 'alter tempmin set
> default'
> 
> What does "rejected" in this message mean?

It says that child03 had no ability to perform the requested
action, in this case setting a default value. It might be better
to reject ALTER on the parent as a whole when any children
doesn't accept any action.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hello,

> > Then the second issue, however I don't have enough idea of how
> > ALTER TABLE works, the complexity would be reduced if acceptance
> > chek for alter "action"s would done on foreign server or data
> > wrapper side, not on the core of ALTER TABLE. It would also be a
> > help to output error messages like above.
> 
> I'm not sure it's worth having such an mechanism inside/outside the PG
> core.  I might misunderstand your concern here, but is it the risk of
> constraint violation?

A bit different:) It's the problem of how and who decides whether
each ALTER action given can be performed on each child. Set of
available actions vary according to the nature of each foreign
table/server/driver and also according to their functional
evolutions made in future, largely independent of the core,
maybe. The core oughtn't to.. couldn't maintain such a function
judging availability of every possible combination of action and
foreign table.

>  If so, I'd like to vote for an idea of avoiding
> that violation by making the FDW in itself perform ExecQual() for each
> tuple retrived from the remote server at the query time.

In my humble opition, it is not so serious problem that
functionally acceptable actions can cause any kind of
inconsistency by the nature of fdw so far. It is well documented
and operators should be aware of such inconsistencies after being
informed of what they did, by the NOTICE messages. I think.

> >> * ALTER FOREIGN TABLE target_table *INHERIT* parent_table: Add the
> >> * target table as a new child of the parent table.
> >> * ALTER FOREIGN TABLE target_table *NO INHERIT* parent_table: Remove the
> >> * target table from the list of children of the parent table.
> >
> > I got it, thank you. It alone seems no probmen but also doesn't
> > seem to be a matter of 'ALTER TABLE'. Could you tell me how this
> > is related to 'ALTER TABLE'?
> 
> These are not related to ALTER TABLE.  [NO] INHERIT parent_table (and
> ADD/DROP CONSTRAINT) are what I think should be newly allowed to apply
> to foreign tables directly.

Thank you, I understood that. I thought it already existed.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/02/21 15:23), Kyotaro HORIGUCHI wrote:
>>>> NOTICE: Child foregn table child01 is affected.
>>>> NOTICE: Child foregn table child02 is affected
>>>> NOTICE: Child foregn table child03 rejected 'alter tempmin set
>>>> default'
>>>>
>>>> What do you think about this? It looks a bit too loud for me
>>>> though...
>>>
>>> I think that's a good idea.
>>
>> I just thought those messages would be shown for the user to readily
>> notice the changes of the structures of child tables that are foreign,
>> done by the recursive altering operation.  But I overlooked the third
>> line:
>>
>> NOTICE: Child foregn table child03 rejected 'alter tempmin set
>> default'
>>
>> What does "rejected" in this message mean?
>
> It says that child03 had no ability to perform the requested
> action, in this case setting a default value. It might be better
> to reject ALTER on the parent as a whole when any children
> doesn't accept any action.

Now understood, thougn I'm not sure it's worth implementing such a 
checking mechanism in the recursive altering operation...

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hello,

At Fri, 21 Feb 2014 16:33:32 +0900, Etsuro Fujita wrote
> (2014/02/21 15:23), Kyotaro HORIGUCHI wrote:
> >>>> NOTICE: Child foregn table child01 is affected.
> >>>> NOTICE: Child foregn table child02 is affected
> >>>> NOTICE: Child foregn table child03 rejected 'alter tempmin set
> >>>> default'
> > It says that child03 had no ability to perform the requested
> > action, in this case setting a default value. It might be better
> > to reject ALTER on the parent as a whole when any children
> > doesn't accept any action.
> 
> Now understood, thougn I'm not sure it's worth implementing such a
> checking mechanism in the recursive altering operation...

Did you mean foreign tables can sometimes silently ignore ALTER
actions which it can't perform? It will cause inconsistency which
operators didn't anticipate. This example uses "add column" for
perspicuitly but all types of action could result like this.

==============
=# ALTER TABLE parent ADD COLUMN x integer;
ALTER TABLE
=# \d parent          Table "public.parent"Column |  Type   |     Modifiers      
--------+---------+--------------------a      | integer | b      | integer | x      | integer | 
Number of child tables: 2 (Use \d+ to list them.)
=# \d child1       Foreign table "public.child1" Column  |  Type   | Modifiers | FDW Options 
----------+---------+-----------+-------------a      | integer | b      | integer | 

=# (Op: Ouch!)
==============

I think this should result as,

==============
=# ALTER TABLE parent ADD COLUMN x integer;
ERROR: Foreign child table child1 could not perform some of the actions.
=#
==============


regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
In addition to an issue pointed out recently by Horiguchi-san, I've
found there is another issue we have to discuss.  That is, we can't
build any parameterized Append paths for an inheritance tree that
contains at least one foreign table, in set_append_rel_pathlist().  This
is because the patch doesn't take into consideration
"reparameterization" for paths for a foreign table, which attempts to
modify these paths to have greater parameterization so that each child
path in the inheritance tree can have the exact same parameterization.
To do so, I think we would need to add code for the foreign-table case
to reparameterize_path().  And I think we should introduce a new FDW
routine, say ReparameterizeForeignPath() because the processing would be
performed best by the FDW itself.

Comments are welcome!

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hello, I tried minimal implementation to do that.

At Tue, 25 Feb 2014 19:45:56 +0900, Etsuro Fujita wrote
> In addition to an issue pointed out recently by Horiguchi-san, I've
> found there is another issue we have to discuss.  That is, we can't
> build any parameterized Append paths for an inheritance tree that
> contains at least one foreign table, in set_append_rel_pathlist().  This
> is because the patch doesn't take into consideration
> "reparameterization" for paths for a foreign table, which attempts to
> modify these paths to have greater parameterization so that each child
> path in the inheritance tree can have the exact same parameterization.
> To do so, I think we would need to add code for the foreign-table case
> to reparameterize_path().  And I think we should introduce a new FDW
> routine, say ReparameterizeForeignPath() because the processing would be
> performed best by the FDW itself.
> 
> Comments are welcome!

I think the problem is foreign childs in inheritance tables
prevents all menber in the inheritance relation from using
parameterized paths, correct?

|=# explain select * from p join (select uname from c1 limit 1) s on s.uname = p.uname;
|                                  QUERY PLAN                                   
|-------------------------------------------------------------------------------
| Hash Join  (cost=0.04..244.10 rows=50 width=58)
|   Hash Cond: (p.uname = c1_1.uname)
|   ->  Append  (cost=0.00..206.01 rows=10012 width=50)
|         ->  Seq Scan on p  (cost=0.00..0.00 rows=1 width=168)
|         ->  Seq Scan on c1  (cost=0.00..204.01 rows=10001 width=50)
|         ->  Foreign Scan on c2  (cost=0.00..2.00 rows=10 width=168)
|               Foreign File: /etc/passwd
|               Foreign File Size: 1851
|   ->  Hash  (cost=0.03..0.03 rows=1 width=8)
|         ->  Limit  (cost=0.00..0.02 rows=1 width=8)
|               ->  Seq Scan on c1 c1_1  (cost=0.00..204.01 rows=10001 width=8)
| Planning time: 1.095 ms

Hmm. I tried minimal implementation to do that. This omits cost
recalculation but seems to work as expected. This seems enough if
cost recalc is trivial here.

diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index b79af7a..18ced04 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -2062,6 +2062,16 @@ reparameterize_path(PlannerInfo *root, Path *path,        case T_SubqueryScan:            return
create_subqueryscan_path(root,rel, path->pathkeys,                                            required_outer);
 
+        case T_ForeignScan:
+            {
+                ForeignPath *fpath = (ForeignPath*) path;
+                ForeignPath *newpath = makeNode(ForeignPath);
+                memcpy(newpath, fpath, sizeof(ForeignPath));
+                newpath->path.param_info =
+                    get_baserel_parampathinfo(root, rel, required_outer);
+                /* cost recalc omitted */
+                return (Path *)newpath;
+            }        default:            break;    }
....

|=# explain select * from p join (select uname from c1 limit 1) s on s.uname = p.uname;
|                                 QUERY PLAN                                 
|----------------------------------------------------------------------------
| Nested Loop  (cost=0.00..10.46 rows=50 width=58)
|   ->  Limit  (cost=0.00..0.02 rows=1 width=8)
|         ->  Seq Scan on c1 c1_1  (cost=0.00..204.01 rows=10001 width=8)
|   ->  Append  (cost=0.00..10.30 rows=12 width=158)
|         ->  Seq Scan on p  (cost=0.00..0.00 rows=1 width=168)
|               Filter: (c1_1.uname = uname)
|         ->  Index Scan using i_c1 on c1  (cost=0.29..8.30 rows=1 width=50)
|               Index Cond: (uname = c1_1.uname)
|         ->  Foreign Scan on c2  (cost=0.00..2.00 rows=10 width=168)
|               Filter: (c1_1.uname = uname)
|               Foreign File: /etc/passwd
|               Foreign File Size: 1851
| Planning time: 2.044 ms


regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hello. As a minimal implementation, I made an attempt that emit
NOTICE message when alter table affects foreign tables. It looks
like following,

| =# alter table passwd add column added int, add column added2 int;
| NOTICE:  This command affects foreign relation "cf1"
| NOTICE:  This command affects foreign relation "cf1"
| ALTER TABLE
| =# select * from passwd;
| ERROR:  missing data for column "added"
| CONTEXT:  COPY cf1, line 1: "root:x:0:0:root:/root:/bin/bash"
| =# 

This seems far better than silently performing the command,
except for the duplicated message:( New bitmap might required to
avoid the duplication..

I made the changes above and below as an attempt in the attached
patch foreign_inherit-v4.patch 

> I think the problem is foreign childs in inheritance tables
> prevents all menber in the inheritance relation from using
> parameterized paths, correct?
...
> Hmm. I tried minimal implementation to do that. This omits cost
> recalculation but seems to work as expected. This seems enough if
> cost recalc is trivial here.

Any comments?

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 8ace8bd..b4e53c1 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -258,6 +258,17 @@ CREATE TABLE products (   even if the value came from the default value definition.  </para>
+  <note>
+   <para>
+    Note that constraints can be defined on foreign tables too, but such
+    constraints are not enforced on insert or update.  Those constraints are
+    "assertive", and work only to tell planner that some kind of optimization
+    such as constraint exclusion can be considerd.  This seems useless, but
+    allows us to use foriegn table as child table (see
+    <xref linkend="ddl-inherit">) to off-load to multiple servers.
+   </para>
+  </note>
+  <sect2 id="ddl-constraints-check-constraints">   <title>Check Constraints</title>
@@ -2017,8 +2028,8 @@ CREATE TABLE capitals (  </para>  <para>
-   In <productname>PostgreSQL</productname>, a table can inherit from
-   zero or more other tables, and a query can reference either all
+   In <productname>PostgreSQL</productname>, a table or foreign table can
+   inherit from zero or more other tables, and a query can reference either all   rows of a table or all rows of a
tableplus all of its descendant tables.   The latter behavior is the default.   For example, the following query finds
thenames of all cities,
 
diff --git a/doc/src/sgml/ref/alter_foreign_table.sgml b/doc/src/sgml/ref/alter_foreign_table.sgml
index 4d8cfc5..f7a382e 100644
--- a/doc/src/sgml/ref/alter_foreign_table.sgml
+++ b/doc/src/sgml/ref/alter_foreign_table.sgml
@@ -42,6 +42,8 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab    ALTER [ COLUMN ]
<replaceableclass="PARAMETER">column_name</replaceable> SET ( <replaceable
class="PARAMETER">attribute_option</replaceable>= <replaceable class="PARAMETER">value</replaceable> [, ... ] )
ALTER[ COLUMN ] <replaceable class="PARAMETER">column_name</replaceable> RESET ( <replaceable
class="PARAMETER">attribute_option</replaceable>[, ... ] )    ALTER [ COLUMN ] <replaceable
class="PARAMETER">column_name</replaceable>OPTIONS ( [ ADD | SET | DROP ] <replaceable
class="PARAMETER">option</replaceable>['<replaceable class="PARAMETER">value</replaceable>'] [, ... ])
 
+    INHERIT <replaceable class="PARAMETER">parent_table</replaceable>
+    NO INHERIT <replaceable class="PARAMETER">parent_table</replaceable>    OWNER TO <replaceable
class="PARAMETER">new_owner</replaceable>   OPTIONS ( [ ADD | SET | DROP ] <replaceable
class="PARAMETER">option</replaceable>['<replaceable class="PARAMETER">value</replaceable>'] [, ... ])</synopsis>
 
@@ -178,6 +180,26 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab   </varlistentry>
 <varlistentry>
 
+    <term><literal>INHERIT <replaceable class="PARAMETER">parent_table</replaceable></literal></term>
+    <listitem>
+     <para>
+      This form adds the target foreign table as a new child of the specified
+      parent table.  The parent table must be a plain table.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term><literal>NO INHERIT <replaceable class="PARAMETER">parent_table</replaceable></literal></term>
+    <listitem>
+     <para>
+      This form removes the target foreign table from the list of children of
+      the specified parent table.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>    <term><literal>OPTIONS ( [ ADD | SET | DROP ] <replaceable class="PARAMETER">option</replaceable>
['<replaceableclass="PARAMETER">value</replaceable>'] [, ... ] )</literal></term>    <listitem>     <para>
 
@@ -306,6 +328,16 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab       </para>
</listitem>     </varlistentry>
 
+
+     <varlistentry>
+      <term><replaceable class="PARAMETER">parent_name</replaceable></term>
+      <listitem>
+       <para>
+        A parent table to associate or de-associate with this foreign table.
+        The parent table must be a plain table.
+       </para>
+      </listitem>
+     </varlistentry>    </variablelist> </refsect1>
diff --git a/doc/src/sgml/ref/create_foreign_table.sgml b/doc/src/sgml/ref/create_foreign_table.sgml
index 06a7087..cc11dee 100644
--- a/doc/src/sgml/ref/create_foreign_table.sgml
+++ b/doc/src/sgml/ref/create_foreign_table.sgml
@@ -22,6 +22,7 @@ CREATE FOREIGN TABLE [ IF NOT EXISTS ] <replaceable class="PARAMETER">table_name    <replaceable
class="PARAMETER">column_name</replaceable><replaceable class="PARAMETER">data_type</replaceable> [ OPTIONS (
<replaceableclass="PARAMETER">option</replaceable> '<replaceable class="PARAMETER">value</replaceable>' [, ... ] ) ] [
COLLATE<replaceable>collation</replaceable> ] [ <replaceable class="PARAMETER">column_constraint</replaceable> [ ... ]
]   [, ... ]] )
 
+[ INHERITS ( <replaceable>parent_table</replaceable> [, ... ] ) ]  SERVER <replaceable
class="parameter">server_name</replaceable>[OPTIONS ( <replaceable class="PARAMETER">option</replaceable> '<replaceable
class="PARAMETER">value</replaceable>'[, ... ] ) ]
 
@@ -159,6 +160,18 @@ CREATE FOREIGN TABLE [ IF NOT EXISTS ] <replaceable class="PARAMETER">table_name   </varlistentry>
 <varlistentry>
 
+    <term><replaceable class="PARAMETER">parent_table</replaceable></term>
+    <listitem>
+     <para>
+      The name of an existing table from which the new foreign table
+      automatically inherits all columns.  The specified parent table
+      must be a plain table.  See <xref linkend="ddl-inherit"> for the
+      details of table inheritance.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>    <term><replaceable class="PARAMETER">server_name</replaceable></term>    <listitem>     <para>
diff --git a/src/backend/catalog/pg_inherits.c b/src/backend/catalog/pg_inherits.c
index f263b42..b4a084c 100644
--- a/src/backend/catalog/pg_inherits.c
+++ b/src/backend/catalog/pg_inherits.c
@@ -256,6 +256,48 @@ has_subclass(Oid relationId)/*
+ * Check wether the inheritance tree contains foreign table(s).
+ */
+bool
+contains_foreign(Oid parentrelId, LOCKMODE lockmode)
+{
+    bool        result = false;
+    List       *tableOIDs;
+    ListCell   *lc;
+
+    /* Find all members of the inheritance tree */
+    tableOIDs = find_all_inheritors(parentrelId, lockmode, NULL);
+
+    /* There are no children */
+    if (list_length(tableOIDs) < 2)
+        return result;
+
+    foreach(lc, tableOIDs)
+    {
+        Oid            childOID = lfirst_oid(lc);
+        Relation    childrel;
+
+        /* Parent should not be foreign */
+        if (childOID == parentrelId)
+            continue;
+
+        /* We already got the needed lock */
+        childrel = heap_open(childOID, NoLock);
+
+        if (childrel->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
+        {
+            /* Found it */
+            result = true;
+        }
+
+        heap_close(childrel, lockmode);
+    }
+
+    return result;
+}
+
+
+/* * Given two type OIDs, determine whether the first is a complex type * (class type) that inherits from the second.
*/
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index a04adea..bb7bdf4 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -115,7 +115,8 @@ static Datum ind_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull); *    analyze_rel() --
analyzeone relation */void
 
-analyze_rel(Oid relid, VacuumStmt *vacstmt, BufferAccessStrategy bstrategy)
+analyze_rel(Oid relid, VacuumStmt *vacstmt, VacuumMode vacmode,
+            BufferAccessStrategy bstrategy){    Relation    onerel;    int            elevel;
@@ -270,7 +271,9 @@ analyze_rel(Oid relid, VacuumStmt *vacstmt, BufferAccessStrategy bstrategy)    /*     * If there
arechild tables, do recursive ANALYZE.     */
 
-    if (onerel->rd_rel->relhassubclass)
+    if (onerel->rd_rel->relhassubclass &&
+        (vacmode == VAC_MODE_SINGLE ||
+         !contains_foreign(RelationGetRelid(onerel), AccessShareLock)))        do_analyze_rel(onerel, vacstmt,
acquirefunc,relpages, true, elevel);    /*
 
@@ -1452,12 +1455,15 @@ acquire_inherited_sample_rows(Relation onerel, int elevel,{    List       *tableOIDs;
Relation  *rels;
 
+    AcquireSampleRowsFunc *acquirefunc;    double       *relblocks;    double        totalblocks;
+    Relation    saved_rel;    int            numrows,                nrels,                i;    ListCell   *lc;
+    bool        isAnalyzable = true;    /*     * Find all members of inheritance set.  We only need AccessShareLock
on
@@ -1486,6 +1492,8 @@ acquire_inherited_sample_rows(Relation onerel, int elevel,     * BlockNumber, so we use double
arithmetic.    */    rels = (Relation *) palloc(list_length(tableOIDs) * sizeof(Relation));
 
+    acquirefunc = (AcquireSampleRowsFunc *) palloc(list_length(tableOIDs)
+                                            * sizeof(AcquireSampleRowsFunc));    relblocks = (double *)
palloc(list_length(tableOIDs)* sizeof(double));    totalblocks = 0;    nrels = 0;
 
@@ -1507,12 +1515,59 @@ acquire_inherited_sample_rows(Relation onerel, int elevel,        }        rels[nrels] =
childrel;
-        relblocks[nrels] = (double) RelationGetNumberOfBlocks(childrel);
+
+        if (childrel->rd_rel->relkind != RELKIND_FOREIGN_TABLE)
+        {
+            acquirefunc[nrels] = acquire_sample_rows;
+            relblocks[nrels] = (double) RelationGetNumberOfBlocks(childrel);
+        }
+        else
+        {
+            /*
+             * For a foreign table, call the FDW's hook function to see whether
+             * it supports analysis.
+             */
+            FdwRoutine *fdwroutine;
+            BlockNumber relpages = 0;
+            bool        ok = false;
+
+            fdwroutine = GetFdwRoutineForRelation(childrel, false);
+            if (fdwroutine->AnalyzeForeignTable != NULL)
+                ok = fdwroutine->AnalyzeForeignTable(childrel,
+                                                     &acquirefunc[nrels],
+                                                     &relpages);
+            if (!ok)
+            {
+                isAnalyzable = false;
+                break;
+            }
+
+            relblocks[nrels] = (double) relpages;
+        }
+        totalblocks += relblocks[nrels];        nrels++;    }    /*
+     * If there is at least one foreign table that cannot be analyzed, give up.
+     */
+    if (!isAnalyzable)
+    {
+        ereport(WARNING,
+                (errmsg("skipping \"%s\" inheritance tree --- cannot analyze foreign table \"%s\"",
+                        RelationGetRelationName(onerel),
+                        RelationGetRelationName(rels[nrels]))));
+        for (i = 0; i < nrels; i++)
+        {
+            Relation    childrel = rels[i];
+
+            heap_close(childrel, NoLock);
+        }
+        return 0;
+    }
+
+    /*     * Now sample rows from each relation, proportionally to its fraction of     * the total block count.  (This
mightbe less than desirable if the child     * rels have radically different free-space percentages, but it's not
 
@@ -1525,6 +1580,7 @@ acquire_inherited_sample_rows(Relation onerel, int elevel,    {        Relation    childrel =
rels[i];       double        childblocks = relblocks[i];
 
+        AcquireSampleRowsFunc childacquirefunc = acquirefunc[i];        if (childblocks > 0)        {
@@ -1540,12 +1596,12 @@ acquire_inherited_sample_rows(Relation onerel, int elevel,                            tdrows;
            /* Fetch a random sample of the child's rows */
 
-                childrows = acquire_sample_rows(childrel,
-                                                elevel,
-                                                rows + numrows,
-                                                childtargrows,
-                                                &trows,
-                                                &tdrows);
+                childrows = childacquirefunc(childrel,
+                                             elevel,
+                                             rows + numrows,
+                                             childtargrows,
+                                             &trows,
+                                             &tdrows);                /* We may need to convert from child's rowtype
toparent's */                if (childrows > 0 &&
 
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 25f01e5..61ea18b 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -311,7 +311,8 @@ static AlteredTableInfo *ATGetQueueEntry(List **wqueue, Relation rel);static void
ATSimplePermissions(Relationrel, int allowed_targets);static void ATWrongRelkindError(Relation rel, int
allowed_targets);staticvoid ATSimpleRecursion(List **wqueue, Relation rel,
 
-                  AlterTableCmd *cmd, bool recurse, LOCKMODE lockmode);
+                  AlterTableCmd *cmd, bool recurse,
+                  bool include_foreign, LOCKMODE lockmode);static void ATTypedTableRecursion(List **wqueue, Relation
rel,AlterTableCmd *cmd,                      LOCKMODE lockmode);static List *find_typed_table_dependencies(Oid typeOid,
constchar *typeName,
 
@@ -467,10 +468,6 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId)        ereport(ERROR,
(errcode(ERRCODE_INVALID_TABLE_DEFINITION),                errmsg("ON COMMIT can only be used on temporary tables")));
 
-    if (stmt->constraints != NIL && relkind == RELKIND_FOREIGN_TABLE)
-        ereport(ERROR,
-                (errcode(ERRCODE_WRONG_OBJECT_TYPE),
-                 errmsg("constraints are not supported on foreign tables")));    /*     * Look up the namespace in
whichwe are supposed to create the relation,
 
@@ -3019,24 +3016,28 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,             * rules.             */
         ATSimplePermissions(rel, ATT_TABLE | ATT_VIEW | ATT_FOREIGN_TABLE);
 
-            ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+            /* Recurses to child tables that are foreign, too */
+            ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode);            /* No command-specific prep
needed*/            pass = cmd->def ? AT_PASS_ADD_CONSTR : AT_PASS_DROP;            break;        case AT_DropNotNull:
 /* ALTER COLUMN DROP NOT NULL */            ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
 
-            ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+            /* Recurses to child tables that are foreign, too */
+            ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode);            /* No command-specific prep
needed*/            pass = AT_PASS_DROP;            break;        case AT_SetNotNull:        /* ALTER COLUMN SET NOT
NULL*/            ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
 
-            ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+            /* Recurses to child tables that are foreign, too */
+            ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode);            /* No command-specific prep
needed*/            pass = AT_PASS_ADD_CONSTR;            break;        case AT_SetStatistics:    /* ALTER COLUMN SET
STATISTICS*/
 
-            ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+            /* Recurses to child tables that are foreign, too */
+            ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode);            /* Performs own permission checks
*/           ATPrepSetStatistics(rel, cmd->name, cmd->def, lockmode);            pass = AT_PASS_MISC;
 
@@ -3049,7 +3050,8 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,            break;        case
AT_SetStorage:       /* ALTER COLUMN SET STORAGE */            ATSimplePermissions(rel, ATT_TABLE | ATT_MATVIEW);
 
-            ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+            /* Don't recurse to child tables that are foreign */
+            ATSimpleRecursion(wqueue, rel, cmd, recurse, false, lockmode);            /* No command-specific prep
needed*/            pass = AT_PASS_MISC;            break;
 
@@ -3067,7 +3069,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,            pass = AT_PASS_ADD_INDEX;
        break;        case AT_AddConstraint:    /* ADD CONSTRAINT */
 
-            ATSimplePermissions(rel, ATT_TABLE);
+            ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);            /* Recursion occurs during execution
phase*/            /* No command-specific prep needed except saving recurse flag */            if (recurse)
 
@@ -3081,7 +3083,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,            pass = AT_PASS_ADD_CONSTR;
         break;        case AT_DropConstraint:    /* DROP CONSTRAINT */
 
-            ATSimplePermissions(rel, ATT_TABLE);
+            ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);            /* Recursion occurs during execution
phase*/            /* No command-specific prep needed except saving recurse flag */            if (recurse)
 
@@ -3149,13 +3151,19 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,            pass = AT_PASS_MISC;
     break;        case AT_AddInherit:        /* INHERIT */
 
-            ATSimplePermissions(rel, ATT_TABLE);
+            ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);            /* This command never recurses */
     ATPrepAddInherit(rel);            pass = AT_PASS_MISC;            break;
 
+        case AT_DropInherit:    /* NO INHERIT */
+            ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
+            /* This command never recurses */
+            /* No command-specific prep needed */
+            pass = AT_PASS_MISC;
+            break;        case AT_AlterConstraint:        /* ALTER CONSTRAINT */
-            ATSimplePermissions(rel, ATT_TABLE);
+            ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);            pass = AT_PASS_MISC;            break;
      case AT_ValidateConstraint:        /* VALIDATE CONSTRAINT */
 
@@ -3184,7 +3192,6 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,        case AT_EnableAlwaysRule:
case AT_EnableReplicaRule:        case AT_DisableRule:
 
-        case AT_DropInherit:    /* NO INHERIT */        case AT_AddOf:            /* OF */        case AT_DropOf: /*
NOTOF */            ATSimplePermissions(rel, ATT_TABLE);
 
@@ -4037,8 +4044,12 @@ ATSimplePermissions(Relation rel, int allowed_targets)        case RELKIND_COMPOSITE_TYPE:
    actual_target = ATT_COMPOSITE_TYPE;            break;
 
+        case RELKIND_FOREIGN_TABLE:            actual_target = ATT_FOREIGN_TABLE;
+            ereport(NOTICE,
+                    (errmsg("This command affects foreign relation \"%s\"",
+                            RelationGetRelationName(rel))));            break;        default:
actual_target= 0;
 
@@ -4125,7 +4136,8 @@ ATWrongRelkindError(Relation rel, int allowed_targets) */static voidATSimpleRecursion(List
**wqueue,Relation rel,
 
-                  AlterTableCmd *cmd, bool recurse, LOCKMODE lockmode)
+                  AlterTableCmd *cmd, bool recurse,
+                  bool include_foreign, LOCKMODE lockmode){    /*     * Propagate to children if desired.  Non-table
relationsnever have
 
@@ -4153,8 +4165,12 @@ ATSimpleRecursion(List **wqueue, Relation rel,                continue;            /*
find_all_inheritorsalready got lock */            childrel = relation_open(childrelid, NoLock);
 
-            CheckTableNotInUse(childrel, "ALTER TABLE");
-            ATPrepCmd(wqueue, childrel, cmd, false, true, lockmode);
+            if (childrel->rd_rel->relkind != RELKIND_FOREIGN_TABLE
+                || include_foreign)
+            {
+                CheckTableNotInUse(childrel, "ALTER TABLE");
+                ATPrepCmd(wqueue, childrel, cmd, false, true, lockmode);
+            }            relation_close(childrel, NoLock);        }    }
@@ -4444,7 +4460,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,    /* At top level,
permissioncheck was done in ATPrepCmd, else do it */    if (recursing)
 
-        ATSimplePermissions(rel, ATT_TABLE);
+        ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);    attrdesc = heap_open(AttributeRelationId,
RowExclusiveLock);
@@ -4859,7 +4875,19 @@ ATPrepAddOids(List **wqueue, Relation rel, bool recurse, AlterTableCmd *cmd, LOC
ATPrepAddColumn(wqueue,rel, recurse, false, cmd, lockmode);    if (recurse)
 
+    {
+        /*
+         * Don't allow to add an OID column to inheritance tree that contains
+         * foreign table(s)
+         */
+      if (contains_foreign(RelationGetRelid(rel), AccessShareLock))
+            ereport(ERROR,
+                    (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+                     errmsg("cannot add OID column to inheritance tree \"%s\" because it contains foreign table(s)",
+                            RelationGetRelationName(rel))));
+        cmd->subtype = AT_AddOidsRecurse;
+    }}/*
@@ -5340,7 +5368,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,    /* At top level, permission
checkwas done in ATPrepCmd, else do it */    if (recursing)
 
-        ATSimplePermissions(rel, ATT_TABLE);
+        ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);    /*     * get the number of the attribute
@@ -5732,7 +5760,14 @@ ATAddCheckConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel,    /* At top level,
permissioncheck was done in ATPrepCmd, else do it */    if (recursing)
 
-        ATSimplePermissions(rel, ATT_TABLE);
+        ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
+
+    /* Don't allow ADD constraint NOT VALID on foreign tables */
+    if (tab->relkind == RELKIND_FOREIGN_TABLE &&
+        constr->skip_validation && !recursing)
+        ereport(ERROR,
+                (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+                 errmsg("NOT VALID is not supported on foreign tables")));    /*     * Call AddRelationNewConstraints
todo the work, making sure it works on
 
@@ -5743,9 +5778,17 @@ ATAddCheckConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel,     * omitted from the
returnedlist, which is what we want: we do not need     * to do any validation work.  That can only happen at child
tables,    * though, since we disallow merging at the top level.
 
-     */
+     *
+     * When propagating a NOT VALID option to children that are foreign tables,
+     * we quietly ignore the option.  Note that this is safe because foreign
+     * tables don't have any children.
+     */
+    constr = copyObject(constr);
+    if (tab->relkind == RELKIND_FOREIGN_TABLE &&
+        constr->skip_validation && recursing)
+        constr->skip_validation = false;    newcons = AddRelationNewConstraints(rel, NIL,
-                                        list_make1(copyObject(constr)),
+                                        list_make1(constr),                                        recursing,
/*allow_merge */                                        !recursing,        /* is_local */
        is_readd);        /* is_internal */
 
@@ -7225,7 +7268,7 @@ ATExecDropConstraint(Relation rel, const char *constrName,    /* At top level, permission check
wasdone in ATPrepCmd, else do it */    if (recursing)
 
-        ATSimplePermissions(rel, ATT_TABLE);
+        ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);    conrel = heap_open(ConstraintRelationId,
RowExclusiveLock);
@@ -7560,7 +7603,10 @@ ATPrepAlterColumnType(List **wqueue,     * alter would put them out of step.     */    if
(recurse)
-        ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+    {
+        /* Recurses to child tables that are foreign, too */
+        ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode);
+    }    else if (!recursing &&             find_inheritance_children(RelationGetRelid(rel), NoLock) != NIL)
ereport(ERROR,
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index ded1841..35b5dd2 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -98,6 +98,7 @@ vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast,       BufferAccessStrategy bstrategy, bool
for_wraparound,bool isTopLevel){    const char *stmttype;
 
+    VacuumMode    vacmode;    volatile bool in_outer_xact,                use_own_xacts;    List       *relations;
@@ -146,6 +147,20 @@ vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast,
ALLOCSET_DEFAULT_MAXSIZE);   /*
 
+     * Identify vacuum mode.  If relid is not InvalidOid, the caller should be
+     * an autovacuum worker.  See the above comments.
+     */
+    if (relid != InvalidOid)
+        vacmode = VAC_MODE_AUTOVACUUM;
+    else
+    {
+        if (!vacstmt->relation)
+            vacmode = VAC_MODE_ALL;
+        else
+            vacmode = VAC_MODE_SINGLE;
+    }
+
+    /*     * If caller didn't give us a buffer strategy object, make one in the     * cross-transaction memory
context.    */
 
@@ -248,7 +263,7 @@ vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast,
PushActiveSnapshot(GetTransactionSnapshot());               }
 
-                analyze_rel(relid, vacstmt, vac_strategy);
+                analyze_rel(relid, vacstmt, vacmode, vac_strategy);                if (use_own_xacts)
{
diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c
index 52dcc72..238bba2 100644
--- a/src/backend/optimizer/prep/prepunion.c
+++ b/src/backend/optimizer/prep/prepunion.c
@@ -1337,11 +1337,12 @@ expand_inherited_rtentry(PlannerInfo *root, RangeTblEntry *rte, Index rti)        /*         *
Buildan RTE for the child, and attach to query's rangetable list.         * We copy most fields of the parent's RTE,
butreplace relation OID,
 
-         * and set inh = false.  Also, set requiredPerms to zero since all
-         * required permissions checks are done on the original RTE.
+         * relkind and set inh = false.  Also, set requiredPerms to zero since
+         * all required permissions checks are done on the original RTE.         */        childrte = copyObject(rte);
      childrte->relid = childOID;
 
+        childrte->relkind = newrelation->rd_rel->relkind;        childrte->inh = false;        childrte->requiredPerms
=0;        parse->rtable = lappend(parse->rtable, childrte);
 
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index b79af7a..18ced04 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -2062,6 +2062,16 @@ reparameterize_path(PlannerInfo *root, Path *path,        case T_SubqueryScan:            return
create_subqueryscan_path(root,rel, path->pathkeys,                                            required_outer);
 
+        case T_ForeignScan:
+            {
+                ForeignPath *fpath = (ForeignPath*) path;
+                ForeignPath *newpath = makeNode(ForeignPath);
+                memcpy(newpath, fpath, sizeof(ForeignPath));
+                newpath->path.param_info =
+                    get_baserel_parampathinfo(root, rel, required_outer);
+                /* cost recalc omitted */
+                return (Path *)newpath;
+            }        default:            break;    }
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index e3060a4..b5e47f4 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -4209,32 +4209,32 @@ AlterForeignServerStmt: ALTER SERVER name foreign_server_version
alter_generic_oCreateForeignTableStmt:       CREATE FOREIGN TABLE qualified_name            '(' OptTableElementList
')'
-            SERVER name create_generic_options
+            OptInherit SERVER name create_generic_options                {                    CreateForeignTableStmt
*n= makeNode(CreateForeignTableStmt);                    $4->relpersistence = RELPERSISTENCE_PERMANENT;
  n->base.relation = $4;                    n->base.tableElts = $6;
 
-                    n->base.inhRelations = NIL;
+                    n->base.inhRelations = $8;                    n->base.if_not_exists = false;                    /*
FDW-specificdata */
 
-                    n->servername = $9;
-                    n->options = $10;
+                    n->servername = $10;
+                    n->options = $11;                    $$ = (Node *) n;                }        | CREATE FOREIGN
TABLEIF_P NOT EXISTS qualified_name            '(' OptTableElementList ')'
 
-            SERVER name create_generic_options
+            OptInherit SERVER name create_generic_options                {                    CreateForeignTableStmt
*n= makeNode(CreateForeignTableStmt);                    $7->relpersistence = RELPERSISTENCE_PERMANENT;
  n->base.relation = $7;                    n->base.tableElts = $9;
 
-                    n->base.inhRelations = NIL;
+                    n->base.inhRelations = $11;                    n->base.if_not_exists = true;                    /*
FDW-specificdata */
 
-                    n->servername = $12;
-                    n->options = $13;
+                    n->servername = $13;
+                    n->options = $14;                    $$ = (Node *) n;                }        ;
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index 1e071d7..955f27c 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -515,12 +515,6 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column)                break;
  case CONSTR_CHECK:
 
-                if (cxt->isforeign)
-                    ereport(ERROR,
-                            (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
-                    errmsg("constraints are not supported on foreign tables"),
-                             parser_errposition(cxt->pstate,
-                                                constraint->location)));                cxt->ckconstraints =
lappend(cxt->ckconstraints,constraint);                break;
 
@@ -529,7 +523,7 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column)                if
(cxt->isforeign)                   ereport(ERROR,                            (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
 
-                    errmsg("constraints are not supported on foreign tables"),
+                    errmsg("primary key or unique constraints are not supported on foreign tables"),
         parser_errposition(cxt->pstate,                                                constraint->location)));
       if (constraint->keys == NIL)
 
@@ -546,7 +540,7 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column)                if
(cxt->isforeign)                   ereport(ERROR,                            (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
 
-                    errmsg("constraints are not supported on foreign tables"),
+                    errmsg("foreign key constraints are not supported on foreign tables"),
parser_errposition(cxt->pstate,                                               constraint->location)));
 
@@ -605,10 +599,14 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column)static
voidtransformTableConstraint(CreateStmtContext*cxt, Constraint *constraint){
 
-    if (cxt->isforeign)
+    if (cxt->isforeign &&
+        (constraint->contype == CONSTR_PRIMARY ||
+         constraint->contype == CONSTR_UNIQUE ||
+         constraint->contype == CONSTR_EXCLUSION ||
+         constraint->contype == CONSTR_FOREIGN))        ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
-                 errmsg("constraints are not supported on foreign tables"),
+                 errmsg("primary key, unique, exclusion, or foreign key constraints are not supported on foreign
tables"),                parser_errposition(cxt->pstate,                                    constraint->location)));
 
diff --git a/src/include/catalog/pg_inherits_fn.h b/src/include/catalog/pg_inherits_fn.h
index 757d849..228573a 100644
--- a/src/include/catalog/pg_inherits_fn.h
+++ b/src/include/catalog/pg_inherits_fn.h
@@ -21,6 +21,7 @@ extern List *find_inheritance_children(Oid parentrelId, LOCKMODE lockmode);extern List
*find_all_inheritors(OidparentrelId, LOCKMODE lockmode,                    List **parents);extern bool has_subclass(Oid
relationId);
+extern bool contains_foreign(Oid parentrelId, LOCKMODE lockmode);extern bool typeInheritsFrom(Oid subclassTypeId, Oid
superclassTypeId);#endif  /* PG_INHERITS_FN_H */
 
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 058dc5f..8253a07 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -140,6 +140,15 @@ extern int    vacuum_multixact_freeze_min_age;extern int    vacuum_multixact_freeze_table_age;
+/* Possible modes for vacuum() */
+typedef enum
+{
+    VAC_MODE_ALL,                /* Vacuum/analyze all relations */
+    VAC_MODE_SINGLE,            /* Vacuum/analyze a specific relation */
+    VAC_MODE_AUTOVACUUM            /* Autovacuum worker */
+} VacuumMode;
+
+/* in commands/vacuum.c */extern void vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast,       BufferAccessStrategy
bstrategy,bool for_wraparound, bool isTopLevel);
 
@@ -174,7 +183,7 @@ extern void lazy_vacuum_rel(Relation onerel, VacuumStmt *vacstmt,
BufferAccessStrategybstrategy);/* in commands/analyze.c */
 
-extern void analyze_rel(Oid relid, VacuumStmt *vacstmt,
+extern void analyze_rel(Oid relid, VacuumStmt *vacstmt, VacuumMode vacmode,            BufferAccessStrategy
bstrategy);externbool std_typanalyze(VacAttrStats *stats);extern double anl_random_fract(void);
 
diff --git a/src/test/regress/expected/foreign_data.out b/src/test/regress/expected/foreign_data.out
index 60506e0..f15d2e1 100644
--- a/src/test/regress/expected/foreign_data.out
+++ b/src/test/regress/expected/foreign_data.out
@@ -750,16 +750,12 @@ CREATE TABLE use_ft1_column_type (x ft1);ALTER FOREIGN TABLE ft1 ALTER COLUMN c8 SET DATA TYPE
integer;   -- ERRORERROR:  cannot alter foreign table "ft1" because column "use_ft1_column_type.x" uses its row
typeDROPTABLE use_ft1_column_type;
 
-ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0); -- ERROR
-ERROR:  constraints are not supported on foreign tables
-LINE 1: ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c...
-                                    ^
+ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0);ALTER FOREIGN TABLE ft1 DROP CONSTRAINT no_const;
           -- ERROR
 
-ERROR:  "ft1" is not a table
+ERROR:  constraint "no_const" of relation "ft1" does not existALTER FOREIGN TABLE ft1 DROP CONSTRAINT IF EXISTS
no_const;
-ERROR:  "ft1" is not a table
-ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c1_check;
-ERROR:  "ft1" is not a table
+NOTICE:  constraint "no_const" of relation "ft1" does not exist, skipping
+ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c9_check;ALTER FOREIGN TABLE ft1 SET WITH OIDS;
--ERRORERROR:  "ft1" is not a tableALTER FOREIGN TABLE ft1 OWNER TO regress_test_role;
 
diff --git a/src/test/regress/sql/foreign_data.sql b/src/test/regress/sql/foreign_data.sql
index f819eb1..2aa5ffe 100644
--- a/src/test/regress/sql/foreign_data.sql
+++ b/src/test/regress/sql/foreign_data.sql
@@ -314,10 +314,10 @@ ALTER FOREIGN TABLE ft1 ALTER COLUMN c8 SET STATISTICS -1;CREATE TABLE use_ft1_column_type (x
ft1);ALTERFOREIGN TABLE ft1 ALTER COLUMN c8 SET DATA TYPE integer;    -- ERRORDROP TABLE use_ft1_column_type;
 
-ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0); -- ERROR
+ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0);ALTER FOREIGN TABLE ft1 DROP CONSTRAINT no_const;
           -- ERRORALTER FOREIGN TABLE ft1 DROP CONSTRAINT IF EXISTS no_const;
 
-ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c1_check;
+ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c9_check;ALTER FOREIGN TABLE ft1 SET WITH OIDS;
--ERRORALTER FOREIGN TABLE ft1 OWNER TO regress_test_role;ALTER FOREIGN TABLE ft1 OPTIONS (DROP delimiter, SET quote
'~',ADD escape '@'); 

Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hello, 

> This seems far better than silently performing the command,
> except for the duplicated message:( New bitmap might required to
> avoid the duplication..

I rewrote it in more tidy way. ATController collects all affected
tables on ATRewriteCatalogs as first stage, then emit NOTICE
message according to the affected relations list. The message
looks like,

| =# alter table passwd alter column uname set default 'hoge';
| NOTICE:  This command affects 2 foreign tables: cf1, cf2
| ALTER TABLE

Do you feel this too large or complicated? I think so a bit..

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 8ace8bd..b4e53c1 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -258,6 +258,17 @@ CREATE TABLE products (   even if the value came from the default value definition.  </para>
+  <note>
+   <para>
+    Note that constraints can be defined on foreign tables too, but such
+    constraints are not enforced on insert or update.  Those constraints are
+    "assertive", and work only to tell planner that some kind of optimization
+    such as constraint exclusion can be considerd.  This seems useless, but
+    allows us to use foriegn table as child table (see
+    <xref linkend="ddl-inherit">) to off-load to multiple servers.
+   </para>
+  </note>
+  <sect2 id="ddl-constraints-check-constraints">   <title>Check Constraints</title>
@@ -2017,8 +2028,8 @@ CREATE TABLE capitals (  </para>  <para>
-   In <productname>PostgreSQL</productname>, a table can inherit from
-   zero or more other tables, and a query can reference either all
+   In <productname>PostgreSQL</productname>, a table or foreign table can
+   inherit from zero or more other tables, and a query can reference either all   rows of a table or all rows of a
tableplus all of its descendant tables.   The latter behavior is the default.   For example, the following query finds
thenames of all cities,
 
diff --git a/doc/src/sgml/ref/alter_foreign_table.sgml b/doc/src/sgml/ref/alter_foreign_table.sgml
index 4d8cfc5..f7a382e 100644
--- a/doc/src/sgml/ref/alter_foreign_table.sgml
+++ b/doc/src/sgml/ref/alter_foreign_table.sgml
@@ -42,6 +42,8 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab    ALTER [ COLUMN ]
<replaceableclass="PARAMETER">column_name</replaceable> SET ( <replaceable
class="PARAMETER">attribute_option</replaceable>= <replaceable class="PARAMETER">value</replaceable> [, ... ] )
ALTER[ COLUMN ] <replaceable class="PARAMETER">column_name</replaceable> RESET ( <replaceable
class="PARAMETER">attribute_option</replaceable>[, ... ] )    ALTER [ COLUMN ] <replaceable
class="PARAMETER">column_name</replaceable>OPTIONS ( [ ADD | SET | DROP ] <replaceable
class="PARAMETER">option</replaceable>['<replaceable class="PARAMETER">value</replaceable>'] [, ... ])
 
+    INHERIT <replaceable class="PARAMETER">parent_table</replaceable>
+    NO INHERIT <replaceable class="PARAMETER">parent_table</replaceable>    OWNER TO <replaceable
class="PARAMETER">new_owner</replaceable>   OPTIONS ( [ ADD | SET | DROP ] <replaceable
class="PARAMETER">option</replaceable>['<replaceable class="PARAMETER">value</replaceable>'] [, ... ])</synopsis>
 
@@ -178,6 +180,26 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab   </varlistentry>
 <varlistentry>
 
+    <term><literal>INHERIT <replaceable class="PARAMETER">parent_table</replaceable></literal></term>
+    <listitem>
+     <para>
+      This form adds the target foreign table as a new child of the specified
+      parent table.  The parent table must be a plain table.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term><literal>NO INHERIT <replaceable class="PARAMETER">parent_table</replaceable></literal></term>
+    <listitem>
+     <para>
+      This form removes the target foreign table from the list of children of
+      the specified parent table.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>    <term><literal>OPTIONS ( [ ADD | SET | DROP ] <replaceable class="PARAMETER">option</replaceable>
['<replaceableclass="PARAMETER">value</replaceable>'] [, ... ] )</literal></term>    <listitem>     <para>
 
@@ -306,6 +328,16 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab       </para>
</listitem>     </varlistentry>
 
+
+     <varlistentry>
+      <term><replaceable class="PARAMETER">parent_name</replaceable></term>
+      <listitem>
+       <para>
+        A parent table to associate or de-associate with this foreign table.
+        The parent table must be a plain table.
+       </para>
+      </listitem>
+     </varlistentry>    </variablelist> </refsect1>
diff --git a/doc/src/sgml/ref/create_foreign_table.sgml b/doc/src/sgml/ref/create_foreign_table.sgml
index 06a7087..cc11dee 100644
--- a/doc/src/sgml/ref/create_foreign_table.sgml
+++ b/doc/src/sgml/ref/create_foreign_table.sgml
@@ -22,6 +22,7 @@ CREATE FOREIGN TABLE [ IF NOT EXISTS ] <replaceable class="PARAMETER">table_name    <replaceable
class="PARAMETER">column_name</replaceable><replaceable class="PARAMETER">data_type</replaceable> [ OPTIONS (
<replaceableclass="PARAMETER">option</replaceable> '<replaceable class="PARAMETER">value</replaceable>' [, ... ] ) ] [
COLLATE<replaceable>collation</replaceable> ] [ <replaceable class="PARAMETER">column_constraint</replaceable> [ ... ]
]   [, ... ]] )
 
+[ INHERITS ( <replaceable>parent_table</replaceable> [, ... ] ) ]  SERVER <replaceable
class="parameter">server_name</replaceable>[OPTIONS ( <replaceable class="PARAMETER">option</replaceable> '<replaceable
class="PARAMETER">value</replaceable>'[, ... ] ) ]
 
@@ -159,6 +160,18 @@ CREATE FOREIGN TABLE [ IF NOT EXISTS ] <replaceable class="PARAMETER">table_name   </varlistentry>
 <varlistentry>
 
+    <term><replaceable class="PARAMETER">parent_table</replaceable></term>
+    <listitem>
+     <para>
+      The name of an existing table from which the new foreign table
+      automatically inherits all columns.  The specified parent table
+      must be a plain table.  See <xref linkend="ddl-inherit"> for the
+      details of table inheritance.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>    <term><replaceable class="PARAMETER">server_name</replaceable></term>    <listitem>     <para>
diff --git a/src/backend/catalog/pg_inherits.c b/src/backend/catalog/pg_inherits.c
index f263b42..b4a084c 100644
--- a/src/backend/catalog/pg_inherits.c
+++ b/src/backend/catalog/pg_inherits.c
@@ -256,6 +256,48 @@ has_subclass(Oid relationId)/*
+ * Check wether the inheritance tree contains foreign table(s).
+ */
+bool
+contains_foreign(Oid parentrelId, LOCKMODE lockmode)
+{
+    bool        result = false;
+    List       *tableOIDs;
+    ListCell   *lc;
+
+    /* Find all members of the inheritance tree */
+    tableOIDs = find_all_inheritors(parentrelId, lockmode, NULL);
+
+    /* There are no children */
+    if (list_length(tableOIDs) < 2)
+        return result;
+
+    foreach(lc, tableOIDs)
+    {
+        Oid            childOID = lfirst_oid(lc);
+        Relation    childrel;
+
+        /* Parent should not be foreign */
+        if (childOID == parentrelId)
+            continue;
+
+        /* We already got the needed lock */
+        childrel = heap_open(childOID, NoLock);
+
+        if (childrel->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
+        {
+            /* Found it */
+            result = true;
+        }
+
+        heap_close(childrel, lockmode);
+    }
+
+    return result;
+}
+
+
+/* * Given two type OIDs, determine whether the first is a complex type * (class type) that inherits from the second.
*/
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index a04adea..bb7bdf4 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -115,7 +115,8 @@ static Datum ind_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull); *    analyze_rel() --
analyzeone relation */void
 
-analyze_rel(Oid relid, VacuumStmt *vacstmt, BufferAccessStrategy bstrategy)
+analyze_rel(Oid relid, VacuumStmt *vacstmt, VacuumMode vacmode,
+            BufferAccessStrategy bstrategy){    Relation    onerel;    int            elevel;
@@ -270,7 +271,9 @@ analyze_rel(Oid relid, VacuumStmt *vacstmt, BufferAccessStrategy bstrategy)    /*     * If there
arechild tables, do recursive ANALYZE.     */
 
-    if (onerel->rd_rel->relhassubclass)
+    if (onerel->rd_rel->relhassubclass &&
+        (vacmode == VAC_MODE_SINGLE ||
+         !contains_foreign(RelationGetRelid(onerel), AccessShareLock)))        do_analyze_rel(onerel, vacstmt,
acquirefunc,relpages, true, elevel);    /*
 
@@ -1452,12 +1455,15 @@ acquire_inherited_sample_rows(Relation onerel, int elevel,{    List       *tableOIDs;
Relation  *rels;
 
+    AcquireSampleRowsFunc *acquirefunc;    double       *relblocks;    double        totalblocks;
+    Relation    saved_rel;    int            numrows,                nrels,                i;    ListCell   *lc;
+    bool        isAnalyzable = true;    /*     * Find all members of inheritance set.  We only need AccessShareLock
on
@@ -1486,6 +1492,8 @@ acquire_inherited_sample_rows(Relation onerel, int elevel,     * BlockNumber, so we use double
arithmetic.    */    rels = (Relation *) palloc(list_length(tableOIDs) * sizeof(Relation));
 
+    acquirefunc = (AcquireSampleRowsFunc *) palloc(list_length(tableOIDs)
+                                            * sizeof(AcquireSampleRowsFunc));    relblocks = (double *)
palloc(list_length(tableOIDs)* sizeof(double));    totalblocks = 0;    nrels = 0;
 
@@ -1507,12 +1515,59 @@ acquire_inherited_sample_rows(Relation onerel, int elevel,        }        rels[nrels] =
childrel;
-        relblocks[nrels] = (double) RelationGetNumberOfBlocks(childrel);
+
+        if (childrel->rd_rel->relkind != RELKIND_FOREIGN_TABLE)
+        {
+            acquirefunc[nrels] = acquire_sample_rows;
+            relblocks[nrels] = (double) RelationGetNumberOfBlocks(childrel);
+        }
+        else
+        {
+            /*
+             * For a foreign table, call the FDW's hook function to see whether
+             * it supports analysis.
+             */
+            FdwRoutine *fdwroutine;
+            BlockNumber relpages = 0;
+            bool        ok = false;
+
+            fdwroutine = GetFdwRoutineForRelation(childrel, false);
+            if (fdwroutine->AnalyzeForeignTable != NULL)
+                ok = fdwroutine->AnalyzeForeignTable(childrel,
+                                                     &acquirefunc[nrels],
+                                                     &relpages);
+            if (!ok)
+            {
+                isAnalyzable = false;
+                break;
+            }
+
+            relblocks[nrels] = (double) relpages;
+        }
+        totalblocks += relblocks[nrels];        nrels++;    }    /*
+     * If there is at least one foreign table that cannot be analyzed, give up.
+     */
+    if (!isAnalyzable)
+    {
+        ereport(WARNING,
+                (errmsg("skipping \"%s\" inheritance tree --- cannot analyze foreign table \"%s\"",
+                        RelationGetRelationName(onerel),
+                        RelationGetRelationName(rels[nrels]))));
+        for (i = 0; i < nrels; i++)
+        {
+            Relation    childrel = rels[i];
+
+            heap_close(childrel, NoLock);
+        }
+        return 0;
+    }
+
+    /*     * Now sample rows from each relation, proportionally to its fraction of     * the total block count.  (This
mightbe less than desirable if the child     * rels have radically different free-space percentages, but it's not
 
@@ -1525,6 +1580,7 @@ acquire_inherited_sample_rows(Relation onerel, int elevel,    {        Relation    childrel =
rels[i];       double        childblocks = relblocks[i];
 
+        AcquireSampleRowsFunc childacquirefunc = acquirefunc[i];        if (childblocks > 0)        {
@@ -1540,12 +1596,12 @@ acquire_inherited_sample_rows(Relation onerel, int elevel,                            tdrows;
            /* Fetch a random sample of the child's rows */
 
-                childrows = acquire_sample_rows(childrel,
-                                                elevel,
-                                                rows + numrows,
-                                                childtargrows,
-                                                &trows,
-                                                &tdrows);
+                childrows = childacquirefunc(childrel,
+                                             elevel,
+                                             rows + numrows,
+                                             childtargrows,
+                                             &trows,
+                                             &tdrows);                /* We may need to convert from child's rowtype
toparent's */                if (childrows > 0 &&
 
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 25f01e5..6859ff0 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -299,26 +299,30 @@ static void validateForeignKeyConstraint(char *conname,static void
createForeignKeyTriggers(Relationrel, Oid refRelOid,                         Constraint *fkconstraint,
      Oid constraintOid, Oid indexOid);
 
+static void PrintForeignNotice(List *affected, Relation rel);static void ATController(Relation rel, List *cmds, bool
recurse,LOCKMODE lockmode);static void ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,          bool
recurse,bool recursing, LOCKMODE lockmode);
 
-static void ATRewriteCatalogs(List **wqueue, LOCKMODE lockmode);
-static void ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
-          AlterTableCmd *cmd, LOCKMODE lockmode);
+static void ATRewriteCatalogs(List **wqueue, List **affectd, LOCKMODE lockmode);
+static void ATExecCmd(List **wqueue, List **affected, AlteredTableInfo *tab,
+                      Relation rel, AlterTableCmd *cmd, LOCKMODE lockmode);static void ATRewriteTables(List **wqueue,
LOCKMODElockmode);static void ATRewriteTable(AlteredTableInfo *tab, Oid OIDNewHeap, LOCKMODE lockmode);static
AlteredTableInfo*ATGetQueueEntry(List **wqueue, Relation rel);
 
+static void ATAddAffectedRelid(List **affected, Oid relid);static void ATSimplePermissions(Relation rel, int
allowed_targets);staticvoid ATWrongRelkindError(Relation rel, int allowed_targets);static void ATSimpleRecursion(List
**wqueue,Relation rel,
 
-                  AlterTableCmd *cmd, bool recurse, LOCKMODE lockmode);
+                  AlterTableCmd *cmd, bool recurse,
+                  bool include_foreign, LOCKMODE lockmode);static void ATTypedTableRecursion(List **wqueue, Relation
rel,AlterTableCmd *cmd,                      LOCKMODE lockmode);static List *find_typed_table_dependencies(Oid typeOid,
constchar *typeName,                              DropBehavior behavior);static void ATPrepAddColumn(List **wqueue,
Relationrel, bool recurse, bool recursing,                AlterTableCmd *cmd, LOCKMODE lockmode);
 
-static void ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
+static void ATExecAddColumn(List **wqueue, List **affected,
+                AlteredTableInfo *tab, Relation rel,                ColumnDef *colDef, bool isOid,                bool
recurse,bool recursing, LOCKMODE lockmode);static void check_for_column_name_collision(Relation rel, const char
*colname);
@@ -341,13 +345,14 @@ static void ATExecSetStorage(Relation rel, const char *colName,                 Node *newValue,
LOCKMODElockmode);static void ATPrepDropColumn(List **wqueue, Relation rel, bool recurse, bool recursing,
 AlterTableCmd *cmd, LOCKMODE lockmode);
 
-static void ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
+static void ATExecDropColumn(List **wqueue, List **affected,
+                 Relation rel, const char *colName,                 DropBehavior behavior,                 bool
recurse,bool recursing,                 bool missing_ok, LOCKMODE lockmode);static void ATExecAddIndex(AlteredTableInfo
*tab,Relation rel,               IndexStmt *stmt, bool is_rebuild, LOCKMODE lockmode);
 
-static void ATExecAddConstraint(List **wqueue,
+static void ATExecAddConstraint(List **wqueue, List **affected,                    AlteredTableInfo *tab, Relation
rel,                   Constraint *newConstraint, bool recurse, bool is_readd,                    LOCKMODE lockmode);
 
@@ -467,10 +472,6 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId)        ereport(ERROR,
(errcode(ERRCODE_INVALID_TABLE_DEFINITION),                errmsg("ON COMMIT can only be used on temporary tables")));
 
-    if (stmt->constraints != NIL && relkind == RELKIND_FOREIGN_TABLE)
-        ereport(ERROR,
-                (errcode(ERRCODE_WRONG_OBJECT_TYPE),
-                 errmsg("constraints are not supported on foreign tables")));    /*     * Look up the namespace in
whichwe are supposed to create the relation,
 
@@ -2940,9 +2941,71 @@ AlterTableGetLockLevel(List *cmds)}static void
+PrintForeignNotice(List *affected, Relation rel)
+{
+    ListCell *l;
+    StringInfo namelist = NULL;
+    bool       rel_is_foreign = false;
+    int           state = 0;
+    int n = 0;
+
+    if (!affected) return;
+
+    foreach (l, affected)
+    {
+        Oid         relid = lfirst_oid(l);
+        Relation r = relation_open(relid, NoLock);
+        if (r->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
+        {
+            if (relid == RelationGetRelid(rel))
+                rel_is_foreign = true;
+
+            switch (state)
+            {
+            case 0:
+                namelist = makeStringInfo();
+                appendStringInfoString(namelist, RelationGetRelationName(r));
+                state = 1;
+                break;
+
+            case 1:
+                if (namelist->len < 32)
+                {
+                    appendStringInfoString(namelist, ", ");
+                    appendStringInfoString(namelist,
+                                           RelationGetRelationName(r));
+                }
+                else
+                {
+                    appendStringInfoString(namelist, "...");
+                    state = 2;
+                }
+                break;
+
+            default:;
+                /* Do nothing */
+            }
+                
+            n++;
+        }
+        relation_close(r, NoLock);
+    }
+
+    /*
+     * Don't show this notice if the relation designated in this command is
+     * the only foreign table affected.
+     */
+    if (n > 1 || (n == 1 && !rel_is_foreign))
+        ereport(NOTICE,
+                (errmsg_plural("This command affects %d foreign table: %s",
+                               "This command affects %d foreign tables: %s",
+                               n, n, namelist->data)));
+}
+
+static voidATController(Relation rel, List *cmds, bool recurse, LOCKMODE lockmode){
-    List       *wqueue = NIL;
+    List       *wqueue = NIL, *affected = NIL;    ListCell   *lcmd;    /* Phase 1: preliminary examination of
commands,create work queue */
 
@@ -2957,10 +3020,14 @@ ATController(Relation rel, List *cmds, bool recurse, LOCKMODE lockmode)    relation_close(rel,
NoLock);   /* Phase 2: update system catalogs */
 
-    ATRewriteCatalogs(&wqueue, lockmode);
+    ATRewriteCatalogs(&wqueue, &affected, lockmode);    /* Phase 3: scan/rewrite tables as needed */
ATRewriteTables(&wqueue,lockmode);
 
+
+    /* Notice if foreign tables are affected by this command */
+    if (affected)
+        PrintForeignNotice(affected, rel);}/*
@@ -3019,24 +3086,28 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,             * rules.             */
         ATSimplePermissions(rel, ATT_TABLE | ATT_VIEW | ATT_FOREIGN_TABLE);
 
-            ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+            /* Recurses to child tables that are foreign, too */
+            ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode);            /* No command-specific prep
needed*/            pass = cmd->def ? AT_PASS_ADD_CONSTR : AT_PASS_DROP;            break;        case AT_DropNotNull:
 /* ALTER COLUMN DROP NOT NULL */            ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
 
-            ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+            /* Recurses to child tables that are foreign, too */
+            ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode);            /* No command-specific prep
needed*/            pass = AT_PASS_DROP;            break;        case AT_SetNotNull:        /* ALTER COLUMN SET NOT
NULL*/            ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
 
-            ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+            /* Recurses to child tables that are foreign, too */
+            ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode);            /* No command-specific prep
needed*/            pass = AT_PASS_ADD_CONSTR;            break;        case AT_SetStatistics:    /* ALTER COLUMN SET
STATISTICS*/
 
-            ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+            /* Recurses to child tables that are foreign, too */
+            ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode);            /* Performs own permission checks
*/           ATPrepSetStatistics(rel, cmd->name, cmd->def, lockmode);            pass = AT_PASS_MISC;
 
@@ -3049,7 +3120,8 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,            break;        case
AT_SetStorage:       /* ALTER COLUMN SET STORAGE */            ATSimplePermissions(rel, ATT_TABLE | ATT_MATVIEW);
 
-            ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+            /* Don't recurse to child tables that are foreign */
+            ATSimpleRecursion(wqueue, rel, cmd, recurse, false, lockmode);            /* No command-specific prep
needed*/            pass = AT_PASS_MISC;            break;
 
@@ -3067,7 +3139,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,            pass = AT_PASS_ADD_INDEX;
        break;        case AT_AddConstraint:    /* ADD CONSTRAINT */
 
-            ATSimplePermissions(rel, ATT_TABLE);
+            ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);            /* Recursion occurs during execution
phase*/            /* No command-specific prep needed except saving recurse flag */            if (recurse)
 
@@ -3081,7 +3153,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,            pass = AT_PASS_ADD_CONSTR;
         break;        case AT_DropConstraint:    /* DROP CONSTRAINT */
 
-            ATSimplePermissions(rel, ATT_TABLE);
+            ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);            /* Recursion occurs during execution
phase*/            /* No command-specific prep needed except saving recurse flag */            if (recurse)
 
@@ -3149,13 +3221,19 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,            pass = AT_PASS_MISC;
     break;        case AT_AddInherit:        /* INHERIT */
 
-            ATSimplePermissions(rel, ATT_TABLE);
+            ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);            /* This command never recurses */
     ATPrepAddInherit(rel);            pass = AT_PASS_MISC;            break;
 
+        case AT_DropInherit:    /* NO INHERIT */
+            ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
+            /* This command never recurses */
+            /* No command-specific prep needed */
+            pass = AT_PASS_MISC;
+            break;        case AT_AlterConstraint:        /* ALTER CONSTRAINT */
-            ATSimplePermissions(rel, ATT_TABLE);
+            ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);            pass = AT_PASS_MISC;            break;
      case AT_ValidateConstraint:        /* VALIDATE CONSTRAINT */
 
@@ -3184,7 +3262,6 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,        case AT_EnableAlwaysRule:
case AT_EnableReplicaRule:        case AT_DisableRule:
 
-        case AT_DropInherit:    /* NO INHERIT */        case AT_AddOf:            /* OF */        case AT_DropOf: /*
NOTOF */            ATSimplePermissions(rel, ATT_TABLE);
 
@@ -3217,7 +3294,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, * conflicts). */static void
-ATRewriteCatalogs(List **wqueue, LOCKMODE lockmode)
+ATRewriteCatalogs(List **wqueue, List **affected, LOCKMODE lockmode){    int            pass;    ListCell   *ltab;
@@ -3248,7 +3325,8 @@ ATRewriteCatalogs(List **wqueue, LOCKMODE lockmode)            rel = relation_open(tab->relid,
NoLock);           foreach(lcmd, subcmds)
 
-                ATExecCmd(wqueue, tab, rel, (AlterTableCmd *) lfirst(lcmd), lockmode);
+                ATExecCmd(wqueue, affected, tab, rel,
+                          (AlterTableCmd *) lfirst(lcmd), lockmode);            /*             * After the ALTER TYPE
pass,do cleanup work (this is not done in
 
@@ -3277,19 +3355,21 @@ ATRewriteCatalogs(List **wqueue, LOCKMODE lockmode) * ATExecCmd: dispatch a subcommand to
appropriateexecution routine */static void
 
-ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
+ATExecCmd(List **wqueue, List **affected, AlteredTableInfo *tab, Relation rel,          AlterTableCmd *cmd, LOCKMODE
lockmode){
+    ATAddAffectedRelid(affected, RelationGetRelid(rel));
+    switch (cmd->subtype)    {        case AT_AddColumn:        /* ADD COLUMN */        case AT_AddColumnToView:
/* add column via CREATE OR REPLACE                                         * VIEW */
 
-            ATExecAddColumn(wqueue, tab, rel, (ColumnDef *) cmd->def,
+            ATExecAddColumn(wqueue, affected, tab, rel, (ColumnDef *) cmd->def,                            false,
false,false, lockmode);            break;        case AT_AddColumnRecurse:
 
-            ATExecAddColumn(wqueue, tab, rel, (ColumnDef *) cmd->def,
+            ATExecAddColumn(wqueue, affected, tab, rel, (ColumnDef *) cmd->def,                            false,
true,false, lockmode);            break;        case AT_ColumnDefault:    /* ALTER COLUMN DEFAULT */
 
@@ -3314,11 +3394,11 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,            ATExecSetStorage(rel,
cmd->name,cmd->def, lockmode);            break;        case AT_DropColumn:        /* DROP COLUMN */
 
-            ATExecDropColumn(wqueue, rel, cmd->name,
+            ATExecDropColumn(wqueue, affected, rel, cmd->name,                     cmd->behavior, false, false,
cmd->missing_ok,lockmode);            break;        case AT_DropColumnRecurse:        /* DROP COLUMN with recursion */
 
-            ATExecDropColumn(wqueue, rel, cmd->name,
+            ATExecDropColumn(wqueue, affected, rel, cmd->name,                      cmd->behavior, true, false,
cmd->missing_ok,lockmode);            break;        case AT_AddIndex:        /* ADD INDEX */
 
@@ -3328,16 +3408,19 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,            ATExecAddIndex(tab,
rel,(IndexStmt *) cmd->def, true, lockmode);            break;        case AT_AddConstraint:    /* ADD CONSTRAINT */
 
-            ATExecAddConstraint(wqueue, tab, rel, (Constraint *) cmd->def,
+            ATExecAddConstraint(wqueue, affected, tab, rel,
+                                (Constraint *) cmd->def,                                 false, false, lockmode);
     break;        case AT_AddConstraintRecurse:    /* ADD CONSTRAINT with recursion */
 
-            ATExecAddConstraint(wqueue, tab, rel, (Constraint *) cmd->def,
+            ATExecAddConstraint(wqueue, affected, tab, rel,
+                                (Constraint *) cmd->def,                                true, false, lockmode);
   break;        case AT_ReAddConstraint:        /* Re-add pre-existing check                                         *
constraint*/
 
-            ATExecAddConstraint(wqueue, tab, rel, (Constraint *) cmd->def,
+            ATExecAddConstraint(wqueue, affected,
+                                tab, rel, (Constraint *) cmd->def,                                false, true,
lockmode);           break;        case AT_AddIndexConstraint:        /* ADD CONSTRAINT USING INDEX */
 
@@ -3383,13 +3466,15 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,        case AT_AddOids:        /*
SETWITH OIDS */            /* Use the ADD COLUMN code, unless prep decided to do nothing */            if (cmd->def !=
NULL)
-                ATExecAddColumn(wqueue, tab, rel, (ColumnDef *) cmd->def,
+                ATExecAddColumn(wqueue, affected,
+                                tab, rel, (ColumnDef *) cmd->def,                                true, false, false,
lockmode);           break;        case AT_AddOidsRecurse:    /* SET WITH OIDS */            /* Use the ADD COLUMN
code,unless prep decided to do nothing */            if (cmd->def != NULL)
 
-                ATExecAddColumn(wqueue, tab, rel, (ColumnDef *) cmd->def,
+                ATExecAddColumn(wqueue, affected,
+                                tab, rel, (ColumnDef *) cmd->def,                                true, true, false,
lockmode);           break;        case AT_DropOids:        /* SET WITHOUT OIDS */
 
@@ -4009,6 +4094,23 @@ ATGetQueueEntry(List **wqueue, Relation rel)}/*
+ * ATAddAffectedRelid: add a relid to affected list.
+ */
+static void
+ATAddAffectedRelid(List **affected, Oid relid)
+{
+    ListCell   *l;
+
+    foreach(l, *affected)
+    {
+        if (lfirst_oid(l) == relid)
+            return;
+    }
+
+    *affected = lappend_oid(*affected, relid);
+}
+
+/* * ATSimplePermissions * * - Ensure that it is a relation (or possibly a view)
@@ -4125,7 +4227,8 @@ ATWrongRelkindError(Relation rel, int allowed_targets) */static voidATSimpleRecursion(List
**wqueue,Relation rel,
 
-                  AlterTableCmd *cmd, bool recurse, LOCKMODE lockmode)
+                  AlterTableCmd *cmd, bool recurse,
+                  bool include_foreign, LOCKMODE lockmode){    /*     * Propagate to children if desired.  Non-table
relationsnever have
 
@@ -4153,8 +4256,12 @@ ATSimpleRecursion(List **wqueue, Relation rel,                continue;            /*
find_all_inheritorsalready got lock */            childrel = relation_open(childrelid, NoLock);
 
-            CheckTableNotInUse(childrel, "ALTER TABLE");
-            ATPrepCmd(wqueue, childrel, cmd, false, true, lockmode);
+            if (childrel->rd_rel->relkind != RELKIND_FOREIGN_TABLE
+                || include_foreign)
+            {
+                CheckTableNotInUse(childrel, "ALTER TABLE");
+                ATPrepCmd(wqueue, childrel, cmd, false, true, lockmode);
+            }            relation_close(childrel, NoLock);        }    }
@@ -4421,7 +4528,8 @@ ATPrepAddColumn(List **wqueue, Relation rel, bool recurse, bool recursing,}static void
-ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
+ATExecAddColumn(List **wqueue, List **affected,
+                AlteredTableInfo *tab, Relation rel,                ColumnDef *colDef, bool isOid,                bool
recurse,bool recursing, LOCKMODE lockmode){
 
@@ -4442,9 +4550,11 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,    ListCell   *child;
AclResult   aclresult;
 
+    ATAddAffectedRelid(affected, myrelid);
+    /* At top level, permission check was done in ATPrepCmd, else do it */    if (recursing)
-        ATSimplePermissions(rel, ATT_TABLE);
+        ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);    attrdesc = heap_open(AttributeRelationId,
RowExclusiveLock);
@@ -4746,7 +4856,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,        childtab =
ATGetQueueEntry(wqueue,childrel);        /* Recurse to child */
 
-        ATExecAddColumn(wqueue, childtab, childrel,
+        ATExecAddColumn(wqueue, affected, childtab, childrel,                        colDef, isOid, recurse, true,
lockmode);       heap_close(childrel, NoLock);
 
@@ -4859,7 +4969,19 @@ ATPrepAddOids(List **wqueue, Relation rel, bool recurse, AlterTableCmd *cmd, LOC
ATPrepAddColumn(wqueue,rel, recurse, false, cmd, lockmode);    if (recurse)
 
+    {
+        /*
+         * Don't allow to add an OID column to inheritance tree that contains
+         * foreign table(s)
+         */
+      if (contains_foreign(RelationGetRelid(rel), AccessShareLock))
+            ereport(ERROR,
+                    (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+                     errmsg("cannot add OID column to inheritance tree \"%s\" because it contains foreign table(s)",
+                            RelationGetRelationName(rel))));
+        cmd->subtype = AT_AddOidsRecurse;
+    }}/*
@@ -5327,7 +5449,8 @@ ATPrepDropColumn(List **wqueue, Relation rel, bool recurse, bool recursing,}static void
-ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
+ATExecDropColumn(List **wqueue, List **affected,
+                 Relation rel, const char *colName,                 DropBehavior behavior,                 bool
recurse,bool recursing,                 bool missing_ok, LOCKMODE lockmode)
 
@@ -5338,9 +5461,11 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,    List       *children;
ObjectAddressobject;
 
+    ATAddAffectedRelid(affected, RelationGetRelid(rel));
+    /* At top level, permission check was done in ATPrepCmd, else do it */    if (recursing)
-        ATSimplePermissions(rel, ATT_TABLE);
+        ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);    /*     * get the number of the attribute
@@ -5426,7 +5551,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,                if
(childatt->attinhcount== 1 && !childatt->attislocal)                {                    /* Time to delete this child
column,too */
 
-                    ATExecDropColumn(wqueue, childrel, colName,
+                    ATExecDropColumn(wqueue, affected, childrel, colName,
behavior,true, true,                                     false, lockmode);                }
 
@@ -5644,12 +5769,15 @@ ATExecAddIndexConstraint(AlteredTableInfo *tab, Relation rel, * ALTER TABLE ADD CONSTRAINT
*/staticvoid
 
-ATExecAddConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel,
+ATExecAddConstraint(List **wqueue, List **affected,
+                    AlteredTableInfo *tab, Relation rel,                    Constraint *newConstraint, bool recurse,
boolis_readd,                    LOCKMODE lockmode){    Assert(IsA(newConstraint, Constraint));
 
+    ATAddAffectedRelid(affected, RelationGetRelid(rel));
+    /*     * Currently, we only expect to see CONSTR_CHECK and CONSTR_FOREIGN nodes     * arriving here (see the
preprocessingdone in parse_utilcmd.c).  Use a
 
@@ -5732,7 +5860,14 @@ ATAddCheckConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel,    /* At top level,
permissioncheck was done in ATPrepCmd, else do it */    if (recursing)
 
-        ATSimplePermissions(rel, ATT_TABLE);
+        ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
+
+    /* Don't allow ADD constraint NOT VALID on foreign tables */
+    if (tab->relkind == RELKIND_FOREIGN_TABLE &&
+        constr->skip_validation && !recursing)
+        ereport(ERROR,
+                (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+                 errmsg("NOT VALID is not supported on foreign tables")));    /*     * Call AddRelationNewConstraints
todo the work, making sure it works on
 
@@ -5743,9 +5878,17 @@ ATAddCheckConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel,     * omitted from the
returnedlist, which is what we want: we do not need     * to do any validation work.  That can only happen at child
tables,    * though, since we disallow merging at the top level.
 
-     */
+     *
+     * When propagating a NOT VALID option to children that are foreign tables,
+     * we quietly ignore the option.  Note that this is safe because foreign
+     * tables don't have any children.
+     */
+    constr = copyObject(constr);
+    if (tab->relkind == RELKIND_FOREIGN_TABLE &&
+        constr->skip_validation && recursing)
+        constr->skip_validation = false;    newcons = AddRelationNewConstraints(rel, NIL,
-                                        list_make1(copyObject(constr)),
+                                        list_make1(constr),                                        recursing,
/*allow_merge */                                        !recursing,        /* is_local */
        is_readd);        /* is_internal */
 
@@ -7225,7 +7368,7 @@ ATExecDropConstraint(Relation rel, const char *constrName,    /* At top level, permission check
wasdone in ATPrepCmd, else do it */    if (recursing)
 
-        ATSimplePermissions(rel, ATT_TABLE);
+        ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);    conrel = heap_open(ConstraintRelationId,
RowExclusiveLock);
@@ -7560,7 +7703,10 @@ ATPrepAlterColumnType(List **wqueue,     * alter would put them out of step.     */    if
(recurse)
-        ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+    {
+        /* Recurses to child tables that are foreign, too */
+        ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode);
+    }    else if (!recursing &&             find_inheritance_children(RelationGetRelid(rel), NoLock) != NIL)
ereport(ERROR,
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index ded1841..35b5dd2 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -98,6 +98,7 @@ vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast,       BufferAccessStrategy bstrategy, bool
for_wraparound,bool isTopLevel){    const char *stmttype;
 
+    VacuumMode    vacmode;    volatile bool in_outer_xact,                use_own_xacts;    List       *relations;
@@ -146,6 +147,20 @@ vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast,
ALLOCSET_DEFAULT_MAXSIZE);   /*
 
+     * Identify vacuum mode.  If relid is not InvalidOid, the caller should be
+     * an autovacuum worker.  See the above comments.
+     */
+    if (relid != InvalidOid)
+        vacmode = VAC_MODE_AUTOVACUUM;
+    else
+    {
+        if (!vacstmt->relation)
+            vacmode = VAC_MODE_ALL;
+        else
+            vacmode = VAC_MODE_SINGLE;
+    }
+
+    /*     * If caller didn't give us a buffer strategy object, make one in the     * cross-transaction memory
context.    */
 
@@ -248,7 +263,7 @@ vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast,
PushActiveSnapshot(GetTransactionSnapshot());               }
 
-                analyze_rel(relid, vacstmt, vac_strategy);
+                analyze_rel(relid, vacstmt, vacmode, vac_strategy);                if (use_own_xacts)
{
diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c
index 52dcc72..238bba2 100644
--- a/src/backend/optimizer/prep/prepunion.c
+++ b/src/backend/optimizer/prep/prepunion.c
@@ -1337,11 +1337,12 @@ expand_inherited_rtentry(PlannerInfo *root, RangeTblEntry *rte, Index rti)        /*         *
Buildan RTE for the child, and attach to query's rangetable list.         * We copy most fields of the parent's RTE,
butreplace relation OID,
 
-         * and set inh = false.  Also, set requiredPerms to zero since all
-         * required permissions checks are done on the original RTE.
+         * relkind and set inh = false.  Also, set requiredPerms to zero since
+         * all required permissions checks are done on the original RTE.         */        childrte = copyObject(rte);
      childrte->relid = childOID;
 
+        childrte->relkind = newrelation->rd_rel->relkind;        childrte->inh = false;        childrte->requiredPerms
=0;        parse->rtable = lappend(parse->rtable, childrte);
 
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index b79af7a..18ced04 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -2062,6 +2062,16 @@ reparameterize_path(PlannerInfo *root, Path *path,        case T_SubqueryScan:            return
create_subqueryscan_path(root,rel, path->pathkeys,                                            required_outer);
 
+        case T_ForeignScan:
+            {
+                ForeignPath *fpath = (ForeignPath*) path;
+                ForeignPath *newpath = makeNode(ForeignPath);
+                memcpy(newpath, fpath, sizeof(ForeignPath));
+                newpath->path.param_info =
+                    get_baserel_parampathinfo(root, rel, required_outer);
+                /* cost recalc omitted */
+                return (Path *)newpath;
+            }        default:            break;    }
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index e3060a4..b5e47f4 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -4209,32 +4209,32 @@ AlterForeignServerStmt: ALTER SERVER name foreign_server_version
alter_generic_oCreateForeignTableStmt:       CREATE FOREIGN TABLE qualified_name            '(' OptTableElementList
')'
-            SERVER name create_generic_options
+            OptInherit SERVER name create_generic_options                {                    CreateForeignTableStmt
*n= makeNode(CreateForeignTableStmt);                    $4->relpersistence = RELPERSISTENCE_PERMANENT;
  n->base.relation = $4;                    n->base.tableElts = $6;
 
-                    n->base.inhRelations = NIL;
+                    n->base.inhRelations = $8;                    n->base.if_not_exists = false;                    /*
FDW-specificdata */
 
-                    n->servername = $9;
-                    n->options = $10;
+                    n->servername = $10;
+                    n->options = $11;                    $$ = (Node *) n;                }        | CREATE FOREIGN
TABLEIF_P NOT EXISTS qualified_name            '(' OptTableElementList ')'
 
-            SERVER name create_generic_options
+            OptInherit SERVER name create_generic_options                {                    CreateForeignTableStmt
*n= makeNode(CreateForeignTableStmt);                    $7->relpersistence = RELPERSISTENCE_PERMANENT;
  n->base.relation = $7;                    n->base.tableElts = $9;
 
-                    n->base.inhRelations = NIL;
+                    n->base.inhRelations = $11;                    n->base.if_not_exists = true;                    /*
FDW-specificdata */
 
-                    n->servername = $12;
-                    n->options = $13;
+                    n->servername = $13;
+                    n->options = $14;                    $$ = (Node *) n;                }        ;
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index 1e071d7..955f27c 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -515,12 +515,6 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column)                break;
  case CONSTR_CHECK:
 
-                if (cxt->isforeign)
-                    ereport(ERROR,
-                            (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
-                    errmsg("constraints are not supported on foreign tables"),
-                             parser_errposition(cxt->pstate,
-                                                constraint->location)));                cxt->ckconstraints =
lappend(cxt->ckconstraints,constraint);                break;
 
@@ -529,7 +523,7 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column)                if
(cxt->isforeign)                   ereport(ERROR,                            (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
 
-                    errmsg("constraints are not supported on foreign tables"),
+                    errmsg("primary key or unique constraints are not supported on foreign tables"),
         parser_errposition(cxt->pstate,                                                constraint->location)));
       if (constraint->keys == NIL)
 
@@ -546,7 +540,7 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column)                if
(cxt->isforeign)                   ereport(ERROR,                            (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
 
-                    errmsg("constraints are not supported on foreign tables"),
+                    errmsg("foreign key constraints are not supported on foreign tables"),
parser_errposition(cxt->pstate,                                               constraint->location)));
 
@@ -605,10 +599,14 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column)static
voidtransformTableConstraint(CreateStmtContext*cxt, Constraint *constraint){
 
-    if (cxt->isforeign)
+    if (cxt->isforeign &&
+        (constraint->contype == CONSTR_PRIMARY ||
+         constraint->contype == CONSTR_UNIQUE ||
+         constraint->contype == CONSTR_EXCLUSION ||
+         constraint->contype == CONSTR_FOREIGN))        ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
-                 errmsg("constraints are not supported on foreign tables"),
+                 errmsg("primary key, unique, exclusion, or foreign key constraints are not supported on foreign
tables"),                parser_errposition(cxt->pstate,                                    constraint->location)));
 
diff --git a/src/include/catalog/pg_inherits_fn.h b/src/include/catalog/pg_inherits_fn.h
index 757d849..228573a 100644
--- a/src/include/catalog/pg_inherits_fn.h
+++ b/src/include/catalog/pg_inherits_fn.h
@@ -21,6 +21,7 @@ extern List *find_inheritance_children(Oid parentrelId, LOCKMODE lockmode);extern List
*find_all_inheritors(OidparentrelId, LOCKMODE lockmode,                    List **parents);extern bool has_subclass(Oid
relationId);
+extern bool contains_foreign(Oid parentrelId, LOCKMODE lockmode);extern bool typeInheritsFrom(Oid subclassTypeId, Oid
superclassTypeId);#endif  /* PG_INHERITS_FN_H */
 
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 058dc5f..8253a07 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -140,6 +140,15 @@ extern int    vacuum_multixact_freeze_min_age;extern int    vacuum_multixact_freeze_table_age;
+/* Possible modes for vacuum() */
+typedef enum
+{
+    VAC_MODE_ALL,                /* Vacuum/analyze all relations */
+    VAC_MODE_SINGLE,            /* Vacuum/analyze a specific relation */
+    VAC_MODE_AUTOVACUUM            /* Autovacuum worker */
+} VacuumMode;
+
+/* in commands/vacuum.c */extern void vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast,       BufferAccessStrategy
bstrategy,bool for_wraparound, bool isTopLevel);
 
@@ -174,7 +183,7 @@ extern void lazy_vacuum_rel(Relation onerel, VacuumStmt *vacstmt,
BufferAccessStrategybstrategy);/* in commands/analyze.c */
 
-extern void analyze_rel(Oid relid, VacuumStmt *vacstmt,
+extern void analyze_rel(Oid relid, VacuumStmt *vacstmt, VacuumMode vacmode,            BufferAccessStrategy
bstrategy);externbool std_typanalyze(VacAttrStats *stats);extern double anl_random_fract(void);
 
diff --git a/src/test/regress/expected/foreign_data.out b/src/test/regress/expected/foreign_data.out
index 60506e0..f15d2e1 100644
--- a/src/test/regress/expected/foreign_data.out
+++ b/src/test/regress/expected/foreign_data.out
@@ -750,16 +750,12 @@ CREATE TABLE use_ft1_column_type (x ft1);ALTER FOREIGN TABLE ft1 ALTER COLUMN c8 SET DATA TYPE
integer;   -- ERRORERROR:  cannot alter foreign table "ft1" because column "use_ft1_column_type.x" uses its row
typeDROPTABLE use_ft1_column_type;
 
-ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0); -- ERROR
-ERROR:  constraints are not supported on foreign tables
-LINE 1: ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c...
-                                    ^
+ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0);ALTER FOREIGN TABLE ft1 DROP CONSTRAINT no_const;
           -- ERROR
 
-ERROR:  "ft1" is not a table
+ERROR:  constraint "no_const" of relation "ft1" does not existALTER FOREIGN TABLE ft1 DROP CONSTRAINT IF EXISTS
no_const;
-ERROR:  "ft1" is not a table
-ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c1_check;
-ERROR:  "ft1" is not a table
+NOTICE:  constraint "no_const" of relation "ft1" does not exist, skipping
+ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c9_check;ALTER FOREIGN TABLE ft1 SET WITH OIDS;
--ERRORERROR:  "ft1" is not a tableALTER FOREIGN TABLE ft1 OWNER TO regress_test_role;
 
diff --git a/src/test/regress/sql/foreign_data.sql b/src/test/regress/sql/foreign_data.sql
index f819eb1..2aa5ffe 100644
--- a/src/test/regress/sql/foreign_data.sql
+++ b/src/test/regress/sql/foreign_data.sql
@@ -314,10 +314,10 @@ ALTER FOREIGN TABLE ft1 ALTER COLUMN c8 SET STATISTICS -1;CREATE TABLE use_ft1_column_type (x
ft1);ALTERFOREIGN TABLE ft1 ALTER COLUMN c8 SET DATA TYPE integer;    -- ERRORDROP TABLE use_ft1_column_type;
 
-ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0); -- ERROR
+ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0);ALTER FOREIGN TABLE ft1 DROP CONSTRAINT no_const;
           -- ERRORALTER FOREIGN TABLE ft1 DROP CONSTRAINT IF EXISTS no_const;
 
-ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c1_check;
+ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c9_check;ALTER FOREIGN TABLE ft1 SET WITH OIDS;
--ERRORALTER FOREIGN TABLE ft1 OWNER TO regress_test_role;ALTER FOREIGN TABLE ft1 OPTIONS (DROP delimiter, SET quote
'~',ADD escape '@'); 

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/03/11 14:07), Kyotaro HORIGUCHI wrote:
>> This seems far better than silently performing the command,
>> except for the duplicated message:( New bitmap might required to
>> avoid the duplication..
>
> I rewrote it in more tidy way. ATController collects all affected
> tables on ATRewriteCatalogs as first stage, then emit NOTICE
> message according to the affected relations list. The message
> looks like,
>
> | =# alter table passwd alter column uname set default 'hoge';
> | NOTICE:  This command affects 2 foreign tables: cf1, cf2
> | ALTER TABLE
>
> Do you feel this too large or complicated? I think so a bit..

No.  I think that would be a useful message for the user.

My feeling is it would be better to show this kind of messages for all 
the affected tables whether or not the affected ones are foreign.  How 
about introducing a VERBOSE option for ALTER TABLE?  Though, I think 
that should be implemented as a separate patch.

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
Hi Horiguchi-san,

Thank you for working this patch!

(2014/03/10 17:29), Kyotaro HORIGUCHI wrote:
> Hello. As a minimal implementation, I made an attempt that emit
> NOTICE message when alter table affects foreign tables. It looks
> like following,
>
> | =# alter table passwd add column added int, add column added2 int;
> | NOTICE:  This command affects foreign relation "cf1"
> | NOTICE:  This command affects foreign relation "cf1"
> | ALTER TABLE
> | =# select * from passwd;
> | ERROR:  missing data for column "added"
> | CONTEXT:  COPY cf1, line 1: "root:x:0:0:root:/root:/bin/bash"
> | =#
>
> This seems far better than silently performing the command,
> except for the duplicated message:( New bitmap might required to
> avoid the duplication..

As I said before, I think it would be better to show this kind of
information on each of the affected tables whether or not the affected
one is foreign.  I also think it would be better to show it only when
the user has specified an option to do so, similar to a VERBOSE option
of other commands.  ISTM this should be implemented as a separate patch.

> I made the changes above and below as an attempt in the attached
> patch foreign_inherit-v4.patch
>
>> I think the problem is foreign childs in inheritance tables
>> prevents all menber in the inheritance relation from using
>> parameterized paths, correct?

Yes, I think so, too.

>> Hmm. I tried minimal implementation to do that. This omits cost
>> recalculation but seems to work as expected. This seems enough if
>> cost recalc is trivial here.

I think we should redo the cost/size estimate, because for example,
greater parameterization leads to a smaller rowcount estimate, if I
understand correctly.  In addition, I think this reparameterization
should be done by the FDW itself, becasuse the FDW has more knowledge
about it than the PG core.  So, I think we should introduce a new FDW
routine for that, say ReparameterizeForeignPath(), as proposed in [1].
Attached is an updated version of the patch.  Due to the above reason, I
removed from the patch the message displaying function you added.

Sorry for the delay.

[1] http://www.postgresql.org/message-id/530C7464.6020006@lab.ntt.co.jp

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hi Fujita-san,

> Thank you for working this patch!

No problem, but my point seems always out of the main target a bit:(

> > | =# alter table passwd add column added int, add column added2 int;
> > | NOTICE:  This command affects foreign relation "cf1"
> > | NOTICE:  This command affects foreign relation "cf1"
> > | ALTER TABLE
> > | =# select * from passwd;
> > | ERROR:  missing data for column "added"
> > | CONTEXT:  COPY cf1, line 1: "root:x:0:0:root:/root:/bin/bash"
> > | =#
> >
> > This seems far better than silently performing the command,
> > except for the duplicated message:( New bitmap might required to
> > avoid the duplication..
> 
> As I said before, I think it would be better to show this kind of
> information on each of the affected tables whether or not the affected
> one is foreign.  I also think it would be better to show it only when
> the user has specified an option to do so, similar to a VERBOSE option
> of other commands.  ISTM this should be implemented as a separate
> patch.

Hmm. I *wish* this kind of indication to be with this patch even
only for foreign tables which would have inconsistency
potentially. Expanding to other objects and/or knobs are no
problem to be added later.

> >> Hmm. I tried minimal implementation to do that. This omits cost
> >> recalculation but seems to work as expected. This seems enough if
> >> cost recalc is trivial here.
> 
> I think we should redo the cost/size estimate, because for example,
> greater parameterization leads to a smaller rowcount estimate, if I
> understand correctly.  In addition, I think this reparameterization
> should be done by the FDW itself, becasuse the FDW has more knowledge
> about it than the PG core.  So, I think we should introduce a new FDW
> routine for that, say ReparameterizeForeignPath(), as proposed in
> [1]. Attached is an updated version of the patch.  Due to the above
> reason, I removed from the patch the message displaying function you
> added.
> 
> Sorry for the delay.
> 
> [1]
> http://www.postgresql.org/message-id/530C7464.6020006@lab.ntt.co.jp

I had a rough look on foreign reparameterize stuff. It seems ok
generally.

By the way, Can I have a simple script to build an environment to
run this on?

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hello,

> By the way, Can I have a simple script to build an environment to
> run this on?

I built test environment and ran the simple test using
postgres_fdw and got parameterized path from v3 patch on the
following operation as shown there, and v6 also gives one, but I
haven't seen the reparameterization of v6 patch work.

# How could I think to have got it work before?

Do you have any idea to make postgreReparameterizeForeignPath on
foreign (child) tables works effectively?

regards,

====
### on pg1/pg2:
create table pu1 (a int not null, b int not null, c int, d text);
create unique index i_pu1_ab on pu1 (a, b);
create unique index i_pu1_c  on pu1 (c);
create table cu11 (like pu1 including all) inherits (pu1);
create table cu12 (like pu1 including all) inherits (pu1);
insert into cu11 (select a / 5, 4 - (a % 5), a, 'cu11' from generate_series(000000, 099999) a);
insert into cu12 (select a / 5, 4 - (a % 5), a, 'cu12' from generate_series(100000, 199999) a);

### on pg1:
create extension postgres_fdw;
create server pg2 foreign data wrapper postgres_fdw options (host '/tmp', port '5433', dbname 'postgres');
create user mapping for current_user server pg2 options (user 'horiguti');
create foreign table _cu11 (a int, b int, c int, d text) server pg2 options (table_name 'cu11', use_remote_estimate
'true');
create foreign table _cu12 (a int, b int, c int, d text) server pg2 options (table_name 'cu12', use_remote_estimate
'true');
create table rpu1 (a int, b int, c int, d text);
alter foreign table _cu11 inherit rpu1;
alter foreign table _cu12 inherit rpu1;
analyze rpu1;

=# explain analyze select pu1.*   from pu1 join rpu1 on (pu1.c = rpu1.c) where pu1.a = 3;
    QUERY PLAN
 
-------------------------------------------------------------------------------Nested Loop  (cost=0.00..2414.57 rows=11
width=19)            (actual time=0.710..7.167 rows=5 loops=1)  ->  Append  (cost=0.00..30.76 rows=11 width=19)
<localside.. ommitted>  ->  Append  (cost=0.00..216.68 rows=3 width=4)              (actual time=0.726..1.419 rows=1
loops=5)       ->  Seq Scan on rpu1  (cost=0.00..0.00 rows=1 width=4)                              (actual
time=0.000..0.000rows=0 loops=5)              Filter: (pu1.c = c)        ->  Foreign Scan on _cu11
(cost=100.30..108.34rows=1 width=4)                                   (actual time=0.602..0.603 rows=1 loops=5)
-> Foreign Scan on _cu12  (cost=100.30..108.34 rows=1 width=4)                                   (actual
time=0.566..0.566rows=0 loops=5)Planning time: 4.452 msTotal runtime: 7.663 ms
 
(15 rows)

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/03/18 18:38), Kyotaro HORIGUCHI wrote:
>> By the way, Can I have a simple script to build an environment to
>> run this on?
>
> I built test environment and ran the simple test using
> postgres_fdw and got parameterized path from v3 patch on the
> following operation as shown there, and v6 also gives one, but I
> haven't seen the reparameterization of v6 patch work.
>
> # How could I think to have got it work before?
>
> Do you have any idea to make postgreReparameterizeForeignPath on
> foreign (child) tables works effectively?

> =# explain analyze select pu1.*
>      from pu1 join rpu1 on (pu1.c = rpu1.c) where pu1.a = 3;

ISTM postgresReparameterizeForeignPath() cannot be called in this query
in principle.  Here is a simple example for the case where the
use_remote_estimate option is true:

# On mydatabase

mydatabase=# CREATE TABLE mytable (id INTEGER, x INTEGER);
CREATE TABLE
mydatabase=# INSERT INTO mytable SELECT x, x FROM generate_series(0,
9999) x;
INSERT 0 10000

# On postgres

postgres=# CREATE TABLE inttable (id INTEGER);
CREATE TABLE
postgres=# INSERT INTO inttable SELECT x FROM generate_series(0, 9999) x;
INSERT 0 10000
postgres=# ANALYZE inttable;
ANALYZE

postgres=# CREATE TABLE patest0 (id INTEGER, x INTEGER);
CREATE TABLE
postgres=# CREATE TABLE patest1 () INHERITS (patest0);
CREATE TABLE
postgres=# INSERT INTO patest1 SELECT x, x FROM generate_series(0, 9999) x;
INSERT 0 10000
postgres=# CREATE INDEX patest1_id_idx ON patest1(id);
CREATE INDEX
postgres=# CREATE SERVER myserver FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (host 'localhost', dbname 'mydatabase');
CREATE SERVER
postgres=# CREATE USER MAPPING FOR PUBLIC SERVER myserver OPTIONS (user
'pgsql');
CREATE USER MAPPING
postgres=# CREATE FOREIGN TABLE patest2 () INHERITS (patest0) SERVER
myserver OPTIONS (table_name 'mytable');
CREATE FOREIGN TABLE
postgres=# ANALYZE patest0;
ANALYZE
postgres=# ANALYZE patest1;
ANALYZE
postgres=# ANALYZE patest2;
ANALYZE
postgres=# EXPLAIN VERBOSE SELECT * FROM patest0 join (SELECT id FROM
inttable LIMIT 1) ss ON patest0.id = ss.id;
                                            QUERY PLAN
-------------------------------------------------------------------------------------------------
  Nested Loop  (cost=0.00..478.36 rows=2 width=12)
    Output: patest0.id, patest0.x, inttable.id
    ->  Limit  (cost=0.00..0.01 rows=1 width=4)
          Output: inttable.id
          ->  Seq Scan on public.inttable  (cost=0.00..145.00 rows=10000
width=4)
                Output: inttable.id
    ->  Append  (cost=0.00..478.31 rows=3 width=8)
          ->  Seq Scan on public.patest0  (cost=0.00..0.00 rows=1 width=8)
                Output: patest0.id, patest0.x
                Filter: (inttable.id = patest0.id)
          ->  Index Scan using patest1_id_idx on public.patest1
(cost=0.29..8.30 rows=1 width=8)
                Output: patest1.id, patest1.x
                Index Cond: (patest1.id = inttable.id)
          ->  Foreign Scan on public.patest2  (cost=100.00..470.00
rows=1 width=8)
                Output: patest2.id, patest2.x
                Remote SQL: SELECT id, x FROM public.mytable WHERE
(($1::integer = id))
  Planning time: 0.233 ms
(17 rows)

I revised the patch.  Patche attached, though I plan to update the
documentation further early next week.

Thanks,

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/03/20 21:59), Etsuro Fujita wrote:
> Here is a simple example for the case where the 
> use_remote_estimate option is true:

Sorry, I incorrectly wrote it.  The following example is for the case
where the option is *false*, as you see.

> # On mydatabase
> 
> mydatabase=# CREATE TABLE mytable (id INTEGER, x INTEGER);
> CREATE TABLE
> mydatabase=# INSERT INTO mytable SELECT x, x FROM generate_series(0, 
> 9999) x;
> INSERT 0 10000
> 
> # On postgres
> 
> postgres=# CREATE TABLE inttable (id INTEGER);
> CREATE TABLE
> postgres=# INSERT INTO inttable SELECT x FROM generate_series(0, 9999) x;
> INSERT 0 10000
> postgres=# ANALYZE inttable;
> ANALYZE
> 
> postgres=# CREATE TABLE patest0 (id INTEGER, x INTEGER);
> CREATE TABLE
> postgres=# CREATE TABLE patest1 () INHERITS (patest0);
> CREATE TABLE
> postgres=# INSERT INTO patest1 SELECT x, x FROM generate_series(0, 9999) x;
> INSERT 0 10000
> postgres=# CREATE INDEX patest1_id_idx ON patest1(id);
> CREATE INDEX
> postgres=# CREATE SERVER myserver FOREIGN DATA WRAPPER postgres_fdw 
> OPTIONS (host 'localhost', dbname 'mydatabase');
> CREATE SERVER
> postgres=# CREATE USER MAPPING FOR PUBLIC SERVER myserver OPTIONS (user 
> 'pgsql');
> CREATE USER MAPPING
> postgres=# CREATE FOREIGN TABLE patest2 () INHERITS (patest0) SERVER 
> myserver OPTIONS (table_name 'mytable');
> CREATE FOREIGN TABLE
> postgres=# ANALYZE patest0;
> ANALYZE
> postgres=# ANALYZE patest1;
> ANALYZE
> postgres=# ANALYZE patest2;
> ANALYZE
> postgres=# EXPLAIN VERBOSE SELECT * FROM patest0 join (SELECT id FROM 
> inttable LIMIT 1) ss ON patest0.id = ss.id;
>                                             QUERY PLAN
> ------------------------------------------------------------------------------------------------- 
> 
>   Nested Loop  (cost=0.00..478.36 rows=2 width=12)
>     Output: patest0.id, patest0.x, inttable.id
>     ->  Limit  (cost=0.00..0.01 rows=1 width=4)
>           Output: inttable.id
>           ->  Seq Scan on public.inttable  (cost=0.00..145.00 rows=10000 
> width=4)
>                 Output: inttable.id
>     ->  Append  (cost=0.00..478.31 rows=3 width=8)
>           ->  Seq Scan on public.patest0  (cost=0.00..0.00 rows=1 width=8)
>                 Output: patest0.id, patest0.x
>                 Filter: (inttable.id = patest0.id)
>           ->  Index Scan using patest1_id_idx on public.patest1 
> (cost=0.29..8.30 rows=1 width=8)
>                 Output: patest1.id, patest1.x
>                 Index Cond: (patest1.id = inttable.id)
>           ->  Foreign Scan on public.patest2  (cost=100.00..470.00 
> rows=1 width=8)
>                 Output: patest2.id, patest2.x
>                 Remote SQL: SELECT id, x FROM public.mytable WHERE 
> (($1::integer = id))
>   Planning time: 0.233 ms
> (17 rows)

Sorry for the delay.

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/03/20 21:59), Etsuro Fujita wrote:
> I revised the patch.  Patche attached, though I plan to update the
> documentation further early next week.

I updated the documentation further and revise the patch further.
Attached is an updated version of the patch.

Thanks,

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hello, I could see reparameterize for foreign path to work
effectively thanks to your advice. The key point was setting
use_remote_estimate to false and existence of another child to
get parameterized path in prior stages.

The overall patch was applied on HEAD and compiled cleanly except
for a warning.

> analyze.c: In function ‘acquire_inherited_sample_rows’:
> analyze.c:1461: warning: unused variable ‘saved_rel’


As for postgres-fdw, the point I felt uneasy in previous patch
was fixed already:) - which was coping with omission of
ReparameterizeForeignPath.

And for file-fdw, you made a change to re-create foreignscan node
instead of the previous copy-and-modify. Is the reason you did it
that you considered the cost of 're-checking whether to
selectively perform binary conversion' is low enough? Or other
reasons?

Finally, although I insist the necessity of the warning for child
foreign tables on alter table, if you belive it to be put off,
I'll compromise by putting a note to CF-app that last judgement
is left to committer.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
Hi Horiguchi-san,

(2014/03/26 17:14), Kyotaro HORIGUCHI wrote:
> The overall patch was applied on HEAD and compiled cleanly except
> for a warning.
> 
>> analyze.c: In function ‘acquire_inherited_sample_rows’:
>> analyze.c:1461: warning: unused variable ‘saved_rel’

I've fixed this in the latest version (v8) of the patch.

> And for file-fdw, you made a change to re-create foreignscan node
> instead of the previous copy-and-modify. Is the reason you did it
> that you considered the cost of 're-checking whether to
> selectively perform binary conversion' is low enough? Or other
> reasons?

The reason is that we get the result of the recheck from
path->fdw_private.  Sorry, I'd forgotten it.  So, I modified the code to
simply call create_foreignscan_path().

> Finally, although I insist the necessity of the warning for child
> foreign tables on alter table, if you belive it to be put off,
> I'll compromise by putting a note to CF-app that last judgement
> is left to committer.

OK  So, if there are no objections of other, I'll mark this patch as
"ready for committer" and do that.

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hello,

> >> analyze.c: In function ‘acquire_inherited_sample_rows’:
> >> analyze.c:1461: warning: unused variable ‘saved_rel’
> 
> I've fixed this in the latest version (v8) of the patch.

Mmm. sorry. I missed v8 patch. Then I had a look on that and have
a question.

You've added a check for relkind of baserel of the foreign path
to be reparameterized. If this should be an assertion - not a
conditional branch -, it would be better to put the assertion in
create_foreignscan_path instead of there.

=====
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -1723,6 +1723,7 @@ create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,                        List
*fdw_private){   ForeignPath *pathnode = makeNode(ForeignPath);
 
+    Assert(rel->rtekind == RTE_RELATION);    pathnode->path.pathtype = T_ForeignScan;    pathnode->path.parent = rel;
=====

> > And for file-fdw, you made a change to re-create foreignscan node
> > instead of the previous copy-and-modify. Is the reason you did it
> > that you considered the cost of 're-checking whether to
> > selectively perform binary conversion' is low enough? Or other
> > reasons?
> 
> The reason is that we get the result of the recheck from
> path->fdw_private.  Sorry, I'd forgotten it.  So, I modified the code to
> simply call create_foreignscan_path().

Anyway you new code seems closer to the basics and the gain by
the previous optimization don't seem to be significant..

> > Finally, although I insist the necessity of the warning for child
> > foreign tables on alter table, if you belive it to be put off,
> > I'll compromise by putting a note to CF-app that last judgement
> > is left to committer.
> 
> OK  So, if there are no objections of other, I'll mark this patch as
> "ready for committer" and do that.

Thank you.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
Hi Horiguchi-san,

(2014/03/27 17:09), Kyotaro HORIGUCHI wrote:
>>>> analyze.c: In function ‘acquire_inherited_sample_rows’:
>>>> analyze.c:1461: warning: unused variable ‘saved_rel’
>>
>> I've fixed this in the latest version (v8) of the patch.
> 
> Mmm. sorry. I missed v8 patch. Then I had a look on that and have
> a question.

Thank you for the review!

> You've added a check for relkind of baserel of the foreign path
> to be reparameterized. If this should be an assertion - not a
> conditional branch -, it would be better to put the assertion in
> create_foreignscan_path instead of there.
> 
> =====
> --- a/src/backend/optimizer/util/pathnode.c
> +++ b/src/backend/optimizer/util/pathnode.c
> @@ -1723,6 +1723,7 @@ create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
>                           List *fdw_private)
>   {
>       ForeignPath *pathnode = makeNode(ForeignPath);
> +    Assert(rel->rtekind == RTE_RELATION);
>   
>       pathnode->path.pathtype = T_ForeignScan;
>       pathnode->path.parent = rel;
> =====

Maybe I'm missing the point, but I don't think it'd be better to put the
assertion in create_foreignscan_path().  And I think it'd be the caller'
responsiblity to ensure that equality, as any other pathnode creation
routine for a baserel in pathnode.c assumes that equality.

>>> And for file-fdw, you made a change to re-create foreignscan node
>>> instead of the previous copy-and-modify. Is the reason you did it
>>> that you considered the cost of 're-checking whether to
>>> selectively perform binary conversion' is low enough? Or other
>>> reasons?
>>
>> The reason is that we get the result of the recheck from
>> path->fdw_private.  Sorry, I'd forgotten it.  So, I modified the code to
>> simply call create_foreignscan_path().
> 
> Anyway you new code seems closer to the basics and the gain by
> the previous optimization don't seem to be significant..

Yeah, that's true.  I have to admit that the previous optimization is
nonsense.

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/03/27 10:49), Etsuro Fujita wrote:
> (2014/03/26 17:14), Kyotaro HORIGUCHI wrote:
>> Finally, although I insist the necessity of the warning for child
>> foreign tables on alter table, if you belive it to be put off,
>> I'll compromise by putting a note to CF-app that last judgement
>> is left to committer.
>
> OK  So, if there are no objections of other, I'll mark this patch as
> "ready for committer" and do that.

I'll do it right now, since there seems to be no objections of others.

I've revised the patch a bit further, mainly the documentation.  Patch
attached.

Note: as mentioned above, Horiguchi-san insisted that warning
functionality for recursive ALTER TABLE operations for inheritance trees
that contain one or more children that are foreign.  However, that
functionality hasn't been included in the patch, since ISTM that that
functionality should be implemented as a separate patch [1].  We leave
this for commiters to decide.

[1] http://www.postgresql.org/message-id/5321ABD2.6000104@lab.ntt.co.jp

Thanks,

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hi,

> >       ForeignPath *pathnode = makeNode(ForeignPath);
> > +    Assert(rel->rtekind == RTE_RELATION);
> >   
> >       pathnode->path.pathtype = T_ForeignScan;
..
> Maybe I'm missing the point, but I don't think it'd be better to put the
> assertion in create_foreignscan_path().  And I think it'd be the caller'
> responsiblity to ensure that equality, as any other pathnode creation
> routine for a baserel in pathnode.c assumes that equality.

Hmm. The assertion (not shown above but you put in
parameterize_path:) seems to say that 'base relation for foreign
paths must be a RTE_RELATION' isn't right? But I don't see
anything putting such a restriction in reparameterize_path
itself. Could you tell me where such a restriction comes from? Or
who needs such a restriction? I think any assertions shouldn't be
anywhere other than where just before needed.

Thoughts?

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/03/28 13:28), Kyotaro HORIGUCHI wrote:
> Hmm. The assertion (not shown above but you put in
> parameterize_path:) seems to say that 'base relation for foreign
> paths must be a RTE_RELATION' isn't right?

I think that's right.  Please see allpaths.c, especially set_rel_pathlist().

For your information, the same assertion can be found in 
create_foreignscan_plan().

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
I've found some bugs.  Attached is an updated version (v10).

Changes:

* When CREATE FOREIGN TABLE INHERITS, don't allow a foreign table to
inherit from parent tables that have OID system columns.

* When CREATE FOREIGN TABLE INHERITS, reset the storage parameter for
inherited columns that have non-defaut values.

* Don't allow CHECK (expr) *NO INHERIT* on foreign tables, since those
tables cannot have child tables.

* Fix and update the documentation.

Thanks,

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
Attached is v11.

Changes:

* Rebased to head
* Improve an error message added to tablecmd.c to match it to existing
ones there
* Improve the documentaion a bit

Thanks,

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/04/02 21:25), Etsuro Fujita wrote:
> Attached is v11.
> 
> Changes:
> 
> * Rebased to head
> * Improve an error message added to tablecmd.c to match it to existing
> ones there
> * Improve the documentaion a bit

I moved this to 2014-06.

Since I've merged with the initial patch by Hanada-san (1) a feature to
allow the inherited stats to be computed by the ANALYZE command and (2)
a new FDW routine for path reparameterization, I put my name on the author.

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hello,

Before continueing discussion, I tried this patch on the current
head.

This patch applies cleanly except one hunk because of a
modification just AFTER it, which did no harm. Finally all
regression tests suceeded.

Attached is the rebased patch of v11 up to the current master.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center
diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c
index 5a4d5aa..5a9aec0 100644
--- a/contrib/file_fdw/file_fdw.c
+++ b/contrib/file_fdw/file_fdw.c
@@ -115,6 +115,11 @@ static void fileGetForeignRelSize(PlannerInfo *root,static void fileGetForeignPaths(PlannerInfo
*root,                   RelOptInfo *baserel,                    Oid foreigntableid);
 
+static ForeignPath *fileReparameterizeForeignPath(PlannerInfo *root,
+                                                  RelOptInfo *baserel,
+                                                  Oid foreigntableid,
+                                                  ForeignPath *path,
+                                                  Relids required_outer);static ForeignScan
*fileGetForeignPlan(PlannerInfo*root,                   RelOptInfo *baserel,                   Oid foreigntableid,
 
@@ -143,7 +148,7 @@ static bool check_selective_binary_conversion(RelOptInfo *baserel,static void
estimate_size(PlannerInfo*root, RelOptInfo *baserel,              FileFdwPlanState *fdw_private);static void
estimate_costs(PlannerInfo*root, RelOptInfo *baserel,
 
-               FileFdwPlanState *fdw_private,
+               FileFdwPlanState *fdw_private, List *join_conds,               Cost *startup_cost, Cost
*total_cost);staticint file_acquire_sample_rows(Relation onerel, int elevel,                         HeapTuple *rows,
inttargrows,
 
@@ -161,6 +166,7 @@ file_fdw_handler(PG_FUNCTION_ARGS)    fdwroutine->GetForeignRelSize = fileGetForeignRelSize;
fdwroutine->GetForeignPaths= fileGetForeignPaths;
 
+    fdwroutine->ReparameterizeForeignPath = fileReparameterizeForeignPath;    fdwroutine->GetForeignPlan =
fileGetForeignPlan;   fdwroutine->ExplainForeignScan = fileExplainForeignScan;    fdwroutine->BeginForeignScan =
fileBeginForeignScan;
@@ -509,7 +515,8 @@ fileGetForeignPaths(PlannerInfo *root,                                          (Node *) columns));
  /* Estimate costs */
 
-    estimate_costs(root, baserel, fdw_private,
+    estimate_costs(root, baserel,
+                   fdw_private, NIL,                   &startup_cost, &total_cost);    /*
@@ -534,6 +541,41 @@ fileGetForeignPaths(PlannerInfo *root,}/*
+ * fileReparameterizeForeignPath
+ *        Attempt to modify a given path to have greater parameterization
+ */
+static ForeignPath *
+fileReparameterizeForeignPath(PlannerInfo *root,
+                              RelOptInfo *baserel,
+                              Oid foreigntableid,
+                              ForeignPath *path,
+                              Relids required_outer)
+{
+    ParamPathInfo *param_info;
+    FileFdwPlanState *fdw_private = (FileFdwPlanState *) baserel->fdw_private;
+    Cost        startup_cost;
+    Cost        total_cost;
+
+    /* Get the ParamPathInfo */
+    param_info = get_baserel_parampathinfo(root, baserel, required_outer);
+
+    /* Redo the cost estimates */
+    estimate_costs(root, baserel,
+                   fdw_private,
+                   param_info->ppi_clauses,
+                   &startup_cost, &total_cost);
+
+    /* Make and return the new path */
+    return create_foreignscan_path(root, baserel,
+                                   param_info->ppi_rows,
+                                   startup_cost,
+                                   total_cost,
+                                   NIL,        /* no pathkeys */
+                                   required_outer,
+                                   path->fdw_private);
+}
+
+/* * fileGetForeignPlan *        Create a ForeignScan plan node for scanning the foreign table */
@@ -962,12 +1004,13 @@ estimate_size(PlannerInfo *root, RelOptInfo *baserel, */static voidestimate_costs(PlannerInfo
*root,RelOptInfo *baserel,
 
-               FileFdwPlanState *fdw_private,
+               FileFdwPlanState *fdw_private, List *join_conds,               Cost *startup_cost, Cost *total_cost){
BlockNumber pages = fdw_private->pages;    double        ntuples = fdw_private->ntuples;    Cost        run_cost = 0;
 
+    QualCost    join_cost;    Cost        cpu_per_tuple;    /*
@@ -978,8 +1021,11 @@ estimate_costs(PlannerInfo *root, RelOptInfo *baserel,     */    run_cost += seq_page_cost *
pages;
-    *startup_cost = baserel->baserestrictcost.startup;
-    cpu_per_tuple = cpu_tuple_cost * 10 + baserel->baserestrictcost.per_tuple;
+    cost_qual_eval(&join_cost, join_conds, root);
+    *startup_cost =
+        (baserel->baserestrictcost.startup + join_cost.startup);
+    cpu_per_tuple = cpu_tuple_cost * 10 +
+        (baserel->baserestrictcost.per_tuple + join_cost.per_tuple);    run_cost += cpu_per_tuple * ntuples;
*total_cost= *startup_cost + run_cost;}
 
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 7dd43a9..3f7f7c2 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -239,6 +239,11 @@ static void postgresGetForeignRelSize(PlannerInfo *root,static void
postgresGetForeignPaths(PlannerInfo*root,                        RelOptInfo *baserel,                        Oid
foreigntableid);
+static ForeignPath *postgresReparameterizeForeignPath(PlannerInfo *root,
+                                                      RelOptInfo *baserel,
+                                                      Oid foreigntableid,
+                                                      ForeignPath *path,
+                                                      Relids required_outer);static ForeignScan
*postgresGetForeignPlan(PlannerInfo*root,                       RelOptInfo *baserel,                       Oid
foreigntableid,
@@ -340,6 +345,7 @@ postgres_fdw_handler(PG_FUNCTION_ARGS)    /* Functions for scanning foreign tables */
routine->GetForeignRelSize= postgresGetForeignRelSize;    routine->GetForeignPaths = postgresGetForeignPaths;
 
+    routine->ReparameterizeForeignPath = postgresReparameterizeForeignPath;    routine->GetForeignPlan =
postgresGetForeignPlan;   routine->BeginForeignScan = postgresBeginForeignScan;    routine->IterateForeignScan =
postgresIterateForeignScan;
@@ -727,6 +733,48 @@ postgresGetForeignPaths(PlannerInfo *root,}/*
+ * postgresReparameterizeForeignPath
+ *        Attempt to modify a given path to have greater parameterization
+ */
+static ForeignPath *
+postgresReparameterizeForeignPath(PlannerInfo *root,
+                                  RelOptInfo *baserel,
+                                  Oid foreigntableid,
+                                  ForeignPath *path,
+                                  Relids required_outer)
+{
+    ParamPathInfo *param_info;
+    double        rows;
+    int            width;
+    Cost        startup_cost;
+    Cost        total_cost;
+
+    /* Get the ParamPathInfo */
+    param_info = get_baserel_parampathinfo(root, baserel, required_outer);
+
+    /* Redo the cost estimates */
+    estimate_path_cost_size(root, baserel,
+                            param_info->ppi_clauses,
+                            &rows, &width,
+                            &startup_cost, &total_cost);
+
+    /*
+     * ppi_rows currently won't get looked at by anything, but still we
+     * may as well ensure that it matches our idea of the rowcount.
+     */
+    param_info->ppi_rows = rows;
+
+    /* Make and return the new path */
+    return create_foreignscan_path(root, baserel,
+                                   rows,
+                                   startup_cost,
+                                   total_cost,
+                                   NIL,        /* no pathkeys */
+                                   required_outer,
+                                   NIL);    /* no fdw_private list */
+}
+
+/* * postgresGetForeignPlan *        Create ForeignScan plan node which implements selected best path */
@@ -1773,11 +1821,8 @@ estimate_path_cost_size(PlannerInfo *root,    }    else    {
-        /*
-         * We don't support join conditions in this mode (hence, no
-         * parameterized paths can be made).
-         */
-        Assert(join_conds == NIL);
+        Selectivity join_sel;
+        QualCost    join_cost;        /* Use rows/width estimates made by set_baserel_size_estimates. */        rows =
baserel->rows;
@@ -1790,17 +1835,29 @@ estimate_path_cost_size(PlannerInfo *root,        retrieved_rows = clamp_row_est(rows /
fpinfo->local_conds_sel);       retrieved_rows = Min(retrieved_rows, baserel->tuples);
 
+        /* Factor in the selectivity of the join_conds */
+        join_sel = clauselist_selectivity(root,
+                                          join_conds,
+                                          baserel->relid,
+                                          JOIN_INNER,
+                                          NULL);
+
+        rows = clamp_row_est(rows * join_sel);
+        /*         * Cost as though this were a seqscan, which is pessimistic.  We
-         * effectively imagine the local_conds are being evaluated remotely,
-         * too.
+         * effectively imagine the local_conds and join_conds are being
+         * evaluated remotely, too.         */        startup_cost = 0;        run_cost = 0;        run_cost +=
seq_page_cost* baserel->pages;
 
-        startup_cost += baserel->baserestrictcost.startup;
-        cpu_per_tuple = cpu_tuple_cost + baserel->baserestrictcost.per_tuple;
+        cost_qual_eval(&join_cost, join_conds, root);
+        startup_cost +=
+            (baserel->baserestrictcost.startup + join_cost.startup);
+        cpu_per_tuple = cpu_tuple_cost +
+            (baserel->baserestrictcost.per_tuple + join_cost.per_tuple);        run_cost += cpu_per_tuple *
baserel->tuples;       total_cost = startup_cost + run_cost;
 
diff --git a/doc/src/sgml/fdwhandler.sgml b/doc/src/sgml/fdwhandler.sgml
index 6b5c8b7..ec1492a 100644
--- a/doc/src/sgml/fdwhandler.sgml
+++ b/doc/src/sgml/fdwhandler.sgml
@@ -162,6 +162,37 @@ GetForeignPaths (PlannerInfo *root,    <para><programlisting>
+ForeignPath *
+ReparameterizeForeignPath (PlannerInfo *root,
+                           RelOptInfo *baserel,
+                           Oid foreigntableid,
+                           ForeignPath *path,
+                           Relids required_outer);
+</programlisting>
+
+     Create an access path for a scan on a foreign table using join
+     clauses, which is called a <quote>parameterized path</>.
+     This is called during query planning.
+     The parameters are as for <function>GetForeignRelSize</>, plus
+     the <structname>ForeignPath</> (previously produced by
+     <function>GetForeignPaths</>), and the IDs of all other tables
+     that provide the join clauses.
+    </para>
+
+    <para>
+     This function must generate a parameterized path for a scan on the
+     foreign table.  The parameterized path must contain cost estimates,
+     and can contain any FDW-private information that is needed to identify
+     the specific scan method intended.  Unlike the other scan-related
+     functions, this function is optional.
+    </para>
+
+    <para>
+     See <xref linkend="fdw-planning"> for additional information.
+    </para>
+
+    <para>
+<programlisting>ForeignScan *GetForeignPlan (PlannerInfo *root,                RelOptInfo *baserel,
@@ -808,10 +839,10 @@ GetForeignServerByName(const char *name, bool missing_ok);    <para>     The FDW callback
functions<function>GetForeignRelSize</>,
 
-     <function>GetForeignPaths</>, <function>GetForeignPlan</>, and
-     <function>PlanForeignModify</> must fit into the workings of the
-     <productname>PostgreSQL</> planner.  Here are some notes about what
-     they must do.
+     <function>GetForeignPaths</>, <function>ReparameterizeForeignPath</>,
+     <function>GetForeignPlan</>, and <function>PlanForeignModify</> must fit
+     into the workings of the <productname>PostgreSQL</> planner.  Here are
+     some notes about what they must do.    </para>    <para>
@@ -841,14 +872,17 @@ GetForeignServerByName(const char *name, bool missing_ok);     to initialize it to NULL when the
<literal>baserel</>node is created.     It is useful for passing information forward from
<function>GetForeignRelSize</>to <function>GetForeignPaths</> and/or
 
-     <function>GetForeignPaths</> to <function>GetForeignPlan</>, thereby
-     avoiding recalculation.
+     <function>GetForeignPaths</> to <function>ReparameterizeForeignPath</>
+     and/or
+     <function>GetForeignPaths</> or <function>ReparameterizeForeignPath</>
+     to <function>GetForeignPlan</>, thereby avoiding recalculation.    </para>    <para>
-     <function>GetForeignPaths</> can identify the meaning of different
-     access paths by storing private information in the
-     <structfield>fdw_private</> field of <structname>ForeignPath</> nodes.
+     <function>GetForeignPaths</> or <function>ReparameterizeForeignPath</>
+     can identify the meaning of different access paths by storing private
+     information in the <structfield>fdw_private</> field of
+     <structname>ForeignPath</> nodes.     <structfield>fdw_private</> is declared as a <type>List</> pointer, but
couldactually contain anything since the core planner does not touch     it.  However, best practice is to use a
representationthat's dumpable
 
@@ -891,10 +925,11 @@ GetForeignServerByName(const char *name, bool missing_ok);     <replaceable>sub_expression</>,
whichit determines can be executed on     the remote server given the locally-evaluated value of the
<replaceable>sub_expression</>. The actual identification of such a
 
-     clause should happen during <function>GetForeignPaths</>, since it would
-     affect the cost estimate for the path.  The path's
-     <structfield>fdw_private</> field would probably include a pointer to
-     the identified clause's <structname>RestrictInfo</> node.  Then
+     clause should happen during <function>GetForeignPaths</> or
+     <function>ReparameterizeForeignPath</>, since it would affect the cost
+     estimate for the path.  The path's <structfield>fdw_private</> field
+     would probably include a pointer to the identified clause's
+     <structname>RestrictInfo</> node.  Then     <function>GetForeignPlan</> would remove that clause from
<literal>scan_clauses</>,    but add the <replaceable>sub_expression</> to <structfield>fdw_exprs</>     to ensure that
itgets massaged into executable form.  It would probably
 
diff --git a/doc/src/sgml/ref/alter_foreign_table.sgml b/doc/src/sgml/ref/alter_foreign_table.sgml
index 4d8cfc5..10461d5 100644
--- a/doc/src/sgml/ref/alter_foreign_table.sgml
+++ b/doc/src/sgml/ref/alter_foreign_table.sgml
@@ -42,6 +42,10 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab    ALTER [ COLUMN ]
<replaceableclass="PARAMETER">column_name</replaceable> SET ( <replaceable
class="PARAMETER">attribute_option</replaceable>= <replaceable class="PARAMETER">value</replaceable> [, ... ] )
ALTER[ COLUMN ] <replaceable class="PARAMETER">column_name</replaceable> RESET ( <replaceable
class="PARAMETER">attribute_option</replaceable>[, ... ] )    ALTER [ COLUMN ] <replaceable
class="PARAMETER">column_name</replaceable>OPTIONS ( [ ADD | SET | DROP ] <replaceable
class="PARAMETER">option</replaceable>['<replaceable class="PARAMETER">value</replaceable>'] [, ... ])
 
+    ADD <replaceable class="PARAMETER">table_constraint</replaceable>
+    DROP CONSTRAINT [ IF EXISTS ]  <replaceable class="PARAMETER">constraint_name</replaceable> [ RESTRICT | CASCADE
]
+    INHERIT <replaceable class="PARAMETER">parent_table</replaceable>
+    NO INHERIT <replaceable class="PARAMETER">parent_table</replaceable>    OWNER TO <replaceable
class="PARAMETER">new_owner</replaceable>   OPTIONS ( [ ADD | SET | DROP ] <replaceable
class="PARAMETER">option</replaceable>['<replaceable class="PARAMETER">value</replaceable>'] [, ... ])</synopsis>
 
@@ -149,6 +153,50 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab   </varlistentry>
 <varlistentry>
 
+    <term><literal>ADD <replaceable class="PARAMETER">table_constraint</replaceable></literal></term>
+    <listitem>
+     <para>
+      This form adds a new constraint to a table using the same syntax as
+      <xref linkend="SQL-CREATEFOREIGNTABLE">.
+      Unlike the case when adding a constraint to a regular table, nothing happens
+      to the underlying storage: this action simply declares that
+      some new constraint holds for all rows in the foreign table.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term><literal>DROP CONSTRAINT [ IF EXISTS ]</literal></term>
+    <listitem>
+     <para>
+      This form drops the specified constraint on a table.
+      If <literal>IF EXISTS</literal> is specified and the constraint
+      does not exist, no error is thrown. In this case a notice is issued instead.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term><literal>INHERIT <replaceable class="PARAMETER">parent_table</replaceable></literal></term>
+    <listitem>
+     <para>
+      This form adds the target foreign table as a new child of the specified
+      parent table.  The parent table must be an ordinary table.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term><literal>NO INHERIT <replaceable class="PARAMETER">parent_table</replaceable></literal></term>
+    <listitem>
+     <para>
+      This form removes the target foreign table from the list of children of
+      the specified parent table.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>    <term><literal>OWNER</literal></term>    <listitem>     <para>
@@ -270,6 +318,24 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab
</varlistentry>    <varlistentry>
 
+      <term><replaceable class="PARAMETER">table_constraint</replaceable></term>
+      <listitem>
+       <para>
+        New table constraint for the table.
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
+      <term><replaceable class="PARAMETER">constraint_name</replaceable></term>
+      <listitem>
+       <para>
+        Name of an existing constraint to drop.
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>      <term><literal>CASCADE</literal></term>      <listitem>       <para>
@@ -290,6 +356,16 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab
</varlistentry>    <varlistentry>
 
+      <term><replaceable class="PARAMETER">parent_table</replaceable></term>
+      <listitem>
+       <para>
+        A parent table to associate or de-associate with this foreign table.
+        The parent table must be an ordinary table.
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>      <term><replaceable class="PARAMETER">new_owner</replaceable></term>      <listitem>
<para>
@@ -319,10 +395,10 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab   <para>
Consistencywith the foreign server is not checked when a column is added    or removed with <literal>ADD
COLUMN</literal>or
 
-    <literal>DROP COLUMN</literal>, a <literal>NOT NULL</> constraint is
-    added, or a column type is changed with <literal>SET DATA TYPE</>.  It is
-    the user's responsibility to ensure that the table definition matches the
-    remote side.
+    <literal>DROP COLUMN</literal>, a <literal>NOT NULL</> constraint or
+    <literal>CHECK</> constraint is added, or a column type is changed with
+    <literal>SET DATA TYPE</>.  It is the user's responsibility to ensure that
+    the table definition matches the remote side.   </para>   <para>
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 69a1e14..eb1352e 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -967,6 +967,13 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceable>   </para>   <para>
+    If a table has any descendant tables that are foreign, a recursive
+    <literal>SET STORAGE</literal> operation will be rejected since it
+    is not permitted to add an <literal>oid</literal> system column to
+    foreign tables.
+   </para>
+
+   <para>    The <literal>TRIGGER</>, <literal>CLUSTER</>, <literal>OWNER</>,    and <literal>TABLESPACE</> actions
neverrecurse to descendant tables;    that is, they always act as though <literal>ONLY</> were specified.
 
@@ -975,6 +982,19 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceable>   </para>   <para>
+    When adding a <literal>CHECK</> constraint with the <literal>NOT VALID
+    </literal> option recursively, an inherited constraint on a descendant
+    table that is foreign will be marked valid without checking
+    consistency with the foreign server.
+   </para>
+
+   <para>
+    A recursive <literal>SET STORAGE</literal> operation will make the
+    storage mode for a descendant table's column unchanged if the table is
+    foreign.
+   </para>
+
+   <para>    Changing any part of a system catalog table is not permitted.   </para>
diff --git a/doc/src/sgml/ref/analyze.sgml b/doc/src/sgml/ref/analyze.sgml
index 08d316a..ec257c5 100644
--- a/doc/src/sgml/ref/analyze.sgml
+++ b/doc/src/sgml/ref/analyze.sgml
@@ -200,6 +200,13 @@ ANALYZE [ VERBOSE ] [ <replaceable class="PARAMETER">table_name</replaceable> [  </para>  <para>
+    The inheritance statistics for a parent table that contains one or more
+    children that are foreign tables are collected only when explicitly
+    selected.  If the foreign table's wrapper does not support
+    <command>ANALYZE</command>, the command prints a warning and does nothing.
+  </para>
+
+  <para>    If the table being analyzed is completely empty, <command>ANALYZE</command>    will not record new
statisticsfor that table.  Any existing statistics    will be retained.
 
diff --git a/doc/src/sgml/ref/create_foreign_table.sgml b/doc/src/sgml/ref/create_foreign_table.sgml
index 4a8cf38..1755b38 100644
--- a/doc/src/sgml/ref/create_foreign_table.sgml
+++ b/doc/src/sgml/ref/create_foreign_table.sgml
@@ -19,9 +19,11 @@ <refsynopsisdiv><synopsis>CREATE FOREIGN TABLE [ IF NOT EXISTS ] <replaceable
class="PARAMETER">table_name</replaceable>( [
 
-    <replaceable class="PARAMETER">column_name</replaceable> <replaceable class="PARAMETER">data_type</replaceable> [
OPTIONS( <replaceable class="PARAMETER">option</replaceable> '<replaceable class="PARAMETER">value</replaceable>' [,
...] ) ] [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable
class="PARAMETER">column_constraint</replaceable>[ ... ] ]
 
+  { <replaceable class="PARAMETER">column_name</replaceable> <replaceable class="PARAMETER">data_type</replaceable> [
OPTIONS( <replaceable class="PARAMETER">option</replaceable> '<replaceable class="PARAMETER">value</replaceable>' [,
...] ) ] [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable
class="PARAMETER">column_constraint</replaceable>[ ... ] ]
 
+    | <replaceable>table_constraint</replaceable> }    [, ... ]] )
+[ INHERITS ( <replaceable>parent_table</replaceable> [, ... ] ) ]  SERVER <replaceable
class="parameter">server_name</replaceable>[OPTIONS ( <replaceable class="PARAMETER">option</replaceable> '<replaceable
class="PARAMETER">value</replaceable>'[, ... ] ) ]
 
@@ -30,7 +32,13 @@ CREATE FOREIGN TABLE [ IF NOT EXISTS ] <replaceable class="PARAMETER">table_name[ CONSTRAINT
<replaceableclass="PARAMETER">constraint_name</replaceable> ]{ NOT NULL |  NULL |
 
+  CHECK ( <replaceable class="PARAMETER">expression</replaceable> ) |  DEFAULT <replaceable>default_expr</replaceable>
}
+
+<phrase>and <replaceable class="PARAMETER">table_constraint</replaceable> is:</phrase>
+
+[ CONSTRAINT <replaceable class="PARAMETER">constraint_name</replaceable> ]
+{ CHECK ( <replaceable class="PARAMETER">expression</replaceable> ) }</synopsis> </refsynopsisdiv>
@@ -138,6 +146,27 @@ CREATE FOREIGN TABLE [ IF NOT EXISTS ] <replaceable class="PARAMETER">table_name   </varlistentry>
 <varlistentry>
 
+    <term><literal>CHECK ( <replaceable class="PARAMETER">expression</replaceable> )</literal></term>
+    <listitem>
+     <para>
+      The <literal>CHECK</> clause specifies an expression producing a
+      Boolean result which each row must satisfy.
+      Expressions evaluating to TRUE or UNKNOWN succeed.
+      A check constraint specified as a column constraint should
+      reference that column's value only, while an expression
+      appearing in a table constraint can reference multiple columns.
+     </para>
+
+     <para>
+      Currently, <literal>CHECK</literal> expressions cannot contain
+      subqueries nor refer to variables other than columns of the
+      current row.  The system column <literal>tableoid</literal>
+      may be referenced, but not any other system column.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>    <term><literal>DEFAULT    <replaceable>default_expr</replaceable></literal></term>    <listitem>
@@ -159,6 +188,18 @@ CREATE FOREIGN TABLE [ IF NOT EXISTS ] <replaceable class="PARAMETER">table_name   </varlistentry>
 <varlistentry>
 
+    <term><replaceable class="PARAMETER">parent_table</replaceable></term>
+    <listitem>
+     <para>
+      The name of an existing table from which the new foreign table
+      automatically inherits all columns.  The specified parent table
+      must be an ordinary table.  See <xref linkend="ddl-inherit"> for the
+      details of table inheritance.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>    <term><replaceable class="PARAMETER">server_name</replaceable></term>    <listitem>     <para>
@@ -187,6 +228,24 @@ CREATE FOREIGN TABLE [ IF NOT EXISTS ] <replaceable class="PARAMETER">table_name </refsect1>
+ <refsect1>
+  <title>Notes</title>
+
+   <para>
+    Constraints on foreign tables are not enforced on insert or update.
+    Those definitions simply declare the constraints hold for all rows
+    in the foreign tables.  It is the user's responsibility to ensure
+    that those definitions match the remote side.  Such constraints are
+    used for some kind of query optimization such as constraint exclusion
+    for partitioned tables (see <xref linkend="ddl-partitioning">).
+   </para>
+
+   <para>
+    Since it is not permitted to add an <literal>oid</> system column to
+    foreign tables, the command will be rejected if any parent tables
+    have <literal>oid</> system columns.
+   </para>
+ </refsect1> <refsect1 id="SQL-CREATEFOREIGNTABLE-examples">  <title>Examples</title>
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 33eef9f..21d32c5 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -2217,6 +2217,12 @@ AddRelationNewConstraints(Relation rel,        if (cdef->contype != CONSTR_CHECK)
continue;
+        if (cdef->is_no_inherit &&
+            rel->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
+            ereport(ERROR,
+                    (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+                     errmsg("CHECK constraints on foreign tables cannot be marked NO INHERIT")));
+        if (cdef->raw_expr != NULL)        {            Assert(cdef->cooked_expr == NULL);
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index c09ca7e..f67273c 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -82,6 +82,7 @@ int            default_statistics_target = 100;/* A few variables that don't seem worth passing
aroundas parameters */static MemoryContext anl_context = NULL;
 
+static VacuumMode vac_mode;static BufferAccessStrategy vac_strategy;
@@ -115,7 +116,10 @@ static Datum ind_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull); *    analyze_rel() --
analyzeone relation */void
 
-analyze_rel(Oid relid, VacuumStmt *vacstmt, BufferAccessStrategy bstrategy)
+analyze_rel(Oid relid,
+            VacuumStmt *vacstmt,
+            VacuumMode vacmode,
+            BufferAccessStrategy bstrategy){    Relation    onerel;    int            elevel;
@@ -129,6 +133,7 @@ analyze_rel(Oid relid, VacuumStmt *vacstmt, BufferAccessStrategy bstrategy)        elevel = DEBUG2;
  /* Set up static variables */
 
+    vac_mode = vacmode;    vac_strategy = bstrategy;    /*
@@ -1452,6 +1457,7 @@ acquire_inherited_sample_rows(Relation onerel, int elevel,{    List       *tableOIDs;    Relation
 *rels;
 
+    AcquireSampleRowsFunc *acquirefunc;    double       *relblocks;    double        totalblocks;    int
numrows,
@@ -1486,6 +1492,8 @@ acquire_inherited_sample_rows(Relation onerel, int elevel,     * BlockNumber, so we use double
arithmetic.    */    rels = (Relation *) palloc(list_length(tableOIDs) * sizeof(Relation));
 
+    acquirefunc = (AcquireSampleRowsFunc *) palloc(list_length(tableOIDs)
+                                            * sizeof(AcquireSampleRowsFunc));    relblocks = (double *)
palloc(list_length(tableOIDs)* sizeof(double));    totalblocks = 0;    nrels = 0;
 
@@ -1507,7 +1515,40 @@ acquire_inherited_sample_rows(Relation onerel, int elevel,        }        rels[nrels] =
childrel;
-        relblocks[nrels] = (double) RelationGetNumberOfBlocks(childrel);
+
+        if (childrel->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
+        {
+            FdwRoutine *fdwroutine;
+            BlockNumber relpages = 0;
+            bool        ok = false;
+
+            /* Ignore unless analyzing a specific inheritance tree */
+            if (vac_mode != VAC_MODE_SINGLE)
+                return 0;
+
+            /* Check whether the FDW supports analysis */
+            fdwroutine = GetFdwRoutineForRelation(childrel, false);
+            if (fdwroutine->AnalyzeForeignTable != NULL)
+                ok = fdwroutine->AnalyzeForeignTable(childrel,
+                                                     &acquirefunc[nrels],
+                                                     &relpages);
+            if (!ok)
+            {
+                /* Give up if the FDW doesn't support analysis */
+                ereport(WARNING,
+                        (errmsg("skipping analyze of inheritance tree \"%s\" --- cannot analyze foreign table
\"%s\"",
+                                RelationGetRelationName(onerel),
+                                RelationGetRelationName(childrel))));
+                return 0;
+            }
+            relblocks[nrels] = (double) relpages;
+        }
+        else
+        {
+            acquirefunc[nrels] = acquire_sample_rows;
+            relblocks[nrels] = (double) RelationGetNumberOfBlocks(childrel);
+        }
+        totalblocks += relblocks[nrels];        nrels++;    }
@@ -1525,6 +1566,7 @@ acquire_inherited_sample_rows(Relation onerel, int elevel,    {        Relation    childrel =
rels[i];       double        childblocks = relblocks[i];
 
+        AcquireSampleRowsFunc childacquirefunc = acquirefunc[i];        if (childblocks > 0)        {
@@ -1540,12 +1582,12 @@ acquire_inherited_sample_rows(Relation onerel, int elevel,                            tdrows;
            /* Fetch a random sample of the child's rows */
 
-                childrows = acquire_sample_rows(childrel,
-                                                elevel,
-                                                rows + numrows,
-                                                childtargrows,
-                                                &trows,
-                                                &tdrows);
+                childrows = childacquirefunc(childrel,
+                                             elevel,
+                                             rows + numrows,
+                                             childtargrows,
+                                             &trows,
+                                             &tdrows);                /* We may need to convert from child's rowtype
toparent's */                if (childrows > 0 &&
 
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 341262b..9e09906 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -310,7 +310,8 @@ static AlteredTableInfo *ATGetQueueEntry(List **wqueue, Relation rel);static void
ATSimplePermissions(Relationrel, int allowed_targets);static void ATWrongRelkindError(Relation rel, int
allowed_targets);staticvoid ATSimpleRecursion(List **wqueue, Relation rel,
 
-                  AlterTableCmd *cmd, bool recurse, LOCKMODE lockmode);
+                  AlterTableCmd *cmd, bool recurse,
+                  bool include_foreign, LOCKMODE lockmode);static void ATTypedTableRecursion(List **wqueue, Relation
rel,AlterTableCmd *cmd,                      LOCKMODE lockmode);static List *find_typed_table_dependencies(Oid typeOid,
constchar *typeName,
 
@@ -466,10 +467,6 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId)        ereport(ERROR,
(errcode(ERRCODE_INVALID_TABLE_DEFINITION),                errmsg("ON COMMIT can only be used on temporary tables")));
 
-    if (stmt->constraints != NIL && relkind == RELKIND_FOREIGN_TABLE)
-        ereport(ERROR,
-                (errcode(ERRCODE_WRONG_OBJECT_TYPE),
-                 errmsg("constraints are not supported on foreign tables")));    /*     * Look up the namespace in
whichwe are supposed to create the relation,
 
@@ -556,6 +553,30 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId)
stmt->relation->relpersistence,                            &inheritOids, &old_constraints, &parentOidCount);
 
+    if (relkind == RELKIND_FOREIGN_TABLE)
+    {
+        /*
+         * Don't allow a foreign table to inherit from parents that have OID
+         * system columns.
+         */
+        if (parentOidCount > 0)
+            ereport(ERROR,
+                    (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+                     errmsg("cannot inherit from relation with OIDs")));
+
+        /*
+         * Reset the storage parameter for inherited attributes that have
+         * non-default values.
+         */
+        foreach(listptr, schema)
+        {
+            ColumnDef  *colDef = lfirst(listptr);
+        
+            if (colDef->storage != 0)
+                colDef->storage = 0;
+        }
+    }
+    /*     * Create a tuple descriptor from the relation schema.  Note that this     * deals with column names, types,
andNOT NULL constraints, but not
 
@@ -3065,24 +3086,28 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,             * rules.             */
         ATSimplePermissions(rel, ATT_TABLE | ATT_VIEW | ATT_FOREIGN_TABLE); 
-            ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+            /* Recurse to child tables that are foreign, too */
+            ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode);            /* No command-specific prep
needed*/            pass = cmd->def ? AT_PASS_ADD_CONSTR : AT_PASS_DROP;            break;        case AT_DropNotNull:
 /* ALTER COLUMN DROP NOT NULL */            ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
 
-            ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+            /* Recurse to child tables that are foreign, too */
+            ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode);            /* No command-specific prep
needed*/            pass = AT_PASS_DROP;            break;        case AT_SetNotNull:        /* ALTER COLUMN SET NOT
NULL*/            ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
 
-            ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+            /* Recurse to child tables that are foreign, too */
+            ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode);            /* No command-specific prep
needed*/            pass = AT_PASS_ADD_CONSTR;            break;        case AT_SetStatistics:    /* ALTER COLUMN SET
STATISTICS*/
 
-            ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+            /* Recurse to child tables that are foreign, too */
+            ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode);            /* Performs own permission checks
*/           ATPrepSetStatistics(rel, cmd->name, cmd->def, lockmode);            pass = AT_PASS_MISC;
 
@@ -3095,7 +3120,8 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,            break;        case
AT_SetStorage:       /* ALTER COLUMN SET STORAGE */            ATSimplePermissions(rel, ATT_TABLE | ATT_MATVIEW);
 
-            ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+            /* Don't recurse to child tables that are foreign */
+            ATSimpleRecursion(wqueue, rel, cmd, recurse, false, lockmode);            /* No command-specific prep
needed*/            pass = AT_PASS_MISC;            break;
 
@@ -3113,7 +3139,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,            pass = AT_PASS_ADD_INDEX;
        break;        case AT_AddConstraint:    /* ADD CONSTRAINT */
 
-            ATSimplePermissions(rel, ATT_TABLE);
+            ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);            /* Recursion occurs during execution
phase*/            /* No command-specific prep needed except saving recurse flag */            if (recurse)
 
@@ -3127,7 +3153,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,            pass = AT_PASS_ADD_CONSTR;
         break;        case AT_DropConstraint:    /* DROP CONSTRAINT */
 
-            ATSimplePermissions(rel, ATT_TABLE);
+            ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);            /* Recursion occurs during execution
phase*/            /* No command-specific prep needed except saving recurse flag */            if (recurse)
 
@@ -3195,11 +3221,17 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,            pass = AT_PASS_MISC;
     break;        case AT_AddInherit:        /* INHERIT */
 
-            ATSimplePermissions(rel, ATT_TABLE);
+            ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);            /* This command never recurses */
     ATPrepAddInherit(rel);            pass = AT_PASS_MISC;            break;
 
+        case AT_DropInherit:    /* NO INHERIT */
+            ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
+            /* This command never recurses */
+            /* No command-specific prep needed */
+            pass = AT_PASS_MISC;
+            break;        case AT_AlterConstraint:        /* ALTER CONSTRAINT */            ATSimplePermissions(rel,
ATT_TABLE);           pass = AT_PASS_MISC;
 
@@ -3233,7 +3265,6 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,        case AT_EnableAlwaysRule:
case AT_EnableReplicaRule:        case AT_DisableRule:
 
-        case AT_DropInherit:    /* NO INHERIT */        case AT_AddOf:            /* OF */        case AT_DropOf: /*
NOTOF */            ATSimplePermissions(rel, ATT_TABLE);
 
@@ -4174,7 +4205,8 @@ ATWrongRelkindError(Relation rel, int allowed_targets) */static voidATSimpleRecursion(List
**wqueue,Relation rel,
 
-                  AlterTableCmd *cmd, bool recurse, LOCKMODE lockmode)
+                  AlterTableCmd *cmd, bool recurse,
+                  bool include_foreign, LOCKMODE lockmode){    /*     * Propagate to children if desired.  Non-table
relationsnever have
 
@@ -4202,8 +4234,12 @@ ATSimpleRecursion(List **wqueue, Relation rel,                continue;            /*
find_all_inheritorsalready got lock */            childrel = relation_open(childrelid, NoLock);
 
-            CheckTableNotInUse(childrel, "ALTER TABLE");
-            ATPrepCmd(wqueue, childrel, cmd, false, true, lockmode);
+            if (childrel->rd_rel->relkind != RELKIND_FOREIGN_TABLE
+                || include_foreign)
+            {
+                CheckTableNotInUse(childrel, "ALTER TABLE");
+                ATPrepCmd(wqueue, childrel, cmd, false, true, lockmode);
+            }            relation_close(childrel, NoLock);        }    }
@@ -4493,7 +4529,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,    /* At top level,
permissioncheck was done in ATPrepCmd, else do it */    if (recursing)
 
-        ATSimplePermissions(rel, ATT_TABLE);
+        ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);    attrdesc = heap_open(AttributeRelationId,
RowExclusiveLock);
@@ -4789,6 +4825,11 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,        /*
find_inheritance_childrenalready got lock */        childrel = heap_open(childrelid, NoLock);
 
+        if (childrel->rd_rel->relkind == RELKIND_FOREIGN_TABLE && isOid)
+            ereport(ERROR,
+                    (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+                     errmsg("cannot add OID column to foreign table \"%s\"",
+                            RelationGetRelationName(childrel))));        CheckTableNotInUse(childrel, "ALTER TABLE");
     /* Find or create work queue entry for this table */
 
@@ -5389,7 +5430,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName,    /* At top level, permission
checkwas done in ATPrepCmd, else do it */    if (recursing)
 
-        ATSimplePermissions(rel, ATT_TABLE);
+        ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);    /*     * get the number of the attribute
@@ -5781,7 +5822,14 @@ ATAddCheckConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel,    /* At top level,
permissioncheck was done in ATPrepCmd, else do it */    if (recursing)
 
-        ATSimplePermissions(rel, ATT_TABLE);
+        ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
+
+    /* Don't allow ADD CONSTRAINT NOT VALID to be applied to foreign tables */
+    if (tab->relkind == RELKIND_FOREIGN_TABLE &&
+        constr->skip_validation && !recursing)
+        ereport(ERROR,
+                (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+                 errmsg("NOT VALID is not supported on foreign tables")));    /*     * Call AddRelationNewConstraints
todo the work, making sure it works on
 
@@ -5792,9 +5840,17 @@ ATAddCheckConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel,     * omitted from the
returnedlist, which is what we want: we do not need     * to do any validation work.  That can only happen at child
tables,    * though, since we disallow merging at the top level.
 
-     */
+     *
+     * When propagating a NOT VALID option to children that are foreign tables,
+     * we quietly ignore the option.  Note that this is safe because foreign
+     * tables don't have any children.
+     */
+    constr = (Constraint *) copyObject(constr);
+    if (tab->relkind == RELKIND_FOREIGN_TABLE &&
+        constr->skip_validation && recursing)
+        constr->skip_validation = false;    newcons = AddRelationNewConstraints(rel, NIL,
-                                        list_make1(copyObject(constr)),
+                                        list_make1(constr),                                        recursing,
/*allow_merge */                                        !recursing,        /* is_local */
        is_readd);        /* is_internal */
 
@@ -7274,7 +7330,7 @@ ATExecDropConstraint(Relation rel, const char *constrName,    /* At top level, permission check
wasdone in ATPrepCmd, else do it */    if (recursing)
 
-        ATSimplePermissions(rel, ATT_TABLE);
+        ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);    conrel = heap_open(ConstraintRelationId,
RowExclusiveLock);
@@ -7609,7 +7665,10 @@ ATPrepAlterColumnType(List **wqueue,     * alter would put them out of step.     */    if
(recurse)
-        ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+    {
+        /* Recurse to child tables that are foreign, too */
+        ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode);
+    }    else if (!recursing &&             find_inheritance_children(RelationGetRelid(rel), NoLock) != NIL)
ereport(ERROR,
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 3d2c739..b5e3ccf 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -61,6 +61,7 @@ int            vacuum_multixact_freeze_table_age;/* A few variables that don't seem worth passing
aroundas parameters */static MemoryContext vac_context = NULL;
 
+static VacuumMode vac_mode;static BufferAccessStrategy vac_strategy;
@@ -146,6 +147,20 @@ vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast,
ALLOCSET_DEFAULT_MAXSIZE);   /*
 
+     * Identify vacuum mode.  If relid is not InvalidOid, the caller should be
+     * an autovacuum worker.  See the above comments.
+     */
+    if (relid != InvalidOid)
+        vac_mode = VAC_MODE_AUTOVACUUM;
+    else
+    {
+        if (!vacstmt->relation)
+            vac_mode = VAC_MODE_ALL;
+        else
+            vac_mode = VAC_MODE_SINGLE;
+    }
+
+    /*     * If caller didn't give us a buffer strategy object, make one in the     * cross-transaction memory
context.    */
 
@@ -248,7 +263,7 @@ vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast,
PushActiveSnapshot(GetTransactionSnapshot());               }
 
-                analyze_rel(relid, vacstmt, vac_strategy);
+                analyze_rel(relid, vacstmt, vac_mode, vac_strategy);                if (use_own_xacts)
{
diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c
index 0410fdd..836fae6 100644
--- a/src/backend/optimizer/prep/prepunion.c
+++ b/src/backend/optimizer/prep/prepunion.c
@@ -1338,11 +1338,12 @@ expand_inherited_rtentry(PlannerInfo *root, RangeTblEntry *rte, Index rti)        /*         *
Buildan RTE for the child, and attach to query's rangetable list.         * We copy most fields of the parent's RTE,
butreplace relation OID,
 
-         * and set inh = false.  Also, set requiredPerms to zero since all
-         * required permissions checks are done on the original RTE.
+         * relkind and set inh = false.  Also, set requiredPerms to zero since
+         * all required permissions checks are done on the original RTE.         */        childrte = copyObject(rte);
      childrte->relid = childOID;
 
+        childrte->relkind = newrelation->rd_rel->relkind;        childrte->inh = false;        childrte->requiredPerms
=0;        parse->rtable = lappend(parse->rtable, childrte);
 
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 4e05dcd..880595f 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -16,6 +16,7 @@#include <math.h>
+#include "foreign/fdwapi.h"#include "miscadmin.h"#include "nodes/nodeFuncs.h"#include "optimizer/clauses.h"
@@ -2062,6 +2063,31 @@ reparameterize_path(PlannerInfo *root, Path *path,        case T_SubqueryScan:            return
create_subqueryscan_path(root,rel, path->pathkeys,                                            required_outer);
 
+        case T_ForeignScan:
+            {
+                ForeignPath *newpath = NULL;
+
+                /* Let the FDW reparameterize the path node if possible */
+                if (rel->fdwroutine->ReparameterizeForeignPath != NULL)
+                {
+                    Index        scan_relid = rel->relid;
+                    RangeTblEntry *rte;
+
+                    /* it should be a base rel... */
+                    Assert(scan_relid > 0);
+                    Assert(rel->rtekind == RTE_RELATION);
+                    rte = planner_rt_fetch(scan_relid, root);
+                    Assert(rte->rtekind == RTE_RELATION);
+
+                    newpath =
+                        rel->fdwroutine->ReparameterizeForeignPath(root,
+                                                                   rel,
+                                                                   rte->relid,
+                                                          (ForeignPath *) path,
+                                                               required_outer);
+                }
+                return (Path *) newpath;
+            }        default:            break;    }
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 605c9b4..a206501 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -4213,32 +4213,32 @@ AlterForeignServerStmt: ALTER SERVER name foreign_server_version
alter_generic_oCreateForeignTableStmt:       CREATE FOREIGN TABLE qualified_name            '(' OptTableElementList
')'
-            SERVER name create_generic_options
+            OptInherit SERVER name create_generic_options                {                    CreateForeignTableStmt
*n= makeNode(CreateForeignTableStmt);                    $4->relpersistence = RELPERSISTENCE_PERMANENT;
  n->base.relation = $4;                    n->base.tableElts = $6;
 
-                    n->base.inhRelations = NIL;
+                    n->base.inhRelations = $8;                    n->base.if_not_exists = false;                    /*
FDW-specificdata */
 
-                    n->servername = $9;
-                    n->options = $10;
+                    n->servername = $10;
+                    n->options = $11;                    $$ = (Node *) n;                }        | CREATE FOREIGN
TABLEIF_P NOT EXISTS qualified_name            '(' OptTableElementList ')'
 
-            SERVER name create_generic_options
+            OptInherit SERVER name create_generic_options                {                    CreateForeignTableStmt
*n= makeNode(CreateForeignTableStmt);                    $7->relpersistence = RELPERSISTENCE_PERMANENT;
  n->base.relation = $7;                    n->base.tableElts = $9;
 
-                    n->base.inhRelations = NIL;
+                    n->base.inhRelations = $11;                    n->base.if_not_exists = true;                    /*
FDW-specificdata */
 
-                    n->servername = $12;
-                    n->options = $13;
+                    n->servername = $13;
+                    n->options = $14;                    $$ = (Node *) n;                }        ;
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index 7c1939f..949392a 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -515,12 +515,6 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column)                break;
  case CONSTR_CHECK:
 
-                if (cxt->isforeign)
-                    ereport(ERROR,
-                            (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
-                    errmsg("constraints are not supported on foreign tables"),
-                             parser_errposition(cxt->pstate,
-                                                constraint->location)));                cxt->ckconstraints =
lappend(cxt->ckconstraints,constraint);                break;
 
@@ -529,7 +523,7 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column)                if
(cxt->isforeign)                   ereport(ERROR,                            (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
 
-                    errmsg("constraints are not supported on foreign tables"),
+                    errmsg("primary key or unique constraints are not supported on foreign tables"),
         parser_errposition(cxt->pstate,                                                constraint->location)));
       if (constraint->keys == NIL)
 
@@ -546,7 +540,7 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column)                if
(cxt->isforeign)                   ereport(ERROR,                            (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
 
-                    errmsg("constraints are not supported on foreign tables"),
+                    errmsg("foreign key constraints are not supported on foreign tables"),
parser_errposition(cxt->pstate,                                               constraint->location)));
 
@@ -605,10 +599,14 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column)static
voidtransformTableConstraint(CreateStmtContext*cxt, Constraint *constraint){
 
-    if (cxt->isforeign)
+    if (cxt->isforeign &&
+        (constraint->contype == CONSTR_PRIMARY ||
+         constraint->contype == CONSTR_UNIQUE ||
+         constraint->contype == CONSTR_EXCLUSION ||
+         constraint->contype == CONSTR_FOREIGN))        ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
-                 errmsg("constraints are not supported on foreign tables"),
+                 errmsg("primary key, unique, exclusion, or foreign key constraints are not supported on foreign
tables"),                parser_errposition(cxt->pstate,                                    constraint->location)));
 
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index d33552a..400e373 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -140,6 +140,15 @@ extern int    vacuum_multixact_freeze_min_age;extern int    vacuum_multixact_freeze_table_age;
+/* Possible modes for vacuum() */
+typedef enum
+{
+    VAC_MODE_ALL,                /* Vacuum/analyze all relations */
+    VAC_MODE_SINGLE,            /* Vacuum/analyze a specific relation */
+    VAC_MODE_AUTOVACUUM            /* Autovacuum worker */
+} VacuumMode;
+
+/* in commands/vacuum.c */extern void vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast,       BufferAccessStrategy
bstrategy,bool for_wraparound, bool isTopLevel);
 
@@ -174,7 +183,9 @@ extern void lazy_vacuum_rel(Relation onerel, VacuumStmt *vacstmt,
BufferAccessStrategybstrategy);/* in commands/analyze.c */
 
-extern void analyze_rel(Oid relid, VacuumStmt *vacstmt,
+extern void analyze_rel(Oid relid,
+            VacuumStmt *vacstmt,
+            VacuumMode vacmode,            BufferAccessStrategy bstrategy);extern bool std_typanalyze(VacAttrStats
*stats);externdouble anl_random_fract(void);
 
diff --git a/src/include/foreign/fdwapi.h b/src/include/foreign/fdwapi.h
index 1b735da..5f3996a 100644
--- a/src/include/foreign/fdwapi.h
+++ b/src/include/foreign/fdwapi.h
@@ -31,6 +31,12 @@ typedef void (*GetForeignPaths_function) (PlannerInfo *root,
            RelOptInfo *baserel,                                                      Oid foreigntableid);
 
+typedef ForeignPath *(*ReparameterizeForeignPath_function) (PlannerInfo *root,
+                                                         RelOptInfo *baserel,
+                                                          Oid foreigntableid,
+                                                            ForeignPath *path,
+                                                      Relids required_outer);
+typedef ForeignScan *(*GetForeignPlan_function) (PlannerInfo *root,
    RelOptInfo *baserel,                                                          Oid foreigntableid,
 
@@ -117,6 +123,7 @@ typedef struct FdwRoutine    /* Functions for scanning foreign tables */
GetForeignRelSize_functionGetForeignRelSize;    GetForeignPaths_function GetForeignPaths;
 
+    ReparameterizeForeignPath_function ReparameterizeForeignPath;    GetForeignPlan_function GetForeignPlan;
BeginForeignScan_functionBeginForeignScan;    IterateForeignScan_function IterateForeignScan;
 
diff --git a/src/test/regress/expected/foreign_data.out b/src/test/regress/expected/foreign_data.out
index ff203b2..36df338 100644
--- a/src/test/regress/expected/foreign_data.out
+++ b/src/test/regress/expected/foreign_data.out
@@ -748,16 +748,12 @@ CREATE TABLE use_ft1_column_type (x ft1);ALTER FOREIGN TABLE ft1 ALTER COLUMN c8 SET DATA TYPE
integer;   -- ERRORERROR:  cannot alter foreign table "ft1" because column "use_ft1_column_type.x" uses its row
typeDROPTABLE use_ft1_column_type;
 
-ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0); -- ERROR
-ERROR:  constraints are not supported on foreign tables
-LINE 1: ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c...
-                                    ^
+ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0);ALTER FOREIGN TABLE ft1 DROP CONSTRAINT no_const;
           -- ERROR
 
-ERROR:  "ft1" is not a table
+ERROR:  constraint "no_const" of relation "ft1" does not existALTER FOREIGN TABLE ft1 DROP CONSTRAINT IF EXISTS
no_const;
-ERROR:  "ft1" is not a table
-ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c1_check;
-ERROR:  "ft1" is not a table
+NOTICE:  constraint "no_const" of relation "ft1" does not exist, skipping
+ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c9_check;ALTER FOREIGN TABLE ft1 SET WITH OIDS;
--ERRORERROR:  "ft1" is not a tableALTER FOREIGN TABLE ft1 OWNER TO regress_test_role;
 
diff --git a/src/test/regress/sql/foreign_data.sql b/src/test/regress/sql/foreign_data.sql
index 0f0869e..8f9ea86 100644
--- a/src/test/regress/sql/foreign_data.sql
+++ b/src/test/regress/sql/foreign_data.sql
@@ -314,10 +314,10 @@ ALTER FOREIGN TABLE ft1 ALTER COLUMN c8 SET STATISTICS -1;CREATE TABLE use_ft1_column_type (x
ft1);ALTERFOREIGN TABLE ft1 ALTER COLUMN c8 SET DATA TYPE integer;    -- ERRORDROP TABLE use_ft1_column_type;
 
-ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0); -- ERROR
+ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0);ALTER FOREIGN TABLE ft1 DROP CONSTRAINT no_const;
           -- ERRORALTER FOREIGN TABLE ft1 DROP CONSTRAINT IF EXISTS no_const;
 
-ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c1_check;
+ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c9_check;ALTER FOREIGN TABLE ft1 SET WITH OIDS;
--ERRORALTER FOREIGN TABLE ft1 OWNER TO regress_test_role;ALTER FOREIGN TABLE ft1 OPTIONS (DROP delimiter, SET quote
'~',ADD escape '@'); 

Re: inherit support for foreign tables

From
Ashutosh Bapat
Date:
Hi,
Selecting tableoid on parent causes an error, "ERROR:  cannot extract system attribute from virtual tuple". The foreign table has an OID which can be reported as tableoid for the rows coming from that foreign table. Do we want to do that?


On Fri, Jun 20, 2014 at 1:34 PM, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
Hello,

Before continueing discussion, I tried this patch on the current
head.

This patch applies cleanly except one hunk because of a
modification just AFTER it, which did no harm. Finally all
regression tests suceeded.

Attached is the rebased patch of v11 up to the current master.

regards,

--
Kyotaro Horiguchi
NTT Open Source Software Center


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers




--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
Hi Ashutosh,

Thank you for the review.

(2014/06/23 18:35), Ashutosh Bapat wrote:
> Hi,
> Selecting tableoid on parent causes an error, "ERROR:  cannot extract
> system attribute from virtual tuple". The foreign table has an OID which
> can be reported as tableoid for the rows coming from that foreign table.
> Do we want to do that?

No.  I think it's a bug.  I'll fix it.

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/06/24 16:30), Etsuro Fujita wrote:
> (2014/06/23 18:35), Ashutosh Bapat wrote:

>> Selecting tableoid on parent causes an error, "ERROR:  cannot extract
>> system attribute from virtual tuple". The foreign table has an OID which
>> can be reported as tableoid for the rows coming from that foreign table.
>> Do we want to do that?
>
> No.  I think it's a bug.  I'll fix it.

Done.  I think this is because create_foreignscan_plan() makes reference
to attr_needed, which isn't computed for inheritance children.  To aboid
this, I've modified create_foreignscan_plan() to see reltargetlist and
baserestrictinfo, instead of attr_needed.  Please find attached an
updated version of the patch.

Sorry for the delay.

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Ashutosh Bapat
Date:
I checked that it's reporting the right tableoid now.

BTW, why aren't you using the tlist passed to this function? I guess create_scan_plan() passes tlist after processing it, so that should be used rather than rel->reltargetlist.


On Mon, Jun 30, 2014 at 12:22 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/06/24 16:30), Etsuro Fujita wrote:
(2014/06/23 18:35), Ashutosh Bapat wrote:

Selecting tableoid on parent causes an error, "ERROR:  cannot extract
system attribute from virtual tuple". The foreign table has an OID which
can be reported as tableoid for the rows coming from that foreign table.
Do we want to do that?

No.  I think it's a bug.  I'll fix it.

Done.  I think this is because create_foreignscan_plan() makes reference to attr_needed, which isn't computed for inheritance children.  To aboid this, I've modified create_foreignscan_plan() to see reltargetlist and baserestrictinfo, instead of attr_needed.  Please find attached an updated version of the patch.


Sorry for the delay.

Best regards,
Etsuro Fujita



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/06/30 17:47), Ashutosh Bapat wrote:
> I checked that it's reporting the right tableoid now.

Thank you for the check.

> BTW, why aren't you using the tlist passed to this function? I guess
> create_scan_plan() passes tlist after processing it, so that should be
> used rather than rel->reltargetlist.

I think that that would be maybe OK, but I think that it would not be 
efficient to use the list to compute attrs_used, because the tlist would 
have more information than rel->reltargetlist in cases where the tlist 
is build through build_physical_tlist().

Thanks,

Best regards,
Etsuro Fujita

>
> On Mon, Jun 30, 2014 at 12:22 PM, Etsuro Fujita
> <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:
>
>     (2014/06/24 16:30), Etsuro Fujita wrote:
>
>         (2014/06/23 18:35), Ashutosh Bapat wrote:
>
>
>             Selecting tableoid on parent causes an error, "ERROR:
>               cannot extract
>             system attribute from virtual tuple". The foreign table has
>             an OID which
>             can be reported as tableoid for the rows coming from that
>             foreign table.
>             Do we want to do that?
>
>
>         No.  I think it's a bug.  I'll fix it.
>
>
>     Done.  I think this is because create_foreignscan_plan() makes
>     reference to attr_needed, which isn't computed for inheritance
>     children.  To aboid this, I've modified create_foreignscan_plan() to
>     see reltargetlist and baserestrictinfo, instead of attr_needed.
>       Please find attached an updated version of the patch.
>
>
>     Sorry for the delay.
>
>     Best regards,
>     Etsuro Fujita
>
>
>
>
> --
> Best Wishes,
> Ashutosh Bapat
> EnterpriseDB Corporation
> The Postgres Database Company



Re: inherit support for foreign tables

From
Ashutosh Bapat
Date:



On Mon, Jun 30, 2014 at 4:17 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/06/30 17:47), Ashutosh Bapat wrote:
I checked that it's reporting the right tableoid now.

Thank you for the check.


BTW, why aren't you using the tlist passed to this function? I guess
create_scan_plan() passes tlist after processing it, so that should be
used rather than rel->reltargetlist.

I think that that would be maybe OK, but I think that it would not be efficient to use the list to compute attrs_used, because the tlist would have more information than rel->reltargetlist in cases where the tlist is build through build_physical_tlist().


In that case, we can call build_relation_tlist() for foreign tables.
 

Thanks,

Best regards,
Etsuro Fujita


On Mon, Jun 30, 2014 at 12:22 PM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:

    (2014/06/24 16:30), Etsuro Fujita wrote:

        (2014/06/23 18:35), Ashutosh Bapat wrote:


            Selecting tableoid on parent causes an error, "ERROR:
              cannot extract
            system attribute from virtual tuple". The foreign table has
            an OID which
            can be reported as tableoid for the rows coming from that
            foreign table.
            Do we want to do that?


        No.  I think it's a bug.  I'll fix it.


    Done.  I think this is because create_foreignscan_plan() makes
    reference to attr_needed, which isn't computed for inheritance
    children.  To aboid this, I've modified create_foreignscan_plan() to
    see reltargetlist and baserestrictinfo, instead of attr_needed.
      Please find attached an updated version of the patch.


    Sorry for the delay.

    Best regards,
    Etsuro Fujita




--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: inherit support for foreign tables

From
Tom Lane
Date:
Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
> Done.  I think this is because create_foreignscan_plan() makes reference 
> to attr_needed, which isn't computed for inheritance children.

I wonder whether it isn't time to change that.  It was coded like that
originally only because calculating the values would've been a waste of
cycles at the time.  But this is at least the third place where it'd be
useful to have attr_needed for child rels.
        regards, tom lane



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/06/30 20:17), Ashutosh Bapat wrote:
> On Mon, Jun 30, 2014 at 4:17 PM, Etsuro Fujita
> <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:

>     (2014/06/30 17:47), Ashutosh Bapat wrote:

>         BTW, why aren't you using the tlist passed to this function? I guess
>         create_scan_plan() passes tlist after processing it, so that
>         should be
>         used rather than rel->reltargetlist.

>     I think that that would be maybe OK, but I think that it would not
>     be efficient to use the list to compute attrs_used, because the
>     tlist would have more information than rel->reltargetlist in cases
>     where the tlist is build through build_physical_tlist().

> In that case, we can call build_relation_tlist() for foreign tables.

Do you mean build_physical_tlist()?

Yeah, we can call build_physical_tlist() (and do that in some cases), 
but if we call the function, it would generate a tlist that contains all 
Vars in the relation, not only those Vars actually needed by the query 
(ie, Vars in reltargetlist), and thus it would take more cycles to 
compute attr_used from the tlist than from reltargetlist.  That' what I 
wanted to say.

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/06/30 22:48), Tom Lane wrote:
> Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
>> Done.  I think this is because create_foreignscan_plan() makes reference
>> to attr_needed, which isn't computed for inheritance children.
> 
> I wonder whether it isn't time to change that.  It was coded like that
> originally only because calculating the values would've been a waste of
> cycles at the time.  But this is at least the third place where it'd be
> useful to have attr_needed for child rels.

+1 for calculating attr_needed for child rels.  (I was wondering too.)

I'll create a separate patch for it.

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hello,

> >  BTW, why aren't you using the tlist passed to this function? I guess
> >> create_scan_plan() passes tlist after processing it, so that should be
> >> used rather than rel->reltargetlist.
> >>
> >
> > I think that that would be maybe OK, but I think that it would not be
> > efficient to use the list to compute attrs_used, because the tlist would
> > have more information than rel->reltargetlist in cases where the tlist is
> > build through build_physical_tlist().
> >
> >
> In that case, we can call build_relation_tlist() for foreign tables.

# is it build_base_rel_tilst() ?

It needs only one effective line added, which would be seemingly
far simple ingoring potential extra calculation for non-child
relations (I suppose getting rid of it needs a foreach on
rel->append_rel_list..) and too-longer physical tlist.  On the
other hand build_relation_tlist() and this patch seem to be in
the same order of computational complexity for the length of the
tlist.

I prefer to use build_base_rel_tlist() for the clarity of
code. But I don't know how high the chance to big physical tlist
and non-child relations become to be an annoyance. Is there any
suggestions?



By the way, I tried xmin and xmax for the file_fdw tables.

postgres=# select tableoid, xmin, xmax, * from passwd1;
tableoid | xmin |    xmax    |     uname     | pass |  uid  |  gid  | ..   16396 |  244 | 4294967295 | root          |
x   |     0 |     0 | root...  16396 |  252 | 4294967295 | bin           | x    |     1 |     1 | bin...  16396 |  284
|4294967295 | daemon        | x    |     2 |     2 | daemon...
 

The xmin and xmax apparently doesn't look sane. After some
investigation, I found that they came from the following place in
heap_form_tuple(), (call stach is show below)

| HeapTupleHeaderSetDatumLength(td, len);
| HeapTupleHeaderSetTypeId(td, tupleDescriptor->tdtypeid);
| HeapTupleHeaderSetTypMod(td, tupleDescriptor->tdtypmod);

HeapTupleHeader is a union of HeapTupleFields and
DatumTupleFields and these macors seem to treat it as the
latter. Then later this tuple seems to be read as the former so
xmin and xmax should have that values.

This seems to be a bug. But I have no idea how to deal with it
for now. I'll take more look into this.

The call stack onto the above heap_form_tuple is as follows.

#0  heap_form_tuple (tupleDescriptor=0x7fc46d2953a8, values=0x10dd1a0, isnull=0x10dd1f8 "") at heaptuple.c:731
#1  ExecCopySlotTuple (slot=0x10dd0e0) at execTuples.c:574
#2  ExecMaterializeSlot (slot=0x10dd0e0) at execTuples.c:760
#3  ForeignNext (node=0x10dc7b0) at nodeForeignscan.c:61
#4  ExecScanFetch (node=0x10dc7b0, accessMtd=0x66599c <ForeignNext>, recheckMtd=0x665a43 <ForeignRecheck>) at
execScan.c:82
#5  ExecScan (node=0x10dc7b0, accessMtd=0x66599c <ForeignNext>, recheckMtd=0x665a43 <ForeignRecheck>) at
execScan.c:167
#6  ExecForeignScan (node=0x10dc7b0) at nodeForeignscan.c:91
#7  ExecProcNode (node=0x10dc7b0) at execProcnode.c:442



regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



Re: inherit support for foreign tables

From
Ashutosh Bapat
Date:
<div dir="ltr">If we are going to change that portion of the code, we may as well go a bit forward and allow any
expressionsto be fetched from a foreign server (obviously, if that server is capable of doing so). It will help, when
wecome to aggregate push-down or whole query push-down (whenever that happens). So, instead of attr_needed, which
restrictsonly the attributes to be fetched, why not to targetlist itself?<br /></div><div class="gmail_extra"><br /><br
/><divclass="gmail_quote">On Mon, Jun 30, 2014 at 7:18 PM, Tom Lane <span dir="ltr"><<a
href="mailto:tgl@sss.pgh.pa.us"target="_blank">tgl@sss.pgh.pa.us</a>></span> wrote:<br /><blockquote
class="gmail_quote"style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">Etsuro Fujita
<<ahref="mailto:fujita.etsuro@lab.ntt.co.jp">fujita.etsuro@lab.ntt.co.jp</a>> writes:<br /> > Done.  I think
thisis because create_foreignscan_plan() makes reference<br /> > to attr_needed, which isn't computed for
inheritancechildren.<br /><br /></div>I wonder whether it isn't time to change that.  It was coded like that<br />
originallyonly because calculating the values would've been a waste of<br /> cycles at the time.  But this is at least
thethird place where it'd be<br /> useful to have attr_needed for child rels.<br /><br />                        
regards,tom lane<br /></blockquote></div><br /><br clear="all" /><br />-- <br /><div dir="ltr">Best Wishes,<br
/>AshutoshBapat<br />EnterpriseDB Corporation<br />The Postgres Database Company<br /></div></div> 

Re: inherit support for foreign tables

From
Ashutosh Bapat
Date:



On Tue, Jul 1, 2014 at 7:39 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/06/30 20:17), Ashutosh Bapat wrote:
On Mon, Jun 30, 2014 at 4:17 PM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:

    (2014/06/30 17:47), Ashutosh Bapat wrote:

        BTW, why aren't you using the tlist passed to this function? I guess
        create_scan_plan() passes tlist after processing it, so that
        should be
        used rather than rel->reltargetlist.

    I think that that would be maybe OK, but I think that it would not
    be efficient to use the list to compute attrs_used, because the
    tlist would have more information than rel->reltargetlist in cases
    where the tlist is build through build_physical_tlist().

In that case, we can call build_relation_tlist() for foreign tables.

Do you mean build_physical_tlist()?


Sorry, I meant build_path_tlist() or disuse_physical_tlist() whichever is appropriate.

We may want to modify use_physical_tlist(), to return false, in case of foreign tables. BTW, it does return false for inheritance trees.

 486     /*
 487      * Can't do it with inheritance cases either (mainly because Append
 488      * doesn't project).
 489      */
 490     if (rel->reloptkind != RELOPT_BASEREL)
 491         return false;

Yeah, we can call build_physical_tlist() (and do that in some cases), but if we call the function, it would generate a tlist that contains all Vars in the relation, not only those Vars actually needed by the query (ie, Vars in reltargetlist), and thus it would take more cycles to compute attr_used from the tlist than from reltargetlist.  That' what I wanted to say.


Thanks,

Best regards,
Etsuro Fujita



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hello,

Sorry, this was no relation with this patch.

ForeignNext materializes the slot, which would be any of physical
and virtual tuple, when system column was requested. If it was a
virtual one, file_fdw makes this, heap_form_tuple generates the
tuple as DatumTuple. The result is a jumble of virtual and
physical. But the returning slot has tts_tuple so the caller
interprets it as a physical one.

Anyway the requests for xmin/xmax could not be prevented
beforehand in current framework, I did rewrite the physical tuple
header so as to really be a physical one.

This would be another patch, so I will put this into next CF if
this don't get any immediate objection.


> By the way, I tried xmin and xmax for the file_fdw tables.
> 
> postgres=# select tableoid, xmin, xmax, * from passwd1;
> tableoid | xmin |    xmax    |     uname     | pass |  uid  |  gid  | .. 
>    16396 |  244 | 4294967295 | root          | x    |     0 |     0 | root...
>    16396 |  252 | 4294967295 | bin           | x    |     1 |     1 | bin...
>    16396 |  284 | 4294967295 | daemon        | x    |     2 |     2 | daemon...
> 
> The xmin and xmax apparently doesn't look sane. After some
> investigation, I found that they came from the following place in
> heap_form_tuple(), (call stach is show below)


regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center
diff --git a/src/backend/executor/nodeForeignscan.c b/src/backend/executor/nodeForeignscan.c
index 9cc5345..728db14 100644
--- a/src/backend/executor/nodeForeignscan.c
+++ b/src/backend/executor/nodeForeignscan.c
@@ -22,6 +22,8 @@ */#include "postgres.h"
+#include "access/transam.h"
+#include "access/htup_details.h"#include "executor/executor.h"#include "executor/nodeForeignscan.h"#include
"foreign/fdwapi.h"
@@ -58,8 +60,21 @@ ForeignNext(ForeignScanState *node)     */    if (plan->fsSystemCol && !TupIsNull(slot))    {
+        bool        was_virtual_tuple = (slot->tts_tuple == NULL);        HeapTuple    tup =
ExecMaterializeSlot(slot);
+        if (was_virtual_tuple)
+        {
+            /*
+             * ExecMaterializeSlot fills the tuple header as a
+             * DatumTupleFields if the slot was a virtual tuple, but a
+             * physical one is needed here. So rewrite the tuple header as
+             * HeapTupleFirelds.
+             */
+            HeapTupleHeaderSetXmin(tup->t_data, FrozenTransactionId);
+            HeapTupleHeaderSetXmax(tup->t_data, InvalidTransactionId);
+            HeapTupleHeaderSetCmin(tup->t_data, FirstCommandId);
+        }        tup->t_tableOid = RelationGetRelid(node->ss.ss_currentRelation);    }

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/07/01 15:13), Ashutosh Bapat wrote:
> On Tue, Jul 1, 2014 at 7:39 AM, Etsuro Fujita
> <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:

> We may want to modify use_physical_tlist(), to return false, in case of
> foreign tables. BTW, it does return false for inheritance trees.

Yeah, but please consider cases where foreign tables are not inheritance 
child rels (and any system columns are requested).

>   486     /*
>   487      * Can't do it with inheritance cases either (mainly because
> Append
>   488      * doesn't project).
>   489      */
>   490     if (rel->reloptkind != RELOPT_BASEREL)
>   491         return false;
>
>     Yeah, we can call build_physical_tlist() (and do that in some
>     cases), but if we call the function, it would generate a tlist that
>     contains all Vars in the relation, not only those Vars actually
>     needed by the query (ie, Vars in reltargetlist), and thus it would
>     take more cycles to compute attr_used from the tlist than from
>     reltargetlist.  That' what I wanted to say.

Maybe I'm missing something, but what's the point of using the tlist, 
not reltargetlist?

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Ashutosh Bapat
Date:



On Tue, Jul 1, 2014 at 12:25 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/07/01 15:13), Ashutosh Bapat wrote:
On Tue, Jul 1, 2014 at 7:39 AM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:

We may want to modify use_physical_tlist(), to return false, in case of
foreign tables. BTW, it does return false for inheritance trees.

Yeah, but please consider cases where foreign tables are not inheritance child rels (and any system columns are requested).


  486     /*
  487      * Can't do it with inheritance cases either (mainly because
Append
  488      * doesn't project).
  489      */
  490     if (rel->reloptkind != RELOPT_BASEREL)
  491         return false;

    Yeah, we can call build_physical_tlist() (and do that in some
    cases), but if we call the function, it would generate a tlist that
    contains all Vars in the relation, not only those Vars actually
    needed by the query (ie, Vars in reltargetlist), and thus it would
    take more cycles to compute attr_used from the tlist than from
    reltargetlist.  That' what I wanted to say.

Maybe I'm missing something, but what's the point of using the tlist, not reltargetlist?


Compliance with other create_*scan_plan() functions. The tlist passed to those functions is sometimes preprocessed in create_scan_plan() and some of the function it calls. If we use reltargetlist directly, we loose that preprocessing. I have not see any of create_*scan_plan() fetch the targetlist directly from RelOptInfo. It is always the one supplied by build_path_tlist() or disuse_physical_tlist() (which in turn calls build_path_tlist()) or build_physical_tlist().
 

Thanks,

Best regards,
Etsuro Fujita



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/07/01 16:04), Ashutosh Bapat wrote:
> On Tue, Jul 1, 2014 at 12:25 PM, Etsuro Fujita
> <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:

>     Maybe I'm missing something, but what's the point of using the
>     tlist, not reltargetlist?

> Compliance with other create_*scan_plan() functions. The tlist passed to
> those functions is sometimes preprocessed in create_scan_plan() and some
> of the function it calls. If we use reltargetlist directly, we loose
> that preprocessing. I have not see any of create_*scan_plan() fetch the
> targetlist directly from RelOptInfo. It is always the one supplied by
> build_path_tlist() or disuse_physical_tlist() (which in turn calls
> build_path_tlist()) or build_physical_tlist().

I've got the point.

As I said upthread, I'll work on calculating attr_needed for child rels, 
and I hope that that will eliminate your concern.

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hi, 

At Tue, 01 Jul 2014 16:30:41 +0900, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote in
<53B263A1.3060107@lab.ntt.co.jp>
> I've got the point.
> 
> As I said upthread, I'll work on calculating attr_needed for child
> rels, and I hope that that will eliminate your concern.

Inheritance tree is expanded far after where attr_needed for the
parent built, in set_base_rel_sizes() in make_one_rel(). I'm
afraid that building all attr_needed for whole inheritance tree
altogether is a bit suffering.

I have wanted the point of inheritance expansion earlier for
another patch. Do you think that rearranging there? Or generate
them individually in crete_foreign_plan()?

Anyway, I'm lookin forward to your next patch. So no answer
needed.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



Re: inherit support for foreign tables

From
Noah Misch
Date:
On Fri, Jun 20, 2014 at 05:04:06PM +0900, Kyotaro HORIGUCHI wrote:
> Attached is the rebased patch of v11 up to the current master.

I've been studying this patch.

SELECT FOR UPDATE on the inheritance parent fails with a can't-happen error
condition, even when SELECT FOR UPDATE on the child foreign table alone would
have succeeded.

The patch adds zero tests.  Add principal tests to postgres_fdw.sql.  Also,
create a child foreign table in foreign_data.sql; this will make dump/reload
tests of the regression database exercise an inheritance tree that includes a
foreign table.

The inheritance section of ddl.sgml should mention child foreign tables, at
least briefly.  Consider mentioning it in the partitioning section, too.


Your chosen ANALYZE behavior is fair, but the messaging from a database-wide
ANALYZE VERBOSE needs work:

INFO:  analyzing "test_foreign_inherit.parent"
INFO:  "parent": scanned 1 of 1 pages, containing 1 live rows and 0 dead rows; 1 rows in sample, 1 estimated total rows
INFO:  analyzing "test_foreign_inherit.parent" inheritance tree
WARNING:  relcache reference leak: relation "child" not closed
WARNING:  relcache reference leak: relation "tchild" not closed
WARNING:  relcache reference leak: relation "parent" not closed

Please arrange to omit the 'analyzing "tablename" inheritance tree' message,
since this ANALYZE actually skipped that task.  (The warnings obviously need a
fix, too.)  I do find it awkward that adding a foreign table to an inheritance
tree will make autovacuum stop collecting statistics on that inheritance tree,
but I can't think of a better policy.


The rest of these review comments are strictly observations; they're not
requests for you to change the patch, but they may deserve more discussion.

We use the term "child table" in many messages.  Should that change to
something more inclusive, now that the child may be a foreign table?  Perhaps
one of "child relation", plain "child", or "child foreign table"/"child table"
depending on the actual object?  "child relation" is robust technically, but
it might add more confusion than it removes.  Varying the message depending on
the actual object feels like a waste.  Opinions?

LOCK TABLE on the inheritance parent locks child foreign tables, but LOCK
TABLE fails when given a foreign table directly.  That's odd, but I see no
cause to change it.

A partition root only accepts an UPDATE command if every child is updatable.
Similarly, "UPDATE ... WHERE CURRENT OF cursor_name" fails if any child does
not support it.  That seems fine.  Incidentally, does anyone have a FDW that
supports WHERE CURRENT OF?


The longstanding behavior of CREATE TABLE INHERITS is to reorder local columns
to match the order found in parents.  That is, both of these tables actually
have columns in the order (a,b):

create table parent (a int, b int);
create table child (b int, a int) inherits (parent);

Ordinary dump/reload always uses CREATE TABLE INHERITS, thereby changing
column order in this way.  (pg_dump --binary-upgrade uses ALTER TABLE INHERIT
and some catalog hacks to avoid doing so.)  I've never liked that dump/reload
can change column order, but it's tolerable if you follow the best practice of
always writing out column lists.  The stakes rise for foreign tables, because
column order is inherently significant to file_fdw and probably to certain
other non-RDBMS FDWs.  If pg_dump changes column order in your file_fdw
foreign table, the table breaks.  I would heartily support making pg_dump
preserve column order for all inheritance children.  That doesn't rise to the
level of being a blocker for this patch, though.


I am attaching rough-hewn tests I used during this review.

Thanks,
nm

--
Noah Misch
EnterpriseDB                                 http://www.enterprisedb.com

Attachment

Re: inherit support for foreign tables

From
Shigeru Hanada
Date:
Hi Fujita-san,

Sorry for leaving this thread for long time.

2014-06-24 16:30 GMT+09:00 Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>:
> (2014/06/23 18:35), Ashutosh Bapat wrote:
>>
>> Hi,
>> Selecting tableoid on parent causes an error, "ERROR:  cannot extract
>> system attribute from virtual tuple". The foreign table has an OID which
>> can be reported as tableoid for the rows coming from that foreign table.
>> Do we want to do that?
>
>
> No.  I think it's a bug.  I'll fix it.

IIUC, you mean that tableoid can't be retrieved when a foreign table
is accessed via parent table, but it sounds strange to me, because one
of main purposes of tableoid is determine actual source table in
appended results.

Am I missing the point?

-- 
Shigeru HANADA



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/07/10 18:12), Shigeru Hanada wrote:
> 2014-06-24 16:30 GMT+09:00 Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>:
>> (2014/06/23 18:35), Ashutosh Bapat wrote:

>>> Selecting tableoid on parent causes an error, "ERROR:  cannot extract
>>> system attribute from virtual tuple". The foreign table has an OID which
>>> can be reported as tableoid for the rows coming from that foreign table.
>>> Do we want to do that?

>> No.  I think it's a bug.  I'll fix it.

> IIUC, you mean that tableoid can't be retrieved when a foreign table
> is accessed via parent table,

No.  What I want to say is that tableoid *can* be retrieved when a 
foreign table is accessed via the parent table.

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/07/11 15:50), Etsuro Fujita wrote:
> (2014/07/10 18:12), Shigeru Hanada wrote:

>> IIUC, you mean that tableoid can't be retrieved when a foreign table
>> is accessed via parent table,
>
> No.  What I want to say is that tableoid *can* be retrieved when a
> foreign table is accessed via the parent table.

In fact, you can do that with v13 [1], but I plan to change the way of 
fixing (see [2]).

Thanks,

[1] http://www.postgresql.org/message-id/53B10914.2010504@lab.ntt.co.jp
[2] http://www.postgresql.org/message-id/53B2188B.4090302@lab.ntt.co.jp

Best regards,
Etsuro Fujita



[BUG?] tuples from file_fdw has strange xids.

From
Kyotaro HORIGUCHI
Date:
Hello, I found that tuples come from file_fdw has strange xmin and xmax.

> postgres=# select tableoid, xmin, xmax, * from passwd1;
> tableoid | xmin |    xmax    |     uname     | pass |  uid  |  gid  | .. 
>    16396 |  244 | 4294967295 | root          | x    |     0 |     0 | root...
>    16396 |  252 | 4294967295 | bin           | x    |     1 |     1 | bin...
>    16396 |  284 | 4294967295 | daemon        | x    |     2 |     2 | daemon...

Back to 9.1 gives the same result.

These xmin and xmax are the simple interpretations of a
DatumTupleFields filled by ExecMaterializedSlot() beeing fed the
source virtual tuple slot from file_fdw. On the other hand,
postgres_fdw gives sane xids (xmin = 2, xmax = 0).

This is because ForeignNext returns physical tuples in which
their headers are DatumTupleFields regardless whether the system
columns are requested or not. The fdw machinary desn't seem to
provide fdw handlers with a means to reject requests for
unavailable system columns, so the tuple header should be fixed
with the sane values as HeapTupleFields.

The patch attached fixes the header of materialized tuple to be
sane (2, 0) if the source slot was a virtual tuple in mechanism().


regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center
diff --git a/src/backend/executor/nodeForeignscan.c b/src/backend/executor/nodeForeignscan.c
index 9cc5345..59dc5f4 100644
--- a/src/backend/executor/nodeForeignscan.c
+++ b/src/backend/executor/nodeForeignscan.c
@@ -22,6 +22,8 @@ */#include "postgres.h"
+#include "access/transam.h"
+#include "access/htup_details.h"#include "executor/executor.h"#include "executor/nodeForeignscan.h"#include
"foreign/fdwapi.h"
@@ -58,8 +60,21 @@ ForeignNext(ForeignScanState *node)     */    if (plan->fsSystemCol && !TupIsNull(slot))    {
+        bool        was_virtual_tuple = (slot->tts_tuple == NULL);        HeapTuple    tup =
ExecMaterializeSlot(slot);
+        if (was_virtual_tuple)
+        {
+            /*
+             * ExecMaterializeSlot fills the tuple header as a
+             * DatumTupleFields if the slot was a virtual tuple, although a
+             * physical one is needed by the callers. So rewrite the tuple
+             * header as a sane HeapTupleFields.
+             */
+            HeapTupleHeaderSetXmin(tup->t_data, FrozenTransactionId);
+            HeapTupleHeaderSetXmax(tup->t_data, InvalidTransactionId);
+            HeapTupleHeaderSetCmin(tup->t_data, FirstCommandId);
+        }        tup->t_tableOid = RelationGetRelid(node->ss.ss_currentRelation);    }

Re: [BUG?] tuples from file_fdw has strange xids.

From
Tom Lane
Date:
Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> writes:
> Hello, I found that tuples come from file_fdw has strange xmin and xmax.

file_fdw isn't documented to return anything useful for xmin/xmax/etc,
so I don't find this surprising.

> The patch attached fixes the header of materialized tuple to be
> sane (2, 0) if the source slot was a virtual tuple in mechanism().

I don't really think it's ForeignNext's place to be doing something
about this.  It seems like a useless expenditure of cycles.  Besides,
this fails to cover e.g. postgres_fdw, which is likewise unconcerned
about what it returns for system columns other than ctid.
        regards, tom lane



Re: [BUG?] tuples from file_fdw has strange xids.

From
Kyotaro HORIGUCHI
Date:
Hello, sorry, it's my bad.

> Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> writes:
> > Hello, I found that tuples come from file_fdw has strange xmin and xmax.
> 
> file_fdw isn't documented to return anything useful for xmin/xmax/etc,
> so I don't find this surprising.
> 
> > The patch attached fixes the header of materialized tuple to be
> > sane (2, 0) if the source slot was a virtual tuple in mechanism().
> 
> I don't really think it's ForeignNext's place to be doing something
> about this.  It seems like a useless expenditure of cycles.  Besides,
> this fails to cover e.g. postgres_fdw, which is likewise unconcerned
> about what it returns for system columns other than ctid.

I somehow misunderstood that postgres_fdw returns (xmin, xmax) =
(2, 0) but I confirmed that xmin, xmax and citd seems insane.

I completely agree with you.

Thank you for giving response for the stupid story.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/07/01 11:10), Etsuro Fujita wrote:
> (2014/06/30 22:48), Tom Lane wrote:
>> Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
>>> Done.  I think this is because create_foreignscan_plan() makes reference
>>> to attr_needed, which isn't computed for inheritance children.
>>
>> I wonder whether it isn't time to change that.  It was coded like that
>> originally only because calculating the values would've been a waste of
>> cycles at the time.  But this is at least the third place where it'd be
>> useful to have attr_needed for child rels.
>
> +1 for calculating attr_needed for child rels.  (I was wondering too.)
>
> I'll create a separate patch for it.

Attached is a WIP patch for that.  The following functions have been
changed to refer to attr_needed:

* check_index_only()
* remove_unused_subquery_outputs()
* check_selective_binary_conversion()

I'll add this to the upcoming commitfest.  If anyone has any time to
glance at it before then, that would be a great help.

Thanks,

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/08/06 20:43), Etsuro Fujita wrote:
>> (2014/06/30 22:48), Tom Lane wrote:
>>> I wonder whether it isn't time to change that.  It was coded like that
>>> originally only because calculating the values would've been a waste of
>>> cycles at the time.  But this is at least the third place where it'd be
>>> useful to have attr_needed for child rels.

> Attached is a WIP patch for that.

I've revised the patch.

Changes:

* Make the code more readable
* Revise the comments
* Cleanup

Please find attached an updated version of the patch.

Thanks,

Best regards,
Etsuro Fujita

Attachment
(2014/08/08 18:51), Etsuro Fujita wrote:
>>> (2014/06/30 22:48), Tom Lane wrote:
>>>> I wonder whether it isn't time to change that.  It was coded like that
>>>> originally only because calculating the values would've been a waste of
>>>> cycles at the time.  But this is at least the third place where it'd be
>>>> useful to have attr_needed for child rels.

> I've revised the patch.

There was a problem with the previous patch, which will be described
below.  Attached is the updated version of the patch addressing that.

The previous patch doesn't cope with some UNION ALL cases properly.  So,
e.g., the server will crash for the following query:

postgres=# create table ta1 (f1 int);
CREATE TABLE
postgres=# create table ta2 (f2 int primary key, f3 int);
CREATE TABLE
postgres=# create table tb1 (f1 int);
CREATE TABLE
postgres=# create table tb2 (f2 int primary key, f3 int);
CREATE TABLE
postgres=# explain verbose select f1 from ((select f1, f2 from (select
f1, f2, f3 from ta1 left join ta2 on f1 = f2 limit 1) ssa) union all
(select f1,
f2 from (select f1, f2, f3 from tb1 left join tb2 on f1 = f2 limit 1)
ssb)) ss;

With the updated version, we get the right result:

postgres=# explain verbose select f1 from ((select f1, f2 from (select
f1, f2, f3 from ta1 left join ta2 on f1 = f2 limit 1) ssa) union all
(select f1,
f2 from (select f1, f2, f3 from tb1 left join tb2 on f1 = f2 limit 1)
ssb)) ss;
                                   QUERY PLAN
--------------------------------------------------------------------------------
 Append  (cost=0.00..0.05 rows=2 width=4)
   ->  Subquery Scan on ssa  (cost=0.00..0.02 rows=1 width=4)
         Output: ssa.f1
         ->  Limit  (cost=0.00..0.01 rows=1 width=4)
               Output: ta1.f1, (NULL::integer), (NULL::integer)
               ->  Seq Scan on public.ta1  (cost=0.00..34.00 rows=2400
width=4)
                     Output: ta1.f1, NULL::integer, NULL::integer
   ->  Subquery Scan on ssb  (cost=0.00..0.02 rows=1 width=4)
         Output: ssb.f1
         ->  Limit  (cost=0.00..0.01 rows=1 width=4)
               Output: tb1.f1, (NULL::integer), (NULL::integer)
               ->  Seq Scan on public.tb1  (cost=0.00..34.00 rows=2400
width=4)
                     Output: tb1.f1, NULL::integer, NULL::integer
 Planning time: 0.453 ms
(14 rows)

While thinking to address this problem, Ashutosh also expressed concern
about the UNION ALL handling in the previous patch in a private email.
Thank you for the review, Ashutosh!

Thanks,

Best regards,
Etsuro Fujita

Attachment
Hi,



On Thu, Aug 14, 2014 at 10:05 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/08/08 18:51), Etsuro Fujita wrote:
>>> (2014/06/30 22:48), Tom Lane wrote:
>>>> I wonder whether it isn't time to change that.  It was coded like that
>>>> originally only because calculating the values would've been a waste of
>>>> cycles at the time.  But this is at least the third place where it'd be
>>>> useful to have attr_needed for child rels.

> I've revised the patch.

There was a problem with the previous patch, which will be described
below.  Attached is the updated version of the patch addressing that.

The previous patch doesn't cope with some UNION ALL cases properly.  So,
e.g., the server will crash for the following query:

postgres=# create table ta1 (f1 int);
CREATE TABLE
postgres=# create table ta2 (f2 int primary key, f3 int);
CREATE TABLE
postgres=# create table tb1 (f1 int);
CREATE TABLE
postgres=# create table tb2 (f2 int primary key, f3 int);
CREATE TABLE
postgres=# explain verbose select f1 from ((select f1, f2 from (select
f1, f2, f3 from ta1 left join ta2 on f1 = f2 limit 1) ssa) union all
(select f1,
f2 from (select f1, f2, f3 from tb1 left join tb2 on f1 = f2 limit 1)
ssb)) ss;

With the updated version, we get the right result:

postgres=# explain verbose select f1 from ((select f1, f2 from (select
f1, f2, f3 from ta1 left join ta2 on f1 = f2 limit 1) ssa) union all
(select f1,
f2 from (select f1, f2, f3 from tb1 left join tb2 on f1 = f2 limit 1)
ssb)) ss;
                                   QUERY PLAN
--------------------------------------------------------------------------------
 Append  (cost=0.00..0.05 rows=2 width=4)
   ->  Subquery Scan on ssa  (cost=0.00..0.02 rows=1 width=4)
         Output: ssa.f1
         ->  Limit  (cost=0.00..0.01 rows=1 width=4)
               Output: ta1.f1, (NULL::integer), (NULL::integer)
               ->  Seq Scan on public.ta1  (cost=0.00..34.00 rows=2400
width=4)
                     Output: ta1.f1, NULL::integer, NULL::integer
   ->  Subquery Scan on ssb  (cost=0.00..0.02 rows=1 width=4)
         Output: ssb.f1
         ->  Limit  (cost=0.00..0.01 rows=1 width=4)
               Output: tb1.f1, (NULL::integer), (NULL::integer)
               ->  Seq Scan on public.tb1  (cost=0.00..34.00 rows=2400
width=4)
                     Output: tb1.f1, NULL::integer, NULL::integer
 Planning time: 0.453 ms
(14 rows)

While thinking to address this problem, Ashutosh also expressed concern
about the UNION ALL handling in the previous patch in a private email.
Thank you for the review, Ashutosh!


Thanks for taking care of this one.

Here are some more comments

attr_needed also has the attributes used in the restriction clauses as seen in distribute_qual_to_rels(), so, it looks unnecessary to call pull_varattnos() on the clauses in baserestrictinfo in functions check_selective_binary_conversion(), remove_unused_subquery_outputs(), check_index_only().

Although in case of RTE_RELATION, the 0th entry in attr_needed corresponds to FirstLowInvalidHeapAttributeNumber + 1, it's always safer to use it is RelOptInfo::min_attr, in case get_relation_info() wants to change assumption or somewhere down the line some other part of code wants to change attr_needed information. It may be unlikely, that it would change, but it will be better to stick to comments in RelOptInfo
 443     AttrNumber  min_attr;       /* smallest attrno of rel (often <0) */
 444     AttrNumber  max_attr;       /* largest attrno of rel */
 445     Relids     *attr_needed;    /* array indexed [min_attr .. max_attr] */
 
Thanks,

Best regards,
Etsuro Fujita



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Hi Ashutish,

(2014/08/14 22:30), Ashutosh Bapat wrote:
> On Thu, Aug 14, 2014 at 10:05 AM, Etsuro Fujita
> <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:
>
>     (2014/08/08 18:51), Etsuro Fujita wrote:
>      >>> (2014/06/30 22:48), Tom Lane wrote:
>      >>>> I wonder whether it isn't time to change that.  It was coded
>     like that
>      >>>> originally only because calculating the values would've been a
>     waste of
>      >>>> cycles at the time.  But this is at least the third place
>     where it'd be
>      >>>> useful to have attr_needed for child rels.
>
>      > I've revised the patch.
>
>     There was a problem with the previous patch, which will be described
>     below.  Attached is the updated version of the patch addressing that.

> Here are some more comments

Thank you for the further review!

> attr_needed also has the attributes used in the restriction clauses as
> seen in distribute_qual_to_rels(), so, it looks unnecessary to call
> pull_varattnos() on the clauses in baserestrictinfo in functions
> check_selective_binary_conversion(), remove_unused_subquery_outputs(),
> check_index_only().

IIUC, I think it's *necessary* to do that in those functions since the
attributes used in the restriction clauses in baserestrictinfo are not
added to attr_needed due the following code in distribute_qual_to_rels.

     /*
      * If it's a join clause (either naturally, or because delayed by
      * outer-join rules), add vars used in the clause to targetlists of
their
      * relations, so that they will be emitted by the plan nodes that scan
      * those relations (else they won't be available at the join node!).
      *
      * Note: if the clause gets absorbed into an EquivalenceClass then this
      * may be unnecessary, but for now we have to do it to cover the case
      * where the EC becomes ec_broken and we end up reinserting the
original
      * clauses into the plan.
      */
     if (bms_membership(relids) == BMS_MULTIPLE)
     {
         List       *vars = pull_var_clause(clause,
                                            PVC_RECURSE_AGGREGATES,
                                            PVC_INCLUDE_PLACEHOLDERS);

         add_vars_to_targetlist(root, vars, relids, false);
         list_free(vars);
     }

> Although in case of RTE_RELATION, the 0th entry in attr_needed
> corresponds to FirstLowInvalidHeapAttributeNumber + 1, it's always safer
> to use it is RelOptInfo::min_attr, in case get_relation_info() wants to
> change assumption or somewhere down the line some other part of code
> wants to change attr_needed information. It may be unlikely, that it
> would change, but it will be better to stick to comments in RelOptInfo
>   443     AttrNumber  min_attr;       /* smallest attrno of rel (often
> <0) */
>   444     AttrNumber  max_attr;       /* largest attrno of rel */
>   445     Relids     *attr_needed;    /* array indexed [min_attr ..
> max_attr] */

Good point!  Attached is the revised version of the patch.

Thanks,

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
Hi Noah,

Thank you for the review!

(2014/07/02 11:23), Noah Misch wrote:
> On Fri, Jun 20, 2014 at 05:04:06PM +0900, Kyotaro HORIGUCHI wrote:
>> Attached is the rebased patch of v11 up to the current master.
>
> I've been studying this patch.
>
> SELECT FOR UPDATE on the inheritance parent fails with a can't-happen error
> condition, even when SELECT FOR UPDATE on the child foreign table alone would
> have succeeded.

To fix this, I've modified the planner and executor so that the planner
adds wholerow as well as ctid and tableoid as resjunk output columns to
the plan for an inheritance tree that contains foreign tables, and that
while the executor uses the ctid and tableoid in the EPQ processing for
child regular tables as before, the executor uses the wholerow and
tableoid for child foreign tables.  Patch attached.  This is based on
the patch [1].

> The patch adds zero tests.  Add principal tests to postgres_fdw.sql.  Also,
> create a child foreign table in foreign_data.sql; this will make dump/reload
> tests of the regression database exercise an inheritance tree that includes a
> foreign table.

Done.  I used your tests as reference.  Thanks!

> The inheritance section of ddl.sgml should mention child foreign tables, at
> least briefly.  Consider mentioning it in the partitioning section, too.

Done.

> Your chosen ANALYZE behavior is fair, but the messaging from a database-wide
> ANALYZE VERBOSE needs work:
>
> INFO:  analyzing "test_foreign_inherit.parent"
> INFO:  "parent": scanned 1 of 1 pages, containing 1 live rows and 0 dead rows; 1 rows in sample, 1 estimated total
rows
> INFO:  analyzing "test_foreign_inherit.parent" inheritance tree
> WARNING:  relcache reference leak: relation "child" not closed
> WARNING:  relcache reference leak: relation "tchild" not closed
> WARNING:  relcache reference leak: relation "parent" not closed
>
> Please arrange to omit the 'analyzing "tablename" inheritance tree' message,
> since this ANALYZE actually skipped that task.  (The warnings obviously need a
> fix, too.)  I do find it awkward that adding a foreign table to an inheritance
> tree will make autovacuum stop collecting statistics on that inheritance tree,
> but I can't think of a better policy.

I think it would be better that this is handled in the same way as an
inheritance tree that turns out to be a singe table that doesn't have
any descendants in acquire_inherited_sample_rows().  That would still
output the message as shown below, but I think that that would be more
consistent with the existing code.  The patch works so.  (The warnings
are also fixed.)

INFO:  analyzing "public.parent"
INFO:  "parent": scanned 0 of 0 pages, containing 0 live rows and 0 dead
rows; 0 rows in sample, 0 estimated total rows
INFO:  analyzing "public.parent" inheritance tree
INFO:  analyzing "pg_catalog.pg_authid"
INFO:  "pg_authid": scanned 1 of 1 pages, containing 1 live rows and 0
dead rows; 1 rows in sample, 1 estimated total rows

> The rest of these review comments are strictly observations; they're not
> requests for you to change the patch, but they may deserve more discussion.

I'd like to give my opinions on those things later on.

Sorry for the long delay.

[1] http://www.postgresql.org/message-id/53F4707C.8030904@lab.ntt.co.jp

Best regards,
Etsuro Fujita

Attachment



On Wed, Aug 20, 2014 at 3:25 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
Hi Ashutish,


(2014/08/14 22:30), Ashutosh Bapat wrote:
On Thu, Aug 14, 2014 at 10:05 AM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:

    (2014/08/08 18:51), Etsuro Fujita wrote:
     >>> (2014/06/30 22:48), Tom Lane wrote:
     >>>> I wonder whether it isn't time to change that.  It was coded
    like that
     >>>> originally only because calculating the values would've been a
    waste of
     >>>> cycles at the time.  But this is at least the third place
    where it'd be
     >>>> useful to have attr_needed for child rels.

     > I've revised the patch.

    There was a problem with the previous patch, which will be described
    below.  Attached is the updated version of the patch addressing that.

Here are some more comments

Thank you for the further review!


attr_needed also has the attributes used in the restriction clauses as
seen in distribute_qual_to_rels(), so, it looks unnecessary to call
pull_varattnos() on the clauses in baserestrictinfo in functions
check_selective_binary_conversion(), remove_unused_subquery_outputs(),
check_index_only().

IIUC, I think it's *necessary* to do that in those functions since the attributes used in the restriction clauses in baserestrictinfo are not added to attr_needed due the following code in distribute_qual_to_rels.


That's right. Thanks for pointing that out.
 
    /*
     * If it's a join clause (either naturally, or because delayed by
     * outer-join rules), add vars used in the clause to targetlists of their
     * relations, so that they will be emitted by the plan nodes that scan
     * those relations (else they won't be available at the join node!).
     *
     * Note: if the clause gets absorbed into an EquivalenceClass then this
     * may be unnecessary, but for now we have to do it to cover the case
     * where the EC becomes ec_broken and we end up reinserting the original
     * clauses into the plan.
     */
    if (bms_membership(relids) == BMS_MULTIPLE)
    {
        List       *vars = pull_var_clause(clause,
                                           PVC_RECURSE_AGGREGATES,
                                           PVC_INCLUDE_PLACEHOLDERS);

        add_vars_to_targetlist(root, vars, relids, false);
        list_free(vars);

    }

Although in case of RTE_RELATION, the 0th entry in attr_needed
corresponds to FirstLowInvalidHeapAttributeNumber + 1, it's always safer
to use it is RelOptInfo::min_attr, in case get_relation_info() wants to
change assumption or somewhere down the line some other part of code
wants to change attr_needed information. It may be unlikely, that it
would change, but it will be better to stick to comments in RelOptInfo
  443     AttrNumber  min_attr;       /* smallest attrno of rel (often
<0) */
  444     AttrNumber  max_attr;       /* largest attrno of rel */
  445     Relids     *attr_needed;    /* array indexed [min_attr ..
max_attr] */

Good point!  Attached is the revised version of the patch.


If the patch is not in the commit-fest, can you please add it there? From my side, the review is done, it should be marked "ready for committer", unless somebody else wants to review.
 

Thanks,

Best regards,
Etsuro Fujita



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
(2014/08/21 13:21), Ashutosh Bapat wrote:
> On Wed, Aug 20, 2014 at 3:25 PM, Etsuro Fujita
> <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:

>     Hi Ashutish,

I am sorry that I mistook your name's spelling.

>     (2014/08/14 22:30), Ashutosh Bapat wrote:
>
>         On Thu, Aug 14, 2014 at 10:05 AM, Etsuro Fujita
>         <fujita.etsuro@lab.ntt.co.jp
>         <mailto:fujita.etsuro@lab.ntt.co.jp>
>         <mailto:fujita.etsuro@lab.ntt.__co.jp
>         <mailto:fujita.etsuro@lab.ntt.co.jp>>> wrote:
>
>              (2014/08/08 18:51), Etsuro Fujita wrote:
>               >>> (2014/06/30 22:48), Tom Lane wrote:
>               >>>> I wonder whether it isn't time to change that.  It
>         was coded
>              like that
>               >>>> originally only because calculating the values
>         would've been a
>              waste of
>               >>>> cycles at the time.  But this is at least the third place
>              where it'd be
>               >>>> useful to have attr_needed for child rels.

>              There was a problem with the previous patch, which will be
>         described
>              below.  Attached is the updated version of the patch
>         addressing that.

>         Here are some more comments

>         attr_needed also has the attributes used in the restriction
>         clauses as
>         seen in distribute_qual_to_rels(), so, it looks unnecessary to call
>         pull_varattnos() on the clauses in baserestrictinfo in functions
>         check_selective_binary___conversion(),
>         remove_unused_subquery___outputs(),
>         check_index_only().

>     IIUC, I think it's *necessary* to do that in those functions since
>     the attributes used in the restriction clauses in baserestrictinfo
>     are not added to attr_needed due the following code in
>     distribute_qual_to_rels.

> That's right. Thanks for pointing that out.

>         Although in case of RTE_RELATION, the 0th entry in attr_needed
>         corresponds to FirstLowInvalidHeapAttributeNu__mber + 1, it's
>         always safer
>         to use it is RelOptInfo::min_attr, in case get_relation_info()
>         wants to
>         change assumption or somewhere down the line some other part of code
>         wants to change attr_needed information. It may be unlikely, that it
>         would change, but it will be better to stick to comments in
>         RelOptInfo
>            443     AttrNumber  min_attr;       /* smallest attrno of rel
>         (often
>         <0) */
>            444     AttrNumber  max_attr;       /* largest attrno of rel */
>            445     Relids     *attr_needed;    /* array indexed [min_attr ..
>         max_attr] */

>     Good point!  Attached is the revised version of the patch.

> If the patch is not in the commit-fest, can you please add it there?

I've already done that:

https://commitfest.postgresql.org/action/patch_view?id=1529

>  From my side, the review is done, it should be marked "ready for
> committer", unless somebody else wants to review.

Many thanks!

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/07/02 11:23), Noah Misch wrote:
> On Fri, Jun 20, 2014 at 05:04:06PM +0900, Kyotaro HORIGUCHI wrote:
>> Attached is the rebased patch of v11 up to the current master.

> The rest of these review comments are strictly observations; they're not
> requests for you to change the patch, but they may deserve more discussion.
>
> We use the term "child table" in many messages.  Should that change to
> something more inclusive, now that the child may be a foreign table?  Perhaps
> one of "child relation", plain "child", or "child foreign table"/"child table"
> depending on the actual object?  "child relation" is robust technically, but
> it might add more confusion than it removes.  Varying the message depending on
> the actual object feels like a waste.  Opinions?

IMHO, I think that "child table" would not confusing users terribly.

> LOCK TABLE on the inheritance parent locks child foreign tables, but LOCK
> TABLE fails when given a foreign table directly.  That's odd, but I see no
> cause to change it.

I agree wth that.

> The longstanding behavior of CREATE TABLE INHERITS is to reorder local columns
> to match the order found in parents.  That is, both of these tables actually
> have columns in the order (a,b):
>
> create table parent (a int, b int);
> create table child (b int, a int) inherits (parent);
>
> Ordinary dump/reload always uses CREATE TABLE INHERITS, thereby changing
> column order in this way.  (pg_dump --binary-upgrade uses ALTER TABLE INHERIT
> and some catalog hacks to avoid doing so.)  I've never liked that dump/reload
> can change column order, but it's tolerable if you follow the best practice of
> always writing out column lists.  The stakes rise for foreign tables, because
> column order is inherently significant to file_fdw and probably to certain
> other non-RDBMS FDWs.  If pg_dump changes column order in your file_fdw
> foreign table, the table breaks.  I would heartily support making pg_dump
> preserve column order for all inheritance children.  That doesn't rise to the
> level of being a blocker for this patch, though.

I agree with that, too.  I think it would be better to add docs for now.

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Noah Misch
Date:
On Wed, Aug 20, 2014 at 08:11:01PM +0900, Etsuro Fujita wrote:
> (2014/07/02 11:23), Noah Misch wrote:
> >Your chosen ANALYZE behavior is fair, but the messaging from a database-wide
> >ANALYZE VERBOSE needs work:
> >
> >INFO:  analyzing "test_foreign_inherit.parent"
> >INFO:  "parent": scanned 1 of 1 pages, containing 1 live rows and 0 dead rows; 1 rows in sample, 1 estimated total
rows
> >INFO:  analyzing "test_foreign_inherit.parent" inheritance tree
> >WARNING:  relcache reference leak: relation "child" not closed
> >WARNING:  relcache reference leak: relation "tchild" not closed
> >WARNING:  relcache reference leak: relation "parent" not closed
> >
> >Please arrange to omit the 'analyzing "tablename" inheritance tree' message,
> >since this ANALYZE actually skipped that task.  (The warnings obviously need a
> >fix, too.)  I do find it awkward that adding a foreign table to an inheritance
> >tree will make autovacuum stop collecting statistics on that inheritance tree,
> >but I can't think of a better policy.
> 
> I think it would be better that this is handled in the same way as
> an inheritance tree that turns out to be a singe table that doesn't
> have any descendants in acquire_inherited_sample_rows().  That would
> still output the message as shown below, but I think that that would
> be more consistent with the existing code.  The patch works so.
> (The warnings are also fixed.)
> 
> INFO:  analyzing "public.parent"
> INFO:  "parent": scanned 0 of 0 pages, containing 0 live rows and 0
> dead rows; 0 rows in sample, 0 estimated total rows
> INFO:  analyzing "public.parent" inheritance tree
> INFO:  analyzing "pg_catalog.pg_authid"
> INFO:  "pg_authid": scanned 1 of 1 pages, containing 1 live rows and
> 0 dead rows; 1 rows in sample, 1 estimated total rows

Today's ANALYZE VERBOSE messaging for former inheritance parents (tables with
relhassubclass = true but no pg_inherits.inhparent links) is deceptive, and I
welcome a fix to omit the spurious message.  As defects go, this is quite
minor.  There's fundamentally no value in collecting inheritance tree
statistics for such a parent, and no PostgreSQL command will do so.

Your proposed behavior for inheritance parents having at least one foreign
table child is more likely to mislead DBAs in practice.  An inheritance tree
genuinely exists, and a different ANALYZE command is quite capable of
collecting statistics on that inheritance tree.  Messaging should reflect the
difference between ANALYZE invocations that do such work and ANALYZE
invocations that skip it.  I'm anticipating a bug report along these lines:
 I saw poor estimates involving a child foreign table, so I ran "ANALYZE VERBOSE", which reported 'INFO:  analyzing
"public.parent"inheritance tree'.  Estimates remained poor, so I scratched my head for awhile before noticing that
pg_statsdidn't report such statistics.  I then ran "ANALYZE VERBOSE parent", saw the same 'INFO:  analyzing
"public.parent"inheritance tree', but this time saw relevant rows in pg_stats.  The message doesn't reflect the actual
behavior.

I'll sympathize with that complaint.  It's a minor point overall, but the code
impact is, I predict, small enough that we may as well get it right.  A
credible alternative is to emit a second message indicating that we skipped
the inheritance tree statistics after all, and why we skipped them.



Re: inherit support for foreign tables

From
Alvaro Herrera
Date:
Noah Misch wrote:

> I'm anticipating a bug report along these lines:
> 
>   I saw poor estimates involving a child foreign table, so I ran "ANALYZE
>   VERBOSE", which reported 'INFO:  analyzing "public.parent" inheritance
>   tree'.  Estimates remained poor, so I scratched my head for awhile before
>   noticing that pg_stats didn't report such statistics.  I then ran "ANALYZE
>   VERBOSE parent", saw the same 'INFO:  analyzing "public.parent" inheritance
>   tree', but this time saw relevant rows in pg_stats.  The message doesn't
>   reflect the actual behavior.
> 
> I'll sympathize with that complaint.

Agreed on that.

> A credible alternative is to emit a second message indicating that we
> skipped the inheritance tree statistics after all, and why we skipped
> them.

That'd be similar to Xorg emitting messages such as

[    53.772] (II) intel(0): RandR 1.2 enabled, ignore the following RandR disabled message.
[    53.800] (--) RandR disabled

I find this very poor.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services






On Thu, Aug 21, 2014 at 3:00 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/08/21 13:21), Ashutosh Bapat wrote:
On Wed, Aug 20, 2014 at 3:25 PM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:

    Hi Ashutish,

I am sorry that I mistook your name's spelling.

    (2014/08/14 22:30), Ashutosh Bapat wrote:

        On Thu, Aug 14, 2014 at 10:05 AM, Etsuro Fujita
        <fujita.etsuro@lab.ntt.co.jp
        <mailto:fujita.etsuro@lab.ntt.co.jp>
        <mailto:fujita.etsuro@lab.ntt.__co.jp

        <mailto:fujita.etsuro@lab.ntt.co.jp>>> wrote:

             (2014/08/08 18:51), Etsuro Fujita wrote:
              >>> (2014/06/30 22:48), Tom Lane wrote:
              >>>> I wonder whether it isn't time to change that.  It
        was coded
             like that
              >>>> originally only because calculating the values
        would've been a
             waste of
              >>>> cycles at the time.  But this is at least the third place
             where it'd be
              >>>> useful to have attr_needed for child rels.

             There was a problem with the previous patch, which will be
        described
             below.  Attached is the updated version of the patch
        addressing that.

        Here are some more comments

        attr_needed also has the attributes used in the restriction
        clauses as
        seen in distribute_qual_to_rels(), so, it looks unnecessary to call
        pull_varattnos() on the clauses in baserestrictinfo in functions
        check_selective_binary___conversion(),
        remove_unused_subquery___outputs(),
        check_index_only().

    IIUC, I think it's *necessary* to do that in those functions since
    the attributes used in the restriction clauses in baserestrictinfo
    are not added to attr_needed due the following code in
    distribute_qual_to_rels.

That's right. Thanks for pointing that out.

        Although in case of RTE_RELATION, the 0th entry in attr_needed
        corresponds to FirstLowInvalidHeapAttributeNu__mber + 1, it's

        always safer
        to use it is RelOptInfo::min_attr, in case get_relation_info()
        wants to
        change assumption or somewhere down the line some other part of code
        wants to change attr_needed information. It may be unlikely, that it
        would change, but it will be better to stick to comments in
        RelOptInfo
           443     AttrNumber  min_attr;       /* smallest attrno of rel
        (often
        <0) */
           444     AttrNumber  max_attr;       /* largest attrno of rel */
           445     Relids     *attr_needed;    /* array indexed [min_attr ..
        max_attr] */

    Good point!  Attached is the revised version of the patch.

If the patch is not in the commit-fest, can you please add it there?

I've already done that:

https://commitfest.postgresql.org/action/patch_view?id=1529


 From my side, the review is done, it should be marked "ready for
committer", unless somebody else wants to review.

Many thanks!


Thanks. Since, I haven't seen anybody else commenting here and I do not have any further comments to make, I have marked it as "ready for committer".
 
Best regards,
Etsuro Fujita



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/08/22 12:58), Alvaro Herrera wrote:
> Noah Misch wrote:
>
>> I'm anticipating a bug report along these lines:
>>
>>    I saw poor estimates involving a child foreign table, so I ran "ANALYZE
>>    VERBOSE", which reported 'INFO:  analyzing "public.parent" inheritance
>>    tree'.  Estimates remained poor, so I scratched my head for awhile before
>>    noticing that pg_stats didn't report such statistics.  I then ran "ANALYZE
>>    VERBOSE parent", saw the same 'INFO:  analyzing "public.parent" inheritance
>>    tree', but this time saw relevant rows in pg_stats.  The message doesn't
>>    reflect the actual behavior.
>>
>> I'll sympathize with that complaint.
>
> Agreed on that.

I've got the point.  Will fix.

Thanks for the comment!

Best regards,
Etsuro Fujita



Hi Fujita-san,

I reviewed the v4 patch, and here are some comments from me.

* After applying this patch, pull_varattnos() should not called in
unnecessary places.  For developers who want list of
columns-to-be-processed for some purpose, it would be nice to mention
when they should use pull_varattnos() and when they should not, maybe
at the comments of pull_varattnos() implementation.

* deparseReturningList() and postgresGetForeignRelSize() in
contrib/postgres_fdw/ also call pull_varattnos() to determine which
column to be in the SELECT clause of remote query.  Shouldn't these be
replaced in the same manner?  Other pull_varattnos() calls are for
restrictions, so IIUC they can't be replaced.

* Through this review I thought up that lazy evaluation approach might
fit attr_needed.  I mean that we leave attr_needed for child relations
empty, and fill it at the first request for it.  This would avoid
useless computation of attr_needed for child relations, though this
idea has been considered but thrown away before...


2014-08-20 18:55 GMT+09:00 Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>:
> Hi Ashutish,
>
>
> (2014/08/14 22:30), Ashutosh Bapat wrote:
>>
>> On Thu, Aug 14, 2014 at 10:05 AM, Etsuro Fujita
>> <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:
>>
>>     (2014/08/08 18:51), Etsuro Fujita wrote:
>>      >>> (2014/06/30 22:48), Tom Lane wrote:
>>      >>>> I wonder whether it isn't time to change that.  It was coded
>>     like that
>>      >>>> originally only because calculating the values would've been a
>>     waste of
>>      >>>> cycles at the time.  But this is at least the third place
>>     where it'd be
>>      >>>> useful to have attr_needed for child rels.
>>
>>      > I've revised the patch.
>>
>>     There was a problem with the previous patch, which will be described
>>     below.  Attached is the updated version of the patch addressing that.
>
>
>> Here are some more comments
>
>
> Thank you for the further review!
>
>
>> attr_needed also has the attributes used in the restriction clauses as
>> seen in distribute_qual_to_rels(), so, it looks unnecessary to call
>> pull_varattnos() on the clauses in baserestrictinfo in functions
>> check_selective_binary_conversion(), remove_unused_subquery_outputs(),
>> check_index_only().
>
>
> IIUC, I think it's *necessary* to do that in those functions since the
> attributes used in the restriction clauses in baserestrictinfo are not added
> to attr_needed due the following code in distribute_qual_to_rels.
>
>     /*
>      * If it's a join clause (either naturally, or because delayed by
>      * outer-join rules), add vars used in the clause to targetlists of
> their
>      * relations, so that they will be emitted by the plan nodes that scan
>      * those relations (else they won't be available at the join node!).
>      *
>      * Note: if the clause gets absorbed into an EquivalenceClass then this
>      * may be unnecessary, but for now we have to do it to cover the case
>      * where the EC becomes ec_broken and we end up reinserting the original
>      * clauses into the plan.
>      */
>     if (bms_membership(relids) == BMS_MULTIPLE)
>     {
>         List       *vars = pull_var_clause(clause,
>                                            PVC_RECURSE_AGGREGATES,
>                                            PVC_INCLUDE_PLACEHOLDERS);
>
>         add_vars_to_targetlist(root, vars, relids, false);
>         list_free(vars);
>
>     }
>
>> Although in case of RTE_RELATION, the 0th entry in attr_needed
>> corresponds to FirstLowInvalidHeapAttributeNumber + 1, it's always safer
>> to use it is RelOptInfo::min_attr, in case get_relation_info() wants to
>> change assumption or somewhere down the line some other part of code
>> wants to change attr_needed information. It may be unlikely, that it
>> would change, but it will be better to stick to comments in RelOptInfo
>>   443     AttrNumber  min_attr;       /* smallest attrno of rel (often
>> <0) */
>>   444     AttrNumber  max_attr;       /* largest attrno of rel */
>>   445     Relids     *attr_needed;    /* array indexed [min_attr ..
>> max_attr] */
>
>
> Good point!  Attached is the revised version of the patch.
>
>
> Thanks,
>
> Best regards,
> Etsuro Fujita



-- 
Shigeru HANADA



Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
> [ attr_needed-v4.patch ]

I looked this over, and TBH I'm rather disappointed.  The patch adds
150 lines of dubiously-correct code in order to save ... uh, well,
actually it *adds* code, because the places that are supposedly getting
a benefit are changed like this:

*** 799,806 **** check_selective_binary_conversion(RelOptInfo *baserel,     }      /* Collect all the attributes needed
forjoins or final output. */
 
!     pull_varattnos((Node *) baserel->reltargetlist, baserel->relid,
!                    &attrs_used);      /* Add all the attributes used by restriction clauses. */     foreach(lc,
baserel->baserestrictinfo)
--- 799,810 ----     }      /* Collect all the attributes needed for joins or final output. */
!     for (i = baserel->min_attr; i <= baserel->max_attr; i++)
!     {
!         if (!bms_is_empty(baserel->attr_needed[i - baserel->min_attr]))
!             attrs_used = bms_add_member(attrs_used,
!                                         i - FirstLowInvalidHeapAttributeNumber);
!     }      /* Add all the attributes used by restriction clauses. */     foreach(lc, baserel->baserestrictinfo)

That's not simpler, it's not easier to understand, and it's probably not
faster either.  We could address some of those complaints by encapsulating
the above loop into a utility function, but still, I come away with the
feeling that it's not worth changing this.  Considering that all the
places that are doing this then proceed to use pull_varattnos to add on
attnos from the restriction clauses, it seems like using pull_varattnos
on the reltargetlist isn't such a bad thing after all.

So I'm inclined to reject this.  It seemed like a good idea in the
abstract, but the concrete result isn't very attractive, and doesn't
seem like an improvement over what we have.
        regards, tom lane



(2014/08/27 3:27), Tom Lane wrote:
> Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
>> [ attr_needed-v4.patch ]
> 
> I looked this over, and TBH I'm rather disappointed.  The patch adds
> 150 lines of dubiously-correct code in order to save ... uh, well,

Just for my study, could you tell me why you think that the code is
"dubiously-correct"?

> Considering that all the
> places that are doing this then proceed to use pull_varattnos to add on
> attnos from the restriction clauses, it seems like using pull_varattnos
> on the reltargetlist isn't such a bad thing after all.

I agree with you on that point.

> So I'm inclined to reject this.  It seemed like a good idea in the
> abstract, but the concrete result isn't very attractive, and doesn't
> seem like an improvement over what we have.

Okay.  I'll withdraw the patch.

Thank you for taking the time to review the patch!

Best regards,
Etsuro Fujita



Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
> (2014/08/27 3:27), Tom Lane wrote:
>> I looked this over, and TBH I'm rather disappointed.  The patch adds
>> 150 lines of dubiously-correct code in order to save ... uh, well,

> Just for my study, could you tell me why you think that the code is
> "dubiously-correct"?

It might be fine; I did not actually review the new
adjust_appendrel_attr_needed code in any detail.  What's scaring me off it
is (1) it's a lot longer and more complicated than I'd thought it would
be, and (2) you already made several bug fixes in it, which is often an
indicator that additional problems lurk.

It's possible there's some other, simpler, way to compute child
attr_needed arrays that would resolve (1) and (2).  However, even if we
had a simple and obviously-correct way to do that, it still seems like
there's not very much benefit to be had after all.  So my thought that
this would be worth doing seems wrong, and I must apologize to you for
sending you chasing down a dead end :-(
        regards, tom lane



(2014/08/27 11:06), Tom Lane wrote:
> Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
>> (2014/08/27 3:27), Tom Lane wrote:
>>> I looked this over, and TBH I'm rather disappointed.  The patch adds
>>> 150 lines of dubiously-correct code in order to save ... uh, well,
> 
>> Just for my study, could you tell me why you think that the code is
>> "dubiously-correct"?
> 
> It might be fine; I did not actually review the new
> adjust_appendrel_attr_needed code in any detail.  What's scaring me off it
> is (1) it's a lot longer and more complicated than I'd thought it would
> be, and (2) you already made several bug fixes in it, which is often an
> indicator that additional problems lurk.

Okay.

> It's possible there's some other, simpler, way to compute child
> attr_needed arrays that would resolve (1) and (2).  However, even if we
> had a simple and obviously-correct way to do that, it still seems like
> there's not very much benefit to be had after all.  So my thought that
> this would be worth doing seems wrong, and I must apologize to you for
> sending you chasing down a dead end :-(

Please don't worry yourself about that!

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/08/22 11:51), Noah Misch wrote:
> On Wed, Aug 20, 2014 at 08:11:01PM +0900, Etsuro Fujita wrote:
>> (2014/07/02 11:23), Noah Misch wrote:
>>> Your chosen ANALYZE behavior is fair, but the messaging from a database-wide
>>> ANALYZE VERBOSE needs work:
>>>
>>> INFO:  analyzing "test_foreign_inherit.parent"
>>> INFO:  "parent": scanned 1 of 1 pages, containing 1 live rows and 0 dead rows; 1 rows in sample, 1 estimated total
rows
>>> INFO:  analyzing "test_foreign_inherit.parent" inheritance tree

>>> Please arrange to omit the 'analyzing "tablename" inheritance tree' message,
>>> since this ANALYZE actually skipped that task.

>> I think it would be better that this is handled in the same way as
>> an inheritance tree that turns out to be a singe table that doesn't
>> have any descendants in acquire_inherited_sample_rows().  That would
>> still output the message as shown below, but I think that that would
>> be more consistent with the existing code.  The patch works so.

> Today's ANALYZE VERBOSE messaging for former inheritance parents (tables with
> relhassubclass = true but no pg_inherits.inhparent links) is deceptive, and I
> welcome a fix to omit the spurious message.  As defects go, this is quite
> minor.  There's fundamentally no value in collecting inheritance tree
> statistics for such a parent, and no PostgreSQL command will do so.
>
> Your proposed behavior for inheritance parents having at least one foreign
> table child is more likely to mislead DBAs in practice.  An inheritance tree
> genuinely exists, and a different ANALYZE command is quite capable of
> collecting statistics on that inheritance tree.  Messaging should reflect the
> difference between ANALYZE invocations that do such work and ANALYZE
> invocations that skip it.  I'm anticipating a bug report along these lines:
>
>    I saw poor estimates involving a child foreign table, so I ran "ANALYZE
>    VERBOSE", which reported 'INFO:  analyzing "public.parent" inheritance
>    tree'.  Estimates remained poor, so I scratched my head for awhile before
>    noticing that pg_stats didn't report such statistics.  I then ran "ANALYZE
>    VERBOSE parent", saw the same 'INFO:  analyzing "public.parent" inheritance
>    tree', but this time saw relevant rows in pg_stats.  The message doesn't
>    reflect the actual behavior.
>
> I'll sympathize with that complaint.  It's a minor point overall, but the code
> impact is, I predict, small enough that we may as well get it right.  A
> credible alternative is to emit a second message indicating that we skipped
> the inheritance tree statistics after all, and why we skipped them.

I'd like to address this by emitting the second message as shown below:

INFO:  analyzing "public.parent"
INFO:  "parent": scanned 0 of 0 pages, containing 0 live rows and 0 dead
rows; 0 rows in sample, 0 estimated total rows
INFO:  analyzing "public.parent" inheritance tree
INFO:  skipping analyze of "public.parent" inheritance tree --- this
inheritance tree contains foreign tables

Attached is the update version of the patch.  Based on the result of
discussions about attr_needed upthread, I've changed the patch so that
create_foreignscan_plan makes reference to reltargetlist, not to
attr_needed.  (So, the patch in [1] isn't required, anymore.)

Other changes:
* Revise code/comments/docs a bit
* Add more regression tests

A separate patch (analyze.patch) handles the former case in a similar way.

[1] http://www.postgresql.org/message-id/53F4707C.8030904@lab.ntt.co.jp

Thanks,

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hello, I have a request with slight significance for the messages.

> I'd like to address this by emitting the second message as shown below:
> 
> INFO:  analyzing "public.parent"
> INFO:  "parent": scanned 0 of 0 pages, containing 0 live rows and 0 dead 
> rows; 0 rows in sample, 0 estimated total rows
> INFO:  analyzing "public.parent" inheritance tree
> INFO:  skipping analyze of "public.parent" inheritance tree --- this 
> inheritance tree contains foreign tables

In acquire_inherited_sample_rows(), the message below is emitted
when the parent explicitly specified in analyze command has at
least one foreign tables.

>    "skipping analyze of \"%s.%s\" inheritance tree --- this
>     inheritance tree contains foreign tables"

This message implicitly asserts (for me) that "A inheritance tree
containing at least one foreign tables *always* cannot be
analyzed" but in reality, we can let it go by specifying the
parent table explicitly. For example, the additional HINT or
DETAIL message would clarify that.

> INFO:  analyzing "public.parent" inheritance tree
> INFO:  skipping analyze of "public.parent" inheritance tree --- this 
> inheritance tree contains foreign tables
+ HINT:  You can analyze this inheritance tree by specifying "public.parent" to analze command

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



Re: inherit support for foreign tables

From
Heikki Linnakangas
Date:
I had a cursory look at this patch and the discussions around this.

ISTM there are actually two new features in this: 1) allow CHECK 
constraints on foreign tables, and 2) support inheritance for foreign 
tables. How about splitting it into two?

- Heikki




Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/09/11 4:32), Heikki Linnakangas wrote:
> I had a cursory look at this patch and the discussions around this.

Thank you!

> ISTM there are actually two new features in this: 1) allow CHECK
> constraints on foreign tables, and 2) support inheritance for foreign
> tables. How about splitting it into two?

That's right.  There are the two in this patch.

I'm not sure if I should split the patch into the two, because 1) won't 
make sense without 2).  As described in the following note added to the 
docs on CREATE FOREIGN TABLE, CHECK constraints on foreign tables are 
intended to support constraint exclusion for partitioned foreign tables:

+     Constraints on foreign tables are not enforced on insert or update.
+     Those definitions simply declare the constraints hold for all rows
+     in the foreign tables.  It is the user's responsibility to ensure
+     that those definitions match the remote side.  Such constraints are
+     used for some kind of query optimization such as constraint exclusion
+     for partitioned tables

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Heikki Linnakangas
Date:
On 09/11/2014 12:22 PM, Etsuro Fujita wrote:
> (2014/09/11 4:32), Heikki Linnakangas wrote:
>> I had a cursory look at this patch and the discussions around this.
>
> Thank you!
>
>> ISTM there are actually two new features in this: 1) allow CHECK
>> constraints on foreign tables, and 2) support inheritance for foreign
>> tables. How about splitting it into two?
>
> That's right.  There are the two in this patch.
>
> I'm not sure if I should split the patch into the two, because 1) won't
> make sense without 2).  As described in the following note added to the
> docs on CREATE FOREIGN TABLE, CHECK constraints on foreign tables are
> intended to support constraint exclusion for partitioned foreign tables:
>
> +     Constraints on foreign tables are not enforced on insert or update.
> +     Those definitions simply declare the constraints hold for all rows
> +     in the foreign tables.  It is the user's responsibility to ensure
> +     that those definitions match the remote side.  Such constraints are
> +     used for some kind of query optimization such as constraint exclusion
> +     for partitioned tables

The planner can do constraint exclusion based on CHECK constraints even 
without inheritance. It's not enabled by default because it can increase 
planning time, but if you set constraint_exclusion=on, it will work.

For example:

postgres=# create table foo (i int4 CHECK (i > 0));
CREATE TABLE
postgres=# explain select * from foo WHERE i < 0;                      QUERY PLAN
------------------------------------------------------ Seq Scan on foo  (cost=0.00..40.00 rows=800 width=4)   Filter:
(i< 0) Planning time: 0.335 ms
 
(3 rows)

postgres=# show constraint_exclusion ; constraint_exclusion
---------------------- partition
(1 row)

postgres=# set constraint_exclusion ='on';
SET
postgres=# explain select * from foo WHERE i < 0;                QUERY PLAN
------------------------------------------ Result  (cost=0.00..0.01 rows=1 width=0)   One-Time Filter: false Planning
time:0.254 ms
 
(3 rows)

postgres=#

- Heikki




Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/09/11 19:46), Heikki Linnakangas wrote:
> On 09/11/2014 12:22 PM, Etsuro Fujita wrote:
>> (2014/09/11 4:32), Heikki Linnakangas wrote:
>>> I had a cursory look at this patch and the discussions around this.
>>
>> Thank you!
>>
>>> ISTM there are actually two new features in this: 1) allow CHECK
>>> constraints on foreign tables, and 2) support inheritance for foreign
>>> tables. How about splitting it into two?
>>
>> That's right.  There are the two in this patch.
>>
>> I'm not sure if I should split the patch into the two, because 1) won't
>> make sense without 2).  As described in the following note added to the
>> docs on CREATE FOREIGN TABLE, CHECK constraints on foreign tables are
>> intended to support constraint exclusion for partitioned foreign tables:
>>
>> +     Constraints on foreign tables are not enforced on insert or update.
>> +     Those definitions simply declare the constraints hold for all rows
>> +     in the foreign tables.  It is the user's responsibility to ensure
>> +     that those definitions match the remote side.  Such constraints are
>> +     used for some kind of query optimization such as constraint
>> exclusion
>> +     for partitioned tables
>
> The planner can do constraint exclusion based on CHECK constraints even
> without inheritance. It's not enabled by default because it can increase
> planning time, but if you set constraint_exclusion=on, it will work.

Exactly!

> For example:
>
> postgres=# create table foo (i int4 CHECK (i > 0));
> CREATE TABLE
> postgres=# explain select * from foo WHERE i < 0;
>                        QUERY PLAN
> ------------------------------------------------------
>   Seq Scan on foo  (cost=0.00..40.00 rows=800 width=4)
>     Filter: (i < 0)
>   Planning time: 0.335 ms
> (3 rows)
>
> postgres=# show constraint_exclusion ;
>   constraint_exclusion
> ----------------------
>   partition
> (1 row)
>
> postgres=# set constraint_exclusion ='on';
> SET
> postgres=# explain select * from foo WHERE i < 0;
>                  QUERY PLAN
> ------------------------------------------
>   Result  (cost=0.00..0.01 rows=1 width=0)
>     One-Time Filter: false
>   Planning time: 0.254 ms
> (3 rows)
>
> postgres=#

Actually, this patch allows the exact same thing to apply to foreign 
tables.  My explanation was insufficient about that.  Sorry for that.

So, should I split the patch into the two?

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Heikki Linnakangas
Date:
On 09/11/2014 02:30 PM, Etsuro Fujita wrote:
> Actually, this patch allows the exact same thing to apply to foreign
> tables.  My explanation was insufficient about that.  Sorry for that.

Great, that's what I thought.

> So, should I split the patch into the two?

Yeah, please do.

- Heikki



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/09/11 20:51), Heikki Linnakangas wrote:
> On 09/11/2014 02:30 PM, Etsuro Fujita wrote:
>> So, should I split the patch into the two?
>
> Yeah, please do.

OK, Will do.

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/09/12 16:30), Etsuro Fujita wrote:
> (2014/09/11 20:51), Heikki Linnakangas wrote:
>> On 09/11/2014 02:30 PM, Etsuro Fujita wrote:
>>> So, should I split the patch into the two?
>>
>> Yeah, please do.
>
> OK, Will do.

Here are separated patches.

fdw-chk.patch  - CHECK constraints on foreign tables
fdw-inh.patch  - table inheritance with foreign tables

The latter has been created on top of [1].  I've addressed the comment
from Horiguchi-san [2] in it also, though I've slightly modified his
proposal.

There is the functionality for path reparameterization for ForeignScan
in the previous patch, but since the functionality would be useful not
only for such table inheritance cases but for UNION ALL cases, I'd add
the functionality as an independent feature maybe to CF 2014-12.

Thanks,

[1] http://www.postgresql.org/message-id/540DA168.3040407@lab.ntt.co.jp
[2]
http://www.postgresql.org/message-id/20140902.142218.253402812.horiguchi.kyotaro@lab.ntt.co.jp

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/08/28 18:00), Etsuro Fujita wrote:
> (2014/08/22 11:51), Noah Misch wrote:
>> Today's ANALYZE VERBOSE messaging for former inheritance parents
>> (tables with
>> relhassubclass = true but no pg_inherits.inhparent links) is
>> deceptive, and I
>> welcome a fix to omit the spurious message.  As defects go, this is quite
>> minor.  There's fundamentally no value in collecting inheritance tree
>> statistics for such a parent, and no PostgreSQL command will do so.

>> A
>> credible alternative is to emit a second message indicating that we
>> skipped
>> the inheritance tree statistics after all, and why we skipped them.

> I'd like to address this by emitting the second message as shown below:

> A separate patch (analyze.patch) handles the former case in a similar way.

I'll add to the upcoming CF, the analyze.patch as an independent item,
which emits a second message indicating that we skipped the inheritance
tree statistics and why we skipped them.

Thanks,

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/10/14 20:00), Etsuro Fujita wrote:
> Here are separated patches.
>
> fdw-chk.patch  - CHECK constraints on foreign tables
> fdw-inh.patch  - table inheritance with foreign tables
>
> The latter has been created on top of [1].

> [1] http://www.postgresql.org/message-id/540DA168.3040407@lab.ntt.co.jp

To be exact, it has been created on top of [1] and fdw-chk.patch.

I noticed that the latter disallows TRUNCATE on inheritance trees that 
contain at least one child foreign table.  But I think it would be 
better to allow it, with the semantics that we quietly ignore the child 
foreign tables and apply the operation to the child plain tables, which 
is the same semantics as ALTER COLUMN SET STORAGE on such inheritance 
trees.  Comments welcome.

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/10/21 17:40), Etsuro Fujita wrote:
> (2014/10/14 20:00), Etsuro Fujita wrote:
>> Here are separated patches.
>>
>> fdw-chk.patch  - CHECK constraints on foreign tables
>> fdw-inh.patch  - table inheritance with foreign tables
>>
>> The latter has been created on top of [1].
>
>> [1] http://www.postgresql.org/message-id/540DA168.3040407@lab.ntt.co.jp
>
> To be exact, it has been created on top of [1] and fdw-chk.patch.
>
> I noticed that the latter disallows TRUNCATE on inheritance trees that
> contain at least one child foreign table.  But I think it would be
> better to allow it, with the semantics that we quietly ignore the child
> foreign tables and apply the operation to the child plain tables, which
> is the same semantics as ALTER COLUMN SET STORAGE on such inheritance
> trees.  Comments welcome.

Done.  And I've also a bit revised regression tests for both patches.
Patches attached.

Thanks,

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hello, I don't fully catch up this topic but tried this one.

> >> Here are separated patches.
> >>
> >> fdw-chk.patch  - CHECK constraints on foreign tables
> >> fdw-inh.patch  - table inheritance with foreign tables
> >>
> >> The latter has been created on top of [1].
> >
> >> [1]
> >> http://www.postgresql.org/message-id/540DA168.3040407@lab.ntt.co.jp
> 
> > To be exact, it has been created on top of [1] and fdw-chk.patch.

I tried both patches on the current head, the newly added
parameter to analyze_rel() hampered them from applying but it is
easy to fix seemingly and almost all the other part was applied
cleanly.

By the way, are these the result of simply splitting of your last
patch, foreign_inherit-v15.patch?

http://www.postgresql.org/message-id/53FEEF94.6040101@lab.ntt.co.jp

The result of apllying whole-in-one version and this splitted
version seem to have many differences. Did you added even other
changes? Or do I understand this patch wrongly?

git diff --numstat 0_foreign_inherit_one 0_foreign_inherit_two 
5       51      contrib/file_fdw/file_fdw.c
10      19      contrib/file_fdw/input/file_fdw.source
18      71      contrib/file_fdw/output/file_fdw.source
19      70      contrib/postgres_fdw/expected/postgres_fdw.out
9       66      contrib/postgres_fdw/postgres_fdw.c
12      35      contrib/postgres_fdw/sql/postgres_fdw.sql
13      48      doc/src/sgml/fdwhandler.sgml
39      39      doc/src/sgml/ref/alter_foreign_table.sgml
4       3       doc/src/sgml/ref/create_foreign_table.sgml
8       0       src/backend/catalog/heap.c
7       3       src/backend/commands/analyze.c
0       7       src/backend/commands/tablecmds.c
1       22      src/backend/optimizer/plan/createplan.c
7       7       src/backend/optimizer/prep/prepunion.c
0       26      src/backend/optimizer/util/pathnode.c
1       1       src/include/commands/vacuum.h
0       7       src/include/foreign/fdwapi.h
19      1       src/test/regress/expected/foreign_data.out
9       2       src/test/regress/sql/foreign_data.sql


> > I noticed that the latter disallows TRUNCATE on inheritance trees that
> > contain at least one child foreign table.  But I think it would be
> > better to allow it, with the semantics that we quietly ignore the
> > child
> > foreign tables and apply the operation to the child plain tables,
> > which
> > is the same semantics as ALTER COLUMN SET STORAGE on such inheritance
> > trees.  Comments welcome.
> 
> Done.  And I've also a bit revised regression tests for both
> patches. Patches attached.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center




Re: inherit support for foreign tables

From
Date:
> (2014/08/28 18:00), Etsuro Fujita wrote:
> > (2014/08/22 11:51), Noah Misch wrote:
> >> Today's ANALYZE VERBOSE messaging for former inheritance parents
> >> (tables with relhassubclass = true but no pg_inherits.inhparent
> >> links) is deceptive, and I welcome a fix to omit the spurious
> >> message.  As defects go, this is quite minor.  There's fundamentally
> >> no value in collecting inheritance tree statistics for such a parent,
> >> and no PostgreSQL command will do so.
> 
> >> A
> >> credible alternative is to emit a second message indicating that we
> >> skipped the inheritance tree statistics after all, and why we skipped
> >> them.
> 
> > I'd like to address this by emitting the second message as shown below:
> 
> > A separate patch (analyze.patch) handles the former case in a similar
> way.
> 
> I'll add to the upcoming CF, the analyze.patch as an independent item,
> which emits a second message indicating that we skipped the inheritance
> tree statistics and why we skipped them.

I did a review of the patch. 
There was no problem.
I confirmed the following.

1. applied cleanly and compilation was without warnings and errors
2. all regress tests was passed ok
3. The message output from ANALYZE VERBOSE.

Following are the SQL which I used to check messages.

create table parent (id serial);
create table child (name text) inherits (parent);
ANALYZE VERBOSE parent ;
drop table child ;
ANALYZE VERBOSE parent ;

Regards,

--
Furuya Osamu


Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
Hi Furuya-san,

(2014/11/07 16:54), furuyao@pm.nttdata.co.jp wrote:
>> (2014/08/28 18:00), Etsuro Fujita wrote:
>>> (2014/08/22 11:51), Noah Misch wrote:
>>>> Today's ANALYZE VERBOSE messaging for former inheritance parents
>>>> (tables with relhassubclass = true but no pg_inherits.inhparent
>>>> links) is deceptive, and I welcome a fix to omit the spurious
>>>> message.  As defects go, this is quite minor.  There's fundamentally
>>>> no value in collecting inheritance tree statistics for such a parent,
>>>> and no PostgreSQL command will do so.
>>
>>>> A
>>>> credible alternative is to emit a second message indicating that we
>>>> skipped the inheritance tree statistics after all, and why we skipped
>>>> them.
>>
>>> I'd like to address this by emitting the second message as shown below:
>>
>>> A separate patch (analyze.patch) handles the former case in a similar
>> way.
>>
>> I'll add to the upcoming CF, the analyze.patch as an independent item,
>> which emits a second message indicating that we skipped the inheritance
>> tree statistics and why we skipped them.
> 
> I did a review of the patch.
> There was no problem.
> I confirmed the following.
> 
> 1. applied cleanly and compilation was without warnings and errors
> 2. all regress tests was passed ok
> 3. The message output from ANALYZE VERBOSE.
> 
> Following are the SQL which I used to check messages.
> 
> create table parent (id serial);
> create table child (name text) inherits (parent);
> ANALYZE VERBOSE parent ;
> drop table child ;
> ANALYZE VERBOSE parent ;

I think that that is a correct test for this patch.

Thanks for the review!

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/11/07 14:57), Kyotaro HORIGUCHI wrote:
>>>> Here are separated patches.
>>>>
>>>> fdw-chk.patch  - CHECK constraints on foreign tables
>>>> fdw-inh.patch  - table inheritance with foreign tables
>>>>
>>>> The latter has been created on top of [1].
>>>
>>>> [1]
>>>> http://www.postgresql.org/message-id/540DA168.3040407@lab.ntt.co.jp
>>
>>> To be exact, it has been created on top of [1] and fdw-chk.patch.
>
> I tried both patches on the current head, the newly added
> parameter to analyze_rel() hampered them from applying but it is
> easy to fix seemingly and almost all the other part was applied
> cleanly.

Thanks for the review!

> By the way, are these the result of simply splitting of your last
> patch, foreign_inherit-v15.patch?
>
> http://www.postgresql.org/message-id/53FEEF94.6040101@lab.ntt.co.jp

The answer is "no".

> The result of apllying whole-in-one version and this splitted
> version seem to have many differences. Did you added even other
> changes? Or do I understand this patch wrongly?

As I said before, I splitted the whole-in-one version into three: 1)
CHECK constraint patch (ie fdw-chk.patch), 2) table inheritance patch
(ie fdw-inh.patch) and 3) path reparameterization patch (not posted).
In addition to that, I slightly modified #1 and #2.

IIUC, #3 would be useful not only for the inheritance cases but for
union all cases.  So, I plan to propose it independently in the next CF.

>>> I noticed that the latter disallows TRUNCATE on inheritance trees that
>>> contain at least one child foreign table.  But I think it would be
>>> better to allow it, with the semantics that we quietly ignore the
>>> child
>>> foreign tables and apply the operation to the child plain tables,
>>> which
>>> is the same semantics as ALTER COLUMN SET STORAGE on such inheritance
>>> trees.  Comments welcome.
>>
>> Done.  And I've also a bit revised regression tests for both
>> patches. Patches attached.

I rebased the patches to the latest head.  Here are updated patches.

Other changes:

* fdw-chk-3.patch: the updated patch revises some ereport messages a
little bit.

* fdw-inh-3.patch: I noticed that there is a doc bug in the previous
patch.  The updated patch fixes that, adds a bit more docs, and revises
regression tests in foreign_data.sql a bit further.

Thanks,

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Ashutosh Bapat
Date:
Hi Fujita-san,
I reviewed fdw-chk-3 patch. Here are my comments

Sanity
--------
1. The patch applies on the latest master using "patch but not by git apply
2. it compiles clean
3. Regression run is clean, including the contrib module regressions

Tests
-------
1. The tests added in file_fdw module look good. We should add tests for CREATE TABLE with CHECK CONSTRAINT also.
2.  For postgres_fdw we need tests to check the behaviour in case the constraints mismatch between the remote table and its local foreign table declaration in case of INSERT, UPDATE and SELECT.
3. In the testcases for postgres_fdw it seems that you have forgotten to add statement after SET constraint_exclusion to 'partition'
4. In test foreign_data there are changes to fix the diffs caused by these changes like below
 ALTER FOREIGN TABLE ft1 DROP CONSTRAINT no_const;               -- ERROR
-ERROR:  "ft1" is not a table
+ERROR:  constraint "no_const" of relation "ft1" does not exist
 ALTER FOREIGN TABLE ft1 DROP CONSTRAINT IF EXISTS no_const;
-ERROR:  "ft1" is not a table
+NOTICE:  constraint "no_const" of relation "ft1" does not exist, skipping
 ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c1_check;
-ERROR:  "ft1" is not a table
+ERROR:  constraint "ft1_c1_check" of relation "ft1" does not exist


Earlier when constraints were not supported for FOREIGN TABLE, these tests made sure the same functionality. So, even though the corresponding constraints were not created on the table (in fact it didn't allow the creation as well). Now that the constraints are allowed, I think the tests for "no_const" (without IF EXISTS) and "ft1_c1_check" are duplicating the same testcase. May be we should review this set of statement in the light of new functionality.

Code and implementation
----------------------------------
The usage of NO INHERIT and NOT VALID with CONSTRAINT on foreign table is blocked, but corresponding documentation entry doesn't mention so. Since foreign tables can not be inherited NO INHERIT option isn't applicable to foreign tables and the constraints on the foreign tables are declarative, hence NOT VALID option is also not applicable. So, I agree with what the code is doing, but that should be reflected in documentation with this explanation.
Rest of the code modifies the condition checks for CHECK CONSTRAINTs on foreign tables, and it looks good to me.



On Fri, Nov 7, 2014 at 5:31 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/11/07 14:57), Kyotaro HORIGUCHI wrote:
Here are separated patches.

fdw-chk.patch  - CHECK constraints on foreign tables
fdw-inh.patch  - table inheritance with foreign tables

The latter has been created on top of [1].

[1]
http://www.postgresql.org/message-id/540DA168.3040407@lab.ntt.co.jp

To be exact, it has been created on top of [1] and fdw-chk.patch.

I tried both patches on the current head, the newly added
parameter to analyze_rel() hampered them from applying but it is
easy to fix seemingly and almost all the other part was applied
cleanly.

Thanks for the review!

By the way, are these the result of simply splitting of your last
patch, foreign_inherit-v15.patch?

http://www.postgresql.org/message-id/53FEEF94.6040101@lab.ntt.co.jp

The answer is "no".

The result of apllying whole-in-one version and this splitted
version seem to have many differences. Did you added even other
changes? Or do I understand this patch wrongly?

As I said before, I splitted the whole-in-one version into three: 1) CHECK constraint patch (ie fdw-chk.patch), 2) table inheritance patch (ie fdw-inh.patch) and 3) path reparameterization patch (not posted). In addition to that, I slightly modified #1 and #2.

IIUC, #3 would be useful not only for the inheritance cases but for union all cases.  So, I plan to propose it independently in the next CF.

I noticed that the latter disallows TRUNCATE on inheritance trees that
contain at least one child foreign table.  But I think it would be
better to allow it, with the semantics that we quietly ignore the
child
foreign tables and apply the operation to the child plain tables,
which
is the same semantics as ALTER COLUMN SET STORAGE on such inheritance
trees.  Comments welcome.

Done.  And I've also a bit revised regression tests for both
patches. Patches attached.

I rebased the patches to the latest head.  Here are updated patches.

Other changes:

* fdw-chk-3.patch: the updated patch revises some ereport messages a little bit.

* fdw-inh-3.patch: I noticed that there is a doc bug in the previous patch.  The updated patch fixes that, adds a bit more docs, and revises regression tests in foreign_data.sql a bit further.


Thanks,

Best regards,
Etsuro Fujita


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers




--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: inherit support for foreign tables

From
Ashutosh Bapat
Date:
Hi Fujita-san,
I tried to apply fdw-inh-3.patch on the latest head from master branch. It failed to apply using both patch and git apply.

"patch" failed to apply because of rejections in contrib/file_fdw/output/file_fdw.source and doc/src/sgml/ref/create_foreign_table.sgml


On Fri, Nov 7, 2014 at 5:31 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/11/07 14:57), Kyotaro HORIGUCHI wrote:
Here are separated patches.

fdw-chk.patch  - CHECK constraints on foreign tables
fdw-inh.patch  - table inheritance with foreign tables

The latter has been created on top of [1].

[1]
http://www.postgresql.org/message-id/540DA168.3040407@lab.ntt.co.jp

To be exact, it has been created on top of [1] and fdw-chk.patch.

I tried both patches on the current head, the newly added
parameter to analyze_rel() hampered them from applying but it is
easy to fix seemingly and almost all the other part was applied
cleanly.

Thanks for the review!

By the way, are these the result of simply splitting of your last
patch, foreign_inherit-v15.patch?

http://www.postgresql.org/message-id/53FEEF94.6040101@lab.ntt.co.jp

The answer is "no".

The result of apllying whole-in-one version and this splitted
version seem to have many differences. Did you added even other
changes? Or do I understand this patch wrongly?

As I said before, I splitted the whole-in-one version into three: 1) CHECK constraint patch (ie fdw-chk.patch), 2) table inheritance patch (ie fdw-inh.patch) and 3) path reparameterization patch (not posted). In addition to that, I slightly modified #1 and #2.

IIUC, #3 would be useful not only for the inheritance cases but for union all cases.  So, I plan to propose it independently in the next CF.

I noticed that the latter disallows TRUNCATE on inheritance trees that
contain at least one child foreign table.  But I think it would be
better to allow it, with the semantics that we quietly ignore the
child
foreign tables and apply the operation to the child plain tables,
which
is the same semantics as ALTER COLUMN SET STORAGE on such inheritance
trees.  Comments welcome.

Done.  And I've also a bit revised regression tests for both
patches. Patches attached.

I rebased the patches to the latest head.  Here are updated patches.

Other changes:

* fdw-chk-3.patch: the updated patch revises some ereport messages a little bit.

* fdw-inh-3.patch: I noticed that there is a doc bug in the previous patch.  The updated patch fixes that, adds a bit more docs, and revises regression tests in foreign_data.sql a bit further.


Thanks,

Best regards,
Etsuro Fujita


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers




--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
Hi Ashutosh,

Thanks for the review!

(2014/11/13 15:23), Ashutosh Bapat wrote:
> I tried to apply fdw-inh-3.patch on the latest head from master branch.
> It failed to apply using both patch and git apply.
>
> "patch" failed to apply because of rejections in
> contrib/file_fdw/output/file_fdw.source and
> doc/src/sgml/ref/create_foreign_table.sgml

As I said upthread, fdw-inh-3.patch has been created on top of [1] and 
fdw-chk-3.patch.  Did you apply these patche first?

[1] https://commitfest.postgresql.org/action/patch_view?id=1599

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Ashutosh Bapat
Date:


On Thu, Nov 13, 2014 at 12:20 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
Hi Ashutosh,

Thanks for the review!

(2014/11/13 15:23), Ashutosh Bapat wrote:
I tried to apply fdw-inh-3.patch on the latest head from master branch.
It failed to apply using both patch and git apply.

"patch" failed to apply because of rejections in
contrib/file_fdw/output/file_fdw.source and
doc/src/sgml/ref/create_foreign_table.sgml

As I said upthread, fdw-inh-3.patch has been created on top of [1] and fdw-chk-3.patch.  Did you apply these patche first?

Oh, sorry, I didn't pay attention to that. I will apply both the patches and review the inheritance patch. Thanks for pointing that out.
 
[1] https://commitfest.postgresql.org/action/patch_view?id=1599

Best regards,
Etsuro Fujita



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/11/12 20:04), Ashutosh Bapat wrote:
> I reviewed fdw-chk-3 patch. Here are my comments

Thanks for the review!

> Tests
> -------
> 1. The tests added in file_fdw module look good. We should add tests for
> CREATE TABLE with CHECK CONSTRAINT also.

Agreed.  I added the tests, and also changed the proposed tests a bit.

> 2.  For postgres_fdw we need tests to check the behaviour in case the
> constraints mismatch between the remote table and its local foreign
> table declaration in case of INSERT, UPDATE and SELECT.

Done.

> 3. In the testcases for postgres_fdw it seems that you have forgotten to
> add statement after SET constraint_exclusion to 'partition'

I added the statement.

> 4. In test foreign_data there are changes to fix the diffs caused by
> these changes like below
>   ALTER FOREIGN TABLE ft1 DROP CONSTRAINT no_const;               -- ERROR
> -ERROR:  "ft1" is not a table
> +ERROR:  constraint "no_const" of relation "ft1" does not exist
>   ALTER FOREIGN TABLE ft1 DROP CONSTRAINT IF EXISTS no_const;
> -ERROR:  "ft1" is not a table
> +NOTICE:  constraint "no_const" of relation "ft1" does not exist, skipping
>   ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c1_check;
> -ERROR:  "ft1" is not a table
> +ERROR:  constraint "ft1_c1_check" of relation "ft1" does not exist

> Earlier when constraints were not supported for FOREIGN TABLE, these
> tests made sure the same functionality. So, even though the
> corresponding constraints were not created on the table (in fact it
> didn't allow the creation as well). Now that the constraints are
> allowed, I think the tests for "no_const" (without IF EXISTS) and
> "ft1_c1_check" are duplicating the same testcase. May be we should
> review this set of statement in the light of new functionality.

Agreed.  I removed the "DROP CONSTRAINT ft1_c1_check" test, and added a
new test for ALTER CONSTRAINT.

> Code and implementation
> ----------------------------------
> The usage of NO INHERIT and NOT VALID with CONSTRAINT on foreign table
> is blocked, but corresponding documentation entry doesn't mention so.
> Since foreign tables can not be inherited NO INHERIT option isn't
> applicable to foreign tables and the constraints on the foreign tables
> are declarative, hence NOT VALID option is also not applicable. So, I
> agree with what the code is doing, but that should be reflected in
> documentation with this explanation.
> Rest of the code modifies the condition checks for CHECK CONSTRAINTs on
> foreign tables, and it looks good to me.

Done.

Other changes:
* Modified one error message that I added in AddRelationNewConstraints,
to match the other message there.
* Revised other docs a little bit.

Attached is an updated version of the patch.

Thanks,

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Ashutosh Bapat
Date:
Hi Fujita-san,
Here are my review comments for patch fdw-inh-3.patch.

Sanity
--------
1. The patch applies cleanly
2. It compiles cleanly.
3. The server regression tests and tests in contrib modules pass.

Tests
-------
1. It seems like you have copied from testcase inherit.sql to postgres_fdw testcase. That's a good thing, but it makes the test quite long. May be we should have two tests in postgres_fdw contrib module, one for simple cases, and other for inheritance. What do you say?

Documentation
--------------------
1. The change in ddl.sgml
-        We will refer to the child tables as partitions, though they
-        are in every way normal <productname>PostgreSQL</> tables.
+        We will refer to the child tables as partitions, though we assume
+        that they are normal <productname>PostgreSQL</> tables.

adds phrase "we assume that", which confuses the intention behind the sentence. The original text is intended to highlight the equivalence between "partition" and "normal table", where as the addition esp. the word "assume" weakens that equivalence. Instead now we have to highlight the equivalence between "partition" and "normal or foreign table". The wording similar to "though they are either normal or foreign tables" should be used there.

2. The wording "some kind of optimization" gives vague picture. May be it can be worded as "Since the constraints are assumed to be true, they are used in constraint-based query optimization like constraint exclusion for partitioned tables.".
+    Those constraints are used in some kind of query optimization such
+    as constraint exclusion for partitioned tables (see
+    <xref linkend="ddl-partitioning">).

Code
-------
1. In the following change
+/*
  * acquire_inherited_sample_rows -- acquire sample rows from inheritance tree
  *
  * This has the same API as acquire_sample_rows, except that rows are
  * collected from all inheritance children as well as the specified table.
- * We fail and return zero if there are no inheritance children.
+ * We fail and return zero if there are no inheritance children or there are
+ * inheritance children that foreign tables.

The addition should be "there are inheritance children that are all foreign tables. Note the addition "are all".

2. The function has_foreign() be better named has_foreign_child()? This function loops through all the tableoids passed even after it has found a foreign table. Why can't we return true immediately after finding the first foreign table, unless the side effects of heap_open() on all the table are required. But I don't see that to be the case, since these tables are locked already through a previous call to heap_open(). In the same function instead of argument name parentrelId may be we should use name parent_oid, so that we use same notation for parent and child table OIDs.

3.  Regarding enum VacuumMode - it's being used only in case of acquire_inherited_sample_rows() and that too, to check only a single value of the three defined there. May be we should just infer that value inside acquire_inherited_sample_rows() or pass a boolean true or false from do_analyse_rel() based on the VacuumStmt. I do not see need for a separate three value enum of which only one value is used and also to pass it down from vacuum() by changing signatures of the minion functions.

4. In postgresGetForeignPlan(), the code added by this patch is required to handle the case when the row mark is placed on a parent table and hence is required to be applied on the child table. We need a comment explaining this. Otherwise, the three step process to get the row mark information isn't clear for a reader.

5. In expand_inherited_rtentry() why do you need a separate variable hasForeign? Instead of using that variable, you can actually set/reset rte->hasForeign since existence of even a single foreign child would mark that member as true. - After typing this, I got the answer when I looked at the function code. Every child's RTE is initially a copy of parent's RTE and hence hasForeign status would be inherited by every child after the first foreign child. I think, this reasoning should be added as comment before assignment to rte->hasForeign at the end of the function.

6. The tests in foreign_data.sql are pretty extensive. Thanks for that. I think, we should also check the effect of each of the following command using \d on appropriate tables.
+CREATE FOREIGN TABLE ft2 () INHERITS (pt1)
+  SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted" 'value');
+ALTER FOREIGN TABLE ft2 NO INHERIT pt1;
+DROP FOREIGN TABLE ft2;
+CREATE FOREIGN TABLE ft2 (
+   c1 integer NOT NULL,
+   c2 text,
+   c3 date
+) SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted" 'value');
+ALTER FOREIGN TABLE ft2 INHERIT pt1;

Rest of the changes look good.


On Thu, Nov 13, 2014 at 12:21 PM, Ashutosh Bapat <ashutosh.bapat@enterprisedb.com> wrote:


On Thu, Nov 13, 2014 at 12:20 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
Hi Ashutosh,

Thanks for the review!

(2014/11/13 15:23), Ashutosh Bapat wrote:
I tried to apply fdw-inh-3.patch on the latest head from master branch.
It failed to apply using both patch and git apply.

"patch" failed to apply because of rejections in
contrib/file_fdw/output/file_fdw.source and
doc/src/sgml/ref/create_foreign_table.sgml

As I said upthread, fdw-inh-3.patch has been created on top of [1] and fdw-chk-3.patch.  Did you apply these patche first?

Oh, sorry, I didn't pay attention to that. I will apply both the patches and review the inheritance patch. Thanks for pointing that out.
 
[1] https://commitfest.postgresql.org/action/patch_view?id=1599

Best regards,
Etsuro Fujita



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: inherit support for foreign tables

From
Ashutosh Bapat
Date:


On Mon, Nov 17, 2014 at 1:25 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/11/12 20:04), Ashutosh Bapat wrote:
I reviewed fdw-chk-3 patch. Here are my comments

Thanks for the review!

Tests
-------
1. The tests added in file_fdw module look good. We should add tests for
CREATE TABLE with CHECK CONSTRAINT also.

Agreed.  I added the tests, and also changed the proposed tests a bit.

2.  For postgres_fdw we need tests to check the behaviour in case the
constraints mismatch between the remote table and its local foreign
table declaration in case of INSERT, UPDATE and SELECT.

Done.

3. In the testcases for postgres_fdw it seems that you have forgotten to
add statement after SET constraint_exclusion to 'partition'

I added the statement.

4. In test foreign_data there are changes to fix the diffs caused by
these changes like below
  ALTER FOREIGN TABLE ft1 DROP CONSTRAINT no_const;               -- ERROR
-ERROR:  "ft1" is not a table
+ERROR:  constraint "no_const" of relation "ft1" does not exist
  ALTER FOREIGN TABLE ft1 DROP CONSTRAINT IF EXISTS no_const;
-ERROR:  "ft1" is not a table
+NOTICE:  constraint "no_const" of relation "ft1" does not exist, skipping
  ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c1_check;
-ERROR:  "ft1" is not a table
+ERROR:  constraint "ft1_c1_check" of relation "ft1" does not exist

Earlier when constraints were not supported for FOREIGN TABLE, these
tests made sure the same functionality. So, even though the
corresponding constraints were not created on the table (in fact it
didn't allow the creation as well). Now that the constraints are
allowed, I think the tests for "no_const" (without IF EXISTS) and
"ft1_c1_check" are duplicating the same testcase. May be we should
review this set of statement in the light of new functionality.

Agreed.  I removed the "DROP CONSTRAINT ft1_c1_check" test, and added a new test for ALTER CONSTRAINT.

Code and implementation
----------------------------------
The usage of NO INHERIT and NOT VALID with CONSTRAINT on foreign table
is blocked, but corresponding documentation entry doesn't mention so.
Since foreign tables can not be inherited NO INHERIT option isn't
applicable to foreign tables and the constraints on the foreign tables
are declarative, hence NOT VALID option is also not applicable. So, I
agree with what the code is doing, but that should be reflected in
documentation with this explanation.
Rest of the code modifies the condition checks for CHECK CONSTRAINTs on
foreign tables, and it looks good to me.

Done.

Other changes:
* Modified one error message that I added in AddRelationNewConstraints, to match the other message there.
* Revised other docs a little bit.

Attached is an updated version of the patch.


I looked at the patch. It looks good now. Once we have the inh patch ready, we can mark this item as ready for commiter.

Thanks,

Best regards,
Etsuro Fujita



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/11/18 18:09), Ashutosh Bapat wrote:
> I looked at the patch. It looks good now. Once we have the inh patch
> ready, we can mark this item as ready for commiter.

Thanks for the review!

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/11/17 17:55), Ashutosh Bapat wrote:
> Here are my review comments for patch fdw-inh-3.patch.

Thanks for the review!

> Tests
> -------
> 1. It seems like you have copied from testcase inherit.sql to
> postgres_fdw testcase. That's a good thing, but it makes the test quite
> long. May be we should have two tests in postgres_fdw contrib module,
> one for simple cases, and other for inheritance. What do you say?

IMO, the test is not so time-consuming, so it doesn't seem worth 
splitting it into two.

> Documentation
> --------------------
> 1. The change in ddl.sgml
> -        We will refer to the child tables as partitions, though they
> -        are in every way normal <productname>PostgreSQL</> tables.
> +        We will refer to the child tables as partitions, though we assume
> +        that they are normal <productname>PostgreSQL</> tables.
>
> adds phrase "we assume that", which confuses the intention behind the
> sentence. The original text is intended to highlight the equivalence
> between "partition" and "normal table", where as the addition esp. the
> word "assume" weakens that equivalence. Instead now we have to highlight
> the equivalence between "partition" and "normal or foreign table". The
> wording similar to "though they are either normal or foreign tables"
> should be used there.

You are right, but I feel that there is something out of place in saying 
that there (5.10.2. Implementing Partitioning) because the procedure 
there has been written based on normal tables only.  Put another way, if 
we say that, I think we'd need to add more docs, describing the syntax 
and/or the corresponding examples for foreign-table cases.  But I'd like 
to leave that for another patch.  So, how about the wording "we assume 
*here* that", instead of "we assume that", as I added the following 
notice in the previous section (5.10.1. Overview)?

@@ -2650,7 +2669,10 @@ VALUES ('Albany', NULL, NULL, 'NY');     table of a single parent table.  The parent table
itselfis normally     empty; it exists just to represent the entire data set.  You should be     familiar with
inheritance(see <xref linkend="ddl-inherit">) before
 
-    attempting to set up partitioning.
+    attempting to set up partitioning.  (The setup and management of
+    partitioned tables illustrated in this section assume that each
+    partition is a normal table.  However, you can do that in a similar way
+    for cases where some or all partitions are foreign tables.)

> 2. The wording "some kind of optimization" gives vague picture. May be
> it can be worded as "Since the constraints are assumed to be true, they
> are used in constraint-based query optimization like constraint
> exclusion for partitioned tables.".
> +    Those constraints are used in some kind of query optimization such
> +    as constraint exclusion for partitioned tables (see
> +    <xref linkend="ddl-partitioning">).

Will follow your revision.

> Code
> -------
> 1. In the following change
> +/*
>    * acquire_inherited_sample_rows -- acquire sample rows from
> inheritance tree
>    *
>    * This has the same API as acquire_sample_rows, except that rows are
>    * collected from all inheritance children as well as the specified table.
> - * We fail and return zero if there are no inheritance children.
> + * We fail and return zero if there are no inheritance children or
> there are
> + * inheritance children that foreign tables.
>
> The addition should be "there are inheritance children that *are all
> *foreign tables. Note the addition "are all".

Sorry, I incorrectly wrote the comment.  What I tried to write is "We 
fail and return zero if there are no inheritance children or if we are 
not in VAC_MODE_SINGLE case and inheritance tree contains at least one 
foreign table.".

> 2. The function has_foreign() be better named has_foreign_child()? This

How about "has_foreign_table"?

> function loops through all the tableoids passed even after it has found
> a foreign table. Why can't we return true immediately after finding the
> first foreign table, unless the side effects of heap_open() on all the
> table are required. But I don't see that to be the case, since these
> tables are locked already through a previous call to heap_open(). In the

Good catch!  Will fix.

> same function instead of argument name parentrelId may be we should use
> name parent_oid, so that we use same notation for parent and child table
> OIDs.

Will fix.

> 3.  Regarding enum VacuumMode - it's being used only in case of
> acquire_inherited_sample_rows() and that too, to check only a single
> value of the three defined there. May be we should just infer that value
> inside acquire_inherited_sample_rows() or pass a boolean true or false
> from do_analyse_rel() based on the VacuumStmt. I do not see need for a
> separate three value enum of which only one value is used and also to
> pass it down from vacuum() by changing signatures of the minion functions.

I introduced that for possible future use.  See the discussion in [1].

> 4. In postgresGetForeignPlan(), the code added by this patch is required
> to handle the case when the row mark is placed on a parent table and
> hence is required to be applied on the child table. We need a comment
> explaining this. Otherwise, the three step process to get the row mark
> information isn't clear for a reader.

Will add the comment.

> 5. In expand_inherited_rtentry() why do you need a separate variable
> hasForeign? Instead of using that variable, you can actually set/reset
> rte->hasForeign since existence of even a single foreign child would
> mark that member as true. - After typing this, I got the answer when I
> looked at the function code. Every child's RTE is initially a copy of
> parent's RTE and hence hasForeign status would be inherited by every
> child after the first foreign child. I think, this reasoning should be
> added as comment before assignment to rte->hasForeign at the end of the
> function.

As you mentioned, I think we could set rte->hasForeign directly during 
the scan for the inheritance set, without the separate variable, 
hasForeign.  But ISTM that it'd improve code readability to set 
rte->hasForeign using the separate variable at the end of the function 
because rte->hasForeign has its meaning only when rte->inh is true and 
because we know whether rte->inh is true, at the end of the function.

> 6. The tests in foreign_data.sql are pretty extensive. Thanks for that.
> I think, we should also check the effect of each of the following
> command using \d on appropriate tables.
> +CREATE FOREIGN TABLE ft2 () INHERITS (pt1)
> +  SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted" 'value');
> +ALTER FOREIGN TABLE ft2 NO INHERIT pt1;
> +DROP FOREIGN TABLE ft2;
> +CREATE FOREIGN TABLE ft2 (
> +   c1 integer NOT NULL,
> +   c2 text,
> +   c3 date
> +) SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted" 'value');
> +ALTER FOREIGN TABLE ft2 INHERIT pt1;

Will fix.

Apart from the above, I noticed that the patch doesn't consider to call 
ExplainForeignModify during EXPLAIN for an inherited UPDATE/DELETE, as 
shown below (note that there are no UPDATE remote queries displayed):

postgres=# explain verbose update parent set a = a * 2 where a = 5;                                     QUERY PLAN
------------------------------------------------------------------------------------- Update on public.parent
(cost=0.00..280.77rows=25 width=10)   ->  Seq Scan on public.parent  (cost=0.00..0.00 rows=1 width=10)         Output:
(parent.a* 2), parent.ctid         Filter: (parent.a = 5)   ->  Foreign Scan on public.ft1  (cost=100.00..140.38
rows=12width=10)         Output: (ft1.a * 2), ft1.ctid         Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE
((a= 
 
5)) FOR UPDATE   ->  Foreign Scan on public.ft2  (cost=100.00..140.38 rows=12 width=10)         Output: (ft2.a * 2),
ft2.ctid        Remote SQL: SELECT a, ctid FROM public.mytable_2 WHERE ((a = 
 
5)) FOR UPDATE
(10 rows)

So, I'd like to modify explain.c to show those queries like this:

postgres=# explain verbose update parent set a = a * 2 where a = 5;                                     QUERY PLAN
------------------------------------------------------------------------------------- Update on public.parent
(cost=0.00..280.77rows=25 width=10)   ->  Seq Scan on public.parent  (cost=0.00..0.00 rows=1 width=10)         Output:
(parent.a* 2), parent.ctid         Filter: (parent.a = 5)   Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid =
$1  ->  Foreign Scan on public.ft1  (cost=100.00..140.38 rows=12 width=10)         Output: (ft1.a * 2), ft1.ctid
Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE ((a = 
 
5)) FOR UPDATE   Remote SQL: UPDATE public.mytable_2 SET a = $2 WHERE ctid = $1   ->  Foreign Scan on public.ft2
(cost=100.00..140.38rows=12 width=10)         Output: (ft2.a * 2), ft2.ctid         Remote SQL: SELECT a, ctid FROM
public.mytable_2WHERE ((a = 
 
5)) FOR UPDATE
(12 rows)

What do you say?

Sorry for the delay.

[1] http://www.postgresql.org/message-id/1316566782-sup-2678@alvh.no-ip.org

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Ashutosh Bapat
Date:


On Thu, Nov 27, 2014 at 3:52 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/11/17 17:55), Ashutosh Bapat wrote:
Here are my review comments for patch fdw-inh-3.patch.

Thanks for the review!

Tests
-------
1. It seems like you have copied from testcase inherit.sql to
postgres_fdw testcase. That's a good thing, but it makes the test quite
long. May be we should have two tests in postgres_fdw contrib module,
one for simple cases, and other for inheritance. What do you say?

IMO, the test is not so time-consuming, so it doesn't seem worth splitting it into two.

I am not worried about the timing but I am worried about the length of the file and hence ease of debugging in case we find any issues there. We will leave that for the commiter to decide.
 

Documentation
--------------------
1. The change in ddl.sgml
-        We will refer to the child tables as partitions, though they
-        are in every way normal <productname>PostgreSQL</> tables.
+        We will refer to the child tables as partitions, though we assume
+        that they are normal <productname>PostgreSQL</> tables.

adds phrase "we assume that", which confuses the intention behind the
sentence. The original text is intended to highlight the equivalence
between "partition" and "normal table", where as the addition esp. the
word "assume" weakens that equivalence. Instead now we have to highlight
the equivalence between "partition" and "normal or foreign table". The
wording similar to "though they are either normal or foreign tables"
should be used there.

You are right, but I feel that there is something out of place in saying that there (5.10.2. Implementing Partitioning) because the procedure there has been written based on normal tables only.  Put another way, if we say that, I think we'd need to add more docs, describing the syntax and/or the corresponding examples for foreign-table cases.  But I'd like to leave that for another patch.  So, how about the wording "we assume *here* that", instead of "we assume that", as I added the following notice in the previous section (5.10.1. Overview)?

@@ -2650,7 +2669,10 @@ VALUES ('Albany', NULL, NULL, 'NY');
     table of a single parent table.  The parent table itself is normally
     empty; it exists just to represent the entire data set.  You should be
     familiar with inheritance (see <xref linkend="ddl-inherit">) before
-    attempting to set up partitioning.
+    attempting to set up partitioning.  (The setup and management of
+    partitioned tables illustrated in this section assume that each
+    partition is a normal table.  However, you can do that in a similar way
+    for cases where some or all partitions are foreign tables.)


This looks ok, though, I would like to see final version of the document. But I think, we will leave that for committer to handle.
 
2. The wording "some kind of optimization" gives vague picture. May be
it can be worded as "Since the constraints are assumed to be true, they
are used in constraint-based query optimization like constraint
exclusion for partitioned tables.".
+    Those constraints are used in some kind of query optimization such
+    as constraint exclusion for partitioned tables (see
+    <xref linkend="ddl-partitioning">).

Will follow your revision.

Thanks.
 

Code
-------
1. In the following change
+/*
   * acquire_inherited_sample_rows -- acquire sample rows from
inheritance tree
   *
   * This has the same API as acquire_sample_rows, except that rows are
   * collected from all inheritance children as well as the specified table.
- * We fail and return zero if there are no inheritance children.
+ * We fail and return zero if there are no inheritance children or
there are
+ * inheritance children that foreign tables.

The addition should be "there are inheritance children that *are all
*foreign tables. Note the addition "are all".

Sorry, I incorrectly wrote the comment.  What I tried to write is "We fail and return zero if there are no inheritance children or if we are not in VAC_MODE_SINGLE case and inheritance tree contains at least one foreign table.".


You might want to use "English" description of VAC_MODE_SINGLE instead of that macro in the comment, so that reader doesn't have to look up VAC_MODE_SINGLE. But I think, we will leave this for the committer.
 
2. The function has_foreign() be better named has_foreign_child()? This

How about "has_foreign_table"?

has_foreign_child() would be better, since these are "children" in the inheritance hierarchy and not mere "table"s.
 

function loops through all the tableoids passed even after it has found
a foreign table. Why can't we return true immediately after finding the
first foreign table, unless the side effects of heap_open() on all the
table are required. But I don't see that to be the case, since these
tables are locked already through a previous call to heap_open(). In the

Good catch!  Will fix.

same function instead of argument name parentrelId may be we should use
name parent_oid, so that we use same notation for parent and child table
OIDs.

Will fix.

Thanks.
 

3.  Regarding enum VacuumMode - it's being used only in case of
acquire_inherited_sample_rows() and that too, to check only a single
value of the three defined there. May be we should just infer that value
inside acquire_inherited_sample_rows() or pass a boolean true or false
from do_analyse_rel() based on the VacuumStmt. I do not see need for a
separate three value enum of which only one value is used and also to
pass it down from vacuum() by changing signatures of the minion functions.

I introduced that for possible future use.  See the discussion in [1].

Will leave it for the commiter to decide.
 

4. In postgresGetForeignPlan(), the code added by this patch is required
to handle the case when the row mark is placed on a parent table and
hence is required to be applied on the child table. We need a comment
explaining this. Otherwise, the three step process to get the row mark
information isn't clear for a reader.

Will add the comment.

5. In expand_inherited_rtentry() why do you need a separate variable
hasForeign? Instead of using that variable, you can actually set/reset
rte->hasForeign since existence of even a single foreign child would
mark that member as true. - After typing this, I got the answer when I
looked at the function code. Every child's RTE is initially a copy of
parent's RTE and hence hasForeign status would be inherited by every
child after the first foreign child. I think, this reasoning should be
added as comment before assignment to rte->hasForeign at the end of the
function.

As you mentioned, I think we could set rte->hasForeign directly during the scan for the inheritance set, without the separate variable, hasForeign.  But ISTM that it'd improve code readability to set rte->hasForeign using the separate variable at the end of the function because rte->hasForeign has its meaning only when rte->inh is true and because we know whether rte->inh is true, at the end of the function.

Fine. Please use "hasForeignChild" instead of just "hasForeign" without a clue as to what is "foreign" here.
 

6. The tests in foreign_data.sql are pretty extensive. Thanks for that.
I think, we should also check the effect of each of the following
command using \d on appropriate tables.
+CREATE FOREIGN TABLE ft2 () INHERITS (pt1)
+  SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted" 'value');
+ALTER FOREIGN TABLE ft2 NO INHERIT pt1;
+DROP FOREIGN TABLE ft2;
+CREATE FOREIGN TABLE ft2 (
+   c1 integer NOT NULL,
+   c2 text,
+   c3 date
+) SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted" 'value');
+ALTER FOREIGN TABLE ft2 INHERIT pt1;

Will fix.

Apart from the above, I noticed that the patch doesn't consider to call ExplainForeignModify during EXPLAIN for an inherited UPDATE/DELETE, as shown below (note that there are no UPDATE remote queries displayed):

postgres=# explain verbose update parent set a = a * 2 where a = 5;
                                     QUERY PLAN
-------------------------------------------------------------------------------------
 Update on public.parent  (cost=0.00..280.77 rows=25 width=10)
   ->  Seq Scan on public.parent  (cost=0.00..0.00 rows=1 width=10)
         Output: (parent.a * 2), parent.ctid
         Filter: (parent.a = 5)
   ->  Foreign Scan on public.ft1  (cost=100.00..140.38 rows=12 width=10)
         Output: (ft1.a * 2), ft1.ctid
         Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE ((a = 5)) FOR UPDATE
   ->  Foreign Scan on public.ft2  (cost=100.00..140.38 rows=12 width=10)
         Output: (ft2.a * 2), ft2.ctid
         Remote SQL: SELECT a, ctid FROM public.mytable_2 WHERE ((a = 5)) FOR UPDATE
(10 rows)

So, I'd like to modify explain.c to show those queries like this:

postgres=# explain verbose update parent set a = a * 2 where a = 5;
                                     QUERY PLAN
-------------------------------------------------------------------------------------
 Update on public.parent  (cost=0.00..280.77 rows=25 width=10)
   ->  Seq Scan on public.parent  (cost=0.00..0.00 rows=1 width=10)
         Output: (parent.a * 2), parent.ctid
         Filter: (parent.a = 5)
   Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1
   ->  Foreign Scan on public.ft1  (cost=100.00..140.38 rows=12 width=10)
         Output: (ft1.a * 2), ft1.ctid
         Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE ((a = 5)) FOR UPDATE
   Remote SQL: UPDATE public.mytable_2 SET a = $2 WHERE ctid = $1
   ->  Foreign Scan on public.ft2  (cost=100.00..140.38 rows=12 width=10)
         Output: (ft2.a * 2), ft2.ctid
         Remote SQL: SELECT a, ctid FROM public.mytable_2 WHERE ((a = 5)) FOR UPDATE
(12 rows)

What do you say?

Two "remote SQL" under a single node would be confusing. Also the node is labelled as "Foreign Scan". It would be confusing to show an "UPDATE" command under this "scan" node.

BTW, I was experimenting with DMLs being executed on multiple FDW server under same transaction and found that the transactions may not be atomic (and may be inconsistent), if one or more of the server fails to commit while rest of them commit the transaction. The reason for this is, we do not "rollback" the already "committed" changes to the foreign server, if one or more of them fail to "commit" a transaction. With foreign tables under inheritance hierarchy a single DML can span across multiple servers and the result may not be atomic (and may be inconsistent). So, either we have to disable DMLs on an inheritance hierarchy which spans multiple servers. OR make sure that such transactions follow 2PC norms.
 

Sorry for the delay.

[1] http://www.postgresql.org/message-id/1316566782-sup-2678@alvh.no-ip.org

Best regards,
Etsuro Fujita



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/11/28 18:14), Ashutosh Bapat wrote:
> On Thu, Nov 27, 2014 at 3:52 PM, Etsuro Fujita
> <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:
>     Apart from the above, I noticed that the patch doesn't consider to
>     call ExplainForeignModify during EXPLAIN for an inherited
>     UPDATE/DELETE, as shown below (note that there are no UPDATE remote
>     queries displayed):

>     So, I'd like to modify explain.c to show those queries like this:

>     postgres=# explain verbose update parent set a = a * 2 where a = 5;
>                                           QUERY PLAN
>     ------------------------------__------------------------------__-------------------------
>       Update on public.parent  (cost=0.00..280.77 rows=25 width=10)
>         ->  Seq Scan on public.parent  (cost=0.00..0.00 rows=1 width=10)
>               Output: (parent.a * 2), parent.ctid
>               Filter: (parent.a = 5)
>         Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1
>         ->  Foreign Scan on public.ft1  (cost=100.00..140.38 rows=12
>     width=10)
>               Output: (ft1.a * 2), ft1.ctid
>               Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE ((a
>     = 5)) FOR UPDATE
>         Remote SQL: UPDATE public.mytable_2 SET a = $2 WHERE ctid = $1
>         ->  Foreign Scan on public.ft2  (cost=100.00..140.38 rows=12
>     width=10)
>               Output: (ft2.a * 2), ft2.ctid
>               Remote SQL: SELECT a, ctid FROM public.mytable_2 WHERE ((a
>     = 5)) FOR UPDATE
>     (12 rows)

> Two "remote SQL" under a single node would be confusing. Also the node
> is labelled as "Foreign Scan". It would be confusing to show an "UPDATE"
> command under this "scan" node.

I thought this as an extention of the existing (ie, non-inherited) case 
(see the below example) to the inherited case.

postgres=# explain verbose update ft1 set a = a * 2 where a = 5;                                     QUERY PLAN
------------------------------------------------------------------------------------- Update on public.ft1
(cost=100.00..140.38rows=12 width=10)   Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1   ->  Foreign
Scanon public.ft1  (cost=100.00..140.38 rows=12 width=10)         Output: (a * 2), ctid         Remote SQL: SELECT a,
ctidFROM public.mytable_1 WHERE ((a = 
 
5)) FOR UPDATE
(5 rows)

I think we should show update commands somewhere for the inherited case 
as for the non-inherited case.  Alternatives to this are welcome.

> BTW, I was experimenting with DMLs being executed on multiple FDW server
> under same transaction and found that the transactions may not be atomic
> (and may be inconsistent), if one or more of the server fails to commit
> while rest of them commit the transaction. The reason for this is, we do
> not "rollback" the already "committed" changes to the foreign server, if
> one or more of them fail to "commit" a transaction. With foreign tables
> under inheritance hierarchy a single DML can span across multiple
> servers and the result may not be atomic (and may be inconsistent). So,

IIUC, even the transactions over the local and the *single* remote 
server are not guaranteed to be executed atomically in the current form.  It is possible that the remote transaction
succeedsand the local one 
 
fails, for example, resulting in data inconsistency between the local 
and the remote.

> either we have to disable DMLs on an inheritance hierarchy which spans
> multiple servers. OR make sure that such transactions follow 2PC norms.

-1 for disabling update queries on such an inheritance hierarchy because 
I think we should leave that to the user's judgment.  And I think 2PC is 
definitely a separate patch.

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Ashutosh Bapat
Date:


On Tue, Dec 2, 2014 at 8:29 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/11/28 18:14), Ashutosh Bapat wrote:
On Thu, Nov 27, 2014 at 3:52 PM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:
    Apart from the above, I noticed that the patch doesn't consider to
    call ExplainForeignModify during EXPLAIN for an inherited
    UPDATE/DELETE, as shown below (note that there are no UPDATE remote
    queries displayed):

    So, I'd like to modify explain.c to show those queries like this:

    postgres=# explain verbose update parent set a = a * 2 where a = 5;
                                          QUERY PLAN
    ------------------------------__------------------------------__-------------------------
      Update on public.parent  (cost=0.00..280.77 rows=25 width=10)
        ->  Seq Scan on public.parent  (cost=0.00..0.00 rows=1 width=10)
              Output: (parent.a * 2), parent.ctid
              Filter: (parent.a = 5)
        Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1
        ->  Foreign Scan on public.ft1  (cost=100.00..140.38 rows=12
    width=10)
              Output: (ft1.a * 2), ft1.ctid
              Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE ((a
    = 5)) FOR UPDATE
        Remote SQL: UPDATE public.mytable_2 SET a = $2 WHERE ctid = $1
        ->  Foreign Scan on public.ft2  (cost=100.00..140.38 rows=12
    width=10)
              Output: (ft2.a * 2), ft2.ctid
              Remote SQL: SELECT a, ctid FROM public.mytable_2 WHERE ((a
    = 5)) FOR UPDATE
    (12 rows)

Two "remote SQL" under a single node would be confusing. Also the node
is labelled as "Foreign Scan". It would be confusing to show an "UPDATE"
command under this "scan" node.

I thought this as an extention of the existing (ie, non-inherited) case (see the below example) to the inherited case.

postgres=# explain verbose update ft1 set a = a * 2 where a = 5;
                                     QUERY PLAN
-------------------------------------------------------------------------------------
 Update on public.ft1  (cost=100.00..140.38 rows=12 width=10)
   Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1
   ->  Foreign Scan on public.ft1  (cost=100.00..140.38 rows=12 width=10)
         Output: (a * 2), ctid
         Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE ((a = 5)) FOR UPDATE
(5 rows)

I think we should show update commands somewhere for the inherited case as for the non-inherited case.  Alternatives to this are welcome.
This is not exactly extension of non-inheritance case. non-inheritance case doesn't show two remote SQLs under the same plan node. May be you can rename the label Remote SQL as Remote UPDATE/INSERT/DELETE (or something to that effect) for the DML command and the Foreign plan node should be renamed to Foreign access node or something to indicate that it does both the scan as well as DML. I am not keen about the actual terminology, but I think a reader of plan shouldn't get confused.

We can leave this for committer's judgement.

BTW, I was experimenting with DMLs being executed on multiple FDW server
under same transaction and found that the transactions may not be atomic
(and may be inconsistent), if one or more of the server fails to commit
while rest of them commit the transaction. The reason for this is, we do
not "rollback" the already "committed" changes to the foreign server, if
one or more of them fail to "commit" a transaction. With foreign tables
under inheritance hierarchy a single DML can span across multiple
servers and the result may not be atomic (and may be inconsistent). So,

IIUC, even the transactions over the local and the *single* remote server are not guaranteed to be executed atomically in the current form.  It is possible that the remote transaction succeeds and the local one fails, for example, resulting in data inconsistency between the local and the remote.

IIUC, while committing transactions involving a single remote server, the steps taken are as follows
1. the local changes are brought to PRE-COMMIT stage, which means that the transaction *will* succeed locally after successful completion of this phase,
2. COMMIT message is sent to the foreign server
3. If step two succeeds, local changes are committed and successful commit is conveyed to the client
4. if step two fails, local changes are rolled back and abort status is conveyed to the client
5. If step 1 itself fails, the remote changes are rolled back.
This is as per one phase commit protocol which guarantees ACID for single foreign data source. So, the changes involving local and a single foreign server seem to be atomic and consistent.

either we have to disable DMLs on an inheritance hierarchy which spans
multiple servers. OR make sure that such transactions follow 2PC norms.

-1 for disabling update queries on such an inheritance hierarchy because I think we should leave that to the user's judgment.  And I think 2PC is definitely a separate patch.

I agree that 2pc is much larger work and is subject for separate patch/es. But it may not be acceptable that changes made within a single command violate atomicity and consistency, which can not be controlled or altered by user intervention. Again, we can leave it for committer's judgement.

Marking this as "ready for committer".


Thanks,

Best regards,
Etsuro Fujita



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: inherit support for foreign tables

From
Ashutosh Bapat
Date:


On Wed, Dec 3, 2014 at 4:05 PM, Ashutosh Bapat <ashutosh.bapat@enterprisedb.com> wrote:


On Tue, Dec 2, 2014 at 8:29 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/11/28 18:14), Ashutosh Bapat wrote:
On Thu, Nov 27, 2014 at 3:52 PM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:
    Apart from the above, I noticed that the patch doesn't consider to
    call ExplainForeignModify during EXPLAIN for an inherited
    UPDATE/DELETE, as shown below (note that there are no UPDATE remote
    queries displayed):

    So, I'd like to modify explain.c to show those queries like this:

    postgres=# explain verbose update parent set a = a * 2 where a = 5;
                                          QUERY PLAN
    ------------------------------__------------------------------__-------------------------
      Update on public.parent  (cost=0.00..280.77 rows=25 width=10)
        ->  Seq Scan on public.parent  (cost=0.00..0.00 rows=1 width=10)
              Output: (parent.a * 2), parent.ctid
              Filter: (parent.a = 5)
        Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1
        ->  Foreign Scan on public.ft1  (cost=100.00..140.38 rows=12
    width=10)
              Output: (ft1.a * 2), ft1.ctid
              Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE ((a
    = 5)) FOR UPDATE
        Remote SQL: UPDATE public.mytable_2 SET a = $2 WHERE ctid = $1
        ->  Foreign Scan on public.ft2  (cost=100.00..140.38 rows=12
    width=10)
              Output: (ft2.a * 2), ft2.ctid
              Remote SQL: SELECT a, ctid FROM public.mytable_2 WHERE ((a
    = 5)) FOR UPDATE
    (12 rows)

Two "remote SQL" under a single node would be confusing. Also the node
is labelled as "Foreign Scan". It would be confusing to show an "UPDATE"
command under this "scan" node.

I thought this as an extention of the existing (ie, non-inherited) case (see the below example) to the inherited case.

postgres=# explain verbose update ft1 set a = a * 2 where a = 5;
                                     QUERY PLAN
-------------------------------------------------------------------------------------
 Update on public.ft1  (cost=100.00..140.38 rows=12 width=10)
   Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1
   ->  Foreign Scan on public.ft1  (cost=100.00..140.38 rows=12 width=10)
         Output: (a * 2), ctid
         Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE ((a = 5)) FOR UPDATE
(5 rows)

I think we should show update commands somewhere for the inherited case as for the non-inherited case.  Alternatives to this are welcome.
This is not exactly extension of non-inheritance case. non-inheritance case doesn't show two remote SQLs under the same plan node. May be you can rename the label Remote SQL as Remote UPDATE/INSERT/DELETE (or something to that effect) for the DML command and the Foreign plan node should be renamed to Foreign access node or something to indicate that it does both the scan as well as DML. I am not keen about the actual terminology, but I think a reader of plan shouldn't get confused.

We can leave this for committer's judgement.

BTW, I was experimenting with DMLs being executed on multiple FDW server
under same transaction and found that the transactions may not be atomic
(and may be inconsistent), if one or more of the server fails to commit
while rest of them commit the transaction. The reason for this is, we do
not "rollback" the already "committed" changes to the foreign server, if
one or more of them fail to "commit" a transaction. With foreign tables
under inheritance hierarchy a single DML can span across multiple
servers and the result may not be atomic (and may be inconsistent). So,

IIUC, even the transactions over the local and the *single* remote server are not guaranteed to be executed atomically in the current form.  It is possible that the remote transaction succeeds and the local one fails, for example, resulting in data inconsistency between the local and the remote.

IIUC, while committing transactions involving a single remote server, the steps taken are as follows
1. the local changes are brought to PRE-COMMIT stage, which means that the transaction *will* succeed locally after successful completion of this phase,
2. COMMIT message is sent to the foreign server
3. If step two succeeds, local changes are committed and successful commit is conveyed to the client
4. if step two fails, local changes are rolled back and abort status is conveyed to the client
5. If step 1 itself fails, the remote changes are rolled back.
This is as per one phase commit protocol which guarantees ACID for single foreign data source. So, the changes involving local and a single foreign server seem to be atomic and consistent.

either we have to disable DMLs on an inheritance hierarchy which spans
multiple servers. OR make sure that such transactions follow 2PC norms.

-1 for disabling update queries on such an inheritance hierarchy because I think we should leave that to the user's judgment.  And I think 2PC is definitely a separate patch.

I agree that 2pc is much larger work and is subject for separate patch/es. But it may not be acceptable that changes made within a single command violate atomicity and consistency, which can not be controlled or altered by user intervention. Again, we can leave it for committer's judgement.

Marking this as "ready for committer".

Sorry, I noticed in the commitfest app, that there are other reviewers as well. Let's wait for them to comment.
 


Thanks,

Best regards,
Etsuro Fujita



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/12/03 19:35), Ashutosh Bapat wrote:
> On Tue, Dec 2, 2014 at 8:29 AM, Etsuro Fujita
> <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:

> This is not exactly extension of non-inheritance case. non-inheritance
> case doesn't show two remote SQLs under the same plan node. May be you
> can rename the label Remote SQL as Remote UPDATE/INSERT/DELETE (or
> something to that effect) for the DML command and the Foreign plan node
> should be renamed to Foreign access node or something to indicate that
> it does both the scan as well as DML. I am not keen about the actual
> terminology, but I think a reader of plan shouldn't get confused.
>
> We can leave this for committer's judgement.

Thanks for the proposal!  I think that would be a good idea.  But I 
think there would be another idea.  An example will be shown below.  We 
show the update commands below the ModifyTable node, not above the 
corresponding ForeignScan nodes, so maybe less confusing.  If there are 
no objections of you and others, I'll update the patch this way.

postgres=# explain verbose update parent set a = a * 2 where a = 5;                                     QUERY PLAN
------------------------------------------------------------------------------------- Update on public.parent
(cost=0.00..280.77rows=25 width=10)   On public.ft1     Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1
On public.ft2     Remote SQL: UPDATE public.mytable_2 SET a = $2 WHERE ctid = $1   ->  Seq Scan on public.parent
(cost=0.00..0.00rows=1 width=10)         Output: (parent.a * 2), parent.ctid         Filter: (parent.a = 5)   ->
ForeignScan on public.ft1  (cost=100.00..140.38 rows=12 width=10)         Output: (ft1.a * 2), ft1.ctid         Remote
SQL:SELECT a, ctid FROM public.mytable_1 WHERE ((a = 
 
5)) FOR UPDATE   ->  Foreign Scan on public.ft2  (cost=100.00..140.38 rows=12 width=10)         Output: (ft2.a * 2),
ft2.ctid        Remote SQL: SELECT a, ctid FROM public.mytable_2 WHERE ((a = 
 
5)) FOR UPDATE
(12 rows)

>     IIUC, even the transactions over the local and the *single* remote
>     server are not guaranteed to be executed atomically in the current
>     form.  It is possible that the remote transaction succeeds and the
>     local one fails, for example, resulting in data inconsistency
>     between the local and the remote.

> IIUC, while committing transactions involving a single remote server,
> the steps taken are as follows
> 1. the local changes are brought to PRE-COMMIT stage, which means that
> the transaction *will* succeed locally after successful completion of
> this phase,
> 2. COMMIT message is sent to the foreign server
> 3. If step two succeeds, local changes are committed and successful
> commit is conveyed to the client
> 4. if step two fails, local changes are rolled back and abort status is
> conveyed to the client
> 5. If step 1 itself fails, the remote changes are rolled back.
> This is as per one phase commit protocol which guarantees ACID for
> single foreign data source. So, the changes involving local and a single
> foreign server seem to be atomic and consistent.

Really?  Maybe I'm missing something, but I don't think the current 
implementation for committing transactions has such a mechanism stated 
in step 1.  So, I think it's possible that the local transaction fails 
in step3 while the remote transaction succeeds, as mentioned above.

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Ashutosh Bapat
Date:


On Thu, Dec 4, 2014 at 9:05 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/12/03 19:35), Ashutosh Bapat wrote:
On Tue, Dec 2, 2014 at 8:29 AM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:

This is not exactly extension of non-inheritance case. non-inheritance
case doesn't show two remote SQLs under the same plan node. May be you
can rename the label Remote SQL as Remote UPDATE/INSERT/DELETE (or
something to that effect) for the DML command and the Foreign plan node
should be renamed to Foreign access node or something to indicate that
it does both the scan as well as DML. I am not keen about the actual
terminology, but I think a reader of plan shouldn't get confused.

We can leave this for committer's judgement.

Thanks for the proposal!  I think that would be a good idea.  But I think there would be another idea.  An example will be shown below.  We show the update commands below the ModifyTable node, not above the corresponding ForeignScan nodes, so maybe less confusing.  If there are no objections of you and others, I'll update the patch this way.

postgres=# explain verbose update parent set a = a * 2 where a = 5;
                                     QUERY PLAN
-------------------------------------------------------------------------------------
 Update on public.parent  (cost=0.00..280.77 rows=25 width=10)
   On public.ft1
     Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1
   On public.ft2
     Remote SQL: UPDATE public.mytable_2 SET a = $2 WHERE ctid = $1
   ->  Seq Scan on public.parent  (cost=0.00..0.00 rows=1 width=10)
         Output: (parent.a * 2), parent.ctid
         Filter: (parent.a = 5)
   ->  Foreign Scan on public.ft1  (cost=100.00..140.38 rows=12 width=10)
         Output: (ft1.a * 2), ft1.ctid
         Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE ((a = 5)) FOR UPDATE
   ->  Foreign Scan on public.ft2  (cost=100.00..140.38 rows=12 width=10)
         Output: (ft2.a * 2), ft2.ctid
         Remote SQL: SELECT a, ctid FROM public.mytable_2 WHERE ((a = 5)) FOR UPDATE
(12 rows)


Looks better.
 
    IIUC, even the transactions over the local and the *single* remote
    server are not guaranteed to be executed atomically in the current
    form.  It is possible that the remote transaction succeeds and the
    local one fails, for example, resulting in data inconsistency
    between the local and the remote.

IIUC, while committing transactions involving a single remote server,
the steps taken are as follows
1. the local changes are brought to PRE-COMMIT stage, which means that
the transaction *will* succeed locally after successful completion of
this phase,
2. COMMIT message is sent to the foreign server
3. If step two succeeds, local changes are committed and successful
commit is conveyed to the client
4. if step two fails, local changes are rolled back and abort status is
conveyed to the client
5. If step 1 itself fails, the remote changes are rolled back.
This is as per one phase commit protocol which guarantees ACID for
single foreign data source. So, the changes involving local and a single
foreign server seem to be atomic and consistent.

Really?  Maybe I'm missing something, but I don't think the current implementation for committing transactions has such a mechanism stated in step 1.  So, I think it's possible that the local transaction fails in step3 while the remote transaction succeeds, as mentioned above.


PFA a script attached which shows this. You may want to check the code in pgfdw_xact_callback() for actions taken by postgres_fdw on various events. CommitTransaction() for how those events are generated. The code there complies with the sequence above.
 

Thanks,

Best regards,
Etsuro Fujita



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: inherit support for foreign tables

From
Ashutosh Bapat
Date:
Sorry, here's the script.

On Thu, Dec 4, 2014 at 10:00 AM, Ashutosh Bapat <ashutosh.bapat@enterprisedb.com> wrote:


On Thu, Dec 4, 2014 at 9:05 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/12/03 19:35), Ashutosh Bapat wrote:
On Tue, Dec 2, 2014 at 8:29 AM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:

This is not exactly extension of non-inheritance case. non-inheritance
case doesn't show two remote SQLs under the same plan node. May be you
can rename the label Remote SQL as Remote UPDATE/INSERT/DELETE (or
something to that effect) for the DML command and the Foreign plan node
should be renamed to Foreign access node or something to indicate that
it does both the scan as well as DML. I am not keen about the actual
terminology, but I think a reader of plan shouldn't get confused.

We can leave this for committer's judgement.

Thanks for the proposal!  I think that would be a good idea.  But I think there would be another idea.  An example will be shown below.  We show the update commands below the ModifyTable node, not above the corresponding ForeignScan nodes, so maybe less confusing.  If there are no objections of you and others, I'll update the patch this way.

postgres=# explain verbose update parent set a = a * 2 where a = 5;
                                     QUERY PLAN
-------------------------------------------------------------------------------------
 Update on public.parent  (cost=0.00..280.77 rows=25 width=10)
   On public.ft1
     Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1
   On public.ft2
     Remote SQL: UPDATE public.mytable_2 SET a = $2 WHERE ctid = $1
   ->  Seq Scan on public.parent  (cost=0.00..0.00 rows=1 width=10)
         Output: (parent.a * 2), parent.ctid
         Filter: (parent.a = 5)
   ->  Foreign Scan on public.ft1  (cost=100.00..140.38 rows=12 width=10)
         Output: (ft1.a * 2), ft1.ctid
         Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE ((a = 5)) FOR UPDATE
   ->  Foreign Scan on public.ft2  (cost=100.00..140.38 rows=12 width=10)
         Output: (ft2.a * 2), ft2.ctid
         Remote SQL: SELECT a, ctid FROM public.mytable_2 WHERE ((a = 5)) FOR UPDATE
(12 rows)


Looks better.
 
    IIUC, even the transactions over the local and the *single* remote
    server are not guaranteed to be executed atomically in the current
    form.  It is possible that the remote transaction succeeds and the
    local one fails, for example, resulting in data inconsistency
    between the local and the remote.

IIUC, while committing transactions involving a single remote server,
the steps taken are as follows
1. the local changes are brought to PRE-COMMIT stage, which means that
the transaction *will* succeed locally after successful completion of
this phase,
2. COMMIT message is sent to the foreign server
3. If step two succeeds, local changes are committed and successful
commit is conveyed to the client
4. if step two fails, local changes are rolled back and abort status is
conveyed to the client
5. If step 1 itself fails, the remote changes are rolled back.
This is as per one phase commit protocol which guarantees ACID for
single foreign data source. So, the changes involving local and a single
foreign server seem to be atomic and consistent.

Really?  Maybe I'm missing something, but I don't think the current implementation for committing transactions has such a mechanism stated in step 1.  So, I think it's possible that the local transaction fails in step3 while the remote transaction succeeds, as mentioned above.


PFA a script attached which shows this. You may want to check the code in pgfdw_xact_callback() for actions taken by postgres_fdw on various events. CommitTransaction() for how those events are generated. The code there complies with the sequence above.
 

Thanks,

Best regards,
Etsuro Fujita



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Attachment

Re: inherit support for foreign tables

From
Noah Misch
Date:
On Thu, Dec 04, 2014 at 10:00:14AM +0530, Ashutosh Bapat wrote:
> On Thu, Dec 4, 2014 at 9:05 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
> > (2014/12/03 19:35), Ashutosh Bapat wrote:
> >> On Tue, Dec 2, 2014 at 8:29 AM, Etsuro Fujita
> >> <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:
> >      IIUC, even the transactions over the local and the *single* remote
> >>     server are not guaranteed to be executed atomically in the current
> >>     form.  It is possible that the remote transaction succeeds and the
> >>     local one fails, for example, resulting in data inconsistency
> >>     between the local and the remote.
> >>
> >
> >  IIUC, while committing transactions involving a single remote server,
> >> the steps taken are as follows
> >> 1. the local changes are brought to PRE-COMMIT stage, which means that
> >> the transaction *will* succeed locally after successful completion of
> >> this phase,
> >> 2. COMMIT message is sent to the foreign server
> >> 3. If step two succeeds, local changes are committed and successful
> >> commit is conveyed to the client
> >> 4. if step two fails, local changes are rolled back and abort status is
> >> conveyed to the client
> >> 5. If step 1 itself fails, the remote changes are rolled back.
> >> This is as per one phase commit protocol which guarantees ACID for
> >> single foreign data source. So, the changes involving local and a single
> >> foreign server seem to be atomic and consistent.
> >>
> >
> > Really?  Maybe I'm missing something, but I don't think the current
> > implementation for committing transactions has such a mechanism stated in
> > step 1.  So, I think it's possible that the local transaction fails in
> > step3 while the remote transaction succeeds, as mentioned above.
> >
> >
> PFA a script attached which shows this. You may want to check the code in
> pgfdw_xact_callback() for actions taken by postgres_fdw on various events.
> CommitTransaction() for how those events are generated. The code there
> complies with the sequence above.

While postgres_fdw delays remote commit to eliminate many causes for having
one server commit while another aborts, other causes remain.  The local
transaction could still fail due to WAL I/O problems or a system crash.  2PC
is the reliable answer, but that was explicitly out of scope for the initial
postgres_fdw write support.  Does this inheritance patch add any atomicity
problem that goes away when one breaks up the inheritance hierarchy and
UPDATEs each table separately?  If not, this limitation is okay.



Re: inherit support for foreign tables

From
David Fetter
Date:
On Thu, Dec 04, 2014 at 12:35:54PM +0900, Etsuro Fujita wrote:
> (2014/12/03 19:35), Ashutosh Bapat wrote:
> >On Tue, Dec 2, 2014 at 8:29 AM, Etsuro Fujita
> ><fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:
> 
> >This is not exactly extension of non-inheritance case. non-inheritance
> >case doesn't show two remote SQLs under the same plan node. May be you
> >can rename the label Remote SQL as Remote UPDATE/INSERT/DELETE (or
> >something to that effect) for the DML command and the Foreign plan node
> >should be renamed to Foreign access node or something to indicate that
> >it does both the scan as well as DML. I am not keen about the actual
> >terminology, but I think a reader of plan shouldn't get confused.
> >
> >We can leave this for committer's judgement.
> 
> Thanks for the proposal!  I think that would be a good idea.  But I think
> there would be another idea.  An example will be shown below.  We show the
> update commands below the ModifyTable node, not above the corresponding
> ForeignScan nodes, so maybe less confusing.  If there are no objections of
> you and others, I'll update the patch this way.
> 
> postgres=# explain verbose update parent set a = a * 2 where a = 5;
>                                      QUERY PLAN
> -------------------------------------------------------------------------------------
>  Update on public.parent  (cost=0.00..280.77 rows=25 width=10)
>    On public.ft1
>      Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1      ^^^^^^^^^^
It occurs to me that the command generated by the FDW might well not
be SQL at all, as is the case with file_fdw and anything else that
talks to a NoSQL engine.

Would it be reasonable to call this "Remote command" or something
similarly generic?

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/12/07 2:02), David Fetter wrote:
> On Thu, Dec 04, 2014 at 12:35:54PM +0900, Etsuro Fujita wrote:
>> But I think
>> there would be another idea.  An example will be shown below.  We show the
>> update commands below the ModifyTable node, not above the corresponding
>> ForeignScan nodes, so maybe less confusing.  If there are no objections of
>> you and others, I'll update the patch this way.
>>
>> postgres=# explain verbose update parent set a = a * 2 where a = 5;
>>                                       QUERY PLAN
>> -------------------------------------------------------------------------------------
>>   Update on public.parent  (cost=0.00..280.77 rows=25 width=10)
>>     On public.ft1
>>       Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1
>         ^^^^^^^^^^
> It occurs to me that the command generated by the FDW might well not
> be SQL at all, as is the case with file_fdw and anything else that
> talks to a NoSQL engine.
>
> Would it be reasonable to call this "Remote command" or something
> similarly generic?

Yeah, but I'd like to propose that this line is shown by the FDW API 
(ie, ExplainForeignModify) as in non-inherited update cases, so that the 
FDW developer can choose the right name.  As for "On public.ft1", I'd 
like to propose that the FDW API also show that by calling a function 
for that introduced into the PG core (Would it be better to use "For" 
rather than "On"?).

Sorry, my explanation was not enough.

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Ashutosh Bapat
Date:


On Sat, Dec 6, 2014 at 9:21 PM, Noah Misch <noah@leadboat.com> wrote:
On Thu, Dec 04, 2014 at 10:00:14AM +0530, Ashutosh Bapat wrote:
> On Thu, Dec 4, 2014 at 9:05 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
> > (2014/12/03 19:35), Ashutosh Bapat wrote:
> >> On Tue, Dec 2, 2014 at 8:29 AM, Etsuro Fujita
> >> <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:
> >      IIUC, even the transactions over the local and the *single* remote
> >>     server are not guaranteed to be executed atomically in the current
> >>     form.  It is possible that the remote transaction succeeds and the
> >>     local one fails, for example, resulting in data inconsistency
> >>     between the local and the remote.
> >>
> >
> >  IIUC, while committing transactions involving a single remote server,
> >> the steps taken are as follows
> >> 1. the local changes are brought to PRE-COMMIT stage, which means that
> >> the transaction *will* succeed locally after successful completion of
> >> this phase,
> >> 2. COMMIT message is sent to the foreign server
> >> 3. If step two succeeds, local changes are committed and successful
> >> commit is conveyed to the client
> >> 4. if step two fails, local changes are rolled back and abort status is
> >> conveyed to the client
> >> 5. If step 1 itself fails, the remote changes are rolled back.
> >> This is as per one phase commit protocol which guarantees ACID for
> >> single foreign data source. So, the changes involving local and a single
> >> foreign server seem to be atomic and consistent.
> >>
> >
> > Really?  Maybe I'm missing something, but I don't think the current
> > implementation for committing transactions has such a mechanism stated in
> > step 1.  So, I think it's possible that the local transaction fails in
> > step3 while the remote transaction succeeds, as mentioned above.
> >
> >
> PFA a script attached which shows this. You may want to check the code in
> pgfdw_xact_callback() for actions taken by postgres_fdw on various events.
> CommitTransaction() for how those events are generated. The code there
> complies with the sequence above.

While postgres_fdw delays remote commit to eliminate many causes for having
one server commit while another aborts, other causes remain.  The local
transaction could still fail due to WAL I/O problems or a system crash.  2PC
is the reliable answer, but that was explicitly out of scope for the initial
postgres_fdw write support.  Does this inheritance patch add any atomicity
problem that goes away when one breaks up the inheritance hierarchy and
UPDATEs each table separately?  If not, this limitation is okay.

If the UPDATES crafted after breaking up the inheritance hierarchy are needed to be run within the same transaction (as the UPDATE on inheritance hierarchy would do), yes, there is atomicity problem.

--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/12/08 15:17), Ashutosh Bapat wrote:
> On Sat, Dec 6, 2014 at 9:21 PM, Noah Misch <noah@leadboat.com
> <mailto:noah@leadboat.com>> wrote:
>     Does this inheritance patch add any
>     atomicity
>     problem that goes away when one breaks up the inheritance hierarchy and
>     UPDATEs each table separately?  If not, this limitation is okay.

> If the UPDATES crafted after breaking up the inheritance hierarchy are
> needed to be run within the same transaction (as the UPDATE on
> inheritance hierarchy would do), yes, there is atomicity problem.

ISTM that your concern would basically a known problem.  Consider the 
following transaction.

BEGIN TRANSACTION;
UPDATE foo SET a = 100;  -- updates on table foo in remote server1
UPDATE bar SET a = 100;  -- updates on table bar in remote server2
COMMIT TRANSACTION;

This transaction would cause the atomicity problem if 
pgfdw_xact_callback() for XACT_EVENT_PRE_COMMIT for foo succeeded and 
then that for bar failed during CommitTransaction().

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
Hi Ashutosh,

Thanks for the review!

(2014/11/28 18:14), Ashutosh Bapat wrote:
> On Thu, Nov 27, 2014 at 3:52 PM, Etsuro Fujita
> <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:
>     (2014/11/17 17:55), Ashutosh Bapat wrote:
>         Here are my review comments for patch fdw-inh-3.patch.

>         Tests
>         -------
>         1. It seems like you have copied from testcase inherit.sql to
>         postgres_fdw testcase. That's a good thing, but it makes the
>         test quite
>         long. May be we should have two tests in postgres_fdw contrib
>         module,
>         one for simple cases, and other for inheritance. What do you say?

>     IMO, the test is not so time-consuming, so it doesn't seem worth
>     splitting it into two.

> I am not worried about the timing but I am worried about the length of
> the file and hence ease of debugging in case we find any issues there.
> We will leave that for the commiter to decide.

OK

>         Documentation
>         --------------------
>         1. The change in ddl.sgml
>         -        We will refer to the child tables as partitions, though
>         they
>         -        are in every way normal <productname>PostgreSQL</> tables.
>         +        We will refer to the child tables as partitions, though
>         we assume
>         +        that they are normal <productname>PostgreSQL</> tables.
>
>         adds phrase "we assume that", which confuses the intention
>         behind the
>         sentence. The original text is intended to highlight the equivalence
>         between "partition" and "normal table", where as the addition
>         esp. the
>         word "assume" weakens that equivalence. Instead now we have to
>         highlight
>         the equivalence between "partition" and "normal or foreign
>         table". The
>         wording similar to "though they are either normal or foreign tables"
>         should be used there.

>     You are right, but I feel that there is something out of place in
>     saying that there (5.10.2. Implementing Partitioning) because the
>     procedure there has been written based on normal tables only.  Put
>     another way, if we say that, I think we'd need to add more docs,
>     describing the syntax and/or the corresponding examples for
>     foreign-table cases.  But I'd like to leave that for another patch.
>     So, how about the wording "we assume *here* that", instead of "we
>     assume that", as I added the following notice in the previous
>     section (5.10.1. Overview)?
>
>     @@ -2650,7 +2669,10 @@ VALUES ('Albany', NULL, NULL, 'NY');
>           table of a single parent table.  The parent table itself is
>     normally
>           empty; it exists just to represent the entire data set.  You
>     should be
>           familiar with inheritance (see <xref linkend="ddl-inherit">)
>     before
>     -    attempting to set up partitioning.
>     +    attempting to set up partitioning.  (The setup and management of
>     +    partitioned tables illustrated in this section assume that each
>     +    partition is a normal table.  However, you can do that in a
>     similar way
>     +    for cases where some or all partitions are foreign tables.)

> This looks ok, though, I would like to see final version of the
> document. But I think, we will leave that for committer to handle.

OK

>         2. The wording "some kind of optimization" gives vague picture.
>         May be
>         it can be worded as "Since the constraints are assumed to be
>         true, they
>         are used in constraint-based query optimization like constraint
>         exclusion for partitioned tables.".
>         +    Those constraints are used in some kind of query
>         optimization such
>         +    as constraint exclusion for partitioned tables (see
>         +    <xref linkend="ddl-partitioning">).

>     Will follow your revision.

Done.

>         Code
>         -------
>         1. In the following change
>         +/*
>             * acquire_inherited_sample_rows -- acquire sample rows from
>         inheritance tree
>             *
>             * This has the same API as acquire_sample_rows, except that
>         rows are
>             * collected from all inheritance children as well as the
>         specified table.
>         - * We fail and return zero if there are no inheritance children.
>         + * We fail and return zero if there are no inheritance children or
>         there are
>         + * inheritance children that foreign tables.
>
>         The addition should be "there are inheritance children that *are all
>         *foreign tables. Note the addition "are all".

>     Sorry, I incorrectly wrote the comment.  What I tried to write is
>     "We fail and return zero if there are no inheritance children or if
>     we are not in VAC_MODE_SINGLE case and inheritance tree contains at
>     least one foreign table.".

> You might want to use "English" description of VAC_MODE_SINGLE instead
> of that macro in the comment, so that reader doesn't have to look up
> VAC_MODE_SINGLE. But I think, we will leave this for the committer.

I corrected the comments and translated the macro into the English
description.

>         2. The function has_foreign() be better named
>         has_foreign_child()? This

>     How about "has_foreign_table"?

> has_foreign_child() would be better, since these are "children" in the
> inheritance hierarchy and not mere "table"s.

Done.  But I renamed it to has_foreign_children() because it sounds more
natural at least to me.

>         function loops through all the tableoids passed even after it
>         has found
>         a foreign table. Why can't we return true immediately after
>         finding the
>         first foreign table, unless the side effects of heap_open() on
>         all the
>         table are required. But I don't see that to be the case, since these
>         tables are locked already through a previous call to
>         heap_open(). In the

>     Good catch!  Will fix.

>         same function instead of argument name parentrelId may be we
>         should use
>         name parent_oid, so that we use same notation for parent and
>         child table
>         OIDs.

>     Will fix.

Done.

>         3.  Regarding enum VacuumMode - it's being used only in case of
>         acquire_inherited_sample_rows(__) and that too, to check only a
>         single
>         value of the three defined there. May be we should just infer
>         that value
>         inside acquire_inherited_sample_rows(__) or pass a boolean true
>         or false
>         from do_analyse_rel() based on the VacuumStmt. I do not see need
>         for a
>         separate three value enum of which only one value is used and
>         also to
>         pass it down from vacuum() by changing signatures of the minion
>         functions.

>     I introduced that for possible future use.  See the discussion in [1].

> Will leave it for the commiter to decide.

I noticed that the signatures need not to be modified, as you said.
Thanks for pointing that out!  So, I revised the patch not to change the
signatures, though I left the enum, renaming it to AnalyzeMode.  Let's
have the committer's review.

>         4. In postgresGetForeignPlan(), the code added by this patch is
>         required
>         to handle the case when the row mark is placed on a parent table and
>         hence is required to be applied on the child table. We need a
>         comment
>         explaining this. Otherwise, the three step process to get the
>         row mark
>         information isn't clear for a reader.

>     Will add the comment.

Done.

>         5. In expand_inherited_rtentry() why do you need a separate variable
>         hasForeign? Instead of using that variable, you can actually
>         set/reset
>         rte->hasForeign since existence of even a single foreign child would
>         mark that member as true. - After typing this, I got the answer
>         when I
>         looked at the function code. Every child's RTE is initially a
>         copy of
>         parent's RTE and hence hasForeign status would be inherited by every
>         child after the first foreign child. I think, this reasoning
>         should be
>         added as comment before assignment to rte->hasForeign at the end
>         of the
>         function.

>     As you mentioned, I think we could set rte->hasForeign directly
>     during the scan for the inheritance set, without the separate
>     variable, hasForeign.  But ISTM that it'd improve code readability
>     to set rte->hasForeign using the separate variable at the end of the
>     function because rte->hasForeign has its meaning only when rte->inh
>     is true and because we know whether rte->inh is true, at the end of
>     the function.

> Fine. Please use "hasForeignChild" instead of just "hasForeign" without
> a clue as to what is "foreign" here.

Done.  But I renamed it to "hasForeignChildren".

>         6. The tests in foreign_data.sql are pretty extensive. Thanks
>         for that.
>         I think, we should also check the effect of each of the following
>         command using \d on appropriate tables.
>         +CREATE FOREIGN TABLE ft2 () INHERITS (pt1)
>         +  SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted"
>         'value');
>         +ALTER FOREIGN TABLE ft2 NO INHERIT pt1;
>         +DROP FOREIGN TABLE ft2;
>         +CREATE FOREIGN TABLE ft2 (
>         +   c1 integer NOT NULL,
>         +   c2 text,
>         +   c3 date
>         +) SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted"
>         'value');
>         +ALTER FOREIGN TABLE ft2 INHERIT pt1;

>     Will fix.

Done.

>     Apart from the above, I noticed that the patch doesn't consider to
>     call ExplainForeignModify during EXPLAIN for an inherited
>     UPDATE/DELETE, as shown below (note that there are no UPDATE remote
>     queries displayed):

Since there seems to be no objections, I updated the patch as proposed
upthread.  Here is an example.

postgres=# explain (format text, verbose) update parent as p set a = a *
2 returning *;
                                    QUERY PLAN
--------------------------------------------------------------------------------
  Update on public.parent p  (cost=0.00..202.33 rows=11 width=10)
    Output: p.a
    For public.ft1 p_1
      Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1
RETURNING a
    For public.ft2 p_2
      Remote SQL: UPDATE public.mytable_2 SET a = $2 WHERE ctid = $1
RETURNING a
    ->  Seq Scan on public.parent p  (cost=0.00..0.00 rows=1 width=10)
          Output: (p.a * 2), p.ctid
    ->  Foreign Scan on public.ft1 p_1  (cost=100.00..101.16 rows=5
width=10)
          Output: (p_1.a * 2), p_1.ctid
          Remote SQL: SELECT a, ctid FROM public.mytable_1 FOR UPDATE
    ->  Foreign Scan on public.ft2 p_2  (cost=100.00..101.16 rows=5
width=10)
          Output: (p_2.a * 2), p_2.ctid
          Remote SQL: SELECT a, ctid FROM public.mytable_2 FOR UPDATE
(14 rows)

Other changes:
* revised regression tests for contrib/file_fdw to refer to tableoid.
* revised docs a bit further.

Attached are updated patches.  Patch fdw-inh-5.patch has been created on
top of patch fdw-chk-5.patch.  Patch fdw-chk-5.patch is basically the
same as the previous one fdw-chk-4.patch, but I slightly modified that
one.  The changes are the following.
* added to foreign_data.sql more tests for your comments.
* revised docs on ALTER FOREIGN TABLE a bit further.

Thanks,

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Ashutosh Bapat
Date:
We haven't heard anything from Horiguchi-san and Hanada-san for almost a week. So, I am fine marking it as "ready for committer". What do you say?


On Wed, Dec 10, 2014 at 8:48 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
Hi Ashutosh,

Thanks for the review!

(2014/11/28 18:14), Ashutosh Bapat wrote:
On Thu, Nov 27, 2014 at 3:52 PM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:
    (2014/11/17 17:55), Ashutosh Bapat wrote:
        Here are my review comments for patch fdw-inh-3.patch.

        Tests
        -------
        1. It seems like you have copied from testcase inherit.sql to
        postgres_fdw testcase. That's a good thing, but it makes the
        test quite
        long. May be we should have two tests in postgres_fdw contrib
        module,
        one for simple cases, and other for inheritance. What do you say?

    IMO, the test is not so time-consuming, so it doesn't seem worth
    splitting it into two.

I am not worried about the timing but I am worried about the length of
the file and hence ease of debugging in case we find any issues there.
We will leave that for the commiter to decide.

OK


        Documentation
        --------------------
        1. The change in ddl.sgml
        -        We will refer to the child tables as partitions, though
        they
        -        are in every way normal <productname>PostgreSQL</> tables.
        +        We will refer to the child tables as partitions, though
        we assume
        +        that they are normal <productname>PostgreSQL</> tables.

        adds phrase "we assume that", which confuses the intention
        behind the
        sentence. The original text is intended to highlight the equivalence
        between "partition" and "normal table", where as the addition
        esp. the
        word "assume" weakens that equivalence. Instead now we have to
        highlight
        the equivalence between "partition" and "normal or foreign
        table". The
        wording similar to "though they are either normal or foreign tables"
        should be used there.

    You are right, but I feel that there is something out of place in
    saying that there (5.10.2. Implementing Partitioning) because the
    procedure there has been written based on normal tables only.  Put
    another way, if we say that, I think we'd need to add more docs,
    describing the syntax and/or the corresponding examples for
    foreign-table cases.  But I'd like to leave that for another patch.
    So, how about the wording "we assume *here* that", instead of "we
    assume that", as I added the following notice in the previous
    section (5.10.1. Overview)?

    @@ -2650,7 +2669,10 @@ VALUES ('Albany', NULL, NULL, 'NY');
          table of a single parent table.  The parent table itself is
    normally
          empty; it exists just to represent the entire data set.  You
    should be
          familiar with inheritance (see <xref linkend="ddl-inherit">)
    before
    -    attempting to set up partitioning.
    +    attempting to set up partitioning.  (The setup and management of
    +    partitioned tables illustrated in this section assume that each
    +    partition is a normal table.  However, you can do that in a
    similar way
    +    for cases where some or all partitions are foreign tables.)

This looks ok, though, I would like to see final version of the
document. But I think, we will leave that for committer to handle.

OK

        2. The wording "some kind of optimization" gives vague picture.
        May be
        it can be worded as "Since the constraints are assumed to be
        true, they
        are used in constraint-based query optimization like constraint
        exclusion for partitioned tables.".
        +    Those constraints are used in some kind of query
        optimization such
        +    as constraint exclusion for partitioned tables (see
        +    <xref linkend="ddl-partitioning">).

    Will follow your revision.

Done.

        Code
        -------
        1. In the following change
        +/*
            * acquire_inherited_sample_rows -- acquire sample rows from
        inheritance tree
            *
            * This has the same API as acquire_sample_rows, except that
        rows are
            * collected from all inheritance children as well as the
        specified table.
        - * We fail and return zero if there are no inheritance children.
        + * We fail and return zero if there are no inheritance children or
        there are
        + * inheritance children that foreign tables.

        The addition should be "there are inheritance children that *are all
        *foreign tables. Note the addition "are all".

    Sorry, I incorrectly wrote the comment.  What I tried to write is
    "We fail and return zero if there are no inheritance children or if
    we are not in VAC_MODE_SINGLE case and inheritance tree contains at
    least one foreign table.".

You might want to use "English" description of VAC_MODE_SINGLE instead
of that macro in the comment, so that reader doesn't have to look up
VAC_MODE_SINGLE. But I think, we will leave this for the committer.

I corrected the comments and translated the macro into the English description.

        2. The function has_foreign() be better named
        has_foreign_child()? This

    How about "has_foreign_table"?

has_foreign_child() would be better, since these are "children" in the
inheritance hierarchy and not mere "table"s.

Done.  But I renamed it to has_foreign_children() because it sounds more natural at least to me.

        function loops through all the tableoids passed even after it
        has found
        a foreign table. Why can't we return true immediately after
        finding the
        first foreign table, unless the side effects of heap_open() on
        all the
        table are required. But I don't see that to be the case, since these
        tables are locked already through a previous call to
        heap_open(). In the

    Good catch!  Will fix.

        same function instead of argument name parentrelId may be we
        should use
        name parent_oid, so that we use same notation for parent and
        child table
        OIDs.

    Will fix.

Done.

        3.  Regarding enum VacuumMode - it's being used only in case of
        acquire_inherited_sample_rows(__) and that too, to check only a
        single
        value of the three defined there. May be we should just infer
        that value
        inside acquire_inherited_sample_rows(__) or pass a boolean true
        or false
        from do_analyse_rel() based on the VacuumStmt. I do not see need
        for a
        separate three value enum of which only one value is used and
        also to
        pass it down from vacuum() by changing signatures of the minion
        functions.

    I introduced that for possible future use.  See the discussion in [1].

Will leave it for the commiter to decide.

I noticed that the signatures need not to be modified, as you said. Thanks for pointing that out!  So, I revised the patch not to change the signatures, though I left the enum, renaming it to AnalyzeMode.  Let's have the committer's review.

        4. In postgresGetForeignPlan(), the code added by this patch is
        required
        to handle the case when the row mark is placed on a parent table and
        hence is required to be applied on the child table. We need a
        comment
        explaining this. Otherwise, the three step process to get the
        row mark
        information isn't clear for a reader.

    Will add the comment.

Done.

        5. In expand_inherited_rtentry() why do you need a separate variable
        hasForeign? Instead of using that variable, you can actually
        set/reset
        rte->hasForeign since existence of even a single foreign child would
        mark that member as true. - After typing this, I got the answer
        when I
        looked at the function code. Every child's RTE is initially a
        copy of
        parent's RTE and hence hasForeign status would be inherited by every
        child after the first foreign child. I think, this reasoning
        should be
        added as comment before assignment to rte->hasForeign at the end
        of the
        function.

    As you mentioned, I think we could set rte->hasForeign directly
    during the scan for the inheritance set, without the separate
    variable, hasForeign.  But ISTM that it'd improve code readability
    to set rte->hasForeign using the separate variable at the end of the
    function because rte->hasForeign has its meaning only when rte->inh
    is true and because we know whether rte->inh is true, at the end of
    the function.

Fine. Please use "hasForeignChild" instead of just "hasForeign" without
a clue as to what is "foreign" here.

Done.  But I renamed it to "hasForeignChildren".

        6. The tests in foreign_data.sql are pretty extensive. Thanks
        for that.
        I think, we should also check the effect of each of the following
        command using \d on appropriate tables.
        +CREATE FOREIGN TABLE ft2 () INHERITS (pt1)
        +  SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted"
        'value');
        +ALTER FOREIGN TABLE ft2 NO INHERIT pt1;
        +DROP FOREIGN TABLE ft2;
        +CREATE FOREIGN TABLE ft2 (
        +   c1 integer NOT NULL,
        +   c2 text,
        +   c3 date
        +) SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted"
        'value');
        +ALTER FOREIGN TABLE ft2 INHERIT pt1;

    Will fix.

Done.

    Apart from the above, I noticed that the patch doesn't consider to
    call ExplainForeignModify during EXPLAIN for an inherited
    UPDATE/DELETE, as shown below (note that there are no UPDATE remote
    queries displayed):

Since there seems to be no objections, I updated the patch as proposed upthread.  Here is an example.

postgres=# explain (format text, verbose) update parent as p set a = a * 2 returning *;
                                   QUERY PLAN
--------------------------------------------------------------------------------
 Update on public.parent p  (cost=0.00..202.33 rows=11 width=10)
   Output: p.a
   For public.ft1 p_1
     Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1 RETURNING a
   For public.ft2 p_2
     Remote SQL: UPDATE public.mytable_2 SET a = $2 WHERE ctid = $1 RETURNING a
   ->  Seq Scan on public.parent p  (cost=0.00..0.00 rows=1 width=10)
         Output: (p.a * 2), p.ctid
   ->  Foreign Scan on public.ft1 p_1  (cost=100.00..101.16 rows=5 width=10)
         Output: (p_1.a * 2), p_1.ctid
         Remote SQL: SELECT a, ctid FROM public.mytable_1 FOR UPDATE
   ->  Foreign Scan on public.ft2 p_2  (cost=100.00..101.16 rows=5 width=10)
         Output: (p_2.a * 2), p_2.ctid
         Remote SQL: SELECT a, ctid FROM public.mytable_2 FOR UPDATE
(14 rows)

Other changes:
* revised regression tests for contrib/file_fdw to refer to tableoid.
* revised docs a bit further.

Attached are updated patches.  Patch fdw-inh-5.patch has been created on top of patch fdw-chk-5.patch.  Patch fdw-chk-5.patch is basically the same as the previous one fdw-chk-4.patch, but I slightly modified that one.  The changes are the following.
* added to foreign_data.sql more tests for your comments.
* revised docs on ALTER FOREIGN TABLE a bit further.


Thanks,

Best regards,
Etsuro Fujita



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
Hi Ashutosh,

Thanks for the review!

(2014/12/10 14:47), Ashutosh Bapat wrote:
> We haven't heard anything from Horiguchi-san and Hanada-san for almost a
> week. So, I am fine marking it as "ready for committer". What do you say?

ISTM that both of them are not against us, so let's ask for the 
committer's review!

Thanks,

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Ashutosh Bapat
Date:
<div dir="ltr">I marked this as ready for committer.<br /></div><div class="gmail_extra"><br /><div
class="gmail_quote">OnThu, Dec 11, 2014 at 8:39 AM, Etsuro Fujita <span dir="ltr"><<a
href="mailto:fujita.etsuro@lab.ntt.co.jp"target="_blank">fujita.etsuro@lab.ntt.co.jp</a>></span> wrote:<br
/><blockquoteclass="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span
class="">HiAshutosh,<br /><br /> Thanks for the review!<br /><br /></span><span class=""> (2014/12/10 14:47), Ashutosh
Bapatwrote:<br /><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Wehaven't heard anything from Horiguchi-san and Hanada-san for almost a<br /> week. So, I am fine marking it as "ready
forcommitter". What do you say?<br /></blockquote><br /></span> ISTM that both of them are not against us, so let's ask
forthe committer's review!<div class="HOEnZb"><div class="h5"><br /><br /> Thanks,<br /><br /> Best regards,<br />
EtsuroFujita<br /></div></div></blockquote></div><br /><br clear="all" /><br />-- <br /><div
class="gmail_signature"><divdir="ltr">Best Wishes,<br />Ashutosh Bapat<br />EnterpriseDB Corporation<br />The Postgres
DatabaseCompany<br /></div></div></div> 

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/12/11 14:54), Ashutosh Bapat wrote:
> I marked this as ready for committer.

Many thanks!

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Michael Paquier
Date:
On Thu, Dec 11, 2014 at 2:54 PM, Ashutosh Bapat
<ashutosh.bapat@enterprisedb.com> wrote:
> On Thu, Dec 11, 2014 at 8:39 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>
> wrote:
>>
>> Hi Ashutosh,
>>
>> Thanks for the review!
>>
>> (2014/12/10 14:47), Ashutosh Bapat wrote:
>>>
>>> We haven't heard anything from Horiguchi-san and Hanada-san for almost a
>>> week. So, I am fine marking it as "ready for committer". What do you say?
Moving this patch to CF 2014-12 with the same status. Let's get a
committer having a look at it.
-- 
Michael



Re: inherit support for foreign tables

From
Tom Lane
Date:
Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
> Attached are updated patches.  Patch fdw-inh-5.patch has been created on 
> top of patch fdw-chk-5.patch.  Patch fdw-chk-5.patch is basically the 
> same as the previous one fdw-chk-4.patch, but I slightly modified that 
> one.  The changes are the following.
> * added to foreign_data.sql more tests for your comments.
> * revised docs on ALTER FOREIGN TABLE a bit further.

I've committed fdw-chk-5.patch with some minor further adjustments;
the most notable one was that I got rid of the error check prohibiting
NO INHERIT, which did not seem to me to have any value.  Attaching such
a clause won't have any effect, but so what?

Have not looked at the other patch yet.
        regards, tom lane



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
(2014/12/18 7:04), Tom Lane wrote:
> Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
>> Attached are updated patches.  Patch fdw-inh-5.patch has been created on
>> top of patch fdw-chk-5.patch.  Patch fdw-chk-5.patch is basically the
>> same as the previous one fdw-chk-4.patch, but I slightly modified that
>> one.  The changes are the following.
>> * added to foreign_data.sql more tests for your comments.
>> * revised docs on ALTER FOREIGN TABLE a bit further.
> 
> I've committed fdw-chk-5.patch with some minor further adjustments;
> the most notable one was that I got rid of the error check prohibiting
> NO INHERIT, which did not seem to me to have any value.  Attaching such
> a clause won't have any effect, but so what?
> 
> Have not looked at the other patch yet.

Thanks!

I added the error check because the other patch, fdw-inh-5.patch,
doesn't allow foreign tables to be inherited and so it seems more
consistent at least to me to do so.

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
On 2014/12/18 7:04, Tom Lane wrote:
> Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
>> Attached are updated patches.  Patch fdw-inh-5.patch has been created on
>> top of patch fdw-chk-5.patch.

> I've committed fdw-chk-5.patch with some minor further adjustments;

> Have not looked at the other patch yet.

I updated the remaining patch correspondingly to the fix [1].  Please
find attached a patch (the patch has been created on top of the patch in
[1]).

I haven't done anything about the issue that postgresGetForeignPlan() is
using get_parse_rowmark(), discussed in eg, [2].  Tom, will you work on
that?

Thanks,

[1] http://www.postgresql.org/message-id/5497BF4C.6080302@lab.ntt.co.jp
[2] http://www.postgresql.org/message-id/18256.1418401027@sss.pgh.pa.us

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Tom Lane
Date:
Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
> I haven't done anything about the issue that postgresGetForeignPlan() is
> using get_parse_rowmark(), discussed in eg, [2].  Tom, will you work on
> that?

Yeah, we need to do something about the PlanRowMark data structure.
Aside from the pre-existing issue in postgres_fdw, we need to fix it
to support inheritance trees in which more than one rowmark method
is being used.  That rte.hasForeignChildren thing is a crock, and
would still be a crock even if it were correctly documented as being
a planner temporary variable (rather than the current implication that
it's always valid).  RangeTblEntry is no place for planner temporaries.

The idea I'd had about that was to convert the markType field into a
bitmask, so that a parent node's markType could represent the logical
OR of the rowmark methods being used by all its children.  I've not
attempted to code this up though, and probably won't get to it until
after Christmas.  One thing that's not clear is what should happen
with ExecRowMark.
        regards, tom lane



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
On 2014/12/23 0:36, Tom Lane wrote:
> Yeah, we need to do something about the PlanRowMark data structure.
> Aside from the pre-existing issue in postgres_fdw, we need to fix it
> to support inheritance trees in which more than one rowmark method
> is being used.  That rte.hasForeignChildren thing is a crock, and
> would still be a crock even if it were correctly documented as being
> a planner temporary variable (rather than the current implication that
> it's always valid).  RangeTblEntry is no place for planner temporaries.

Agreed.

> The idea I'd had about that was to convert the markType field into a
> bitmask, so that a parent node's markType could represent the logical
> OR of the rowmark methods being used by all its children.  I've not
> attempted to code this up though, and probably won't get to it until
> after Christmas.  One thing that's not clear is what should happen
> with ExecRowMark.

That seems like a good idea, as parent PlanRowMarks are ignored at runtime.

Aside from the above, I noticed that the patch has a bug in handling
ExecRowMarks/ExecAuxRowMarks for foreign tables in inheritance trees
during the EPQ processing.:-(  Attached is an updated version of the
patch to fix that, which has been created on top of [1], as said before.

Thanks,

[1] http://www.postgresql.org/message-id/5497BF4C.6080302@lab.ntt.co.jp

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
On 2014/12/23 0:36, Tom Lane wrote:
> Yeah, we need to do something about the PlanRowMark data structure.
> Aside from the pre-existing issue in postgres_fdw, we need to fix it
> to support inheritance trees in which more than one rowmark method
> is being used.  That rte.hasForeignChildren thing is a crock,

> The idea I'd had about that was to convert the markType field into a
> bitmask, so that a parent node's markType could represent the logical
> OR of the rowmark methods being used by all its children.  I've not
> attempted to code this up though, and probably won't get to it until
> after Christmas.  One thing that's not clear is what should happen
> with ExecRowMark.

As I said before, that seems to me like a good idea.  So I'll update the
patch based on that if you're okey with it.  Or you've found any problem
concerning the above idea?

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Michael Paquier
Date:
<div dir="ltr"><br /><div class="gmail_extra"><br /><div class="gmail_quote">On Thu, Jan 15, 2015 at 4:35 PM, Etsuro
Fujita<span dir="ltr"><<a href="mailto:fujita.etsuro@lab.ntt.co.jp"
target="_blank">fujita.etsuro@lab.ntt.co.jp</a>></span>wrote:<br /><blockquote class="gmail_quote" style="margin:0 0
0.8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""></span>As I said before, that seems to me like a good
idea. So I'll update the<br /> patch based on that if you're okey with it.  Or you've found any problem<br />
concerningthe above idea?<br /></blockquote></div><br /></div><div class="gmail_extra">Patch moved to CF 2015-02 with
samestatus, "Ready for committer".<br />-- <br /><div class="gmail_signature">Michael<br /></div></div></div> 

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
On 2015/01/15 16:35, Etsuro Fujita wrote:
> On 2014/12/23 0:36, Tom Lane wrote:
>> Yeah, we need to do something about the PlanRowMark data structure.
>> Aside from the pre-existing issue in postgres_fdw, we need to fix it
>> to support inheritance trees in which more than one rowmark method
>> is being used.  That rte.hasForeignChildren thing is a crock,
>
>> The idea I'd had about that was to convert the markType field into a
>> bitmask, so that a parent node's markType could represent the logical
>> OR of the rowmark methods being used by all its children.

> As I said before, that seems to me like a good idea.  So I'll update the
> patch based on that if you're okey with it.

Done based on your ideas: (a) add a field to PlanRowMark to record the
original lock strength to fix the postgres_fdw issue and (b) convert its
markType field into a bitmask to support the inheritance trees.  I think
that both work well and that (a) is useful for the other places.

Patch attached, which has been created on top of [1].

Best regards,
Etsuro Fujita

[1] http://www.postgresql.org/message-id/54BCBBF8.3020103@lab.ntt.co.jp

Attachment

Re: inherit support for foreign tables

From
Tom Lane
Date:
Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
> I noticed that the latter disallows TRUNCATE on inheritance trees that 
> contain at least one child foreign table.  But I think it would be 
> better to allow it, with the semantics that we quietly ignore the child 
> foreign tables and apply the operation to the child plain tables, which 
> is the same semantics as ALTER COLUMN SET STORAGE on such inheritance 
> trees.  Comments welcome.

I've been working through the foreign table inheritance patch, and found
the code that makes the above happen.  I don't think this is a good idea
at all.  In the first place, successful TRUNCATE should leave the table
empty, not "well, we'll make it empty if we feel up to that".  In the
second place, someday we might want to make TRUNCATE actually work for
foreign tables (at least for FDWs that want to support it).  If we did,
we would have a backwards-compatibility hazard, because suddenly a
TRUNCATE on an inheritance tree that includes a foreign table would have
different non-error effects than before.

I think we should just throw error in this case.

BTW, the SET STORAGE comparison is bogus as well.  I see no reason that
we shouldn't just allow SET STORAGE on foreign tables.  It's probably
not going to have any effect, but so what?  And again, if we did ever
find a use for that, we'd have a compatibility problem if inherited SET
STORAGE has a pre-existing behavior that it skips foreign children.

In the same vein, I'm planning to take out the existing prohibition on
marking CHECK constraints on foreign tables NOT VALID.  That likewise
creates a corner case for inheritance trees for no obviously good reason.
It was reasonable to be conservative about whether to allow that so long
as there were no side-effects; but putting warts into the behavior of
inheritance trees to preserve the prohibition is not a good outcome.
        regards, tom lane



Re: inherit support for foreign tables

From
Tom Lane
Date:
Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
> [ fdw-inh-8.patch ]

I've committed this with some substantial rearrangements, notably:

* I thought that if we were doing this at all, we should go all the way
and allow foreign tables to be both inheritance parents and children.

* As I mentioned earlier, I got rid of a few unnecessary restrictions on
foreign tables so as to avoid introducing warts into inheritance behavior.
In particular, we now allow NOT VALID CHECK constraints (and hence ALTER
... VALIDATE CONSTRAINT), ALTER SET STORAGE, and ALTER SET WITH/WITHOUT
OIDS.  These are probably no-ops anyway for foreign tables, though
conceivably an FDW might choose to implement some behavior for STORAGE
or OIDs.

* I did not like the EXPLAIN changes at all; in the first place they
resulted in invalid JSON output (there could be multiple fields of the
Update plan object with identical labels), and in the second place it
seemed like a bad idea to rely on FDWs to change the behavior of their
ExplainModifyTarget functions.  I've refactored that so that explain.c
remains responsible for getting the grouping right.  Also, as I said
earlier, it seemed like a good idea to produce subgroups identifying
all the target tables not only the foreign ones.

* I fooled around with the PlanRowMark changes some more, mainly with
the idea that we might soon allow FDWs to use rowmark methods other than
ROW_MARK_COPY.  The planner now has just one place where a rel's rowmark
method is chosen, so as to centralize anything we need to do there.
        regards, tom lane



Re: inherit support for foreign tables

From
Robert Haas
Date:
On Sun, Mar 22, 2015 at 1:57 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
>> [ fdw-inh-8.patch ]
>
> I've committed this with some substantial rearrangements, notably:

I'm really glad this is going in!  Thanks to to Shigeru Hanada and
Etsuro Fujita for working on this, to you (Tom) for putting in the
time to get it committed, and of course to the reviewers Ashutosh
Bapat and Kyotaro Horiguchi for their time and effort.

In a way, I believe we can think of this as the beginnings of a
sharding story for PostgreSQL.  A lot more work is needed, of course
-- join and aggregate pushdown are high on my personal list -- but
it's a start.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: inherit support for foreign tables

From
Ashutosh Bapat
Date:


On Mon, Mar 23, 2015 at 12:09 AM, Robert Haas <robertmhaas@gmail.com> wrote:
On Sun, Mar 22, 2015 at 1:57 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
>> [ fdw-inh-8.patch ]
>
> I've committed this with some substantial rearrangements, notably:

I'm really glad this is going in!  Thanks to to Shigeru Hanada and
Etsuro Fujita for working on this, to you (Tom) for putting in the
time to get it committed, and of course to the reviewers Ashutosh
Bapat and Kyotaro Horiguchi for their time and effort.

In a way, I believe we can think of this as the beginnings of a
sharding story for PostgreSQL.  A lot more work is needed, of course
-- join and aggregate pushdown are high on my personal list -- but
it's a start.


+1.
 
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
On 2015/03/23 2:57, Tom Lane wrote:
> I've committed this with some substantial rearrangements, notably:

Thanks for taking the time to committing the patch!

Thanks for the work, Hanada-san!  And thank you everyone for the reviews
and comments, especially Ashutosh, Horiguchi-san and Noah!

> * I fooled around with the PlanRowMark changes some more, mainly with
> the idea that we might soon allow FDWs to use rowmark methods other than
> ROW_MARK_COPY.  The planner now has just one place where a rel's rowmark
> method is chosen, so as to centralize anything we need to do there.

Will work on this issue.

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
On 2015/03/23 2:57, Tom Lane wrote:
> Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
>> [ fdw-inh-8.patch ]
>
> I've committed this with some substantial rearrangements, notably:
>
> * I thought that if we were doing this at all, we should go all the way
> and allow foreign tables to be both inheritance parents and children.

I found that when setting a foreign table to be the parent of an
inheritance set that only contains foreign tables, SELECT FOR UPDATE on
the inheritance parent fails with a can't-happen error condition.  Here
is an example:

$ createdb mydb
$ psql mydb
psql (9.5devel)
Type "help" for help.

mydb=# create table t1 (c1 int);
CREATE TABLE
mydb=# create table t2 (c1 int);
CREATE TABLE

$ psql postgres
psql (9.5devel)
Type "help" for help.

postgres=# create extension postgres_fdw;
CREATE EXTENSION
postgres=# create server myserver foreign data wrapper postgres_fdw
options (dbname 'mydb');
CREATE SERVER
postgres=# create user mapping for current_user server myserver;
CREATE USER MAPPING
postgres=# create foreign table ft1 (c1 int) server myserver options
(table_name 't1');
CREATE FOREIGN TABLE
postgres=# create foreign table ft2 (c1 int) server myserver options
(table_name 't2');
CREATE FOREIGN TABLE
postgres=# alter foreign table ft2 inherit ft1;
ALTER FOREIGN TABLE
postgres=# select * from ft1 for update;
ERROR:  could not find junk tableoid1 column

I think this is a bug.  Attached is a patch fixing this issue.

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Jim Nasby
Date:
On 4/14/15 5:49 AM, Etsuro Fujita wrote:
> postgres=# create foreign table ft1 (c1 int) server myserver options
> (table_name 't1');
> CREATE FOREIGN TABLE
> postgres=# create foreign table ft2 (c1 int) server myserver options
> (table_name 't2');
> CREATE FOREIGN TABLE
> postgres=# alter foreign table ft2 inherit ft1;
> ALTER FOREIGN TABLE
> postgres=# select * from ft1 for update;
> ERROR:  could not find junk tableoid1 column
> 
> I think this is a bug.  Attached is a patch fixing this issue.

What happens when the foreign side breaks the inheritance? Does the FDW
somehow know to check that fact for each query?

What do you gain from having the local table have inheritance?

Basically, I think we have to be very careful about implementing
features that imply the local side knows something about the persisted
state of the remote side (in this case, whether there's inheritance).
Anything like that sets us up for synchronization problems.
-- 
Jim Nasby, Data Architect, Blue Treble Consulting
Data in Trouble? Get it in Treble! http://BlueTreble.com



Re: inherit support for foreign tables

From
Alvaro Herrera
Date:
Jim Nasby wrote:
> On 4/14/15 5:49 AM, Etsuro Fujita wrote:
> > postgres=# create foreign table ft1 (c1 int) server myserver options
> > (table_name 't1');
> > CREATE FOREIGN TABLE
> > postgres=# create foreign table ft2 (c1 int) server myserver options
> > (table_name 't2');
> > CREATE FOREIGN TABLE
> > postgres=# alter foreign table ft2 inherit ft1;
> > ALTER FOREIGN TABLE
> > postgres=# select * from ft1 for update;
> > ERROR:  could not find junk tableoid1 column
> > 
> > I think this is a bug.  Attached is a patch fixing this issue.
> 
> What happens when the foreign side breaks the inheritance? Does the FDW
> somehow know to check that fact for each query?

This is a meaningless question.  The remote tables don't have to have an
inheritance relationship already; only the local side sees them as
connected.

I think the real question is whether we're now (I mean after this patch)
emitting useless tableoid columns that we didn't previously have.  I
think the answer is yes, and if so I think we need a smaller comb to fix
the problem.  This one seems rather large.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hi,

Before suppressing the symptom, I doubt the necessity and/or
validity of giving foreign tables an ability to be a parent. Is
there any reasonable usage for the ability?

I think we should choose to inhibit foreign tables from becoming
a parent rather than leaving it allowed then taking measures for
the consequent symptom.


regards,

At Tue, 14 Apr 2015 15:52:18 -0300, Alvaro Herrera <alvherre@2ndquadrant.com> wrote in
<20150414185218.GX4369@alvh.no-ip.org>
> Jim Nasby wrote:
> > On 4/14/15 5:49 AM, Etsuro Fujita wrote:
> > > postgres=# create foreign table ft1 (c1 int) server myserver options
> > > (table_name 't1');
> > > CREATE FOREIGN TABLE
> > > postgres=# create foreign table ft2 (c1 int) server myserver options
> > > (table_name 't2');
> > > CREATE FOREIGN TABLE
> > > postgres=# alter foreign table ft2 inherit ft1;
> > > ALTER FOREIGN TABLE
> > > postgres=# select * from ft1 for update;
> > > ERROR:  could not find junk tableoid1 column
> > > 
> > > I think this is a bug.  Attached is a patch fixing this issue.
> > 
> > What happens when the foreign side breaks the inheritance? Does the FDW
> > somehow know to check that fact for each query?
> 
> This is a meaningless question.  The remote tables don't have to have an
> inheritance relationship already; only the local side sees them as
> connected.
> 
> I think the real question is whether we're now (I mean after this patch)
> emitting useless tableoid columns that we didn't previously have.  I
> think the answer is yes, and if so I think we need a smaller comb to fix
> the problem.  This one seems rather large.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center




Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
On 2015/04/15 3:52, Alvaro Herrera wrote:
>> On 4/14/15 5:49 AM, Etsuro Fujita wrote:
>>> postgres=# create foreign table ft1 (c1 int) server myserver options
>>> (table_name 't1');
>>> CREATE FOREIGN TABLE
>>> postgres=# create foreign table ft2 (c1 int) server myserver options
>>> (table_name 't2');
>>> CREATE FOREIGN TABLE
>>> postgres=# alter foreign table ft2 inherit ft1;
>>> ALTER FOREIGN TABLE
>>> postgres=# select * from ft1 for update;
>>> ERROR:  could not find junk tableoid1 column
>>>
>>> I think this is a bug.  Attached is a patch fixing this issue.

> I think the real question is whether we're now (I mean after this patch)
> emitting useless tableoid columns that we didn't previously have.  I
> think the answer is yes, and if so I think we need a smaller comb to fix
> the problem.  This one seems rather large.

My answer for that would be *no* because I think tableoid would be 
needed for EvalPlanQual checking in more complex SELECT FOR UPDATE on 
the inheritance or UPDATE/DELETE involving the inheritance as a source 
table.  Also, if we allow the FDW to change the behavior of SELECT FOR 
UPDATE so as to match the local semantics exactly, which I'm working on 
in another thread, I think tableoid would also be needed for the actual 
row locking.

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hello,

At Thu, 16 Apr 2015 12:20:47 +0900, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote in
<552F2A8F.2090406@lab.ntt.co.jp>
> On 2015/04/15 3:52, Alvaro Herrera wrote:
> >> On 4/14/15 5:49 AM, Etsuro Fujita wrote:
> >>> postgres=# create foreign table ft1 (c1 int) server myserver options
> >>> (table_name 't1');
> >>> CREATE FOREIGN TABLE
> >>> postgres=# create foreign table ft2 (c1 int) server myserver options
> >>> (table_name 't2');
> >>> CREATE FOREIGN TABLE
> >>> postgres=# alter foreign table ft2 inherit ft1;
> >>> ALTER FOREIGN TABLE
> >>> postgres=# select * from ft1 for update;
> >>> ERROR:  could not find junk tableoid1 column
> >>>
> >>> I think this is a bug.  Attached is a patch fixing this issue.
> 
> > I think the real question is whether we're now (I mean after this
> > patch)
> > emitting useless tableoid columns that we didn't previously have.  I
> > think the answer is yes, and if so I think we need a smaller comb to
> > fix
> > the problem.  This one seems rather large.
> 
> My answer for that would be *no* because I think tableoid would be
> needed for EvalPlanQual checking in more complex SELECT FOR UPDATE on
> the inheritance or UPDATE/DELETE involving the inheritance as a source
> table.  Also, if we allow the FDW to change the behavior of SELECT FOR
> UPDATE so as to match the local semantics exactly, which I'm working
> on in another thread, I think tableoid would also be needed for the
> actual row locking.

Given the parent foreign talbes, surely they need tableoids for
such usage. The patch preserves the condition rc->isParent so it
newly affects exactly only parent foreign tables for now.

Before the parent foreign tables introduced, ROW_MARK_COPY and
RTE_RELATION are mutually exclusive so didn't need, or cannot
have tableoid.  But now it intorduces an rte with ROW_MARK_COPY &
RTE_RELATION and there seems no reason for parent tables in any
kind not to have tableoid. After such consideration, I came to
think that the patch is a reasonable fix, not mere a workaround.

Thoughts?

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center




Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
On 2015/03/23 2:57, Tom Lane wrote:
> Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
>> [ fdw-inh-8.patch ]
>
> I've committed this with some substantial rearrangements, notably:

> * As I mentioned earlier, I got rid of a few unnecessary restrictions on
> foreign tables so as to avoid introducing warts into inheritance behavior.
> In particular, we now allow NOT VALID CHECK constraints (and hence ALTER
> ... VALIDATE CONSTRAINT), ALTER SET STORAGE, and ALTER SET WITH/WITHOUT
> OIDS.  These are probably no-ops anyway for foreign tables, though
> conceivably an FDW might choose to implement some behavior for STORAGE
> or OIDs.

I agree with you on this point.  However, ISTM there is a bug in
handling OIDs on foreign tables; while we now allow for ALTER SET
WITH/WITHOUT OIDS, we still don't allow the default_with_oids parameter
for foreign tables.  I think that since CREATE FOREIGN TABLE should be
consistent with ALTER FOREIGN TABLE, we should also allow the parameter
for foreign tables.  Attached is a patch for that.

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
David Fetter
Date:
On Wed, Apr 15, 2015 at 09:35:05AM +0900, Kyotaro HORIGUCHI wrote:
> Hi,
> 
> Before suppressing the symptom, I doubt the necessity and/or
> validity of giving foreign tables an ability to be a parent. Is
> there any reasonable usage for the ability?
> 
> I think we should choose to inhibit foreign tables from becoming
> a parent rather than leaving it allowed then taking measures for
> the consequent symptom.

I have a use case for having foreign tables as non-leaf nodes in a
partitioning hierarchy, namely geographic.  One might have a table at
HQ called foo_world, then partitions under it called foo_jp, foo_us,
etc., in one level, foo_us_ca, foo_us_pa, etc. in the next level, and
on down, each in general in a separate data center.

Is there something essential about having non-leaf nodes as foreign
tables that's a problem here?

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate



Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
On 2015/04/16 16:05, Etsuro Fujita wrote:
> On 2015/03/23 2:57, Tom Lane wrote:
>> Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
>>> [ fdw-inh-8.patch ]
>>
>> I've committed this with some substantial rearrangements, notably:
>
>> * As I mentioned earlier, I got rid of a few unnecessary restrictions on
>> foreign tables so as to avoid introducing warts into inheritance behavior.
>> In particular, we now allow NOT VALID CHECK constraints (and hence ALTER
>> ... VALIDATE CONSTRAINT), ALTER SET STORAGE, and ALTER SET WITH/WITHOUT
>> OIDS.  These are probably no-ops anyway for foreign tables, though
>> conceivably an FDW might choose to implement some behavior for STORAGE
>> or OIDs.
>
> I agree with you on this point.  However, ISTM there is a bug in
> handling OIDs on foreign tables; while we now allow for ALTER SET
> WITH/WITHOUT OIDS, we still don't allow the default_with_oids parameter
> for foreign tables.  I think that since CREATE FOREIGN TABLE should be
> consistent with ALTER FOREIGN TABLE, we should also allow the parameter
> for foreign tables.  Attached is a patch for that.

I also updated docs.  Attached is an updated version of the patch.

Best regards,
Etsuro Fujita

Attachment

Re: inherit support for foreign tables

From
Stephen Frost
Date:
Etsuro,

* Etsuro Fujita (fujita.etsuro@lab.ntt.co.jp) wrote:
> postgres=# select * from ft1 for update;
> ERROR:  could not find junk tableoid1 column
>
> I think this is a bug.  Attached is a patch fixing this issue.

Pushed, thanks!
Stephen

Re: inherit support for foreign tables

From
Robert Haas
Date:
On Tue, Apr 14, 2015 at 8:35 PM, Kyotaro HORIGUCHI
<horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> Before suppressing the symptom, I doubt the necessity and/or
> validity of giving foreign tables an ability to be a parent. Is
> there any reasonable usage for the ability?

Gee, I don't see why that would be unreasonable or invalid

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company



Re: inherit support for foreign tables

From
Kyotaro HORIGUCHI
Date:
Hello,

At Thu, 16 Apr 2015 14:43:33 -0700, David Fetter <david@fetter.org> wrote in <20150416214333.GA797@fetter.org>
> On Wed, Apr 15, 2015 at 09:35:05AM +0900, Kyotaro HORIGUCHI wrote:
> > Hi,
> > 
> > Before suppressing the symptom, I doubt the necessity and/or
> > validity of giving foreign tables an ability to be a parent. Is
> > there any reasonable usage for the ability?
...
> I have a use case for having foreign tables as non-leaf nodes in a
> partitioning hierarchy, namely geographic.

Ah, I see. I understood the case of intermediate nodes. I agree
that it is quite natural.

>  One might have a table at
> HQ called foo_world, then partitions under it called foo_jp, foo_us,
> etc., in one level, foo_us_ca, foo_us_pa, etc. in the next level, and
> on down, each in general in a separate data center.
> 
> Is there something essential about having non-leaf nodes as foreign
> tables that's a problem here?

No. I'm convinced of the necessity. Sorry for the noise.


At Wed, 22 Apr 2015 17:00:10 -0400, Robert Haas <robertmhaas@gmail.com> wrote in
<CA+TgmobZVHp3D9wWCV8QJc+qGDu7=tEKNCbXOwijZKhjuCmRWg@mail.gmail.com>
> Gee, I don't see why that would be unreasonable or invalid

Hmm. Yes, as mentioned above, there's no reason to refuse
non-leaf foregin tables. I didn't understood the real cause of
the problem and thought that not allowing foreign *root* tables
seem better than tweaking elsewhere. But that thought found to be
totally a garbage :(

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center




Re: inherit support for foreign tables

From
Etsuro Fujita
Date:
On 2015/04/23 0:34, Stephen Frost wrote:
> * Etsuro Fujita (fujita.etsuro@lab.ntt.co.jp) wrote:
>> postgres=# select * from ft1 for update;
>> ERROR:  could not find junk tableoid1 column
>>
>> I think this is a bug.  Attached is a patch fixing this issue.
>
> Pushed, thanks!

Thank you.

Best regards,
Etsuro Fujita



Re: inherit support for foreign tables

From
Tom Lane
Date:
Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
> On 2015/04/16 16:05, Etsuro Fujita wrote:
>> I agree with you on this point.  However, ISTM there is a bug in
>> handling OIDs on foreign tables; while we now allow for ALTER SET
>> WITH/WITHOUT OIDS, we still don't allow the default_with_oids parameter
>> for foreign tables.  I think that since CREATE FOREIGN TABLE should be
>> consistent with ALTER FOREIGN TABLE, we should also allow the parameter
>> for foreign tables.  Attached is a patch for that.

> I also updated docs.  Attached is an updated version of the patch.

I believe that we intentionally did not do this, and here is why not:
existing pg_dump files assume that default_with_oids doesn't affect any
relation type except plain tables.  pg_backup_archiver.c only bothers
to change the GUC when about to dump a plain table, and otherwise leaves
it at its previous value.  That means if we apply a patch like this, it's
entirely possible that pg_dump/pg_restore will result in foreign tables
accidentally acquiring OID columns.

Since default_with_oids is really only meant as a backwards-compatibility
hack, I don't personally have a problem with restricting it to act only on
plain tables.  However, it might be a good idea to explicitly document
this interaction in a code comment to prevent anyone from re-inventing
this idea...  I'll go do that.
        regards, tom lane



Re: inherit support for foreign tables

From
Andres Freund
Date:
On 2015-04-25 20:47:04 -0400, Tom Lane wrote:
> Since default_with_oids is really only meant as a backwards-compatibility
> hack, I don't personally have a problem with restricting it to act only on
> plain tables.

FWIW, I think we're getting pretty close to the point, or are there
even, where we can remove default_with_oids. So not adding complications
because of it sounds good to me.

Greetings,

Andres Freund



Re: inherit support for foreign tables

From
Tom Lane
Date:
Andres Freund <andres@anarazel.de> writes:
> FWIW, I think we're getting pretty close to the point, or are there
> even, where we can remove default_with_oids. So not adding complications
> because of it sounds good to me.

Well, pg_dump uses it --- so the argument about not breaking old dump
scripts would apply to any attempt to remove it entirely.  But I don't
have a problem with saying that its semantics are frozen.
        regards, tom lane