Thread: inherit support for foreign tables
Hi hackers,
I'd like to propose adding inheritance support for foriegn tables.
David Fetter mentioned this feature last July, but it seems stalled.
http://www.postgresql.org/message-id/20130719005601.GA5760@fetter.org
Supporting inheritance by foreign tables allows us to distribute query
to remote servers by using foreign tables as partition table of a
(perhaps ordinary) table. For this purpose, I think that constraint
exclusion is necessary.
As result of extending Devid's patch for PoC, and AFAIS we need these changes:
1) Add INHERITS(rel, ...) clause to CREATE/ALTER FOREIGN TABLE
Apperantly we need to add new syntax to define parent table(s) of a
foreign table. We have options about the position of INHERIT clause,
but I'd prefer before SERVER clause because having options specific to
foreign tables at the tail would be most extensible.
a) CREATE FOREIGN TABLE child (...) INHERITS(p1, p2) SERVER server;
b) CREATE FOREIGN TABLE child (...) SERVER server INHERITS(p1, p2);
2) Allow foreign tables to have CHECK constraints
Like NOT NULL, I think we don't need to enforce the check duroing
INSERT/UPDATE against foreign table.
3) Allow foreign table as a child node of Append
Currently prepunion.c assumes that children of Append have
RELKIND_RELATION as relkind always, so we need to set relkind of child
RTE explicitly.
Please see attached PoC patch. I'll enhance implementation, tests and
document and submit the patch for the next CF.
Regards,
--
Shigeru HANADA
Attachment
Shigeru Hanada <shigeru.hanada@gmail.com> writes: > I'd like to propose adding inheritance support for foriegn tables. > David Fetter mentioned this feature last July, but it seems stalled. > http://www.postgresql.org/message-id/20130719005601.GA5760@fetter.org The discussion there pointed out that not enough consideration had been given to interactions with other commands. I'm not really satisfied with your analysis here. In particular: > 2) Allow foreign tables to have CHECK constraints > Like NOT NULL, I think we don't need to enforce the check duroing > INSERT/UPDATE against foreign table. Really? It's one thing to say that somebody who adds a CHECK constraint to a foreign table is responsible to make sure that the foreign data will satisfy the constraint. It feels like a different thing to say that ALTER TABLE ADD CONSTRAINT applied to a parent table will silently assume that some child table that happens to be foreign doesn't need any enforcement. Perhaps more to the point, inheritance trees are the main place where the planner depends on the assumption that CHECK constraints represent reality. Are we really prepared to say that it's the user's fault if the planner generates an incorrect plan on the strength of a CHECK constraint that's not actually satisfied by the foreign data? If so, that had better be documented by this patch. But for a project that refuses to let people create a local CHECK or FOREIGN KEY constraint without mechanically checking it, it seems pretty darn weird to be so laissez-faire about constraints on foreign data. regards, tom lane
On Thu, Nov 14, 2013 at 12:28 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> 2) Allow foreign tables to have CHECK constraints >> Like NOT NULL, I think we don't need to enforce the check duroing >> INSERT/UPDATE against foreign table. > > Really? It's one thing to say that somebody who adds a CHECK constraint > to a foreign table is responsible to make sure that the foreign data will > satisfy the constraint. It feels like a different thing to say that ALTER > TABLE ADD CONSTRAINT applied to a parent table will silently assume that > some child table that happens to be foreign doesn't need any enforcement. > > Perhaps more to the point, inheritance trees are the main place where the > planner depends on the assumption that CHECK constraints represent > reality. Are we really prepared to say that it's the user's fault if the > planner generates an incorrect plan on the strength of a CHECK constraint > that's not actually satisfied by the foreign data? If so, that had better > be documented by this patch. But for a project that refuses to let people > create a local CHECK or FOREIGN KEY constraint without mechanically > checking it, it seems pretty darn weird to be so laissez-faire about > constraints on foreign data. I can see both sides of this issue. We certainly have no way to force the remote side to enforce CHECK constraints defined on the local side, because the remote side could also be accepting writes from other sources that don't have any matching constraint. But having said that, I can't see any particularly principled reason why we shouldn't at least check the new rows we insert ourselves. After all, we could be in the situation proposed by KaiGai Kohei, where the foreign data wrapper API is being used as a surrogate storage engine API - i.e. there are no writers to the foreign side except ourselves. In that situation, it would seem odd to randomly fail to enforce the constraints. On the other hand, the performance costs of checking every row bound for the remote table could be quite steep. Consider an update on an inheritance hierarchy that sets a = a + 1 for every row. If we don't worry about verifying that the resulting rows satisfy all local-side constraints, we can potentially ship a single update statement to the remote server and let it do all the work there. But if we DO have to worry about that, then we're going to have to ship every updated row over the wire in at least one direction, if not both. If the purpose of adding CHECK constraints was to enable constraint exclusion, that's a mighty steep price to pay for it. I think it's been previously proposed that we have some version of a CHECK constraint that effectively acts as an assertion for query optimization purposes, but isn't actually enforced by the system. I can see that being useful in a variety of situations, including this one. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, Nov 14, 2013 at 12:28 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> 2) Allow foreign tables to have CHECK constraints
>>> Like NOT NULL, I think we don't need to enforce the check duroing
>>> INSERT/UPDATE against foreign table.
>> Really?
> I think it's been previously proposed that we have some version of a
> CHECK constraint that effectively acts as an assertion for query
> optimization purposes, but isn't actually enforced by the system. I
> can see that being useful in a variety of situations, including this
> one.
Yeah, I think it would be much smarter to provide a different syntax
to explicitly represent the notion that we're only assuming the condition
is true, and not trying to enforce it.
regards, tom lane
2013/11/18 Tom Lane <tgl@sss.pgh.pa.us>: > Robert Haas <robertmhaas@gmail.com> writes: >> I think it's been previously proposed that we have some version of a >> CHECK constraint that effectively acts as an assertion for query >> optimization purposes, but isn't actually enforced by the system. I >> can see that being useful in a variety of situations, including this >> one. > > Yeah, I think it would be much smarter to provide a different syntax > to explicitly represent the notion that we're only assuming the condition > is true, and not trying to enforce it. I'd like to revisit this feature. Explicit notation to represent not-enforcing (assertive?) is an interesting idea. I'm not sure which word is appropriate, but for example, let's use the word ASSERTIVE as a new constraint attribute. CREATE TABLE parent ( id int NOT NULL, group int NOT NULL, name text ); CREATE FOREIGN TABLE child_grp1 ( /* no additional column */ ) INHERITS (parent) SERVER server1; ALTER TABLE child_grp1 ADD CONSTRAINT chk_group1 CHECK (group = 1) ASSERTIVE; If ASSERTIVE is specified, it's not guaranteed that the constraint is enforced completely, so it should be treated as a hint for planner. As Robert mentioned, enforcing as much as we can during INSERT/UPDATE is one option about this issue. In addition, an idea which I can't throw away is to assume that all constraints defined on foreign tables as ASSERTIVE. Foreign tables potentially have dangers to have "wrong" data by updating source data not through foreign tables. This is not specific to an FDW, so IMO constraints defined on foreign tables are basically ASSERTIVE. Of course PG can try to maintain data correct, but always somebody might break it. Besides CHECK constraints, currently NOT NULL constraints are virtually ASSERTIVE (not enforcing). Should it also be noted explicitly? Thoughts? -- Shigeru HANADA
From PostgreSQL manual: "A serious limitation of the inheritance feature is that indexes (including unique constraints) and foreign key constraints only apply to single tables, not to their inheritance children." But is it possible to use index for derived table at all? Why sequential search is used for derived table in the example below: create table base_table (x integer primary key); create table derived_table (y integer) inherits (base_table); insert into base_table values (1); insert into derived_table values (2,2); create index derived_index on derived_table(x); explain select * from base_table where x>=0; QUERY PLAN ---------------------------------------------------------------------------------------------- Append (cost=0.14..4.56 rows=81width=4) -> Index Only Scan using base_table_pkey on base_table (cost=0.14..3.55 rows=80 width=4) Index Cond: (x >= 0) -> Seq Scan on derived_table (cost=0.00..1.01 rows=1 width=4) Filter: (x >= 0) (5 rows)
On Tue, Jan 14, 2014 at 12:07 PM, knizhnik <knizhnik@garret.ru> wrote: > But is it possible to use index for derived table at all? Yes, the planner will do an index scan when it makes sense. > Why sequential search is used for derived table in the example below: > insert into derived_table values (2,2); > create index derived_index on derived_table(x); > explain select * from base_table where x>=0; With only 1 row in the table, the planner decides there's no point in scanning the index. Try with more realistic data. Regards, Marti
Hi all, 2014/1/14 Shigeru Hanada <shigeru.hanada@gmail.com>: > I'd like to revisit this feature. Attached patch allows a foreign table to be a child of a table. It also allows foreign tables to have CHECK constraints. These changes provide us a chance to propagate query load to multiple servers via constraint exclusion. If FDW supports async operation against remote server, parallel processing (not stable but read-only case would be find) can be achieved, though overhead of FDW mechanism is still there. Though this would be debatable, in current implementation, constraints defined on a foreign table (now only NOT NULL and CHECK are supported) are not enforced during INSERT or UPDATE executed against foreign tables. This means that retrieved data might violates the constraints defined on local side. This is debatable issue because integrity of data is important for DBMS, but in the first cut, it is just documented as a note. Because I don't see practical case to have a foreign table as a parent, and it avoid a problem about recursive ALTER TABLE operation, foreign tables can't be a parent. An example of such problem is adding constraint which is not unsupported for foreign tables to the parent of foreign table. Propagated operation can be applied to ordinary tables in the inheritance tree, but can't be to foreign tables. If we allow foreign tables to be parent, it's difficult to process ordinary tables below foreign tables in current traffic cop mechanism. For other commands recursively processed such as ANALYZE, foreign tables in the leaf of inheritance tree are ignored. Any comments or questions are welcome. -- Shigeru HANADA
Attachment
On 11/18/13, 8:36 AM, Robert Haas wrote: > On the other hand, the performance costs of checking every row bound > for the remote table could be quite steep. Consider an update on an > inheritance hierarchy that sets a = a + 1 for every row. If we don't > worry about verifying that the resulting rows satisfy all local-side > constraints, we can potentially ship a single update statement to the > remote server and let it do all the work there. But if we DO have to > worry about that, then we're going to have to ship every updated row > over the wire in at least one direction, if not both. If the purpose > of adding CHECK constraints was to enable constraint exclusion, that's > a mighty steep price to pay for it. A sophisticated enough FDW could verify that the appropriate check already existed in tho foreign side, or it could do somethinglike: BEGIN; UPDATE SET ... WHERE <where> SELECT EXISTS( SELECT 1 WHERE <where> AND NOT (<check condition>) ); And then rollback if the SELECT returns true. But obviously you can't always do that, so I think there's a place for both true constraints and "suggested constraints". -- Jim C. Nasby, Data Architect jim@nasby.net 512.569.9461 (cell) http://jim.nasby.net
(2014/01/14 18:24), Shigeru Hanada wrote: > 2013/11/18 Tom Lane <tgl@sss.pgh.pa.us>: >> Robert Haas <robertmhaas@gmail.com> writes: >>> I think it's been previously proposed that we have some version of a >>> CHECK constraint that effectively acts as an assertion for query >>> optimization purposes, but isn't actually enforced by the system. I >>> can see that being useful in a variety of situations, including this >>> one. >> >> Yeah, I think it would be much smarter to provide a different syntax >> to explicitly represent the notion that we're only assuming the condition >> is true, and not trying to enforce it. > > I'd like to revisit this feature. > > Explicit notation to represent not-enforcing (assertive?) is an > interesting idea. I'm not sure which word is appropriate, but for > example, let's use the word ASSERTIVE as a new constraint attribute. > > CREATE TABLE parent ( > id int NOT NULL, > group int NOT NULL, > name text > ); > > CREATE FOREIGN TABLE child_grp1 ( > /* no additional column */ > ) INHERITS (parent) SERVER server1; > ALTER TABLE child_grp1 ADD CONSTRAINT chk_group1 CHECK (group = 1) ASSERTIVE; > > If ASSERTIVE is specified, it's not guaranteed that the constraint is > enforced completely, so it should be treated as a hint for planner. > As Robert mentioned, enforcing as much as we can during INSERT/UPDATE > is one option about this issue. > > In addition, an idea which I can't throw away is to assume that all > constraints defined on foreign tables as ASSERTIVE. Foreign tables > potentially have dangers to have "wrong" data by updating source data > not through foreign tables. This is not specific to an FDW, so IMO > constraints defined on foreign tables are basically ASSERTIVE. Of > course PG can try to maintain data correct, but always somebody might > break it. > qu > Does it make sense to apply "assertive" CHECK constraint on the qual of ForeignScan to filter out tuples with violated values at the local side, as if row-level security feature doing. It enables to handle a situation that planner expects only "clean" tuples are returned but FDW driver is unavailable to anomalies. Probably, this additional check can be turned on/off on the fly, if FDW driver has a way to inform the core system its capability, like FDW_CAN_ENFORCE_CHECK_CONSTRAINT that informs planner to skip local checks. > Besides CHECK constraints, currently NOT NULL constraints are > virtually ASSERTIVE (not enforcing). Should it also be noted > explicitly? > Backward compatibility.... NOT NULL [ASSERTIVE] might be an option. Thanks, -- OSS Promotion Center / The PG-Strom Project KaiGai Kohei <kaigai@ak.jp.nec.com>
Thanks for the comments. 2014/1/21 KaiGai Kohei <kaigai@ak.jp.nec.com>: >> In addition, an idea which I can't throw away is to assume that all >> constraints defined on foreign tables as ASSERTIVE. Foreign tables >> potentially have dangers to have "wrong" data by updating source data >> not through foreign tables. This is not specific to an FDW, so IMO >> constraints defined on foreign tables are basically ASSERTIVE. Of >> course PG can try to maintain data correct, but always somebody might >> break it. >> qu >> > Does it make sense to apply "assertive" CHECK constraint on the qual > of ForeignScan to filter out tuples with violated values at the local > side, as if row-level security feature doing. > It enables to handle a situation that planner expects only "clean" > tuples are returned but FDW driver is unavailable to anomalies. > > Probably, this additional check can be turned on/off on the fly, > if FDW driver has a way to inform the core system its capability, > like FDW_CAN_ENFORCE_CHECK_CONSTRAINT that informs planner to skip > local checks. Hmm, IIUC you mean that local users can't (or don't need to) know that data which violates the local constraints exist on remote side. Applying constraints to the data which is modified through FDW would be necessary as well. In that design, FDW is a bidirectional filter which provides these features: 1) Don't push wrong data into remote data source, by applying local constraints to the result of the modifying query executed on local PG.This is not perfect filter, because remote constraintsdon't mapped automatically or perfectly (imagine constraints which is available on remote but is not supported in PG). 2) Don't retrieve wrong data from remote to local PG, by applying local constraints I have a concern about consistency. It has not been supported, but let's think of Aggregate push-down invoked by a query below. SELECT count(*) FROM remote_table; If this query was fully pushed down, the result is the # of records exist on remote side, but the result would be # of valid records when we don't push down the aggregate. This would confuse users. >> Besides CHECK constraints, currently NOT NULL constraints are >> virtually ASSERTIVE (not enforcing). Should it also be noted >> explicitly? >> > Backward compatibility…. Yep, backward compatibility (especially visible ones to users) should be minimal, ideally zero. > NOT NULL [ASSERTIVE] might be an option. Treating [ASSERTIVE | NOT ASSERTIVE] like DEFERRABLE, and allow ingASSERTIVE for only foreign tables? It makes sense, though we need consider exclusiveness . But It needs to default to ASSERTIVE on foreign tables, and NOT ASSERTIVE (means "forced") on others. Isn't is too complicated? CREATE FOREIGN TABLE foo ( id int NOT NULL ASSERTIVE CHECK (id > 1) ASSERTIVE, … CONSTRAINT chk_foo_name_upper CHECK(upper(name) = name) ASSERTIVE ) SERVER server; BTW, I noticed that this is like push-down-able expressions in JOIN/WHERE. We need to check a CHECK constraint defined on a foreign tables contains only expressions which have same semantics as remote side (in practice, built-in and immutable)? -- Shigeru HANADA
(2014/01/21 11:44), Shigeru Hanada wrote: > Thanks for the comments. > > 2014/1/21 KaiGai Kohei <kaigai@ak.jp.nec.com>: >>> In addition, an idea which I can't throw away is to assume that all >>> constraints defined on foreign tables as ASSERTIVE. Foreign tables >>> potentially have dangers to have "wrong" data by updating source data >>> not through foreign tables. This is not specific to an FDW, so IMO >>> constraints defined on foreign tables are basically ASSERTIVE. Of >>> course PG can try to maintain data correct, but always somebody might >>> break it. >>> qu >>> >> Does it make sense to apply "assertive" CHECK constraint on the qual >> of ForeignScan to filter out tuples with violated values at the local >> side, as if row-level security feature doing. >> It enables to handle a situation that planner expects only "clean" >> tuples are returned but FDW driver is unavailable to anomalies. >> >> Probably, this additional check can be turned on/off on the fly, >> if FDW driver has a way to inform the core system its capability, >> like FDW_CAN_ENFORCE_CHECK_CONSTRAINT that informs planner to skip >> local checks. > > Hmm, IIUC you mean that local users can't (or don't need to) know that > data which violates the local constraints exist on remote side. > Applying constraints to the data which is modified through FDW would > be necessary as well. In that design, FDW is a bidirectional filter > which provides these features: > > 1) Don't push wrong data into remote data source, by applying local > constraints to the result of the modifying query executed on local PG. > This is not perfect filter, because remote constraints don't mapped > automatically or perfectly (imagine constraints which is available on > remote but is not supported in PG). > 2) Don't retrieve wrong data from remote to local PG, by applying > local constraints > Yes. (1) can be done with ExecConstraints prior to FDW callback on UPDATE or INSERT, even not a perfect solution because of side-channel on the remote data source. For (2), my proposition tries to drop retrieved violated tuples, however, the result is same. > I have a concern about consistency. It has not been supported, but > let's think of Aggregate push-down invoked by a query below. > > SELECT count(*) FROM remote_table; > > If this query was fully pushed down, the result is the # of records > exist on remote side, but the result would be # of valid records when > we don't push down the aggregate. This would confuse users. > Hmm. In this case, FDW driver needs to be responsible to push-down the additional check quals into remote side, so it does not work transparently towards the ForeignScan. It might be a little bit complicated suggestion for the beginning of the efforts. >>> Besides CHECK constraints, currently NOT NULL constraints are >>> virtually ASSERTIVE (not enforcing). Should it also be noted >>> explicitly? >>> >> Backward compatibility…. > > Yep, backward compatibility (especially visible ones to users) should > be minimal, ideally zero. > >> NOT NULL [ASSERTIVE] might be an option. > > Treating [ASSERTIVE | NOT ASSERTIVE] like DEFERRABLE, and allow > ingASSERTIVE for only foreign tables? It makes sense, though we need > consider exclusiveness . But It needs to default to ASSERTIVE on > foreign tables, and NOT ASSERTIVE (means "forced") on others. Isn't > is too complicated? > I think it is not easy to implement assertive checks, except for foreign tables, because all the write stuff to regular tables are managed by PostgreSQL itself. So, it is a good first step to add support "ASSERTIVE" CHECK on foreign table only, and to enforce FDW drivers nothing special from my personal sense. How about committer's opinion? Thanks, -- OSS Promotion Center / The PG-Strom Project KaiGai Kohei <kaigai@ak.jp.nec.com>
On Mon, Jan 20, 2014 at 9:44 PM, Shigeru Hanada
<shigeru.hanada@gmail.com> wrote:
> Thanks for the comments.
>
> 2014/1/21 KaiGai Kohei <kaigai@ak.jp.nec.com>:
>>> In addition, an idea which I can't throw away is to assume that all
>>> constraints defined on foreign tables as ASSERTIVE. Foreign tables
>>> potentially have dangers to have "wrong" data by updating source data
>>> not through foreign tables. This is not specific to an FDW, so IMO
>>> constraints defined on foreign tables are basically ASSERTIVE. Of
>>> course PG can try to maintain data correct, but always somebody might
>>> break it.
>>> qu
>>>
>> Does it make sense to apply "assertive" CHECK constraint on the qual
>> of ForeignScan to filter out tuples with violated values at the local
>> side, as if row-level security feature doing.
>> It enables to handle a situation that planner expects only "clean"
>> tuples are returned but FDW driver is unavailable to anomalies.
>>
>> Probably, this additional check can be turned on/off on the fly,
>> if FDW driver has a way to inform the core system its capability,
>> like FDW_CAN_ENFORCE_CHECK_CONSTRAINT that informs planner to skip
>> local checks.
>
> Hmm, IIUC you mean that local users can't (or don't need to) know that
> data which violates the local constraints exist on remote side.
> Applying constraints to the data which is modified through FDW would
> be necessary as well. In that design, FDW is a bidirectional filter
> which provides these features:
>
> 1) Don't push wrong data into remote data source, by applying local
> constraints to the result of the modifying query executed on local PG.
> This is not perfect filter, because remote constraints don't mapped
> automatically or perfectly (imagine constraints which is available on
> remote but is not supported in PG).
> 2) Don't retrieve wrong data from remote to local PG, by applying
> local constraints
>
> I have a concern about consistency. It has not been supported, but
> let's think of Aggregate push-down invoked by a query below.
>
> SELECT count(*) FROM remote_table;
>
> If this query was fully pushed down, the result is the # of records
> exist on remote side, but the result would be # of valid records when
> we don't push down the aggregate. This would confuse users.
>
>>> Besides CHECK constraints, currently NOT NULL constraints are
>>> virtually ASSERTIVE (not enforcing). Should it also be noted
>>> explicitly?
>>>
>> Backward compatibility….
>
> Yep, backward compatibility (especially visible ones to users) should
> be minimal, ideally zero.
>
>> NOT NULL [ASSERTIVE] might be an option.
>
> Treating [ASSERTIVE | NOT ASSERTIVE] like DEFERRABLE, and allow
> ingASSERTIVE for only foreign tables? It makes sense, though we need
> consider exclusiveness . But It needs to default to ASSERTIVE on
> foreign tables, and NOT ASSERTIVE (means "forced") on others. Isn't
> is too complicated?
>
> CREATE FOREIGN TABLE foo (
> id int NOT NULL ASSERTIVE CHECK (id > 1) ASSERTIVE,
> …
> CONSTRAINT chk_foo_name_upper CHECK (upper(name) = name) ASSERTIVE
> ) SERVER server;
>
> BTW, I noticed that this is like push-down-able expressions in
> JOIN/WHERE. We need to check a CHECK constraint defined on a foreign
> tables contains only expressions which have same semantics as remote
> side (in practice, built-in and immutable)?
I don't think that that ASSERTIVE is going to fly, because "assertive"
means (sayeth the Google) "having or showing a confident and forceful
personality", which is not what we mean here. It's tempting to do
something like try to replace the keyword "check" with "assume" or
"assert" or (stretching) "assertion", but that would require whichever
one we picked to be a fully-reserved keyword, which I can't think is
going to get much support here, for entirely understandable reasons.
So I think we should look for another option.
Currently, constraints can be marked NO INHERIT (though this seems to
have not been fully documented, as the ALTER TABLE page doesn't
mention it anywhere) or NOT VALID, so I'm thinking maybe we should go
with something along those lines. Some ideas:
- NO CHECK. The idea of writing CHECK (id > 1) NO CHECK is pretty
hilarious, though.
- NO VALIDATE. But then people need to understand that NOT VALID
means "we didn't validate it yet" while "no validate" means "we don't
ever intend to validate it", which could be confusing.
- NO ENFORCE. Requires a new (probably unreserved) keyword.
- NOT VALIDATED or NOT CHECKED. Same problems as NO CHECK and NO
VALIDATE, respectively, plus now we have to create a new keyword.
Another idea is to apply an extensible-options syntax to constraints,
like we do for EXPLAIN, VACUUM, etc. Like maybe:
CHECK (id > 1) OPTIONS (enforced false, valid true)
Yet another idea is to consider validity a three-state property:
either the constraint is valid (because we have checked it and are
enforcing it), or it is not valid (because we are enforcing it but
have not checked the pre-existing data), or it is assumed true
(because we are not checking or enforcing it but are believing it
anyway). So then we could have a syntax like this:
CHECK (id > 1) VALIDATE { ON | OFF | ASSERTION }
Other ideas?
One thing that's bugging me a bit about this whole line of attack is
that, in the first instance, the whole goal here is to support
inheritance hierarchies that mix ordinary tables with foreign tables.
If you have a table with children some of which are inherited and
others of which are not inherited, you're very likely going to want
your constraints enforced for real on the children that are tables and
assumed true on the children that are foreign tables, and none of what
we're talking about here gets us to that, because we normally want the
constraints to be identical throughout the inheritance hierarchy.
Maybe there's some way around that, but I'm back to wondering if it
wouldn't be better to simply silently force any constraints on a
foreign-table into assertion mode. That could be done without any new
syntax at all, and frankly I think it's what people are going to want
more often than not.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes:
> One thing that's bugging me a bit about this whole line of attack is
> that, in the first instance, the whole goal here is to support
> inheritance hierarchies that mix ordinary tables with foreign tables.
> If you have a table with children some of which are inherited and
> others of which are not inherited, you're very likely going to want
> your constraints enforced for real on the children that are tables and
> assumed true on the children that are foreign tables, and none of what
> we're talking about here gets us to that, because we normally want the
> constraints to be identical throughout the inheritance hierarchy.
There's a nearby thread that's addressing this same question, in which
I make the case (again) that the right thing for postgres_fdw constraints
is that they're just assumed true. So I'm not sure why this conversation
is proposing to implement a lot of mechanism to do something different
from that.
regards, tom lane
On Tue, Jan 21, 2014 at 3:00 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> One thing that's bugging me a bit about this whole line of attack is >> that, in the first instance, the whole goal here is to support >> inheritance hierarchies that mix ordinary tables with foreign tables. >> If you have a table with children some of which are inherited and >> others of which are not inherited, you're very likely going to want >> your constraints enforced for real on the children that are tables and >> assumed true on the children that are foreign tables, and none of what >> we're talking about here gets us to that, because we normally want the >> constraints to be identical throughout the inheritance hierarchy. > > There's a nearby thread that's addressing this same question, in which > I make the case (again) that the right thing for postgres_fdw constraints > is that they're just assumed true. So I'm not sure why this conversation > is proposing to implement a lot of mechanism to do something different > from that. /me scratches head. Because the other guy named Tom Lane took the opposite position on the second message on this thread, dated 11/14/13? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
(2014/01/22 4:09), Robert Haas wrote:
> On Mon, Jan 20, 2014 at 9:44 PM, Shigeru Hanada
> <shigeru.hanada@gmail.com> wrote:
>> Thanks for the comments.
>>
>> 2014/1/21 KaiGai Kohei <kaigai@ak.jp.nec.com>:
>>>> In addition, an idea which I can't throw away is to assume that all
>>>> constraints defined on foreign tables as ASSERTIVE. Foreign tables
>>>> potentially have dangers to have "wrong" data by updating source data
>>>> not through foreign tables. This is not specific to an FDW, so IMO
>>>> constraints defined on foreign tables are basically ASSERTIVE. Of
>>>> course PG can try to maintain data correct, but always somebody might
>>>> break it.
>>>> qu
>>>>
>>> Does it make sense to apply "assertive" CHECK constraint on the qual
>>> of ForeignScan to filter out tuples with violated values at the local
>>> side, as if row-level security feature doing.
>>> It enables to handle a situation that planner expects only "clean"
>>> tuples are returned but FDW driver is unavailable to anomalies.
>>>
>>> Probably, this additional check can be turned on/off on the fly,
>>> if FDW driver has a way to inform the core system its capability,
>>> like FDW_CAN_ENFORCE_CHECK_CONSTRAINT that informs planner to skip
>>> local checks.
>>
>> Hmm, IIUC you mean that local users can't (or don't need to) know that
>> data which violates the local constraints exist on remote side.
>> Applying constraints to the data which is modified through FDW would
>> be necessary as well. In that design, FDW is a bidirectional filter
>> which provides these features:
>>
>> 1) Don't push wrong data into remote data source, by applying local
>> constraints to the result of the modifying query executed on local PG.
>> This is not perfect filter, because remote constraints don't mapped
>> automatically or perfectly (imagine constraints which is available on
>> remote but is not supported in PG).
>> 2) Don't retrieve wrong data from remote to local PG, by applying
>> local constraints
>>
>> I have a concern about consistency. It has not been supported, but
>> let's think of Aggregate push-down invoked by a query below.
>>
>> SELECT count(*) FROM remote_table;
>>
>> If this query was fully pushed down, the result is the # of records
>> exist on remote side, but the result would be # of valid records when
>> we don't push down the aggregate. This would confuse users.
>>
>>>> Besides CHECK constraints, currently NOT NULL constraints are
>>>> virtually ASSERTIVE (not enforcing). Should it also be noted
>>>> explicitly?
>>>>
>>> Backward compatibility….
>>
>> Yep, backward compatibility (especially visible ones to users) should
>> be minimal, ideally zero.
>>
>>> NOT NULL [ASSERTIVE] might be an option.
>>
>> Treating [ASSERTIVE | NOT ASSERTIVE] like DEFERRABLE, and allow
>> ingASSERTIVE for only foreign tables? It makes sense, though we need
>> consider exclusiveness . But It needs to default to ASSERTIVE on
>> foreign tables, and NOT ASSERTIVE (means "forced") on others. Isn't
>> is too complicated?
>>
>> CREATE FOREIGN TABLE foo (
>> id int NOT NULL ASSERTIVE CHECK (id > 1) ASSERTIVE,
>> …
>> CONSTRAINT chk_foo_name_upper CHECK (upper(name) = name) ASSERTIVE
>> ) SERVER server;
>>
>> BTW, I noticed that this is like push-down-able expressions in
>> JOIN/WHERE. We need to check a CHECK constraint defined on a foreign
>> tables contains only expressions which have same semantics as remote
>> side (in practice, built-in and immutable)?
>
> I don't think that that ASSERTIVE is going to fly, because "assertive"
> means (sayeth the Google) "having or showing a confident and forceful
> personality", which is not what we mean here. It's tempting to do
> something like try to replace the keyword "check" with "assume" or
> "assert" or (stretching) "assertion", but that would require whichever
> one we picked to be a fully-reserved keyword, which I can't think is
> going to get much support here, for entirely understandable reasons.
> So I think we should look for another option.
>
> Currently, constraints can be marked NO INHERIT (though this seems to
> have not been fully documented, as the ALTER TABLE page doesn't
> mention it anywhere) or NOT VALID, so I'm thinking maybe we should go
> with something along those lines. Some ideas:
>
> - NO CHECK. The idea of writing CHECK (id > 1) NO CHECK is pretty
> hilarious, though.
> - NO VALIDATE. But then people need to understand that NOT VALID
> means "we didn't validate it yet" while "no validate" means "we don't
> ever intend to validate it", which could be confusing.
> - NO ENFORCE. Requires a new (probably unreserved) keyword.
> - NOT VALIDATED or NOT CHECKED. Same problems as NO CHECK and NO
> VALIDATE, respectively, plus now we have to create a new keyword.
>
> Another idea is to apply an extensible-options syntax to constraints,
> like we do for EXPLAIN, VACUUM, etc. Like maybe:
>
> CHECK (id > 1) OPTIONS (enforced false, valid true)
>
> Yet another idea is to consider validity a three-state property:
> either the constraint is valid (because we have checked it and are
> enforcing it), or it is not valid (because we are enforcing it but
> have not checked the pre-existing data), or it is assumed true
> (because we are not checking or enforcing it but are believing it
> anyway). So then we could have a syntax like this:
>
> CHECK (id > 1) VALIDATE { ON | OFF | ASSERTION }
>
> Other ideas?
>
> One thing that's bugging me a bit about this whole line of attack is
> that, in the first instance, the whole goal here is to support
> inheritance hierarchies that mix ordinary tables with foreign tables.
> If you have a table with children some of which are inherited and
> others of which are not inherited, you're very likely going to want
> your constraints enforced for real on the children that are tables and
> assumed true on the children that are foreign tables, and none of what
> we're talking about here gets us to that, because we normally want the
> constraints to be identical throughout the inheritance hierarchy.
> Maybe there's some way around that, but I'm back to wondering if it
> wouldn't be better to simply silently force any constraints on a
> foreign-table into assertion mode. That could be done without any new
> syntax at all, and frankly I think it's what people are going to want
> more often than not.
I'd like to vote for the idea of silently forcing any constraints on a
foreign-table into assertion mode. No new syntax and better documentation.
Thanks,
Best regards,
Etsuro Fujita
Hi Hanada-san, While still reviwing this patch, I feel this patch has given enough consideration to interactions with other commands, but found the following incorrect? behabior: postgres=# CREATE TABLE product (id INTEGER, description TEXT); CREATE TABLE postgres=# CREATE FOREIGN TABLE product1 () INHERITS (product) SERVER fs OPTIONS (filename '/home/foo/product1.csv', format 'csv'); CREATE FOREIGN TABLE postgres=# ALTER TABLE product ALTER COLUMN description SET STORAGE EXTERNAL; ERROR: "product1" is not a table or materialized view ISTN the ALTER TABLE simple recursion mechanism (ie ATSimpleRecursion()) should be modified for the ALTER COLUMN SET STORAGE case. I just wanted to quickly tell you this for you to take time to consider. Thanks, Best regards, Etsuro Fujita
2014-01-27 Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>: > While still reviwing this patch, I feel this patch has given enough > consideration to interactions with other commands, but found the following > incorrect? behabior: > > postgres=# CREATE TABLE product (id INTEGER, description TEXT); > CREATE TABLE > postgres=# CREATE FOREIGN TABLE product1 () INHERITS (product) SERVER fs > OPTIONS (filename '/home/foo/product1.csv', format 'csv'); > CREATE FOREIGN TABLE > postgres=# ALTER TABLE product ALTER COLUMN description SET STORAGE > EXTERNAL; > ERROR: "product1" is not a table or materialized view > > ISTN the ALTER TABLE simple recursion mechanism (ie ATSimpleRecursion()) > should be modified for the ALTER COLUMN SET STORAGE case. > > I just wanted to quickly tell you this for you to take time to consider. Thanks for the review. It must be an oversight, so I'll fix it up soon. -- Shigeru HANADA
On Mon, Jan 27, 2014 at 05:06:19PM +0900, Etsuro Fujita wrote: > Hi Hanada-san, > > While still reviwing this patch, I feel this patch has given enough > consideration to interactions with other commands, but found the > following incorrect? behabior: > > postgres=# CREATE TABLE product (id INTEGER, description TEXT); > CREATE TABLE > postgres=# CREATE FOREIGN TABLE product1 () INHERITS (product) > SERVER fs OPTIONS (filename '/home/foo/product1.csv', format 'csv'); > CREATE FOREIGN TABLE > postgres=# ALTER TABLE product ALTER COLUMN description SET STORAGE > EXTERNAL; > ERROR: "product1" is not a table or materialized view > > ISTN the ALTER TABLE simple recursion mechanism (ie > ATSimpleRecursion()) should be modified for the ALTER COLUMN SET > STORAGE case. This points to a larger discussion about what precisely foreign tables can and cannot inherit from local ones. I don't think that a generic solution will be satisfactory, as the PostgreSQL FDW could, at least in principle, support many more than the CSV FDW, as shown above. In my estimation, the outcome of discussion above is not a blocker for this patch. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
Sent from my iPad > On 27-Jan-2014, at 21:03, David Fetter <david@fetter.org> wrote: > >> On Mon, Jan 27, 2014 at 05:06:19PM +0900, Etsuro Fujita wrote: >> Hi Hanada-san, >> >> While still reviwing this patch, I feel this patch has given enough >> consideration to interactions with other commands, but found the >> following incorrect? behabior: >> >> postgres=# CREATE TABLE product (id INTEGER, description TEXT); >> CREATE TABLE >> postgres=# CREATE FOREIGN TABLE product1 () INHERITS (product) >> SERVER fs OPTIONS (filename '/home/foo/product1.csv', format 'csv'); >> CREATE FOREIGN TABLE >> postgres=# ALTER TABLE product ALTER COLUMN description SET STORAGE >> EXTERNAL; >> ERROR: "product1" is not a table or materialized view >> >> ISTN the ALTER TABLE simple recursion mechanism (ie >> ATSimpleRecursion()) should be modified for the ALTER COLUMN SET >> STORAGE case. > > This points to a larger discussion about what precisely foreign tables > can and cannot inherit from local ones. I don't think that a generic > solution will be satisfactory, as the PostgreSQL FDW could, at least > in principle, support many more than the CSV FDW, as shown above. > > In my estimation, the outcome of discussion above is not a blocker for > this I wonder what shall be the cases when foreign table is on a server which does not support *all* SQL features. Does a FDW need to have the possible inherit options mentioned in its documentation for this patch? Regards, Atri
(2014/01/28 0:55), Atri Sharma wrote: >> On 27-Jan-2014, at 21:03, David Fetter <david@fetter.org> wrote: >>> On Mon, Jan 27, 2014 at 05:06:19PM +0900, Etsuro Fujita wrote: >>> Hi Hanada-san, >>> >>> While still reviwing this patch, I feel this patch has given enough >>> consideration to interactions with other commands, but found the >>> following incorrect? behabior: >>> >>> postgres=# CREATE TABLE product (id INTEGER, description TEXT); >>> CREATE TABLE >>> postgres=# CREATE FOREIGN TABLE product1 () INHERITS (product) >>> SERVER fs OPTIONS (filename '/home/foo/product1.csv', format 'csv'); >>> CREATE FOREIGN TABLE >>> postgres=# ALTER TABLE product ALTER COLUMN description SET STORAGE >>> EXTERNAL; >>> ERROR: "product1" is not a table or materialized view >>> >>> ISTN the ALTER TABLE simple recursion mechanism (ie >>> ATSimpleRecursion()) should be modified for the ALTER COLUMN SET >>> STORAGE case. >> >> This points to a larger discussion about what precisely foreign tables >> can and cannot inherit from local ones. I don't think that a generic >> solution will be satisfactory, as the PostgreSQL FDW could, at least >> in principle, support many more than the CSV FDW, as shown above. >> >> In my estimation, the outcome of discussion above is not a blocker for >> this I just thought that among the structures that local tables can alter, the ones that foreign tables also can by ALTER FOREIGN TABLE are inherited, and the others are not inherited. So for the case as shown above, I thought that we silently ignore executing the ALTER COLUMN SET STORAGE command for the foreign table. I wonder that would be the first step. > I wonder what shall be the cases when foreign table is on a server which does not support *all* SQL features. > > Does a FDW need to have the possible inherit options mentioned in its documentation for this patch? The answer is no, in my understanding. The altering operation simply declares some chages for foreign tables in the inheritance tree and does nothing to the underlying storages. Thanks, Best regards, Etsuro Fujita
> > I wonder what shall be the cases when foreign table is on a server which > does not support *all* SQL features. > > > > Does a FDW need to have the possible inherit options mentioned in its > documentation for this patch? > > The answer is no, in my understanding. The altering operation simply > declares some chages for foreign tables in the inheritance tree and does > nothing to the underlying storages. > It seems to me Atri mention about the idea to enforce constraint when partitioned foreign table was referenced... BTW, isn't it feasible to consign FDW drivers its decision? If a foreign table has a check constraint (X BETWEEN 101 AND 200), postgres_fdw will be capable to run this check on the remote server, however, file_fdw will not be capable. It is usual job of them when qualifiers are attached on Path node. Probably, things to do is simple. If the backend appends the configured check constraint as if a user-given qualifier, FDW driver will handle it appropriately. Thanks, -- NEC OSS Promotion Center / PG-Strom Project KaiGai Kohei <kaigai@ak.jp.nec.com> > -----Original Message----- > From: pgsql-hackers-owner@postgresql.org > [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Etsuro Fujita > Sent: Tuesday, January 28, 2014 1:08 PM > To: Atri Sharma; David Fetter > Cc: pgsql-hackers@postgresql.org > Subject: Re: [HACKERS] inherit support for foreign tables > > (2014/01/28 0:55), Atri Sharma wrote: > >> On 27-Jan-2014, at 21:03, David Fetter <david@fetter.org> wrote: > >>> On Mon, Jan 27, 2014 at 05:06:19PM +0900, Etsuro Fujita wrote: > >>> Hi Hanada-san, > >>> > >>> While still reviwing this patch, I feel this patch has given enough > >>> consideration to interactions with other commands, but found the > >>> following incorrect? behabior: > >>> > >>> postgres=# CREATE TABLE product (id INTEGER, description TEXT); > >>> CREATE TABLE postgres=# CREATE FOREIGN TABLE product1 () INHERITS > >>> (product) SERVER fs OPTIONS (filename '/home/foo/product1.csv', > >>> format 'csv'); CREATE FOREIGN TABLE postgres=# ALTER TABLE product > >>> ALTER COLUMN description SET STORAGE EXTERNAL; > >>> ERROR: "product1" is not a table or materialized view > >>> > >>> ISTN the ALTER TABLE simple recursion mechanism (ie > >>> ATSimpleRecursion()) should be modified for the ALTER COLUMN SET > >>> STORAGE case. > >> > >> This points to a larger discussion about what precisely foreign > >> tables can and cannot inherit from local ones. I don't think that a > >> generic solution will be satisfactory, as the PostgreSQL FDW could, > >> at least in principle, support many more than the CSV FDW, as shown above. > >> > >> In my estimation, the outcome of discussion above is not a blocker > >> for this > > I just thought that among the structures that local tables can alter, the > ones that foreign tables also can by ALTER FOREIGN TABLE are inherited, > and the others are not inherited. So for the case as shown above, I thought > that we silently ignore executing the ALTER COLUMN SET STORAGE command for > the foreign table. I wonder that would be the first step. > > > I wonder what shall be the cases when foreign table is on a server which > does not support *all* SQL features. > > > > Does a FDW need to have the possible inherit options mentioned in its > documentation for this patch? > > The answer is no, in my understanding. The altering operation simply > declares some chages for foreign tables in the inheritance tree and does > nothing to the underlying storages. > > Thanks, > > Best regards, > Etsuro Fujita > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make > changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers
(2014/01/27 21:52), Shigeru Hanada wrote:
> 2014-01-27 Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>:
>> While still reviwing this patch, I feel this patch has given enough
>> consideration to interactions with other commands, but found the following
>> incorrect? behabior:
>>
>> postgres=# CREATE TABLE product (id INTEGER, description TEXT);
>> CREATE TABLE
>> postgres=# CREATE FOREIGN TABLE product1 () INHERITS (product) SERVER fs
>> OPTIONS (filename '/home/foo/product1.csv', format 'csv');
>> CREATE FOREIGN TABLE
>> postgres=# ALTER TABLE product ALTER COLUMN description SET STORAGE
>> EXTERNAL;
>> ERROR: "product1" is not a table or materialized view
>>
>> ISTN the ALTER TABLE simple recursion mechanism (ie ATSimpleRecursion())
>> should be modified for the ALTER COLUMN SET STORAGE case.
>>
>> I just wanted to quickly tell you this for you to take time to consider.
>
> Thanks for the review. It must be an oversight, so I'll fix it up soon.
>
It seems little bit complex than I expected. Currently foreign tables
deny ALTER TABLE SET STORAGE with message like below, because foreign
tables don't have storage in the meaning of PG heap tables.
ERROR: "pgbench1_accounts_c1" is not a table or materialized view
At the moment we don't use attstorage for foreign tables, so allowing
SET STORAGE against foreign tables never introduce visible change
except \d+ output of foreign tables. But IMO such operation should
not allowed because users would be confused. So I changed
ATExecSetStorage() to skip on foreign tables. This allows us to emit
ALTER TABLE SET STORAGE against ordinary tables in upper level of
inheritance tree, but it have effect on only ordinary tables in the
tree.
This also allows direct ALTER FOREIGN TABLE SET STORAGE against
foreign table but the command is silently ignored. SET STORAGE
support for foreign tables is not documented because it may confuse
users.
With attached patch, SET STORAGE against wrong relations produces
message like this. Is it confusing to mention foreign table here?
ERROR: "pgbench1_accounts_pkey" is not a table, materialized view, or
foreign table
One other idea is to support SET STORAGE against foreign tables though
it has no effect. This makes all tables and foreign tables to have
same storage option in same column. It might be more understandable
behavior for users.
Thoughts?
FYI, here are lists of ALTER TABLE features which is allowed/denied
for foreign tables.
Allowed
- ADD|DROP COLUMN
- SET DATA TYPE
- SET|DROP NOT NULL
- SET STATISTICS
- SET (attribute_option = value)
- RESET (atrribute_option)
- SET|DROP DEFAULT
- ADD table_constraint
- ALTER CONSTRAINT
- DROP CONSTRAINT
- [NO] INHERIT parent_table
- OWNER
- RENAME
- SET SCHEMA
- SET STORAGE
- moved to here by attached patch
Denied
- ADD table_constraint_using_index
- VALIDATE CONSTRAINT
- DISABLE|ENABLE TRIGGER
- DISABLE|ENABLE RULE
- CLUSTER ON
- SET WITHOUT CLUSTER
- SET WITH|WITHOUT OIDS
- SET (storage_parameter = value)
- RESET (storage_parameter)
- OF type
- NOT OF
- SET TABLESPACE
- REPLICA IDENTITY
--
Shigeru HANADA
--
Shigeru HANADA
Attachment
Sent from my iPad On 28-Jan-2014, at 10:04, Kouhei Kaigai <kaigai@ak.jp.nec.com> wrote: >>> I wonder what shall be the cases when foreign table is on a server which >> does not support *all* SQL features. >>> >>> Does a FDW need to have the possible inherit options mentioned in its >> documentation for this patch? >> >> The answer is no, in my understanding. The altering operation simply >> declares some chages for foreign tables in the inheritance tree and does >> nothing to the underlying storages. > It seems to me Atri mention about the idea to enforce constraint when > partitioned foreign table was referenced... > > BTW, isn't it feasible to consign FDW drivers its decision? > If a foreign table has a check constraint (X BETWEEN 101 AND 200), > postgres_fdw will be capable to run this check on the remote server, > however, file_fdw will not be capable. It is usual job of them when > qualifiers are attached on Path node. > Probably, things to do is simple. If the backend appends the configured > check constraint as if a user-given qualifier, FDW driver will handle Hi, I think that pushing it to FDW driver is the best approach. The FDW driver knows whether or not the foreign server supportsthe said feature,hence,the user should be abstracted from that. I agree that the foreign constraint can be added as a user defined qualifier and dealt with accordingly. Regards, Atri
Shigeru Hanada <shigeru.hanada@gmail.com> writes:
> At the moment we don't use attstorage for foreign tables, so allowing
> SET STORAGE against foreign tables never introduce visible change
> except \d+ output of foreign tables. But IMO such operation should
> not allowed because users would be confused. So I changed
> ATExecSetStorage() to skip on foreign tables.
I think this is totally misguided. Who's to say that some weird FDW
might not pay attention to attstorage? I could imagine a file-based
FDW using that to decide whether to compress columns, for instance.
Admittedly, the chances of that aren't large, but it's pretty hard
to argue that going out of our way to prevent it is a useful activity.
regards, tom lane
On Thu, Jan 30, 2014 at 11:04 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Shigeru Hanada <shigeru.hanada@gmail.com> writes: >> At the moment we don't use attstorage for foreign tables, so allowing >> SET STORAGE against foreign tables never introduce visible change >> except \d+ output of foreign tables. But IMO such operation should >> not allowed because users would be confused. So I changed >> ATExecSetStorage() to skip on foreign tables. > > I think this is totally misguided. Who's to say that some weird FDW > might not pay attention to attstorage? I could imagine a file-based > FDW using that to decide whether to compress columns, for instance. > Admittedly, the chances of that aren't large, but it's pretty hard > to argue that going out of our way to prevent it is a useful activity. I think that's a pretty tenuous position. There are already FDW-specific options sufficient to let a particular FDW store whatever kinds of options it likes; letting the user set options that were only ever intended to be applied to tables just because we can seems sort of dubious. I'm tempted by the idea of continuing to disallow SET STORAGE on an unvarnished foreign table, but allowing it on an inheritance hierarchy that contains at least one real table, with the semantics that we quietly ignore the foreign tables and apply the operation to the plain tables. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, Jan 30, 2014 at 11:04 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I think this is totally misguided. Who's to say that some weird FDW
>> might not pay attention to attstorage? I could imagine a file-based
>> FDW using that to decide whether to compress columns, for instance.
>> Admittedly, the chances of that aren't large, but it's pretty hard
>> to argue that going out of our way to prevent it is a useful activity.
> I think that's a pretty tenuous position. There are already
> FDW-specific options sufficient to let a particular FDW store whatever
> kinds of options it likes; letting the user set options that were only
> ever intended to be applied to tables just because we can seems sort
> of dubious. I'm tempted by the idea of continuing to disallow SET
> STORAGE on an unvarnished foreign table, but allowing it on an
> inheritance hierarchy that contains at least one real table, with the
> semantics that we quietly ignore the foreign tables and apply the
> operation to the plain tables.
[ shrug... ] By far the easiest implementation of that is just to apply
the catalog change to all of them. According to your assumptions, it'll
be a no-op on the foreign tables anyway.
regards, tom lane
On Thu, Jan 30, 2014 at 5:05 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> On Thu, Jan 30, 2014 at 11:04 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >>> I think this is totally misguided. Who's to say that some weird FDW >>> might not pay attention to attstorage? I could imagine a file-based >>> FDW using that to decide whether to compress columns, for instance. >>> Admittedly, the chances of that aren't large, but it's pretty hard >>> to argue that going out of our way to prevent it is a useful activity. > >> I think that's a pretty tenuous position. There are already >> FDW-specific options sufficient to let a particular FDW store whatever >> kinds of options it likes; letting the user set options that were only >> ever intended to be applied to tables just because we can seems sort >> of dubious. I'm tempted by the idea of continuing to disallow SET >> STORAGE on an unvarnished foreign table, but allowing it on an >> inheritance hierarchy that contains at least one real table, with the >> semantics that we quietly ignore the foreign tables and apply the >> operation to the plain tables. > > [ shrug... ] By far the easiest implementation of that is just to apply > the catalog change to all of them. According to your assumptions, it'll > be a no-op on the foreign tables anyway. Well, there's some point to that, too, I suppose. What do others think? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
* Robert Haas (robertmhaas@gmail.com) wrote:
> On Thu, Jan 30, 2014 at 5:05 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > Robert Haas <robertmhaas@gmail.com> writes:
> >> On Thu, Jan 30, 2014 at 11:04 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >>> I think this is totally misguided. Who's to say that some weird FDW
> >>> might not pay attention to attstorage? I could imagine a file-based
> >>> FDW using that to decide whether to compress columns, for instance.
> >>> Admittedly, the chances of that aren't large, but it's pretty hard
> >>> to argue that going out of our way to prevent it is a useful activity.
> >
> >> I think that's a pretty tenuous position. There are already
> >> FDW-specific options sufficient to let a particular FDW store whatever
> >> kinds of options it likes; letting the user set options that were only
> >> ever intended to be applied to tables just because we can seems sort
> >> of dubious. I'm tempted by the idea of continuing to disallow SET
> >> STORAGE on an unvarnished foreign table, but allowing it on an
> >> inheritance hierarchy that contains at least one real table, with the
> >> semantics that we quietly ignore the foreign tables and apply the
> >> operation to the plain tables.
> >
> > [ shrug... ] By far the easiest implementation of that is just to apply
> > the catalog change to all of them. According to your assumptions, it'll
> > be a no-op on the foreign tables anyway.
>
> Well, there's some point to that, too, I suppose. What do others think?
I agree that using the FDW-specific options is the right approach and
disallowing those to be set on foreign tables makes sense. I don't
particularly like the idea of applying changes during inheiritance
which we wouldn't allow the user to do directly. While it might be a
no-op and no FDW might use it, it'd still be confusing.
Thanks,
Stephen
Stephen Frost <sfrost@snowman.net> writes:
> I agree that using the FDW-specific options is the right approach and
> disallowing those to be set on foreign tables makes sense. I don't
> particularly like the idea of applying changes during inheiritance
> which we wouldn't allow the user to do directly. While it might be a
> no-op and no FDW might use it, it'd still be confusing.
If we don't allow it directly, why not? Seems rather arbitrary.
regards, tom lane
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > I agree that using the FDW-specific options is the right approach and
> > disallowing those to be set on foreign tables makes sense. I don't
> > particularly like the idea of applying changes during inheiritance
> > which we wouldn't allow the user to do directly. While it might be a
> > no-op and no FDW might use it, it'd still be confusing.
>
> If we don't allow it directly, why not? Seems rather arbitrary.
I'm apparently missing something here. From my perspective, they're
either allowed or they're not and if they aren't allowed then they
shouldn't be allowed to happen. I'd view this like a CHECK constraint-
if it's a foreign table then it can't have a value for SET STORAGE. I
don't see why it would be 'ok' to allow it to be set if we're setting it
as part of inheiritance but otherwise not allow it to be set.
Thanks,
Stephen
(2014/01/31 9:56), Robert Haas wrote: > On Thu, Jan 30, 2014 at 5:05 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> Robert Haas <robertmhaas@gmail.com> writes: >>> On Thu, Jan 30, 2014 at 11:04 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >>>> I think this is totally misguided. Who's to say that some weird FDW >>>> might not pay attention to attstorage? I could imagine a file-based >>>> FDW using that to decide whether to compress columns, for instance. >>>> Admittedly, the chances of that aren't large, but it's pretty hard >>>> to argue that going out of our way to prevent it is a useful activity. >> >>> I think that's a pretty tenuous position. There are already >>> FDW-specific options sufficient to let a particular FDW store whatever >>> kinds of options it likes; letting the user set options that were only >>> ever intended to be applied to tables just because we can seems sort >>> of dubious. I'm tempted by the idea of continuing to disallow SET >>> STORAGE on an unvarnished foreign table, but allowing it on an >>> inheritance hierarchy that contains at least one real table, with the >>> semantics that we quietly ignore the foreign tables and apply the >>> operation to the plain tables. >> >> [ shrug... ] By far the easiest implementation of that is just to apply >> the catalog change to all of them. According to your assumptions, it'll >> be a no-op on the foreign tables anyway. > > Well, there's some point to that, too, I suppose. What do others think? Allowing ALTER COLUMN SET STORAGE on foreign tables would make sense if for example, "SELECT * INTO local_table FROM foreign_table" did create a new local table of columns having the storage types associated with those of a foreign table? Thanks, Best regards, Etsuro Fujita
On Sun, Feb 2, 2014 at 10:15 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote: > Allowing ALTER COLUMN SET STORAGE on foreign tables would make sense if for > example, "SELECT * INTO local_table FROM foreign_table" did create a new > local table of columns having the storage types associated with those of a > foreign table? Seems like a pretty weak argument. It's not that we can't find strange corner cases where applying SET STORAGE to a foreign table doesn't do something; it's that they *are* strange corner cases. The options as we normally don't understand them just aren't sensible in this context, and a good deal of work has been put into an alternative options framework, which is what authors of FDWs ought to be using. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
(2014/02/04 20:56), Robert Haas wrote: > On Sun, Feb 2, 2014 at 10:15 PM, Etsuro Fujita > <fujita.etsuro@lab.ntt.co.jp> wrote: >> Allowing ALTER COLUMN SET STORAGE on foreign tables would make sense if for >> example, "SELECT * INTO local_table FROM foreign_table" did create a new >> local table of columns having the storage types associated with those of a >> foreign table? > > Seems like a pretty weak argument. It's not that we can't find > strange corner cases where applying SET STORAGE to a foreign table > doesn't do something; it's that they *are* strange corner cases. The > options as we normally don't understand them just aren't sensible in > this context, and a good deal of work has been put into an alternative > options framework, which is what authors of FDWs ought to be using. I just wanted to discuss the possiblity of allowing SET STORAGE on a foreign table, but I've got the point. I'll resume the patch review. Thanks, Best regards, Etsuro Fujita
Hi Hanada-san, Sorry for the delay. (2014/01/30 14:01), Shigeru Hanada wrote: >> 2014-01-27 Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>: >>> While still reviwing this patch, I feel this patch has given enough >>> consideration to interactions with other commands, but found the following >>> incorrect? behabior: >>> >>> postgres=# CREATE TABLE product (id INTEGER, description TEXT); >>> CREATE TABLE >>> postgres=# CREATE FOREIGN TABLE product1 () INHERITS (product) SERVER fs >>> OPTIONS (filename '/home/foo/product1.csv', format 'csv'); >>> CREATE FOREIGN TABLE >>> postgres=# ALTER TABLE product ALTER COLUMN description SET STORAGE >>> EXTERNAL; >>> ERROR: "product1" is not a table or materialized view >>> >>> ISTN the ALTER TABLE simple recursion mechanism (ie ATSimpleRecursion()) >>> should be modified for the ALTER COLUMN SET STORAGE case. > It seems little bit complex than I expected. Currently foreign tables > deny ALTER TABLE SET STORAGE with message like below, because foreign > tables don't have storage in the meaning of PG heap tables. > > ERROR: "pgbench1_accounts_c1" is not a table or materialized view > > At the moment we don't use attstorage for foreign tables, so allowing > SET STORAGE against foreign tables never introduce visible change > except \d+ output of foreign tables. But IMO such operation should > not allowed because users would be confused. So I changed > ATExecSetStorage() to skip on foreign tables. This allows us to emit > ALTER TABLE SET STORAGE against ordinary tables in upper level of > inheritance tree, but it have effect on only ordinary tables in the > tree. > > This also allows direct ALTER FOREIGN TABLE SET STORAGE against > foreign table but the command is silently ignored. SET STORAGE > support for foreign tables is not documented because it may confuse > users. > > With attached patch, SET STORAGE against wrong relations produces > message like this. Is it confusing to mention foreign table here? > > ERROR: "pgbench1_accounts_pkey" is not a table, materialized view, or > foreign table ITSM that would be confusing to users. So, I've modified the patch so that we continue to disallow SET STORAGE on a foreign table *in the same manner as before*, but, as your patch does, allow it on an inheritance hierarchy that contains foreign tables, with the semantics that we quietly ignore the foreign tables and apply the operation to the plain tables, by modifying the ALTER TABLE simple recursion mechanism. Attached is the updated version of the patch. I'll continue the patch review a bit more, though the patch looks good generally to me except for the abobe issue and the way that the ANALYZE command works. For the latter, if there are no objections, I'll merge the ANALYZE patch [1] with your patch. Thanks, [1] http://www.postgresql.org/message-id/52EB10AC.4070307@lab.ntt.co.jp Best regards, Etsuro Fujita
Attachment
(2014/02/07 21:31), Etsuro Fujita wrote: > So, I've modified the patch so > that we continue to disallow SET STORAGE on a foreign table *in the same > manner as before*, but, as your patch does, allow it on an inheritance > hierarchy that contains foreign tables, with the semantics that we > quietly ignore the foreign tables and apply the operation to the plain > tables, by modifying the ALTER TABLE simple recursion mechanism. > Attached is the updated version of the patch. > > I'll continue the patch review a bit more, though the patch looks good > generally to me except for the abobe issue and the way that the ANALYZE > command works. While reviwing the patch, I've found some issues on interactions with other commands, other than the SET STORAGE command. * ADD table_constraint NOT VALID: the patch allows ADD table_constraint *NOT VALID* to be set on a foreign table as well as an inheritance hierarchy that contains foreign tables. But I think it would be better to disallow ADD table_constraint *NOT VALID* on a foreign table, but allow it on an inheritance hierarchy that contains foreign tables, with the semantics that we apply the operation to the plain tables and apply the transformed operation *ADD table_constraint* to the foreign tables. * VALIDATE CONSTRAINT constraint_name: though the patch disallow the direct operation on foreign tables, it allows the operation on an inheritance hierarchy that contains foreign tables, and the operation fails if there is at least one foreign table that has at least one NOT VALID constraint. I think it would be better to modify the patch so that we allow the operation on an inheritance hierarchy that contains foreign tables, with the semantics that we quietly ignore the foreign tables and apply the operation on the plain tables just like the semantics of SET STORAGE. (Note that the foreign tables wouldn't have NOT VALID constraints by diallowing ADD table_constraint *NOT VALID* on foreign tables as mentioned above.) * SET WITH OIDS: though the patch disallow the direct operation on foreign tables, it allows the operation on an inheritance hierarchy that contains foreign tables, and that operation is successfully done on foreign tables. I think it would be better to modify the patch so that we allow it on an inheritance hierarchy that contains foreign tables, with the semantics that we quietly ignore the foreign tables and apply the operation to the plain tables just like the semantics of SET STORAGE. Comments are wellcome! Thanks, Best regards, Etsuro Fujita
(2014/02/10 21:00), Etsuro Fujita wrote:
> (2014/02/07 21:31), Etsuro Fujita wrote:
>> So, I've modified the patch so
>> that we continue to disallow SET STORAGE on a foreign table *in the same
>> manner as before*, but, as your patch does, allow it on an inheritance
>> hierarchy that contains foreign tables, with the semantics that we
>> quietly ignore the foreign tables and apply the operation to the plain
>> tables, by modifying the ALTER TABLE simple recursion mechanism.
> While reviwing the patch, I've found some issues on interactions with
> other commands, other than the SET STORAGE command.
> * ADD table_constraint NOT VALID: the patch allows ADD table_constraint
> *NOT VALID* to be set on a foreign table as well as an inheritance
> hierarchy that contains foreign tables. But I think it would be better
> to disallow ADD table_constraint *NOT VALID* on a foreign table, but
> allow it on an inheritance hierarchy that contains foreign tables, with
> the semantics that we apply the operation to the plain tables and apply
> the transformed operation *ADD table_constraint* to the foreign tables.
Done.
> * VALIDATE CONSTRAINT constraint_name:
I've modified the patch so that though we continue to disallow the
operation on a foreign table, we allow it on an inheritance hierarchy
that contains foreign tables. In that case, the to-be-validated
constraints on the plain tables are validated by the operation, as
before, and the constraints on the foreign tables are ignored. Note
that the constraints on the foreign tables are not NOT VALID due to the
spec mentioned above.
> * SET WITH OIDS: though the patch disallow the direct operation on
> foreign tables, it allows the operation on an inheritance hierarchy that
> contains foreign tables, and that operation is successfully done on
> foreign tables. I think it would be better to modify the patch so that
> we allow it on an inheritance hierarchy that contains foreign tables,
> with the semantics that we quietly ignore the foreign tables and apply
> the operation to the plain tables just like the semantics of SET STORAGE.
I noticed this breaks inheritance hierarchy. To avoid this, I've
modified the patch to disallow SET WITH OIDS on an inheritance hierarchy
that contains at least one foreign table.
The other changes:
- if (stmt->constraints != NIL && relkind == RELKIND_FOREIGN_TABLE)
- ereport(ERROR,
- (errcode(ERRCODE_WRONG_OBJECT_TYPE),
- errmsg("constraints are not supported
on foreign tables")));
+/*
+ * Shouldn't this have been checked in parser?
+ */
+ if (relkind == RELKIND_FOREIGN_TABLE)
+ {
+ ListCell *lc;
+ foreach(lc, stmt->constraints)
+ {
+ NewConstraint *nc = lfirst(lc);
+
+ if (nc->contype != CONSTR_CHECK &&
+ nc->contype != CONSTR_DEFAULT &&
+ nc->contype != CONSTR_NULL &&
+ nc->contype != CONSTR_NOTNULL)
+ ereport(ERROR,
+
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("only check
constraints are supported on foreign tables")));
+ }
+ }
ISTM we wouldn't need the above check in DefineRelation(), so I've
removed the check from the patch. And I've modified the vague error
messages in parse_utilcmd.c.
There seems to be no objections, I've merged the ANALYZE patch [1] with
your patch. I noticed that the patch in [1] isn't efficient. (To
ANALYZE one inheritance tree that contains foreign tables, the patch in
[1] calls the AnalyzeForeignTable() routine two times for each such
foreign table.) So, the updated version has been merged that calls the
routine only once for each such foreign table.
Todo:
ISTM the documentation would need to be updated further.
Thanks,
[1] http://www.postgresql.org/message-id/52EB10AC.4070307@lab.ntt.co.jp
Best regards,
Etsuro Fujita
Attachment
Hi Fujita-san, Thanks for the reviewing. 2014-02-10 21:00 GMT+09:00 Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>: > (2014/02/07 21:31), Etsuro Fujita wrote: >> So, I've modified the patch so >> that we continue to disallow SET STORAGE on a foreign table *in the same >> manner as before*, but, as your patch does, allow it on an inheritance >> hierarchy that contains foreign tables, with the semantics that we >> quietly ignore the foreign tables and apply the operation to the plain >> tables, by modifying the ALTER TABLE simple recursion mechanism. >> Attached is the updated version of the patch. I'm not sure that allowing ALTER TABLE against parent table affects descendants even some of them are foreign table. I think the rule should be simple enough to understand for users, of course it should be also consistent and have backward compatibility. If foreign table can be modified through inheritance tree, this kind of change can be done. 1) create foreign table as a child of a ordinary table 2) run ALTER TABLE parent, the foreign table is also changed 3) remove foreign table from the inheritance tree by ALTER TABLE child NO INHERIT parent 4) here we can't do same thing as 2), because it is not a child anymore. So IMO we should determine which ALTER TABLE features are allowed to foreign tables, and allow them regardless of the recursivity. Comments? -- Shigeru HANADA
> From: Shigeru Hanada [mailto:shigeru.hanada@gmail.com] > 2014-02-10 21:00 GMT+09:00 Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>: > > (2014/02/07 21:31), Etsuro Fujita wrote: > >> So, I've modified the patch so > >> that we continue to disallow SET STORAGE on a foreign table *in the > >> same manner as before*, but, as your patch does, allow it on an > >> inheritance hierarchy that contains foreign tables, with the > >> semantics that we quietly ignore the foreign tables and apply the > >> operation to the plain tables, by modifying the ALTER TABLE simple > recursion mechanism. > >> Attached is the updated version of the patch. > > I'm not sure that allowing ALTER TABLE against parent table affects > descendants even some of them are foreign table. I think the rule should > be simple enough to understand for users, of course it should be also > consistent and have backward compatibility. Yeah, the understandability is important. But I think the flexibility is also important. In other words, I think it isa bit too inflexible that we disallow e.g., SET STORAGE to be set on an inheritance tree that contains foreign table(s)because we disallow SET STORAGE to be set on foreign tables directly. > If foreign table can be modified through inheritance tree, this kind of > change can be done. > > 1) create foreign table as a child of a ordinary table > 2) run ALTER TABLE parent, the foreign table is also changed > 3) remove foreign table from the inheritance tree by ALTER TABLE child NO > INHERIT parent > 4) here we can't do same thing as 2), because it is not a child anymore. > > So IMO we should determine which ALTER TABLE features are allowed to foreign > tables, and allow them regardless of the recursivity. What I think should be newly allowed to be set on foreign tables is * ADD table_constraint * DROP CONSTRAINT * [NO] INHERIT parent_table Thanks, Best regards, Etsuro Fujita
Hello, > 2014-02-10 21:00 GMT+09:00 Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>: > > (2014/02/07 21:31), Etsuro Fujita wrote: > >> So, I've modified the patch so > >> that we continue to disallow SET STORAGE on a foreign table *in the same > >> manner as before*, but, as your patch does, allow it on an inheritance > >> hierarchy that contains foreign tables, with the semantics that we > >> quietly ignore the foreign tables and apply the operation to the plain > >> tables, by modifying the ALTER TABLE simple recursion mechanism. > >> Attached is the updated version of the patch. > > I'm not sure that allowing ALTER TABLE against parent table affects > descendants even some of them are foreign table. I think the rule > should be simple enough to understand for users, of course it should > be also consistent and have backward compatibility. Could you guess any use cases in which we are happy with ALTER TABLE's inheritance tree walking? IMHO, ALTER FOREIGN TABLE always comes with some changes of the data source so implicitly invoking of such commands should be defaultly turned off. If the ALTER'ing the whole familiy is deadfully useful for certain cases, it might be explicitly turned on by some syntax added to CREATE FOREIGN TABLE or ALTER TABLE. It would looks like following, CREATE FOREIGN TABLE ft1 () INHERITS (pt1 ALLOW_INHERITED_ALTER, pt2); ALTER TABLE INCLUDE FOREIGN CHILDREN parent ADD COLUMN add1 integer; These looks quite bad :-( but also seem quite better than accidentially ALTER'ing foreign children. If foreign tables were allowed to ALTER'ed with 'ALTER TABLE', some reconstruction between 'ALTER TABLE' and 'ALTER FOREIGN TABLE' would be needed. > If foreign table can be modified through inheritance tree, this kind > of change can be done. > > 1) create foreign table as a child of a ordinary table > 2) run ALTER TABLE parent, the foreign table is also changed > 3) remove foreign table from the inheritance tree by ALTER TABLE child > NO INHERIT parent > 4) here we can't do same thing as 2), because it is not a child anymore. > > So IMO we should determine which ALTER TABLE features are allowed to > foreign tables, and allow them regardless of the recursivity. > > Comments? regards, -- Kyotaro Horiguchi NTT Open Source Software Center
Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> writes:
> Could you guess any use cases in which we are happy with ALTER
> TABLE's inheritance tree walking? IMHO, ALTER FOREIGN TABLE
> always comes with some changes of the data source so implicitly
> invoking of such commands should be defaultly turned off. If the
> ALTER'ing the whole familiy is deadfully useful for certain
> cases, it might be explicitly turned on by some syntax added to
> CREATE FOREIGN TABLE or ALTER TABLE.
> It would looks like following,
> CREATE FOREIGN TABLE ft1 () INHERITS (pt1 ALLOW_INHERITED_ALTER, pt2);
> ALTER TABLE INCLUDE FOREIGN CHILDREN parent ADD COLUMN add1 integer;
Ugh. This adds a lot of logical complication without much real use,
IMO. The problems that have been discussed in this thread are that
certain types of ALTER are sensible to apply to foreign tables and
others are not --- so how does a one-size-fits-all switch help that?
Also, there are some types of ALTER for which recursion to children
*must* occur or things become inconsistent --- ADD COLUMN itself is
an example, and ADD CONSTRAINT is another. It would not be acceptable
for an ALLOW_INHERITED_ALTER switch to suppress that, but if the
switch is leaky then it's even less consistent and harder to explain.
regards, tom lane
Hello, At Tue, 18 Feb 2014 09:49:23 -0500, Tom Lane wrote > Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> writes: > > Could you guess any use cases in which we are happy with ALTER > > TABLE's inheritance tree walking? IMHO, ALTER FOREIGN TABLE > > always comes with some changes of the data source so implicitly > > invoking of such commands should be defaultly turned off. If the > > ALTER'ing the whole familiy is deadfully useful for certain > > cases, it might be explicitly turned on by some syntax added to > > CREATE FOREIGN TABLE or ALTER TABLE. > > > It would looks like following, > > > CREATE FOREIGN TABLE ft1 () INHERITS (pt1 ALLOW_INHERITED_ALTER, pt2); > > > ALTER TABLE INCLUDE FOREIGN CHILDREN parent ADD COLUMN add1 integer; > > Ugh. This adds a lot of logical complication without much real use, > IMO. Yeah! That's exactry the words I wanted for the expamples. Sorry for bothering you. I also don't see any use case driving us to overcome the extra complexity. > The problems that have been discussed in this thread are that > certain types of ALTER are sensible to apply to foreign tables and > others are not --- so how does a one-size-fits-all switch help that? None, I think. I intended only to display what this ALTER's inheritance walkthrough brings in. > Also, there are some types of ALTER for which recursion to children > *must* occur or things become inconsistent --- ADD COLUMN itself is > an example, and ADD CONSTRAINT is another. It would not be acceptable > for an ALLOW_INHERITED_ALTER switch to suppress that, but if the > switch is leaky then it's even less consistent and harder to explain. Exactly. It is the problem came with allowing to implicit ALTER on foreign children, Shigeru's last message seems to referred to. At Tue, 18 Feb 2014 14:47:51 +0900, Shigeru Hanada wrote > I'm not sure that allowing ALTER TABLE against parent table affects > descendants even some of them are foreign table. Avoiding misunderstandings, my opinion on whether ALTER may applie to foreign chidlren is 'no'. regards, -- Kyotaro Horiguchi NTT Open Source Software Center
Hello, At Tue, 18 Feb 2014 19:24:50 +0900, "Etsuro Fujita" wrote > > From: Shigeru Hanada [mailto:shigeru.hanada@gmail.com] > > 2014-02-10 21:00 GMT+09:00 Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>: > > > (2014/02/07 21:31), Etsuro Fujita wrote: > > >> So, I've modified the patch so > > >> that we continue to disallow SET STORAGE on a foreign table *in the > > >> same manner as before*, but, as your patch does, allow it on an > > >> inheritance hierarchy that contains foreign tables, with the > > >> semantics that we quietly ignore the foreign tables and apply the > > >> operation to the plain tables, by modifying the ALTER TABLE simple > > recursion mechanism. > > >> Attached is the updated version of the patch. > > > > I'm not sure that allowing ALTER TABLE against parent table affects > > descendants even some of them are foreign table. I think the rule should > > be simple enough to understand for users, of course it should be also > > consistent and have backward compatibility. > > Yeah, the understandability is important. But I think the > flexibility is also important. In other words, I think it is a > bit too inflexible that we disallow e.g., SET STORAGE to be set > on an inheritance tree that contains foreign table(s) because > we disallow SET STORAGE to be set on foreign tables directly. What use case needs such a flexibility precedes the lost behavior predictivity of ALTER TABLE and/or code "maintenancibility"(more ordinally words must be...) ? I don't agree with the idea that ALTER TABLE implicitly affects foreign children for the reason in the upthread. Also turning on/off feature implemented as special syntax seems little hope. If you still want that, I suppose ALTER FOREIGN TABLE is usable chainging so as to walk through the inheritance tree and affects only foreign tables along the way. It can naturally affects foregin tables with no unanticipated behavor. Thoughts? > > If foreign table can be modified through inheritance tree, this kind of > > change can be done. > > > > 1) create foreign table as a child of a ordinary table > > 2) run ALTER TABLE parent, the foreign table is also changed > > 3) remove foreign table from the inheritance tree by ALTER TABLE child NO > > INHERIT parent > > 4) here we can't do same thing as 2), because it is not a child anymore. > > > > So IMO we should determine which ALTER TABLE features are allowed to foreign > > tables, and allow them regardless of the recursivity. > > What I think should be newly allowed to be set on foreign tables is > > * ADD table_constraint > * DROP CONSTRAINT As of 9.3 http://www.postgresql.org/docs/9.3/static/sql-alterforeigntable.html > Consistency with the foreign server is not checked when a > column is added or removed with ADD COLUMN or DROP COLUMN, a > NOT NULL constraint is added, or a column type is changed with > SET DATA TYPE. It is the user's responsibility to ensure that > the table definition matches the remote side. So I belive implicit and automatic application of any constraint on foreign childs are considerably danger. > * [NO] INHERIT parent_table Is this usable for inheritance foreign children? NO INHERIT removes all foreign children but INHERIT is nop. regards, -- Kyotaro Horiguchi NTT Open Source Software Center
Hi Horiguchi-san, 2014-02-18 19:29 GMT+09:00 Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>: > Could you guess any use cases in which we are happy with ALTER > TABLE's inheritance tree walking? IMHO, ALTER FOREIGN TABLE > always comes with some changes of the data source so implicitly > invoking of such commands should be defaultly turned off. Imagine a case that foreign data source have attributes (A, B, C, D) but foreign tables and their parent ware defined as (A, B, C). If user wants to use D as well, ALTER TABLE parent ADD COLUMN D type would be useful (rather necessary?) to keep consistency. Changing data type from compatible one (i.e., int to numeric, varchar(n) to text), adding CHECK/NOT NULL constraint would be also possible. -- Shigeru HANADA
(2014/02/19 12:12), Kyotaro HORIGUCHI wrote: > At Tue, 18 Feb 2014 19:24:50 +0900, "Etsuro Fujita" wrote >>> From: Shigeru Hanada [mailto:shigeru.hanada@gmail.com] >>> I'm not sure that allowing ALTER TABLE against parent table affects >>> descendants even some of them are foreign table. I think the rule should >>> be simple enough to understand for users, of course it should be also >>> consistent and have backward compatibility. >> >> Yeah, the understandability is important. But I think the >> flexibility is also important. In other words, I think it is a >> bit too inflexible that we disallow e.g., SET STORAGE to be set >> on an inheritance tree that contains foreign table(s) because >> we disallow SET STORAGE to be set on foreign tables directly. > > What use case needs such a flexibility precedes the lost behavior > predictivity of ALTER TABLE and/or code "maintenancibility"(more > ordinally words must be...) ? I don't agree with the idea that > ALTER TABLE implicitly affects foreign children for the reason in > the upthread. Also turning on/off feature implemented as special > syntax seems little hope. It is just my personal opinion, but I think it would be convenient for users to alter inheritance trees that contain foreign tables the same way as inheritance trees that don't contain any foreign tables, without making user conscious of the inheritance trees contains foreign tables or not. Imagine we have an inheritance tree that contains only plain tables and then add a foreign table as a child of the inheritance tree. Without the flexiblity, we would need to changethe way of altering the structure of the inheritance tree (e.g., ADD CONSTRAINT) to a totally different one, immediately when adding the foreign table. I don't think that would be easy to use. >> What I think should be newly allowed to be set on foreign tables is >> >> * ADD table_constraint >> * DROP CONSTRAINT > > As of 9.3 > > http://www.postgresql.org/docs/9.3/static/sql-alterforeigntable.html > >> Consistency with the foreign server is not checked when a >> column is added or removed with ADD COLUMN or DROP COLUMN, a >> NOT NULL constraint is added, or a column type is changed with >> SET DATA TYPE. It is the user's responsibility to ensure that >> the table definition matches the remote side. > > So I belive implicit and automatic application of any constraint > on foreign childs are considerably danger. We spent a lot of time discussing this issue, and the consensus is that it's users' fault if there are some tuples on the remote side violating a given constraint, as mentioned in the documentation. >> * [NO] INHERIT parent_table > > Is this usable for inheritance foreign children? NO INHERIT > removes all foreign children but INHERIT is nop. I didn't express clearly. Sorry for that. Let me explain about it. * ALTER FOREIGN TABLE target_table *INHERIT* parent_table: Add the target table as a new child of the parent table. * ALTER FOREIGN TABLE target_table *NO INHERIT* parent_table: Remove the target table from the list of children of the parent table. Thanks, Best regards, Etsuro Fujita
Hello, > It is just my personal opinion, but I think it would be convenient for > users to alter inheritance trees that contain foreign tables the same > way as inheritance trees that don't contain any foreign tables, > without making user conscious of the inheritance trees contains > foreign tables or not. Imagine we have an inheritance tree that > contains only plain tables and then add a foreign table as a child of > the inheritance tree. Without the flexiblity, we would need to change > the way of altering the structure of the inheritance tree (e.g., ADD > CONSTRAINT) to a totally different one, immediately when adding the > foreign table. I don't think that would be easy to use. I personally don't see significant advantages such a flexibility. Although my concerns here are only two points, unanticipated application and "maintenancibility". I gave a consideration on these issues again. Then, I think it could be enough by giving feedback to operators for the first issue. =# ALTER TABLE parent ADD CHECK (tempmin < tempmax), ALTER tempmin SET NOT NULL, ALTER tempmin SET DEFAULT 0; NOTICE: Child foregn table child01 is affected. NOTICE: Child foregn table child02 is affected NOTICE: Child foregn table child03 rejected 'alter tempmin set default' What do you think about this? It looks a bit too loud for me though... Then the second issue, however I don't have enough idea of how ALTER TABLE works, the complexity would be reduced if acceptance chek for alter "action"s would done on foreign server or data wrapper side, not on the core of ALTER TABLE. It would also be a help to output error messages like above. However, (NO)INHERIT looks a little different.. > > http://www.postgresql.org/docs/9.3/static/sql-alterforeigntable.html > > > >> Consistency with the foreign server is not checked when a > >> column is added or removed with ADD COLUMN or DROP COLUMN, a > >> NOT NULL constraint is added, or a column type is changed with > >> SET DATA TYPE. It is the user's responsibility to ensure that > >> the table definition matches the remote side. > > > > So I belive implicit and automatic application of any constraint > > on foreign childs are considerably danger. > > We spent a lot of time discussing this issue, and the consensus is > that it's users' fault if there are some tuples on the remote side > violating a given constraint, as mentioned in the documentation. I'm worried about not that but the changes and possible inconsistency would take place behind operators' backs. And this looks to cause such inconsistencies for me. > >> * [NO] INHERIT parent_table > > > > Is this usable for inheritance foreign children? NO INHERIT > > removes all foreign children but INHERIT is nop. > > I didn't express clearly. Sorry for that. Let me explain about it. > > * ALTER FOREIGN TABLE target_table *INHERIT* parent_table: Add the > * target table as a new child of the parent table. > * ALTER FOREIGN TABLE target_table *NO INHERIT* parent_table: Remove the > * target table from the list of children of the parent table. I got it, thank you. It alone seems no probmen but also doesn't seem to be a matter of 'ALTER TABLE'. Could you tell me how this is related to 'ALTER TABLE'? regards, -- Kyotaro Horiguchi NTT Open Source Software Center
Hi, At Wed, 19 Feb 2014 16:17:05 +0900, Shigeru Hanada wrote > 2014-02-18 19:29 GMT+09:00 Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>: > > Could you guess any use cases in which we are happy with ALTER > > TABLE's inheritance tree walking? IMHO, ALTER FOREIGN TABLE > > always comes with some changes of the data source so implicitly > > invoking of such commands should be defaultly turned off. > > Imagine a case that foreign data source have attributes (A, B, C, D) > but foreign tables and their parent ware defined as (A, B, C). If > user wants to use D as well, ALTER TABLE parent ADD COLUMN D type > would be useful (rather necessary?) to keep consistency. Hmm. I seems to me an issue of mis-configuration at first step. However, my anxiety is - as in my message just before - ALTER'ing foreign table definitions without any notice to operatos and irregular or random logic on check applicability(?) of ALTER actions. > Changing data type from compatible one (i.e., int to numeric, > varchar(n) to text), adding CHECK/NOT NULL constraint would be also > possible. I see, thank you. Changing data types are surely valuable but also seems difficult to check validity:( Anyway, I gave a second thought on this issue. Please have a look on that. regards, -- Kyotaro Horiguchi NTT Open Source Software Center
(2014/02/20 15:47), Kyotaro HORIGUCHI wrote: > Although my concerns here are only two points, > unanticipated application and "maintenancibility". I gave a > consideration on these issues again. Sorry, I misunderstood what you mean by "unanticipated application". > Then, I think it could be enough by giving feedback to operators > for the first issue. > > =# ALTER TABLE parent ADD CHECK (tempmin < tempmax), > ALTER tempmin SET NOT NULL, > ALTER tempmin SET DEFAULT 0; > NOTICE: Child foregn table child01 is affected. > NOTICE: Child foregn table child02 is affected > NOTICE: Child foregn table child03 rejected 'alter tempmin set default' > > What do you think about this? It looks a bit too loud for me > though... I think that's a good idea. What do others think? > Then the second issue, however I don't have enough idea of how > ALTER TABLE works, the complexity would be reduced if acceptance > chek for alter "action"s would done on foreign server or data > wrapper side, not on the core of ALTER TABLE. It would also be a > help to output error messages like above. I'm not sure it's worth having such an mechanism inside/outside the PG core. I might misunderstand your concern here, but is it the risk of constraint violation? If so, I'd like to vote for an idea of avoiding that violation by making the FDW in itself perform ExecQual() for each tuple retrived from the remote server at the query time. >> We spent a lot of time discussing this issue, and the consensus is >> that it's users' fault if there are some tuples on the remote side >> violating a given constraint, as mentioned in the documentation. > > I'm worried about not that but the changes and possible > inconsistency would take place behind operators' backs. And this > looks to cause such inconsistencies for me. That is what you mean by "unanticipated application", right? >>>> * [NO] INHERIT parent_table >>> >>> Is this usable for inheritance foreign children? NO INHERIT >>> removes all foreign children but INHERIT is nop. >> >> I didn't express clearly. Sorry for that. Let me explain about it. >> >> * ALTER FOREIGN TABLE target_table *INHERIT* parent_table: Add the >> * target table as a new child of the parent table. >> * ALTER FOREIGN TABLE target_table *NO INHERIT* parent_table: Remove the >> * target table from the list of children of the parent table. > > I got it, thank you. It alone seems no probmen but also doesn't > seem to be a matter of 'ALTER TABLE'. Could you tell me how this > is related to 'ALTER TABLE'? These are not related to ALTER TABLE. [NO] INHERIT parent_table (and ADD/DROP CONSTRAINT) are what I think should be newly allowed to apply to foreign tables directly. Thanks, Best regards, Etsuro Fujita
(2014/02/20 19:55), Etsuro Fujita wrote: > (2014/02/20 15:47), Kyotaro HORIGUCHI wrote: >> Although my concerns here are only two points, >> unanticipated application and "maintenancibility". I gave a >> consideration on these issues again. > > Sorry, I misunderstood what you mean by "unanticipated application". > >> Then, I think it could be enough by giving feedback to operators >> for the first issue. >> >> =# ALTER TABLE parent ADD CHECK (tempmin < tempmax), >> ALTER tempmin SET NOT NULL, >> ALTER tempmin SET DEFAULT 0; >> NOTICE: Child foregn table child01 is affected. >> NOTICE: Child foregn table child02 is affected >> NOTICE: Child foregn table child03 rejected 'alter tempmin set default' >> >> What do you think about this? It looks a bit too loud for me >> though... > > I think that's a good idea. I just thought those messages would be shown for the user to readily notice the changes of the structures of child tables that are foreign, done by the recursive altering operation. But I overlooked the third line: NOTICE: Child foregn table child03 rejected 'alter tempmin set default' What does "rejected" in this message mean? Thanks, Best regards, Etsuro Fujita
Hi, > >> NOTICE: Child foregn table child01 is affected. > >> NOTICE: Child foregn table child02 is affected > >> NOTICE: Child foregn table child03 rejected 'alter tempmin set > >> default' > >> > >> What do you think about this? It looks a bit too loud for me > >> though... > > > > I think that's a good idea. > > I just thought those messages would be shown for the user to readily > notice the changes of the structures of child tables that are foreign, > done by the recursive altering operation. But I overlooked the third > line: > > NOTICE: Child foregn table child03 rejected 'alter tempmin set > default' > > What does "rejected" in this message mean? It says that child03 had no ability to perform the requested action, in this case setting a default value. It might be better to reject ALTER on the parent as a whole when any children doesn't accept any action. regards, -- Kyotaro Horiguchi NTT Open Source Software Center
Hello, > > Then the second issue, however I don't have enough idea of how > > ALTER TABLE works, the complexity would be reduced if acceptance > > chek for alter "action"s would done on foreign server or data > > wrapper side, not on the core of ALTER TABLE. It would also be a > > help to output error messages like above. > > I'm not sure it's worth having such an mechanism inside/outside the PG > core. I might misunderstand your concern here, but is it the risk of > constraint violation? A bit different:) It's the problem of how and who decides whether each ALTER action given can be performed on each child. Set of available actions vary according to the nature of each foreign table/server/driver and also according to their functional evolutions made in future, largely independent of the core, maybe. The core oughtn't to.. couldn't maintain such a function judging availability of every possible combination of action and foreign table. > If so, I'd like to vote for an idea of avoiding > that violation by making the FDW in itself perform ExecQual() for each > tuple retrived from the remote server at the query time. In my humble opition, it is not so serious problem that functionally acceptable actions can cause any kind of inconsistency by the nature of fdw so far. It is well documented and operators should be aware of such inconsistencies after being informed of what they did, by the NOTICE messages. I think. > >> * ALTER FOREIGN TABLE target_table *INHERIT* parent_table: Add the > >> * target table as a new child of the parent table. > >> * ALTER FOREIGN TABLE target_table *NO INHERIT* parent_table: Remove the > >> * target table from the list of children of the parent table. > > > > I got it, thank you. It alone seems no probmen but also doesn't > > seem to be a matter of 'ALTER TABLE'. Could you tell me how this > > is related to 'ALTER TABLE'? > > These are not related to ALTER TABLE. [NO] INHERIT parent_table (and > ADD/DROP CONSTRAINT) are what I think should be newly allowed to apply > to foreign tables directly. Thank you, I understood that. I thought it already existed. regards, -- Kyotaro Horiguchi NTT Open Source Software Center
(2014/02/21 15:23), Kyotaro HORIGUCHI wrote: >>>> NOTICE: Child foregn table child01 is affected. >>>> NOTICE: Child foregn table child02 is affected >>>> NOTICE: Child foregn table child03 rejected 'alter tempmin set >>>> default' >>>> >>>> What do you think about this? It looks a bit too loud for me >>>> though... >>> >>> I think that's a good idea. >> >> I just thought those messages would be shown for the user to readily >> notice the changes of the structures of child tables that are foreign, >> done by the recursive altering operation. But I overlooked the third >> line: >> >> NOTICE: Child foregn table child03 rejected 'alter tempmin set >> default' >> >> What does "rejected" in this message mean? > > It says that child03 had no ability to perform the requested > action, in this case setting a default value. It might be better > to reject ALTER on the parent as a whole when any children > doesn't accept any action. Now understood, thougn I'm not sure it's worth implementing such a checking mechanism in the recursive altering operation... Thanks, Best regards, Etsuro Fujita
Hello, At Fri, 21 Feb 2014 16:33:32 +0900, Etsuro Fujita wrote > (2014/02/21 15:23), Kyotaro HORIGUCHI wrote: > >>>> NOTICE: Child foregn table child01 is affected. > >>>> NOTICE: Child foregn table child02 is affected > >>>> NOTICE: Child foregn table child03 rejected 'alter tempmin set > >>>> default' > > It says that child03 had no ability to perform the requested > > action, in this case setting a default value. It might be better > > to reject ALTER on the parent as a whole when any children > > doesn't accept any action. > > Now understood, thougn I'm not sure it's worth implementing such a > checking mechanism in the recursive altering operation... Did you mean foreign tables can sometimes silently ignore ALTER actions which it can't perform? It will cause inconsistency which operators didn't anticipate. This example uses "add column" for perspicuitly but all types of action could result like this. ============== =# ALTER TABLE parent ADD COLUMN x integer; ALTER TABLE =# \d parent Table "public.parent"Column | Type | Modifiers --------+---------+--------------------a | integer | b | integer | x | integer | Number of child tables: 2 (Use \d+ to list them.) =# \d child1 Foreign table "public.child1" Column | Type | Modifiers | FDW Options ----------+---------+-----------+-------------a | integer | b | integer | =# (Op: Ouch!) ============== I think this should result as, ============== =# ALTER TABLE parent ADD COLUMN x integer; ERROR: Foreign child table child1 could not perform some of the actions. =# ============== regards, -- Kyotaro Horiguchi NTT Open Source Software Center
In addition to an issue pointed out recently by Horiguchi-san, I've found there is another issue we have to discuss. That is, we can't build any parameterized Append paths for an inheritance tree that contains at least one foreign table, in set_append_rel_pathlist(). This is because the patch doesn't take into consideration "reparameterization" for paths for a foreign table, which attempts to modify these paths to have greater parameterization so that each child path in the inheritance tree can have the exact same parameterization. To do so, I think we would need to add code for the foreign-table case to reparameterize_path(). And I think we should introduce a new FDW routine, say ReparameterizeForeignPath() because the processing would be performed best by the FDW itself. Comments are welcome! Thanks, Best regards, Etsuro Fujita
Hello, I tried minimal implementation to do that.
At Tue, 25 Feb 2014 19:45:56 +0900, Etsuro Fujita wrote
> In addition to an issue pointed out recently by Horiguchi-san, I've
> found there is another issue we have to discuss. That is, we can't
> build any parameterized Append paths for an inheritance tree that
> contains at least one foreign table, in set_append_rel_pathlist(). This
> is because the patch doesn't take into consideration
> "reparameterization" for paths for a foreign table, which attempts to
> modify these paths to have greater parameterization so that each child
> path in the inheritance tree can have the exact same parameterization.
> To do so, I think we would need to add code for the foreign-table case
> to reparameterize_path(). And I think we should introduce a new FDW
> routine, say ReparameterizeForeignPath() because the processing would be
> performed best by the FDW itself.
>
> Comments are welcome!
I think the problem is foreign childs in inheritance tables
prevents all menber in the inheritance relation from using
parameterized paths, correct?
|=# explain select * from p join (select uname from c1 limit 1) s on s.uname = p.uname;
| QUERY PLAN
|-------------------------------------------------------------------------------
| Hash Join (cost=0.04..244.10 rows=50 width=58)
| Hash Cond: (p.uname = c1_1.uname)
| -> Append (cost=0.00..206.01 rows=10012 width=50)
| -> Seq Scan on p (cost=0.00..0.00 rows=1 width=168)
| -> Seq Scan on c1 (cost=0.00..204.01 rows=10001 width=50)
| -> Foreign Scan on c2 (cost=0.00..2.00 rows=10 width=168)
| Foreign File: /etc/passwd
| Foreign File Size: 1851
| -> Hash (cost=0.03..0.03 rows=1 width=8)
| -> Limit (cost=0.00..0.02 rows=1 width=8)
| -> Seq Scan on c1 c1_1 (cost=0.00..204.01 rows=10001 width=8)
| Planning time: 1.095 ms
Hmm. I tried minimal implementation to do that. This omits cost
recalculation but seems to work as expected. This seems enough if
cost recalc is trivial here.
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index b79af7a..18ced04 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -2062,6 +2062,16 @@ reparameterize_path(PlannerInfo *root, Path *path, case T_SubqueryScan: return
create_subqueryscan_path(root,rel, path->pathkeys, required_outer);
+ case T_ForeignScan:
+ {
+ ForeignPath *fpath = (ForeignPath*) path;
+ ForeignPath *newpath = makeNode(ForeignPath);
+ memcpy(newpath, fpath, sizeof(ForeignPath));
+ newpath->path.param_info =
+ get_baserel_parampathinfo(root, rel, required_outer);
+ /* cost recalc omitted */
+ return (Path *)newpath;
+ } default: break; }
....
|=# explain select * from p join (select uname from c1 limit 1) s on s.uname = p.uname;
| QUERY PLAN
|----------------------------------------------------------------------------
| Nested Loop (cost=0.00..10.46 rows=50 width=58)
| -> Limit (cost=0.00..0.02 rows=1 width=8)
| -> Seq Scan on c1 c1_1 (cost=0.00..204.01 rows=10001 width=8)
| -> Append (cost=0.00..10.30 rows=12 width=158)
| -> Seq Scan on p (cost=0.00..0.00 rows=1 width=168)
| Filter: (c1_1.uname = uname)
| -> Index Scan using i_c1 on c1 (cost=0.29..8.30 rows=1 width=50)
| Index Cond: (uname = c1_1.uname)
| -> Foreign Scan on c2 (cost=0.00..2.00 rows=10 width=168)
| Filter: (c1_1.uname = uname)
| Foreign File: /etc/passwd
| Foreign File Size: 1851
| Planning time: 2.044 ms
regards,
--
Kyotaro Horiguchi
NTT Open Source Software Center
Hello. As a minimal implementation, I made an attempt that emit
NOTICE message when alter table affects foreign tables. It looks
like following,
| =# alter table passwd add column added int, add column added2 int;
| NOTICE: This command affects foreign relation "cf1"
| NOTICE: This command affects foreign relation "cf1"
| ALTER TABLE
| =# select * from passwd;
| ERROR: missing data for column "added"
| CONTEXT: COPY cf1, line 1: "root:x:0:0:root:/root:/bin/bash"
| =#
This seems far better than silently performing the command,
except for the duplicated message:( New bitmap might required to
avoid the duplication..
I made the changes above and below as an attempt in the attached
patch foreign_inherit-v4.patch
> I think the problem is foreign childs in inheritance tables
> prevents all menber in the inheritance relation from using
> parameterized paths, correct?
...
> Hmm. I tried minimal implementation to do that. This omits cost
> recalculation but seems to work as expected. This seems enough if
> cost recalc is trivial here.
Any comments?
regards,
--
Kyotaro Horiguchi
NTT Open Source Software Center
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 8ace8bd..b4e53c1 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -258,6 +258,17 @@ CREATE TABLE products ( even if the value came from the default value definition. </para>
+ <note>
+ <para>
+ Note that constraints can be defined on foreign tables too, but such
+ constraints are not enforced on insert or update. Those constraints are
+ "assertive", and work only to tell planner that some kind of optimization
+ such as constraint exclusion can be considerd. This seems useless, but
+ allows us to use foriegn table as child table (see
+ <xref linkend="ddl-inherit">) to off-load to multiple servers.
+ </para>
+ </note>
+ <sect2 id="ddl-constraints-check-constraints"> <title>Check Constraints</title>
@@ -2017,8 +2028,8 @@ CREATE TABLE capitals ( </para> <para>
- In <productname>PostgreSQL</productname>, a table can inherit from
- zero or more other tables, and a query can reference either all
+ In <productname>PostgreSQL</productname>, a table or foreign table can
+ inherit from zero or more other tables, and a query can reference either all rows of a table or all rows of a
tableplus all of its descendant tables. The latter behavior is the default. For example, the following query finds
thenames of all cities,
diff --git a/doc/src/sgml/ref/alter_foreign_table.sgml b/doc/src/sgml/ref/alter_foreign_table.sgml
index 4d8cfc5..f7a382e 100644
--- a/doc/src/sgml/ref/alter_foreign_table.sgml
+++ b/doc/src/sgml/ref/alter_foreign_table.sgml
@@ -42,6 +42,8 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab ALTER [ COLUMN ]
<replaceableclass="PARAMETER">column_name</replaceable> SET ( <replaceable
class="PARAMETER">attribute_option</replaceable>= <replaceable class="PARAMETER">value</replaceable> [, ... ] )
ALTER[ COLUMN ] <replaceable class="PARAMETER">column_name</replaceable> RESET ( <replaceable
class="PARAMETER">attribute_option</replaceable>[, ... ] ) ALTER [ COLUMN ] <replaceable
class="PARAMETER">column_name</replaceable>OPTIONS ( [ ADD | SET | DROP ] <replaceable
class="PARAMETER">option</replaceable>['<replaceable class="PARAMETER">value</replaceable>'] [, ... ])
+ INHERIT <replaceable class="PARAMETER">parent_table</replaceable>
+ NO INHERIT <replaceable class="PARAMETER">parent_table</replaceable> OWNER TO <replaceable
class="PARAMETER">new_owner</replaceable> OPTIONS ( [ ADD | SET | DROP ] <replaceable
class="PARAMETER">option</replaceable>['<replaceable class="PARAMETER">value</replaceable>'] [, ... ])</synopsis>
@@ -178,6 +180,26 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab </varlistentry>
<varlistentry>
+ <term><literal>INHERIT <replaceable class="PARAMETER">parent_table</replaceable></literal></term>
+ <listitem>
+ <para>
+ This form adds the target foreign table as a new child of the specified
+ parent table. The parent table must be a plain table.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>NO INHERIT <replaceable class="PARAMETER">parent_table</replaceable></literal></term>
+ <listitem>
+ <para>
+ This form removes the target foreign table from the list of children of
+ the specified parent table.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry> <term><literal>OPTIONS ( [ ADD | SET | DROP ] <replaceable class="PARAMETER">option</replaceable>
['<replaceableclass="PARAMETER">value</replaceable>'] [, ... ] )</literal></term> <listitem> <para>
@@ -306,6 +328,16 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab </para>
</listitem> </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="PARAMETER">parent_name</replaceable></term>
+ <listitem>
+ <para>
+ A parent table to associate or de-associate with this foreign table.
+ The parent table must be a plain table.
+ </para>
+ </listitem>
+ </varlistentry> </variablelist> </refsect1>
diff --git a/doc/src/sgml/ref/create_foreign_table.sgml b/doc/src/sgml/ref/create_foreign_table.sgml
index 06a7087..cc11dee 100644
--- a/doc/src/sgml/ref/create_foreign_table.sgml
+++ b/doc/src/sgml/ref/create_foreign_table.sgml
@@ -22,6 +22,7 @@ CREATE FOREIGN TABLE [ IF NOT EXISTS ] <replaceable class="PARAMETER">table_name <replaceable
class="PARAMETER">column_name</replaceable><replaceable class="PARAMETER">data_type</replaceable> [ OPTIONS (
<replaceableclass="PARAMETER">option</replaceable> '<replaceable class="PARAMETER">value</replaceable>' [, ... ] ) ] [
COLLATE<replaceable>collation</replaceable> ] [ <replaceable class="PARAMETER">column_constraint</replaceable> [ ... ]
] [, ... ]] )
+[ INHERITS ( <replaceable>parent_table</replaceable> [, ... ] ) ] SERVER <replaceable
class="parameter">server_name</replaceable>[OPTIONS ( <replaceable class="PARAMETER">option</replaceable> '<replaceable
class="PARAMETER">value</replaceable>'[, ... ] ) ]
@@ -159,6 +160,18 @@ CREATE FOREIGN TABLE [ IF NOT EXISTS ] <replaceable class="PARAMETER">table_name </varlistentry>
<varlistentry>
+ <term><replaceable class="PARAMETER">parent_table</replaceable></term>
+ <listitem>
+ <para>
+ The name of an existing table from which the new foreign table
+ automatically inherits all columns. The specified parent table
+ must be a plain table. See <xref linkend="ddl-inherit"> for the
+ details of table inheritance.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry> <term><replaceable class="PARAMETER">server_name</replaceable></term> <listitem> <para>
diff --git a/src/backend/catalog/pg_inherits.c b/src/backend/catalog/pg_inherits.c
index f263b42..b4a084c 100644
--- a/src/backend/catalog/pg_inherits.c
+++ b/src/backend/catalog/pg_inherits.c
@@ -256,6 +256,48 @@ has_subclass(Oid relationId)/*
+ * Check wether the inheritance tree contains foreign table(s).
+ */
+bool
+contains_foreign(Oid parentrelId, LOCKMODE lockmode)
+{
+ bool result = false;
+ List *tableOIDs;
+ ListCell *lc;
+
+ /* Find all members of the inheritance tree */
+ tableOIDs = find_all_inheritors(parentrelId, lockmode, NULL);
+
+ /* There are no children */
+ if (list_length(tableOIDs) < 2)
+ return result;
+
+ foreach(lc, tableOIDs)
+ {
+ Oid childOID = lfirst_oid(lc);
+ Relation childrel;
+
+ /* Parent should not be foreign */
+ if (childOID == parentrelId)
+ continue;
+
+ /* We already got the needed lock */
+ childrel = heap_open(childOID, NoLock);
+
+ if (childrel->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
+ {
+ /* Found it */
+ result = true;
+ }
+
+ heap_close(childrel, lockmode);
+ }
+
+ return result;
+}
+
+
+/* * Given two type OIDs, determine whether the first is a complex type * (class type) that inherits from the second.
*/
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index a04adea..bb7bdf4 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -115,7 +115,8 @@ static Datum ind_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull); * analyze_rel() --
analyzeone relation */void
-analyze_rel(Oid relid, VacuumStmt *vacstmt, BufferAccessStrategy bstrategy)
+analyze_rel(Oid relid, VacuumStmt *vacstmt, VacuumMode vacmode,
+ BufferAccessStrategy bstrategy){ Relation onerel; int elevel;
@@ -270,7 +271,9 @@ analyze_rel(Oid relid, VacuumStmt *vacstmt, BufferAccessStrategy bstrategy) /* * If there
arechild tables, do recursive ANALYZE. */
- if (onerel->rd_rel->relhassubclass)
+ if (onerel->rd_rel->relhassubclass &&
+ (vacmode == VAC_MODE_SINGLE ||
+ !contains_foreign(RelationGetRelid(onerel), AccessShareLock))) do_analyze_rel(onerel, vacstmt,
acquirefunc,relpages, true, elevel); /*
@@ -1452,12 +1455,15 @@ acquire_inherited_sample_rows(Relation onerel, int elevel,{ List *tableOIDs;
Relation *rels;
+ AcquireSampleRowsFunc *acquirefunc; double *relblocks; double totalblocks;
+ Relation saved_rel; int numrows, nrels, i; ListCell *lc;
+ bool isAnalyzable = true; /* * Find all members of inheritance set. We only need AccessShareLock
on
@@ -1486,6 +1492,8 @@ acquire_inherited_sample_rows(Relation onerel, int elevel, * BlockNumber, so we use double
arithmetic. */ rels = (Relation *) palloc(list_length(tableOIDs) * sizeof(Relation));
+ acquirefunc = (AcquireSampleRowsFunc *) palloc(list_length(tableOIDs)
+ * sizeof(AcquireSampleRowsFunc)); relblocks = (double *)
palloc(list_length(tableOIDs)* sizeof(double)); totalblocks = 0; nrels = 0;
@@ -1507,12 +1515,59 @@ acquire_inherited_sample_rows(Relation onerel, int elevel, } rels[nrels] =
childrel;
- relblocks[nrels] = (double) RelationGetNumberOfBlocks(childrel);
+
+ if (childrel->rd_rel->relkind != RELKIND_FOREIGN_TABLE)
+ {
+ acquirefunc[nrels] = acquire_sample_rows;
+ relblocks[nrels] = (double) RelationGetNumberOfBlocks(childrel);
+ }
+ else
+ {
+ /*
+ * For a foreign table, call the FDW's hook function to see whether
+ * it supports analysis.
+ */
+ FdwRoutine *fdwroutine;
+ BlockNumber relpages = 0;
+ bool ok = false;
+
+ fdwroutine = GetFdwRoutineForRelation(childrel, false);
+ if (fdwroutine->AnalyzeForeignTable != NULL)
+ ok = fdwroutine->AnalyzeForeignTable(childrel,
+ &acquirefunc[nrels],
+ &relpages);
+ if (!ok)
+ {
+ isAnalyzable = false;
+ break;
+ }
+
+ relblocks[nrels] = (double) relpages;
+ }
+ totalblocks += relblocks[nrels]; nrels++; } /*
+ * If there is at least one foreign table that cannot be analyzed, give up.
+ */
+ if (!isAnalyzable)
+ {
+ ereport(WARNING,
+ (errmsg("skipping \"%s\" inheritance tree --- cannot analyze foreign table \"%s\"",
+ RelationGetRelationName(onerel),
+ RelationGetRelationName(rels[nrels]))));
+ for (i = 0; i < nrels; i++)
+ {
+ Relation childrel = rels[i];
+
+ heap_close(childrel, NoLock);
+ }
+ return 0;
+ }
+
+ /* * Now sample rows from each relation, proportionally to its fraction of * the total block count. (This
mightbe less than desirable if the child * rels have radically different free-space percentages, but it's not
@@ -1525,6 +1580,7 @@ acquire_inherited_sample_rows(Relation onerel, int elevel, { Relation childrel =
rels[i]; double childblocks = relblocks[i];
+ AcquireSampleRowsFunc childacquirefunc = acquirefunc[i]; if (childblocks > 0) {
@@ -1540,12 +1596,12 @@ acquire_inherited_sample_rows(Relation onerel, int elevel, tdrows;
/* Fetch a random sample of the child's rows */
- childrows = acquire_sample_rows(childrel,
- elevel,
- rows + numrows,
- childtargrows,
- &trows,
- &tdrows);
+ childrows = childacquirefunc(childrel,
+ elevel,
+ rows + numrows,
+ childtargrows,
+ &trows,
+ &tdrows); /* We may need to convert from child's rowtype
toparent's */ if (childrows > 0 &&
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 25f01e5..61ea18b 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -311,7 +311,8 @@ static AlteredTableInfo *ATGetQueueEntry(List **wqueue, Relation rel);static void
ATSimplePermissions(Relationrel, int allowed_targets);static void ATWrongRelkindError(Relation rel, int
allowed_targets);staticvoid ATSimpleRecursion(List **wqueue, Relation rel,
- AlterTableCmd *cmd, bool recurse, LOCKMODE lockmode);
+ AlterTableCmd *cmd, bool recurse,
+ bool include_foreign, LOCKMODE lockmode);static void ATTypedTableRecursion(List **wqueue, Relation
rel,AlterTableCmd *cmd, LOCKMODE lockmode);static List *find_typed_table_dependencies(Oid typeOid,
constchar *typeName,
@@ -467,10 +468,6 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId) ereport(ERROR,
(errcode(ERRCODE_INVALID_TABLE_DEFINITION), errmsg("ON COMMIT can only be used on temporary tables")));
- if (stmt->constraints != NIL && relkind == RELKIND_FOREIGN_TABLE)
- ereport(ERROR,
- (errcode(ERRCODE_WRONG_OBJECT_TYPE),
- errmsg("constraints are not supported on foreign tables"))); /* * Look up the namespace in
whichwe are supposed to create the relation,
@@ -3019,24 +3016,28 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, * rules. */
ATSimplePermissions(rel, ATT_TABLE | ATT_VIEW | ATT_FOREIGN_TABLE);
- ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+ /* Recurses to child tables that are foreign, too */
+ ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode); /* No command-specific prep
needed*/ pass = cmd->def ? AT_PASS_ADD_CONSTR : AT_PASS_DROP; break; case AT_DropNotNull:
/* ALTER COLUMN DROP NOT NULL */ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
- ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+ /* Recurses to child tables that are foreign, too */
+ ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode); /* No command-specific prep
needed*/ pass = AT_PASS_DROP; break; case AT_SetNotNull: /* ALTER COLUMN SET NOT
NULL*/ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
- ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+ /* Recurses to child tables that are foreign, too */
+ ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode); /* No command-specific prep
needed*/ pass = AT_PASS_ADD_CONSTR; break; case AT_SetStatistics: /* ALTER COLUMN SET
STATISTICS*/
- ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+ /* Recurses to child tables that are foreign, too */
+ ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode); /* Performs own permission checks
*/ ATPrepSetStatistics(rel, cmd->name, cmd->def, lockmode); pass = AT_PASS_MISC;
@@ -3049,7 +3050,8 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, break; case
AT_SetStorage: /* ALTER COLUMN SET STORAGE */ ATSimplePermissions(rel, ATT_TABLE | ATT_MATVIEW);
- ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+ /* Don't recurse to child tables that are foreign */
+ ATSimpleRecursion(wqueue, rel, cmd, recurse, false, lockmode); /* No command-specific prep
needed*/ pass = AT_PASS_MISC; break;
@@ -3067,7 +3069,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, pass = AT_PASS_ADD_INDEX;
break; case AT_AddConstraint: /* ADD CONSTRAINT */
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE); /* Recursion occurs during execution
phase*/ /* No command-specific prep needed except saving recurse flag */ if (recurse)
@@ -3081,7 +3083,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, pass = AT_PASS_ADD_CONSTR;
break; case AT_DropConstraint: /* DROP CONSTRAINT */
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE); /* Recursion occurs during execution
phase*/ /* No command-specific prep needed except saving recurse flag */ if (recurse)
@@ -3149,13 +3151,19 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, pass = AT_PASS_MISC;
break; case AT_AddInherit: /* INHERIT */
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE); /* This command never recurses */
ATPrepAddInherit(rel); pass = AT_PASS_MISC; break;
+ case AT_DropInherit: /* NO INHERIT */
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
+ /* This command never recurses */
+ /* No command-specific prep needed */
+ pass = AT_PASS_MISC;
+ break; case AT_AlterConstraint: /* ALTER CONSTRAINT */
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE); pass = AT_PASS_MISC; break;
case AT_ValidateConstraint: /* VALIDATE CONSTRAINT */
@@ -3184,7 +3192,6 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, case AT_EnableAlwaysRule:
case AT_EnableReplicaRule: case AT_DisableRule:
- case AT_DropInherit: /* NO INHERIT */ case AT_AddOf: /* OF */ case AT_DropOf: /*
NOTOF */ ATSimplePermissions(rel, ATT_TABLE);
@@ -4037,8 +4044,12 @@ ATSimplePermissions(Relation rel, int allowed_targets) case RELKIND_COMPOSITE_TYPE:
actual_target = ATT_COMPOSITE_TYPE; break;
+ case RELKIND_FOREIGN_TABLE: actual_target = ATT_FOREIGN_TABLE;
+ ereport(NOTICE,
+ (errmsg("This command affects foreign relation \"%s\"",
+ RelationGetRelationName(rel)))); break; default:
actual_target= 0;
@@ -4125,7 +4136,8 @@ ATWrongRelkindError(Relation rel, int allowed_targets) */static voidATSimpleRecursion(List
**wqueue,Relation rel,
- AlterTableCmd *cmd, bool recurse, LOCKMODE lockmode)
+ AlterTableCmd *cmd, bool recurse,
+ bool include_foreign, LOCKMODE lockmode){ /* * Propagate to children if desired. Non-table
relationsnever have
@@ -4153,8 +4165,12 @@ ATSimpleRecursion(List **wqueue, Relation rel, continue; /*
find_all_inheritorsalready got lock */ childrel = relation_open(childrelid, NoLock);
- CheckTableNotInUse(childrel, "ALTER TABLE");
- ATPrepCmd(wqueue, childrel, cmd, false, true, lockmode);
+ if (childrel->rd_rel->relkind != RELKIND_FOREIGN_TABLE
+ || include_foreign)
+ {
+ CheckTableNotInUse(childrel, "ALTER TABLE");
+ ATPrepCmd(wqueue, childrel, cmd, false, true, lockmode);
+ } relation_close(childrel, NoLock); } }
@@ -4444,7 +4460,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, /* At top level,
permissioncheck was done in ATPrepCmd, else do it */ if (recursing)
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE); attrdesc = heap_open(AttributeRelationId,
RowExclusiveLock);
@@ -4859,7 +4875,19 @@ ATPrepAddOids(List **wqueue, Relation rel, bool recurse, AlterTableCmd *cmd, LOC
ATPrepAddColumn(wqueue,rel, recurse, false, cmd, lockmode); if (recurse)
+ {
+ /*
+ * Don't allow to add an OID column to inheritance tree that contains
+ * foreign table(s)
+ */
+ if (contains_foreign(RelationGetRelid(rel), AccessShareLock))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot add OID column to inheritance tree \"%s\" because it contains foreign table(s)",
+ RelationGetRelationName(rel))));
+ cmd->subtype = AT_AddOidsRecurse;
+ }}/*
@@ -5340,7 +5368,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName, /* At top level, permission
checkwas done in ATPrepCmd, else do it */ if (recursing)
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE); /* * get the number of the attribute
@@ -5732,7 +5760,14 @@ ATAddCheckConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel, /* At top level,
permissioncheck was done in ATPrepCmd, else do it */ if (recursing)
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
+
+ /* Don't allow ADD constraint NOT VALID on foreign tables */
+ if (tab->relkind == RELKIND_FOREIGN_TABLE &&
+ constr->skip_validation && !recursing)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("NOT VALID is not supported on foreign tables"))); /* * Call AddRelationNewConstraints
todo the work, making sure it works on
@@ -5743,9 +5778,17 @@ ATAddCheckConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel, * omitted from the
returnedlist, which is what we want: we do not need * to do any validation work. That can only happen at child
tables, * though, since we disallow merging at the top level.
- */
+ *
+ * When propagating a NOT VALID option to children that are foreign tables,
+ * we quietly ignore the option. Note that this is safe because foreign
+ * tables don't have any children.
+ */
+ constr = copyObject(constr);
+ if (tab->relkind == RELKIND_FOREIGN_TABLE &&
+ constr->skip_validation && recursing)
+ constr->skip_validation = false; newcons = AddRelationNewConstraints(rel, NIL,
- list_make1(copyObject(constr)),
+ list_make1(constr), recursing,
/*allow_merge */ !recursing, /* is_local */
is_readd); /* is_internal */
@@ -7225,7 +7268,7 @@ ATExecDropConstraint(Relation rel, const char *constrName, /* At top level, permission check
wasdone in ATPrepCmd, else do it */ if (recursing)
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE); conrel = heap_open(ConstraintRelationId,
RowExclusiveLock);
@@ -7560,7 +7603,10 @@ ATPrepAlterColumnType(List **wqueue, * alter would put them out of step. */ if
(recurse)
- ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+ {
+ /* Recurses to child tables that are foreign, too */
+ ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode);
+ } else if (!recursing && find_inheritance_children(RelationGetRelid(rel), NoLock) != NIL)
ereport(ERROR,
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index ded1841..35b5dd2 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -98,6 +98,7 @@ vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast, BufferAccessStrategy bstrategy, bool
for_wraparound,bool isTopLevel){ const char *stmttype;
+ VacuumMode vacmode; volatile bool in_outer_xact, use_own_xacts; List *relations;
@@ -146,6 +147,20 @@ vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast,
ALLOCSET_DEFAULT_MAXSIZE); /*
+ * Identify vacuum mode. If relid is not InvalidOid, the caller should be
+ * an autovacuum worker. See the above comments.
+ */
+ if (relid != InvalidOid)
+ vacmode = VAC_MODE_AUTOVACUUM;
+ else
+ {
+ if (!vacstmt->relation)
+ vacmode = VAC_MODE_ALL;
+ else
+ vacmode = VAC_MODE_SINGLE;
+ }
+
+ /* * If caller didn't give us a buffer strategy object, make one in the * cross-transaction memory
context. */
@@ -248,7 +263,7 @@ vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast,
PushActiveSnapshot(GetTransactionSnapshot()); }
- analyze_rel(relid, vacstmt, vac_strategy);
+ analyze_rel(relid, vacstmt, vacmode, vac_strategy); if (use_own_xacts)
{
diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c
index 52dcc72..238bba2 100644
--- a/src/backend/optimizer/prep/prepunion.c
+++ b/src/backend/optimizer/prep/prepunion.c
@@ -1337,11 +1337,12 @@ expand_inherited_rtentry(PlannerInfo *root, RangeTblEntry *rte, Index rti) /* *
Buildan RTE for the child, and attach to query's rangetable list. * We copy most fields of the parent's RTE,
butreplace relation OID,
- * and set inh = false. Also, set requiredPerms to zero since all
- * required permissions checks are done on the original RTE.
+ * relkind and set inh = false. Also, set requiredPerms to zero since
+ * all required permissions checks are done on the original RTE. */ childrte = copyObject(rte);
childrte->relid = childOID;
+ childrte->relkind = newrelation->rd_rel->relkind; childrte->inh = false; childrte->requiredPerms
=0; parse->rtable = lappend(parse->rtable, childrte);
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index b79af7a..18ced04 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -2062,6 +2062,16 @@ reparameterize_path(PlannerInfo *root, Path *path, case T_SubqueryScan: return
create_subqueryscan_path(root,rel, path->pathkeys, required_outer);
+ case T_ForeignScan:
+ {
+ ForeignPath *fpath = (ForeignPath*) path;
+ ForeignPath *newpath = makeNode(ForeignPath);
+ memcpy(newpath, fpath, sizeof(ForeignPath));
+ newpath->path.param_info =
+ get_baserel_parampathinfo(root, rel, required_outer);
+ /* cost recalc omitted */
+ return (Path *)newpath;
+ } default: break; }
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index e3060a4..b5e47f4 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -4209,32 +4209,32 @@ AlterForeignServerStmt: ALTER SERVER name foreign_server_version
alter_generic_oCreateForeignTableStmt: CREATE FOREIGN TABLE qualified_name '(' OptTableElementList
')'
- SERVER name create_generic_options
+ OptInherit SERVER name create_generic_options { CreateForeignTableStmt
*n= makeNode(CreateForeignTableStmt); $4->relpersistence = RELPERSISTENCE_PERMANENT;
n->base.relation = $4; n->base.tableElts = $6;
- n->base.inhRelations = NIL;
+ n->base.inhRelations = $8; n->base.if_not_exists = false; /*
FDW-specificdata */
- n->servername = $9;
- n->options = $10;
+ n->servername = $10;
+ n->options = $11; $$ = (Node *) n; } | CREATE FOREIGN
TABLEIF_P NOT EXISTS qualified_name '(' OptTableElementList ')'
- SERVER name create_generic_options
+ OptInherit SERVER name create_generic_options { CreateForeignTableStmt
*n= makeNode(CreateForeignTableStmt); $7->relpersistence = RELPERSISTENCE_PERMANENT;
n->base.relation = $7; n->base.tableElts = $9;
- n->base.inhRelations = NIL;
+ n->base.inhRelations = $11; n->base.if_not_exists = true; /*
FDW-specificdata */
- n->servername = $12;
- n->options = $13;
+ n->servername = $13;
+ n->options = $14; $$ = (Node *) n; } ;
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index 1e071d7..955f27c 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -515,12 +515,6 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column) break;
case CONSTR_CHECK:
- if (cxt->isforeign)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("constraints are not supported on foreign tables"),
- parser_errposition(cxt->pstate,
- constraint->location))); cxt->ckconstraints =
lappend(cxt->ckconstraints,constraint); break;
@@ -529,7 +523,7 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column) if
(cxt->isforeign) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("constraints are not supported on foreign tables"),
+ errmsg("primary key or unique constraints are not supported on foreign tables"),
parser_errposition(cxt->pstate, constraint->location)));
if (constraint->keys == NIL)
@@ -546,7 +540,7 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column) if
(cxt->isforeign) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("constraints are not supported on foreign tables"),
+ errmsg("foreign key constraints are not supported on foreign tables"),
parser_errposition(cxt->pstate, constraint->location)));
@@ -605,10 +599,14 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column)static
voidtransformTableConstraint(CreateStmtContext*cxt, Constraint *constraint){
- if (cxt->isforeign)
+ if (cxt->isforeign &&
+ (constraint->contype == CONSTR_PRIMARY ||
+ constraint->contype == CONSTR_UNIQUE ||
+ constraint->contype == CONSTR_EXCLUSION ||
+ constraint->contype == CONSTR_FOREIGN)) ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("constraints are not supported on foreign tables"),
+ errmsg("primary key, unique, exclusion, or foreign key constraints are not supported on foreign
tables"), parser_errposition(cxt->pstate, constraint->location)));
diff --git a/src/include/catalog/pg_inherits_fn.h b/src/include/catalog/pg_inherits_fn.h
index 757d849..228573a 100644
--- a/src/include/catalog/pg_inherits_fn.h
+++ b/src/include/catalog/pg_inherits_fn.h
@@ -21,6 +21,7 @@ extern List *find_inheritance_children(Oid parentrelId, LOCKMODE lockmode);extern List
*find_all_inheritors(OidparentrelId, LOCKMODE lockmode, List **parents);extern bool has_subclass(Oid
relationId);
+extern bool contains_foreign(Oid parentrelId, LOCKMODE lockmode);extern bool typeInheritsFrom(Oid subclassTypeId, Oid
superclassTypeId);#endif /* PG_INHERITS_FN_H */
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 058dc5f..8253a07 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -140,6 +140,15 @@ extern int vacuum_multixact_freeze_min_age;extern int vacuum_multixact_freeze_table_age;
+/* Possible modes for vacuum() */
+typedef enum
+{
+ VAC_MODE_ALL, /* Vacuum/analyze all relations */
+ VAC_MODE_SINGLE, /* Vacuum/analyze a specific relation */
+ VAC_MODE_AUTOVACUUM /* Autovacuum worker */
+} VacuumMode;
+
+/* in commands/vacuum.c */extern void vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast, BufferAccessStrategy
bstrategy,bool for_wraparound, bool isTopLevel);
@@ -174,7 +183,7 @@ extern void lazy_vacuum_rel(Relation onerel, VacuumStmt *vacstmt,
BufferAccessStrategybstrategy);/* in commands/analyze.c */
-extern void analyze_rel(Oid relid, VacuumStmt *vacstmt,
+extern void analyze_rel(Oid relid, VacuumStmt *vacstmt, VacuumMode vacmode, BufferAccessStrategy
bstrategy);externbool std_typanalyze(VacAttrStats *stats);extern double anl_random_fract(void);
diff --git a/src/test/regress/expected/foreign_data.out b/src/test/regress/expected/foreign_data.out
index 60506e0..f15d2e1 100644
--- a/src/test/regress/expected/foreign_data.out
+++ b/src/test/regress/expected/foreign_data.out
@@ -750,16 +750,12 @@ CREATE TABLE use_ft1_column_type (x ft1);ALTER FOREIGN TABLE ft1 ALTER COLUMN c8 SET DATA TYPE
integer; -- ERRORERROR: cannot alter foreign table "ft1" because column "use_ft1_column_type.x" uses its row
typeDROPTABLE use_ft1_column_type;
-ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0); -- ERROR
-ERROR: constraints are not supported on foreign tables
-LINE 1: ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c...
- ^
+ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0);ALTER FOREIGN TABLE ft1 DROP CONSTRAINT no_const;
-- ERROR
-ERROR: "ft1" is not a table
+ERROR: constraint "no_const" of relation "ft1" does not existALTER FOREIGN TABLE ft1 DROP CONSTRAINT IF EXISTS
no_const;
-ERROR: "ft1" is not a table
-ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c1_check;
-ERROR: "ft1" is not a table
+NOTICE: constraint "no_const" of relation "ft1" does not exist, skipping
+ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c9_check;ALTER FOREIGN TABLE ft1 SET WITH OIDS;
--ERRORERROR: "ft1" is not a tableALTER FOREIGN TABLE ft1 OWNER TO regress_test_role;
diff --git a/src/test/regress/sql/foreign_data.sql b/src/test/regress/sql/foreign_data.sql
index f819eb1..2aa5ffe 100644
--- a/src/test/regress/sql/foreign_data.sql
+++ b/src/test/regress/sql/foreign_data.sql
@@ -314,10 +314,10 @@ ALTER FOREIGN TABLE ft1 ALTER COLUMN c8 SET STATISTICS -1;CREATE TABLE use_ft1_column_type (x
ft1);ALTERFOREIGN TABLE ft1 ALTER COLUMN c8 SET DATA TYPE integer; -- ERRORDROP TABLE use_ft1_column_type;
-ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0); -- ERROR
+ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0);ALTER FOREIGN TABLE ft1 DROP CONSTRAINT no_const;
-- ERRORALTER FOREIGN TABLE ft1 DROP CONSTRAINT IF EXISTS no_const;
-ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c1_check;
+ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c9_check;ALTER FOREIGN TABLE ft1 SET WITH OIDS;
--ERRORALTER FOREIGN TABLE ft1 OWNER TO regress_test_role;ALTER FOREIGN TABLE ft1 OPTIONS (DROP delimiter, SET quote
'~',ADD escape '@');
Hello,
> This seems far better than silently performing the command,
> except for the duplicated message:( New bitmap might required to
> avoid the duplication..
I rewrote it in more tidy way. ATController collects all affected
tables on ATRewriteCatalogs as first stage, then emit NOTICE
message according to the affected relations list. The message
looks like,
| =# alter table passwd alter column uname set default 'hoge';
| NOTICE: This command affects 2 foreign tables: cf1, cf2
| ALTER TABLE
Do you feel this too large or complicated? I think so a bit..
regards,
--
Kyotaro Horiguchi
NTT Open Source Software Center
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 8ace8bd..b4e53c1 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -258,6 +258,17 @@ CREATE TABLE products ( even if the value came from the default value definition. </para>
+ <note>
+ <para>
+ Note that constraints can be defined on foreign tables too, but such
+ constraints are not enforced on insert or update. Those constraints are
+ "assertive", and work only to tell planner that some kind of optimization
+ such as constraint exclusion can be considerd. This seems useless, but
+ allows us to use foriegn table as child table (see
+ <xref linkend="ddl-inherit">) to off-load to multiple servers.
+ </para>
+ </note>
+ <sect2 id="ddl-constraints-check-constraints"> <title>Check Constraints</title>
@@ -2017,8 +2028,8 @@ CREATE TABLE capitals ( </para> <para>
- In <productname>PostgreSQL</productname>, a table can inherit from
- zero or more other tables, and a query can reference either all
+ In <productname>PostgreSQL</productname>, a table or foreign table can
+ inherit from zero or more other tables, and a query can reference either all rows of a table or all rows of a
tableplus all of its descendant tables. The latter behavior is the default. For example, the following query finds
thenames of all cities,
diff --git a/doc/src/sgml/ref/alter_foreign_table.sgml b/doc/src/sgml/ref/alter_foreign_table.sgml
index 4d8cfc5..f7a382e 100644
--- a/doc/src/sgml/ref/alter_foreign_table.sgml
+++ b/doc/src/sgml/ref/alter_foreign_table.sgml
@@ -42,6 +42,8 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab ALTER [ COLUMN ]
<replaceableclass="PARAMETER">column_name</replaceable> SET ( <replaceable
class="PARAMETER">attribute_option</replaceable>= <replaceable class="PARAMETER">value</replaceable> [, ... ] )
ALTER[ COLUMN ] <replaceable class="PARAMETER">column_name</replaceable> RESET ( <replaceable
class="PARAMETER">attribute_option</replaceable>[, ... ] ) ALTER [ COLUMN ] <replaceable
class="PARAMETER">column_name</replaceable>OPTIONS ( [ ADD | SET | DROP ] <replaceable
class="PARAMETER">option</replaceable>['<replaceable class="PARAMETER">value</replaceable>'] [, ... ])
+ INHERIT <replaceable class="PARAMETER">parent_table</replaceable>
+ NO INHERIT <replaceable class="PARAMETER">parent_table</replaceable> OWNER TO <replaceable
class="PARAMETER">new_owner</replaceable> OPTIONS ( [ ADD | SET | DROP ] <replaceable
class="PARAMETER">option</replaceable>['<replaceable class="PARAMETER">value</replaceable>'] [, ... ])</synopsis>
@@ -178,6 +180,26 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab </varlistentry>
<varlistentry>
+ <term><literal>INHERIT <replaceable class="PARAMETER">parent_table</replaceable></literal></term>
+ <listitem>
+ <para>
+ This form adds the target foreign table as a new child of the specified
+ parent table. The parent table must be a plain table.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>NO INHERIT <replaceable class="PARAMETER">parent_table</replaceable></literal></term>
+ <listitem>
+ <para>
+ This form removes the target foreign table from the list of children of
+ the specified parent table.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry> <term><literal>OPTIONS ( [ ADD | SET | DROP ] <replaceable class="PARAMETER">option</replaceable>
['<replaceableclass="PARAMETER">value</replaceable>'] [, ... ] )</literal></term> <listitem> <para>
@@ -306,6 +328,16 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab </para>
</listitem> </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="PARAMETER">parent_name</replaceable></term>
+ <listitem>
+ <para>
+ A parent table to associate or de-associate with this foreign table.
+ The parent table must be a plain table.
+ </para>
+ </listitem>
+ </varlistentry> </variablelist> </refsect1>
diff --git a/doc/src/sgml/ref/create_foreign_table.sgml b/doc/src/sgml/ref/create_foreign_table.sgml
index 06a7087..cc11dee 100644
--- a/doc/src/sgml/ref/create_foreign_table.sgml
+++ b/doc/src/sgml/ref/create_foreign_table.sgml
@@ -22,6 +22,7 @@ CREATE FOREIGN TABLE [ IF NOT EXISTS ] <replaceable class="PARAMETER">table_name <replaceable
class="PARAMETER">column_name</replaceable><replaceable class="PARAMETER">data_type</replaceable> [ OPTIONS (
<replaceableclass="PARAMETER">option</replaceable> '<replaceable class="PARAMETER">value</replaceable>' [, ... ] ) ] [
COLLATE<replaceable>collation</replaceable> ] [ <replaceable class="PARAMETER">column_constraint</replaceable> [ ... ]
] [, ... ]] )
+[ INHERITS ( <replaceable>parent_table</replaceable> [, ... ] ) ] SERVER <replaceable
class="parameter">server_name</replaceable>[OPTIONS ( <replaceable class="PARAMETER">option</replaceable> '<replaceable
class="PARAMETER">value</replaceable>'[, ... ] ) ]
@@ -159,6 +160,18 @@ CREATE FOREIGN TABLE [ IF NOT EXISTS ] <replaceable class="PARAMETER">table_name </varlistentry>
<varlistentry>
+ <term><replaceable class="PARAMETER">parent_table</replaceable></term>
+ <listitem>
+ <para>
+ The name of an existing table from which the new foreign table
+ automatically inherits all columns. The specified parent table
+ must be a plain table. See <xref linkend="ddl-inherit"> for the
+ details of table inheritance.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry> <term><replaceable class="PARAMETER">server_name</replaceable></term> <listitem> <para>
diff --git a/src/backend/catalog/pg_inherits.c b/src/backend/catalog/pg_inherits.c
index f263b42..b4a084c 100644
--- a/src/backend/catalog/pg_inherits.c
+++ b/src/backend/catalog/pg_inherits.c
@@ -256,6 +256,48 @@ has_subclass(Oid relationId)/*
+ * Check wether the inheritance tree contains foreign table(s).
+ */
+bool
+contains_foreign(Oid parentrelId, LOCKMODE lockmode)
+{
+ bool result = false;
+ List *tableOIDs;
+ ListCell *lc;
+
+ /* Find all members of the inheritance tree */
+ tableOIDs = find_all_inheritors(parentrelId, lockmode, NULL);
+
+ /* There are no children */
+ if (list_length(tableOIDs) < 2)
+ return result;
+
+ foreach(lc, tableOIDs)
+ {
+ Oid childOID = lfirst_oid(lc);
+ Relation childrel;
+
+ /* Parent should not be foreign */
+ if (childOID == parentrelId)
+ continue;
+
+ /* We already got the needed lock */
+ childrel = heap_open(childOID, NoLock);
+
+ if (childrel->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
+ {
+ /* Found it */
+ result = true;
+ }
+
+ heap_close(childrel, lockmode);
+ }
+
+ return result;
+}
+
+
+/* * Given two type OIDs, determine whether the first is a complex type * (class type) that inherits from the second.
*/
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index a04adea..bb7bdf4 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -115,7 +115,8 @@ static Datum ind_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull); * analyze_rel() --
analyzeone relation */void
-analyze_rel(Oid relid, VacuumStmt *vacstmt, BufferAccessStrategy bstrategy)
+analyze_rel(Oid relid, VacuumStmt *vacstmt, VacuumMode vacmode,
+ BufferAccessStrategy bstrategy){ Relation onerel; int elevel;
@@ -270,7 +271,9 @@ analyze_rel(Oid relid, VacuumStmt *vacstmt, BufferAccessStrategy bstrategy) /* * If there
arechild tables, do recursive ANALYZE. */
- if (onerel->rd_rel->relhassubclass)
+ if (onerel->rd_rel->relhassubclass &&
+ (vacmode == VAC_MODE_SINGLE ||
+ !contains_foreign(RelationGetRelid(onerel), AccessShareLock))) do_analyze_rel(onerel, vacstmt,
acquirefunc,relpages, true, elevel); /*
@@ -1452,12 +1455,15 @@ acquire_inherited_sample_rows(Relation onerel, int elevel,{ List *tableOIDs;
Relation *rels;
+ AcquireSampleRowsFunc *acquirefunc; double *relblocks; double totalblocks;
+ Relation saved_rel; int numrows, nrels, i; ListCell *lc;
+ bool isAnalyzable = true; /* * Find all members of inheritance set. We only need AccessShareLock
on
@@ -1486,6 +1492,8 @@ acquire_inherited_sample_rows(Relation onerel, int elevel, * BlockNumber, so we use double
arithmetic. */ rels = (Relation *) palloc(list_length(tableOIDs) * sizeof(Relation));
+ acquirefunc = (AcquireSampleRowsFunc *) palloc(list_length(tableOIDs)
+ * sizeof(AcquireSampleRowsFunc)); relblocks = (double *)
palloc(list_length(tableOIDs)* sizeof(double)); totalblocks = 0; nrels = 0;
@@ -1507,12 +1515,59 @@ acquire_inherited_sample_rows(Relation onerel, int elevel, } rels[nrels] =
childrel;
- relblocks[nrels] = (double) RelationGetNumberOfBlocks(childrel);
+
+ if (childrel->rd_rel->relkind != RELKIND_FOREIGN_TABLE)
+ {
+ acquirefunc[nrels] = acquire_sample_rows;
+ relblocks[nrels] = (double) RelationGetNumberOfBlocks(childrel);
+ }
+ else
+ {
+ /*
+ * For a foreign table, call the FDW's hook function to see whether
+ * it supports analysis.
+ */
+ FdwRoutine *fdwroutine;
+ BlockNumber relpages = 0;
+ bool ok = false;
+
+ fdwroutine = GetFdwRoutineForRelation(childrel, false);
+ if (fdwroutine->AnalyzeForeignTable != NULL)
+ ok = fdwroutine->AnalyzeForeignTable(childrel,
+ &acquirefunc[nrels],
+ &relpages);
+ if (!ok)
+ {
+ isAnalyzable = false;
+ break;
+ }
+
+ relblocks[nrels] = (double) relpages;
+ }
+ totalblocks += relblocks[nrels]; nrels++; } /*
+ * If there is at least one foreign table that cannot be analyzed, give up.
+ */
+ if (!isAnalyzable)
+ {
+ ereport(WARNING,
+ (errmsg("skipping \"%s\" inheritance tree --- cannot analyze foreign table \"%s\"",
+ RelationGetRelationName(onerel),
+ RelationGetRelationName(rels[nrels]))));
+ for (i = 0; i < nrels; i++)
+ {
+ Relation childrel = rels[i];
+
+ heap_close(childrel, NoLock);
+ }
+ return 0;
+ }
+
+ /* * Now sample rows from each relation, proportionally to its fraction of * the total block count. (This
mightbe less than desirable if the child * rels have radically different free-space percentages, but it's not
@@ -1525,6 +1580,7 @@ acquire_inherited_sample_rows(Relation onerel, int elevel, { Relation childrel =
rels[i]; double childblocks = relblocks[i];
+ AcquireSampleRowsFunc childacquirefunc = acquirefunc[i]; if (childblocks > 0) {
@@ -1540,12 +1596,12 @@ acquire_inherited_sample_rows(Relation onerel, int elevel, tdrows;
/* Fetch a random sample of the child's rows */
- childrows = acquire_sample_rows(childrel,
- elevel,
- rows + numrows,
- childtargrows,
- &trows,
- &tdrows);
+ childrows = childacquirefunc(childrel,
+ elevel,
+ rows + numrows,
+ childtargrows,
+ &trows,
+ &tdrows); /* We may need to convert from child's rowtype
toparent's */ if (childrows > 0 &&
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 25f01e5..6859ff0 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -299,26 +299,30 @@ static void validateForeignKeyConstraint(char *conname,static void
createForeignKeyTriggers(Relationrel, Oid refRelOid, Constraint *fkconstraint,
Oid constraintOid, Oid indexOid);
+static void PrintForeignNotice(List *affected, Relation rel);static void ATController(Relation rel, List *cmds, bool
recurse,LOCKMODE lockmode);static void ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, bool
recurse,bool recursing, LOCKMODE lockmode);
-static void ATRewriteCatalogs(List **wqueue, LOCKMODE lockmode);
-static void ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
- AlterTableCmd *cmd, LOCKMODE lockmode);
+static void ATRewriteCatalogs(List **wqueue, List **affectd, LOCKMODE lockmode);
+static void ATExecCmd(List **wqueue, List **affected, AlteredTableInfo *tab,
+ Relation rel, AlterTableCmd *cmd, LOCKMODE lockmode);static void ATRewriteTables(List **wqueue,
LOCKMODElockmode);static void ATRewriteTable(AlteredTableInfo *tab, Oid OIDNewHeap, LOCKMODE lockmode);static
AlteredTableInfo*ATGetQueueEntry(List **wqueue, Relation rel);
+static void ATAddAffectedRelid(List **affected, Oid relid);static void ATSimplePermissions(Relation rel, int
allowed_targets);staticvoid ATWrongRelkindError(Relation rel, int allowed_targets);static void ATSimpleRecursion(List
**wqueue,Relation rel,
- AlterTableCmd *cmd, bool recurse, LOCKMODE lockmode);
+ AlterTableCmd *cmd, bool recurse,
+ bool include_foreign, LOCKMODE lockmode);static void ATTypedTableRecursion(List **wqueue, Relation
rel,AlterTableCmd *cmd, LOCKMODE lockmode);static List *find_typed_table_dependencies(Oid typeOid,
constchar *typeName, DropBehavior behavior);static void ATPrepAddColumn(List **wqueue,
Relationrel, bool recurse, bool recursing, AlterTableCmd *cmd, LOCKMODE lockmode);
-static void ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
+static void ATExecAddColumn(List **wqueue, List **affected,
+ AlteredTableInfo *tab, Relation rel, ColumnDef *colDef, bool isOid, bool
recurse,bool recursing, LOCKMODE lockmode);static void check_for_column_name_collision(Relation rel, const char
*colname);
@@ -341,13 +345,14 @@ static void ATExecSetStorage(Relation rel, const char *colName, Node *newValue,
LOCKMODElockmode);static void ATPrepDropColumn(List **wqueue, Relation rel, bool recurse, bool recursing,
AlterTableCmd *cmd, LOCKMODE lockmode);
-static void ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
+static void ATExecDropColumn(List **wqueue, List **affected,
+ Relation rel, const char *colName, DropBehavior behavior, bool
recurse,bool recursing, bool missing_ok, LOCKMODE lockmode);static void ATExecAddIndex(AlteredTableInfo
*tab,Relation rel, IndexStmt *stmt, bool is_rebuild, LOCKMODE lockmode);
-static void ATExecAddConstraint(List **wqueue,
+static void ATExecAddConstraint(List **wqueue, List **affected, AlteredTableInfo *tab, Relation
rel, Constraint *newConstraint, bool recurse, bool is_readd, LOCKMODE lockmode);
@@ -467,10 +472,6 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId) ereport(ERROR,
(errcode(ERRCODE_INVALID_TABLE_DEFINITION), errmsg("ON COMMIT can only be used on temporary tables")));
- if (stmt->constraints != NIL && relkind == RELKIND_FOREIGN_TABLE)
- ereport(ERROR,
- (errcode(ERRCODE_WRONG_OBJECT_TYPE),
- errmsg("constraints are not supported on foreign tables"))); /* * Look up the namespace in
whichwe are supposed to create the relation,
@@ -2940,9 +2941,71 @@ AlterTableGetLockLevel(List *cmds)}static void
+PrintForeignNotice(List *affected, Relation rel)
+{
+ ListCell *l;
+ StringInfo namelist = NULL;
+ bool rel_is_foreign = false;
+ int state = 0;
+ int n = 0;
+
+ if (!affected) return;
+
+ foreach (l, affected)
+ {
+ Oid relid = lfirst_oid(l);
+ Relation r = relation_open(relid, NoLock);
+ if (r->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
+ {
+ if (relid == RelationGetRelid(rel))
+ rel_is_foreign = true;
+
+ switch (state)
+ {
+ case 0:
+ namelist = makeStringInfo();
+ appendStringInfoString(namelist, RelationGetRelationName(r));
+ state = 1;
+ break;
+
+ case 1:
+ if (namelist->len < 32)
+ {
+ appendStringInfoString(namelist, ", ");
+ appendStringInfoString(namelist,
+ RelationGetRelationName(r));
+ }
+ else
+ {
+ appendStringInfoString(namelist, "...");
+ state = 2;
+ }
+ break;
+
+ default:;
+ /* Do nothing */
+ }
+
+ n++;
+ }
+ relation_close(r, NoLock);
+ }
+
+ /*
+ * Don't show this notice if the relation designated in this command is
+ * the only foreign table affected.
+ */
+ if (n > 1 || (n == 1 && !rel_is_foreign))
+ ereport(NOTICE,
+ (errmsg_plural("This command affects %d foreign table: %s",
+ "This command affects %d foreign tables: %s",
+ n, n, namelist->data)));
+}
+
+static voidATController(Relation rel, List *cmds, bool recurse, LOCKMODE lockmode){
- List *wqueue = NIL;
+ List *wqueue = NIL, *affected = NIL; ListCell *lcmd; /* Phase 1: preliminary examination of
commands,create work queue */
@@ -2957,10 +3020,14 @@ ATController(Relation rel, List *cmds, bool recurse, LOCKMODE lockmode) relation_close(rel,
NoLock); /* Phase 2: update system catalogs */
- ATRewriteCatalogs(&wqueue, lockmode);
+ ATRewriteCatalogs(&wqueue, &affected, lockmode); /* Phase 3: scan/rewrite tables as needed */
ATRewriteTables(&wqueue,lockmode);
+
+ /* Notice if foreign tables are affected by this command */
+ if (affected)
+ PrintForeignNotice(affected, rel);}/*
@@ -3019,24 +3086,28 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, * rules. */
ATSimplePermissions(rel, ATT_TABLE | ATT_VIEW | ATT_FOREIGN_TABLE);
- ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+ /* Recurses to child tables that are foreign, too */
+ ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode); /* No command-specific prep
needed*/ pass = cmd->def ? AT_PASS_ADD_CONSTR : AT_PASS_DROP; break; case AT_DropNotNull:
/* ALTER COLUMN DROP NOT NULL */ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
- ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+ /* Recurses to child tables that are foreign, too */
+ ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode); /* No command-specific prep
needed*/ pass = AT_PASS_DROP; break; case AT_SetNotNull: /* ALTER COLUMN SET NOT
NULL*/ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
- ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+ /* Recurses to child tables that are foreign, too */
+ ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode); /* No command-specific prep
needed*/ pass = AT_PASS_ADD_CONSTR; break; case AT_SetStatistics: /* ALTER COLUMN SET
STATISTICS*/
- ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+ /* Recurses to child tables that are foreign, too */
+ ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode); /* Performs own permission checks
*/ ATPrepSetStatistics(rel, cmd->name, cmd->def, lockmode); pass = AT_PASS_MISC;
@@ -3049,7 +3120,8 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, break; case
AT_SetStorage: /* ALTER COLUMN SET STORAGE */ ATSimplePermissions(rel, ATT_TABLE | ATT_MATVIEW);
- ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+ /* Don't recurse to child tables that are foreign */
+ ATSimpleRecursion(wqueue, rel, cmd, recurse, false, lockmode); /* No command-specific prep
needed*/ pass = AT_PASS_MISC; break;
@@ -3067,7 +3139,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, pass = AT_PASS_ADD_INDEX;
break; case AT_AddConstraint: /* ADD CONSTRAINT */
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE); /* Recursion occurs during execution
phase*/ /* No command-specific prep needed except saving recurse flag */ if (recurse)
@@ -3081,7 +3153,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, pass = AT_PASS_ADD_CONSTR;
break; case AT_DropConstraint: /* DROP CONSTRAINT */
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE); /* Recursion occurs during execution
phase*/ /* No command-specific prep needed except saving recurse flag */ if (recurse)
@@ -3149,13 +3221,19 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, pass = AT_PASS_MISC;
break; case AT_AddInherit: /* INHERIT */
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE); /* This command never recurses */
ATPrepAddInherit(rel); pass = AT_PASS_MISC; break;
+ case AT_DropInherit: /* NO INHERIT */
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
+ /* This command never recurses */
+ /* No command-specific prep needed */
+ pass = AT_PASS_MISC;
+ break; case AT_AlterConstraint: /* ALTER CONSTRAINT */
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE); pass = AT_PASS_MISC; break;
case AT_ValidateConstraint: /* VALIDATE CONSTRAINT */
@@ -3184,7 +3262,6 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, case AT_EnableAlwaysRule:
case AT_EnableReplicaRule: case AT_DisableRule:
- case AT_DropInherit: /* NO INHERIT */ case AT_AddOf: /* OF */ case AT_DropOf: /*
NOTOF */ ATSimplePermissions(rel, ATT_TABLE);
@@ -3217,7 +3294,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, * conflicts). */static void
-ATRewriteCatalogs(List **wqueue, LOCKMODE lockmode)
+ATRewriteCatalogs(List **wqueue, List **affected, LOCKMODE lockmode){ int pass; ListCell *ltab;
@@ -3248,7 +3325,8 @@ ATRewriteCatalogs(List **wqueue, LOCKMODE lockmode) rel = relation_open(tab->relid,
NoLock); foreach(lcmd, subcmds)
- ATExecCmd(wqueue, tab, rel, (AlterTableCmd *) lfirst(lcmd), lockmode);
+ ATExecCmd(wqueue, affected, tab, rel,
+ (AlterTableCmd *) lfirst(lcmd), lockmode); /* * After the ALTER TYPE
pass,do cleanup work (this is not done in
@@ -3277,19 +3355,21 @@ ATRewriteCatalogs(List **wqueue, LOCKMODE lockmode) * ATExecCmd: dispatch a subcommand to
appropriateexecution routine */static void
-ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel,
+ATExecCmd(List **wqueue, List **affected, AlteredTableInfo *tab, Relation rel, AlterTableCmd *cmd, LOCKMODE
lockmode){
+ ATAddAffectedRelid(affected, RelationGetRelid(rel));
+ switch (cmd->subtype) { case AT_AddColumn: /* ADD COLUMN */ case AT_AddColumnToView:
/* add column via CREATE OR REPLACE * VIEW */
- ATExecAddColumn(wqueue, tab, rel, (ColumnDef *) cmd->def,
+ ATExecAddColumn(wqueue, affected, tab, rel, (ColumnDef *) cmd->def, false,
false,false, lockmode); break; case AT_AddColumnRecurse:
- ATExecAddColumn(wqueue, tab, rel, (ColumnDef *) cmd->def,
+ ATExecAddColumn(wqueue, affected, tab, rel, (ColumnDef *) cmd->def, false,
true,false, lockmode); break; case AT_ColumnDefault: /* ALTER COLUMN DEFAULT */
@@ -3314,11 +3394,11 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel, ATExecSetStorage(rel,
cmd->name,cmd->def, lockmode); break; case AT_DropColumn: /* DROP COLUMN */
- ATExecDropColumn(wqueue, rel, cmd->name,
+ ATExecDropColumn(wqueue, affected, rel, cmd->name, cmd->behavior, false, false,
cmd->missing_ok,lockmode); break; case AT_DropColumnRecurse: /* DROP COLUMN with recursion */
- ATExecDropColumn(wqueue, rel, cmd->name,
+ ATExecDropColumn(wqueue, affected, rel, cmd->name, cmd->behavior, true, false,
cmd->missing_ok,lockmode); break; case AT_AddIndex: /* ADD INDEX */
@@ -3328,16 +3408,19 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel, ATExecAddIndex(tab,
rel,(IndexStmt *) cmd->def, true, lockmode); break; case AT_AddConstraint: /* ADD CONSTRAINT */
- ATExecAddConstraint(wqueue, tab, rel, (Constraint *) cmd->def,
+ ATExecAddConstraint(wqueue, affected, tab, rel,
+ (Constraint *) cmd->def, false, false, lockmode);
break; case AT_AddConstraintRecurse: /* ADD CONSTRAINT with recursion */
- ATExecAddConstraint(wqueue, tab, rel, (Constraint *) cmd->def,
+ ATExecAddConstraint(wqueue, affected, tab, rel,
+ (Constraint *) cmd->def, true, false, lockmode);
break; case AT_ReAddConstraint: /* Re-add pre-existing check *
constraint*/
- ATExecAddConstraint(wqueue, tab, rel, (Constraint *) cmd->def,
+ ATExecAddConstraint(wqueue, affected,
+ tab, rel, (Constraint *) cmd->def, false, true,
lockmode); break; case AT_AddIndexConstraint: /* ADD CONSTRAINT USING INDEX */
@@ -3383,13 +3466,15 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel, case AT_AddOids: /*
SETWITH OIDS */ /* Use the ADD COLUMN code, unless prep decided to do nothing */ if (cmd->def !=
NULL)
- ATExecAddColumn(wqueue, tab, rel, (ColumnDef *) cmd->def,
+ ATExecAddColumn(wqueue, affected,
+ tab, rel, (ColumnDef *) cmd->def, true, false, false,
lockmode); break; case AT_AddOidsRecurse: /* SET WITH OIDS */ /* Use the ADD COLUMN
code,unless prep decided to do nothing */ if (cmd->def != NULL)
- ATExecAddColumn(wqueue, tab, rel, (ColumnDef *) cmd->def,
+ ATExecAddColumn(wqueue, affected,
+ tab, rel, (ColumnDef *) cmd->def, true, true, false,
lockmode); break; case AT_DropOids: /* SET WITHOUT OIDS */
@@ -4009,6 +4094,23 @@ ATGetQueueEntry(List **wqueue, Relation rel)}/*
+ * ATAddAffectedRelid: add a relid to affected list.
+ */
+static void
+ATAddAffectedRelid(List **affected, Oid relid)
+{
+ ListCell *l;
+
+ foreach(l, *affected)
+ {
+ if (lfirst_oid(l) == relid)
+ return;
+ }
+
+ *affected = lappend_oid(*affected, relid);
+}
+
+/* * ATSimplePermissions * * - Ensure that it is a relation (or possibly a view)
@@ -4125,7 +4227,8 @@ ATWrongRelkindError(Relation rel, int allowed_targets) */static voidATSimpleRecursion(List
**wqueue,Relation rel,
- AlterTableCmd *cmd, bool recurse, LOCKMODE lockmode)
+ AlterTableCmd *cmd, bool recurse,
+ bool include_foreign, LOCKMODE lockmode){ /* * Propagate to children if desired. Non-table
relationsnever have
@@ -4153,8 +4256,12 @@ ATSimpleRecursion(List **wqueue, Relation rel, continue; /*
find_all_inheritorsalready got lock */ childrel = relation_open(childrelid, NoLock);
- CheckTableNotInUse(childrel, "ALTER TABLE");
- ATPrepCmd(wqueue, childrel, cmd, false, true, lockmode);
+ if (childrel->rd_rel->relkind != RELKIND_FOREIGN_TABLE
+ || include_foreign)
+ {
+ CheckTableNotInUse(childrel, "ALTER TABLE");
+ ATPrepCmd(wqueue, childrel, cmd, false, true, lockmode);
+ } relation_close(childrel, NoLock); } }
@@ -4421,7 +4528,8 @@ ATPrepAddColumn(List **wqueue, Relation rel, bool recurse, bool recursing,}static void
-ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel,
+ATExecAddColumn(List **wqueue, List **affected,
+ AlteredTableInfo *tab, Relation rel, ColumnDef *colDef, bool isOid, bool
recurse,bool recursing, LOCKMODE lockmode){
@@ -4442,9 +4550,11 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, ListCell *child;
AclResult aclresult;
+ ATAddAffectedRelid(affected, myrelid);
+ /* At top level, permission check was done in ATPrepCmd, else do it */ if (recursing)
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE); attrdesc = heap_open(AttributeRelationId,
RowExclusiveLock);
@@ -4746,7 +4856,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, childtab =
ATGetQueueEntry(wqueue,childrel); /* Recurse to child */
- ATExecAddColumn(wqueue, childtab, childrel,
+ ATExecAddColumn(wqueue, affected, childtab, childrel, colDef, isOid, recurse, true,
lockmode); heap_close(childrel, NoLock);
@@ -4859,7 +4969,19 @@ ATPrepAddOids(List **wqueue, Relation rel, bool recurse, AlterTableCmd *cmd, LOC
ATPrepAddColumn(wqueue,rel, recurse, false, cmd, lockmode); if (recurse)
+ {
+ /*
+ * Don't allow to add an OID column to inheritance tree that contains
+ * foreign table(s)
+ */
+ if (contains_foreign(RelationGetRelid(rel), AccessShareLock))
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot add OID column to inheritance tree \"%s\" because it contains foreign table(s)",
+ RelationGetRelationName(rel))));
+ cmd->subtype = AT_AddOidsRecurse;
+ }}/*
@@ -5327,7 +5449,8 @@ ATPrepDropColumn(List **wqueue, Relation rel, bool recurse, bool recursing,}static void
-ATExecDropColumn(List **wqueue, Relation rel, const char *colName,
+ATExecDropColumn(List **wqueue, List **affected,
+ Relation rel, const char *colName, DropBehavior behavior, bool
recurse,bool recursing, bool missing_ok, LOCKMODE lockmode)
@@ -5338,9 +5461,11 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName, List *children;
ObjectAddressobject;
+ ATAddAffectedRelid(affected, RelationGetRelid(rel));
+ /* At top level, permission check was done in ATPrepCmd, else do it */ if (recursing)
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE); /* * get the number of the attribute
@@ -5426,7 +5551,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName, if
(childatt->attinhcount== 1 && !childatt->attislocal) { /* Time to delete this child
column,too */
- ATExecDropColumn(wqueue, childrel, colName,
+ ATExecDropColumn(wqueue, affected, childrel, colName,
behavior,true, true, false, lockmode); }
@@ -5644,12 +5769,15 @@ ATExecAddIndexConstraint(AlteredTableInfo *tab, Relation rel, * ALTER TABLE ADD CONSTRAINT
*/staticvoid
-ATExecAddConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel,
+ATExecAddConstraint(List **wqueue, List **affected,
+ AlteredTableInfo *tab, Relation rel, Constraint *newConstraint, bool recurse,
boolis_readd, LOCKMODE lockmode){ Assert(IsA(newConstraint, Constraint));
+ ATAddAffectedRelid(affected, RelationGetRelid(rel));
+ /* * Currently, we only expect to see CONSTR_CHECK and CONSTR_FOREIGN nodes * arriving here (see the
preprocessingdone in parse_utilcmd.c). Use a
@@ -5732,7 +5860,14 @@ ATAddCheckConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel, /* At top level,
permissioncheck was done in ATPrepCmd, else do it */ if (recursing)
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
+
+ /* Don't allow ADD constraint NOT VALID on foreign tables */
+ if (tab->relkind == RELKIND_FOREIGN_TABLE &&
+ constr->skip_validation && !recursing)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("NOT VALID is not supported on foreign tables"))); /* * Call AddRelationNewConstraints
todo the work, making sure it works on
@@ -5743,9 +5878,17 @@ ATAddCheckConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel, * omitted from the
returnedlist, which is what we want: we do not need * to do any validation work. That can only happen at child
tables, * though, since we disallow merging at the top level.
- */
+ *
+ * When propagating a NOT VALID option to children that are foreign tables,
+ * we quietly ignore the option. Note that this is safe because foreign
+ * tables don't have any children.
+ */
+ constr = copyObject(constr);
+ if (tab->relkind == RELKIND_FOREIGN_TABLE &&
+ constr->skip_validation && recursing)
+ constr->skip_validation = false; newcons = AddRelationNewConstraints(rel, NIL,
- list_make1(copyObject(constr)),
+ list_make1(constr), recursing,
/*allow_merge */ !recursing, /* is_local */
is_readd); /* is_internal */
@@ -7225,7 +7368,7 @@ ATExecDropConstraint(Relation rel, const char *constrName, /* At top level, permission check
wasdone in ATPrepCmd, else do it */ if (recursing)
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE); conrel = heap_open(ConstraintRelationId,
RowExclusiveLock);
@@ -7560,7 +7703,10 @@ ATPrepAlterColumnType(List **wqueue, * alter would put them out of step. */ if
(recurse)
- ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+ {
+ /* Recurses to child tables that are foreign, too */
+ ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode);
+ } else if (!recursing && find_inheritance_children(RelationGetRelid(rel), NoLock) != NIL)
ereport(ERROR,
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index ded1841..35b5dd2 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -98,6 +98,7 @@ vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast, BufferAccessStrategy bstrategy, bool
for_wraparound,bool isTopLevel){ const char *stmttype;
+ VacuumMode vacmode; volatile bool in_outer_xact, use_own_xacts; List *relations;
@@ -146,6 +147,20 @@ vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast,
ALLOCSET_DEFAULT_MAXSIZE); /*
+ * Identify vacuum mode. If relid is not InvalidOid, the caller should be
+ * an autovacuum worker. See the above comments.
+ */
+ if (relid != InvalidOid)
+ vacmode = VAC_MODE_AUTOVACUUM;
+ else
+ {
+ if (!vacstmt->relation)
+ vacmode = VAC_MODE_ALL;
+ else
+ vacmode = VAC_MODE_SINGLE;
+ }
+
+ /* * If caller didn't give us a buffer strategy object, make one in the * cross-transaction memory
context. */
@@ -248,7 +263,7 @@ vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast,
PushActiveSnapshot(GetTransactionSnapshot()); }
- analyze_rel(relid, vacstmt, vac_strategy);
+ analyze_rel(relid, vacstmt, vacmode, vac_strategy); if (use_own_xacts)
{
diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c
index 52dcc72..238bba2 100644
--- a/src/backend/optimizer/prep/prepunion.c
+++ b/src/backend/optimizer/prep/prepunion.c
@@ -1337,11 +1337,12 @@ expand_inherited_rtentry(PlannerInfo *root, RangeTblEntry *rte, Index rti) /* *
Buildan RTE for the child, and attach to query's rangetable list. * We copy most fields of the parent's RTE,
butreplace relation OID,
- * and set inh = false. Also, set requiredPerms to zero since all
- * required permissions checks are done on the original RTE.
+ * relkind and set inh = false. Also, set requiredPerms to zero since
+ * all required permissions checks are done on the original RTE. */ childrte = copyObject(rte);
childrte->relid = childOID;
+ childrte->relkind = newrelation->rd_rel->relkind; childrte->inh = false; childrte->requiredPerms
=0; parse->rtable = lappend(parse->rtable, childrte);
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index b79af7a..18ced04 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -2062,6 +2062,16 @@ reparameterize_path(PlannerInfo *root, Path *path, case T_SubqueryScan: return
create_subqueryscan_path(root,rel, path->pathkeys, required_outer);
+ case T_ForeignScan:
+ {
+ ForeignPath *fpath = (ForeignPath*) path;
+ ForeignPath *newpath = makeNode(ForeignPath);
+ memcpy(newpath, fpath, sizeof(ForeignPath));
+ newpath->path.param_info =
+ get_baserel_parampathinfo(root, rel, required_outer);
+ /* cost recalc omitted */
+ return (Path *)newpath;
+ } default: break; }
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index e3060a4..b5e47f4 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -4209,32 +4209,32 @@ AlterForeignServerStmt: ALTER SERVER name foreign_server_version
alter_generic_oCreateForeignTableStmt: CREATE FOREIGN TABLE qualified_name '(' OptTableElementList
')'
- SERVER name create_generic_options
+ OptInherit SERVER name create_generic_options { CreateForeignTableStmt
*n= makeNode(CreateForeignTableStmt); $4->relpersistence = RELPERSISTENCE_PERMANENT;
n->base.relation = $4; n->base.tableElts = $6;
- n->base.inhRelations = NIL;
+ n->base.inhRelations = $8; n->base.if_not_exists = false; /*
FDW-specificdata */
- n->servername = $9;
- n->options = $10;
+ n->servername = $10;
+ n->options = $11; $$ = (Node *) n; } | CREATE FOREIGN
TABLEIF_P NOT EXISTS qualified_name '(' OptTableElementList ')'
- SERVER name create_generic_options
+ OptInherit SERVER name create_generic_options { CreateForeignTableStmt
*n= makeNode(CreateForeignTableStmt); $7->relpersistence = RELPERSISTENCE_PERMANENT;
n->base.relation = $7; n->base.tableElts = $9;
- n->base.inhRelations = NIL;
+ n->base.inhRelations = $11; n->base.if_not_exists = true; /*
FDW-specificdata */
- n->servername = $12;
- n->options = $13;
+ n->servername = $13;
+ n->options = $14; $$ = (Node *) n; } ;
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index 1e071d7..955f27c 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -515,12 +515,6 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column) break;
case CONSTR_CHECK:
- if (cxt->isforeign)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("constraints are not supported on foreign tables"),
- parser_errposition(cxt->pstate,
- constraint->location))); cxt->ckconstraints =
lappend(cxt->ckconstraints,constraint); break;
@@ -529,7 +523,7 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column) if
(cxt->isforeign) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("constraints are not supported on foreign tables"),
+ errmsg("primary key or unique constraints are not supported on foreign tables"),
parser_errposition(cxt->pstate, constraint->location)));
if (constraint->keys == NIL)
@@ -546,7 +540,7 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column) if
(cxt->isforeign) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("constraints are not supported on foreign tables"),
+ errmsg("foreign key constraints are not supported on foreign tables"),
parser_errposition(cxt->pstate, constraint->location)));
@@ -605,10 +599,14 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column)static
voidtransformTableConstraint(CreateStmtContext*cxt, Constraint *constraint){
- if (cxt->isforeign)
+ if (cxt->isforeign &&
+ (constraint->contype == CONSTR_PRIMARY ||
+ constraint->contype == CONSTR_UNIQUE ||
+ constraint->contype == CONSTR_EXCLUSION ||
+ constraint->contype == CONSTR_FOREIGN)) ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("constraints are not supported on foreign tables"),
+ errmsg("primary key, unique, exclusion, or foreign key constraints are not supported on foreign
tables"), parser_errposition(cxt->pstate, constraint->location)));
diff --git a/src/include/catalog/pg_inherits_fn.h b/src/include/catalog/pg_inherits_fn.h
index 757d849..228573a 100644
--- a/src/include/catalog/pg_inherits_fn.h
+++ b/src/include/catalog/pg_inherits_fn.h
@@ -21,6 +21,7 @@ extern List *find_inheritance_children(Oid parentrelId, LOCKMODE lockmode);extern List
*find_all_inheritors(OidparentrelId, LOCKMODE lockmode, List **parents);extern bool has_subclass(Oid
relationId);
+extern bool contains_foreign(Oid parentrelId, LOCKMODE lockmode);extern bool typeInheritsFrom(Oid subclassTypeId, Oid
superclassTypeId);#endif /* PG_INHERITS_FN_H */
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 058dc5f..8253a07 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -140,6 +140,15 @@ extern int vacuum_multixact_freeze_min_age;extern int vacuum_multixact_freeze_table_age;
+/* Possible modes for vacuum() */
+typedef enum
+{
+ VAC_MODE_ALL, /* Vacuum/analyze all relations */
+ VAC_MODE_SINGLE, /* Vacuum/analyze a specific relation */
+ VAC_MODE_AUTOVACUUM /* Autovacuum worker */
+} VacuumMode;
+
+/* in commands/vacuum.c */extern void vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast, BufferAccessStrategy
bstrategy,bool for_wraparound, bool isTopLevel);
@@ -174,7 +183,7 @@ extern void lazy_vacuum_rel(Relation onerel, VacuumStmt *vacstmt,
BufferAccessStrategybstrategy);/* in commands/analyze.c */
-extern void analyze_rel(Oid relid, VacuumStmt *vacstmt,
+extern void analyze_rel(Oid relid, VacuumStmt *vacstmt, VacuumMode vacmode, BufferAccessStrategy
bstrategy);externbool std_typanalyze(VacAttrStats *stats);extern double anl_random_fract(void);
diff --git a/src/test/regress/expected/foreign_data.out b/src/test/regress/expected/foreign_data.out
index 60506e0..f15d2e1 100644
--- a/src/test/regress/expected/foreign_data.out
+++ b/src/test/regress/expected/foreign_data.out
@@ -750,16 +750,12 @@ CREATE TABLE use_ft1_column_type (x ft1);ALTER FOREIGN TABLE ft1 ALTER COLUMN c8 SET DATA TYPE
integer; -- ERRORERROR: cannot alter foreign table "ft1" because column "use_ft1_column_type.x" uses its row
typeDROPTABLE use_ft1_column_type;
-ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0); -- ERROR
-ERROR: constraints are not supported on foreign tables
-LINE 1: ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c...
- ^
+ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0);ALTER FOREIGN TABLE ft1 DROP CONSTRAINT no_const;
-- ERROR
-ERROR: "ft1" is not a table
+ERROR: constraint "no_const" of relation "ft1" does not existALTER FOREIGN TABLE ft1 DROP CONSTRAINT IF EXISTS
no_const;
-ERROR: "ft1" is not a table
-ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c1_check;
-ERROR: "ft1" is not a table
+NOTICE: constraint "no_const" of relation "ft1" does not exist, skipping
+ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c9_check;ALTER FOREIGN TABLE ft1 SET WITH OIDS;
--ERRORERROR: "ft1" is not a tableALTER FOREIGN TABLE ft1 OWNER TO regress_test_role;
diff --git a/src/test/regress/sql/foreign_data.sql b/src/test/regress/sql/foreign_data.sql
index f819eb1..2aa5ffe 100644
--- a/src/test/regress/sql/foreign_data.sql
+++ b/src/test/regress/sql/foreign_data.sql
@@ -314,10 +314,10 @@ ALTER FOREIGN TABLE ft1 ALTER COLUMN c8 SET STATISTICS -1;CREATE TABLE use_ft1_column_type (x
ft1);ALTERFOREIGN TABLE ft1 ALTER COLUMN c8 SET DATA TYPE integer; -- ERRORDROP TABLE use_ft1_column_type;
-ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0); -- ERROR
+ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0);ALTER FOREIGN TABLE ft1 DROP CONSTRAINT no_const;
-- ERRORALTER FOREIGN TABLE ft1 DROP CONSTRAINT IF EXISTS no_const;
-ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c1_check;
+ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c9_check;ALTER FOREIGN TABLE ft1 SET WITH OIDS;
--ERRORALTER FOREIGN TABLE ft1 OWNER TO regress_test_role;ALTER FOREIGN TABLE ft1 OPTIONS (DROP delimiter, SET quote
'~',ADD escape '@');
(2014/03/11 14:07), Kyotaro HORIGUCHI wrote: >> This seems far better than silently performing the command, >> except for the duplicated message:( New bitmap might required to >> avoid the duplication.. > > I rewrote it in more tidy way. ATController collects all affected > tables on ATRewriteCatalogs as first stage, then emit NOTICE > message according to the affected relations list. The message > looks like, > > | =# alter table passwd alter column uname set default 'hoge'; > | NOTICE: This command affects 2 foreign tables: cf1, cf2 > | ALTER TABLE > > Do you feel this too large or complicated? I think so a bit.. No. I think that would be a useful message for the user. My feeling is it would be better to show this kind of messages for all the affected tables whether or not the affected ones are foreign. How about introducing a VERBOSE option for ALTER TABLE? Though, I think that should be implemented as a separate patch. Thanks, Best regards, Etsuro Fujita
Hi Horiguchi-san, Thank you for working this patch! (2014/03/10 17:29), Kyotaro HORIGUCHI wrote: > Hello. As a minimal implementation, I made an attempt that emit > NOTICE message when alter table affects foreign tables. It looks > like following, > > | =# alter table passwd add column added int, add column added2 int; > | NOTICE: This command affects foreign relation "cf1" > | NOTICE: This command affects foreign relation "cf1" > | ALTER TABLE > | =# select * from passwd; > | ERROR: missing data for column "added" > | CONTEXT: COPY cf1, line 1: "root:x:0:0:root:/root:/bin/bash" > | =# > > This seems far better than silently performing the command, > except for the duplicated message:( New bitmap might required to > avoid the duplication.. As I said before, I think it would be better to show this kind of information on each of the affected tables whether or not the affected one is foreign. I also think it would be better to show it only when the user has specified an option to do so, similar to a VERBOSE option of other commands. ISTM this should be implemented as a separate patch. > I made the changes above and below as an attempt in the attached > patch foreign_inherit-v4.patch > >> I think the problem is foreign childs in inheritance tables >> prevents all menber in the inheritance relation from using >> parameterized paths, correct? Yes, I think so, too. >> Hmm. I tried minimal implementation to do that. This omits cost >> recalculation but seems to work as expected. This seems enough if >> cost recalc is trivial here. I think we should redo the cost/size estimate, because for example, greater parameterization leads to a smaller rowcount estimate, if I understand correctly. In addition, I think this reparameterization should be done by the FDW itself, becasuse the FDW has more knowledge about it than the PG core. So, I think we should introduce a new FDW routine for that, say ReparameterizeForeignPath(), as proposed in [1]. Attached is an updated version of the patch. Due to the above reason, I removed from the patch the message displaying function you added. Sorry for the delay. [1] http://www.postgresql.org/message-id/530C7464.6020006@lab.ntt.co.jp Best regards, Etsuro Fujita
Attachment
Hi Fujita-san, > Thank you for working this patch! No problem, but my point seems always out of the main target a bit:( > > | =# alter table passwd add column added int, add column added2 int; > > | NOTICE: This command affects foreign relation "cf1" > > | NOTICE: This command affects foreign relation "cf1" > > | ALTER TABLE > > | =# select * from passwd; > > | ERROR: missing data for column "added" > > | CONTEXT: COPY cf1, line 1: "root:x:0:0:root:/root:/bin/bash" > > | =# > > > > This seems far better than silently performing the command, > > except for the duplicated message:( New bitmap might required to > > avoid the duplication.. > > As I said before, I think it would be better to show this kind of > information on each of the affected tables whether or not the affected > one is foreign. I also think it would be better to show it only when > the user has specified an option to do so, similar to a VERBOSE option > of other commands. ISTM this should be implemented as a separate > patch. Hmm. I *wish* this kind of indication to be with this patch even only for foreign tables which would have inconsistency potentially. Expanding to other objects and/or knobs are no problem to be added later. > >> Hmm. I tried minimal implementation to do that. This omits cost > >> recalculation but seems to work as expected. This seems enough if > >> cost recalc is trivial here. > > I think we should redo the cost/size estimate, because for example, > greater parameterization leads to a smaller rowcount estimate, if I > understand correctly. In addition, I think this reparameterization > should be done by the FDW itself, becasuse the FDW has more knowledge > about it than the PG core. So, I think we should introduce a new FDW > routine for that, say ReparameterizeForeignPath(), as proposed in > [1]. Attached is an updated version of the patch. Due to the above > reason, I removed from the patch the message displaying function you > added. > > Sorry for the delay. > > [1] > http://www.postgresql.org/message-id/530C7464.6020006@lab.ntt.co.jp I had a rough look on foreign reparameterize stuff. It seems ok generally. By the way, Can I have a simple script to build an environment to run this on? regards, -- Kyotaro Horiguchi NTT Open Source Software Center
Hello,
> By the way, Can I have a simple script to build an environment to
> run this on?
I built test environment and ran the simple test using
postgres_fdw and got parameterized path from v3 patch on the
following operation as shown there, and v6 also gives one, but I
haven't seen the reparameterization of v6 patch work.
# How could I think to have got it work before?
Do you have any idea to make postgreReparameterizeForeignPath on
foreign (child) tables works effectively?
regards,
====
### on pg1/pg2:
create table pu1 (a int not null, b int not null, c int, d text);
create unique index i_pu1_ab on pu1 (a, b);
create unique index i_pu1_c on pu1 (c);
create table cu11 (like pu1 including all) inherits (pu1);
create table cu12 (like pu1 including all) inherits (pu1);
insert into cu11 (select a / 5, 4 - (a % 5), a, 'cu11' from generate_series(000000, 099999) a);
insert into cu12 (select a / 5, 4 - (a % 5), a, 'cu12' from generate_series(100000, 199999) a);
### on pg1:
create extension postgres_fdw;
create server pg2 foreign data wrapper postgres_fdw options (host '/tmp', port '5433', dbname 'postgres');
create user mapping for current_user server pg2 options (user 'horiguti');
create foreign table _cu11 (a int, b int, c int, d text) server pg2 options (table_name 'cu11', use_remote_estimate
'true');
create foreign table _cu12 (a int, b int, c int, d text) server pg2 options (table_name 'cu12', use_remote_estimate
'true');
create table rpu1 (a int, b int, c int, d text);
alter foreign table _cu11 inherit rpu1;
alter foreign table _cu12 inherit rpu1;
analyze rpu1;
=# explain analyze select pu1.* from pu1 join rpu1 on (pu1.c = rpu1.c) where pu1.a = 3;
QUERY PLAN
-------------------------------------------------------------------------------Nested Loop (cost=0.00..2414.57 rows=11
width=19) (actual time=0.710..7.167 rows=5 loops=1) -> Append (cost=0.00..30.76 rows=11 width=19)
<localside.. ommitted> -> Append (cost=0.00..216.68 rows=3 width=4) (actual time=0.726..1.419 rows=1
loops=5) -> Seq Scan on rpu1 (cost=0.00..0.00 rows=1 width=4) (actual
time=0.000..0.000rows=0 loops=5) Filter: (pu1.c = c) -> Foreign Scan on _cu11
(cost=100.30..108.34rows=1 width=4) (actual time=0.602..0.603 rows=1 loops=5)
-> Foreign Scan on _cu12 (cost=100.30..108.34 rows=1 width=4) (actual
time=0.566..0.566rows=0 loops=5)Planning time: 4.452 msTotal runtime: 7.663 ms
(15 rows)
--
Kyotaro Horiguchi
NTT Open Source Software Center
(2014/03/18 18:38), Kyotaro HORIGUCHI wrote:
>> By the way, Can I have a simple script to build an environment to
>> run this on?
>
> I built test environment and ran the simple test using
> postgres_fdw and got parameterized path from v3 patch on the
> following operation as shown there, and v6 also gives one, but I
> haven't seen the reparameterization of v6 patch work.
>
> # How could I think to have got it work before?
>
> Do you have any idea to make postgreReparameterizeForeignPath on
> foreign (child) tables works effectively?
> =# explain analyze select pu1.*
> from pu1 join rpu1 on (pu1.c = rpu1.c) where pu1.a = 3;
ISTM postgresReparameterizeForeignPath() cannot be called in this query
in principle. Here is a simple example for the case where the
use_remote_estimate option is true:
# On mydatabase
mydatabase=# CREATE TABLE mytable (id INTEGER, x INTEGER);
CREATE TABLE
mydatabase=# INSERT INTO mytable SELECT x, x FROM generate_series(0,
9999) x;
INSERT 0 10000
# On postgres
postgres=# CREATE TABLE inttable (id INTEGER);
CREATE TABLE
postgres=# INSERT INTO inttable SELECT x FROM generate_series(0, 9999) x;
INSERT 0 10000
postgres=# ANALYZE inttable;
ANALYZE
postgres=# CREATE TABLE patest0 (id INTEGER, x INTEGER);
CREATE TABLE
postgres=# CREATE TABLE patest1 () INHERITS (patest0);
CREATE TABLE
postgres=# INSERT INTO patest1 SELECT x, x FROM generate_series(0, 9999) x;
INSERT 0 10000
postgres=# CREATE INDEX patest1_id_idx ON patest1(id);
CREATE INDEX
postgres=# CREATE SERVER myserver FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (host 'localhost', dbname 'mydatabase');
CREATE SERVER
postgres=# CREATE USER MAPPING FOR PUBLIC SERVER myserver OPTIONS (user
'pgsql');
CREATE USER MAPPING
postgres=# CREATE FOREIGN TABLE patest2 () INHERITS (patest0) SERVER
myserver OPTIONS (table_name 'mytable');
CREATE FOREIGN TABLE
postgres=# ANALYZE patest0;
ANALYZE
postgres=# ANALYZE patest1;
ANALYZE
postgres=# ANALYZE patest2;
ANALYZE
postgres=# EXPLAIN VERBOSE SELECT * FROM patest0 join (SELECT id FROM
inttable LIMIT 1) ss ON patest0.id = ss.id;
QUERY PLAN
-------------------------------------------------------------------------------------------------
Nested Loop (cost=0.00..478.36 rows=2 width=12)
Output: patest0.id, patest0.x, inttable.id
-> Limit (cost=0.00..0.01 rows=1 width=4)
Output: inttable.id
-> Seq Scan on public.inttable (cost=0.00..145.00 rows=10000
width=4)
Output: inttable.id
-> Append (cost=0.00..478.31 rows=3 width=8)
-> Seq Scan on public.patest0 (cost=0.00..0.00 rows=1 width=8)
Output: patest0.id, patest0.x
Filter: (inttable.id = patest0.id)
-> Index Scan using patest1_id_idx on public.patest1
(cost=0.29..8.30 rows=1 width=8)
Output: patest1.id, patest1.x
Index Cond: (patest1.id = inttable.id)
-> Foreign Scan on public.patest2 (cost=100.00..470.00
rows=1 width=8)
Output: patest2.id, patest2.x
Remote SQL: SELECT id, x FROM public.mytable WHERE
(($1::integer = id))
Planning time: 0.233 ms
(17 rows)
I revised the patch. Patche attached, though I plan to update the
documentation further early next week.
Thanks,
Best regards,
Etsuro Fujita
Attachment
(2014/03/20 21:59), Etsuro Fujita wrote: > Here is a simple example for the case where the > use_remote_estimate option is true: Sorry, I incorrectly wrote it. The following example is for the case where the option is *false*, as you see. > # On mydatabase > > mydatabase=# CREATE TABLE mytable (id INTEGER, x INTEGER); > CREATE TABLE > mydatabase=# INSERT INTO mytable SELECT x, x FROM generate_series(0, > 9999) x; > INSERT 0 10000 > > # On postgres > > postgres=# CREATE TABLE inttable (id INTEGER); > CREATE TABLE > postgres=# INSERT INTO inttable SELECT x FROM generate_series(0, 9999) x; > INSERT 0 10000 > postgres=# ANALYZE inttable; > ANALYZE > > postgres=# CREATE TABLE patest0 (id INTEGER, x INTEGER); > CREATE TABLE > postgres=# CREATE TABLE patest1 () INHERITS (patest0); > CREATE TABLE > postgres=# INSERT INTO patest1 SELECT x, x FROM generate_series(0, 9999) x; > INSERT 0 10000 > postgres=# CREATE INDEX patest1_id_idx ON patest1(id); > CREATE INDEX > postgres=# CREATE SERVER myserver FOREIGN DATA WRAPPER postgres_fdw > OPTIONS (host 'localhost', dbname 'mydatabase'); > CREATE SERVER > postgres=# CREATE USER MAPPING FOR PUBLIC SERVER myserver OPTIONS (user > 'pgsql'); > CREATE USER MAPPING > postgres=# CREATE FOREIGN TABLE patest2 () INHERITS (patest0) SERVER > myserver OPTIONS (table_name 'mytable'); > CREATE FOREIGN TABLE > postgres=# ANALYZE patest0; > ANALYZE > postgres=# ANALYZE patest1; > ANALYZE > postgres=# ANALYZE patest2; > ANALYZE > postgres=# EXPLAIN VERBOSE SELECT * FROM patest0 join (SELECT id FROM > inttable LIMIT 1) ss ON patest0.id = ss.id; > QUERY PLAN > ------------------------------------------------------------------------------------------------- > > Nested Loop (cost=0.00..478.36 rows=2 width=12) > Output: patest0.id, patest0.x, inttable.id > -> Limit (cost=0.00..0.01 rows=1 width=4) > Output: inttable.id > -> Seq Scan on public.inttable (cost=0.00..145.00 rows=10000 > width=4) > Output: inttable.id > -> Append (cost=0.00..478.31 rows=3 width=8) > -> Seq Scan on public.patest0 (cost=0.00..0.00 rows=1 width=8) > Output: patest0.id, patest0.x > Filter: (inttable.id = patest0.id) > -> Index Scan using patest1_id_idx on public.patest1 > (cost=0.29..8.30 rows=1 width=8) > Output: patest1.id, patest1.x > Index Cond: (patest1.id = inttable.id) > -> Foreign Scan on public.patest2 (cost=100.00..470.00 > rows=1 width=8) > Output: patest2.id, patest2.x > Remote SQL: SELECT id, x FROM public.mytable WHERE > (($1::integer = id)) > Planning time: 0.233 ms > (17 rows) Sorry for the delay. Best regards, Etsuro Fujita
(2014/03/20 21:59), Etsuro Fujita wrote: > I revised the patch. Patche attached, though I plan to update the > documentation further early next week. I updated the documentation further and revise the patch further. Attached is an updated version of the patch. Thanks, Best regards, Etsuro Fujita
Attachment
Hello, I could see reparameterize for foreign path to work effectively thanks to your advice. The key point was setting use_remote_estimate to false and existence of another child to get parameterized path in prior stages. The overall patch was applied on HEAD and compiled cleanly except for a warning. > analyze.c: In function ‘acquire_inherited_sample_rows’: > analyze.c:1461: warning: unused variable ‘saved_rel’ As for postgres-fdw, the point I felt uneasy in previous patch was fixed already:) - which was coping with omission of ReparameterizeForeignPath. And for file-fdw, you made a change to re-create foreignscan node instead of the previous copy-and-modify. Is the reason you did it that you considered the cost of 're-checking whether to selectively perform binary conversion' is low enough? Or other reasons? Finally, although I insist the necessity of the warning for child foreign tables on alter table, if you belive it to be put off, I'll compromise by putting a note to CF-app that last judgement is left to committer. regards, -- Kyotaro Horiguchi NTT Open Source Software Center
Hi Horiguchi-san, (2014/03/26 17:14), Kyotaro HORIGUCHI wrote: > The overall patch was applied on HEAD and compiled cleanly except > for a warning. > >> analyze.c: In function ‘acquire_inherited_sample_rows’: >> analyze.c:1461: warning: unused variable ‘saved_rel’ I've fixed this in the latest version (v8) of the patch. > And for file-fdw, you made a change to re-create foreignscan node > instead of the previous copy-and-modify. Is the reason you did it > that you considered the cost of 're-checking whether to > selectively perform binary conversion' is low enough? Or other > reasons? The reason is that we get the result of the recheck from path->fdw_private. Sorry, I'd forgotten it. So, I modified the code to simply call create_foreignscan_path(). > Finally, although I insist the necessity of the warning for child > foreign tables on alter table, if you belive it to be put off, > I'll compromise by putting a note to CF-app that last judgement > is left to committer. OK So, if there are no objections of other, I'll mark this patch as "ready for committer" and do that. Thanks, Best regards, Etsuro Fujita
Hello,
> >> analyze.c: In function ‘acquire_inherited_sample_rows’:
> >> analyze.c:1461: warning: unused variable ‘saved_rel’
>
> I've fixed this in the latest version (v8) of the patch.
Mmm. sorry. I missed v8 patch. Then I had a look on that and have
a question.
You've added a check for relkind of baserel of the foreign path
to be reparameterized. If this should be an assertion - not a
conditional branch -, it would be better to put the assertion in
create_foreignscan_path instead of there.
=====
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -1723,6 +1723,7 @@ create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel, List
*fdw_private){ ForeignPath *pathnode = makeNode(ForeignPath);
+ Assert(rel->rtekind == RTE_RELATION); pathnode->path.pathtype = T_ForeignScan; pathnode->path.parent = rel;
=====
> > And for file-fdw, you made a change to re-create foreignscan node
> > instead of the previous copy-and-modify. Is the reason you did it
> > that you considered the cost of 're-checking whether to
> > selectively perform binary conversion' is low enough? Or other
> > reasons?
>
> The reason is that we get the result of the recheck from
> path->fdw_private. Sorry, I'd forgotten it. So, I modified the code to
> simply call create_foreignscan_path().
Anyway you new code seems closer to the basics and the gain by
the previous optimization don't seem to be significant..
> > Finally, although I insist the necessity of the warning for child
> > foreign tables on alter table, if you belive it to be put off,
> > I'll compromise by putting a note to CF-app that last judgement
> > is left to committer.
>
> OK So, if there are no objections of other, I'll mark this patch as
> "ready for committer" and do that.
Thank you.
regards,
--
Kyotaro Horiguchi
NTT Open Source Software Center
Hi Horiguchi-san,
(2014/03/27 17:09), Kyotaro HORIGUCHI wrote:
>>>> analyze.c: In function ‘acquire_inherited_sample_rows’:
>>>> analyze.c:1461: warning: unused variable ‘saved_rel’
>>
>> I've fixed this in the latest version (v8) of the patch.
>
> Mmm. sorry. I missed v8 patch. Then I had a look on that and have
> a question.
Thank you for the review!
> You've added a check for relkind of baserel of the foreign path
> to be reparameterized. If this should be an assertion - not a
> conditional branch -, it would be better to put the assertion in
> create_foreignscan_path instead of there.
>
> =====
> --- a/src/backend/optimizer/util/pathnode.c
> +++ b/src/backend/optimizer/util/pathnode.c
> @@ -1723,6 +1723,7 @@ create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
> List *fdw_private)
> {
> ForeignPath *pathnode = makeNode(ForeignPath);
> + Assert(rel->rtekind == RTE_RELATION);
>
> pathnode->path.pathtype = T_ForeignScan;
> pathnode->path.parent = rel;
> =====
Maybe I'm missing the point, but I don't think it'd be better to put the
assertion in create_foreignscan_path(). And I think it'd be the caller'
responsiblity to ensure that equality, as any other pathnode creation
routine for a baserel in pathnode.c assumes that equality.
>>> And for file-fdw, you made a change to re-create foreignscan node
>>> instead of the previous copy-and-modify. Is the reason you did it
>>> that you considered the cost of 're-checking whether to
>>> selectively perform binary conversion' is low enough? Or other
>>> reasons?
>>
>> The reason is that we get the result of the recheck from
>> path->fdw_private. Sorry, I'd forgotten it. So, I modified the code to
>> simply call create_foreignscan_path().
>
> Anyway you new code seems closer to the basics and the gain by
> the previous optimization don't seem to be significant..
Yeah, that's true. I have to admit that the previous optimization is
nonsense.
Thanks,
Best regards,
Etsuro Fujita
(2014/03/27 10:49), Etsuro Fujita wrote: > (2014/03/26 17:14), Kyotaro HORIGUCHI wrote: >> Finally, although I insist the necessity of the warning for child >> foreign tables on alter table, if you belive it to be put off, >> I'll compromise by putting a note to CF-app that last judgement >> is left to committer. > > OK So, if there are no objections of other, I'll mark this patch as > "ready for committer" and do that. I'll do it right now, since there seems to be no objections of others. I've revised the patch a bit further, mainly the documentation. Patch attached. Note: as mentioned above, Horiguchi-san insisted that warning functionality for recursive ALTER TABLE operations for inheritance trees that contain one or more children that are foreign. However, that functionality hasn't been included in the patch, since ISTM that that functionality should be implemented as a separate patch [1]. We leave this for commiters to decide. [1] http://www.postgresql.org/message-id/5321ABD2.6000104@lab.ntt.co.jp Thanks, Best regards, Etsuro Fujita
Attachment
Hi, > > ForeignPath *pathnode = makeNode(ForeignPath); > > + Assert(rel->rtekind == RTE_RELATION); > > > > pathnode->path.pathtype = T_ForeignScan; .. > Maybe I'm missing the point, but I don't think it'd be better to put the > assertion in create_foreignscan_path(). And I think it'd be the caller' > responsiblity to ensure that equality, as any other pathnode creation > routine for a baserel in pathnode.c assumes that equality. Hmm. The assertion (not shown above but you put in parameterize_path:) seems to say that 'base relation for foreign paths must be a RTE_RELATION' isn't right? But I don't see anything putting such a restriction in reparameterize_path itself. Could you tell me where such a restriction comes from? Or who needs such a restriction? I think any assertions shouldn't be anywhere other than where just before needed. Thoughts? regards, -- Kyotaro Horiguchi NTT Open Source Software Center
(2014/03/28 13:28), Kyotaro HORIGUCHI wrote: > Hmm. The assertion (not shown above but you put in > parameterize_path:) seems to say that 'base relation for foreign > paths must be a RTE_RELATION' isn't right? I think that's right. Please see allpaths.c, especially set_rel_pathlist(). For your information, the same assertion can be found in create_foreignscan_plan(). Thanks, Best regards, Etsuro Fujita
I've found some bugs. Attached is an updated version (v10). Changes: * When CREATE FOREIGN TABLE INHERITS, don't allow a foreign table to inherit from parent tables that have OID system columns. * When CREATE FOREIGN TABLE INHERITS, reset the storage parameter for inherited columns that have non-defaut values. * Don't allow CHECK (expr) *NO INHERIT* on foreign tables, since those tables cannot have child tables. * Fix and update the documentation. Thanks, Best regards, Etsuro Fujita
Attachment
Attached is v11. Changes: * Rebased to head * Improve an error message added to tablecmd.c to match it to existing ones there * Improve the documentaion a bit Thanks, Best regards, Etsuro Fujita
Attachment
(2014/04/02 21:25), Etsuro Fujita wrote: > Attached is v11. > > Changes: > > * Rebased to head > * Improve an error message added to tablecmd.c to match it to existing > ones there > * Improve the documentaion a bit I moved this to 2014-06. Since I've merged with the initial patch by Hanada-san (1) a feature to allow the inherited stats to be computed by the ANALYZE command and (2) a new FDW routine for path reparameterization, I put my name on the author. Thanks, Best regards, Etsuro Fujita
Hello,
Before continueing discussion, I tried this patch on the current
head.
This patch applies cleanly except one hunk because of a
modification just AFTER it, which did no harm. Finally all
regression tests suceeded.
Attached is the rebased patch of v11 up to the current master.
regards,
--
Kyotaro Horiguchi
NTT Open Source Software Center
diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c
index 5a4d5aa..5a9aec0 100644
--- a/contrib/file_fdw/file_fdw.c
+++ b/contrib/file_fdw/file_fdw.c
@@ -115,6 +115,11 @@ static void fileGetForeignRelSize(PlannerInfo *root,static void fileGetForeignPaths(PlannerInfo
*root, RelOptInfo *baserel, Oid foreigntableid);
+static ForeignPath *fileReparameterizeForeignPath(PlannerInfo *root,
+ RelOptInfo *baserel,
+ Oid foreigntableid,
+ ForeignPath *path,
+ Relids required_outer);static ForeignScan
*fileGetForeignPlan(PlannerInfo*root, RelOptInfo *baserel, Oid foreigntableid,
@@ -143,7 +148,7 @@ static bool check_selective_binary_conversion(RelOptInfo *baserel,static void
estimate_size(PlannerInfo*root, RelOptInfo *baserel, FileFdwPlanState *fdw_private);static void
estimate_costs(PlannerInfo*root, RelOptInfo *baserel,
- FileFdwPlanState *fdw_private,
+ FileFdwPlanState *fdw_private, List *join_conds, Cost *startup_cost, Cost
*total_cost);staticint file_acquire_sample_rows(Relation onerel, int elevel, HeapTuple *rows,
inttargrows,
@@ -161,6 +166,7 @@ file_fdw_handler(PG_FUNCTION_ARGS) fdwroutine->GetForeignRelSize = fileGetForeignRelSize;
fdwroutine->GetForeignPaths= fileGetForeignPaths;
+ fdwroutine->ReparameterizeForeignPath = fileReparameterizeForeignPath; fdwroutine->GetForeignPlan =
fileGetForeignPlan; fdwroutine->ExplainForeignScan = fileExplainForeignScan; fdwroutine->BeginForeignScan =
fileBeginForeignScan;
@@ -509,7 +515,8 @@ fileGetForeignPaths(PlannerInfo *root, (Node *) columns));
/* Estimate costs */
- estimate_costs(root, baserel, fdw_private,
+ estimate_costs(root, baserel,
+ fdw_private, NIL, &startup_cost, &total_cost); /*
@@ -534,6 +541,41 @@ fileGetForeignPaths(PlannerInfo *root,}/*
+ * fileReparameterizeForeignPath
+ * Attempt to modify a given path to have greater parameterization
+ */
+static ForeignPath *
+fileReparameterizeForeignPath(PlannerInfo *root,
+ RelOptInfo *baserel,
+ Oid foreigntableid,
+ ForeignPath *path,
+ Relids required_outer)
+{
+ ParamPathInfo *param_info;
+ FileFdwPlanState *fdw_private = (FileFdwPlanState *) baserel->fdw_private;
+ Cost startup_cost;
+ Cost total_cost;
+
+ /* Get the ParamPathInfo */
+ param_info = get_baserel_parampathinfo(root, baserel, required_outer);
+
+ /* Redo the cost estimates */
+ estimate_costs(root, baserel,
+ fdw_private,
+ param_info->ppi_clauses,
+ &startup_cost, &total_cost);
+
+ /* Make and return the new path */
+ return create_foreignscan_path(root, baserel,
+ param_info->ppi_rows,
+ startup_cost,
+ total_cost,
+ NIL, /* no pathkeys */
+ required_outer,
+ path->fdw_private);
+}
+
+/* * fileGetForeignPlan * Create a ForeignScan plan node for scanning the foreign table */
@@ -962,12 +1004,13 @@ estimate_size(PlannerInfo *root, RelOptInfo *baserel, */static voidestimate_costs(PlannerInfo
*root,RelOptInfo *baserel,
- FileFdwPlanState *fdw_private,
+ FileFdwPlanState *fdw_private, List *join_conds, Cost *startup_cost, Cost *total_cost){
BlockNumber pages = fdw_private->pages; double ntuples = fdw_private->ntuples; Cost run_cost = 0;
+ QualCost join_cost; Cost cpu_per_tuple; /*
@@ -978,8 +1021,11 @@ estimate_costs(PlannerInfo *root, RelOptInfo *baserel, */ run_cost += seq_page_cost *
pages;
- *startup_cost = baserel->baserestrictcost.startup;
- cpu_per_tuple = cpu_tuple_cost * 10 + baserel->baserestrictcost.per_tuple;
+ cost_qual_eval(&join_cost, join_conds, root);
+ *startup_cost =
+ (baserel->baserestrictcost.startup + join_cost.startup);
+ cpu_per_tuple = cpu_tuple_cost * 10 +
+ (baserel->baserestrictcost.per_tuple + join_cost.per_tuple); run_cost += cpu_per_tuple * ntuples;
*total_cost= *startup_cost + run_cost;}
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 7dd43a9..3f7f7c2 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -239,6 +239,11 @@ static void postgresGetForeignRelSize(PlannerInfo *root,static void
postgresGetForeignPaths(PlannerInfo*root, RelOptInfo *baserel, Oid
foreigntableid);
+static ForeignPath *postgresReparameterizeForeignPath(PlannerInfo *root,
+ RelOptInfo *baserel,
+ Oid foreigntableid,
+ ForeignPath *path,
+ Relids required_outer);static ForeignScan
*postgresGetForeignPlan(PlannerInfo*root, RelOptInfo *baserel, Oid
foreigntableid,
@@ -340,6 +345,7 @@ postgres_fdw_handler(PG_FUNCTION_ARGS) /* Functions for scanning foreign tables */
routine->GetForeignRelSize= postgresGetForeignRelSize; routine->GetForeignPaths = postgresGetForeignPaths;
+ routine->ReparameterizeForeignPath = postgresReparameterizeForeignPath; routine->GetForeignPlan =
postgresGetForeignPlan; routine->BeginForeignScan = postgresBeginForeignScan; routine->IterateForeignScan =
postgresIterateForeignScan;
@@ -727,6 +733,48 @@ postgresGetForeignPaths(PlannerInfo *root,}/*
+ * postgresReparameterizeForeignPath
+ * Attempt to modify a given path to have greater parameterization
+ */
+static ForeignPath *
+postgresReparameterizeForeignPath(PlannerInfo *root,
+ RelOptInfo *baserel,
+ Oid foreigntableid,
+ ForeignPath *path,
+ Relids required_outer)
+{
+ ParamPathInfo *param_info;
+ double rows;
+ int width;
+ Cost startup_cost;
+ Cost total_cost;
+
+ /* Get the ParamPathInfo */
+ param_info = get_baserel_parampathinfo(root, baserel, required_outer);
+
+ /* Redo the cost estimates */
+ estimate_path_cost_size(root, baserel,
+ param_info->ppi_clauses,
+ &rows, &width,
+ &startup_cost, &total_cost);
+
+ /*
+ * ppi_rows currently won't get looked at by anything, but still we
+ * may as well ensure that it matches our idea of the rowcount.
+ */
+ param_info->ppi_rows = rows;
+
+ /* Make and return the new path */
+ return create_foreignscan_path(root, baserel,
+ rows,
+ startup_cost,
+ total_cost,
+ NIL, /* no pathkeys */
+ required_outer,
+ NIL); /* no fdw_private list */
+}
+
+/* * postgresGetForeignPlan * Create ForeignScan plan node which implements selected best path */
@@ -1773,11 +1821,8 @@ estimate_path_cost_size(PlannerInfo *root, } else {
- /*
- * We don't support join conditions in this mode (hence, no
- * parameterized paths can be made).
- */
- Assert(join_conds == NIL);
+ Selectivity join_sel;
+ QualCost join_cost; /* Use rows/width estimates made by set_baserel_size_estimates. */ rows =
baserel->rows;
@@ -1790,17 +1835,29 @@ estimate_path_cost_size(PlannerInfo *root, retrieved_rows = clamp_row_est(rows /
fpinfo->local_conds_sel); retrieved_rows = Min(retrieved_rows, baserel->tuples);
+ /* Factor in the selectivity of the join_conds */
+ join_sel = clauselist_selectivity(root,
+ join_conds,
+ baserel->relid,
+ JOIN_INNER,
+ NULL);
+
+ rows = clamp_row_est(rows * join_sel);
+ /* * Cost as though this were a seqscan, which is pessimistic. We
- * effectively imagine the local_conds are being evaluated remotely,
- * too.
+ * effectively imagine the local_conds and join_conds are being
+ * evaluated remotely, too. */ startup_cost = 0; run_cost = 0; run_cost +=
seq_page_cost* baserel->pages;
- startup_cost += baserel->baserestrictcost.startup;
- cpu_per_tuple = cpu_tuple_cost + baserel->baserestrictcost.per_tuple;
+ cost_qual_eval(&join_cost, join_conds, root);
+ startup_cost +=
+ (baserel->baserestrictcost.startup + join_cost.startup);
+ cpu_per_tuple = cpu_tuple_cost +
+ (baserel->baserestrictcost.per_tuple + join_cost.per_tuple); run_cost += cpu_per_tuple *
baserel->tuples; total_cost = startup_cost + run_cost;
diff --git a/doc/src/sgml/fdwhandler.sgml b/doc/src/sgml/fdwhandler.sgml
index 6b5c8b7..ec1492a 100644
--- a/doc/src/sgml/fdwhandler.sgml
+++ b/doc/src/sgml/fdwhandler.sgml
@@ -162,6 +162,37 @@ GetForeignPaths (PlannerInfo *root, <para><programlisting>
+ForeignPath *
+ReparameterizeForeignPath (PlannerInfo *root,
+ RelOptInfo *baserel,
+ Oid foreigntableid,
+ ForeignPath *path,
+ Relids required_outer);
+</programlisting>
+
+ Create an access path for a scan on a foreign table using join
+ clauses, which is called a <quote>parameterized path</>.
+ This is called during query planning.
+ The parameters are as for <function>GetForeignRelSize</>, plus
+ the <structname>ForeignPath</> (previously produced by
+ <function>GetForeignPaths</>), and the IDs of all other tables
+ that provide the join clauses.
+ </para>
+
+ <para>
+ This function must generate a parameterized path for a scan on the
+ foreign table. The parameterized path must contain cost estimates,
+ and can contain any FDW-private information that is needed to identify
+ the specific scan method intended. Unlike the other scan-related
+ functions, this function is optional.
+ </para>
+
+ <para>
+ See <xref linkend="fdw-planning"> for additional information.
+ </para>
+
+ <para>
+<programlisting>ForeignScan *GetForeignPlan (PlannerInfo *root, RelOptInfo *baserel,
@@ -808,10 +839,10 @@ GetForeignServerByName(const char *name, bool missing_ok); <para> The FDW callback
functions<function>GetForeignRelSize</>,
- <function>GetForeignPaths</>, <function>GetForeignPlan</>, and
- <function>PlanForeignModify</> must fit into the workings of the
- <productname>PostgreSQL</> planner. Here are some notes about what
- they must do.
+ <function>GetForeignPaths</>, <function>ReparameterizeForeignPath</>,
+ <function>GetForeignPlan</>, and <function>PlanForeignModify</> must fit
+ into the workings of the <productname>PostgreSQL</> planner. Here are
+ some notes about what they must do. </para> <para>
@@ -841,14 +872,17 @@ GetForeignServerByName(const char *name, bool missing_ok); to initialize it to NULL when the
<literal>baserel</>node is created. It is useful for passing information forward from
<function>GetForeignRelSize</>to <function>GetForeignPaths</> and/or
- <function>GetForeignPaths</> to <function>GetForeignPlan</>, thereby
- avoiding recalculation.
+ <function>GetForeignPaths</> to <function>ReparameterizeForeignPath</>
+ and/or
+ <function>GetForeignPaths</> or <function>ReparameterizeForeignPath</>
+ to <function>GetForeignPlan</>, thereby avoiding recalculation. </para> <para>
- <function>GetForeignPaths</> can identify the meaning of different
- access paths by storing private information in the
- <structfield>fdw_private</> field of <structname>ForeignPath</> nodes.
+ <function>GetForeignPaths</> or <function>ReparameterizeForeignPath</>
+ can identify the meaning of different access paths by storing private
+ information in the <structfield>fdw_private</> field of
+ <structname>ForeignPath</> nodes. <structfield>fdw_private</> is declared as a <type>List</> pointer, but
couldactually contain anything since the core planner does not touch it. However, best practice is to use a
representationthat's dumpable
@@ -891,10 +925,11 @@ GetForeignServerByName(const char *name, bool missing_ok); <replaceable>sub_expression</>,
whichit determines can be executed on the remote server given the locally-evaluated value of the
<replaceable>sub_expression</>. The actual identification of such a
- clause should happen during <function>GetForeignPaths</>, since it would
- affect the cost estimate for the path. The path's
- <structfield>fdw_private</> field would probably include a pointer to
- the identified clause's <structname>RestrictInfo</> node. Then
+ clause should happen during <function>GetForeignPaths</> or
+ <function>ReparameterizeForeignPath</>, since it would affect the cost
+ estimate for the path. The path's <structfield>fdw_private</> field
+ would probably include a pointer to the identified clause's
+ <structname>RestrictInfo</> node. Then <function>GetForeignPlan</> would remove that clause from
<literal>scan_clauses</>, but add the <replaceable>sub_expression</> to <structfield>fdw_exprs</> to ensure that
itgets massaged into executable form. It would probably
diff --git a/doc/src/sgml/ref/alter_foreign_table.sgml b/doc/src/sgml/ref/alter_foreign_table.sgml
index 4d8cfc5..10461d5 100644
--- a/doc/src/sgml/ref/alter_foreign_table.sgml
+++ b/doc/src/sgml/ref/alter_foreign_table.sgml
@@ -42,6 +42,10 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab ALTER [ COLUMN ]
<replaceableclass="PARAMETER">column_name</replaceable> SET ( <replaceable
class="PARAMETER">attribute_option</replaceable>= <replaceable class="PARAMETER">value</replaceable> [, ... ] )
ALTER[ COLUMN ] <replaceable class="PARAMETER">column_name</replaceable> RESET ( <replaceable
class="PARAMETER">attribute_option</replaceable>[, ... ] ) ALTER [ COLUMN ] <replaceable
class="PARAMETER">column_name</replaceable>OPTIONS ( [ ADD | SET | DROP ] <replaceable
class="PARAMETER">option</replaceable>['<replaceable class="PARAMETER">value</replaceable>'] [, ... ])
+ ADD <replaceable class="PARAMETER">table_constraint</replaceable>
+ DROP CONSTRAINT [ IF EXISTS ] <replaceable class="PARAMETER">constraint_name</replaceable> [ RESTRICT | CASCADE
]
+ INHERIT <replaceable class="PARAMETER">parent_table</replaceable>
+ NO INHERIT <replaceable class="PARAMETER">parent_table</replaceable> OWNER TO <replaceable
class="PARAMETER">new_owner</replaceable> OPTIONS ( [ ADD | SET | DROP ] <replaceable
class="PARAMETER">option</replaceable>['<replaceable class="PARAMETER">value</replaceable>'] [, ... ])</synopsis>
@@ -149,6 +153,50 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab </varlistentry>
<varlistentry>
+ <term><literal>ADD <replaceable class="PARAMETER">table_constraint</replaceable></literal></term>
+ <listitem>
+ <para>
+ This form adds a new constraint to a table using the same syntax as
+ <xref linkend="SQL-CREATEFOREIGNTABLE">.
+ Unlike the case when adding a constraint to a regular table, nothing happens
+ to the underlying storage: this action simply declares that
+ some new constraint holds for all rows in the foreign table.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>DROP CONSTRAINT [ IF EXISTS ]</literal></term>
+ <listitem>
+ <para>
+ This form drops the specified constraint on a table.
+ If <literal>IF EXISTS</literal> is specified and the constraint
+ does not exist, no error is thrown. In this case a notice is issued instead.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>INHERIT <replaceable class="PARAMETER">parent_table</replaceable></literal></term>
+ <listitem>
+ <para>
+ This form adds the target foreign table as a new child of the specified
+ parent table. The parent table must be an ordinary table.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>NO INHERIT <replaceable class="PARAMETER">parent_table</replaceable></literal></term>
+ <listitem>
+ <para>
+ This form removes the target foreign table from the list of children of
+ the specified parent table.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry> <term><literal>OWNER</literal></term> <listitem> <para>
@@ -270,6 +318,24 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab
</varlistentry> <varlistentry>
+ <term><replaceable class="PARAMETER">table_constraint</replaceable></term>
+ <listitem>
+ <para>
+ New table constraint for the table.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><replaceable class="PARAMETER">constraint_name</replaceable></term>
+ <listitem>
+ <para>
+ Name of an existing constraint to drop.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry> <term><literal>CASCADE</literal></term> <listitem> <para>
@@ -290,6 +356,16 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab
</varlistentry> <varlistentry>
+ <term><replaceable class="PARAMETER">parent_table</replaceable></term>
+ <listitem>
+ <para>
+ A parent table to associate or de-associate with this foreign table.
+ The parent table must be an ordinary table.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry> <term><replaceable class="PARAMETER">new_owner</replaceable></term> <listitem>
<para>
@@ -319,10 +395,10 @@ ALTER FOREIGN TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceab <para>
Consistencywith the foreign server is not checked when a column is added or removed with <literal>ADD
COLUMN</literal>or
- <literal>DROP COLUMN</literal>, a <literal>NOT NULL</> constraint is
- added, or a column type is changed with <literal>SET DATA TYPE</>. It is
- the user's responsibility to ensure that the table definition matches the
- remote side.
+ <literal>DROP COLUMN</literal>, a <literal>NOT NULL</> constraint or
+ <literal>CHECK</> constraint is added, or a column type is changed with
+ <literal>SET DATA TYPE</>. It is the user's responsibility to ensure that
+ the table definition matches the remote side. </para> <para>
diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml
index 69a1e14..eb1352e 100644
--- a/doc/src/sgml/ref/alter_table.sgml
+++ b/doc/src/sgml/ref/alter_table.sgml
@@ -967,6 +967,13 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceable> </para> <para>
+ If a table has any descendant tables that are foreign, a recursive
+ <literal>SET STORAGE</literal> operation will be rejected since it
+ is not permitted to add an <literal>oid</literal> system column to
+ foreign tables.
+ </para>
+
+ <para> The <literal>TRIGGER</>, <literal>CLUSTER</>, <literal>OWNER</>, and <literal>TABLESPACE</> actions
neverrecurse to descendant tables; that is, they always act as though <literal>ONLY</> were specified.
@@ -975,6 +982,19 @@ ALTER TABLE [ IF EXISTS ] <replaceable class="PARAMETER">name</replaceable> </para> <para>
+ When adding a <literal>CHECK</> constraint with the <literal>NOT VALID
+ </literal> option recursively, an inherited constraint on a descendant
+ table that is foreign will be marked valid without checking
+ consistency with the foreign server.
+ </para>
+
+ <para>
+ A recursive <literal>SET STORAGE</literal> operation will make the
+ storage mode for a descendant table's column unchanged if the table is
+ foreign.
+ </para>
+
+ <para> Changing any part of a system catalog table is not permitted. </para>
diff --git a/doc/src/sgml/ref/analyze.sgml b/doc/src/sgml/ref/analyze.sgml
index 08d316a..ec257c5 100644
--- a/doc/src/sgml/ref/analyze.sgml
+++ b/doc/src/sgml/ref/analyze.sgml
@@ -200,6 +200,13 @@ ANALYZE [ VERBOSE ] [ <replaceable class="PARAMETER">table_name</replaceable> [ </para> <para>
+ The inheritance statistics for a parent table that contains one or more
+ children that are foreign tables are collected only when explicitly
+ selected. If the foreign table's wrapper does not support
+ <command>ANALYZE</command>, the command prints a warning and does nothing.
+ </para>
+
+ <para> If the table being analyzed is completely empty, <command>ANALYZE</command> will not record new
statisticsfor that table. Any existing statistics will be retained.
diff --git a/doc/src/sgml/ref/create_foreign_table.sgml b/doc/src/sgml/ref/create_foreign_table.sgml
index 4a8cf38..1755b38 100644
--- a/doc/src/sgml/ref/create_foreign_table.sgml
+++ b/doc/src/sgml/ref/create_foreign_table.sgml
@@ -19,9 +19,11 @@ <refsynopsisdiv><synopsis>CREATE FOREIGN TABLE [ IF NOT EXISTS ] <replaceable
class="PARAMETER">table_name</replaceable>( [
- <replaceable class="PARAMETER">column_name</replaceable> <replaceable class="PARAMETER">data_type</replaceable> [
OPTIONS( <replaceable class="PARAMETER">option</replaceable> '<replaceable class="PARAMETER">value</replaceable>' [,
...] ) ] [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable
class="PARAMETER">column_constraint</replaceable>[ ... ] ]
+ { <replaceable class="PARAMETER">column_name</replaceable> <replaceable class="PARAMETER">data_type</replaceable> [
OPTIONS( <replaceable class="PARAMETER">option</replaceable> '<replaceable class="PARAMETER">value</replaceable>' [,
...] ) ] [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable
class="PARAMETER">column_constraint</replaceable>[ ... ] ]
+ | <replaceable>table_constraint</replaceable> } [, ... ]] )
+[ INHERITS ( <replaceable>parent_table</replaceable> [, ... ] ) ] SERVER <replaceable
class="parameter">server_name</replaceable>[OPTIONS ( <replaceable class="PARAMETER">option</replaceable> '<replaceable
class="PARAMETER">value</replaceable>'[, ... ] ) ]
@@ -30,7 +32,13 @@ CREATE FOREIGN TABLE [ IF NOT EXISTS ] <replaceable class="PARAMETER">table_name[ CONSTRAINT
<replaceableclass="PARAMETER">constraint_name</replaceable> ]{ NOT NULL | NULL |
+ CHECK ( <replaceable class="PARAMETER">expression</replaceable> ) | DEFAULT <replaceable>default_expr</replaceable>
}
+
+<phrase>and <replaceable class="PARAMETER">table_constraint</replaceable> is:</phrase>
+
+[ CONSTRAINT <replaceable class="PARAMETER">constraint_name</replaceable> ]
+{ CHECK ( <replaceable class="PARAMETER">expression</replaceable> ) }</synopsis> </refsynopsisdiv>
@@ -138,6 +146,27 @@ CREATE FOREIGN TABLE [ IF NOT EXISTS ] <replaceable class="PARAMETER">table_name </varlistentry>
<varlistentry>
+ <term><literal>CHECK ( <replaceable class="PARAMETER">expression</replaceable> )</literal></term>
+ <listitem>
+ <para>
+ The <literal>CHECK</> clause specifies an expression producing a
+ Boolean result which each row must satisfy.
+ Expressions evaluating to TRUE or UNKNOWN succeed.
+ A check constraint specified as a column constraint should
+ reference that column's value only, while an expression
+ appearing in a table constraint can reference multiple columns.
+ </para>
+
+ <para>
+ Currently, <literal>CHECK</literal> expressions cannot contain
+ subqueries nor refer to variables other than columns of the
+ current row. The system column <literal>tableoid</literal>
+ may be referenced, but not any other system column.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry> <term><literal>DEFAULT <replaceable>default_expr</replaceable></literal></term> <listitem>
@@ -159,6 +188,18 @@ CREATE FOREIGN TABLE [ IF NOT EXISTS ] <replaceable class="PARAMETER">table_name </varlistentry>
<varlistentry>
+ <term><replaceable class="PARAMETER">parent_table</replaceable></term>
+ <listitem>
+ <para>
+ The name of an existing table from which the new foreign table
+ automatically inherits all columns. The specified parent table
+ must be an ordinary table. See <xref linkend="ddl-inherit"> for the
+ details of table inheritance.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry> <term><replaceable class="PARAMETER">server_name</replaceable></term> <listitem> <para>
@@ -187,6 +228,24 @@ CREATE FOREIGN TABLE [ IF NOT EXISTS ] <replaceable class="PARAMETER">table_name </refsect1>
+ <refsect1>
+ <title>Notes</title>
+
+ <para>
+ Constraints on foreign tables are not enforced on insert or update.
+ Those definitions simply declare the constraints hold for all rows
+ in the foreign tables. It is the user's responsibility to ensure
+ that those definitions match the remote side. Such constraints are
+ used for some kind of query optimization such as constraint exclusion
+ for partitioned tables (see <xref linkend="ddl-partitioning">).
+ </para>
+
+ <para>
+ Since it is not permitted to add an <literal>oid</> system column to
+ foreign tables, the command will be rejected if any parent tables
+ have <literal>oid</> system columns.
+ </para>
+ </refsect1> <refsect1 id="SQL-CREATEFOREIGNTABLE-examples"> <title>Examples</title>
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 33eef9f..21d32c5 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -2217,6 +2217,12 @@ AddRelationNewConstraints(Relation rel, if (cdef->contype != CONSTR_CHECK)
continue;
+ if (cdef->is_no_inherit &&
+ rel->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("CHECK constraints on foreign tables cannot be marked NO INHERIT")));
+ if (cdef->raw_expr != NULL) { Assert(cdef->cooked_expr == NULL);
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index c09ca7e..f67273c 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -82,6 +82,7 @@ int default_statistics_target = 100;/* A few variables that don't seem worth passing
aroundas parameters */static MemoryContext anl_context = NULL;
+static VacuumMode vac_mode;static BufferAccessStrategy vac_strategy;
@@ -115,7 +116,10 @@ static Datum ind_fetch_func(VacAttrStatsP stats, int rownum, bool *isNull); * analyze_rel() --
analyzeone relation */void
-analyze_rel(Oid relid, VacuumStmt *vacstmt, BufferAccessStrategy bstrategy)
+analyze_rel(Oid relid,
+ VacuumStmt *vacstmt,
+ VacuumMode vacmode,
+ BufferAccessStrategy bstrategy){ Relation onerel; int elevel;
@@ -129,6 +133,7 @@ analyze_rel(Oid relid, VacuumStmt *vacstmt, BufferAccessStrategy bstrategy) elevel = DEBUG2;
/* Set up static variables */
+ vac_mode = vacmode; vac_strategy = bstrategy; /*
@@ -1452,6 +1457,7 @@ acquire_inherited_sample_rows(Relation onerel, int elevel,{ List *tableOIDs; Relation
*rels;
+ AcquireSampleRowsFunc *acquirefunc; double *relblocks; double totalblocks; int
numrows,
@@ -1486,6 +1492,8 @@ acquire_inherited_sample_rows(Relation onerel, int elevel, * BlockNumber, so we use double
arithmetic. */ rels = (Relation *) palloc(list_length(tableOIDs) * sizeof(Relation));
+ acquirefunc = (AcquireSampleRowsFunc *) palloc(list_length(tableOIDs)
+ * sizeof(AcquireSampleRowsFunc)); relblocks = (double *)
palloc(list_length(tableOIDs)* sizeof(double)); totalblocks = 0; nrels = 0;
@@ -1507,7 +1515,40 @@ acquire_inherited_sample_rows(Relation onerel, int elevel, } rels[nrels] =
childrel;
- relblocks[nrels] = (double) RelationGetNumberOfBlocks(childrel);
+
+ if (childrel->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
+ {
+ FdwRoutine *fdwroutine;
+ BlockNumber relpages = 0;
+ bool ok = false;
+
+ /* Ignore unless analyzing a specific inheritance tree */
+ if (vac_mode != VAC_MODE_SINGLE)
+ return 0;
+
+ /* Check whether the FDW supports analysis */
+ fdwroutine = GetFdwRoutineForRelation(childrel, false);
+ if (fdwroutine->AnalyzeForeignTable != NULL)
+ ok = fdwroutine->AnalyzeForeignTable(childrel,
+ &acquirefunc[nrels],
+ &relpages);
+ if (!ok)
+ {
+ /* Give up if the FDW doesn't support analysis */
+ ereport(WARNING,
+ (errmsg("skipping analyze of inheritance tree \"%s\" --- cannot analyze foreign table
\"%s\"",
+ RelationGetRelationName(onerel),
+ RelationGetRelationName(childrel))));
+ return 0;
+ }
+ relblocks[nrels] = (double) relpages;
+ }
+ else
+ {
+ acquirefunc[nrels] = acquire_sample_rows;
+ relblocks[nrels] = (double) RelationGetNumberOfBlocks(childrel);
+ }
+ totalblocks += relblocks[nrels]; nrels++; }
@@ -1525,6 +1566,7 @@ acquire_inherited_sample_rows(Relation onerel, int elevel, { Relation childrel =
rels[i]; double childblocks = relblocks[i];
+ AcquireSampleRowsFunc childacquirefunc = acquirefunc[i]; if (childblocks > 0) {
@@ -1540,12 +1582,12 @@ acquire_inherited_sample_rows(Relation onerel, int elevel, tdrows;
/* Fetch a random sample of the child's rows */
- childrows = acquire_sample_rows(childrel,
- elevel,
- rows + numrows,
- childtargrows,
- &trows,
- &tdrows);
+ childrows = childacquirefunc(childrel,
+ elevel,
+ rows + numrows,
+ childtargrows,
+ &trows,
+ &tdrows); /* We may need to convert from child's rowtype
toparent's */ if (childrows > 0 &&
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 341262b..9e09906 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -310,7 +310,8 @@ static AlteredTableInfo *ATGetQueueEntry(List **wqueue, Relation rel);static void
ATSimplePermissions(Relationrel, int allowed_targets);static void ATWrongRelkindError(Relation rel, int
allowed_targets);staticvoid ATSimpleRecursion(List **wqueue, Relation rel,
- AlterTableCmd *cmd, bool recurse, LOCKMODE lockmode);
+ AlterTableCmd *cmd, bool recurse,
+ bool include_foreign, LOCKMODE lockmode);static void ATTypedTableRecursion(List **wqueue, Relation
rel,AlterTableCmd *cmd, LOCKMODE lockmode);static List *find_typed_table_dependencies(Oid typeOid,
constchar *typeName,
@@ -466,10 +467,6 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId) ereport(ERROR,
(errcode(ERRCODE_INVALID_TABLE_DEFINITION), errmsg("ON COMMIT can only be used on temporary tables")));
- if (stmt->constraints != NIL && relkind == RELKIND_FOREIGN_TABLE)
- ereport(ERROR,
- (errcode(ERRCODE_WRONG_OBJECT_TYPE),
- errmsg("constraints are not supported on foreign tables"))); /* * Look up the namespace in
whichwe are supposed to create the relation,
@@ -556,6 +553,30 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId)
stmt->relation->relpersistence, &inheritOids, &old_constraints, &parentOidCount);
+ if (relkind == RELKIND_FOREIGN_TABLE)
+ {
+ /*
+ * Don't allow a foreign table to inherit from parents that have OID
+ * system columns.
+ */
+ if (parentOidCount > 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot inherit from relation with OIDs")));
+
+ /*
+ * Reset the storage parameter for inherited attributes that have
+ * non-default values.
+ */
+ foreach(listptr, schema)
+ {
+ ColumnDef *colDef = lfirst(listptr);
+
+ if (colDef->storage != 0)
+ colDef->storage = 0;
+ }
+ }
+ /* * Create a tuple descriptor from the relation schema. Note that this * deals with column names, types,
andNOT NULL constraints, but not
@@ -3065,24 +3086,28 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, * rules. */
ATSimplePermissions(rel, ATT_TABLE | ATT_VIEW | ATT_FOREIGN_TABLE);
- ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+ /* Recurse to child tables that are foreign, too */
+ ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode); /* No command-specific prep
needed*/ pass = cmd->def ? AT_PASS_ADD_CONSTR : AT_PASS_DROP; break; case AT_DropNotNull:
/* ALTER COLUMN DROP NOT NULL */ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
- ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+ /* Recurse to child tables that are foreign, too */
+ ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode); /* No command-specific prep
needed*/ pass = AT_PASS_DROP; break; case AT_SetNotNull: /* ALTER COLUMN SET NOT
NULL*/ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
- ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+ /* Recurse to child tables that are foreign, too */
+ ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode); /* No command-specific prep
needed*/ pass = AT_PASS_ADD_CONSTR; break; case AT_SetStatistics: /* ALTER COLUMN SET
STATISTICS*/
- ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+ /* Recurse to child tables that are foreign, too */
+ ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode); /* Performs own permission checks
*/ ATPrepSetStatistics(rel, cmd->name, cmd->def, lockmode); pass = AT_PASS_MISC;
@@ -3095,7 +3120,8 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, break; case
AT_SetStorage: /* ALTER COLUMN SET STORAGE */ ATSimplePermissions(rel, ATT_TABLE | ATT_MATVIEW);
- ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+ /* Don't recurse to child tables that are foreign */
+ ATSimpleRecursion(wqueue, rel, cmd, recurse, false, lockmode); /* No command-specific prep
needed*/ pass = AT_PASS_MISC; break;
@@ -3113,7 +3139,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, pass = AT_PASS_ADD_INDEX;
break; case AT_AddConstraint: /* ADD CONSTRAINT */
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE); /* Recursion occurs during execution
phase*/ /* No command-specific prep needed except saving recurse flag */ if (recurse)
@@ -3127,7 +3153,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, pass = AT_PASS_ADD_CONSTR;
break; case AT_DropConstraint: /* DROP CONSTRAINT */
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE); /* Recursion occurs during execution
phase*/ /* No command-specific prep needed except saving recurse flag */ if (recurse)
@@ -3195,11 +3221,17 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, pass = AT_PASS_MISC;
break; case AT_AddInherit: /* INHERIT */
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE); /* This command never recurses */
ATPrepAddInherit(rel); pass = AT_PASS_MISC; break;
+ case AT_DropInherit: /* NO INHERIT */
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
+ /* This command never recurses */
+ /* No command-specific prep needed */
+ pass = AT_PASS_MISC;
+ break; case AT_AlterConstraint: /* ALTER CONSTRAINT */ ATSimplePermissions(rel,
ATT_TABLE); pass = AT_PASS_MISC;
@@ -3233,7 +3265,6 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, case AT_EnableAlwaysRule:
case AT_EnableReplicaRule: case AT_DisableRule:
- case AT_DropInherit: /* NO INHERIT */ case AT_AddOf: /* OF */ case AT_DropOf: /*
NOTOF */ ATSimplePermissions(rel, ATT_TABLE);
@@ -4174,7 +4205,8 @@ ATWrongRelkindError(Relation rel, int allowed_targets) */static voidATSimpleRecursion(List
**wqueue,Relation rel,
- AlterTableCmd *cmd, bool recurse, LOCKMODE lockmode)
+ AlterTableCmd *cmd, bool recurse,
+ bool include_foreign, LOCKMODE lockmode){ /* * Propagate to children if desired. Non-table
relationsnever have
@@ -4202,8 +4234,12 @@ ATSimpleRecursion(List **wqueue, Relation rel, continue; /*
find_all_inheritorsalready got lock */ childrel = relation_open(childrelid, NoLock);
- CheckTableNotInUse(childrel, "ALTER TABLE");
- ATPrepCmd(wqueue, childrel, cmd, false, true, lockmode);
+ if (childrel->rd_rel->relkind != RELKIND_FOREIGN_TABLE
+ || include_foreign)
+ {
+ CheckTableNotInUse(childrel, "ALTER TABLE");
+ ATPrepCmd(wqueue, childrel, cmd, false, true, lockmode);
+ } relation_close(childrel, NoLock); } }
@@ -4493,7 +4529,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, /* At top level,
permissioncheck was done in ATPrepCmd, else do it */ if (recursing)
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE); attrdesc = heap_open(AttributeRelationId,
RowExclusiveLock);
@@ -4789,6 +4825,11 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, /*
find_inheritance_childrenalready got lock */ childrel = heap_open(childrelid, NoLock);
+ if (childrel->rd_rel->relkind == RELKIND_FOREIGN_TABLE && isOid)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot add OID column to foreign table \"%s\"",
+ RelationGetRelationName(childrel)))); CheckTableNotInUse(childrel, "ALTER TABLE");
/* Find or create work queue entry for this table */
@@ -5389,7 +5430,7 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName, /* At top level, permission
checkwas done in ATPrepCmd, else do it */ if (recursing)
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE); /* * get the number of the attribute
@@ -5781,7 +5822,14 @@ ATAddCheckConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel, /* At top level,
permissioncheck was done in ATPrepCmd, else do it */ if (recursing)
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE);
+
+ /* Don't allow ADD CONSTRAINT NOT VALID to be applied to foreign tables */
+ if (tab->relkind == RELKIND_FOREIGN_TABLE &&
+ constr->skip_validation && !recursing)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("NOT VALID is not supported on foreign tables"))); /* * Call AddRelationNewConstraints
todo the work, making sure it works on
@@ -5792,9 +5840,17 @@ ATAddCheckConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel, * omitted from the
returnedlist, which is what we want: we do not need * to do any validation work. That can only happen at child
tables, * though, since we disallow merging at the top level.
- */
+ *
+ * When propagating a NOT VALID option to children that are foreign tables,
+ * we quietly ignore the option. Note that this is safe because foreign
+ * tables don't have any children.
+ */
+ constr = (Constraint *) copyObject(constr);
+ if (tab->relkind == RELKIND_FOREIGN_TABLE &&
+ constr->skip_validation && recursing)
+ constr->skip_validation = false; newcons = AddRelationNewConstraints(rel, NIL,
- list_make1(copyObject(constr)),
+ list_make1(constr), recursing,
/*allow_merge */ !recursing, /* is_local */
is_readd); /* is_internal */
@@ -7274,7 +7330,7 @@ ATExecDropConstraint(Relation rel, const char *constrName, /* At top level, permission check
wasdone in ATPrepCmd, else do it */ if (recursing)
- ATSimplePermissions(rel, ATT_TABLE);
+ ATSimplePermissions(rel, ATT_TABLE | ATT_FOREIGN_TABLE); conrel = heap_open(ConstraintRelationId,
RowExclusiveLock);
@@ -7609,7 +7665,10 @@ ATPrepAlterColumnType(List **wqueue, * alter would put them out of step. */ if
(recurse)
- ATSimpleRecursion(wqueue, rel, cmd, recurse, lockmode);
+ {
+ /* Recurse to child tables that are foreign, too */
+ ATSimpleRecursion(wqueue, rel, cmd, recurse, true, lockmode);
+ } else if (!recursing && find_inheritance_children(RelationGetRelid(rel), NoLock) != NIL)
ereport(ERROR,
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 3d2c739..b5e3ccf 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -61,6 +61,7 @@ int vacuum_multixact_freeze_table_age;/* A few variables that don't seem worth passing
aroundas parameters */static MemoryContext vac_context = NULL;
+static VacuumMode vac_mode;static BufferAccessStrategy vac_strategy;
@@ -146,6 +147,20 @@ vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast,
ALLOCSET_DEFAULT_MAXSIZE); /*
+ * Identify vacuum mode. If relid is not InvalidOid, the caller should be
+ * an autovacuum worker. See the above comments.
+ */
+ if (relid != InvalidOid)
+ vac_mode = VAC_MODE_AUTOVACUUM;
+ else
+ {
+ if (!vacstmt->relation)
+ vac_mode = VAC_MODE_ALL;
+ else
+ vac_mode = VAC_MODE_SINGLE;
+ }
+
+ /* * If caller didn't give us a buffer strategy object, make one in the * cross-transaction memory
context. */
@@ -248,7 +263,7 @@ vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast,
PushActiveSnapshot(GetTransactionSnapshot()); }
- analyze_rel(relid, vacstmt, vac_strategy);
+ analyze_rel(relid, vacstmt, vac_mode, vac_strategy); if (use_own_xacts)
{
diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c
index 0410fdd..836fae6 100644
--- a/src/backend/optimizer/prep/prepunion.c
+++ b/src/backend/optimizer/prep/prepunion.c
@@ -1338,11 +1338,12 @@ expand_inherited_rtentry(PlannerInfo *root, RangeTblEntry *rte, Index rti) /* *
Buildan RTE for the child, and attach to query's rangetable list. * We copy most fields of the parent's RTE,
butreplace relation OID,
- * and set inh = false. Also, set requiredPerms to zero since all
- * required permissions checks are done on the original RTE.
+ * relkind and set inh = false. Also, set requiredPerms to zero since
+ * all required permissions checks are done on the original RTE. */ childrte = copyObject(rte);
childrte->relid = childOID;
+ childrte->relkind = newrelation->rd_rel->relkind; childrte->inh = false; childrte->requiredPerms
=0; parse->rtable = lappend(parse->rtable, childrte);
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 4e05dcd..880595f 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -16,6 +16,7 @@#include <math.h>
+#include "foreign/fdwapi.h"#include "miscadmin.h"#include "nodes/nodeFuncs.h"#include "optimizer/clauses.h"
@@ -2062,6 +2063,31 @@ reparameterize_path(PlannerInfo *root, Path *path, case T_SubqueryScan: return
create_subqueryscan_path(root,rel, path->pathkeys, required_outer);
+ case T_ForeignScan:
+ {
+ ForeignPath *newpath = NULL;
+
+ /* Let the FDW reparameterize the path node if possible */
+ if (rel->fdwroutine->ReparameterizeForeignPath != NULL)
+ {
+ Index scan_relid = rel->relid;
+ RangeTblEntry *rte;
+
+ /* it should be a base rel... */
+ Assert(scan_relid > 0);
+ Assert(rel->rtekind == RTE_RELATION);
+ rte = planner_rt_fetch(scan_relid, root);
+ Assert(rte->rtekind == RTE_RELATION);
+
+ newpath =
+ rel->fdwroutine->ReparameterizeForeignPath(root,
+ rel,
+ rte->relid,
+ (ForeignPath *) path,
+ required_outer);
+ }
+ return (Path *) newpath;
+ } default: break; }
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 605c9b4..a206501 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -4213,32 +4213,32 @@ AlterForeignServerStmt: ALTER SERVER name foreign_server_version
alter_generic_oCreateForeignTableStmt: CREATE FOREIGN TABLE qualified_name '(' OptTableElementList
')'
- SERVER name create_generic_options
+ OptInherit SERVER name create_generic_options { CreateForeignTableStmt
*n= makeNode(CreateForeignTableStmt); $4->relpersistence = RELPERSISTENCE_PERMANENT;
n->base.relation = $4; n->base.tableElts = $6;
- n->base.inhRelations = NIL;
+ n->base.inhRelations = $8; n->base.if_not_exists = false; /*
FDW-specificdata */
- n->servername = $9;
- n->options = $10;
+ n->servername = $10;
+ n->options = $11; $$ = (Node *) n; } | CREATE FOREIGN
TABLEIF_P NOT EXISTS qualified_name '(' OptTableElementList ')'
- SERVER name create_generic_options
+ OptInherit SERVER name create_generic_options { CreateForeignTableStmt
*n= makeNode(CreateForeignTableStmt); $7->relpersistence = RELPERSISTENCE_PERMANENT;
n->base.relation = $7; n->base.tableElts = $9;
- n->base.inhRelations = NIL;
+ n->base.inhRelations = $11; n->base.if_not_exists = true; /*
FDW-specificdata */
- n->servername = $12;
- n->options = $13;
+ n->servername = $13;
+ n->options = $14; $$ = (Node *) n; } ;
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index 7c1939f..949392a 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -515,12 +515,6 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column) break;
case CONSTR_CHECK:
- if (cxt->isforeign)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("constraints are not supported on foreign tables"),
- parser_errposition(cxt->pstate,
- constraint->location))); cxt->ckconstraints =
lappend(cxt->ckconstraints,constraint); break;
@@ -529,7 +523,7 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column) if
(cxt->isforeign) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("constraints are not supported on foreign tables"),
+ errmsg("primary key or unique constraints are not supported on foreign tables"),
parser_errposition(cxt->pstate, constraint->location)));
if (constraint->keys == NIL)
@@ -546,7 +540,7 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column) if
(cxt->isforeign) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("constraints are not supported on foreign tables"),
+ errmsg("foreign key constraints are not supported on foreign tables"),
parser_errposition(cxt->pstate, constraint->location)));
@@ -605,10 +599,14 @@ transformColumnDefinition(CreateStmtContext *cxt, ColumnDef *column)static
voidtransformTableConstraint(CreateStmtContext*cxt, Constraint *constraint){
- if (cxt->isforeign)
+ if (cxt->isforeign &&
+ (constraint->contype == CONSTR_PRIMARY ||
+ constraint->contype == CONSTR_UNIQUE ||
+ constraint->contype == CONSTR_EXCLUSION ||
+ constraint->contype == CONSTR_FOREIGN)) ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("constraints are not supported on foreign tables"),
+ errmsg("primary key, unique, exclusion, or foreign key constraints are not supported on foreign
tables"), parser_errposition(cxt->pstate, constraint->location)));
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index d33552a..400e373 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -140,6 +140,15 @@ extern int vacuum_multixact_freeze_min_age;extern int vacuum_multixact_freeze_table_age;
+/* Possible modes for vacuum() */
+typedef enum
+{
+ VAC_MODE_ALL, /* Vacuum/analyze all relations */
+ VAC_MODE_SINGLE, /* Vacuum/analyze a specific relation */
+ VAC_MODE_AUTOVACUUM /* Autovacuum worker */
+} VacuumMode;
+
+/* in commands/vacuum.c */extern void vacuum(VacuumStmt *vacstmt, Oid relid, bool do_toast, BufferAccessStrategy
bstrategy,bool for_wraparound, bool isTopLevel);
@@ -174,7 +183,9 @@ extern void lazy_vacuum_rel(Relation onerel, VacuumStmt *vacstmt,
BufferAccessStrategybstrategy);/* in commands/analyze.c */
-extern void analyze_rel(Oid relid, VacuumStmt *vacstmt,
+extern void analyze_rel(Oid relid,
+ VacuumStmt *vacstmt,
+ VacuumMode vacmode, BufferAccessStrategy bstrategy);extern bool std_typanalyze(VacAttrStats
*stats);externdouble anl_random_fract(void);
diff --git a/src/include/foreign/fdwapi.h b/src/include/foreign/fdwapi.h
index 1b735da..5f3996a 100644
--- a/src/include/foreign/fdwapi.h
+++ b/src/include/foreign/fdwapi.h
@@ -31,6 +31,12 @@ typedef void (*GetForeignPaths_function) (PlannerInfo *root,
RelOptInfo *baserel, Oid foreigntableid);
+typedef ForeignPath *(*ReparameterizeForeignPath_function) (PlannerInfo *root,
+ RelOptInfo *baserel,
+ Oid foreigntableid,
+ ForeignPath *path,
+ Relids required_outer);
+typedef ForeignScan *(*GetForeignPlan_function) (PlannerInfo *root,
RelOptInfo *baserel, Oid foreigntableid,
@@ -117,6 +123,7 @@ typedef struct FdwRoutine /* Functions for scanning foreign tables */
GetForeignRelSize_functionGetForeignRelSize; GetForeignPaths_function GetForeignPaths;
+ ReparameterizeForeignPath_function ReparameterizeForeignPath; GetForeignPlan_function GetForeignPlan;
BeginForeignScan_functionBeginForeignScan; IterateForeignScan_function IterateForeignScan;
diff --git a/src/test/regress/expected/foreign_data.out b/src/test/regress/expected/foreign_data.out
index ff203b2..36df338 100644
--- a/src/test/regress/expected/foreign_data.out
+++ b/src/test/regress/expected/foreign_data.out
@@ -748,16 +748,12 @@ CREATE TABLE use_ft1_column_type (x ft1);ALTER FOREIGN TABLE ft1 ALTER COLUMN c8 SET DATA TYPE
integer; -- ERRORERROR: cannot alter foreign table "ft1" because column "use_ft1_column_type.x" uses its row
typeDROPTABLE use_ft1_column_type;
-ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0); -- ERROR
-ERROR: constraints are not supported on foreign tables
-LINE 1: ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c...
- ^
+ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0);ALTER FOREIGN TABLE ft1 DROP CONSTRAINT no_const;
-- ERROR
-ERROR: "ft1" is not a table
+ERROR: constraint "no_const" of relation "ft1" does not existALTER FOREIGN TABLE ft1 DROP CONSTRAINT IF EXISTS
no_const;
-ERROR: "ft1" is not a table
-ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c1_check;
-ERROR: "ft1" is not a table
+NOTICE: constraint "no_const" of relation "ft1" does not exist, skipping
+ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c9_check;ALTER FOREIGN TABLE ft1 SET WITH OIDS;
--ERRORERROR: "ft1" is not a tableALTER FOREIGN TABLE ft1 OWNER TO regress_test_role;
diff --git a/src/test/regress/sql/foreign_data.sql b/src/test/regress/sql/foreign_data.sql
index 0f0869e..8f9ea86 100644
--- a/src/test/regress/sql/foreign_data.sql
+++ b/src/test/regress/sql/foreign_data.sql
@@ -314,10 +314,10 @@ ALTER FOREIGN TABLE ft1 ALTER COLUMN c8 SET STATISTICS -1;CREATE TABLE use_ft1_column_type (x
ft1);ALTERFOREIGN TABLE ft1 ALTER COLUMN c8 SET DATA TYPE integer; -- ERRORDROP TABLE use_ft1_column_type;
-ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0); -- ERROR
+ALTER FOREIGN TABLE ft1 ADD CONSTRAINT ft1_c9_check CHECK (c9 < 0);ALTER FOREIGN TABLE ft1 DROP CONSTRAINT no_const;
-- ERRORALTER FOREIGN TABLE ft1 DROP CONSTRAINT IF EXISTS no_const;
-ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c1_check;
+ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c9_check;ALTER FOREIGN TABLE ft1 SET WITH OIDS;
--ERRORALTER FOREIGN TABLE ft1 OWNER TO regress_test_role;ALTER FOREIGN TABLE ft1 OPTIONS (DROP delimiter, SET quote
'~',ADD escape '@');
Hi,
Selecting tableoid on parent causes an error, "ERROR: cannot extract system attribute from virtual tuple". The foreign table has an OID which can be reported as tableoid for the rows coming from that foreign table. Do we want to do that?On Fri, Jun 20, 2014 at 1:34 PM, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
Hello,
Before continueing discussion, I tried this patch on the current
head.
This patch applies cleanly except one hunk because of a
modification just AFTER it, which did no harm. Finally all
regression tests suceeded.
Attached is the rebased patch of v11 up to the current master.
regards,
--
Kyotaro Horiguchi
NTT Open Source Software Center
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Hi Ashutosh, Thank you for the review. (2014/06/23 18:35), Ashutosh Bapat wrote: > Hi, > Selecting tableoid on parent causes an error, "ERROR: cannot extract > system attribute from virtual tuple". The foreign table has an OID which > can be reported as tableoid for the rows coming from that foreign table. > Do we want to do that? No. I think it's a bug. I'll fix it. Thanks, Best regards, Etsuro Fujita
(2014/06/24 16:30), Etsuro Fujita wrote: > (2014/06/23 18:35), Ashutosh Bapat wrote: >> Selecting tableoid on parent causes an error, "ERROR: cannot extract >> system attribute from virtual tuple". The foreign table has an OID which >> can be reported as tableoid for the rows coming from that foreign table. >> Do we want to do that? > > No. I think it's a bug. I'll fix it. Done. I think this is because create_foreignscan_plan() makes reference to attr_needed, which isn't computed for inheritance children. To aboid this, I've modified create_foreignscan_plan() to see reltargetlist and baserestrictinfo, instead of attr_needed. Please find attached an updated version of the patch. Sorry for the delay. Best regards, Etsuro Fujita
Attachment
I checked that it's reporting the right tableoid now.
BTW, why aren't you using the tlist passed to this function? I guess create_scan_plan() passes tlist after processing it, so that should be used rather than rel->reltargetlist.On Mon, Jun 30, 2014 at 12:22 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/06/24 16:30), Etsuro Fujita wrote:(2014/06/23 18:35), Ashutosh Bapat wrote:Done. I think this is because create_foreignscan_plan() makes reference to attr_needed, which isn't computed for inheritance children. To aboid this, I've modified create_foreignscan_plan() to see reltargetlist and baserestrictinfo, instead of attr_needed. Please find attached an updated version of the patch.Selecting tableoid on parent causes an error, "ERROR: cannot extract
system attribute from virtual tuple". The foreign table has an OID which
can be reported as tableoid for the rows coming from that foreign table.
Do we want to do that?
No. I think it's a bug. I'll fix it.
Sorry for the delay.
Best regards,
Etsuro Fujita
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
(2014/06/30 17:47), Ashutosh Bapat wrote: > I checked that it's reporting the right tableoid now. Thank you for the check. > BTW, why aren't you using the tlist passed to this function? I guess > create_scan_plan() passes tlist after processing it, so that should be > used rather than rel->reltargetlist. I think that that would be maybe OK, but I think that it would not be efficient to use the list to compute attrs_used, because the tlist would have more information than rel->reltargetlist in cases where the tlist is build through build_physical_tlist(). Thanks, Best regards, Etsuro Fujita > > On Mon, Jun 30, 2014 at 12:22 PM, Etsuro Fujita > <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote: > > (2014/06/24 16:30), Etsuro Fujita wrote: > > (2014/06/23 18:35), Ashutosh Bapat wrote: > > > Selecting tableoid on parent causes an error, "ERROR: > cannot extract > system attribute from virtual tuple". The foreign table has > an OID which > can be reported as tableoid for the rows coming from that > foreign table. > Do we want to do that? > > > No. I think it's a bug. I'll fix it. > > > Done. I think this is because create_foreignscan_plan() makes > reference to attr_needed, which isn't computed for inheritance > children. To aboid this, I've modified create_foreignscan_plan() to > see reltargetlist and baserestrictinfo, instead of attr_needed. > Please find attached an updated version of the patch. > > > Sorry for the delay. > > Best regards, > Etsuro Fujita > > > > > -- > Best Wishes, > Ashutosh Bapat > EnterpriseDB Corporation > The Postgres Database Company
On Mon, Jun 30, 2014 at 4:17 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
In that case, we can call build_relation_tlist() for foreign tables.
(2014/06/30 17:47), Ashutosh Bapat wrote:Thank you for the check.I checked that it's reporting the right tableoid now.I think that that would be maybe OK, but I think that it would not be efficient to use the list to compute attrs_used, because the tlist would have more information than rel->reltargetlist in cases where the tlist is build through build_physical_tlist().BTW, why aren't you using the tlist passed to this function? I guess
create_scan_plan() passes tlist after processing it, so that should be
used rather than rel->reltargetlist.
In that case, we can call build_relation_tlist() for foreign tables.
Thanks,
Best regards,
Etsuro FujitaOn Mon, Jun 30, 2014 at 12:22 PM, Etsuro Fujita<fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:
(2014/06/24 16:30), Etsuro Fujita wrote:
(2014/06/23 18:35), Ashutosh Bapat wrote:
Selecting tableoid on parent causes an error, "ERROR:
cannot extract
system attribute from virtual tuple". The foreign table has
an OID which
can be reported as tableoid for the rows coming from that
foreign table.
Do we want to do that?
No. I think it's a bug. I'll fix it.
Done. I think this is because create_foreignscan_plan() makes
reference to attr_needed, which isn't computed for inheritance
children. To aboid this, I've modified create_foreignscan_plan() to
see reltargetlist and baserestrictinfo, instead of attr_needed.
Please find attached an updated version of the patch.
Sorry for the delay.
Best regards,
Etsuro Fujita
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
> Done. I think this is because create_foreignscan_plan() makes reference
> to attr_needed, which isn't computed for inheritance children.
I wonder whether it isn't time to change that. It was coded like that
originally only because calculating the values would've been a waste of
cycles at the time. But this is at least the third place where it'd be
useful to have attr_needed for child rels.
regards, tom lane
(2014/06/30 20:17), Ashutosh Bapat wrote: > On Mon, Jun 30, 2014 at 4:17 PM, Etsuro Fujita > <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote: > (2014/06/30 17:47), Ashutosh Bapat wrote: > BTW, why aren't you using the tlist passed to this function? I guess > create_scan_plan() passes tlist after processing it, so that > should be > used rather than rel->reltargetlist. > I think that that would be maybe OK, but I think that it would not > be efficient to use the list to compute attrs_used, because the > tlist would have more information than rel->reltargetlist in cases > where the tlist is build through build_physical_tlist(). > In that case, we can call build_relation_tlist() for foreign tables. Do you mean build_physical_tlist()? Yeah, we can call build_physical_tlist() (and do that in some cases), but if we call the function, it would generate a tlist that contains all Vars in the relation, not only those Vars actually needed by the query (ie, Vars in reltargetlist), and thus it would take more cycles to compute attr_used from the tlist than from reltargetlist. That' what I wanted to say. Thanks, Best regards, Etsuro Fujita
(2014/06/30 22:48), Tom Lane wrote: > Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes: >> Done. I think this is because create_foreignscan_plan() makes reference >> to attr_needed, which isn't computed for inheritance children. > > I wonder whether it isn't time to change that. It was coded like that > originally only because calculating the values would've been a waste of > cycles at the time. But this is at least the third place where it'd be > useful to have attr_needed for child rels. +1 for calculating attr_needed for child rels. (I was wondering too.) I'll create a separate patch for it. Thanks, Best regards, Etsuro Fujita
Hello, > > BTW, why aren't you using the tlist passed to this function? I guess > >> create_scan_plan() passes tlist after processing it, so that should be > >> used rather than rel->reltargetlist. > >> > > > > I think that that would be maybe OK, but I think that it would not be > > efficient to use the list to compute attrs_used, because the tlist would > > have more information than rel->reltargetlist in cases where the tlist is > > build through build_physical_tlist(). > > > > > In that case, we can call build_relation_tlist() for foreign tables. # is it build_base_rel_tilst() ? It needs only one effective line added, which would be seemingly far simple ingoring potential extra calculation for non-child relations (I suppose getting rid of it needs a foreach on rel->append_rel_list..) and too-longer physical tlist. On the other hand build_relation_tlist() and this patch seem to be in the same order of computational complexity for the length of the tlist. I prefer to use build_base_rel_tlist() for the clarity of code. But I don't know how high the chance to big physical tlist and non-child relations become to be an annoyance. Is there any suggestions? By the way, I tried xmin and xmax for the file_fdw tables. postgres=# select tableoid, xmin, xmax, * from passwd1; tableoid | xmin | xmax | uname | pass | uid | gid | .. 16396 | 244 | 4294967295 | root | x | 0 | 0 | root... 16396 | 252 | 4294967295 | bin | x | 1 | 1 | bin... 16396 | 284 |4294967295 | daemon | x | 2 | 2 | daemon... The xmin and xmax apparently doesn't look sane. After some investigation, I found that they came from the following place in heap_form_tuple(), (call stach is show below) | HeapTupleHeaderSetDatumLength(td, len); | HeapTupleHeaderSetTypeId(td, tupleDescriptor->tdtypeid); | HeapTupleHeaderSetTypMod(td, tupleDescriptor->tdtypmod); HeapTupleHeader is a union of HeapTupleFields and DatumTupleFields and these macors seem to treat it as the latter. Then later this tuple seems to be read as the former so xmin and xmax should have that values. This seems to be a bug. But I have no idea how to deal with it for now. I'll take more look into this. The call stack onto the above heap_form_tuple is as follows. #0 heap_form_tuple (tupleDescriptor=0x7fc46d2953a8, values=0x10dd1a0, isnull=0x10dd1f8 "") at heaptuple.c:731 #1 ExecCopySlotTuple (slot=0x10dd0e0) at execTuples.c:574 #2 ExecMaterializeSlot (slot=0x10dd0e0) at execTuples.c:760 #3 ForeignNext (node=0x10dc7b0) at nodeForeignscan.c:61 #4 ExecScanFetch (node=0x10dc7b0, accessMtd=0x66599c <ForeignNext>, recheckMtd=0x665a43 <ForeignRecheck>) at execScan.c:82 #5 ExecScan (node=0x10dc7b0, accessMtd=0x66599c <ForeignNext>, recheckMtd=0x665a43 <ForeignRecheck>) at execScan.c:167 #6 ExecForeignScan (node=0x10dc7b0) at nodeForeignscan.c:91 #7 ExecProcNode (node=0x10dc7b0) at execProcnode.c:442 regards, -- Kyotaro Horiguchi NTT Open Source Software Center
<div dir="ltr">If we are going to change that portion of the code, we may as well go a bit forward and allow any expressionsto be fetched from a foreign server (obviously, if that server is capable of doing so). It will help, when wecome to aggregate push-down or whole query push-down (whenever that happens). So, instead of attr_needed, which restrictsonly the attributes to be fetched, why not to targetlist itself?<br /></div><div class="gmail_extra"><br /><br /><divclass="gmail_quote">On Mon, Jun 30, 2014 at 7:18 PM, Tom Lane <span dir="ltr"><<a href="mailto:tgl@sss.pgh.pa.us"target="_blank">tgl@sss.pgh.pa.us</a>></span> wrote:<br /><blockquote class="gmail_quote"style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">Etsuro Fujita <<ahref="mailto:fujita.etsuro@lab.ntt.co.jp">fujita.etsuro@lab.ntt.co.jp</a>> writes:<br /> > Done. I think thisis because create_foreignscan_plan() makes reference<br /> > to attr_needed, which isn't computed for inheritancechildren.<br /><br /></div>I wonder whether it isn't time to change that. It was coded like that<br /> originallyonly because calculating the values would've been a waste of<br /> cycles at the time. But this is at least thethird place where it'd be<br /> useful to have attr_needed for child rels.<br /><br /> regards,tom lane<br /></blockquote></div><br /><br clear="all" /><br />-- <br /><div dir="ltr">Best Wishes,<br />AshutoshBapat<br />EnterpriseDB Corporation<br />The Postgres Database Company<br /></div></div>
On Tue, Jul 1, 2014 at 7:39 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/06/30 20:17), Ashutosh Bapat wrote:On Mon, Jun 30, 2014 at 4:17 PM, Etsuro Fujita<fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:(2014/06/30 17:47), Ashutosh Bapat wrote:Do you mean build_physical_tlist()?BTW, why aren't you using the tlist passed to this function? I guess
create_scan_plan() passes tlist after processing it, so that
should be
used rather than rel->reltargetlist.I think that that would be maybe OK, but I think that it would not
be efficient to use the list to compute attrs_used, because the
tlist would have more information than rel->reltargetlist in cases
where the tlist is build through build_physical_tlist().In that case, we can call build_relation_tlist() for foreign tables.
Sorry, I meant build_path_tlist() or disuse_physical_tlist() whichever is appropriate.
We may want to modify use_physical_tlist(), to return false, in case of foreign tables. BTW, it does return false for inheritance trees.
486 /*
487 * Can't do it with inheritance cases either (mainly because Append
488 * doesn't project).
489 */
490 if (rel->reloptkind != RELOPT_BASEREL)
491 return false;
486 /*
487 * Can't do it with inheritance cases either (mainly because Append
488 * doesn't project).
489 */
490 if (rel->reloptkind != RELOPT_BASEREL)
491 return false;
Yeah, we can call build_physical_tlist() (and do that in some cases), but if we call the function, it would generate a tlist that contains all Vars in the relation, not only those Vars actually needed by the query (ie, Vars in reltargetlist), and thus it would take more cycles to compute attr_used from the tlist than from reltargetlist. That' what I wanted to say.
Thanks,
Best regards,
Etsuro Fujita
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Hello,
Sorry, this was no relation with this patch.
ForeignNext materializes the slot, which would be any of physical
and virtual tuple, when system column was requested. If it was a
virtual one, file_fdw makes this, heap_form_tuple generates the
tuple as DatumTuple. The result is a jumble of virtual and
physical. But the returning slot has tts_tuple so the caller
interprets it as a physical one.
Anyway the requests for xmin/xmax could not be prevented
beforehand in current framework, I did rewrite the physical tuple
header so as to really be a physical one.
This would be another patch, so I will put this into next CF if
this don't get any immediate objection.
> By the way, I tried xmin and xmax for the file_fdw tables.
>
> postgres=# select tableoid, xmin, xmax, * from passwd1;
> tableoid | xmin | xmax | uname | pass | uid | gid | ..
> 16396 | 244 | 4294967295 | root | x | 0 | 0 | root...
> 16396 | 252 | 4294967295 | bin | x | 1 | 1 | bin...
> 16396 | 284 | 4294967295 | daemon | x | 2 | 2 | daemon...
>
> The xmin and xmax apparently doesn't look sane. After some
> investigation, I found that they came from the following place in
> heap_form_tuple(), (call stach is show below)
regards,
--
Kyotaro Horiguchi
NTT Open Source Software Center
diff --git a/src/backend/executor/nodeForeignscan.c b/src/backend/executor/nodeForeignscan.c
index 9cc5345..728db14 100644
--- a/src/backend/executor/nodeForeignscan.c
+++ b/src/backend/executor/nodeForeignscan.c
@@ -22,6 +22,8 @@ */#include "postgres.h"
+#include "access/transam.h"
+#include "access/htup_details.h"#include "executor/executor.h"#include "executor/nodeForeignscan.h"#include
"foreign/fdwapi.h"
@@ -58,8 +60,21 @@ ForeignNext(ForeignScanState *node) */ if (plan->fsSystemCol && !TupIsNull(slot)) {
+ bool was_virtual_tuple = (slot->tts_tuple == NULL); HeapTuple tup =
ExecMaterializeSlot(slot);
+ if (was_virtual_tuple)
+ {
+ /*
+ * ExecMaterializeSlot fills the tuple header as a
+ * DatumTupleFields if the slot was a virtual tuple, but a
+ * physical one is needed here. So rewrite the tuple header as
+ * HeapTupleFirelds.
+ */
+ HeapTupleHeaderSetXmin(tup->t_data, FrozenTransactionId);
+ HeapTupleHeaderSetXmax(tup->t_data, InvalidTransactionId);
+ HeapTupleHeaderSetCmin(tup->t_data, FirstCommandId);
+ } tup->t_tableOid = RelationGetRelid(node->ss.ss_currentRelation); }
(2014/07/01 15:13), Ashutosh Bapat wrote: > On Tue, Jul 1, 2014 at 7:39 AM, Etsuro Fujita > <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote: > We may want to modify use_physical_tlist(), to return false, in case of > foreign tables. BTW, it does return false for inheritance trees. Yeah, but please consider cases where foreign tables are not inheritance child rels (and any system columns are requested). > 486 /* > 487 * Can't do it with inheritance cases either (mainly because > Append > 488 * doesn't project). > 489 */ > 490 if (rel->reloptkind != RELOPT_BASEREL) > 491 return false; > > Yeah, we can call build_physical_tlist() (and do that in some > cases), but if we call the function, it would generate a tlist that > contains all Vars in the relation, not only those Vars actually > needed by the query (ie, Vars in reltargetlist), and thus it would > take more cycles to compute attr_used from the tlist than from > reltargetlist. That' what I wanted to say. Maybe I'm missing something, but what's the point of using the tlist, not reltargetlist? Thanks, Best regards, Etsuro Fujita
On Tue, Jul 1, 2014 at 12:25 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/07/01 15:13), Ashutosh Bapat wrote:On Tue, Jul 1, 2014 at 7:39 AM, Etsuro Fujita<fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:Yeah, but please consider cases where foreign tables are not inheritance child rels (and any system columns are requested).We may want to modify use_physical_tlist(), to return false, in case of
foreign tables. BTW, it does return false for inheritance trees.Maybe I'm missing something, but what's the point of using the tlist, not reltargetlist?486 /*
487 * Can't do it with inheritance cases either (mainly because
Append
488 * doesn't project).
489 */
490 if (rel->reloptkind != RELOPT_BASEREL)
491 return false;
Yeah, we can call build_physical_tlist() (and do that in some
cases), but if we call the function, it would generate a tlist that
contains all Vars in the relation, not only those Vars actually
needed by the query (ie, Vars in reltargetlist), and thus it would
take more cycles to compute attr_used from the tlist than from
reltargetlist. That' what I wanted to say.
Compliance with other create_*scan_plan() functions. The tlist passed to those functions is sometimes preprocessed in create_scan_plan() and some of the function it calls. If we use reltargetlist directly, we loose that preprocessing. I have not see any of create_*scan_plan() fetch the targetlist directly from RelOptInfo. It is always the one supplied by build_path_tlist() or disuse_physical_tlist() (which in turn calls build_path_tlist()) or build_physical_tlist().
Thanks,
Best regards,
Etsuro Fujita
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
(2014/07/01 16:04), Ashutosh Bapat wrote: > On Tue, Jul 1, 2014 at 12:25 PM, Etsuro Fujita > <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote: > Maybe I'm missing something, but what's the point of using the > tlist, not reltargetlist? > Compliance with other create_*scan_plan() functions. The tlist passed to > those functions is sometimes preprocessed in create_scan_plan() and some > of the function it calls. If we use reltargetlist directly, we loose > that preprocessing. I have not see any of create_*scan_plan() fetch the > targetlist directly from RelOptInfo. It is always the one supplied by > build_path_tlist() or disuse_physical_tlist() (which in turn calls > build_path_tlist()) or build_physical_tlist(). I've got the point. As I said upthread, I'll work on calculating attr_needed for child rels, and I hope that that will eliminate your concern. Thanks, Best regards, Etsuro Fujita
Hi, At Tue, 01 Jul 2014 16:30:41 +0900, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote in <53B263A1.3060107@lab.ntt.co.jp> > I've got the point. > > As I said upthread, I'll work on calculating attr_needed for child > rels, and I hope that that will eliminate your concern. Inheritance tree is expanded far after where attr_needed for the parent built, in set_base_rel_sizes() in make_one_rel(). I'm afraid that building all attr_needed for whole inheritance tree altogether is a bit suffering. I have wanted the point of inheritance expansion earlier for another patch. Do you think that rearranging there? Or generate them individually in crete_foreign_plan()? Anyway, I'm lookin forward to your next patch. So no answer needed. regards, -- Kyotaro Horiguchi NTT Open Source Software Center
On Fri, Jun 20, 2014 at 05:04:06PM +0900, Kyotaro HORIGUCHI wrote: > Attached is the rebased patch of v11 up to the current master. I've been studying this patch. SELECT FOR UPDATE on the inheritance parent fails with a can't-happen error condition, even when SELECT FOR UPDATE on the child foreign table alone would have succeeded. The patch adds zero tests. Add principal tests to postgres_fdw.sql. Also, create a child foreign table in foreign_data.sql; this will make dump/reload tests of the regression database exercise an inheritance tree that includes a foreign table. The inheritance section of ddl.sgml should mention child foreign tables, at least briefly. Consider mentioning it in the partitioning section, too. Your chosen ANALYZE behavior is fair, but the messaging from a database-wide ANALYZE VERBOSE needs work: INFO: analyzing "test_foreign_inherit.parent" INFO: "parent": scanned 1 of 1 pages, containing 1 live rows and 0 dead rows; 1 rows in sample, 1 estimated total rows INFO: analyzing "test_foreign_inherit.parent" inheritance tree WARNING: relcache reference leak: relation "child" not closed WARNING: relcache reference leak: relation "tchild" not closed WARNING: relcache reference leak: relation "parent" not closed Please arrange to omit the 'analyzing "tablename" inheritance tree' message, since this ANALYZE actually skipped that task. (The warnings obviously need a fix, too.) I do find it awkward that adding a foreign table to an inheritance tree will make autovacuum stop collecting statistics on that inheritance tree, but I can't think of a better policy. The rest of these review comments are strictly observations; they're not requests for you to change the patch, but they may deserve more discussion. We use the term "child table" in many messages. Should that change to something more inclusive, now that the child may be a foreign table? Perhaps one of "child relation", plain "child", or "child foreign table"/"child table" depending on the actual object? "child relation" is robust technically, but it might add more confusion than it removes. Varying the message depending on the actual object feels like a waste. Opinions? LOCK TABLE on the inheritance parent locks child foreign tables, but LOCK TABLE fails when given a foreign table directly. That's odd, but I see no cause to change it. A partition root only accepts an UPDATE command if every child is updatable. Similarly, "UPDATE ... WHERE CURRENT OF cursor_name" fails if any child does not support it. That seems fine. Incidentally, does anyone have a FDW that supports WHERE CURRENT OF? The longstanding behavior of CREATE TABLE INHERITS is to reorder local columns to match the order found in parents. That is, both of these tables actually have columns in the order (a,b): create table parent (a int, b int); create table child (b int, a int) inherits (parent); Ordinary dump/reload always uses CREATE TABLE INHERITS, thereby changing column order in this way. (pg_dump --binary-upgrade uses ALTER TABLE INHERIT and some catalog hacks to avoid doing so.) I've never liked that dump/reload can change column order, but it's tolerable if you follow the best practice of always writing out column lists. The stakes rise for foreign tables, because column order is inherently significant to file_fdw and probably to certain other non-RDBMS FDWs. If pg_dump changes column order in your file_fdw foreign table, the table breaks. I would heartily support making pg_dump preserve column order for all inheritance children. That doesn't rise to the level of being a blocker for this patch, though. I am attaching rough-hewn tests I used during this review. Thanks, nm -- Noah Misch EnterpriseDB http://www.enterprisedb.com
Attachment
Hi Fujita-san, Sorry for leaving this thread for long time. 2014-06-24 16:30 GMT+09:00 Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>: > (2014/06/23 18:35), Ashutosh Bapat wrote: >> >> Hi, >> Selecting tableoid on parent causes an error, "ERROR: cannot extract >> system attribute from virtual tuple". The foreign table has an OID which >> can be reported as tableoid for the rows coming from that foreign table. >> Do we want to do that? > > > No. I think it's a bug. I'll fix it. IIUC, you mean that tableoid can't be retrieved when a foreign table is accessed via parent table, but it sounds strange to me, because one of main purposes of tableoid is determine actual source table in appended results. Am I missing the point? -- Shigeru HANADA
(2014/07/10 18:12), Shigeru Hanada wrote: > 2014-06-24 16:30 GMT+09:00 Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>: >> (2014/06/23 18:35), Ashutosh Bapat wrote: >>> Selecting tableoid on parent causes an error, "ERROR: cannot extract >>> system attribute from virtual tuple". The foreign table has an OID which >>> can be reported as tableoid for the rows coming from that foreign table. >>> Do we want to do that? >> No. I think it's a bug. I'll fix it. > IIUC, you mean that tableoid can't be retrieved when a foreign table > is accessed via parent table, No. What I want to say is that tableoid *can* be retrieved when a foreign table is accessed via the parent table. Thanks, Best regards, Etsuro Fujita
(2014/07/11 15:50), Etsuro Fujita wrote: > (2014/07/10 18:12), Shigeru Hanada wrote: >> IIUC, you mean that tableoid can't be retrieved when a foreign table >> is accessed via parent table, > > No. What I want to say is that tableoid *can* be retrieved when a > foreign table is accessed via the parent table. In fact, you can do that with v13 [1], but I plan to change the way of fixing (see [2]). Thanks, [1] http://www.postgresql.org/message-id/53B10914.2010504@lab.ntt.co.jp [2] http://www.postgresql.org/message-id/53B2188B.4090302@lab.ntt.co.jp Best regards, Etsuro Fujita
Hello, I found that tuples come from file_fdw has strange xmin and xmax.
> postgres=# select tableoid, xmin, xmax, * from passwd1;
> tableoid | xmin | xmax | uname | pass | uid | gid | ..
> 16396 | 244 | 4294967295 | root | x | 0 | 0 | root...
> 16396 | 252 | 4294967295 | bin | x | 1 | 1 | bin...
> 16396 | 284 | 4294967295 | daemon | x | 2 | 2 | daemon...
Back to 9.1 gives the same result.
These xmin and xmax are the simple interpretations of a
DatumTupleFields filled by ExecMaterializedSlot() beeing fed the
source virtual tuple slot from file_fdw. On the other hand,
postgres_fdw gives sane xids (xmin = 2, xmax = 0).
This is because ForeignNext returns physical tuples in which
their headers are DatumTupleFields regardless whether the system
columns are requested or not. The fdw machinary desn't seem to
provide fdw handlers with a means to reject requests for
unavailable system columns, so the tuple header should be fixed
with the sane values as HeapTupleFields.
The patch attached fixes the header of materialized tuple to be
sane (2, 0) if the source slot was a virtual tuple in mechanism().
regards,
--
Kyotaro Horiguchi
NTT Open Source Software Center
diff --git a/src/backend/executor/nodeForeignscan.c b/src/backend/executor/nodeForeignscan.c
index 9cc5345..59dc5f4 100644
--- a/src/backend/executor/nodeForeignscan.c
+++ b/src/backend/executor/nodeForeignscan.c
@@ -22,6 +22,8 @@ */#include "postgres.h"
+#include "access/transam.h"
+#include "access/htup_details.h"#include "executor/executor.h"#include "executor/nodeForeignscan.h"#include
"foreign/fdwapi.h"
@@ -58,8 +60,21 @@ ForeignNext(ForeignScanState *node) */ if (plan->fsSystemCol && !TupIsNull(slot)) {
+ bool was_virtual_tuple = (slot->tts_tuple == NULL); HeapTuple tup =
ExecMaterializeSlot(slot);
+ if (was_virtual_tuple)
+ {
+ /*
+ * ExecMaterializeSlot fills the tuple header as a
+ * DatumTupleFields if the slot was a virtual tuple, although a
+ * physical one is needed by the callers. So rewrite the tuple
+ * header as a sane HeapTupleFields.
+ */
+ HeapTupleHeaderSetXmin(tup->t_data, FrozenTransactionId);
+ HeapTupleHeaderSetXmax(tup->t_data, InvalidTransactionId);
+ HeapTupleHeaderSetCmin(tup->t_data, FirstCommandId);
+ } tup->t_tableOid = RelationGetRelid(node->ss.ss_currentRelation); }
Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> writes:
> Hello, I found that tuples come from file_fdw has strange xmin and xmax.
file_fdw isn't documented to return anything useful for xmin/xmax/etc,
so I don't find this surprising.
> The patch attached fixes the header of materialized tuple to be
> sane (2, 0) if the source slot was a virtual tuple in mechanism().
I don't really think it's ForeignNext's place to be doing something
about this. It seems like a useless expenditure of cycles. Besides,
this fails to cover e.g. postgres_fdw, which is likewise unconcerned
about what it returns for system columns other than ctid.
regards, tom lane
Hello, sorry, it's my bad. > Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> writes: > > Hello, I found that tuples come from file_fdw has strange xmin and xmax. > > file_fdw isn't documented to return anything useful for xmin/xmax/etc, > so I don't find this surprising. > > > The patch attached fixes the header of materialized tuple to be > > sane (2, 0) if the source slot was a virtual tuple in mechanism(). > > I don't really think it's ForeignNext's place to be doing something > about this. It seems like a useless expenditure of cycles. Besides, > this fails to cover e.g. postgres_fdw, which is likewise unconcerned > about what it returns for system columns other than ctid. I somehow misunderstood that postgres_fdw returns (xmin, xmax) = (2, 0) but I confirmed that xmin, xmax and citd seems insane. I completely agree with you. Thank you for giving response for the stupid story. regards, -- Kyotaro Horiguchi NTT Open Source Software Center
(2014/07/01 11:10), Etsuro Fujita wrote: > (2014/06/30 22:48), Tom Lane wrote: >> Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes: >>> Done. I think this is because create_foreignscan_plan() makes reference >>> to attr_needed, which isn't computed for inheritance children. >> >> I wonder whether it isn't time to change that. It was coded like that >> originally only because calculating the values would've been a waste of >> cycles at the time. But this is at least the third place where it'd be >> useful to have attr_needed for child rels. > > +1 for calculating attr_needed for child rels. (I was wondering too.) > > I'll create a separate patch for it. Attached is a WIP patch for that. The following functions have been changed to refer to attr_needed: * check_index_only() * remove_unused_subquery_outputs() * check_selective_binary_conversion() I'll add this to the upcoming commitfest. If anyone has any time to glance at it before then, that would be a great help. Thanks, Best regards, Etsuro Fujita
Attachment
(2014/08/06 20:43), Etsuro Fujita wrote: >> (2014/06/30 22:48), Tom Lane wrote: >>> I wonder whether it isn't time to change that. It was coded like that >>> originally only because calculating the values would've been a waste of >>> cycles at the time. But this is at least the third place where it'd be >>> useful to have attr_needed for child rels. > Attached is a WIP patch for that. I've revised the patch. Changes: * Make the code more readable * Revise the comments * Cleanup Please find attached an updated version of the patch. Thanks, Best regards, Etsuro Fujita
Attachment
Compute attr_needed for child relations (was Re: inherit support for foreign tables)
From
Etsuro Fujita
Date:
(2014/08/08 18:51), Etsuro Fujita wrote:
>>> (2014/06/30 22:48), Tom Lane wrote:
>>>> I wonder whether it isn't time to change that. It was coded like that
>>>> originally only because calculating the values would've been a waste of
>>>> cycles at the time. But this is at least the third place where it'd be
>>>> useful to have attr_needed for child rels.
> I've revised the patch.
There was a problem with the previous patch, which will be described
below. Attached is the updated version of the patch addressing that.
The previous patch doesn't cope with some UNION ALL cases properly. So,
e.g., the server will crash for the following query:
postgres=# create table ta1 (f1 int);
CREATE TABLE
postgres=# create table ta2 (f2 int primary key, f3 int);
CREATE TABLE
postgres=# create table tb1 (f1 int);
CREATE TABLE
postgres=# create table tb2 (f2 int primary key, f3 int);
CREATE TABLE
postgres=# explain verbose select f1 from ((select f1, f2 from (select
f1, f2, f3 from ta1 left join ta2 on f1 = f2 limit 1) ssa) union all
(select f1,
f2 from (select f1, f2, f3 from tb1 left join tb2 on f1 = f2 limit 1)
ssb)) ss;
With the updated version, we get the right result:
postgres=# explain verbose select f1 from ((select f1, f2 from (select
f1, f2, f3 from ta1 left join ta2 on f1 = f2 limit 1) ssa) union all
(select f1,
f2 from (select f1, f2, f3 from tb1 left join tb2 on f1 = f2 limit 1)
ssb)) ss;
QUERY PLAN
--------------------------------------------------------------------------------
Append (cost=0.00..0.05 rows=2 width=4)
-> Subquery Scan on ssa (cost=0.00..0.02 rows=1 width=4)
Output: ssa.f1
-> Limit (cost=0.00..0.01 rows=1 width=4)
Output: ta1.f1, (NULL::integer), (NULL::integer)
-> Seq Scan on public.ta1 (cost=0.00..34.00 rows=2400
width=4)
Output: ta1.f1, NULL::integer, NULL::integer
-> Subquery Scan on ssb (cost=0.00..0.02 rows=1 width=4)
Output: ssb.f1
-> Limit (cost=0.00..0.01 rows=1 width=4)
Output: tb1.f1, (NULL::integer), (NULL::integer)
-> Seq Scan on public.tb1 (cost=0.00..34.00 rows=2400
width=4)
Output: tb1.f1, NULL::integer, NULL::integer
Planning time: 0.453 ms
(14 rows)
While thinking to address this problem, Ashutosh also expressed concern
about the UNION ALL handling in the previous patch in a private email.
Thank you for the review, Ashutosh!
Thanks,
Best regards,
Etsuro Fujita
Attachment
Re: Compute attr_needed for child relations (was Re: inherit support for foreign tables)
From
Ashutosh Bapat
Date:
Hi,
--
On Thu, Aug 14, 2014 at 10:05 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/08/08 18:51), Etsuro Fujita wrote:
>>> (2014/06/30 22:48), Tom Lane wrote:
>>>> I wonder whether it isn't time to change that. It was coded like that
>>>> originally only because calculating the values would've been a waste of
>>>> cycles at the time. But this is at least the third place where it'd be
>>>> useful to have attr_needed for child rels.
> I've revised the patch.
There was a problem with the previous patch, which will be described
below. Attached is the updated version of the patch addressing that.
The previous patch doesn't cope with some UNION ALL cases properly. So,
e.g., the server will crash for the following query:
postgres=# create table ta1 (f1 int);
CREATE TABLE
postgres=# create table ta2 (f2 int primary key, f3 int);
CREATE TABLE
postgres=# create table tb1 (f1 int);
CREATE TABLE
postgres=# create table tb2 (f2 int primary key, f3 int);
CREATE TABLE
postgres=# explain verbose select f1 from ((select f1, f2 from (select
f1, f2, f3 from ta1 left join ta2 on f1 = f2 limit 1) ssa) union all
(select f1,
f2 from (select f1, f2, f3 from tb1 left join tb2 on f1 = f2 limit 1)
ssb)) ss;
With the updated version, we get the right result:
postgres=# explain verbose select f1 from ((select f1, f2 from (select
f1, f2, f3 from ta1 left join ta2 on f1 = f2 limit 1) ssa) union all
(select f1,
f2 from (select f1, f2, f3 from tb1 left join tb2 on f1 = f2 limit 1)
ssb)) ss;
QUERY PLAN
--------------------------------------------------------------------------------
Append (cost=0.00..0.05 rows=2 width=4)
-> Subquery Scan on ssa (cost=0.00..0.02 rows=1 width=4)
Output: ssa.f1
-> Limit (cost=0.00..0.01 rows=1 width=4)
Output: ta1.f1, (NULL::integer), (NULL::integer)
-> Seq Scan on public.ta1 (cost=0.00..34.00 rows=2400
width=4)
Output: ta1.f1, NULL::integer, NULL::integer
-> Subquery Scan on ssb (cost=0.00..0.02 rows=1 width=4)
Output: ssb.f1
-> Limit (cost=0.00..0.01 rows=1 width=4)
Output: tb1.f1, (NULL::integer), (NULL::integer)
-> Seq Scan on public.tb1 (cost=0.00..34.00 rows=2400
width=4)
Output: tb1.f1, NULL::integer, NULL::integer
Planning time: 0.453 ms
(14 rows)
While thinking to address this problem, Ashutosh also expressed concern
about the UNION ALL handling in the previous patch in a private email.
Thank you for the review, Ashutosh!
Thanks for taking care of this one.
Here are some more comments
attr_needed also has the attributes used in the restriction clauses as seen in distribute_qual_to_rels(), so, it looks unnecessary to call pull_varattnos() on the clauses in baserestrictinfo in functions check_selective_binary_conversion(), remove_unused_subquery_outputs(), check_index_only().
Although in case of RTE_RELATION, the 0th entry in attr_needed corresponds to FirstLowInvalidHeapAttributeNumber + 1, it's always safer to use it is RelOptInfo::min_attr, in case get_relation_info() wants to change assumption or somewhere down the line some other part of code wants to change attr_needed information. It may be unlikely, that it would change, but it will be better to stick to comments in RelOptInfo
443 AttrNumber min_attr; /* smallest attrno of rel (often <0) */
444 AttrNumber max_attr; /* largest attrno of rel */
445 Relids *attr_needed; /* array indexed [min_attr .. max_attr] */
Here are some more comments
attr_needed also has the attributes used in the restriction clauses as seen in distribute_qual_to_rels(), so, it looks unnecessary to call pull_varattnos() on the clauses in baserestrictinfo in functions check_selective_binary_conversion(), remove_unused_subquery_outputs(), check_index_only().
Although in case of RTE_RELATION, the 0th entry in attr_needed corresponds to FirstLowInvalidHeapAttributeNumber + 1, it's always safer to use it is RelOptInfo::min_attr, in case get_relation_info() wants to change assumption or somewhere down the line some other part of code wants to change attr_needed information. It may be unlikely, that it would change, but it will be better to stick to comments in RelOptInfo
443 AttrNumber min_attr; /* smallest attrno of rel (often <0) */
444 AttrNumber max_attr; /* largest attrno of rel */
445 Relids *attr_needed; /* array indexed [min_attr .. max_attr] */
Thanks,
Best regards,
Etsuro Fujita
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Re: Compute attr_needed for child relations (was Re: inherit support for foreign tables)
From
Etsuro Fujita
Date:
Hi Ashutish,
(2014/08/14 22:30), Ashutosh Bapat wrote:
> On Thu, Aug 14, 2014 at 10:05 AM, Etsuro Fujita
> <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:
>
> (2014/08/08 18:51), Etsuro Fujita wrote:
> >>> (2014/06/30 22:48), Tom Lane wrote:
> >>>> I wonder whether it isn't time to change that. It was coded
> like that
> >>>> originally only because calculating the values would've been a
> waste of
> >>>> cycles at the time. But this is at least the third place
> where it'd be
> >>>> useful to have attr_needed for child rels.
>
> > I've revised the patch.
>
> There was a problem with the previous patch, which will be described
> below. Attached is the updated version of the patch addressing that.
> Here are some more comments
Thank you for the further review!
> attr_needed also has the attributes used in the restriction clauses as
> seen in distribute_qual_to_rels(), so, it looks unnecessary to call
> pull_varattnos() on the clauses in baserestrictinfo in functions
> check_selective_binary_conversion(), remove_unused_subquery_outputs(),
> check_index_only().
IIUC, I think it's *necessary* to do that in those functions since the
attributes used in the restriction clauses in baserestrictinfo are not
added to attr_needed due the following code in distribute_qual_to_rels.
/*
* If it's a join clause (either naturally, or because delayed by
* outer-join rules), add vars used in the clause to targetlists of
their
* relations, so that they will be emitted by the plan nodes that scan
* those relations (else they won't be available at the join node!).
*
* Note: if the clause gets absorbed into an EquivalenceClass then this
* may be unnecessary, but for now we have to do it to cover the case
* where the EC becomes ec_broken and we end up reinserting the
original
* clauses into the plan.
*/
if (bms_membership(relids) == BMS_MULTIPLE)
{
List *vars = pull_var_clause(clause,
PVC_RECURSE_AGGREGATES,
PVC_INCLUDE_PLACEHOLDERS);
add_vars_to_targetlist(root, vars, relids, false);
list_free(vars);
}
> Although in case of RTE_RELATION, the 0th entry in attr_needed
> corresponds to FirstLowInvalidHeapAttributeNumber + 1, it's always safer
> to use it is RelOptInfo::min_attr, in case get_relation_info() wants to
> change assumption or somewhere down the line some other part of code
> wants to change attr_needed information. It may be unlikely, that it
> would change, but it will be better to stick to comments in RelOptInfo
> 443 AttrNumber min_attr; /* smallest attrno of rel (often
> <0) */
> 444 AttrNumber max_attr; /* largest attrno of rel */
> 445 Relids *attr_needed; /* array indexed [min_attr ..
> max_attr] */
Good point! Attached is the revised version of the patch.
Thanks,
Best regards,
Etsuro Fujita
Attachment
Hi Noah, Thank you for the review! (2014/07/02 11:23), Noah Misch wrote: > On Fri, Jun 20, 2014 at 05:04:06PM +0900, Kyotaro HORIGUCHI wrote: >> Attached is the rebased patch of v11 up to the current master. > > I've been studying this patch. > > SELECT FOR UPDATE on the inheritance parent fails with a can't-happen error > condition, even when SELECT FOR UPDATE on the child foreign table alone would > have succeeded. To fix this, I've modified the planner and executor so that the planner adds wholerow as well as ctid and tableoid as resjunk output columns to the plan for an inheritance tree that contains foreign tables, and that while the executor uses the ctid and tableoid in the EPQ processing for child regular tables as before, the executor uses the wholerow and tableoid for child foreign tables. Patch attached. This is based on the patch [1]. > The patch adds zero tests. Add principal tests to postgres_fdw.sql. Also, > create a child foreign table in foreign_data.sql; this will make dump/reload > tests of the regression database exercise an inheritance tree that includes a > foreign table. Done. I used your tests as reference. Thanks! > The inheritance section of ddl.sgml should mention child foreign tables, at > least briefly. Consider mentioning it in the partitioning section, too. Done. > Your chosen ANALYZE behavior is fair, but the messaging from a database-wide > ANALYZE VERBOSE needs work: > > INFO: analyzing "test_foreign_inherit.parent" > INFO: "parent": scanned 1 of 1 pages, containing 1 live rows and 0 dead rows; 1 rows in sample, 1 estimated total rows > INFO: analyzing "test_foreign_inherit.parent" inheritance tree > WARNING: relcache reference leak: relation "child" not closed > WARNING: relcache reference leak: relation "tchild" not closed > WARNING: relcache reference leak: relation "parent" not closed > > Please arrange to omit the 'analyzing "tablename" inheritance tree' message, > since this ANALYZE actually skipped that task. (The warnings obviously need a > fix, too.) I do find it awkward that adding a foreign table to an inheritance > tree will make autovacuum stop collecting statistics on that inheritance tree, > but I can't think of a better policy. I think it would be better that this is handled in the same way as an inheritance tree that turns out to be a singe table that doesn't have any descendants in acquire_inherited_sample_rows(). That would still output the message as shown below, but I think that that would be more consistent with the existing code. The patch works so. (The warnings are also fixed.) INFO: analyzing "public.parent" INFO: "parent": scanned 0 of 0 pages, containing 0 live rows and 0 dead rows; 0 rows in sample, 0 estimated total rows INFO: analyzing "public.parent" inheritance tree INFO: analyzing "pg_catalog.pg_authid" INFO: "pg_authid": scanned 1 of 1 pages, containing 1 live rows and 0 dead rows; 1 rows in sample, 1 estimated total rows > The rest of these review comments are strictly observations; they're not > requests for you to change the patch, but they may deserve more discussion. I'd like to give my opinions on those things later on. Sorry for the long delay. [1] http://www.postgresql.org/message-id/53F4707C.8030904@lab.ntt.co.jp Best regards, Etsuro Fujita
Attachment
Re: Compute attr_needed for child relations (was Re: inherit support for foreign tables)
From
Ashutosh Bapat
Date:
On Wed, Aug 20, 2014 at 3:25 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
Hi Ashutish,
(2014/08/14 22:30), Ashutosh Bapat wrote:On Thu, Aug 14, 2014 at 10:05 AM, Etsuro Fujita<fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:
(2014/08/08 18:51), Etsuro Fujita wrote:
>>> (2014/06/30 22:48), Tom Lane wrote:
>>>> I wonder whether it isn't time to change that. It was coded
like that
>>>> originally only because calculating the values would've been a
waste of
>>>> cycles at the time. But this is at least the third place
where it'd be
>>>> useful to have attr_needed for child rels.
> I've revised the patch.
There was a problem with the previous patch, which will be described
below. Attached is the updated version of the patch addressing that.Thank you for the further review!Here are some more commentsIIUC, I think it's *necessary* to do that in those functions since the attributes used in the restriction clauses in baserestrictinfo are not added to attr_needed due the following code in distribute_qual_to_rels.attr_needed also has the attributes used in the restriction clauses as
seen in distribute_qual_to_rels(), so, it looks unnecessary to call
pull_varattnos() on the clauses in baserestrictinfo in functions
check_selective_binary_conversion(), remove_unused_subquery_outputs(),
check_index_only().
That's right. Thanks for pointing that out.
/*
* If it's a join clause (either naturally, or because delayed by
* outer-join rules), add vars used in the clause to targetlists of their
* relations, so that they will be emitted by the plan nodes that scan
* those relations (else they won't be available at the join node!).
*
* Note: if the clause gets absorbed into an EquivalenceClass then this
* may be unnecessary, but for now we have to do it to cover the case
* where the EC becomes ec_broken and we end up reinserting the original
* clauses into the plan.
*/
if (bms_membership(relids) == BMS_MULTIPLE)
{
List *vars = pull_var_clause(clause,
PVC_RECURSE_AGGREGATES,
PVC_INCLUDE_PLACEHOLDERS);
add_vars_to_targetlist(root, vars, relids, false);
list_free(vars);Good point! Attached is the revised version of the patch.
}Although in case of RTE_RELATION, the 0th entry in attr_needed
corresponds to FirstLowInvalidHeapAttributeNumber + 1, it's always safer
to use it is RelOptInfo::min_attr, in case get_relation_info() wants to
change assumption or somewhere down the line some other part of code
wants to change attr_needed information. It may be unlikely, that it
would change, but it will be better to stick to comments in RelOptInfo
443 AttrNumber min_attr; /* smallest attrno of rel (often
<0) */
444 AttrNumber max_attr; /* largest attrno of rel */
445 Relids *attr_needed; /* array indexed [min_attr ..
max_attr] */
If the patch is not in the commit-fest, can you please add it there? From my side, the review is done, it should be marked "ready for committer", unless somebody else wants to review.
Thanks,
Best regards,
Etsuro Fujita
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Re: Compute attr_needed for child relations (was Re: inherit support for foreign tables)
From
Etsuro Fujita
Date:
(2014/08/21 13:21), Ashutosh Bapat wrote: > On Wed, Aug 20, 2014 at 3:25 PM, Etsuro Fujita > <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote: > Hi Ashutish, I am sorry that I mistook your name's spelling. > (2014/08/14 22:30), Ashutosh Bapat wrote: > > On Thu, Aug 14, 2014 at 10:05 AM, Etsuro Fujita > <fujita.etsuro@lab.ntt.co.jp > <mailto:fujita.etsuro@lab.ntt.co.jp> > <mailto:fujita.etsuro@lab.ntt.__co.jp > <mailto:fujita.etsuro@lab.ntt.co.jp>>> wrote: > > (2014/08/08 18:51), Etsuro Fujita wrote: > >>> (2014/06/30 22:48), Tom Lane wrote: > >>>> I wonder whether it isn't time to change that. It > was coded > like that > >>>> originally only because calculating the values > would've been a > waste of > >>>> cycles at the time. But this is at least the third place > where it'd be > >>>> useful to have attr_needed for child rels. > There was a problem with the previous patch, which will be > described > below. Attached is the updated version of the patch > addressing that. > Here are some more comments > attr_needed also has the attributes used in the restriction > clauses as > seen in distribute_qual_to_rels(), so, it looks unnecessary to call > pull_varattnos() on the clauses in baserestrictinfo in functions > check_selective_binary___conversion(), > remove_unused_subquery___outputs(), > check_index_only(). > IIUC, I think it's *necessary* to do that in those functions since > the attributes used in the restriction clauses in baserestrictinfo > are not added to attr_needed due the following code in > distribute_qual_to_rels. > That's right. Thanks for pointing that out. > Although in case of RTE_RELATION, the 0th entry in attr_needed > corresponds to FirstLowInvalidHeapAttributeNu__mber + 1, it's > always safer > to use it is RelOptInfo::min_attr, in case get_relation_info() > wants to > change assumption or somewhere down the line some other part of code > wants to change attr_needed information. It may be unlikely, that it > would change, but it will be better to stick to comments in > RelOptInfo > 443 AttrNumber min_attr; /* smallest attrno of rel > (often > <0) */ > 444 AttrNumber max_attr; /* largest attrno of rel */ > 445 Relids *attr_needed; /* array indexed [min_attr .. > max_attr] */ > Good point! Attached is the revised version of the patch. > If the patch is not in the commit-fest, can you please add it there? I've already done that: https://commitfest.postgresql.org/action/patch_view?id=1529 > From my side, the review is done, it should be marked "ready for > committer", unless somebody else wants to review. Many thanks! Best regards, Etsuro Fujita
(2014/07/02 11:23), Noah Misch wrote: > On Fri, Jun 20, 2014 at 05:04:06PM +0900, Kyotaro HORIGUCHI wrote: >> Attached is the rebased patch of v11 up to the current master. > The rest of these review comments are strictly observations; they're not > requests for you to change the patch, but they may deserve more discussion. > > We use the term "child table" in many messages. Should that change to > something more inclusive, now that the child may be a foreign table? Perhaps > one of "child relation", plain "child", or "child foreign table"/"child table" > depending on the actual object? "child relation" is robust technically, but > it might add more confusion than it removes. Varying the message depending on > the actual object feels like a waste. Opinions? IMHO, I think that "child table" would not confusing users terribly. > LOCK TABLE on the inheritance parent locks child foreign tables, but LOCK > TABLE fails when given a foreign table directly. That's odd, but I see no > cause to change it. I agree wth that. > The longstanding behavior of CREATE TABLE INHERITS is to reorder local columns > to match the order found in parents. That is, both of these tables actually > have columns in the order (a,b): > > create table parent (a int, b int); > create table child (b int, a int) inherits (parent); > > Ordinary dump/reload always uses CREATE TABLE INHERITS, thereby changing > column order in this way. (pg_dump --binary-upgrade uses ALTER TABLE INHERIT > and some catalog hacks to avoid doing so.) I've never liked that dump/reload > can change column order, but it's tolerable if you follow the best practice of > always writing out column lists. The stakes rise for foreign tables, because > column order is inherently significant to file_fdw and probably to certain > other non-RDBMS FDWs. If pg_dump changes column order in your file_fdw > foreign table, the table breaks. I would heartily support making pg_dump > preserve column order for all inheritance children. That doesn't rise to the > level of being a blocker for this patch, though. I agree with that, too. I think it would be better to add docs for now. Thanks, Best regards, Etsuro Fujita
On Wed, Aug 20, 2014 at 08:11:01PM +0900, Etsuro Fujita wrote: > (2014/07/02 11:23), Noah Misch wrote: > >Your chosen ANALYZE behavior is fair, but the messaging from a database-wide > >ANALYZE VERBOSE needs work: > > > >INFO: analyzing "test_foreign_inherit.parent" > >INFO: "parent": scanned 1 of 1 pages, containing 1 live rows and 0 dead rows; 1 rows in sample, 1 estimated total rows > >INFO: analyzing "test_foreign_inherit.parent" inheritance tree > >WARNING: relcache reference leak: relation "child" not closed > >WARNING: relcache reference leak: relation "tchild" not closed > >WARNING: relcache reference leak: relation "parent" not closed > > > >Please arrange to omit the 'analyzing "tablename" inheritance tree' message, > >since this ANALYZE actually skipped that task. (The warnings obviously need a > >fix, too.) I do find it awkward that adding a foreign table to an inheritance > >tree will make autovacuum stop collecting statistics on that inheritance tree, > >but I can't think of a better policy. > > I think it would be better that this is handled in the same way as > an inheritance tree that turns out to be a singe table that doesn't > have any descendants in acquire_inherited_sample_rows(). That would > still output the message as shown below, but I think that that would > be more consistent with the existing code. The patch works so. > (The warnings are also fixed.) > > INFO: analyzing "public.parent" > INFO: "parent": scanned 0 of 0 pages, containing 0 live rows and 0 > dead rows; 0 rows in sample, 0 estimated total rows > INFO: analyzing "public.parent" inheritance tree > INFO: analyzing "pg_catalog.pg_authid" > INFO: "pg_authid": scanned 1 of 1 pages, containing 1 live rows and > 0 dead rows; 1 rows in sample, 1 estimated total rows Today's ANALYZE VERBOSE messaging for former inheritance parents (tables with relhassubclass = true but no pg_inherits.inhparent links) is deceptive, and I welcome a fix to omit the spurious message. As defects go, this is quite minor. There's fundamentally no value in collecting inheritance tree statistics for such a parent, and no PostgreSQL command will do so. Your proposed behavior for inheritance parents having at least one foreign table child is more likely to mislead DBAs in practice. An inheritance tree genuinely exists, and a different ANALYZE command is quite capable of collecting statistics on that inheritance tree. Messaging should reflect the difference between ANALYZE invocations that do such work and ANALYZE invocations that skip it. I'm anticipating a bug report along these lines: I saw poor estimates involving a child foreign table, so I ran "ANALYZE VERBOSE", which reported 'INFO: analyzing "public.parent"inheritance tree'. Estimates remained poor, so I scratched my head for awhile before noticing that pg_statsdidn't report such statistics. I then ran "ANALYZE VERBOSE parent", saw the same 'INFO: analyzing "public.parent"inheritance tree', but this time saw relevant rows in pg_stats. The message doesn't reflect the actual behavior. I'll sympathize with that complaint. It's a minor point overall, but the code impact is, I predict, small enough that we may as well get it right. A credible alternative is to emit a second message indicating that we skipped the inheritance tree statistics after all, and why we skipped them.
Noah Misch wrote: > I'm anticipating a bug report along these lines: > > I saw poor estimates involving a child foreign table, so I ran "ANALYZE > VERBOSE", which reported 'INFO: analyzing "public.parent" inheritance > tree'. Estimates remained poor, so I scratched my head for awhile before > noticing that pg_stats didn't report such statistics. I then ran "ANALYZE > VERBOSE parent", saw the same 'INFO: analyzing "public.parent" inheritance > tree', but this time saw relevant rows in pg_stats. The message doesn't > reflect the actual behavior. > > I'll sympathize with that complaint. Agreed on that. > A credible alternative is to emit a second message indicating that we > skipped the inheritance tree statistics after all, and why we skipped > them. That'd be similar to Xorg emitting messages such as [ 53.772] (II) intel(0): RandR 1.2 enabled, ignore the following RandR disabled message. [ 53.800] (--) RandR disabled I find this very poor. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: Compute attr_needed for child relations (was Re: inherit support for foreign tables)
From
Ashutosh Bapat
Date:
On Thu, Aug 21, 2014 at 3:00 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/08/21 13:21), Ashutosh Bapat wrote:On Wed, Aug 20, 2014 at 3:25 PM, Etsuro Fujita<fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:Hi Ashutish,
I am sorry that I mistook your name's spelling.(2014/08/14 22:30), Ashutosh Bapat wrote:<mailto:fujita.etsuro@lab.ntt.__co.jp
On Thu, Aug 14, 2014 at 10:05 AM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp
<mailto:fujita.etsuro@lab.ntt.co.jp>
<mailto:fujita.etsuro@lab.ntt.co.jp>>> wrote:
(2014/08/08 18:51), Etsuro Fujita wrote:
>>> (2014/06/30 22:48), Tom Lane wrote:
>>>> I wonder whether it isn't time to change that. It
was coded
like that
>>>> originally only because calculating the values
would've been a
waste of
>>>> cycles at the time. But this is at least the third place
where it'd be
>>>> useful to have attr_needed for child rels.There was a problem with the previous patch, which will be
described
below. Attached is the updated version of the patch
addressing that.Here are some more commentsattr_needed also has the attributes used in the restrictioncheck_selective_binary___conversion(),
clauses as
seen in distribute_qual_to_rels(), so, it looks unnecessary to call
pull_varattnos() on the clauses in baserestrictinfo in functions
remove_unused_subquery___outputs(),
check_index_only().IIUC, I think it's *necessary* to do that in those functions since
the attributes used in the restriction clauses in baserestrictinfo
are not added to attr_needed due the following code in
distribute_qual_to_rels.That's right. Thanks for pointing that out.Although in case of RTE_RELATION, the 0th entry in attr_neededcorresponds to FirstLowInvalidHeapAttributeNu__mber + 1, it's
always safer
to use it is RelOptInfo::min_attr, in case get_relation_info()
wants to
change assumption or somewhere down the line some other part of code
wants to change attr_needed information. It may be unlikely, that it
would change, but it will be better to stick to comments in
RelOptInfo
443 AttrNumber min_attr; /* smallest attrno of rel
(often
<0) */
444 AttrNumber max_attr; /* largest attrno of rel */
445 Relids *attr_needed; /* array indexed [min_attr ..
max_attr] */I've already done that:Good point! Attached is the revised version of the patch.If the patch is not in the commit-fest, can you please add it there?
https://commitfest.postgresql.org/action/patch_view?id=1529
Many thanks!From my side, the review is done, it should be marked "ready for
committer", unless somebody else wants to review.
Thanks. Since, I haven't seen anybody else commenting here and I do not have any further comments to make, I have marked it as "ready for committer".
Best regards,
Etsuro Fujita
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
(2014/08/22 12:58), Alvaro Herrera wrote: > Noah Misch wrote: > >> I'm anticipating a bug report along these lines: >> >> I saw poor estimates involving a child foreign table, so I ran "ANALYZE >> VERBOSE", which reported 'INFO: analyzing "public.parent" inheritance >> tree'. Estimates remained poor, so I scratched my head for awhile before >> noticing that pg_stats didn't report such statistics. I then ran "ANALYZE >> VERBOSE parent", saw the same 'INFO: analyzing "public.parent" inheritance >> tree', but this time saw relevant rows in pg_stats. The message doesn't >> reflect the actual behavior. >> >> I'll sympathize with that complaint. > > Agreed on that. I've got the point. Will fix. Thanks for the comment! Best regards, Etsuro Fujita
Re: Compute attr_needed for child relations (was Re: inherit support for foreign tables)
From
Shigeru Hanada
Date:
Hi Fujita-san,
I reviewed the v4 patch, and here are some comments from me.
* After applying this patch, pull_varattnos() should not called in
unnecessary places. For developers who want list of
columns-to-be-processed for some purpose, it would be nice to mention
when they should use pull_varattnos() and when they should not, maybe
at the comments of pull_varattnos() implementation.
* deparseReturningList() and postgresGetForeignRelSize() in
contrib/postgres_fdw/ also call pull_varattnos() to determine which
column to be in the SELECT clause of remote query. Shouldn't these be
replaced in the same manner? Other pull_varattnos() calls are for
restrictions, so IIUC they can't be replaced.
* Through this review I thought up that lazy evaluation approach might
fit attr_needed. I mean that we leave attr_needed for child relations
empty, and fill it at the first request for it. This would avoid
useless computation of attr_needed for child relations, though this
idea has been considered but thrown away before...
2014-08-20 18:55 GMT+09:00 Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>:
> Hi Ashutish,
>
>
> (2014/08/14 22:30), Ashutosh Bapat wrote:
>>
>> On Thu, Aug 14, 2014 at 10:05 AM, Etsuro Fujita
>> <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:
>>
>> (2014/08/08 18:51), Etsuro Fujita wrote:
>> >>> (2014/06/30 22:48), Tom Lane wrote:
>> >>>> I wonder whether it isn't time to change that. It was coded
>> like that
>> >>>> originally only because calculating the values would've been a
>> waste of
>> >>>> cycles at the time. But this is at least the third place
>> where it'd be
>> >>>> useful to have attr_needed for child rels.
>>
>> > I've revised the patch.
>>
>> There was a problem with the previous patch, which will be described
>> below. Attached is the updated version of the patch addressing that.
>
>
>> Here are some more comments
>
>
> Thank you for the further review!
>
>
>> attr_needed also has the attributes used in the restriction clauses as
>> seen in distribute_qual_to_rels(), so, it looks unnecessary to call
>> pull_varattnos() on the clauses in baserestrictinfo in functions
>> check_selective_binary_conversion(), remove_unused_subquery_outputs(),
>> check_index_only().
>
>
> IIUC, I think it's *necessary* to do that in those functions since the
> attributes used in the restriction clauses in baserestrictinfo are not added
> to attr_needed due the following code in distribute_qual_to_rels.
>
> /*
> * If it's a join clause (either naturally, or because delayed by
> * outer-join rules), add vars used in the clause to targetlists of
> their
> * relations, so that they will be emitted by the plan nodes that scan
> * those relations (else they won't be available at the join node!).
> *
> * Note: if the clause gets absorbed into an EquivalenceClass then this
> * may be unnecessary, but for now we have to do it to cover the case
> * where the EC becomes ec_broken and we end up reinserting the original
> * clauses into the plan.
> */
> if (bms_membership(relids) == BMS_MULTIPLE)
> {
> List *vars = pull_var_clause(clause,
> PVC_RECURSE_AGGREGATES,
> PVC_INCLUDE_PLACEHOLDERS);
>
> add_vars_to_targetlist(root, vars, relids, false);
> list_free(vars);
>
> }
>
>> Although in case of RTE_RELATION, the 0th entry in attr_needed
>> corresponds to FirstLowInvalidHeapAttributeNumber + 1, it's always safer
>> to use it is RelOptInfo::min_attr, in case get_relation_info() wants to
>> change assumption or somewhere down the line some other part of code
>> wants to change attr_needed information. It may be unlikely, that it
>> would change, but it will be better to stick to comments in RelOptInfo
>> 443 AttrNumber min_attr; /* smallest attrno of rel (often
>> <0) */
>> 444 AttrNumber max_attr; /* largest attrno of rel */
>> 445 Relids *attr_needed; /* array indexed [min_attr ..
>> max_attr] */
>
>
> Good point! Attached is the revised version of the patch.
>
>
> Thanks,
>
> Best regards,
> Etsuro Fujita
--
Shigeru HANADA
Re: Compute attr_needed for child relations (was Re: inherit support for foreign tables)
From
Tom Lane
Date:
Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
> [ attr_needed-v4.patch ]
I looked this over, and TBH I'm rather disappointed. The patch adds
150 lines of dubiously-correct code in order to save ... uh, well,
actually it *adds* code, because the places that are supposedly getting
a benefit are changed like this:
*** 799,806 **** check_selective_binary_conversion(RelOptInfo *baserel, } /* Collect all the attributes needed
forjoins or final output. */
! pull_varattnos((Node *) baserel->reltargetlist, baserel->relid,
! &attrs_used); /* Add all the attributes used by restriction clauses. */ foreach(lc,
baserel->baserestrictinfo)
--- 799,810 ---- } /* Collect all the attributes needed for joins or final output. */
! for (i = baserel->min_attr; i <= baserel->max_attr; i++)
! {
! if (!bms_is_empty(baserel->attr_needed[i - baserel->min_attr]))
! attrs_used = bms_add_member(attrs_used,
! i - FirstLowInvalidHeapAttributeNumber);
! } /* Add all the attributes used by restriction clauses. */ foreach(lc, baserel->baserestrictinfo)
That's not simpler, it's not easier to understand, and it's probably not
faster either. We could address some of those complaints by encapsulating
the above loop into a utility function, but still, I come away with the
feeling that it's not worth changing this. Considering that all the
places that are doing this then proceed to use pull_varattnos to add on
attnos from the restriction clauses, it seems like using pull_varattnos
on the reltargetlist isn't such a bad thing after all.
So I'm inclined to reject this. It seemed like a good idea in the
abstract, but the concrete result isn't very attractive, and doesn't
seem like an improvement over what we have.
regards, tom lane
Re: Compute attr_needed for child relations (was Re: inherit support for foreign tables)
From
Etsuro Fujita
Date:
(2014/08/27 3:27), Tom Lane wrote: > Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes: >> [ attr_needed-v4.patch ] > > I looked this over, and TBH I'm rather disappointed. The patch adds > 150 lines of dubiously-correct code in order to save ... uh, well, Just for my study, could you tell me why you think that the code is "dubiously-correct"? > Considering that all the > places that are doing this then proceed to use pull_varattnos to add on > attnos from the restriction clauses, it seems like using pull_varattnos > on the reltargetlist isn't such a bad thing after all. I agree with you on that point. > So I'm inclined to reject this. It seemed like a good idea in the > abstract, but the concrete result isn't very attractive, and doesn't > seem like an improvement over what we have. Okay. I'll withdraw the patch. Thank you for taking the time to review the patch! Best regards, Etsuro Fujita
Re: Compute attr_needed for child relations (was Re: inherit support for foreign tables)
From
Tom Lane
Date:
Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
> (2014/08/27 3:27), Tom Lane wrote:
>> I looked this over, and TBH I'm rather disappointed. The patch adds
>> 150 lines of dubiously-correct code in order to save ... uh, well,
> Just for my study, could you tell me why you think that the code is
> "dubiously-correct"?
It might be fine; I did not actually review the new
adjust_appendrel_attr_needed code in any detail. What's scaring me off it
is (1) it's a lot longer and more complicated than I'd thought it would
be, and (2) you already made several bug fixes in it, which is often an
indicator that additional problems lurk.
It's possible there's some other, simpler, way to compute child
attr_needed arrays that would resolve (1) and (2). However, even if we
had a simple and obviously-correct way to do that, it still seems like
there's not very much benefit to be had after all. So my thought that
this would be worth doing seems wrong, and I must apologize to you for
sending you chasing down a dead end :-(
regards, tom lane
Re: Compute attr_needed for child relations (was Re: inherit support for foreign tables)
From
Etsuro Fujita
Date:
(2014/08/27 11:06), Tom Lane wrote: > Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes: >> (2014/08/27 3:27), Tom Lane wrote: >>> I looked this over, and TBH I'm rather disappointed. The patch adds >>> 150 lines of dubiously-correct code in order to save ... uh, well, > >> Just for my study, could you tell me why you think that the code is >> "dubiously-correct"? > > It might be fine; I did not actually review the new > adjust_appendrel_attr_needed code in any detail. What's scaring me off it > is (1) it's a lot longer and more complicated than I'd thought it would > be, and (2) you already made several bug fixes in it, which is often an > indicator that additional problems lurk. Okay. > It's possible there's some other, simpler, way to compute child > attr_needed arrays that would resolve (1) and (2). However, even if we > had a simple and obviously-correct way to do that, it still seems like > there's not very much benefit to be had after all. So my thought that > this would be worth doing seems wrong, and I must apologize to you for > sending you chasing down a dead end :-( Please don't worry yourself about that! Thanks, Best regards, Etsuro Fujita
(2014/08/22 11:51), Noah Misch wrote: > On Wed, Aug 20, 2014 at 08:11:01PM +0900, Etsuro Fujita wrote: >> (2014/07/02 11:23), Noah Misch wrote: >>> Your chosen ANALYZE behavior is fair, but the messaging from a database-wide >>> ANALYZE VERBOSE needs work: >>> >>> INFO: analyzing "test_foreign_inherit.parent" >>> INFO: "parent": scanned 1 of 1 pages, containing 1 live rows and 0 dead rows; 1 rows in sample, 1 estimated total rows >>> INFO: analyzing "test_foreign_inherit.parent" inheritance tree >>> Please arrange to omit the 'analyzing "tablename" inheritance tree' message, >>> since this ANALYZE actually skipped that task. >> I think it would be better that this is handled in the same way as >> an inheritance tree that turns out to be a singe table that doesn't >> have any descendants in acquire_inherited_sample_rows(). That would >> still output the message as shown below, but I think that that would >> be more consistent with the existing code. The patch works so. > Today's ANALYZE VERBOSE messaging for former inheritance parents (tables with > relhassubclass = true but no pg_inherits.inhparent links) is deceptive, and I > welcome a fix to omit the spurious message. As defects go, this is quite > minor. There's fundamentally no value in collecting inheritance tree > statistics for such a parent, and no PostgreSQL command will do so. > > Your proposed behavior for inheritance parents having at least one foreign > table child is more likely to mislead DBAs in practice. An inheritance tree > genuinely exists, and a different ANALYZE command is quite capable of > collecting statistics on that inheritance tree. Messaging should reflect the > difference between ANALYZE invocations that do such work and ANALYZE > invocations that skip it. I'm anticipating a bug report along these lines: > > I saw poor estimates involving a child foreign table, so I ran "ANALYZE > VERBOSE", which reported 'INFO: analyzing "public.parent" inheritance > tree'. Estimates remained poor, so I scratched my head for awhile before > noticing that pg_stats didn't report such statistics. I then ran "ANALYZE > VERBOSE parent", saw the same 'INFO: analyzing "public.parent" inheritance > tree', but this time saw relevant rows in pg_stats. The message doesn't > reflect the actual behavior. > > I'll sympathize with that complaint. It's a minor point overall, but the code > impact is, I predict, small enough that we may as well get it right. A > credible alternative is to emit a second message indicating that we skipped > the inheritance tree statistics after all, and why we skipped them. I'd like to address this by emitting the second message as shown below: INFO: analyzing "public.parent" INFO: "parent": scanned 0 of 0 pages, containing 0 live rows and 0 dead rows; 0 rows in sample, 0 estimated total rows INFO: analyzing "public.parent" inheritance tree INFO: skipping analyze of "public.parent" inheritance tree --- this inheritance tree contains foreign tables Attached is the update version of the patch. Based on the result of discussions about attr_needed upthread, I've changed the patch so that create_foreignscan_plan makes reference to reltargetlist, not to attr_needed. (So, the patch in [1] isn't required, anymore.) Other changes: * Revise code/comments/docs a bit * Add more regression tests A separate patch (analyze.patch) handles the former case in a similar way. [1] http://www.postgresql.org/message-id/53F4707C.8030904@lab.ntt.co.jp Thanks, Best regards, Etsuro Fujita
Attachment
Hello, I have a request with slight significance for the messages. > I'd like to address this by emitting the second message as shown below: > > INFO: analyzing "public.parent" > INFO: "parent": scanned 0 of 0 pages, containing 0 live rows and 0 dead > rows; 0 rows in sample, 0 estimated total rows > INFO: analyzing "public.parent" inheritance tree > INFO: skipping analyze of "public.parent" inheritance tree --- this > inheritance tree contains foreign tables In acquire_inherited_sample_rows(), the message below is emitted when the parent explicitly specified in analyze command has at least one foreign tables. > "skipping analyze of \"%s.%s\" inheritance tree --- this > inheritance tree contains foreign tables" This message implicitly asserts (for me) that "A inheritance tree containing at least one foreign tables *always* cannot be analyzed" but in reality, we can let it go by specifying the parent table explicitly. For example, the additional HINT or DETAIL message would clarify that. > INFO: analyzing "public.parent" inheritance tree > INFO: skipping analyze of "public.parent" inheritance tree --- this > inheritance tree contains foreign tables + HINT: You can analyze this inheritance tree by specifying "public.parent" to analze command -- Kyotaro Horiguchi NTT Open Source Software Center
I had a cursory look at this patch and the discussions around this. ISTM there are actually two new features in this: 1) allow CHECK constraints on foreign tables, and 2) support inheritance for foreign tables. How about splitting it into two? - Heikki
(2014/09/11 4:32), Heikki Linnakangas wrote: > I had a cursory look at this patch and the discussions around this. Thank you! > ISTM there are actually two new features in this: 1) allow CHECK > constraints on foreign tables, and 2) support inheritance for foreign > tables. How about splitting it into two? That's right. There are the two in this patch. I'm not sure if I should split the patch into the two, because 1) won't make sense without 2). As described in the following note added to the docs on CREATE FOREIGN TABLE, CHECK constraints on foreign tables are intended to support constraint exclusion for partitioned foreign tables: + Constraints on foreign tables are not enforced on insert or update. + Those definitions simply declare the constraints hold for all rows + in the foreign tables. It is the user's responsibility to ensure + that those definitions match the remote side. Such constraints are + used for some kind of query optimization such as constraint exclusion + for partitioned tables Thanks, Best regards, Etsuro Fujita
On 09/11/2014 12:22 PM, Etsuro Fujita wrote: > (2014/09/11 4:32), Heikki Linnakangas wrote: >> I had a cursory look at this patch and the discussions around this. > > Thank you! > >> ISTM there are actually two new features in this: 1) allow CHECK >> constraints on foreign tables, and 2) support inheritance for foreign >> tables. How about splitting it into two? > > That's right. There are the two in this patch. > > I'm not sure if I should split the patch into the two, because 1) won't > make sense without 2). As described in the following note added to the > docs on CREATE FOREIGN TABLE, CHECK constraints on foreign tables are > intended to support constraint exclusion for partitioned foreign tables: > > + Constraints on foreign tables are not enforced on insert or update. > + Those definitions simply declare the constraints hold for all rows > + in the foreign tables. It is the user's responsibility to ensure > + that those definitions match the remote side. Such constraints are > + used for some kind of query optimization such as constraint exclusion > + for partitioned tables The planner can do constraint exclusion based on CHECK constraints even without inheritance. It's not enabled by default because it can increase planning time, but if you set constraint_exclusion=on, it will work. For example: postgres=# create table foo (i int4 CHECK (i > 0)); CREATE TABLE postgres=# explain select * from foo WHERE i < 0; QUERY PLAN ------------------------------------------------------ Seq Scan on foo (cost=0.00..40.00 rows=800 width=4) Filter: (i< 0) Planning time: 0.335 ms (3 rows) postgres=# show constraint_exclusion ; constraint_exclusion ---------------------- partition (1 row) postgres=# set constraint_exclusion ='on'; SET postgres=# explain select * from foo WHERE i < 0; QUERY PLAN ------------------------------------------ Result (cost=0.00..0.01 rows=1 width=0) One-Time Filter: false Planning time:0.254 ms (3 rows) postgres=# - Heikki
(2014/09/11 19:46), Heikki Linnakangas wrote: > On 09/11/2014 12:22 PM, Etsuro Fujita wrote: >> (2014/09/11 4:32), Heikki Linnakangas wrote: >>> I had a cursory look at this patch and the discussions around this. >> >> Thank you! >> >>> ISTM there are actually two new features in this: 1) allow CHECK >>> constraints on foreign tables, and 2) support inheritance for foreign >>> tables. How about splitting it into two? >> >> That's right. There are the two in this patch. >> >> I'm not sure if I should split the patch into the two, because 1) won't >> make sense without 2). As described in the following note added to the >> docs on CREATE FOREIGN TABLE, CHECK constraints on foreign tables are >> intended to support constraint exclusion for partitioned foreign tables: >> >> + Constraints on foreign tables are not enforced on insert or update. >> + Those definitions simply declare the constraints hold for all rows >> + in the foreign tables. It is the user's responsibility to ensure >> + that those definitions match the remote side. Such constraints are >> + used for some kind of query optimization such as constraint >> exclusion >> + for partitioned tables > > The planner can do constraint exclusion based on CHECK constraints even > without inheritance. It's not enabled by default because it can increase > planning time, but if you set constraint_exclusion=on, it will work. Exactly! > For example: > > postgres=# create table foo (i int4 CHECK (i > 0)); > CREATE TABLE > postgres=# explain select * from foo WHERE i < 0; > QUERY PLAN > ------------------------------------------------------ > Seq Scan on foo (cost=0.00..40.00 rows=800 width=4) > Filter: (i < 0) > Planning time: 0.335 ms > (3 rows) > > postgres=# show constraint_exclusion ; > constraint_exclusion > ---------------------- > partition > (1 row) > > postgres=# set constraint_exclusion ='on'; > SET > postgres=# explain select * from foo WHERE i < 0; > QUERY PLAN > ------------------------------------------ > Result (cost=0.00..0.01 rows=1 width=0) > One-Time Filter: false > Planning time: 0.254 ms > (3 rows) > > postgres=# Actually, this patch allows the exact same thing to apply to foreign tables. My explanation was insufficient about that. Sorry for that. So, should I split the patch into the two? Thanks, Best regards, Etsuro Fujita
On 09/11/2014 02:30 PM, Etsuro Fujita wrote: > Actually, this patch allows the exact same thing to apply to foreign > tables. My explanation was insufficient about that. Sorry for that. Great, that's what I thought. > So, should I split the patch into the two? Yeah, please do. - Heikki
(2014/09/11 20:51), Heikki Linnakangas wrote: > On 09/11/2014 02:30 PM, Etsuro Fujita wrote: >> So, should I split the patch into the two? > > Yeah, please do. OK, Will do. Thanks, Best regards, Etsuro Fujita
(2014/09/12 16:30), Etsuro Fujita wrote: > (2014/09/11 20:51), Heikki Linnakangas wrote: >> On 09/11/2014 02:30 PM, Etsuro Fujita wrote: >>> So, should I split the patch into the two? >> >> Yeah, please do. > > OK, Will do. Here are separated patches. fdw-chk.patch - CHECK constraints on foreign tables fdw-inh.patch - table inheritance with foreign tables The latter has been created on top of [1]. I've addressed the comment from Horiguchi-san [2] in it also, though I've slightly modified his proposal. There is the functionality for path reparameterization for ForeignScan in the previous patch, but since the functionality would be useful not only for such table inheritance cases but for UNION ALL cases, I'd add the functionality as an independent feature maybe to CF 2014-12. Thanks, [1] http://www.postgresql.org/message-id/540DA168.3040407@lab.ntt.co.jp [2] http://www.postgresql.org/message-id/20140902.142218.253402812.horiguchi.kyotaro@lab.ntt.co.jp Best regards, Etsuro Fujita
Attachment
(2014/08/28 18:00), Etsuro Fujita wrote: > (2014/08/22 11:51), Noah Misch wrote: >> Today's ANALYZE VERBOSE messaging for former inheritance parents >> (tables with >> relhassubclass = true but no pg_inherits.inhparent links) is >> deceptive, and I >> welcome a fix to omit the spurious message. As defects go, this is quite >> minor. There's fundamentally no value in collecting inheritance tree >> statistics for such a parent, and no PostgreSQL command will do so. >> A >> credible alternative is to emit a second message indicating that we >> skipped >> the inheritance tree statistics after all, and why we skipped them. > I'd like to address this by emitting the second message as shown below: > A separate patch (analyze.patch) handles the former case in a similar way. I'll add to the upcoming CF, the analyze.patch as an independent item, which emits a second message indicating that we skipped the inheritance tree statistics and why we skipped them. Thanks, Best regards, Etsuro Fujita
Attachment
(2014/10/14 20:00), Etsuro Fujita wrote: > Here are separated patches. > > fdw-chk.patch - CHECK constraints on foreign tables > fdw-inh.patch - table inheritance with foreign tables > > The latter has been created on top of [1]. > [1] http://www.postgresql.org/message-id/540DA168.3040407@lab.ntt.co.jp To be exact, it has been created on top of [1] and fdw-chk.patch. I noticed that the latter disallows TRUNCATE on inheritance trees that contain at least one child foreign table. But I think it would be better to allow it, with the semantics that we quietly ignore the child foreign tables and apply the operation to the child plain tables, which is the same semantics as ALTER COLUMN SET STORAGE on such inheritance trees. Comments welcome. Thanks, Best regards, Etsuro Fujita
(2014/10/21 17:40), Etsuro Fujita wrote: > (2014/10/14 20:00), Etsuro Fujita wrote: >> Here are separated patches. >> >> fdw-chk.patch - CHECK constraints on foreign tables >> fdw-inh.patch - table inheritance with foreign tables >> >> The latter has been created on top of [1]. > >> [1] http://www.postgresql.org/message-id/540DA168.3040407@lab.ntt.co.jp > > To be exact, it has been created on top of [1] and fdw-chk.patch. > > I noticed that the latter disallows TRUNCATE on inheritance trees that > contain at least one child foreign table. But I think it would be > better to allow it, with the semantics that we quietly ignore the child > foreign tables and apply the operation to the child plain tables, which > is the same semantics as ALTER COLUMN SET STORAGE on such inheritance > trees. Comments welcome. Done. And I've also a bit revised regression tests for both patches. Patches attached. Thanks, Best regards, Etsuro Fujita
Attachment
Hello, I don't fully catch up this topic but tried this one. > >> Here are separated patches. > >> > >> fdw-chk.patch - CHECK constraints on foreign tables > >> fdw-inh.patch - table inheritance with foreign tables > >> > >> The latter has been created on top of [1]. > > > >> [1] > >> http://www.postgresql.org/message-id/540DA168.3040407@lab.ntt.co.jp > > > To be exact, it has been created on top of [1] and fdw-chk.patch. I tried both patches on the current head, the newly added parameter to analyze_rel() hampered them from applying but it is easy to fix seemingly and almost all the other part was applied cleanly. By the way, are these the result of simply splitting of your last patch, foreign_inherit-v15.patch? http://www.postgresql.org/message-id/53FEEF94.6040101@lab.ntt.co.jp The result of apllying whole-in-one version and this splitted version seem to have many differences. Did you added even other changes? Or do I understand this patch wrongly? git diff --numstat 0_foreign_inherit_one 0_foreign_inherit_two 5 51 contrib/file_fdw/file_fdw.c 10 19 contrib/file_fdw/input/file_fdw.source 18 71 contrib/file_fdw/output/file_fdw.source 19 70 contrib/postgres_fdw/expected/postgres_fdw.out 9 66 contrib/postgres_fdw/postgres_fdw.c 12 35 contrib/postgres_fdw/sql/postgres_fdw.sql 13 48 doc/src/sgml/fdwhandler.sgml 39 39 doc/src/sgml/ref/alter_foreign_table.sgml 4 3 doc/src/sgml/ref/create_foreign_table.sgml 8 0 src/backend/catalog/heap.c 7 3 src/backend/commands/analyze.c 0 7 src/backend/commands/tablecmds.c 1 22 src/backend/optimizer/plan/createplan.c 7 7 src/backend/optimizer/prep/prepunion.c 0 26 src/backend/optimizer/util/pathnode.c 1 1 src/include/commands/vacuum.h 0 7 src/include/foreign/fdwapi.h 19 1 src/test/regress/expected/foreign_data.out 9 2 src/test/regress/sql/foreign_data.sql > > I noticed that the latter disallows TRUNCATE on inheritance trees that > > contain at least one child foreign table. But I think it would be > > better to allow it, with the semantics that we quietly ignore the > > child > > foreign tables and apply the operation to the child plain tables, > > which > > is the same semantics as ALTER COLUMN SET STORAGE on such inheritance > > trees. Comments welcome. > > Done. And I've also a bit revised regression tests for both > patches. Patches attached. regards, -- Kyotaro Horiguchi NTT Open Source Software Center
> (2014/08/28 18:00), Etsuro Fujita wrote: > > (2014/08/22 11:51), Noah Misch wrote: > >> Today's ANALYZE VERBOSE messaging for former inheritance parents > >> (tables with relhassubclass = true but no pg_inherits.inhparent > >> links) is deceptive, and I welcome a fix to omit the spurious > >> message. As defects go, this is quite minor. There's fundamentally > >> no value in collecting inheritance tree statistics for such a parent, > >> and no PostgreSQL command will do so. > > >> A > >> credible alternative is to emit a second message indicating that we > >> skipped the inheritance tree statistics after all, and why we skipped > >> them. > > > I'd like to address this by emitting the second message as shown below: > > > A separate patch (analyze.patch) handles the former case in a similar > way. > > I'll add to the upcoming CF, the analyze.patch as an independent item, > which emits a second message indicating that we skipped the inheritance > tree statistics and why we skipped them. I did a review of the patch. There was no problem. I confirmed the following. 1. applied cleanly and compilation was without warnings and errors 2. all regress tests was passed ok 3. The message output from ANALYZE VERBOSE. Following are the SQL which I used to check messages. create table parent (id serial); create table child (name text) inherits (parent); ANALYZE VERBOSE parent ; drop table child ; ANALYZE VERBOSE parent ; Regards, -- Furuya Osamu
Hi Furuya-san, (2014/11/07 16:54), furuyao@pm.nttdata.co.jp wrote: >> (2014/08/28 18:00), Etsuro Fujita wrote: >>> (2014/08/22 11:51), Noah Misch wrote: >>>> Today's ANALYZE VERBOSE messaging for former inheritance parents >>>> (tables with relhassubclass = true but no pg_inherits.inhparent >>>> links) is deceptive, and I welcome a fix to omit the spurious >>>> message. As defects go, this is quite minor. There's fundamentally >>>> no value in collecting inheritance tree statistics for such a parent, >>>> and no PostgreSQL command will do so. >> >>>> A >>>> credible alternative is to emit a second message indicating that we >>>> skipped the inheritance tree statistics after all, and why we skipped >>>> them. >> >>> I'd like to address this by emitting the second message as shown below: >> >>> A separate patch (analyze.patch) handles the former case in a similar >> way. >> >> I'll add to the upcoming CF, the analyze.patch as an independent item, >> which emits a second message indicating that we skipped the inheritance >> tree statistics and why we skipped them. > > I did a review of the patch. > There was no problem. > I confirmed the following. > > 1. applied cleanly and compilation was without warnings and errors > 2. all regress tests was passed ok > 3. The message output from ANALYZE VERBOSE. > > Following are the SQL which I used to check messages. > > create table parent (id serial); > create table child (name text) inherits (parent); > ANALYZE VERBOSE parent ; > drop table child ; > ANALYZE VERBOSE parent ; I think that that is a correct test for this patch. Thanks for the review! Best regards, Etsuro Fujita
(2014/11/07 14:57), Kyotaro HORIGUCHI wrote: >>>> Here are separated patches. >>>> >>>> fdw-chk.patch - CHECK constraints on foreign tables >>>> fdw-inh.patch - table inheritance with foreign tables >>>> >>>> The latter has been created on top of [1]. >>> >>>> [1] >>>> http://www.postgresql.org/message-id/540DA168.3040407@lab.ntt.co.jp >> >>> To be exact, it has been created on top of [1] and fdw-chk.patch. > > I tried both patches on the current head, the newly added > parameter to analyze_rel() hampered them from applying but it is > easy to fix seemingly and almost all the other part was applied > cleanly. Thanks for the review! > By the way, are these the result of simply splitting of your last > patch, foreign_inherit-v15.patch? > > http://www.postgresql.org/message-id/53FEEF94.6040101@lab.ntt.co.jp The answer is "no". > The result of apllying whole-in-one version and this splitted > version seem to have many differences. Did you added even other > changes? Or do I understand this patch wrongly? As I said before, I splitted the whole-in-one version into three: 1) CHECK constraint patch (ie fdw-chk.patch), 2) table inheritance patch (ie fdw-inh.patch) and 3) path reparameterization patch (not posted). In addition to that, I slightly modified #1 and #2. IIUC, #3 would be useful not only for the inheritance cases but for union all cases. So, I plan to propose it independently in the next CF. >>> I noticed that the latter disallows TRUNCATE on inheritance trees that >>> contain at least one child foreign table. But I think it would be >>> better to allow it, with the semantics that we quietly ignore the >>> child >>> foreign tables and apply the operation to the child plain tables, >>> which >>> is the same semantics as ALTER COLUMN SET STORAGE on such inheritance >>> trees. Comments welcome. >> >> Done. And I've also a bit revised regression tests for both >> patches. Patches attached. I rebased the patches to the latest head. Here are updated patches. Other changes: * fdw-chk-3.patch: the updated patch revises some ereport messages a little bit. * fdw-inh-3.patch: I noticed that there is a doc bug in the previous patch. The updated patch fixes that, adds a bit more docs, and revises regression tests in foreign_data.sql a bit further. Thanks, Best regards, Etsuro Fujita
Attachment
Hi Fujita-san,
I reviewed fdw-chk-3 patch. Here are my comments--------
1. The patch applies on the latest master using "patch but not by git apply
-------
4. In test foreign_data there are changes to fix the diffs caused by these changes like below
ALTER FOREIGN TABLE ft1 DROP CONSTRAINT no_const; -- ERROR
-ERROR: "ft1" is not a table
+ERROR: constraint "no_const" of relation "ft1" does not exist
ALTER FOREIGN TABLE ft1 DROP CONSTRAINT IF EXISTS no_const;
-ERROR: "ft1" is not a table
+NOTICE: constraint "no_const" of relation "ft1" does not exist, skipping
ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c1_check;
-ERROR: "ft1" is not a table
+ERROR: constraint "ft1_c1_check" of relation "ft1" does not exist
ALTER FOREIGN TABLE ft1 DROP CONSTRAINT no_const; -- ERROR
-ERROR: "ft1" is not a table
+ERROR: constraint "no_const" of relation "ft1" does not exist
ALTER FOREIGN TABLE ft1 DROP CONSTRAINT IF EXISTS no_const;
-ERROR: "ft1" is not a table
+NOTICE: constraint "no_const" of relation "ft1" does not exist, skipping
ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c1_check;
-ERROR: "ft1" is not a table
+ERROR: constraint "ft1_c1_check" of relation "ft1" does not exist
Earlier when constraints were not supported for FOREIGN TABLE, these tests made sure the same functionality. So, even though the corresponding constraints were not created on the table (in fact it didn't allow the creation as well). Now that the constraints are allowed, I think the tests for "no_const" (without IF EXISTS) and "ft1_c1_check" are duplicating the same testcase. May be we should review this set of statement in the light of new functionality.
----------------------------------
On Fri, Nov 7, 2014 at 5:31 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/11/07 14:57), Kyotaro HORIGUCHI wrote:Here are separated patches.
fdw-chk.patch - CHECK constraints on foreign tables
fdw-inh.patch - table inheritance with foreign tables
The latter has been created on top of [1].[1]
http://www.postgresql.org/message-id/540DA168.3040407@lab.ntt.co.jpTo be exact, it has been created on top of [1] and fdw-chk.patch.
I tried both patches on the current head, the newly added
parameter to analyze_rel() hampered them from applying but it is
easy to fix seemingly and almost all the other part was applied
cleanly.
Thanks for the review!By the way, are these the result of simply splitting of your last
patch, foreign_inherit-v15.patch?
http://www.postgresql.org/message-id/53FEEF94.6040101@lab.ntt.co.jp
The answer is "no".The result of apllying whole-in-one version and this splitted
version seem to have many differences. Did you added even other
changes? Or do I understand this patch wrongly?
As I said before, I splitted the whole-in-one version into three: 1) CHECK constraint patch (ie fdw-chk.patch), 2) table inheritance patch (ie fdw-inh.patch) and 3) path reparameterization patch (not posted). In addition to that, I slightly modified #1 and #2.
IIUC, #3 would be useful not only for the inheritance cases but for union all cases. So, I plan to propose it independently in the next CF.I noticed that the latter disallows TRUNCATE on inheritance trees that
contain at least one child foreign table. But I think it would be
better to allow it, with the semantics that we quietly ignore the
child
foreign tables and apply the operation to the child plain tables,
which
is the same semantics as ALTER COLUMN SET STORAGE on such inheritance
trees. Comments welcome.
Done. And I've also a bit revised regression tests for both
patches. Patches attached.
I rebased the patches to the latest head. Here are updated patches.
Other changes:
* fdw-chk-3.patch: the updated patch revises some ereport messages a little bit.
* fdw-inh-3.patch: I noticed that there is a doc bug in the previous patch. The updated patch fixes that, adds a bit more docs, and revises regression tests in foreign_data.sql a bit further.
Thanks,
Best regards,
Etsuro Fujita
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Hi Fujita-san,
I tried to apply fdw-inh-3.patch on the latest head from master branch. It failed to apply using both patch and git apply."patch" failed to apply because of rejections in contrib/file_fdw/output/file_fdw.source and doc/src/sgml/ref/create_foreign_table.sgml
On Fri, Nov 7, 2014 at 5:31 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/11/07 14:57), Kyotaro HORIGUCHI wrote:Here are separated patches.
fdw-chk.patch - CHECK constraints on foreign tables
fdw-inh.patch - table inheritance with foreign tables
The latter has been created on top of [1].[1]
http://www.postgresql.org/message-id/540DA168.3040407@lab.ntt.co.jpTo be exact, it has been created on top of [1] and fdw-chk.patch.
I tried both patches on the current head, the newly added
parameter to analyze_rel() hampered them from applying but it is
easy to fix seemingly and almost all the other part was applied
cleanly.
Thanks for the review!By the way, are these the result of simply splitting of your last
patch, foreign_inherit-v15.patch?
http://www.postgresql.org/message-id/53FEEF94.6040101@lab.ntt.co.jp
The answer is "no".The result of apllying whole-in-one version and this splitted
version seem to have many differences. Did you added even other
changes? Or do I understand this patch wrongly?
As I said before, I splitted the whole-in-one version into three: 1) CHECK constraint patch (ie fdw-chk.patch), 2) table inheritance patch (ie fdw-inh.patch) and 3) path reparameterization patch (not posted). In addition to that, I slightly modified #1 and #2.
IIUC, #3 would be useful not only for the inheritance cases but for union all cases. So, I plan to propose it independently in the next CF.I noticed that the latter disallows TRUNCATE on inheritance trees that
contain at least one child foreign table. But I think it would be
better to allow it, with the semantics that we quietly ignore the
child
foreign tables and apply the operation to the child plain tables,
which
is the same semantics as ALTER COLUMN SET STORAGE on such inheritance
trees. Comments welcome.
Done. And I've also a bit revised regression tests for both
patches. Patches attached.
I rebased the patches to the latest head. Here are updated patches.
Other changes:
* fdw-chk-3.patch: the updated patch revises some ereport messages a little bit.
* fdw-inh-3.patch: I noticed that there is a doc bug in the previous patch. The updated patch fixes that, adds a bit more docs, and revises regression tests in foreign_data.sql a bit further.
Thanks,
Best regards,
Etsuro Fujita
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Hi Ashutosh, Thanks for the review! (2014/11/13 15:23), Ashutosh Bapat wrote: > I tried to apply fdw-inh-3.patch on the latest head from master branch. > It failed to apply using both patch and git apply. > > "patch" failed to apply because of rejections in > contrib/file_fdw/output/file_fdw.source and > doc/src/sgml/ref/create_foreign_table.sgml As I said upthread, fdw-inh-3.patch has been created on top of [1] and fdw-chk-3.patch. Did you apply these patche first? [1] https://commitfest.postgresql.org/action/patch_view?id=1599 Best regards, Etsuro Fujita
On Thu, Nov 13, 2014 at 12:20 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
Hi Ashutosh,
Thanks for the review!
(2014/11/13 15:23), Ashutosh Bapat wrote:I tried to apply fdw-inh-3.patch on the latest head from master branch.
It failed to apply using both patch and git apply.
"patch" failed to apply because of rejections in
contrib/file_fdw/output/file_fdw.source and
doc/src/sgml/ref/create_foreign_table.sgml
As I said upthread, fdw-inh-3.patch has been created on top of [1] and fdw-chk-3.patch. Did you apply these patche first?
Oh, sorry, I didn't pay attention to that. I will apply both the patches and review the inheritance patch. Thanks for pointing that out.
[1] https://commitfest.postgresql.org/action/patch_view?id=1599
Best regards,
Etsuro Fujita
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
(2014/11/12 20:04), Ashutosh Bapat wrote: > I reviewed fdw-chk-3 patch. Here are my comments Thanks for the review! > Tests > ------- > 1. The tests added in file_fdw module look good. We should add tests for > CREATE TABLE with CHECK CONSTRAINT also. Agreed. I added the tests, and also changed the proposed tests a bit. > 2. For postgres_fdw we need tests to check the behaviour in case the > constraints mismatch between the remote table and its local foreign > table declaration in case of INSERT, UPDATE and SELECT. Done. > 3. In the testcases for postgres_fdw it seems that you have forgotten to > add statement after SET constraint_exclusion to 'partition' I added the statement. > 4. In test foreign_data there are changes to fix the diffs caused by > these changes like below > ALTER FOREIGN TABLE ft1 DROP CONSTRAINT no_const; -- ERROR > -ERROR: "ft1" is not a table > +ERROR: constraint "no_const" of relation "ft1" does not exist > ALTER FOREIGN TABLE ft1 DROP CONSTRAINT IF EXISTS no_const; > -ERROR: "ft1" is not a table > +NOTICE: constraint "no_const" of relation "ft1" does not exist, skipping > ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c1_check; > -ERROR: "ft1" is not a table > +ERROR: constraint "ft1_c1_check" of relation "ft1" does not exist > Earlier when constraints were not supported for FOREIGN TABLE, these > tests made sure the same functionality. So, even though the > corresponding constraints were not created on the table (in fact it > didn't allow the creation as well). Now that the constraints are > allowed, I think the tests for "no_const" (without IF EXISTS) and > "ft1_c1_check" are duplicating the same testcase. May be we should > review this set of statement in the light of new functionality. Agreed. I removed the "DROP CONSTRAINT ft1_c1_check" test, and added a new test for ALTER CONSTRAINT. > Code and implementation > ---------------------------------- > The usage of NO INHERIT and NOT VALID with CONSTRAINT on foreign table > is blocked, but corresponding documentation entry doesn't mention so. > Since foreign tables can not be inherited NO INHERIT option isn't > applicable to foreign tables and the constraints on the foreign tables > are declarative, hence NOT VALID option is also not applicable. So, I > agree with what the code is doing, but that should be reflected in > documentation with this explanation. > Rest of the code modifies the condition checks for CHECK CONSTRAINTs on > foreign tables, and it looks good to me. Done. Other changes: * Modified one error message that I added in AddRelationNewConstraints, to match the other message there. * Revised other docs a little bit. Attached is an updated version of the patch. Thanks, Best regards, Etsuro Fujita
Attachment
Hi Fujita-san,
Here are my review comments for patch fdw-inh-3.patch.Sanity
--------
-------
--------------------
- We will refer to the child tables as partitions, though they
- are in every way normal <productname>PostgreSQL</> tables.
+ We will refer to the child tables as partitions, though we assume
+ that they are normal <productname>PostgreSQL</> tables.
+ Those constraints are used in some kind of query optimization such
+ as constraint exclusion for partitioned tables (see
+ <xref linkend="ddl-partitioning">).
-------
+/*
* acquire_inherited_sample_rows -- acquire sample rows from inheritance tree
*
* This has the same API as acquire_sample_rows, except that rows are
* collected from all inheritance children as well as the specified table.
- * We fail and return zero if there are no inheritance children.
+ * We fail and return zero if there are no inheritance children or there are
+ * inheritance children that foreign tables.
+CREATE FOREIGN TABLE ft2 () INHERITS (pt1)
+ SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted" 'value');
+ALTER FOREIGN TABLE ft2 NO INHERIT pt1;
+DROP FOREIGN TABLE ft2;
+CREATE FOREIGN TABLE ft2 (
+ c1 integer NOT NULL,
+ c2 text,
+ c3 date
+) SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted" 'value');
+ALTER FOREIGN TABLE ft2 INHERIT pt1;
Rest of the changes look good.
On Thu, Nov 13, 2014 at 12:21 PM, Ashutosh Bapat <ashutosh.bapat@enterprisedb.com> wrote:
On Thu, Nov 13, 2014 at 12:20 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:Hi Ashutosh,
Thanks for the review!
(2014/11/13 15:23), Ashutosh Bapat wrote:I tried to apply fdw-inh-3.patch on the latest head from master branch.
It failed to apply using both patch and git apply.
"patch" failed to apply because of rejections in
contrib/file_fdw/output/file_fdw.source and
doc/src/sgml/ref/create_foreign_table.sgml
As I said upthread, fdw-inh-3.patch has been created on top of [1] and fdw-chk-3.patch. Did you apply these patche first?Oh, sorry, I didn't pay attention to that. I will apply both the patches and review the inheritance patch. Thanks for pointing that out.[1] https://commitfest.postgresql.org/action/patch_view?id=1599
Best regards,
Etsuro Fujita
--Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
On Mon, Nov 17, 2014 at 1:25 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/11/12 20:04), Ashutosh Bapat wrote:I reviewed fdw-chk-3 patch. Here are my comments
Thanks for the review!Tests
-------
1. The tests added in file_fdw module look good. We should add tests for
CREATE TABLE with CHECK CONSTRAINT also.
Agreed. I added the tests, and also changed the proposed tests a bit.2. For postgres_fdw we need tests to check the behaviour in case the
constraints mismatch between the remote table and its local foreign
table declaration in case of INSERT, UPDATE and SELECT.
Done.3. In the testcases for postgres_fdw it seems that you have forgotten to
add statement after SET constraint_exclusion to 'partition'
I added the statement.4. In test foreign_data there are changes to fix the diffs caused by
these changes like below
ALTER FOREIGN TABLE ft1 DROP CONSTRAINT no_const; -- ERROR
-ERROR: "ft1" is not a table
+ERROR: constraint "no_const" of relation "ft1" does not exist
ALTER FOREIGN TABLE ft1 DROP CONSTRAINT IF EXISTS no_const;
-ERROR: "ft1" is not a table
+NOTICE: constraint "no_const" of relation "ft1" does not exist, skipping
ALTER FOREIGN TABLE ft1 DROP CONSTRAINT ft1_c1_check;
-ERROR: "ft1" is not a table
+ERROR: constraint "ft1_c1_check" of relation "ft1" does not existEarlier when constraints were not supported for FOREIGN TABLE, these
tests made sure the same functionality. So, even though the
corresponding constraints were not created on the table (in fact it
didn't allow the creation as well). Now that the constraints are
allowed, I think the tests for "no_const" (without IF EXISTS) and
"ft1_c1_check" are duplicating the same testcase. May be we should
review this set of statement in the light of new functionality.
Agreed. I removed the "DROP CONSTRAINT ft1_c1_check" test, and added a new test for ALTER CONSTRAINT.Code and implementation
----------------------------------
The usage of NO INHERIT and NOT VALID with CONSTRAINT on foreign table
is blocked, but corresponding documentation entry doesn't mention so.
Since foreign tables can not be inherited NO INHERIT option isn't
applicable to foreign tables and the constraints on the foreign tables
are declarative, hence NOT VALID option is also not applicable. So, I
agree with what the code is doing, but that should be reflected in
documentation with this explanation.
Rest of the code modifies the condition checks for CHECK CONSTRAINTs on
foreign tables, and it looks good to me.
Done.
Other changes:
* Modified one error message that I added in AddRelationNewConstraints, to match the other message there.
* Revised other docs a little bit.
Attached is an updated version of the patch.
I looked at the patch. It looks good now. Once we have the inh patch ready, we can mark this item as ready for commiter.
Thanks,
Best regards,
Etsuro Fujita
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
(2014/11/18 18:09), Ashutosh Bapat wrote: > I looked at the patch. It looks good now. Once we have the inh patch > ready, we can mark this item as ready for commiter. Thanks for the review! Best regards, Etsuro Fujita
(2014/11/17 17:55), Ashutosh Bapat wrote:
> Here are my review comments for patch fdw-inh-3.patch.
Thanks for the review!
> Tests
> -------
> 1. It seems like you have copied from testcase inherit.sql to
> postgres_fdw testcase. That's a good thing, but it makes the test quite
> long. May be we should have two tests in postgres_fdw contrib module,
> one for simple cases, and other for inheritance. What do you say?
IMO, the test is not so time-consuming, so it doesn't seem worth
splitting it into two.
> Documentation
> --------------------
> 1. The change in ddl.sgml
> - We will refer to the child tables as partitions, though they
> - are in every way normal <productname>PostgreSQL</> tables.
> + We will refer to the child tables as partitions, though we assume
> + that they are normal <productname>PostgreSQL</> tables.
>
> adds phrase "we assume that", which confuses the intention behind the
> sentence. The original text is intended to highlight the equivalence
> between "partition" and "normal table", where as the addition esp. the
> word "assume" weakens that equivalence. Instead now we have to highlight
> the equivalence between "partition" and "normal or foreign table". The
> wording similar to "though they are either normal or foreign tables"
> should be used there.
You are right, but I feel that there is something out of place in saying
that there (5.10.2. Implementing Partitioning) because the procedure
there has been written based on normal tables only. Put another way, if
we say that, I think we'd need to add more docs, describing the syntax
and/or the corresponding examples for foreign-table cases. But I'd like
to leave that for another patch. So, how about the wording "we assume
*here* that", instead of "we assume that", as I added the following
notice in the previous section (5.10.1. Overview)?
@@ -2650,7 +2669,10 @@ VALUES ('Albany', NULL, NULL, 'NY'); table of a single parent table. The parent table
itselfis normally empty; it exists just to represent the entire data set. You should be familiar with
inheritance(see <xref linkend="ddl-inherit">) before
- attempting to set up partitioning.
+ attempting to set up partitioning. (The setup and management of
+ partitioned tables illustrated in this section assume that each
+ partition is a normal table. However, you can do that in a similar way
+ for cases where some or all partitions are foreign tables.)
> 2. The wording "some kind of optimization" gives vague picture. May be
> it can be worded as "Since the constraints are assumed to be true, they
> are used in constraint-based query optimization like constraint
> exclusion for partitioned tables.".
> + Those constraints are used in some kind of query optimization such
> + as constraint exclusion for partitioned tables (see
> + <xref linkend="ddl-partitioning">).
Will follow your revision.
> Code
> -------
> 1. In the following change
> +/*
> * acquire_inherited_sample_rows -- acquire sample rows from
> inheritance tree
> *
> * This has the same API as acquire_sample_rows, except that rows are
> * collected from all inheritance children as well as the specified table.
> - * We fail and return zero if there are no inheritance children.
> + * We fail and return zero if there are no inheritance children or
> there are
> + * inheritance children that foreign tables.
>
> The addition should be "there are inheritance children that *are all
> *foreign tables. Note the addition "are all".
Sorry, I incorrectly wrote the comment. What I tried to write is "We
fail and return zero if there are no inheritance children or if we are
not in VAC_MODE_SINGLE case and inheritance tree contains at least one
foreign table.".
> 2. The function has_foreign() be better named has_foreign_child()? This
How about "has_foreign_table"?
> function loops through all the tableoids passed even after it has found
> a foreign table. Why can't we return true immediately after finding the
> first foreign table, unless the side effects of heap_open() on all the
> table are required. But I don't see that to be the case, since these
> tables are locked already through a previous call to heap_open(). In the
Good catch! Will fix.
> same function instead of argument name parentrelId may be we should use
> name parent_oid, so that we use same notation for parent and child table
> OIDs.
Will fix.
> 3. Regarding enum VacuumMode - it's being used only in case of
> acquire_inherited_sample_rows() and that too, to check only a single
> value of the three defined there. May be we should just infer that value
> inside acquire_inherited_sample_rows() or pass a boolean true or false
> from do_analyse_rel() based on the VacuumStmt. I do not see need for a
> separate three value enum of which only one value is used and also to
> pass it down from vacuum() by changing signatures of the minion functions.
I introduced that for possible future use. See the discussion in [1].
> 4. In postgresGetForeignPlan(), the code added by this patch is required
> to handle the case when the row mark is placed on a parent table and
> hence is required to be applied on the child table. We need a comment
> explaining this. Otherwise, the three step process to get the row mark
> information isn't clear for a reader.
Will add the comment.
> 5. In expand_inherited_rtentry() why do you need a separate variable
> hasForeign? Instead of using that variable, you can actually set/reset
> rte->hasForeign since existence of even a single foreign child would
> mark that member as true. - After typing this, I got the answer when I
> looked at the function code. Every child's RTE is initially a copy of
> parent's RTE and hence hasForeign status would be inherited by every
> child after the first foreign child. I think, this reasoning should be
> added as comment before assignment to rte->hasForeign at the end of the
> function.
As you mentioned, I think we could set rte->hasForeign directly during
the scan for the inheritance set, without the separate variable,
hasForeign. But ISTM that it'd improve code readability to set
rte->hasForeign using the separate variable at the end of the function
because rte->hasForeign has its meaning only when rte->inh is true and
because we know whether rte->inh is true, at the end of the function.
> 6. The tests in foreign_data.sql are pretty extensive. Thanks for that.
> I think, we should also check the effect of each of the following
> command using \d on appropriate tables.
> +CREATE FOREIGN TABLE ft2 () INHERITS (pt1)
> + SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted" 'value');
> +ALTER FOREIGN TABLE ft2 NO INHERIT pt1;
> +DROP FOREIGN TABLE ft2;
> +CREATE FOREIGN TABLE ft2 (
> + c1 integer NOT NULL,
> + c2 text,
> + c3 date
> +) SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted" 'value');
> +ALTER FOREIGN TABLE ft2 INHERIT pt1;
Will fix.
Apart from the above, I noticed that the patch doesn't consider to call
ExplainForeignModify during EXPLAIN for an inherited UPDATE/DELETE, as
shown below (note that there are no UPDATE remote queries displayed):
postgres=# explain verbose update parent set a = a * 2 where a = 5; QUERY PLAN
------------------------------------------------------------------------------------- Update on public.parent
(cost=0.00..280.77rows=25 width=10) -> Seq Scan on public.parent (cost=0.00..0.00 rows=1 width=10) Output:
(parent.a* 2), parent.ctid Filter: (parent.a = 5) -> Foreign Scan on public.ft1 (cost=100.00..140.38
rows=12width=10) Output: (ft1.a * 2), ft1.ctid Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE
((a=
5)) FOR UPDATE -> Foreign Scan on public.ft2 (cost=100.00..140.38 rows=12 width=10) Output: (ft2.a * 2),
ft2.ctid Remote SQL: SELECT a, ctid FROM public.mytable_2 WHERE ((a =
5)) FOR UPDATE
(10 rows)
So, I'd like to modify explain.c to show those queries like this:
postgres=# explain verbose update parent set a = a * 2 where a = 5; QUERY PLAN
------------------------------------------------------------------------------------- Update on public.parent
(cost=0.00..280.77rows=25 width=10) -> Seq Scan on public.parent (cost=0.00..0.00 rows=1 width=10) Output:
(parent.a* 2), parent.ctid Filter: (parent.a = 5) Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid =
$1 -> Foreign Scan on public.ft1 (cost=100.00..140.38 rows=12 width=10) Output: (ft1.a * 2), ft1.ctid
Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE ((a =
5)) FOR UPDATE Remote SQL: UPDATE public.mytable_2 SET a = $2 WHERE ctid = $1 -> Foreign Scan on public.ft2
(cost=100.00..140.38rows=12 width=10) Output: (ft2.a * 2), ft2.ctid Remote SQL: SELECT a, ctid FROM
public.mytable_2WHERE ((a =
5)) FOR UPDATE
(12 rows)
What do you say?
Sorry for the delay.
[1] http://www.postgresql.org/message-id/1316566782-sup-2678@alvh.no-ip.org
Best regards,
Etsuro Fujita
On Thu, Nov 27, 2014 at 3:52 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/11/17 17:55), Ashutosh Bapat wrote:Here are my review comments for patch fdw-inh-3.patch.
Thanks for the review!Tests
-------
1. It seems like you have copied from testcase inherit.sql to
postgres_fdw testcase. That's a good thing, but it makes the test quite
long. May be we should have two tests in postgres_fdw contrib module,
one for simple cases, and other for inheritance. What do you say?
IMO, the test is not so time-consuming, so it doesn't seem worth splitting it into two.
I am not worried about the timing but I am worried about the length of the file and hence ease of debugging in case we find any issues there. We will leave that for the commiter to decide.
Documentation
--------------------
1. The change in ddl.sgml
- We will refer to the child tables as partitions, though they
- are in every way normal <productname>PostgreSQL</> tables.
+ We will refer to the child tables as partitions, though we assume
+ that they are normal <productname>PostgreSQL</> tables.
adds phrase "we assume that", which confuses the intention behind the
sentence. The original text is intended to highlight the equivalence
between "partition" and "normal table", where as the addition esp. the
word "assume" weakens that equivalence. Instead now we have to highlight
the equivalence between "partition" and "normal or foreign table". The
wording similar to "though they are either normal or foreign tables"
should be used there.
You are right, but I feel that there is something out of place in saying that there (5.10.2. Implementing Partitioning) because the procedure there has been written based on normal tables only. Put another way, if we say that, I think we'd need to add more docs, describing the syntax and/or the corresponding examples for foreign-table cases. But I'd like to leave that for another patch. So, how about the wording "we assume *here* that", instead of "we assume that", as I added the following notice in the previous section (5.10.1. Overview)?
@@ -2650,7 +2669,10 @@ VALUES ('Albany', NULL, NULL, 'NY');
table of a single parent table. The parent table itself is normally
empty; it exists just to represent the entire data set. You should be
familiar with inheritance (see <xref linkend="ddl-inherit">) before
- attempting to set up partitioning.
+ attempting to set up partitioning. (The setup and management of
+ partitioned tables illustrated in this section assume that each
+ partition is a normal table. However, you can do that in a similar way
+ for cases where some or all partitions are foreign tables.)
This looks ok, though, I would like to see final version of the document. But I think, we will leave that for committer to handle.
2. The wording "some kind of optimization" gives vague picture. May be
it can be worded as "Since the constraints are assumed to be true, they
are used in constraint-based query optimization like constraint
exclusion for partitioned tables.".
+ Those constraints are used in some kind of query optimization such
+ as constraint exclusion for partitioned tables (see
+ <xref linkend="ddl-partitioning">).
Will follow your revision.
Thanks.
Code
-------
1. In the following change
+/*
* acquire_inherited_sample_rows -- acquire sample rows from
inheritance tree
*
* This has the same API as acquire_sample_rows, except that rows are
* collected from all inheritance children as well as the specified table.
- * We fail and return zero if there are no inheritance children.
+ * We fail and return zero if there are no inheritance children or
there are
+ * inheritance children that foreign tables.
The addition should be "there are inheritance children that *are all
*foreign tables. Note the addition "are all".
Sorry, I incorrectly wrote the comment. What I tried to write is "We fail and return zero if there are no inheritance children or if we are not in VAC_MODE_SINGLE case and inheritance tree contains at least one foreign table.".
You might want to use "English" description of VAC_MODE_SINGLE instead of that macro in the comment, so that reader doesn't have to look up VAC_MODE_SINGLE. But I think, we will leave this for the committer.
2. The function has_foreign() be better named has_foreign_child()? This
How about "has_foreign_table"?
has_foreign_child() would be better, since these are "children" in the inheritance hierarchy and not mere "table"s.
function loops through all the tableoids passed even after it has found
a foreign table. Why can't we return true immediately after finding the
first foreign table, unless the side effects of heap_open() on all the
table are required. But I don't see that to be the case, since these
tables are locked already through a previous call to heap_open(). In the
Good catch! Will fix.same function instead of argument name parentrelId may be we should use
name parent_oid, so that we use same notation for parent and child table
OIDs.
Will fix.
Thanks.
3. Regarding enum VacuumMode - it's being used only in case of
acquire_inherited_sample_rows() and that too, to check only a single
value of the three defined there. May be we should just infer that value
inside acquire_inherited_sample_rows() or pass a boolean true or false
from do_analyse_rel() based on the VacuumStmt. I do not see need for a
separate three value enum of which only one value is used and also to
pass it down from vacuum() by changing signatures of the minion functions.
I introduced that for possible future use. See the discussion in [1].
Will leave it for the commiter to decide.
4. In postgresGetForeignPlan(), the code added by this patch is required
to handle the case when the row mark is placed on a parent table and
hence is required to be applied on the child table. We need a comment
explaining this. Otherwise, the three step process to get the row mark
information isn't clear for a reader.
Will add the comment.5. In expand_inherited_rtentry() why do you need a separate variable
hasForeign? Instead of using that variable, you can actually set/reset
rte->hasForeign since existence of even a single foreign child would
mark that member as true. - After typing this, I got the answer when I
looked at the function code. Every child's RTE is initially a copy of
parent's RTE and hence hasForeign status would be inherited by every
child after the first foreign child. I think, this reasoning should be
added as comment before assignment to rte->hasForeign at the end of the
function.
As you mentioned, I think we could set rte->hasForeign directly during the scan for the inheritance set, without the separate variable, hasForeign. But ISTM that it'd improve code readability to set rte->hasForeign using the separate variable at the end of the function because rte->hasForeign has its meaning only when rte->inh is true and because we know whether rte->inh is true, at the end of the function.
Fine. Please use "hasForeignChild" instead of just "hasForeign" without a clue as to what is "foreign" here.
6. The tests in foreign_data.sql are pretty extensive. Thanks for that.
I think, we should also check the effect of each of the following
command using \d on appropriate tables.
+CREATE FOREIGN TABLE ft2 () INHERITS (pt1)
+ SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted" 'value');
+ALTER FOREIGN TABLE ft2 NO INHERIT pt1;
+DROP FOREIGN TABLE ft2;
+CREATE FOREIGN TABLE ft2 (
+ c1 integer NOT NULL,
+ c2 text,
+ c3 date
+) SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted" 'value');
+ALTER FOREIGN TABLE ft2 INHERIT pt1;
Will fix.
Apart from the above, I noticed that the patch doesn't consider to call ExplainForeignModify during EXPLAIN for an inherited UPDATE/DELETE, as shown below (note that there are no UPDATE remote queries displayed):
postgres=# explain verbose update parent set a = a * 2 where a = 5;
QUERY PLAN
-------------------------------------------------------------------------------------
Update on public.parent (cost=0.00..280.77 rows=25 width=10)
-> Seq Scan on public.parent (cost=0.00..0.00 rows=1 width=10)
Output: (parent.a * 2), parent.ctid
Filter: (parent.a = 5)
-> Foreign Scan on public.ft1 (cost=100.00..140.38 rows=12 width=10)
Output: (ft1.a * 2), ft1.ctid
Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE ((a = 5)) FOR UPDATE
-> Foreign Scan on public.ft2 (cost=100.00..140.38 rows=12 width=10)
Output: (ft2.a * 2), ft2.ctid
Remote SQL: SELECT a, ctid FROM public.mytable_2 WHERE ((a = 5)) FOR UPDATE
(10 rows)
So, I'd like to modify explain.c to show those queries like this:
postgres=# explain verbose update parent set a = a * 2 where a = 5;
QUERY PLAN
-------------------------------------------------------------------------------------
Update on public.parent (cost=0.00..280.77 rows=25 width=10)
-> Seq Scan on public.parent (cost=0.00..0.00 rows=1 width=10)
Output: (parent.a * 2), parent.ctid
Filter: (parent.a = 5)
Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1
-> Foreign Scan on public.ft1 (cost=100.00..140.38 rows=12 width=10)
Output: (ft1.a * 2), ft1.ctid
Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE ((a = 5)) FOR UPDATE
Remote SQL: UPDATE public.mytable_2 SET a = $2 WHERE ctid = $1
-> Foreign Scan on public.ft2 (cost=100.00..140.38 rows=12 width=10)
Output: (ft2.a * 2), ft2.ctid
Remote SQL: SELECT a, ctid FROM public.mytable_2 WHERE ((a = 5)) FOR UPDATE
(12 rows)
What do you say?
Two "remote SQL" under a single node would be confusing. Also the node is labelled as "Foreign Scan". It would be confusing to show an "UPDATE" command under this "scan" node.
BTW, I was experimenting with DMLs being executed on multiple FDW server under same transaction and found that the transactions may not be atomic (and may be inconsistent), if one or more of the server fails to commit while rest of them commit the transaction. The reason for this is, we do not "rollback" the already "committed" changes to the foreign server, if one or more of them fail to "commit" a transaction. With foreign tables under inheritance hierarchy a single DML can span across multiple servers and the result may not be atomic (and may be inconsistent). So, either we have to disable DMLs on an inheritance hierarchy which spans multiple servers. OR make sure that such transactions follow 2PC norms.
Sorry for the delay.
[1] http://www.postgresql.org/message-id/1316566782-sup-2678@alvh.no-ip.org
Best regards,
Etsuro Fujita
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
(2014/11/28 18:14), Ashutosh Bapat wrote: > On Thu, Nov 27, 2014 at 3:52 PM, Etsuro Fujita > <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote: > Apart from the above, I noticed that the patch doesn't consider to > call ExplainForeignModify during EXPLAIN for an inherited > UPDATE/DELETE, as shown below (note that there are no UPDATE remote > queries displayed): > So, I'd like to modify explain.c to show those queries like this: > postgres=# explain verbose update parent set a = a * 2 where a = 5; > QUERY PLAN > ------------------------------__------------------------------__------------------------- > Update on public.parent (cost=0.00..280.77 rows=25 width=10) > -> Seq Scan on public.parent (cost=0.00..0.00 rows=1 width=10) > Output: (parent.a * 2), parent.ctid > Filter: (parent.a = 5) > Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1 > -> Foreign Scan on public.ft1 (cost=100.00..140.38 rows=12 > width=10) > Output: (ft1.a * 2), ft1.ctid > Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE ((a > = 5)) FOR UPDATE > Remote SQL: UPDATE public.mytable_2 SET a = $2 WHERE ctid = $1 > -> Foreign Scan on public.ft2 (cost=100.00..140.38 rows=12 > width=10) > Output: (ft2.a * 2), ft2.ctid > Remote SQL: SELECT a, ctid FROM public.mytable_2 WHERE ((a > = 5)) FOR UPDATE > (12 rows) > Two "remote SQL" under a single node would be confusing. Also the node > is labelled as "Foreign Scan". It would be confusing to show an "UPDATE" > command under this "scan" node. I thought this as an extention of the existing (ie, non-inherited) case (see the below example) to the inherited case. postgres=# explain verbose update ft1 set a = a * 2 where a = 5; QUERY PLAN ------------------------------------------------------------------------------------- Update on public.ft1 (cost=100.00..140.38rows=12 width=10) Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1 -> Foreign Scanon public.ft1 (cost=100.00..140.38 rows=12 width=10) Output: (a * 2), ctid Remote SQL: SELECT a, ctidFROM public.mytable_1 WHERE ((a = 5)) FOR UPDATE (5 rows) I think we should show update commands somewhere for the inherited case as for the non-inherited case. Alternatives to this are welcome. > BTW, I was experimenting with DMLs being executed on multiple FDW server > under same transaction and found that the transactions may not be atomic > (and may be inconsistent), if one or more of the server fails to commit > while rest of them commit the transaction. The reason for this is, we do > not "rollback" the already "committed" changes to the foreign server, if > one or more of them fail to "commit" a transaction. With foreign tables > under inheritance hierarchy a single DML can span across multiple > servers and the result may not be atomic (and may be inconsistent). So, IIUC, even the transactions over the local and the *single* remote server are not guaranteed to be executed atomically in the current form. It is possible that the remote transaction succeedsand the local one fails, for example, resulting in data inconsistency between the local and the remote. > either we have to disable DMLs on an inheritance hierarchy which spans > multiple servers. OR make sure that such transactions follow 2PC norms. -1 for disabling update queries on such an inheritance hierarchy because I think we should leave that to the user's judgment. And I think 2PC is definitely a separate patch. Thanks, Best regards, Etsuro Fujita
On Tue, Dec 2, 2014 at 8:29 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/11/28 18:14), Ashutosh Bapat wrote:On Thu, Nov 27, 2014 at 3:52 PM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:
Apart from the above, I noticed that the patch doesn't consider to
call ExplainForeignModify during EXPLAIN for an inherited
UPDATE/DELETE, as shown below (note that there are no UPDATE remote
queries displayed):So, I'd like to modify explain.c to show those queries like this:postgres=# explain verbose update parent set a = a * 2 where a = 5;
QUERY PLAN
------------------------------__------------------------------__-------------------------
Update on public.parent (cost=0.00..280.77 rows=25 width=10)
-> Seq Scan on public.parent (cost=0.00..0.00 rows=1 width=10)
Output: (parent.a * 2), parent.ctid
Filter: (parent.a = 5)
Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1
-> Foreign Scan on public.ft1 (cost=100.00..140.38 rows=12
width=10)
Output: (ft1.a * 2), ft1.ctid
Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE ((a
= 5)) FOR UPDATE
Remote SQL: UPDATE public.mytable_2 SET a = $2 WHERE ctid = $1
-> Foreign Scan on public.ft2 (cost=100.00..140.38 rows=12
width=10)
Output: (ft2.a * 2), ft2.ctid
Remote SQL: SELECT a, ctid FROM public.mytable_2 WHERE ((a
= 5)) FOR UPDATE
(12 rows)Two "remote SQL" under a single node would be confusing. Also the node
is labelled as "Foreign Scan". It would be confusing to show an "UPDATE"
command under this "scan" node.
I thought this as an extention of the existing (ie, non-inherited) case (see the below example) to the inherited case.
postgres=# explain verbose update ft1 set a = a * 2 where a = 5;
QUERY PLAN
-------------------------------------------------------------------------------------
Update on public.ft1 (cost=100.00..140.38 rows=12 width=10)
Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1
-> Foreign Scan on public.ft1 (cost=100.00..140.38 rows=12 width=10)
Output: (a * 2), ctid
Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE ((a = 5)) FOR UPDATE
(5 rows)
I think we should show update commands somewhere for the inherited case as for the non-inherited case. Alternatives to this are welcome.
This is not exactly extension of non-inheritance case. non-inheritance case doesn't show two remote SQLs under the same plan node. May be you can rename the label Remote SQL as Remote UPDATE/INSERT/DELETE (or something to that effect) for the DML command and the Foreign plan node should be renamed to Foreign access node or something to indicate that it does both the scan as well as DML. I am not keen about the actual terminology, but I think a reader of plan shouldn't get confused.
We can leave this for committer's judgement.
BTW, I was experimenting with DMLs being executed on multiple FDW server
under same transaction and found that the transactions may not be atomic
(and may be inconsistent), if one or more of the server fails to commit
while rest of them commit the transaction. The reason for this is, we do
not "rollback" the already "committed" changes to the foreign server, if
one or more of them fail to "commit" a transaction. With foreign tables
under inheritance hierarchy a single DML can span across multiple
servers and the result may not be atomic (and may be inconsistent). So,
IIUC, even the transactions over the local and the *single* remote server are not guaranteed to be executed atomically in the current form. It is possible that the remote transaction succeeds and the local one fails, for example, resulting in data inconsistency between the local and the remote.
IIUC, while committing transactions involving a single remote server, the steps taken are as follows
1. the local changes are brought to PRE-COMMIT stage, which means that the transaction *will* succeed locally after successful completion of this phase,
1. the local changes are brought to PRE-COMMIT stage, which means that the transaction *will* succeed locally after successful completion of this phase,
2. COMMIT message is sent to the foreign server
3. If step two succeeds, local changes are committed and successful commit is conveyed to the client
4. if step two fails, local changes are rolled back and abort status is conveyed to the client
5. If step 1 itself fails, the remote changes are rolled back.
This is as per one phase commit protocol which guarantees ACID for single foreign data source. So, the changes involving local and a single foreign server seem to be atomic and consistent.
either we have to disable DMLs on an inheritance hierarchy which spans
multiple servers. OR make sure that such transactions follow 2PC norms.
-1 for disabling update queries on such an inheritance hierarchy because I think we should leave that to the user's judgment. And I think 2PC is definitely a separate patch.
I agree that 2pc is much larger work and is subject for separate patch/es. But it may not be acceptable that changes made within a single command violate atomicity and consistency, which can not be controlled or altered by user intervention. Again, we can leave it for committer's judgement.
Marking this as "ready for committer".
Thanks,
Best regards,
Etsuro Fujita
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
On Wed, Dec 3, 2014 at 4:05 PM, Ashutosh Bapat <ashutosh.bapat@enterprisedb.com> wrote:
On Tue, Dec 2, 2014 at 8:29 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:(2014/11/28 18:14), Ashutosh Bapat wrote:On Thu, Nov 27, 2014 at 3:52 PM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:
Apart from the above, I noticed that the patch doesn't consider to
call ExplainForeignModify during EXPLAIN for an inherited
UPDATE/DELETE, as shown below (note that there are no UPDATE remote
queries displayed):So, I'd like to modify explain.c to show those queries like this:postgres=# explain verbose update parent set a = a * 2 where a = 5;
QUERY PLAN
------------------------------__------------------------------__-------------------------
Update on public.parent (cost=0.00..280.77 rows=25 width=10)
-> Seq Scan on public.parent (cost=0.00..0.00 rows=1 width=10)
Output: (parent.a * 2), parent.ctid
Filter: (parent.a = 5)
Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1
-> Foreign Scan on public.ft1 (cost=100.00..140.38 rows=12
width=10)
Output: (ft1.a * 2), ft1.ctid
Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE ((a
= 5)) FOR UPDATE
Remote SQL: UPDATE public.mytable_2 SET a = $2 WHERE ctid = $1
-> Foreign Scan on public.ft2 (cost=100.00..140.38 rows=12
width=10)
Output: (ft2.a * 2), ft2.ctid
Remote SQL: SELECT a, ctid FROM public.mytable_2 WHERE ((a
= 5)) FOR UPDATE
(12 rows)Two "remote SQL" under a single node would be confusing. Also the node
is labelled as "Foreign Scan". It would be confusing to show an "UPDATE"
command under this "scan" node.
I thought this as an extention of the existing (ie, non-inherited) case (see the below example) to the inherited case.
postgres=# explain verbose update ft1 set a = a * 2 where a = 5;
QUERY PLAN
-------------------------------------------------------------------------------------
Update on public.ft1 (cost=100.00..140.38 rows=12 width=10)
Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1
-> Foreign Scan on public.ft1 (cost=100.00..140.38 rows=12 width=10)
Output: (a * 2), ctid
Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE ((a = 5)) FOR UPDATE
(5 rows)
I think we should show update commands somewhere for the inherited case as for the non-inherited case. Alternatives to this are welcome.This is not exactly extension of non-inheritance case. non-inheritance case doesn't show two remote SQLs under the same plan node. May be you can rename the label Remote SQL as Remote UPDATE/INSERT/DELETE (or something to that effect) for the DML command and the Foreign plan node should be renamed to Foreign access node or something to indicate that it does both the scan as well as DML. I am not keen about the actual terminology, but I think a reader of plan shouldn't get confused.We can leave this for committer's judgement.BTW, I was experimenting with DMLs being executed on multiple FDW server
under same transaction and found that the transactions may not be atomic
(and may be inconsistent), if one or more of the server fails to commit
while rest of them commit the transaction. The reason for this is, we do
not "rollback" the already "committed" changes to the foreign server, if
one or more of them fail to "commit" a transaction. With foreign tables
under inheritance hierarchy a single DML can span across multiple
servers and the result may not be atomic (and may be inconsistent). So,
IIUC, even the transactions over the local and the *single* remote server are not guaranteed to be executed atomically in the current form. It is possible that the remote transaction succeeds and the local one fails, for example, resulting in data inconsistency between the local and the remote.IIUC, while committing transactions involving a single remote server, the steps taken are as follows
1. the local changes are brought to PRE-COMMIT stage, which means that the transaction *will* succeed locally after successful completion of this phase,2. COMMIT message is sent to the foreign server3. If step two succeeds, local changes are committed and successful commit is conveyed to the client4. if step two fails, local changes are rolled back and abort status is conveyed to the client5. If step 1 itself fails, the remote changes are rolled back.This is as per one phase commit protocol which guarantees ACID for single foreign data source. So, the changes involving local and a single foreign server seem to be atomic and consistent.either we have to disable DMLs on an inheritance hierarchy which spans
multiple servers. OR make sure that such transactions follow 2PC norms.
-1 for disabling update queries on such an inheritance hierarchy because I think we should leave that to the user's judgment. And I think 2PC is definitely a separate patch.I agree that 2pc is much larger work and is subject for separate patch/es. But it may not be acceptable that changes made within a single command violate atomicity and consistency, which can not be controlled or altered by user intervention. Again, we can leave it for committer's judgement.Marking this as "ready for committer".
Sorry, I noticed in the commitfest app, that there are other reviewers as well. Let's wait for them to comment.
Thanks,
Best regards,
Etsuro Fujita
--Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
(2014/12/03 19:35), Ashutosh Bapat wrote: > On Tue, Dec 2, 2014 at 8:29 AM, Etsuro Fujita > <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote: > This is not exactly extension of non-inheritance case. non-inheritance > case doesn't show two remote SQLs under the same plan node. May be you > can rename the label Remote SQL as Remote UPDATE/INSERT/DELETE (or > something to that effect) for the DML command and the Foreign plan node > should be renamed to Foreign access node or something to indicate that > it does both the scan as well as DML. I am not keen about the actual > terminology, but I think a reader of plan shouldn't get confused. > > We can leave this for committer's judgement. Thanks for the proposal! I think that would be a good idea. But I think there would be another idea. An example will be shown below. We show the update commands below the ModifyTable node, not above the corresponding ForeignScan nodes, so maybe less confusing. If there are no objections of you and others, I'll update the patch this way. postgres=# explain verbose update parent set a = a * 2 where a = 5; QUERY PLAN ------------------------------------------------------------------------------------- Update on public.parent (cost=0.00..280.77rows=25 width=10) On public.ft1 Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1 On public.ft2 Remote SQL: UPDATE public.mytable_2 SET a = $2 WHERE ctid = $1 -> Seq Scan on public.parent (cost=0.00..0.00rows=1 width=10) Output: (parent.a * 2), parent.ctid Filter: (parent.a = 5) -> ForeignScan on public.ft1 (cost=100.00..140.38 rows=12 width=10) Output: (ft1.a * 2), ft1.ctid Remote SQL:SELECT a, ctid FROM public.mytable_1 WHERE ((a = 5)) FOR UPDATE -> Foreign Scan on public.ft2 (cost=100.00..140.38 rows=12 width=10) Output: (ft2.a * 2), ft2.ctid Remote SQL: SELECT a, ctid FROM public.mytable_2 WHERE ((a = 5)) FOR UPDATE (12 rows) > IIUC, even the transactions over the local and the *single* remote > server are not guaranteed to be executed atomically in the current > form. It is possible that the remote transaction succeeds and the > local one fails, for example, resulting in data inconsistency > between the local and the remote. > IIUC, while committing transactions involving a single remote server, > the steps taken are as follows > 1. the local changes are brought to PRE-COMMIT stage, which means that > the transaction *will* succeed locally after successful completion of > this phase, > 2. COMMIT message is sent to the foreign server > 3. If step two succeeds, local changes are committed and successful > commit is conveyed to the client > 4. if step two fails, local changes are rolled back and abort status is > conveyed to the client > 5. If step 1 itself fails, the remote changes are rolled back. > This is as per one phase commit protocol which guarantees ACID for > single foreign data source. So, the changes involving local and a single > foreign server seem to be atomic and consistent. Really? Maybe I'm missing something, but I don't think the current implementation for committing transactions has such a mechanism stated in step 1. So, I think it's possible that the local transaction fails in step3 while the remote transaction succeeds, as mentioned above. Thanks, Best regards, Etsuro Fujita
On Thu, Dec 4, 2014 at 9:05 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
(2014/12/03 19:35), Ashutosh Bapat wrote:On Tue, Dec 2, 2014 at 8:29 AM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:This is not exactly extension of non-inheritance case. non-inheritance
case doesn't show two remote SQLs under the same plan node. May be you
can rename the label Remote SQL as Remote UPDATE/INSERT/DELETE (or
something to that effect) for the DML command and the Foreign plan node
should be renamed to Foreign access node or something to indicate that
it does both the scan as well as DML. I am not keen about the actual
terminology, but I think a reader of plan shouldn't get confused.
We can leave this for committer's judgement.
Thanks for the proposal! I think that would be a good idea. But I think there would be another idea. An example will be shown below. We show the update commands below the ModifyTable node, not above the corresponding ForeignScan nodes, so maybe less confusing. If there are no objections of you and others, I'll update the patch this way.
postgres=# explain verbose update parent set a = a * 2 where a = 5;
QUERY PLAN
-------------------------------------------------------------------------------------
Update on public.parent (cost=0.00..280.77 rows=25 width=10)
On public.ft1
Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1
On public.ft2
Remote SQL: UPDATE public.mytable_2 SET a = $2 WHERE ctid = $1
-> Seq Scan on public.parent (cost=0.00..0.00 rows=1 width=10)
Output: (parent.a * 2), parent.ctid
Filter: (parent.a = 5)
-> Foreign Scan on public.ft1 (cost=100.00..140.38 rows=12 width=10)
Output: (ft1.a * 2), ft1.ctid
Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE ((a = 5)) FOR UPDATE
-> Foreign Scan on public.ft2 (cost=100.00..140.38 rows=12 width=10)
Output: (ft2.a * 2), ft2.ctid
Remote SQL: SELECT a, ctid FROM public.mytable_2 WHERE ((a = 5)) FOR UPDATE
(12 rows)
Looks better.
IIUC, even the transactions over the local and the *single* remote
server are not guaranteed to be executed atomically in the current
form. It is possible that the remote transaction succeeds and the
local one fails, for example, resulting in data inconsistency
between the local and the remote.IIUC, while committing transactions involving a single remote server,
the steps taken are as follows
1. the local changes are brought to PRE-COMMIT stage, which means that
the transaction *will* succeed locally after successful completion of
this phase,
2. COMMIT message is sent to the foreign server
3. If step two succeeds, local changes are committed and successful
commit is conveyed to the client
4. if step two fails, local changes are rolled back and abort status is
conveyed to the client
5. If step 1 itself fails, the remote changes are rolled back.
This is as per one phase commit protocol which guarantees ACID for
single foreign data source. So, the changes involving local and a single
foreign server seem to be atomic and consistent.
Really? Maybe I'm missing something, but I don't think the current implementation for committing transactions has such a mechanism stated in step 1. So, I think it's possible that the local transaction fails in step3 while the remote transaction succeeds, as mentioned above.
PFA a script attached which shows this. You may want to check the code in pgfdw_xact_callback() for actions taken by postgres_fdw on various events. CommitTransaction() for how those events are generated. The code there complies with the sequence above.
Thanks,
Best regards,
Etsuro Fujita
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Sorry, here's the script.
On Thu, Dec 4, 2014 at 10:00 AM, Ashutosh Bapat <ashutosh.bapat@enterprisedb.com> wrote:
On Thu, Dec 4, 2014 at 9:05 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:(2014/12/03 19:35), Ashutosh Bapat wrote:On Tue, Dec 2, 2014 at 8:29 AM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:This is not exactly extension of non-inheritance case. non-inheritance
case doesn't show two remote SQLs under the same plan node. May be you
can rename the label Remote SQL as Remote UPDATE/INSERT/DELETE (or
something to that effect) for the DML command and the Foreign plan node
should be renamed to Foreign access node or something to indicate that
it does both the scan as well as DML. I am not keen about the actual
terminology, but I think a reader of plan shouldn't get confused.
We can leave this for committer's judgement.
Thanks for the proposal! I think that would be a good idea. But I think there would be another idea. An example will be shown below. We show the update commands below the ModifyTable node, not above the corresponding ForeignScan nodes, so maybe less confusing. If there are no objections of you and others, I'll update the patch this way.
postgres=# explain verbose update parent set a = a * 2 where a = 5;
QUERY PLAN
-------------------------------------------------------------------------------------
Update on public.parent (cost=0.00..280.77 rows=25 width=10)
On public.ft1
Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1
On public.ft2
Remote SQL: UPDATE public.mytable_2 SET a = $2 WHERE ctid = $1
-> Seq Scan on public.parent (cost=0.00..0.00 rows=1 width=10)
Output: (parent.a * 2), parent.ctid
Filter: (parent.a = 5)
-> Foreign Scan on public.ft1 (cost=100.00..140.38 rows=12 width=10)
Output: (ft1.a * 2), ft1.ctid
Remote SQL: SELECT a, ctid FROM public.mytable_1 WHERE ((a = 5)) FOR UPDATE
-> Foreign Scan on public.ft2 (cost=100.00..140.38 rows=12 width=10)
Output: (ft2.a * 2), ft2.ctid
Remote SQL: SELECT a, ctid FROM public.mytable_2 WHERE ((a = 5)) FOR UPDATE
(12 rows)Looks better.IIUC, even the transactions over the local and the *single* remote
server are not guaranteed to be executed atomically in the current
form. It is possible that the remote transaction succeeds and the
local one fails, for example, resulting in data inconsistency
between the local and the remote.IIUC, while committing transactions involving a single remote server,
the steps taken are as follows
1. the local changes are brought to PRE-COMMIT stage, which means that
the transaction *will* succeed locally after successful completion of
this phase,
2. COMMIT message is sent to the foreign server
3. If step two succeeds, local changes are committed and successful
commit is conveyed to the client
4. if step two fails, local changes are rolled back and abort status is
conveyed to the client
5. If step 1 itself fails, the remote changes are rolled back.
This is as per one phase commit protocol which guarantees ACID for
single foreign data source. So, the changes involving local and a single
foreign server seem to be atomic and consistent.
Really? Maybe I'm missing something, but I don't think the current implementation for committing transactions has such a mechanism stated in step 1. So, I think it's possible that the local transaction fails in step3 while the remote transaction succeeds, as mentioned above.PFA a script attached which shows this. You may want to check the code in pgfdw_xact_callback() for actions taken by postgres_fdw on various events. CommitTransaction() for how those events are generated. The code there complies with the sequence above.
Thanks,
Best regards,
Etsuro Fujita
--Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Attachment
On Thu, Dec 04, 2014 at 10:00:14AM +0530, Ashutosh Bapat wrote: > On Thu, Dec 4, 2014 at 9:05 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote: > > (2014/12/03 19:35), Ashutosh Bapat wrote: > >> On Tue, Dec 2, 2014 at 8:29 AM, Etsuro Fujita > >> <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote: > > IIUC, even the transactions over the local and the *single* remote > >> server are not guaranteed to be executed atomically in the current > >> form. It is possible that the remote transaction succeeds and the > >> local one fails, for example, resulting in data inconsistency > >> between the local and the remote. > >> > > > > IIUC, while committing transactions involving a single remote server, > >> the steps taken are as follows > >> 1. the local changes are brought to PRE-COMMIT stage, which means that > >> the transaction *will* succeed locally after successful completion of > >> this phase, > >> 2. COMMIT message is sent to the foreign server > >> 3. If step two succeeds, local changes are committed and successful > >> commit is conveyed to the client > >> 4. if step two fails, local changes are rolled back and abort status is > >> conveyed to the client > >> 5. If step 1 itself fails, the remote changes are rolled back. > >> This is as per one phase commit protocol which guarantees ACID for > >> single foreign data source. So, the changes involving local and a single > >> foreign server seem to be atomic and consistent. > >> > > > > Really? Maybe I'm missing something, but I don't think the current > > implementation for committing transactions has such a mechanism stated in > > step 1. So, I think it's possible that the local transaction fails in > > step3 while the remote transaction succeeds, as mentioned above. > > > > > PFA a script attached which shows this. You may want to check the code in > pgfdw_xact_callback() for actions taken by postgres_fdw on various events. > CommitTransaction() for how those events are generated. The code there > complies with the sequence above. While postgres_fdw delays remote commit to eliminate many causes for having one server commit while another aborts, other causes remain. The local transaction could still fail due to WAL I/O problems or a system crash. 2PC is the reliable answer, but that was explicitly out of scope for the initial postgres_fdw write support. Does this inheritance patch add any atomicity problem that goes away when one breaks up the inheritance hierarchy and UPDATEs each table separately? If not, this limitation is okay.
On Thu, Dec 04, 2014 at 12:35:54PM +0900, Etsuro Fujita wrote: > (2014/12/03 19:35), Ashutosh Bapat wrote: > >On Tue, Dec 2, 2014 at 8:29 AM, Etsuro Fujita > ><fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote: > > >This is not exactly extension of non-inheritance case. non-inheritance > >case doesn't show two remote SQLs under the same plan node. May be you > >can rename the label Remote SQL as Remote UPDATE/INSERT/DELETE (or > >something to that effect) for the DML command and the Foreign plan node > >should be renamed to Foreign access node or something to indicate that > >it does both the scan as well as DML. I am not keen about the actual > >terminology, but I think a reader of plan shouldn't get confused. > > > >We can leave this for committer's judgement. > > Thanks for the proposal! I think that would be a good idea. But I think > there would be another idea. An example will be shown below. We show the > update commands below the ModifyTable node, not above the corresponding > ForeignScan nodes, so maybe less confusing. If there are no objections of > you and others, I'll update the patch this way. > > postgres=# explain verbose update parent set a = a * 2 where a = 5; > QUERY PLAN > ------------------------------------------------------------------------------------- > Update on public.parent (cost=0.00..280.77 rows=25 width=10) > On public.ft1 > Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1 ^^^^^^^^^^ It occurs to me that the command generated by the FDW might well not be SQL at all, as is the case with file_fdw and anything else that talks to a NoSQL engine. Would it be reasonable to call this "Remote command" or something similarly generic? Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
(2014/12/07 2:02), David Fetter wrote: > On Thu, Dec 04, 2014 at 12:35:54PM +0900, Etsuro Fujita wrote: >> But I think >> there would be another idea. An example will be shown below. We show the >> update commands below the ModifyTable node, not above the corresponding >> ForeignScan nodes, so maybe less confusing. If there are no objections of >> you and others, I'll update the patch this way. >> >> postgres=# explain verbose update parent set a = a * 2 where a = 5; >> QUERY PLAN >> ------------------------------------------------------------------------------------- >> Update on public.parent (cost=0.00..280.77 rows=25 width=10) >> On public.ft1 >> Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1 > ^^^^^^^^^^ > It occurs to me that the command generated by the FDW might well not > be SQL at all, as is the case with file_fdw and anything else that > talks to a NoSQL engine. > > Would it be reasonable to call this "Remote command" or something > similarly generic? Yeah, but I'd like to propose that this line is shown by the FDW API (ie, ExplainForeignModify) as in non-inherited update cases, so that the FDW developer can choose the right name. As for "On public.ft1", I'd like to propose that the FDW API also show that by calling a function for that introduced into the PG core (Would it be better to use "For" rather than "On"?). Sorry, my explanation was not enough. Best regards, Etsuro Fujita
On Sat, Dec 6, 2014 at 9:21 PM, Noah Misch <noah@leadboat.com> wrote:
On Thu, Dec 04, 2014 at 10:00:14AM +0530, Ashutosh Bapat wrote:
> On Thu, Dec 4, 2014 at 9:05 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
> > (2014/12/03 19:35), Ashutosh Bapat wrote:
> >> On Tue, Dec 2, 2014 at 8:29 AM, Etsuro Fujita
> >> <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:While postgres_fdw delays remote commit to eliminate many causes for having> > IIUC, even the transactions over the local and the *single* remote
> >> server are not guaranteed to be executed atomically in the current
> >> form. It is possible that the remote transaction succeeds and the
> >> local one fails, for example, resulting in data inconsistency
> >> between the local and the remote.
> >>
> >
> > IIUC, while committing transactions involving a single remote server,
> >> the steps taken are as follows
> >> 1. the local changes are brought to PRE-COMMIT stage, which means that
> >> the transaction *will* succeed locally after successful completion of
> >> this phase,
> >> 2. COMMIT message is sent to the foreign server
> >> 3. If step two succeeds, local changes are committed and successful
> >> commit is conveyed to the client
> >> 4. if step two fails, local changes are rolled back and abort status is
> >> conveyed to the client
> >> 5. If step 1 itself fails, the remote changes are rolled back.
> >> This is as per one phase commit protocol which guarantees ACID for
> >> single foreign data source. So, the changes involving local and a single
> >> foreign server seem to be atomic and consistent.
> >>
> >
> > Really? Maybe I'm missing something, but I don't think the current
> > implementation for committing transactions has such a mechanism stated in
> > step 1. So, I think it's possible that the local transaction fails in
> > step3 while the remote transaction succeeds, as mentioned above.
> >
> >
> PFA a script attached which shows this. You may want to check the code in
> pgfdw_xact_callback() for actions taken by postgres_fdw on various events.
> CommitTransaction() for how those events are generated. The code there
> complies with the sequence above.
one server commit while another aborts, other causes remain. The local
transaction could still fail due to WAL I/O problems or a system crash. 2PC
is the reliable answer, but that was explicitly out of scope for the initial
postgres_fdw write support. Does this inheritance patch add any atomicity
problem that goes away when one breaks up the inheritance hierarchy and
UPDATEs each table separately? If not, this limitation is okay.
If the UPDATES crafted after breaking up the inheritance hierarchy are needed to be run within the same transaction (as the UPDATE on inheritance hierarchy would do), yes, there is atomicity problem.
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
(2014/12/08 15:17), Ashutosh Bapat wrote: > On Sat, Dec 6, 2014 at 9:21 PM, Noah Misch <noah@leadboat.com > <mailto:noah@leadboat.com>> wrote: > Does this inheritance patch add any > atomicity > problem that goes away when one breaks up the inheritance hierarchy and > UPDATEs each table separately? If not, this limitation is okay. > If the UPDATES crafted after breaking up the inheritance hierarchy are > needed to be run within the same transaction (as the UPDATE on > inheritance hierarchy would do), yes, there is atomicity problem. ISTM that your concern would basically a known problem. Consider the following transaction. BEGIN TRANSACTION; UPDATE foo SET a = 100; -- updates on table foo in remote server1 UPDATE bar SET a = 100; -- updates on table bar in remote server2 COMMIT TRANSACTION; This transaction would cause the atomicity problem if pgfdw_xact_callback() for XACT_EVENT_PRE_COMMIT for foo succeeded and then that for bar failed during CommitTransaction(). Thanks, Best regards, Etsuro Fujita
Hi Ashutosh,
Thanks for the review!
(2014/11/28 18:14), Ashutosh Bapat wrote:
> On Thu, Nov 27, 2014 at 3:52 PM, Etsuro Fujita
> <fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:
> (2014/11/17 17:55), Ashutosh Bapat wrote:
> Here are my review comments for patch fdw-inh-3.patch.
> Tests
> -------
> 1. It seems like you have copied from testcase inherit.sql to
> postgres_fdw testcase. That's a good thing, but it makes the
> test quite
> long. May be we should have two tests in postgres_fdw contrib
> module,
> one for simple cases, and other for inheritance. What do you say?
> IMO, the test is not so time-consuming, so it doesn't seem worth
> splitting it into two.
> I am not worried about the timing but I am worried about the length of
> the file and hence ease of debugging in case we find any issues there.
> We will leave that for the commiter to decide.
OK
> Documentation
> --------------------
> 1. The change in ddl.sgml
> - We will refer to the child tables as partitions, though
> they
> - are in every way normal <productname>PostgreSQL</> tables.
> + We will refer to the child tables as partitions, though
> we assume
> + that they are normal <productname>PostgreSQL</> tables.
>
> adds phrase "we assume that", which confuses the intention
> behind the
> sentence. The original text is intended to highlight the equivalence
> between "partition" and "normal table", where as the addition
> esp. the
> word "assume" weakens that equivalence. Instead now we have to
> highlight
> the equivalence between "partition" and "normal or foreign
> table". The
> wording similar to "though they are either normal or foreign tables"
> should be used there.
> You are right, but I feel that there is something out of place in
> saying that there (5.10.2. Implementing Partitioning) because the
> procedure there has been written based on normal tables only. Put
> another way, if we say that, I think we'd need to add more docs,
> describing the syntax and/or the corresponding examples for
> foreign-table cases. But I'd like to leave that for another patch.
> So, how about the wording "we assume *here* that", instead of "we
> assume that", as I added the following notice in the previous
> section (5.10.1. Overview)?
>
> @@ -2650,7 +2669,10 @@ VALUES ('Albany', NULL, NULL, 'NY');
> table of a single parent table. The parent table itself is
> normally
> empty; it exists just to represent the entire data set. You
> should be
> familiar with inheritance (see <xref linkend="ddl-inherit">)
> before
> - attempting to set up partitioning.
> + attempting to set up partitioning. (The setup and management of
> + partitioned tables illustrated in this section assume that each
> + partition is a normal table. However, you can do that in a
> similar way
> + for cases where some or all partitions are foreign tables.)
> This looks ok, though, I would like to see final version of the
> document. But I think, we will leave that for committer to handle.
OK
> 2. The wording "some kind of optimization" gives vague picture.
> May be
> it can be worded as "Since the constraints are assumed to be
> true, they
> are used in constraint-based query optimization like constraint
> exclusion for partitioned tables.".
> + Those constraints are used in some kind of query
> optimization such
> + as constraint exclusion for partitioned tables (see
> + <xref linkend="ddl-partitioning">).
> Will follow your revision.
Done.
> Code
> -------
> 1. In the following change
> +/*
> * acquire_inherited_sample_rows -- acquire sample rows from
> inheritance tree
> *
> * This has the same API as acquire_sample_rows, except that
> rows are
> * collected from all inheritance children as well as the
> specified table.
> - * We fail and return zero if there are no inheritance children.
> + * We fail and return zero if there are no inheritance children or
> there are
> + * inheritance children that foreign tables.
>
> The addition should be "there are inheritance children that *are all
> *foreign tables. Note the addition "are all".
> Sorry, I incorrectly wrote the comment. What I tried to write is
> "We fail and return zero if there are no inheritance children or if
> we are not in VAC_MODE_SINGLE case and inheritance tree contains at
> least one foreign table.".
> You might want to use "English" description of VAC_MODE_SINGLE instead
> of that macro in the comment, so that reader doesn't have to look up
> VAC_MODE_SINGLE. But I think, we will leave this for the committer.
I corrected the comments and translated the macro into the English
description.
> 2. The function has_foreign() be better named
> has_foreign_child()? This
> How about "has_foreign_table"?
> has_foreign_child() would be better, since these are "children" in the
> inheritance hierarchy and not mere "table"s.
Done. But I renamed it to has_foreign_children() because it sounds more
natural at least to me.
> function loops through all the tableoids passed even after it
> has found
> a foreign table. Why can't we return true immediately after
> finding the
> first foreign table, unless the side effects of heap_open() on
> all the
> table are required. But I don't see that to be the case, since these
> tables are locked already through a previous call to
> heap_open(). In the
> Good catch! Will fix.
> same function instead of argument name parentrelId may be we
> should use
> name parent_oid, so that we use same notation for parent and
> child table
> OIDs.
> Will fix.
Done.
> 3. Regarding enum VacuumMode - it's being used only in case of
> acquire_inherited_sample_rows(__) and that too, to check only a
> single
> value of the three defined there. May be we should just infer
> that value
> inside acquire_inherited_sample_rows(__) or pass a boolean true
> or false
> from do_analyse_rel() based on the VacuumStmt. I do not see need
> for a
> separate three value enum of which only one value is used and
> also to
> pass it down from vacuum() by changing signatures of the minion
> functions.
> I introduced that for possible future use. See the discussion in [1].
> Will leave it for the commiter to decide.
I noticed that the signatures need not to be modified, as you said.
Thanks for pointing that out! So, I revised the patch not to change the
signatures, though I left the enum, renaming it to AnalyzeMode. Let's
have the committer's review.
> 4. In postgresGetForeignPlan(), the code added by this patch is
> required
> to handle the case when the row mark is placed on a parent table and
> hence is required to be applied on the child table. We need a
> comment
> explaining this. Otherwise, the three step process to get the
> row mark
> information isn't clear for a reader.
> Will add the comment.
Done.
> 5. In expand_inherited_rtentry() why do you need a separate variable
> hasForeign? Instead of using that variable, you can actually
> set/reset
> rte->hasForeign since existence of even a single foreign child would
> mark that member as true. - After typing this, I got the answer
> when I
> looked at the function code. Every child's RTE is initially a
> copy of
> parent's RTE and hence hasForeign status would be inherited by every
> child after the first foreign child. I think, this reasoning
> should be
> added as comment before assignment to rte->hasForeign at the end
> of the
> function.
> As you mentioned, I think we could set rte->hasForeign directly
> during the scan for the inheritance set, without the separate
> variable, hasForeign. But ISTM that it'd improve code readability
> to set rte->hasForeign using the separate variable at the end of the
> function because rte->hasForeign has its meaning only when rte->inh
> is true and because we know whether rte->inh is true, at the end of
> the function.
> Fine. Please use "hasForeignChild" instead of just "hasForeign" without
> a clue as to what is "foreign" here.
Done. But I renamed it to "hasForeignChildren".
> 6. The tests in foreign_data.sql are pretty extensive. Thanks
> for that.
> I think, we should also check the effect of each of the following
> command using \d on appropriate tables.
> +CREATE FOREIGN TABLE ft2 () INHERITS (pt1)
> + SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted"
> 'value');
> +ALTER FOREIGN TABLE ft2 NO INHERIT pt1;
> +DROP FOREIGN TABLE ft2;
> +CREATE FOREIGN TABLE ft2 (
> + c1 integer NOT NULL,
> + c2 text,
> + c3 date
> +) SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted"
> 'value');
> +ALTER FOREIGN TABLE ft2 INHERIT pt1;
> Will fix.
Done.
> Apart from the above, I noticed that the patch doesn't consider to
> call ExplainForeignModify during EXPLAIN for an inherited
> UPDATE/DELETE, as shown below (note that there are no UPDATE remote
> queries displayed):
Since there seems to be no objections, I updated the patch as proposed
upthread. Here is an example.
postgres=# explain (format text, verbose) update parent as p set a = a *
2 returning *;
QUERY PLAN
--------------------------------------------------------------------------------
Update on public.parent p (cost=0.00..202.33 rows=11 width=10)
Output: p.a
For public.ft1 p_1
Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1
RETURNING a
For public.ft2 p_2
Remote SQL: UPDATE public.mytable_2 SET a = $2 WHERE ctid = $1
RETURNING a
-> Seq Scan on public.parent p (cost=0.00..0.00 rows=1 width=10)
Output: (p.a * 2), p.ctid
-> Foreign Scan on public.ft1 p_1 (cost=100.00..101.16 rows=5
width=10)
Output: (p_1.a * 2), p_1.ctid
Remote SQL: SELECT a, ctid FROM public.mytable_1 FOR UPDATE
-> Foreign Scan on public.ft2 p_2 (cost=100.00..101.16 rows=5
width=10)
Output: (p_2.a * 2), p_2.ctid
Remote SQL: SELECT a, ctid FROM public.mytable_2 FOR UPDATE
(14 rows)
Other changes:
* revised regression tests for contrib/file_fdw to refer to tableoid.
* revised docs a bit further.
Attached are updated patches. Patch fdw-inh-5.patch has been created on
top of patch fdw-chk-5.patch. Patch fdw-chk-5.patch is basically the
same as the previous one fdw-chk-4.patch, but I slightly modified that
one. The changes are the following.
* added to foreign_data.sql more tests for your comments.
* revised docs on ALTER FOREIGN TABLE a bit further.
Thanks,
Best regards,
Etsuro Fujita
Attachment
We haven't heard anything from Horiguchi-san and Hanada-san for almost a week. So, I am fine marking it as "ready for committer". What do you say?
On Wed, Dec 10, 2014 at 8:48 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote:
Hi Ashutosh,
Thanks for the review!
(2014/11/28 18:14), Ashutosh Bapat wrote:On Thu, Nov 27, 2014 at 3:52 PM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp <mailto:fujita.etsuro@lab.ntt.co.jp>> wrote:
(2014/11/17 17:55), Ashutosh Bapat wrote:
Here are my review comments for patch fdw-inh-3.patch.Tests
-------
1. It seems like you have copied from testcase inherit.sql to
postgres_fdw testcase. That's a good thing, but it makes the
test quite
long. May be we should have two tests in postgres_fdw contrib
module,
one for simple cases, and other for inheritance. What do you say?IMO, the test is not so time-consuming, so it doesn't seem worth
splitting it into two.I am not worried about the timing but I am worried about the length of
the file and hence ease of debugging in case we find any issues there.
We will leave that for the commiter to decide.
OKOKDocumentation
--------------------
1. The change in ddl.sgml
- We will refer to the child tables as partitions, though
they
- are in every way normal <productname>PostgreSQL</> tables.
+ We will refer to the child tables as partitions, though
we assume
+ that they are normal <productname>PostgreSQL</> tables.
adds phrase "we assume that", which confuses the intention
behind the
sentence. The original text is intended to highlight the equivalence
between "partition" and "normal table", where as the addition
esp. the
word "assume" weakens that equivalence. Instead now we have to
highlight
the equivalence between "partition" and "normal or foreign
table". The
wording similar to "though they are either normal or foreign tables"
should be used there.You are right, but I feel that there is something out of place in
saying that there (5.10.2. Implementing Partitioning) because the
procedure there has been written based on normal tables only. Put
another way, if we say that, I think we'd need to add more docs,
describing the syntax and/or the corresponding examples for
foreign-table cases. But I'd like to leave that for another patch.
So, how about the wording "we assume *here* that", instead of "we
assume that", as I added the following notice in the previous
section (5.10.1. Overview)?
@@ -2650,7 +2669,10 @@ VALUES ('Albany', NULL, NULL, 'NY');
table of a single parent table. The parent table itself is
normally
empty; it exists just to represent the entire data set. You
should be
familiar with inheritance (see <xref linkend="ddl-inherit">)
before
- attempting to set up partitioning.
+ attempting to set up partitioning. (The setup and management of
+ partitioned tables illustrated in this section assume that each
+ partition is a normal table. However, you can do that in a
similar way
+ for cases where some or all partitions are foreign tables.)This looks ok, though, I would like to see final version of the
document. But I think, we will leave that for committer to handle.2. The wording "some kind of optimization" gives vague picture.
May be
it can be worded as "Since the constraints are assumed to be
true, they
are used in constraint-based query optimization like constraint
exclusion for partitioned tables.".
+ Those constraints are used in some kind of query
optimization such
+ as constraint exclusion for partitioned tables (see
+ <xref linkend="ddl-partitioning">).Will follow your revision.
Done.Code
-------
1. In the following change
+/*
* acquire_inherited_sample_rows -- acquire sample rows from
inheritance tree
*
* This has the same API as acquire_sample_rows, except that
rows are
* collected from all inheritance children as well as the
specified table.
- * We fail and return zero if there are no inheritance children.
+ * We fail and return zero if there are no inheritance children or
there are
+ * inheritance children that foreign tables.
The addition should be "there are inheritance children that *are all
*foreign tables. Note the addition "are all".Sorry, I incorrectly wrote the comment. What I tried to write is
"We fail and return zero if there are no inheritance children or if
we are not in VAC_MODE_SINGLE case and inheritance tree contains at
least one foreign table.".You might want to use "English" description of VAC_MODE_SINGLE instead
of that macro in the comment, so that reader doesn't have to look up
VAC_MODE_SINGLE. But I think, we will leave this for the committer.
I corrected the comments and translated the macro into the English description.2. The function has_foreign() be better named
has_foreign_child()? ThisHow about "has_foreign_table"?has_foreign_child() would be better, since these are "children" in the
inheritance hierarchy and not mere "table"s.
Done. But I renamed it to has_foreign_children() because it sounds more natural at least to me.function loops through all the tableoids passed even after it
has found
a foreign table. Why can't we return true immediately after
finding the
first foreign table, unless the side effects of heap_open() on
all the
table are required. But I don't see that to be the case, since these
tables are locked already through a previous call to
heap_open(). In theGood catch! Will fix.same function instead of argument name parentrelId may be we
should use
name parent_oid, so that we use same notation for parent and
child table
OIDs.Will fix.
Done.3. Regarding enum VacuumMode - it's being used only in case of
acquire_inherited_sample_rows(__) and that too, to check only a
single
value of the three defined there. May be we should just infer
that value
inside acquire_inherited_sample_rows(__) or pass a boolean true
or false
from do_analyse_rel() based on the VacuumStmt. I do not see need
for a
separate three value enum of which only one value is used and
also to
pass it down from vacuum() by changing signatures of the minion
functions.I introduced that for possible future use. See the discussion in [1].Will leave it for the commiter to decide.
I noticed that the signatures need not to be modified, as you said. Thanks for pointing that out! So, I revised the patch not to change the signatures, though I left the enum, renaming it to AnalyzeMode. Let's have the committer's review.4. In postgresGetForeignPlan(), the code added by this patch is
required
to handle the case when the row mark is placed on a parent table and
hence is required to be applied on the child table. We need a
comment
explaining this. Otherwise, the three step process to get the
row mark
information isn't clear for a reader.Will add the comment.
Done.5. In expand_inherited_rtentry() why do you need a separate variable
hasForeign? Instead of using that variable, you can actually
set/reset
rte->hasForeign since existence of even a single foreign child would
mark that member as true. - After typing this, I got the answer
when I
looked at the function code. Every child's RTE is initially a
copy of
parent's RTE and hence hasForeign status would be inherited by every
child after the first foreign child. I think, this reasoning
should be
added as comment before assignment to rte->hasForeign at the end
of the
function.As you mentioned, I think we could set rte->hasForeign directly
during the scan for the inheritance set, without the separate
variable, hasForeign. But ISTM that it'd improve code readability
to set rte->hasForeign using the separate variable at the end of the
function because rte->hasForeign has its meaning only when rte->inh
is true and because we know whether rte->inh is true, at the end of
the function.Fine. Please use "hasForeignChild" instead of just "hasForeign" without
a clue as to what is "foreign" here.
Done. But I renamed it to "hasForeignChildren".6. The tests in foreign_data.sql are pretty extensive. Thanks
for that.
I think, we should also check the effect of each of the following
command using \d on appropriate tables.
+CREATE FOREIGN TABLE ft2 () INHERITS (pt1)
+ SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted"
'value');
+ALTER FOREIGN TABLE ft2 NO INHERIT pt1;
+DROP FOREIGN TABLE ft2;
+CREATE FOREIGN TABLE ft2 (
+ c1 integer NOT NULL,
+ c2 text,
+ c3 date
+) SERVER s0 OPTIONS (delimiter ',', quote '"', "be quoted"
'value');
+ALTER FOREIGN TABLE ft2 INHERIT pt1;Will fix.
Done.Apart from the above, I noticed that the patch doesn't consider to
call ExplainForeignModify during EXPLAIN for an inherited
UPDATE/DELETE, as shown below (note that there are no UPDATE remote
queries displayed):
Since there seems to be no objections, I updated the patch as proposed upthread. Here is an example.
postgres=# explain (format text, verbose) update parent as p set a = a * 2 returning *;
QUERY PLAN
--------------------------------------------------------------------------------
Update on public.parent p (cost=0.00..202.33 rows=11 width=10)
Output: p.a
For public.ft1 p_1
Remote SQL: UPDATE public.mytable_1 SET a = $2 WHERE ctid = $1 RETURNING a
For public.ft2 p_2
Remote SQL: UPDATE public.mytable_2 SET a = $2 WHERE ctid = $1 RETURNING a
-> Seq Scan on public.parent p (cost=0.00..0.00 rows=1 width=10)
Output: (p.a * 2), p.ctid
-> Foreign Scan on public.ft1 p_1 (cost=100.00..101.16 rows=5 width=10)
Output: (p_1.a * 2), p_1.ctid
Remote SQL: SELECT a, ctid FROM public.mytable_1 FOR UPDATE
-> Foreign Scan on public.ft2 p_2 (cost=100.00..101.16 rows=5 width=10)
Output: (p_2.a * 2), p_2.ctid
Remote SQL: SELECT a, ctid FROM public.mytable_2 FOR UPDATE
(14 rows)
Other changes:
* revised regression tests for contrib/file_fdw to refer to tableoid.
* revised docs a bit further.
Attached are updated patches. Patch fdw-inh-5.patch has been created on top of patch fdw-chk-5.patch. Patch fdw-chk-5.patch is basically the same as the previous one fdw-chk-4.patch, but I slightly modified that one. The changes are the following.
* added to foreign_data.sql more tests for your comments.
* revised docs on ALTER FOREIGN TABLE a bit further.
Thanks,
Best regards,
Etsuro Fujita
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Hi Ashutosh, Thanks for the review! (2014/12/10 14:47), Ashutosh Bapat wrote: > We haven't heard anything from Horiguchi-san and Hanada-san for almost a > week. So, I am fine marking it as "ready for committer". What do you say? ISTM that both of them are not against us, so let's ask for the committer's review! Thanks, Best regards, Etsuro Fujita
<div dir="ltr">I marked this as ready for committer.<br /></div><div class="gmail_extra"><br /><div class="gmail_quote">OnThu, Dec 11, 2014 at 8:39 AM, Etsuro Fujita <span dir="ltr"><<a href="mailto:fujita.etsuro@lab.ntt.co.jp"target="_blank">fujita.etsuro@lab.ntt.co.jp</a>></span> wrote:<br /><blockquoteclass="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">HiAshutosh,<br /><br /> Thanks for the review!<br /><br /></span><span class=""> (2014/12/10 14:47), Ashutosh Bapatwrote:<br /><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Wehaven't heard anything from Horiguchi-san and Hanada-san for almost a<br /> week. So, I am fine marking it as "ready forcommitter". What do you say?<br /></blockquote><br /></span> ISTM that both of them are not against us, so let's ask forthe committer's review!<div class="HOEnZb"><div class="h5"><br /><br /> Thanks,<br /><br /> Best regards,<br /> EtsuroFujita<br /></div></div></blockquote></div><br /><br clear="all" /><br />-- <br /><div class="gmail_signature"><divdir="ltr">Best Wishes,<br />Ashutosh Bapat<br />EnterpriseDB Corporation<br />The Postgres DatabaseCompany<br /></div></div></div>
(2014/12/11 14:54), Ashutosh Bapat wrote: > I marked this as ready for committer. Many thanks! Best regards, Etsuro Fujita
On Thu, Dec 11, 2014 at 2:54 PM, Ashutosh Bapat <ashutosh.bapat@enterprisedb.com> wrote: > On Thu, Dec 11, 2014 at 8:39 AM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> > wrote: >> >> Hi Ashutosh, >> >> Thanks for the review! >> >> (2014/12/10 14:47), Ashutosh Bapat wrote: >>> >>> We haven't heard anything from Horiguchi-san and Hanada-san for almost a >>> week. So, I am fine marking it as "ready for committer". What do you say? Moving this patch to CF 2014-12 with the same status. Let's get a committer having a look at it. -- Michael
Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
> Attached are updated patches. Patch fdw-inh-5.patch has been created on
> top of patch fdw-chk-5.patch. Patch fdw-chk-5.patch is basically the
> same as the previous one fdw-chk-4.patch, but I slightly modified that
> one. The changes are the following.
> * added to foreign_data.sql more tests for your comments.
> * revised docs on ALTER FOREIGN TABLE a bit further.
I've committed fdw-chk-5.patch with some minor further adjustments;
the most notable one was that I got rid of the error check prohibiting
NO INHERIT, which did not seem to me to have any value. Attaching such
a clause won't have any effect, but so what?
Have not looked at the other patch yet.
regards, tom lane
(2014/12/18 7:04), Tom Lane wrote: > Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes: >> Attached are updated patches. Patch fdw-inh-5.patch has been created on >> top of patch fdw-chk-5.patch. Patch fdw-chk-5.patch is basically the >> same as the previous one fdw-chk-4.patch, but I slightly modified that >> one. The changes are the following. >> * added to foreign_data.sql more tests for your comments. >> * revised docs on ALTER FOREIGN TABLE a bit further. > > I've committed fdw-chk-5.patch with some minor further adjustments; > the most notable one was that I got rid of the error check prohibiting > NO INHERIT, which did not seem to me to have any value. Attaching such > a clause won't have any effect, but so what? > > Have not looked at the other patch yet. Thanks! I added the error check because the other patch, fdw-inh-5.patch, doesn't allow foreign tables to be inherited and so it seems more consistent at least to me to do so. Best regards, Etsuro Fujita
On 2014/12/18 7:04, Tom Lane wrote: > Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes: >> Attached are updated patches. Patch fdw-inh-5.patch has been created on >> top of patch fdw-chk-5.patch. > I've committed fdw-chk-5.patch with some minor further adjustments; > Have not looked at the other patch yet. I updated the remaining patch correspondingly to the fix [1]. Please find attached a patch (the patch has been created on top of the patch in [1]). I haven't done anything about the issue that postgresGetForeignPlan() is using get_parse_rowmark(), discussed in eg, [2]. Tom, will you work on that? Thanks, [1] http://www.postgresql.org/message-id/5497BF4C.6080302@lab.ntt.co.jp [2] http://www.postgresql.org/message-id/18256.1418401027@sss.pgh.pa.us Best regards, Etsuro Fujita
Attachment
Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
> I haven't done anything about the issue that postgresGetForeignPlan() is
> using get_parse_rowmark(), discussed in eg, [2]. Tom, will you work on
> that?
Yeah, we need to do something about the PlanRowMark data structure.
Aside from the pre-existing issue in postgres_fdw, we need to fix it
to support inheritance trees in which more than one rowmark method
is being used. That rte.hasForeignChildren thing is a crock, and
would still be a crock even if it were correctly documented as being
a planner temporary variable (rather than the current implication that
it's always valid). RangeTblEntry is no place for planner temporaries.
The idea I'd had about that was to convert the markType field into a
bitmask, so that a parent node's markType could represent the logical
OR of the rowmark methods being used by all its children. I've not
attempted to code this up though, and probably won't get to it until
after Christmas. One thing that's not clear is what should happen
with ExecRowMark.
regards, tom lane
On 2014/12/23 0:36, Tom Lane wrote: > Yeah, we need to do something about the PlanRowMark data structure. > Aside from the pre-existing issue in postgres_fdw, we need to fix it > to support inheritance trees in which more than one rowmark method > is being used. That rte.hasForeignChildren thing is a crock, and > would still be a crock even if it were correctly documented as being > a planner temporary variable (rather than the current implication that > it's always valid). RangeTblEntry is no place for planner temporaries. Agreed. > The idea I'd had about that was to convert the markType field into a > bitmask, so that a parent node's markType could represent the logical > OR of the rowmark methods being used by all its children. I've not > attempted to code this up though, and probably won't get to it until > after Christmas. One thing that's not clear is what should happen > with ExecRowMark. That seems like a good idea, as parent PlanRowMarks are ignored at runtime. Aside from the above, I noticed that the patch has a bug in handling ExecRowMarks/ExecAuxRowMarks for foreign tables in inheritance trees during the EPQ processing.:-( Attached is an updated version of the patch to fix that, which has been created on top of [1], as said before. Thanks, [1] http://www.postgresql.org/message-id/5497BF4C.6080302@lab.ntt.co.jp Best regards, Etsuro Fujita
Attachment
On 2014/12/23 0:36, Tom Lane wrote: > Yeah, we need to do something about the PlanRowMark data structure. > Aside from the pre-existing issue in postgres_fdw, we need to fix it > to support inheritance trees in which more than one rowmark method > is being used. That rte.hasForeignChildren thing is a crock, > The idea I'd had about that was to convert the markType field into a > bitmask, so that a parent node's markType could represent the logical > OR of the rowmark methods being used by all its children. I've not > attempted to code this up though, and probably won't get to it until > after Christmas. One thing that's not clear is what should happen > with ExecRowMark. As I said before, that seems to me like a good idea. So I'll update the patch based on that if you're okey with it. Or you've found any problem concerning the above idea? Best regards, Etsuro Fujita
<div dir="ltr"><br /><div class="gmail_extra"><br /><div class="gmail_quote">On Thu, Jan 15, 2015 at 4:35 PM, Etsuro Fujita<span dir="ltr"><<a href="mailto:fujita.etsuro@lab.ntt.co.jp" target="_blank">fujita.etsuro@lab.ntt.co.jp</a>></span>wrote:<br /><blockquote class="gmail_quote" style="margin:0 0 0.8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""></span>As I said before, that seems to me like a good idea. So I'll update the<br /> patch based on that if you're okey with it. Or you've found any problem<br /> concerningthe above idea?<br /></blockquote></div><br /></div><div class="gmail_extra">Patch moved to CF 2015-02 with samestatus, "Ready for committer".<br />-- <br /><div class="gmail_signature">Michael<br /></div></div></div>
On 2015/01/15 16:35, Etsuro Fujita wrote: > On 2014/12/23 0:36, Tom Lane wrote: >> Yeah, we need to do something about the PlanRowMark data structure. >> Aside from the pre-existing issue in postgres_fdw, we need to fix it >> to support inheritance trees in which more than one rowmark method >> is being used. That rte.hasForeignChildren thing is a crock, > >> The idea I'd had about that was to convert the markType field into a >> bitmask, so that a parent node's markType could represent the logical >> OR of the rowmark methods being used by all its children. > As I said before, that seems to me like a good idea. So I'll update the > patch based on that if you're okey with it. Done based on your ideas: (a) add a field to PlanRowMark to record the original lock strength to fix the postgres_fdw issue and (b) convert its markType field into a bitmask to support the inheritance trees. I think that both work well and that (a) is useful for the other places. Patch attached, which has been created on top of [1]. Best regards, Etsuro Fujita [1] http://www.postgresql.org/message-id/54BCBBF8.3020103@lab.ntt.co.jp
Attachment
Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
> I noticed that the latter disallows TRUNCATE on inheritance trees that
> contain at least one child foreign table. But I think it would be
> better to allow it, with the semantics that we quietly ignore the child
> foreign tables and apply the operation to the child plain tables, which
> is the same semantics as ALTER COLUMN SET STORAGE on such inheritance
> trees. Comments welcome.
I've been working through the foreign table inheritance patch, and found
the code that makes the above happen. I don't think this is a good idea
at all. In the first place, successful TRUNCATE should leave the table
empty, not "well, we'll make it empty if we feel up to that". In the
second place, someday we might want to make TRUNCATE actually work for
foreign tables (at least for FDWs that want to support it). If we did,
we would have a backwards-compatibility hazard, because suddenly a
TRUNCATE on an inheritance tree that includes a foreign table would have
different non-error effects than before.
I think we should just throw error in this case.
BTW, the SET STORAGE comparison is bogus as well. I see no reason that
we shouldn't just allow SET STORAGE on foreign tables. It's probably
not going to have any effect, but so what? And again, if we did ever
find a use for that, we'd have a compatibility problem if inherited SET
STORAGE has a pre-existing behavior that it skips foreign children.
In the same vein, I'm planning to take out the existing prohibition on
marking CHECK constraints on foreign tables NOT VALID. That likewise
creates a corner case for inheritance trees for no obviously good reason.
It was reasonable to be conservative about whether to allow that so long
as there were no side-effects; but putting warts into the behavior of
inheritance trees to preserve the prohibition is not a good outcome.
regards, tom lane
Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
> [ fdw-inh-8.patch ]
I've committed this with some substantial rearrangements, notably:
* I thought that if we were doing this at all, we should go all the way
and allow foreign tables to be both inheritance parents and children.
* As I mentioned earlier, I got rid of a few unnecessary restrictions on
foreign tables so as to avoid introducing warts into inheritance behavior.
In particular, we now allow NOT VALID CHECK constraints (and hence ALTER
... VALIDATE CONSTRAINT), ALTER SET STORAGE, and ALTER SET WITH/WITHOUT
OIDS. These are probably no-ops anyway for foreign tables, though
conceivably an FDW might choose to implement some behavior for STORAGE
or OIDs.
* I did not like the EXPLAIN changes at all; in the first place they
resulted in invalid JSON output (there could be multiple fields of the
Update plan object with identical labels), and in the second place it
seemed like a bad idea to rely on FDWs to change the behavior of their
ExplainModifyTarget functions. I've refactored that so that explain.c
remains responsible for getting the grouping right. Also, as I said
earlier, it seemed like a good idea to produce subgroups identifying
all the target tables not only the foreign ones.
* I fooled around with the PlanRowMark changes some more, mainly with
the idea that we might soon allow FDWs to use rowmark methods other than
ROW_MARK_COPY. The planner now has just one place where a rel's rowmark
method is chosen, so as to centralize anything we need to do there.
regards, tom lane
On Sun, Mar 22, 2015 at 1:57 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes: >> [ fdw-inh-8.patch ] > > I've committed this with some substantial rearrangements, notably: I'm really glad this is going in! Thanks to to Shigeru Hanada and Etsuro Fujita for working on this, to you (Tom) for putting in the time to get it committed, and of course to the reviewers Ashutosh Bapat and Kyotaro Horiguchi for their time and effort. In a way, I believe we can think of this as the beginnings of a sharding story for PostgreSQL. A lot more work is needed, of course -- join and aggregate pushdown are high on my personal list -- but it's a start. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Mon, Mar 23, 2015 at 12:09 AM, Robert Haas <robertmhaas@gmail.com> wrote:
On Sun, Mar 22, 2015 at 1:57 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
>> [ fdw-inh-8.patch ]
>
> I've committed this with some substantial rearrangements, notably:
I'm really glad this is going in! Thanks to to Shigeru Hanada and
Etsuro Fujita for working on this, to you (Tom) for putting in the
time to get it committed, and of course to the reviewers Ashutosh
Bapat and Kyotaro Horiguchi for their time and effort.
In a way, I believe we can think of this as the beginnings of a
sharding story for PostgreSQL. A lot more work is needed, of course
-- join and aggregate pushdown are high on my personal list -- but
it's a start.
+1.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company
On 2015/03/23 2:57, Tom Lane wrote: > I've committed this with some substantial rearrangements, notably: Thanks for taking the time to committing the patch! Thanks for the work, Hanada-san! And thank you everyone for the reviews and comments, especially Ashutosh, Horiguchi-san and Noah! > * I fooled around with the PlanRowMark changes some more, mainly with > the idea that we might soon allow FDWs to use rowmark methods other than > ROW_MARK_COPY. The planner now has just one place where a rel's rowmark > method is chosen, so as to centralize anything we need to do there. Will work on this issue. Best regards, Etsuro Fujita
On 2015/03/23 2:57, Tom Lane wrote: > Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes: >> [ fdw-inh-8.patch ] > > I've committed this with some substantial rearrangements, notably: > > * I thought that if we were doing this at all, we should go all the way > and allow foreign tables to be both inheritance parents and children. I found that when setting a foreign table to be the parent of an inheritance set that only contains foreign tables, SELECT FOR UPDATE on the inheritance parent fails with a can't-happen error condition. Here is an example: $ createdb mydb $ psql mydb psql (9.5devel) Type "help" for help. mydb=# create table t1 (c1 int); CREATE TABLE mydb=# create table t2 (c1 int); CREATE TABLE $ psql postgres psql (9.5devel) Type "help" for help. postgres=# create extension postgres_fdw; CREATE EXTENSION postgres=# create server myserver foreign data wrapper postgres_fdw options (dbname 'mydb'); CREATE SERVER postgres=# create user mapping for current_user server myserver; CREATE USER MAPPING postgres=# create foreign table ft1 (c1 int) server myserver options (table_name 't1'); CREATE FOREIGN TABLE postgres=# create foreign table ft2 (c1 int) server myserver options (table_name 't2'); CREATE FOREIGN TABLE postgres=# alter foreign table ft2 inherit ft1; ALTER FOREIGN TABLE postgres=# select * from ft1 for update; ERROR: could not find junk tableoid1 column I think this is a bug. Attached is a patch fixing this issue. Best regards, Etsuro Fujita
Attachment
On 4/14/15 5:49 AM, Etsuro Fujita wrote: > postgres=# create foreign table ft1 (c1 int) server myserver options > (table_name 't1'); > CREATE FOREIGN TABLE > postgres=# create foreign table ft2 (c1 int) server myserver options > (table_name 't2'); > CREATE FOREIGN TABLE > postgres=# alter foreign table ft2 inherit ft1; > ALTER FOREIGN TABLE > postgres=# select * from ft1 for update; > ERROR: could not find junk tableoid1 column > > I think this is a bug. Attached is a patch fixing this issue. What happens when the foreign side breaks the inheritance? Does the FDW somehow know to check that fact for each query? What do you gain from having the local table have inheritance? Basically, I think we have to be very careful about implementing features that imply the local side knows something about the persisted state of the remote side (in this case, whether there's inheritance). Anything like that sets us up for synchronization problems. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com
Jim Nasby wrote: > On 4/14/15 5:49 AM, Etsuro Fujita wrote: > > postgres=# create foreign table ft1 (c1 int) server myserver options > > (table_name 't1'); > > CREATE FOREIGN TABLE > > postgres=# create foreign table ft2 (c1 int) server myserver options > > (table_name 't2'); > > CREATE FOREIGN TABLE > > postgres=# alter foreign table ft2 inherit ft1; > > ALTER FOREIGN TABLE > > postgres=# select * from ft1 for update; > > ERROR: could not find junk tableoid1 column > > > > I think this is a bug. Attached is a patch fixing this issue. > > What happens when the foreign side breaks the inheritance? Does the FDW > somehow know to check that fact for each query? This is a meaningless question. The remote tables don't have to have an inheritance relationship already; only the local side sees them as connected. I think the real question is whether we're now (I mean after this patch) emitting useless tableoid columns that we didn't previously have. I think the answer is yes, and if so I think we need a smaller comb to fix the problem. This one seems rather large. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Hi, Before suppressing the symptom, I doubt the necessity and/or validity of giving foreign tables an ability to be a parent. Is there any reasonable usage for the ability? I think we should choose to inhibit foreign tables from becoming a parent rather than leaving it allowed then taking measures for the consequent symptom. regards, At Tue, 14 Apr 2015 15:52:18 -0300, Alvaro Herrera <alvherre@2ndquadrant.com> wrote in <20150414185218.GX4369@alvh.no-ip.org> > Jim Nasby wrote: > > On 4/14/15 5:49 AM, Etsuro Fujita wrote: > > > postgres=# create foreign table ft1 (c1 int) server myserver options > > > (table_name 't1'); > > > CREATE FOREIGN TABLE > > > postgres=# create foreign table ft2 (c1 int) server myserver options > > > (table_name 't2'); > > > CREATE FOREIGN TABLE > > > postgres=# alter foreign table ft2 inherit ft1; > > > ALTER FOREIGN TABLE > > > postgres=# select * from ft1 for update; > > > ERROR: could not find junk tableoid1 column > > > > > > I think this is a bug. Attached is a patch fixing this issue. > > > > What happens when the foreign side breaks the inheritance? Does the FDW > > somehow know to check that fact for each query? > > This is a meaningless question. The remote tables don't have to have an > inheritance relationship already; only the local side sees them as > connected. > > I think the real question is whether we're now (I mean after this patch) > emitting useless tableoid columns that we didn't previously have. I > think the answer is yes, and if so I think we need a smaller comb to fix > the problem. This one seems rather large. -- Kyotaro Horiguchi NTT Open Source Software Center
On 2015/04/15 3:52, Alvaro Herrera wrote: >> On 4/14/15 5:49 AM, Etsuro Fujita wrote: >>> postgres=# create foreign table ft1 (c1 int) server myserver options >>> (table_name 't1'); >>> CREATE FOREIGN TABLE >>> postgres=# create foreign table ft2 (c1 int) server myserver options >>> (table_name 't2'); >>> CREATE FOREIGN TABLE >>> postgres=# alter foreign table ft2 inherit ft1; >>> ALTER FOREIGN TABLE >>> postgres=# select * from ft1 for update; >>> ERROR: could not find junk tableoid1 column >>> >>> I think this is a bug. Attached is a patch fixing this issue. > I think the real question is whether we're now (I mean after this patch) > emitting useless tableoid columns that we didn't previously have. I > think the answer is yes, and if so I think we need a smaller comb to fix > the problem. This one seems rather large. My answer for that would be *no* because I think tableoid would be needed for EvalPlanQual checking in more complex SELECT FOR UPDATE on the inheritance or UPDATE/DELETE involving the inheritance as a source table. Also, if we allow the FDW to change the behavior of SELECT FOR UPDATE so as to match the local semantics exactly, which I'm working on in another thread, I think tableoid would also be needed for the actual row locking. Best regards, Etsuro Fujita
Hello, At Thu, 16 Apr 2015 12:20:47 +0900, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote in <552F2A8F.2090406@lab.ntt.co.jp> > On 2015/04/15 3:52, Alvaro Herrera wrote: > >> On 4/14/15 5:49 AM, Etsuro Fujita wrote: > >>> postgres=# create foreign table ft1 (c1 int) server myserver options > >>> (table_name 't1'); > >>> CREATE FOREIGN TABLE > >>> postgres=# create foreign table ft2 (c1 int) server myserver options > >>> (table_name 't2'); > >>> CREATE FOREIGN TABLE > >>> postgres=# alter foreign table ft2 inherit ft1; > >>> ALTER FOREIGN TABLE > >>> postgres=# select * from ft1 for update; > >>> ERROR: could not find junk tableoid1 column > >>> > >>> I think this is a bug. Attached is a patch fixing this issue. > > > I think the real question is whether we're now (I mean after this > > patch) > > emitting useless tableoid columns that we didn't previously have. I > > think the answer is yes, and if so I think we need a smaller comb to > > fix > > the problem. This one seems rather large. > > My answer for that would be *no* because I think tableoid would be > needed for EvalPlanQual checking in more complex SELECT FOR UPDATE on > the inheritance or UPDATE/DELETE involving the inheritance as a source > table. Also, if we allow the FDW to change the behavior of SELECT FOR > UPDATE so as to match the local semantics exactly, which I'm working > on in another thread, I think tableoid would also be needed for the > actual row locking. Given the parent foreign talbes, surely they need tableoids for such usage. The patch preserves the condition rc->isParent so it newly affects exactly only parent foreign tables for now. Before the parent foreign tables introduced, ROW_MARK_COPY and RTE_RELATION are mutually exclusive so didn't need, or cannot have tableoid. But now it intorduces an rte with ROW_MARK_COPY & RTE_RELATION and there seems no reason for parent tables in any kind not to have tableoid. After such consideration, I came to think that the patch is a reasonable fix, not mere a workaround. Thoughts? regards, -- Kyotaro Horiguchi NTT Open Source Software Center
On 2015/03/23 2:57, Tom Lane wrote: > Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes: >> [ fdw-inh-8.patch ] > > I've committed this with some substantial rearrangements, notably: > * As I mentioned earlier, I got rid of a few unnecessary restrictions on > foreign tables so as to avoid introducing warts into inheritance behavior. > In particular, we now allow NOT VALID CHECK constraints (and hence ALTER > ... VALIDATE CONSTRAINT), ALTER SET STORAGE, and ALTER SET WITH/WITHOUT > OIDS. These are probably no-ops anyway for foreign tables, though > conceivably an FDW might choose to implement some behavior for STORAGE > or OIDs. I agree with you on this point. However, ISTM there is a bug in handling OIDs on foreign tables; while we now allow for ALTER SET WITH/WITHOUT OIDS, we still don't allow the default_with_oids parameter for foreign tables. I think that since CREATE FOREIGN TABLE should be consistent with ALTER FOREIGN TABLE, we should also allow the parameter for foreign tables. Attached is a patch for that. Best regards, Etsuro Fujita
Attachment
On Wed, Apr 15, 2015 at 09:35:05AM +0900, Kyotaro HORIGUCHI wrote: > Hi, > > Before suppressing the symptom, I doubt the necessity and/or > validity of giving foreign tables an ability to be a parent. Is > there any reasonable usage for the ability? > > I think we should choose to inhibit foreign tables from becoming > a parent rather than leaving it allowed then taking measures for > the consequent symptom. I have a use case for having foreign tables as non-leaf nodes in a partitioning hierarchy, namely geographic. One might have a table at HQ called foo_world, then partitions under it called foo_jp, foo_us, etc., in one level, foo_us_ca, foo_us_pa, etc. in the next level, and on down, each in general in a separate data center. Is there something essential about having non-leaf nodes as foreign tables that's a problem here? Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
On 2015/04/16 16:05, Etsuro Fujita wrote: > On 2015/03/23 2:57, Tom Lane wrote: >> Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes: >>> [ fdw-inh-8.patch ] >> >> I've committed this with some substantial rearrangements, notably: > >> * As I mentioned earlier, I got rid of a few unnecessary restrictions on >> foreign tables so as to avoid introducing warts into inheritance behavior. >> In particular, we now allow NOT VALID CHECK constraints (and hence ALTER >> ... VALIDATE CONSTRAINT), ALTER SET STORAGE, and ALTER SET WITH/WITHOUT >> OIDS. These are probably no-ops anyway for foreign tables, though >> conceivably an FDW might choose to implement some behavior for STORAGE >> or OIDs. > > I agree with you on this point. However, ISTM there is a bug in > handling OIDs on foreign tables; while we now allow for ALTER SET > WITH/WITHOUT OIDS, we still don't allow the default_with_oids parameter > for foreign tables. I think that since CREATE FOREIGN TABLE should be > consistent with ALTER FOREIGN TABLE, we should also allow the parameter > for foreign tables. Attached is a patch for that. I also updated docs. Attached is an updated version of the patch. Best regards, Etsuro Fujita
Attachment
Etsuro, * Etsuro Fujita (fujita.etsuro@lab.ntt.co.jp) wrote: > postgres=# select * from ft1 for update; > ERROR: could not find junk tableoid1 column > > I think this is a bug. Attached is a patch fixing this issue. Pushed, thanks! Stephen
On Tue, Apr 14, 2015 at 8:35 PM, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote: > Before suppressing the symptom, I doubt the necessity and/or > validity of giving foreign tables an ability to be a parent. Is > there any reasonable usage for the ability? Gee, I don't see why that would be unreasonable or invalid -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Hello, At Thu, 16 Apr 2015 14:43:33 -0700, David Fetter <david@fetter.org> wrote in <20150416214333.GA797@fetter.org> > On Wed, Apr 15, 2015 at 09:35:05AM +0900, Kyotaro HORIGUCHI wrote: > > Hi, > > > > Before suppressing the symptom, I doubt the necessity and/or > > validity of giving foreign tables an ability to be a parent. Is > > there any reasonable usage for the ability? ... > I have a use case for having foreign tables as non-leaf nodes in a > partitioning hierarchy, namely geographic. Ah, I see. I understood the case of intermediate nodes. I agree that it is quite natural. > One might have a table at > HQ called foo_world, then partitions under it called foo_jp, foo_us, > etc., in one level, foo_us_ca, foo_us_pa, etc. in the next level, and > on down, each in general in a separate data center. > > Is there something essential about having non-leaf nodes as foreign > tables that's a problem here? No. I'm convinced of the necessity. Sorry for the noise. At Wed, 22 Apr 2015 17:00:10 -0400, Robert Haas <robertmhaas@gmail.com> wrote in <CA+TgmobZVHp3D9wWCV8QJc+qGDu7=tEKNCbXOwijZKhjuCmRWg@mail.gmail.com> > Gee, I don't see why that would be unreasonable or invalid Hmm. Yes, as mentioned above, there's no reason to refuse non-leaf foregin tables. I didn't understood the real cause of the problem and thought that not allowing foreign *root* tables seem better than tweaking elsewhere. But that thought found to be totally a garbage :( regards, -- Kyotaro Horiguchi NTT Open Source Software Center
On 2015/04/23 0:34, Stephen Frost wrote: > * Etsuro Fujita (fujita.etsuro@lab.ntt.co.jp) wrote: >> postgres=# select * from ft1 for update; >> ERROR: could not find junk tableoid1 column >> >> I think this is a bug. Attached is a patch fixing this issue. > > Pushed, thanks! Thank you. Best regards, Etsuro Fujita
Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> writes:
> On 2015/04/16 16:05, Etsuro Fujita wrote:
>> I agree with you on this point. However, ISTM there is a bug in
>> handling OIDs on foreign tables; while we now allow for ALTER SET
>> WITH/WITHOUT OIDS, we still don't allow the default_with_oids parameter
>> for foreign tables. I think that since CREATE FOREIGN TABLE should be
>> consistent with ALTER FOREIGN TABLE, we should also allow the parameter
>> for foreign tables. Attached is a patch for that.
> I also updated docs. Attached is an updated version of the patch.
I believe that we intentionally did not do this, and here is why not:
existing pg_dump files assume that default_with_oids doesn't affect any
relation type except plain tables. pg_backup_archiver.c only bothers
to change the GUC when about to dump a plain table, and otherwise leaves
it at its previous value. That means if we apply a patch like this, it's
entirely possible that pg_dump/pg_restore will result in foreign tables
accidentally acquiring OID columns.
Since default_with_oids is really only meant as a backwards-compatibility
hack, I don't personally have a problem with restricting it to act only on
plain tables. However, it might be a good idea to explicitly document
this interaction in a code comment to prevent anyone from re-inventing
this idea... I'll go do that.
regards, tom lane
On 2015-04-25 20:47:04 -0400, Tom Lane wrote: > Since default_with_oids is really only meant as a backwards-compatibility > hack, I don't personally have a problem with restricting it to act only on > plain tables. FWIW, I think we're getting pretty close to the point, or are there even, where we can remove default_with_oids. So not adding complications because of it sounds good to me. Greetings, Andres Freund
Andres Freund <andres@anarazel.de> writes:
> FWIW, I think we're getting pretty close to the point, or are there
> even, where we can remove default_with_oids. So not adding complications
> because of it sounds good to me.
Well, pg_dump uses it --- so the argument about not breaking old dump
scripts would apply to any attempt to remove it entirely. But I don't
have a problem with saying that its semantics are frozen.
regards, tom lane