Thread: [HACKERS] foreign table creation and NOT VALID check constraints
In f27a6b15e656 (9.6 & later), we decided to "Mark CHECK constraints declared NOT VALID valid if created with table." In retrospect, constraints on foreign tables should have been excluded from consideration in that commit, because the thinking behind the aforementioned commit (that the constraint is trivially validated because the newly created table contains no data) does not equally apply to the foreign tables case. Should we do something about that? Thanks, Amit
On 1 August 2017 at 07:16, Amit Langote <Langote_Amit_f8@lab.ntt.co.jp> wrote: > In f27a6b15e656 (9.6 & later), we decided to "Mark CHECK constraints > declared NOT VALID valid if created with table." In retrospect, > constraints on foreign tables should have been excluded from consideration > in that commit, because the thinking behind the aforementioned commit > (that the constraint is trivially validated because the newly created > table contains no data) does not equally apply to the foreign tables case. > > Should we do something about that? In what way does it not apply? Do you have a failure case? -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2017/08/01 15:22, Simon Riggs wrote: > On 1 August 2017 at 07:16, Amit Langote <Langote_Amit_f8@lab.ntt.co.jp> wrote: >> In f27a6b15e656 (9.6 & later), we decided to "Mark CHECK constraints >> declared NOT VALID valid if created with table." In retrospect, >> constraints on foreign tables should have been excluded from consideration >> in that commit, because the thinking behind the aforementioned commit >> (that the constraint is trivially validated because the newly created >> table contains no data) does not equally apply to the foreign tables case. >> >> Should we do something about that? > > In what way does it not apply? Do you have a failure case? Sorry for not mentioning the details. I was thinking that a foreign table starts containing the data of the remote object it points to the moment it's created (unlike local tables which contain no data to begin with). If a user is not sure whether a particular constraint being created in the same command holds for the remote data, they may mark it as NOT VALID and hope that the system treats the constraint as such until such a time that they mark it valid by running ALTER TABLE VALIDATE CONSTRAINT. Since the planner is the only consumer of pg_constraint.convalidated, that means the user expects the planner to ignore such a constraint. Since f27a6b15e656, users are no longer able to expect so. For example: create extension postgres_fdw; create server loopback foreign data wrapper postgres_fdw options (dbname 'postgres'); create table foo (a) as select 1; create foreign table ffoo ( a int, constraint check_a check (a = 0) not valid ) server loopback options (table_name 'foo'); set constraint_exclusion to on; -- constraint check_a will exclude the table select * from ffoo where a = 1;a --- (0 rows) explain select * from ffoo where a = 1; QUERY PLAN ------------------------------------------Result (cost=0.00..0.00 rows=0 width=4) One-Time Filter: false (2 rows) The above "issue" doesn't occur if the constraint is created after-the-fact, whereby it's possible to specify NOT VALID and have the system actually store it in the catalog as such. alter foreign table ffoo drop constraint check_a; alter foreign table ffoo add constraint check_a check (a = 0) not valid; select * from ffoo where a = 1;a ---1 (1 row) explain select * from ffoo where a = 1; QUERY PLAN -------------------------------------------------------------Foreign Scan on ffoo (cost=100.00..146.86 rows=15 width=4) Maybe, this is not such a big matter though, because the core system doesn't actually enforce any constraints of the foreign tables locally anyway (which is a documented fact), but my concern is the possibility of some considering this a POLA violation. Lack of complaints so far perhaps means nobody did. Thanks, Amit
On 1 August 2017 at 08:37, Amit Langote <Langote_Amit_f8@lab.ntt.co.jp> wrote: > On 2017/08/01 15:22, Simon Riggs wrote: >> On 1 August 2017 at 07:16, Amit Langote <Langote_Amit_f8@lab.ntt.co.jp> wrote: >>> In f27a6b15e656 (9.6 & later), we decided to "Mark CHECK constraints >>> declared NOT VALID valid if created with table." In retrospect, >>> constraints on foreign tables should have been excluded from consideration >>> in that commit, because the thinking behind the aforementioned commit >>> (that the constraint is trivially validated because the newly created >>> table contains no data) does not equally apply to the foreign tables case. >>> >>> Should we do something about that? >> >> In what way does it not apply? Do you have a failure case? > > Sorry for not mentioning the details. > > I was thinking that a foreign table starts containing the data of the > remote object it points to the moment it's created (unlike local tables > which contain no data to begin with). If a user is not sure whether a > particular constraint being created in the same command holds for the > remote data, they may mark it as NOT VALID and hope that the system treats > the constraint as such until such a time that they mark it valid by > running ALTER TABLE VALIDATE CONSTRAINT. Since the planner is the only > consumer of pg_constraint.convalidated, that means the user expects the > planner to ignore such a constraint. Since f27a6b15e656, users are no > longer able to expect so. For Foreign Tables, it sounds like an issue. Don't think it exists for normal tables. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2017/08/01 17:54, Simon Riggs wrote: > On 1 August 2017 at 08:37, Amit Langote <Langote_Amit_f8@lab.ntt.co.jp> wrote: >> On 2017/08/01 15:22, Simon Riggs wrote: >>> On 1 August 2017 at 07:16, Amit Langote <Langote_Amit_f8@lab.ntt.co.jp> wrote: >>>> In f27a6b15e656 (9.6 & later), we decided to "Mark CHECK constraints >>>> declared NOT VALID valid if created with table." In retrospect, >>>> constraints on foreign tables should have been excluded from consideration >>>> in that commit, because the thinking behind the aforementioned commit >>>> (that the constraint is trivially validated because the newly created >>>> table contains no data) does not equally apply to the foreign tables case. >>>> >>>> Should we do something about that? >>> >>> In what way does it not apply? Do you have a failure case? >> >> Sorry for not mentioning the details. >> >> I was thinking that a foreign table starts containing the data of the >> remote object it points to the moment it's created (unlike local tables >> which contain no data to begin with). If a user is not sure whether a >> particular constraint being created in the same command holds for the >> remote data, they may mark it as NOT VALID and hope that the system treats >> the constraint as such until such a time that they mark it valid by >> running ALTER TABLE VALIDATE CONSTRAINT. Since the planner is the only >> consumer of pg_constraint.convalidated, that means the user expects the >> planner to ignore such a constraint. Since f27a6b15e656, users are no >> longer able to expect so. > > For Foreign Tables, it sounds like an issue. Don't think it exists for > normal tables. Yes. I was saying in my first email that we should not disregard user's request to mark a constraint NOT VALID if the table in question is a *foreign table*. So, it's OK that commit f27a6b15e656 changed things to ignore NOT VALID if it's in the following command: create table foo (a int, constraint check_a check (a > 0) not valid); But, not OK in the following command: create foreign table ffoo ( a int, constraint check_a check (a > 0) not valid ) server loopback options (table_name 'foo'); Thanks, Amit
On Tue, Aug 1, 2017 at 2:41 PM, Amit Langote <Langote_Amit_f8@lab.ntt.co.jp> wrote: > On 2017/08/01 17:54, Simon Riggs wrote: >> On 1 August 2017 at 08:37, Amit Langote <Langote_Amit_f8@lab.ntt.co.jp> wrote: >>> On 2017/08/01 15:22, Simon Riggs wrote: >>>> On 1 August 2017 at 07:16, Amit Langote <Langote_Amit_f8@lab.ntt.co.jp> wrote: >>>>> In f27a6b15e656 (9.6 & later), we decided to "Mark CHECK constraints >>>>> declared NOT VALID valid if created with table." In retrospect, >>>>> constraints on foreign tables should have been excluded from consideration >>>>> in that commit, because the thinking behind the aforementioned commit >>>>> (that the constraint is trivially validated because the newly created >>>>> table contains no data) does not equally apply to the foreign tables case. >>>>> >>>>> Should we do something about that? >>>> >>>> In what way does it not apply? Do you have a failure case? >>> >>> Sorry for not mentioning the details. >>> >>> I was thinking that a foreign table starts containing the data of the >>> remote object it points to the moment it's created (unlike local tables >>> which contain no data to begin with). If a user is not sure whether a >>> particular constraint being created in the same command holds for the >>> remote data, they may mark it as NOT VALID and hope that the system treats >>> the constraint as such until such a time that they mark it valid by >>> running ALTER TABLE VALIDATE CONSTRAINT. Since the planner is the only >>> consumer of pg_constraint.convalidated, that means the user expects the >>> planner to ignore such a constraint. Since f27a6b15e656, users are no >>> longer able to expect so. >> >> For Foreign Tables, it sounds like an issue. Don't think it exists for >> normal tables. > > Yes. I was saying in my first email that we should not disregard user's > request to mark a constraint NOT VALID if the table in question is a > *foreign table*. > > So, it's OK that commit f27a6b15e656 changed things to ignore NOT VALID if > it's in the following command: > > create table foo (a int, constraint check_a check (a > 0) not valid); > > But, not OK in the following command: > > create foreign table ffoo ( > a int, > constraint check_a check (a > 0) not valid > ) server loopback options (table_name 'foo'); > If the user has specified "not valid" for a constraint on the foreign table, there is high chance that s/he is aware of the fact that the remote table that the foreign table points to has some rows which will violet the constraint. So, +1. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company
On Wed, Aug 2, 2017 at 3:46 AM, Ashutosh Bapat <ashutosh.bapat@enterprisedb.com> wrote: > If the user has specified "not valid" for a constraint on the foreign > table, there is high chance that s/he is aware of the fact that the > remote table that the foreign table points to has some rows which will > violet the constraint. So, +1. +1 from me, too. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 2017/08/02 20:40, Robert Haas wrote: > On Wed, Aug 2, 2017 at 3:46 AM, Ashutosh Bapat > <ashutosh.bapat@enterprisedb.com> wrote: >> If the user has specified "not valid" for a constraint on the foreign >> table, there is high chance that s/he is aware of the fact that the >> remote table that the foreign table points to has some rows which will >> violet the constraint. So, +1. > > +1 from me, too. Alright, thanks. Attached is a patch. I think this could be considered a bug-fix, backpatchable to 9.6 which introduced this behavior change [1]. Thoughts? Thanks, Amit [1] http://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=f27a6b15e656 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Attachment
On Wed, Aug 2, 2017 at 9:41 PM, Amit Langote <Langote_Amit_f8@lab.ntt.co.jp> wrote: > On 2017/08/02 20:40, Robert Haas wrote: >> On Wed, Aug 2, 2017 at 3:46 AM, Ashutosh Bapat >> <ashutosh.bapat@enterprisedb.com> wrote: >>> If the user has specified "not valid" for a constraint on the foreign >>> table, there is high chance that s/he is aware of the fact that the >>> remote table that the foreign table points to has some rows which will >>> violet the constraint. So, +1. >> >> +1 from me, too. > > Alright, thanks. > > Attached is a patch. I think this could be considered a bug-fix, > backpatchable to 9.6 which introduced this behavior change [1]. I could go either way on that. It's not inconceivable somebody could be unhappy about seeing this behavior change in a minor release. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes:
> On Wed, Aug 2, 2017 at 9:41 PM, Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> Attached is a patch. I think this could be considered a bug-fix,
>> backpatchable to 9.6 which introduced this behavior change [1].
> I could go either way on that. It's not inconceivable somebody could
> be unhappy about seeing this behavior change in a minor release.
FWIW, I vote with the camp that this is a clear bug and needs to be
fixed. 9.6 broke a behavior that could be relied on before that.
We do not normally hesitate to fix regressions in minor releases.
(That's not a vote for the patch as submitted; I haven't reviewed it.
But we need to fix this.)
regards, tom lane
On Thu, Aug 3, 2017 at 12:35 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> On Wed, Aug 2, 2017 at 9:41 PM, Amit Langote >> <Langote_Amit_f8@lab.ntt.co.jp> wrote: >>> Attached is a patch. I think this could be considered a bug-fix, >>> backpatchable to 9.6 which introduced this behavior change [1]. > >> I could go either way on that. It's not inconceivable somebody could >> be unhappy about seeing this behavior change in a minor release. > > FWIW, I vote with the camp that this is a clear bug and needs to be > fixed. 9.6 broke a behavior that could be relied on before that. > We do not normally hesitate to fix regressions in minor releases. > > (That's not a vote for the patch as submitted; I haven't reviewed it. > But we need to fix this.) OK. I'm going to commit and back-patch the substantive fix with a comment change, but I'm not going to include Amit's documentation changes for now because I'm not sure they are going to be sufficiently clear. There's not a lot of context for them where he put them. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 2017/08/04 2:13, Robert Haas wrote: > On Thu, Aug 3, 2017 at 12:35 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> Robert Haas <robertmhaas@gmail.com> writes: >>> On Wed, Aug 2, 2017 at 9:41 PM, Amit Langote >>> <Langote_Amit_f8@lab.ntt.co.jp> wrote: >>>> Attached is a patch. I think this could be considered a bug-fix, >>>> backpatchable to 9.6 which introduced this behavior change [1]. >> >>> I could go either way on that. It's not inconceivable somebody could >>> be unhappy about seeing this behavior change in a minor release. >> >> FWIW, I vote with the camp that this is a clear bug and needs to be >> fixed. 9.6 broke a behavior that could be relied on before that. >> We do not normally hesitate to fix regressions in minor releases. >> >> (That's not a vote for the patch as submitted; I haven't reviewed it. >> But we need to fix this.) > > OK. I'm going to commit and back-patch the substantive fix with a > comment change, but I'm not going to include Amit's documentation > changes for now because I'm not sure they are going to be sufficiently > clear. There's not a lot of context for them where he put them. Thanks for committing the code changes. About the documentation changes, it seems that the only places where any description of NOT VALID appears is ALTER TABLE, ALTER FOREIGN TABLE, and ALTER DOMAIN references pages. Even if the CREATE (FOREIGN) TABLE syntax allows it, NOT VALID does not appear in the syntax synopsis, so it seems kind of a hidden feature. Maybe, we should fix that first (if at all). Thanks, Amit
On Thu, Aug 3, 2017 at 9:29 PM, Amit Langote <Langote_Amit_f8@lab.ntt.co.jp> wrote: > Thanks for committing the code changes. > > About the documentation changes, it seems that the only places where any > description of NOT VALID appears is ALTER TABLE, ALTER FOREIGN TABLE, and > ALTER DOMAIN references pages. Even if the CREATE (FOREIGN) TABLE syntax > allows it, NOT VALID does not appear in the syntax synopsis, so it seems > kind of a hidden feature. Maybe, we should fix that first (if at all). Yeah. If somebody wants to do a general survey of how NOT VALID is documented and improve that, cool. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company